Virtual machine migration methods, scheduling systems and related products

Abstract

This application provides a virtual machine migration method, a scheduling system, and related products, which can improve virtual machine migration efficiency. The method described above can be applied to a scheduling system. Specifically, a first server includes multiple NUMA nodes, each hosting one or more virtual machines. When the scheduling system detects that the first NUMA node on the first server cannot meet the normal operation requirements of the first virtual machine, or that the CPU and memory resources of the first NUMA node are unevenly allocated, the scheduling system will migrate one or more virtual machines from the first NUMA node to a second NUMA node on the first server.

Description

Technical Field

[0001] This application relates to the field of cloud computing, and in particular to a virtual machine migration method, scheduling system and related products. Background Technology

[0002] As the number of central processing unit (CPU) cores on servers continues to increase, the bus between the CPU and memory controller in traditional symmetric multi-processing (SMP) architectures becomes a bottleneck for system performance. Therefore, servers can adopt a non-uniform memory access (NUMA) architecture.

[0003] In a NUMA architecture, a server consists of multiple NUMA nodes. Each NUMA node typically comprises a physical CPU and its own local memory, and each NUMA node can host one or more virtual machines. In practical applications, different NUMA nodes within a server may host different virtual machines, leading to varying loads on each NUMA node. For example, some NUMA nodes may be heavily loaded, while others may be lightly loaded. Resource hotspots may appear on heavily loaded NUMA nodes, thus affecting the normal operation of the virtual machines on those nodes.

[0004] To address the aforementioned issues, current methods include cross-host migration to move virtual machines on NUMA nodes experiencing resource hotspots to other servers, thus alleviating the resource hotspot situation on those NUMA nodes. For example, when NUMA node 1 on server 1 experiences a resource hotspot, some virtual machines on NUMA node 1 can be migrated to NUMA node 2 on server 2 via the network between server 1 and server 2. However, since the network between server 1 and server 2 also needs to maintain normal business communication between them, this method will increase migration time when the amount of data being migrated is large, thereby impacting the services running on the migrated virtual machines. Summary of the Invention

[0005] This application provides a virtual machine migration method, scheduling system, and related products, which can improve virtual machine migration efficiency.

[0006] Firstly, this application provides a virtual machine migration method. This method is applied to a scheduling system. Specifically, a first server includes multiple NUMA nodes, each hosting one or more virtual machines. When the scheduling system detects that the first NUMA node on the first server cannot meet the normal operation requirements of the first virtual machine, or that the CPU and memory resources of the first NUMA node are unevenly allocated, the scheduling system will migrate one or more virtual machines from the first NUMA node to a second NUMA node on the first server.

[0007] It is understandable that when the first NUMA node cannot meet the normal operation requirements of the first virtual machine, migrating one or more virtual machines from the first NUMA node to the second NUMA node not only allows the migrated virtual machines to run normally, but also reduces the resource consumption of the first NUMA node, enabling other virtual machines on the first NUMA node to run normally. When the CPU and memory resources of the first NUMA node are unevenly distributed, migrating one or more virtual machines from the first NUMA node to the second NUMA node can balance the allocation of CPU and memory resources on the first NUMA node, reducing fragmented resources on the first NUMA node, and consequently reducing fragmented resources on the first server, thus reducing resource waste and performance degradation on the first server. Furthermore, compared to cross-host migration, migrating virtual machines between NUMA nodes within the first server has higher migration efficiency.

[0008] In one possible implementation, the scheduling system detects that the first NUMA node on the first server cannot meet the normal operation requirements of the first virtual machine, including one or more of the following situations: First, when the CPU and / or memory resources of the first NUMA node are hotspots, it is determined that the first NUMA node cannot meet the normal operation requirements of the first virtual machine. Here, a resource hotspot refers to a bottleneck in the performance of the first NUMA node due to excessive concentrated use of resources, thereby affecting the normal operation of the virtual machine on the first NUMA node. Second, when the quality of service (QoS) of the application running on the first virtual machine deteriorates, it is determined that the first NUMA node cannot meet the normal operation requirements of the first virtual machine. Here, application QoS deterioration means that the application's QoS is lower than the user's minimum QoS requirements for the application. Application QoS refers to the latency and waiting time when the virtual machine provides application services to the user.

[0009] In one possible implementation, the scheduling system migrates one or more virtual machines on a first NUMA node to a second NUMA node on a first server, including: the scheduling system migrates a second virtual machine on the first NUMA node to the second NUMA node, wherein the priority of the second virtual machine is lower than a preset priority.

[0010] The priority of a virtual machine is set by the user based on the application running on the virtual machine. Generally, a higher virtual machine priority indicates a higher requirement for the service experience of the application running on the virtual machine, meaning that the normal operation of the virtual machine needs to be prioritized. In the above implementation, the priority of the second virtual machine to be migrated is lower than the preset priority, which means that the user's requirement for the service experience of the application running on the second virtual machine is relatively low, and the user cannot perceive (or can accept) the impact of migrating the second virtual machine on the service experience. On the other hand, for the virtual machines on the first NUMA node with a priority higher than the preset priority, migrating the second virtual machine can quickly restore the normal operation of the virtual machines with a priority higher than the preset priority.

[0011] In one possible implementation, the scheduling system can migrate the second virtual machine on the first NUMA node to the second NUMA node through the following steps: the scheduling system creates a third virtual machine on the second NUMA node and establishes a local communication channel between the second virtual machine and the third virtual machine, and then migrates the state and data of the second virtual machine to the third virtual machine through the aforementioned local communication channel.

[0012] In the above implementation, the local communication channel can be, for example, a UNIX socket channel. Therefore, by using the local communication channel between the second virtual machine and the third virtual machine to migrate the second virtual machine, the consumption of network bandwidth is reduced and the migration efficiency of the second virtual machine is improved.

[0013] In one possible implementation, the second virtual machine is bound to one or more pass-through devices plugged into the first server. These pass-through devices can be devices plugged into the first server via peripheral component interconnect (PCI) slots or peripheral component interconnect express (PCIe) slots, such as network cards, hard drives, graphics processing units (GPUs), neural network processing units (NPUs), or one or more of these.

[0014] For passthrough devices (hereinafter referred to as "virtual passthrough devices") bound (also known as "mounted") to the second virtual machine, the scheduling system can migrate the virtual passthrough device to the third virtual machine through the following steps: the scheduling system unmounts the virtual passthrough device from the second virtual machine and remounts the virtual passthrough device to the third virtual machine. This allows for the migration of virtual passthrough devices without creating a new virtual passthrough device for the third virtual machine, thus improving the migration efficiency of the second virtual machine.

[0015] In one possible implementation, before migrating one or more virtual machines on the first NUMA node to the second NUMA node, the scheduling system selects the second NUMA node from the multiple NUMA nodes contained in the first server based on at least one of the following: the load of each NUMA node on the first server, the priority of the virtual machines carried on each NUMA node, and the fragmented resources existing on the first server after the migration.

[0016] By implementing the above method, the NUMA node with low load, no virtual machines with a priority higher than the preset priority, and minimal fragmented resources on the first server after migration can be selected as the second NUMA node. In this way, the migration will not cause virtual machines on the second NUMA node to malfunction, nor will it increase fragmented resources on the first server.

[0017] In one possible implementation, before migrating one or more virtual machines on the first NUMA node to the second NUMA node, the scheduling system also determines whether local migration requirements are met. Specifically, if local migration requirements are met, the scheduling system migrates one or more virtual machines from the first NUMA node to the second NUMA node. If local migration requirements are not met, the scheduling system may migrate one or more virtual machines from the first NUMA node to the NUMA node of the second server, or it may migrate one or more virtual machines from the first NUMA node to the second NUMA node.

[0018] The aforementioned local migration requirements include one or more of the following: the first server includes a NUMA node capable of supporting the normal operation of one or more virtual machines; or, the first NUMA node hosts one or more important virtual machines, wherein the important virtual machines are virtual machines with a priority greater than the preset priority; or, the QoS of the aforementioned one or more important virtual machines is degraded.

[0019] By implementing the above method, the scheduling system can prioritize virtual machine migration between NUMA nodes within the first server. Since the local migration of virtual machines is highly efficient, the migration of virtual machines can be achieved more quickly.

[0020] Secondly, this application provides a scheduling system. This scheduling system is used to manage the migration of virtual machines on a first server. Specifically, the first server includes multiple NUMA nodes, each NUMA node hosting one or more virtual machines. The scheduling system includes a first scheduling module. The first scheduling module is used to detect if the first NUMA node on the first server cannot meet the normal operation requirements of the first virtual machine, or if the CPU and memory resources of the first NUMA node are unevenly allocated, and then migrate one or more virtual machines from the first NUMA node to a second NUMA node on the first server.

[0021] In one possible implementation, the first scheduling module is used to determine that the first NUMA node cannot meet the normal operation of the first virtual machine when the CPU resources and / or memory resources of the first NUMA node are hotspots. Here, a resource hotspot refers to a bottleneck in the performance of the first NUMA node caused by excessive concentrated use of resources, which in turn affects the normal operation of the virtual machine on the first NUMA node.

[0022] In another possible implementation, the first scheduling module is used to determine that the first NUMA node cannot meet the normal operation of the first virtual machine when the QoS of the application running on the first virtual machine deteriorates. Here, the QoS deterioration of the application means that the QoS of the application is lower than the minimum QoS requirement of the user for the application. The QoS of the application refers to the latency and waiting time when the virtual machine provides application services to the user.

[0023] In one possible implementation, the first scheduling module is used to migrate the second virtual machine on the first NUMA node to the second NUMA node, wherein the priority of the second virtual machine is lower than the preset priority.

[0024] In one possible implementation, the first scheduling module is used to create a third virtual machine on the second NUMA node and establish a local communication channel between the second virtual machine and the third virtual machine. Then, through the aforementioned local communication channel, the state and data of the second virtual machine are migrated to the third virtual machine.

[0025] In one possible implementation, the second virtual machine is bound to one or more passthrough devices plugged into the first server. These passthrough devices can be devices plugged into the first server, such as network cards, hard drives, GPUs, NPUs, or one or more other types, including PCI slots or PCIe slots. For the passthrough devices bound (also referred to as "mounted") to the second virtual machine (hereinafter referred to as "virtual passthrough devices"), the first scheduling module is used to unmount the virtual passthrough devices from the second virtual machine and remount them onto the third virtual machine. This allows for the migration of virtual passthrough devices.

[0026] In one possible implementation, the first scheduling module is further configured to select a second NUMA node from the multiple NUMA nodes contained in the first server based on at least one of the following: the load of each NUMA node on the first server, the priority of the virtual machines carried on each NUMA node, and the fragmented resources existing on the first server after migration.

[0027] In one possible implementation, the first scheduling module determines whether the local migration requirement is met, and when the requirement is met, migrates one or more virtual machines on the first NUMA node to the second NUMA node. Optionally, when the local migration requirement is not met, the first scheduling module may also migrate one or more virtual machines on the first NUMA node to the second NUMA node.

[0028] The aforementioned local migration requirements include one or more of the following: the first server includes a NUMA node capable of supporting the normal operation of one or more virtual machines; or, the first NUMA node hosts one or more important virtual machines, wherein the important virtual machines are virtual machines with a priority greater than the preset priority; or, the QoS of the aforementioned one or more important virtual machines is degraded.

[0029] In one possible implementation, the scheduling system further includes a second scheduling module. When local migration requirements are not met, the second scheduling module migrates one or more virtual machines from the first NUMA node to the NUMA node of the second server.

[0030] Thirdly, this application provides a computing device. The computing device includes a processor and a memory, the processor executing instructions stored in the memory to cause the computing device to perform some or all of the methods described in the first aspect and any implementation thereof.

[0031] Fourthly, this application provides a computing device cluster. The computing device cluster includes at least one computing device, each computing device including a processor and a memory. The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, causing the computing device cluster to perform some or all of the methods described in the first aspect and any implementation thereof.

[0032] Fifthly, this application provides a computer program product containing instructions. This computer program product may be a software or program product containing instructions that can run on a computing device or be stored on any usable medium. When the computer program product is run on a computing device, it causes the computing device to perform some or all of the methods described in the first aspect and any implementation thereof.

[0033] Sixthly, this application provides a computer-readable storage medium. The computer storage medium includes computer program instructions that, when executed by a computing device, cause the computing device to perform some or all of the methods described in the first aspect and any implementation thereof. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of a virtual machine migration scenario provided in this application;

[0035] Figure 2 This is a schematic diagram of the structure of a server provided in this application;

[0036] Figure 3 This is a schematic diagram of another server structure provided in this application;

[0037] Figure 4 This is a schematic diagram illustrating a deployment method for a server and scheduling system provided in this application;

[0038] Figure 5 This is a schematic diagram illustrating another deployment method for the server and scheduling system provided in this application;

[0039] Figure 6 This is a flowchart illustrating a virtual machine migration method provided in this application;

[0040] Figure 7 This is a schematic diagram of the structure of a scheduling system provided in this application;

[0041] Figure 8 This is a schematic diagram of the structure of a computing device provided in this application;

[0042] Figure 9 This is a schematic diagram of the structure of a computing device cluster provided in this application;

[0043] Figure 10 This is a schematic diagram of another computing device cluster provided in this application. Detailed Implementation

[0044] To address the issue of low virtual machine migration efficiency, this application provides a virtual machine migration method. This method can be applied to a scheduling system. Specifically, when a first virtual machine needs to be migrated from a first NUMA node, the scheduling system can migrate the first virtual machine to a second NUMA node on the local server. This achieves a method for migrating virtual machines between NUMA nodes within a server, improving the migration efficiency of virtual machines.

[0045] The technical solution provided in this application will be described below with reference to the accompanying drawings.

[0046] See Figure 1 , Figure 1 This illustrates a virtual machine migration scenario to which this application applies. For example... Figure 1 As shown, this scenario includes server 100 and scheduling system 200. The number of servers 100 can be one or more; for ease of description later, Figure 1 The diagram shows m servers 100, namely server 1, server 2, ..., server m, where m is an integer greater than 1. A communication connection is established between the servers 100 and the scheduling system 200. This communication connection can be a wired connection, such as Ethernet, a line, or cable; or a wireless connection, such as Wi-Fi, a cellular network, or the Internet or a local area network (LAN) that simultaneously supports wired and wireless connections. This application does not limit the specific type of connection. The servers 100 and the scheduling system 200 are described below:

[0047] (1) Server 100

[0048] Server 100 may be a central server in a cloud data center, an edge server, or a local server in a local data center, and may also include terminal devices such as desktop computers and laptops, which are not limited in this application.

[0049] Server 100 adopts a NUMA architecture, which is a memory architecture designed for multi-processor computers, capable of solving memory access latency and bandwidth bottlenecks in multi-processor systems. For example... Figure 2 As shown, server 100 includes multiple NUMA nodes 110, each of which has an independent physical CPU and local memory. Typically, a NUMA node 110 contains one physical CPU, which includes one or more CPU cores, also known as processors. The processor accesses the local NUMA node memory faster than it accesses the memory of other NUMA nodes. This reduces memory access to other NUMA nodes, thereby reducing latency and bandwidth bottlenecks in remote memory access.

[0050] Optionally, server 100 may also be provided with one or more slots, such as PCI slots, PCIe slots, dual inline memory modules (DIMM) slots, universal serial bus (USB) slots, etc. Through these slots, one or more devices such as network cards, hard drives, GPUs, NPUs, etc., can be plugged into server 100 to enhance the performance of server 100. For ease of description, the devices plugged into server 100 through the slots will be referred to as "pass-through devices 120" below.

[0051] The various components on server 100 (such as NUMA node 110 and pass-through device 120) can be connected via bus 130. In other words, bus 130 can include a path for transmitting information between the various components of server 100. In specific implementations, bus 130 can include one or more of the following: PCI bus, PCIe bus, Extended Industry Standard Architecture (EISA) bus, etc. Bus 130 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 2 The character is represented by only one line, but this does not mean that server 100 has only one bus or one type of bus.

[0052] In addition, server 100 can also connect to one or more storage devices in the storage resource pool via a network. These storage devices include disks, solid-state drives (SSDs), shingled magnetic recording hard drives, optical discs, and one or more of the following: a wide area network (WAN) or a local area network (LAN). WANs include leased lines, the Internet, multi-protocol label switching (MPLS) networks, virtual private networks (VPNs), and one or more of the following: Ethernet, wireless local area networks (WLANs), virtual local area networks (VLANs), and one or more of the following: server 100 can also connect to one or more network devices in the network resource pool via the aforementioned network. These network devices include network interface cards (NICs), load balancers, switches, routers, and one or more of the following. For ease of description, the devices connected to server 100 via the aforementioned network (such as the storage devices and network devices mentioned above) will be referred to as "remote device 300".

[0053] As you can understand, the above content introduces server 100 from the perspective of the hardware layer. The following will introduce server 100 from the perspective of the software layer.

[0054] like Figure 3 As shown, the software layer of server 100 includes an operating system (hereinafter referred to as the "host operating system") installed and running on server 100. The host operating system contains a virtual machine monitor (VMM) 140. The virtual machine monitor 140 can create one or more virtual machines (VMs) 111 on the NUMA nodes 110 included in server 100, and manage the created virtual machines 111. A virtual machine 111 refers to a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment. Specifically, the virtual machine monitor 140 has one or more of the following functions: ① Virtualizing the physical CPU and local memory of the NUMA node 110 to provide a portion of the physical CPU and local memory of the NUMA node 110 (hereinafter referred to as "virtual CPU" and "virtual memory") to the virtual machine 111. ② Virtualize one or more passthrough devices 120 inserted on server 100, and provide a portion of the passthrough devices 120 (hereinafter referred to as "virtual passthrough devices") to virtual machine 111 through mounting (also known as "binding"). Here, "mounting" means that virtual machine manager 140 connects the virtual passthrough devices to virtual machine 111 by configuring the mapping relationship between the virtual passthrough devices and virtual machine 111, so that virtual machine 111 can access and use the virtual passthrough devices. ③ Virtualize one or more remote devices 300 connected to server 100, and provide a portion of the remote devices 300 (hereinafter referred to as "virtual remote devices") to virtual machine 111 through mounting. Here, "mounting" means that virtual machine manager 140 connects the virtual remote devices to virtual machine 111 by configuring the mapping relationship between the virtual remote devices and virtual machine 111, so that virtual machine 111 can access and use the virtual passthrough devices. ④ Install an operating system (hereinafter referred to as "virtual machine operating system") on virtual machine 111, wherein one or more applications can be installed and run on the virtual machine operating system. ⑤ Enable communication between different virtual machines 111. ⑥ Enable communication between virtual machine 111 and external devices (such as scheduling system 200). ⑦ Delete virtual machine 111. It is understood that the above items are merely illustrative examples of the functions that virtual machine manager 140 may have. In addition to the functions mentioned above, virtual machine manager 140 also has other functions, which will not be described in detail here for the sake of brevity.

[0055] (2) Scheduling System 200

[0056] Because the virtual machine manager 140 allocates resources to virtual machines 111 at the NUMA node 110 level, the virtual machines 111 hosted on each NUMA node 110 within the server 100 are not identical, resulting in different loads on each NUMA node 110. This can lead to the following situations:

[0057] ① Resource hotspot of NUMA node 110. Here, "resource hotspot of NUMA node 110" refers to the phenomenon where one or more resources, such as the physical CPU or local memory of NUMA node 110, become a performance bottleneck due to excessive concentrated usage. It should be understood that when NUMA node 110 experiences a resource hotspot, it will affect the normal operation of virtual machine 111 on NUMA node 110.

[0058] For example, suppose server 100 includes two NUMA nodes 110, namely NUMA node 1 and NUMA node 2. Both NUMA node 1 and NUMA node 2 have 16 physical CPU cores and 32GB of local memory. If three virtual machines 111 need to be deployed on server 100: virtual machine 1, virtual machine 2, and virtual machine 3, where virtual machine 1 and virtual machine 2 are each 8 cores and 16GB of memory, and virtual machine 3 is 12 cores and 24GB of memory, then the virtual machine manager 140 can deploy virtual machine 1 and virtual machine 2 on NUMA node 1 and virtual machine 3 on NUMA node 2. In this case, the load on NUMA node 1 is greater than the load on NUMA node 2. Moreover, for NUMA node 1, if virtual machine 1 and virtual machine 2 frequently access their respective virtual memory within a certain period of time, it will cause the memory bandwidth of NUMA node 1 to become a bottleneck, i.e., a memory hotspot of NUMA node 1.

[0059] ② The uneven distribution of CPU and memory resources on NUMA node 110 results in unallocated resources becoming fragmented resources. Fragmented resources refer to resources that cannot be effectively allocated or utilized, including one or more of CPU and memory resources. It should be understood that when there are many fragmented resources on server 100, it will cause resource waste and performance degradation on server 100.

[0060] Using the example above, suppose we continue to deploy virtual machine 4 on NUMA node 2. Virtual machine 4 has 4 cores and 4GB of memory. After deployment, the only unallocated resources on NUMA node 2 are 4GB of memory, which makes it impossible to deploy any more virtual machines on NUMA node 2. In this case, the unallocated memory will become fragmented resources.

[0061] ③ QoS degradation of virtual machine 111. QoS degradation of virtual machine 111 refers to the application's QoS being lower than the user's minimum QoS requirement for that application. Application QoS refers to the latency and waiting time when virtual machine 111 provides application services to the user. In other words, QoS degradation of virtual machine 111 will affect the applications running on it, thereby reducing the user experience.

[0062] When one or more of the above three situations occur, the scheduling system 200 can migrate one or more virtual machines 111 on the server 100 from their current NUMA node 110 to other local NUMA nodes 110, thereby adjusting the load of each NUMA node 110 within the server 100 and avoiding the occurrence of the above three situations.

[0063] Furthermore, if one or more of the above situations still exist after the migration is completed, the scheduling system 200 can also migrate one or more virtual machines 111 on the server to the NUMA node of another server 100.

[0064] In addition, when the I / O read / write speed of the aforementioned server is slow or the power consumption is high, the scheduling system 200 will also migrate one or more virtual machines 111 on the server to the NUMA node of other servers 100.

[0065] In a specific implementation, the scheduling system 200 can be deployed on a single computing device or a cluster of computing devices. These computing devices can include servers, such as central servers or edge servers in a cloud data center, or local servers in a local data center, as well as terminal devices such as desktop computers and laptops; this application does not limit the scope of these. Furthermore, when the scheduling system 200 is deployed on a single computing device, the computing device containing the scheduling system 200 can belong to the same computing device cluster as the server 100, or it can belong to a different computing device cluster. Similarly, when the scheduling system 200 is deployed on a computing device cluster, the computing device cluster containing the scheduling system 200 and the computing device cluster containing the server 100 can be the same computing device cluster or different computing device clusters.

[0066] As one possible implementation, the scheduling system 200 is deployed in a cloud data center. In this case, the scheduling system 200 can provide cloud services to users based on the basic resources (including computing resources, storage resources, and network resources) provided by the cloud data center. Accordingly, users can register an account on the cloud management platform, log in to the cloud management platform through their account, and purchase the aforementioned cloud services. After successful purchase, the scheduling system 200 can be used to schedule virtual machine migrations for one or more servers 100 owned by the user.

[0067] It is worth noting that the deployment methods of server 100 and scheduling system 200, in addition to referring to... Figure 1 Besides the separate deployment method shown, other deployment methods can also be used depending on the specific application scenario and business needs. For example, ... Figure 4 As shown, the scheduling system 200 can be deployed as a single unit with the server 100; that is, the scheduling system 200 can be deployed on the server 100 as a software module (such as a plugin or component). For example, the scheduling system 200 can logically be divided into multiple functional modules (such as the first scheduling module and the second scheduling module mentioned later). Therefore, as... Figure 5 As shown, the scheduling system 200 can also have one part (such as the first scheduling module) deployed on the server 100, and another part (such as the second scheduling module) deployed outside the server 100. For ease of description, the following embodiments use... Figure 1 The deployment method shown will be introduced.

[0068] Next, combined Figure 6 The flowchart illustrating the virtual machine migration method provides a more detailed explanation of the virtual machine migration process implemented by the scheduling system 200. For ease of description, the following example only illustrates the migration of a single virtual machine 111 (hereinafter referred to as the "first source virtual machine"), where the server 100 where the first source virtual machine resides is referred to as the "source server". Figure 6 As shown, the scheduling system 200 can migrate the first source virtual machine through the following steps.

[0069] Step 101: The scheduling system 200 obtains the running data of the source server.

[0070] The source server can be Figure 1 Any one of the m servers 100 shown. The source server's operational data is used to reflect one or more aspects related to the source server, such as its operational status, resource usage, and resource allocation. Specifically, the source server's operational data may include one or more of the following categories: source server topology data, source server performance data, and performance data of virtual machines 111 on the source server.

[0071] The topology data of the source server is used to reflect the resource allocation of the source server. For example, the topology data of the source server includes one or more of the following: ① the resources owned by the source server; wherein, the resources owned by the source server can be represented by one or more of the following information: relevant information about the CPU resources owned by the source server (such as the specifications and number of CPU cores), relevant information about the memory resources owned by the source server (such as memory size, memory bandwidth, etc.), relevant information about the pass-through device 120 mounted on the source server (such as the identifier and specifications of the pass-through device 120, etc.), and relevant information about the remote device 300 connected to the source server (such as the identifier and specifications of the remote device 300, etc.). ② Information related to the NUMA node 110 contained in the source server, such as the identifier of the NUMA node 110 and the resources owned by the NUMA node 110; wherein, the identifier of the NUMA node 110 may include the name, ID and other information that can identify the NUMA node 110; the resources owned by the NUMA node 110 can be represented by one or more of the following aspects: information related to the CPU cores contained in the physical CPU of the NUMA node 110 (such as the specifications and number of CPU cores), and information related to local memory (such as memory size, memory bandwidth, etc.). ③ Information related to the virtual machine 111 carried on the NUMA node 110, such as the identifier of virtual machine 111, the priority of virtual machine 111, and the resources owned by virtual machine 111; wherein, the identifier of virtual machine 111 may include information that can identify virtual machine 111, such as the name and ID of virtual machine 111; the priority of virtual machine 111 is set by the user based on the application running on virtual machine 111, the higher the priority of virtual machine 111, the more important it is to ensure the normal operation of virtual machine to meet the user's requirements for service experience; the resources owned by virtual machine 111 can be represented by one or more of the following aspects of information: information related to virtual CPU (such as the specifications and number of CPU cores), information related to virtual memory (such as memory size, memory bandwidth, etc.), information related to virtual passthrough devices (such as the mapping relationship between virtual passthrough devices and virtual machine 111), and information related to virtual remote devices (such as the mapping relationship between virtual remote devices and virtual machine 111).

[0072] The performance data of the source server is used to reflect the operating status and resource usage of the source server. For example, the performance data of the source server includes one or more of the following: ① The usage of each CPU core among the multiple CPU cores owned by the source server, wherein the CPU core usage can be represented by one or more parameters such as CPU utilization rate, CPU temperature, and instructions per second (IPS). ② The memory usage of each NUMA node 110 within the source server, wherein the memory usage can be represented by one or more parameters such as the size of the memory owned by the source server, the size of the used memory, the size of the free memory, memory bandwidth, memory utilization rate, cache hit rate, and cache miss rate. ③ The usage of the passthrough devices 120 mounted on the source server. Taking hard drives and network interface cards (NICs) as examples, the hard drive usage can be represented by one or more parameters such as the total capacity of the hard drive, the used capacity, the free capacity, disk utilization rate, input / output (I / O) read / write rate, and average response time. The NIC usage can be represented by one or more parameters such as the maximum bandwidth available to the NUMA node, the actual bandwidth used, bandwidth utilization rate, throughput, network latency, data transmission rate, and packet loss rate. ④ Usage of remote device 300 connected to the source server. It should be understood that the usage of remote device 300 is similar to that of pass-through device 120, and for simplicity, it will not be elaborated upon here. ⑤ Source server I / O read / write rate, i.e., the speed at which the source server transfers data between itself and storage devices (such as hard drives, disks, etc.) per unit of time. ⑥ Source server power consumption.

[0073] The performance data of virtual machine 111 on the source server is used to reflect the running status and resource usage of virtual machine 111. For example, the performance data of virtual machine 111 includes at least one or more of the following: ① The usage of the virtual CPU owned by virtual machine 111, wherein the virtual CPU usage can be represented by one or more parameters such as CPU utilization, CPU temperature, and IPS. ② The usage of virtual memory owned by virtual machine 111, wherein the virtual memory usage can be represented by one or more parameters such as virtual memory size, used memory size, free memory size, memory bandwidth, memory utilization, cache hit rate, and cache miss rate. ③ The usage of virtual passthrough devices mounted on virtual machine 111. Taking virtual hard disk and virtual network card mounted on virtual machine 111 as an example, the virtual hard disk usage can be represented by one or more parameters such as total virtual hard disk capacity, used capacity, free capacity, disk utilization, I / O read / write rate, and average response time. The virtual network card usage can be represented by one or more parameters such as maximum bandwidth, actual bandwidth used, bandwidth utilization, throughput, network latency, data transmission rate, and packet loss rate. ④ Usage of the virtual remote device mounted on virtual machine 111. It should be understood that the usage of the virtual remote device is similar to that of the virtual passthrough device, and for the sake of simplicity, it will not be described here.

[0074] In its implementation, the scheduling system 200 can obtain the runtime data of the source server in the following way: the source server is configured with runtime data collection tools, such as one or more tools like numactl, numastat, perf, libvirt, and kvmtop. These tools can collect the runtime data of server 100 and periodically send the collected runtime data of server 100 to the scheduling system 200. Thus, the scheduling system 200 can obtain the runtime data of server 100.

[0075] Step 102: The scheduling system 200 determines the migration scenario that the source server meets based on the source server's operating data.

[0076] The migration scenarios that the source server meets include one or more of the following: ① resource hotspots of one or more NUMA nodes 110; ② QoS degradation of one or more virtual machines 111; ③ unbalanced allocation of CPU and memory resources of one or more NUMA nodes 110; ④ I / O read / write rate of the source server is less than the preset read / write rate; ⑤ power of the source server is greater than the preset power.

[0077] (1) Regarding migration scenario ①, the scheduling system 200 can determine whether the source server meets the migration scenario in the following ways:

[0078] Taking a NUMA node 110 as an example, since the resources of NUMA node 110 include physical CPU and local memory, the resource hotspots of NUMA node 110 include CPU hotspots and / or memory hotspots of NUMA node 110.

[0079] As mentioned above, the source server's operational data includes the usage of each CPU core among the multiple CPU cores owned by the source server, which CPU cores are included in the physical CPU of NUMA node 110, and the usage of memory owned by NUMA node 110.

[0080] Therefore, the scheduling system 200 can determine the usage of multiple CPU cores in the physical CPU of the NUMA node 110 and the usage of memory in the NUMA node 110 based on the operating data of the source server. Then, based on the usage of multiple CPU cores in the physical CPU of the NUMA node 110, it can determine whether the CPU of the NUMA node 110 is a hotspot, and based on the usage of memory in the NUMA node 110, it can determine whether the memory of the NUMA node 110 is a hotspot.

[0081] To facilitate understanding, the following example uses CPU utilization rate to represent the CPU core usage on NUMA node 110 and memory bandwidth to represent the memory usage on NUMA node 110, to further explain how the scheduling system 200 determines whether the resources of NUMA node 110 are hotspots based on the CPU core and memory usage of NUMA node 110.

[0082] Based on the operating data of the source server, the scheduling system 200 determines the utilization rate of each CPU core in the physical CPU of the NUMA node 110, and thus determines the CPU utilization rate of the NUMA node 110. The CPU utilization rate of the NUMA node 110 is the average of the utilization rates of all CPU cores in its physical CPU. When the CPU utilization rate of the NUMA node 110 is greater than a preset CPU utilization rate, the scheduling system 200 identifies the NUMA node 110 as a CPU hotspot. Conversely, when the CPU utilization rate of the NUMA node 110 is less than or equal to the preset CPU utilization rate, the scheduling system 200 determines that the NUMA node 110's CPU does not exhibit a hotspot.

[0083] The scheduling system 200 also determines the memory bandwidth of the NUMA node 110 based on the operating data of the server 100. When the memory bandwidth of the NUMA node 110 is greater than the preset memory bandwidth, the scheduling system 200 determines that the NUMA node 110 has a memory hotspot. Conversely, when the memory bandwidth of the NUMA node 110 is less than or equal to the preset memory bandwidth, the scheduling system 200 determines that the NUMA node 110 does not have a memory hotspot.

[0084] It is worth noting that the preset CPU utilization and preset memory bandwidth involved in the above steps can be set by the user or by the scheduling system 200 according to the specifications of the source server. For example, the scheduling system 200 can pre-collect the maximum CPU utilization and maximum memory bandwidth allowed by servers of the same specifications, and then set the preset CPU utilization and preset memory bandwidth according to the maximum CPU utilization and maximum memory bandwidth. The preset CPU utilization can be set to a value less than the maximum CPU utilization, and the preset memory bandwidth can be set to a value less than the maximum memory bandwidth.

[0085] It is understood that the above content is only an example of one possible implementation of the scheduling system 200 in determining whether the resources of NUMA node 110 are hotspots. In practical applications, the scheduling system 200 can also determine whether the resources of NUMA node 110 are hotspots based on other parameters that can represent the usage of CPU cores and memory of NUMA node 110 (such as CPU core IPS, memory utilization, etc.). For the sake of brevity, this will not be elaborated here.

[0086] (2) Regarding migration scenario ②, the scheduling system 200 can determine whether the source server meets the requirements of the migration scenario in the following ways:

[0087] Taking a virtual machine 111 as an example, the scheduling system 200 can obtain the performance data and priority of the virtual machine 111 based on the running data of the source server, and then determine whether the QoS of the virtual machine 111 has deteriorated based on the information obtained above.

[0088] More specifically, the scheduling system 200 is equipped with a QoS detection engine, which is a component used to detect whether the QoS of a virtual machine has degraded based on the virtual machine's performance data and priority. Therefore, the scheduling system 200 can input the performance data and priority of virtual machine 111 into the QoS detection engine, and then the QoS detection engine will determine whether the QoS of virtual machine 111 has degraded based on the performance data and priority of virtual machine 111.

[0089] As one possible implementation, a QoS detection engine includes a QoS detection model and / or QoS detection rules. The QoS detection model can be an existing industry model capable of assessing whether a virtual machine's QoS has deteriorated based on its performance data, such as a weighted scoring model, decision tree model, random forest model, or neural network model. The QoS detection rules can be user-inputted rules capable of assessing whether a virtual machine's QoS has deteriorated based on its performance data.

[0090] To facilitate understanding, the following three examples will further illustrate how the QoS detection engine determines whether the QoS of virtual machine 111 has deteriorated based on its performance data and priority:

[0091] Example 1) The QoS detection engine uses a QoS detection model to determine whether the QoS of virtual machine 111 has deteriorated.

[0092] Taking a weighted average QoS detection model as an example, after the QoS detection engine obtains the performance data of virtual machine 111, it first preprocesses the performance data (such as removing outliers and normalizing it), and then inputs the processed performance data of virtual machine 111 into the QoS detection model. Upon receiving the performance data of virtual machine 111, the QoS detection model determines the weight corresponding to the performance data of virtual machine 111, and then calculates the QoS score of virtual machine 111 based on the performance data and its corresponding weight. When the QoS score of virtual machine 111 is greater than a first preset score, it means that the QoS of virtual machine 111 is likely to be degraded. In this case, the output result of the QoS detection model is "QoS degraded" for virtual machine 111. When the QoS score of virtual machine 111 is less than or equal to the first preset score, it means that the QoS of virtual machine 111 is likely to be degraded. In this case, the output result of the QoS detection model is "QoS not degraded" for virtual machine 111. The first preset score can be set by the user or by the QoS detection engine according to the priority of virtual machine 111. The higher the priority of virtual machine 111, the smaller the value of the first preset score.

[0093] It is understandable that, in addition to the weighted average model, there are other types of QoS detection models. How other types of models calculate the QoS score of virtual machine 111 will not be discussed in detail here.

[0094] Example 2) The QoS detection engine uses QoS detection rules to determine whether the QoS of virtual machine 111 has deteriorated.

[0095] QoS detection rules include one or more judgment conditions and corresponding scores. Therefore, the QoS detection engine can determine whether the performance data of virtual machine 111 meets the judgment conditions in the QoS detection rules, and calculate the QoS score of virtual machine 111 based on the judgment result.

[0096] For example, suppose virtual machine 111 has an initial QoS score of 100, and the QoS detection rules include: if the virtual machine's CPU utilization is greater than 80%, deduct 10 points; if the virtual machine's memory utilization is greater than 90%, deduct 15 points. Based on these QoS detection rules, the QoS detection engine needs to determine whether virtual machine 111's CPU utilization is greater than 80% and its memory utilization is greater than 90%, and then deduct the corresponding points from virtual machine 111's initial QoS score. The remaining score is the QoS score of virtual machine 111. When virtual machine 111's QoS score is less than a second preset score, virtual machine 111 is determined to have degraded QoS; when virtual machine 111's QoS score is greater than or equal to the second preset score, virtual machine 111's QoS is determined to be undegraded. The second preset score can be set by the user or by the QoS detection engine based on the priority of virtual machine 111. The higher the priority of virtual machine 111, the larger the value of the second preset score.

[0097] It is understood that the above example is merely an illustrative illustration of a possible QoS detection rule. In practical applications, QoS detection rules may include other content, which is not limited in this application.

[0098] Example 3) The QoS detection engine uses the QoS detection model and QoS detection rules to determine whether the QoS of virtual machine 111 has deteriorated.

[0099] The QoS detection engine uses both a QoS detection model and QoS detection rules to determine whether the QoS of virtual machine 111 has degraded (hereinafter, the result determined by the QoS detection model is referred to as the "first result," and the result determined by the QoS detection rules is referred to as the "second result"). If the first result and the second result are inconsistent, then virtual machine 111 is determined to have degraded QoS. If the first result and the second result are consistent and both indicate that virtual machine 111 has degraded QoS, then virtual machine 111 is determined to have degraded QoS. If both the first result and the second result indicate that virtual machine 111 has not degraded QoS, then virtual machine 111 has not degraded QoS.

[0100] It is understood that the above content only introduces one possible implementation of the QoS detection engine using the QoS detection model and QoS detection rules to determine whether the QoS of virtual machine 111 has deteriorated. In addition to the above method, it can also be implemented in other ways, and this application does not limit it.

[0101] (3) Regarding migration condition ③, the scheduling system 200 can determine whether the migration scenario is met in the following ways:

[0102] As one possible implementation, the scheduling system 200 determines whether the source server meets migration condition ③ based on the resource fragmentation rate of the source server. The resource fragmentation rate of the source server is the proportion of fragmented resources to total resources. When the resource fragmentation rate of the source server is greater than a preset fragmentation rate, it indicates that the source server currently has a large amount of fragmented resources. This is caused by an imbalance in CPU and memory resources among one or more NUMA nodes on the source server; in other words, the source server meets migration condition ③. Conversely, when the resource fragmentation rate of the source server is greater than the preset fragmentation rate, it indicates that the allocation of CPU and memory resources among the NUMA nodes within the source server is relatively balanced; in other words, the source server does not meet migration condition ③.

[0103] The scheduling system 200 determines the resource fragmentation rate of the source server in the following way: Based on the operating data of the source server, the scheduling system 200 determines the load of each NUMA node 110 within the source server. The load of the NUMA node 110 reflects the resource allocation of the NUMA node 110, i.e., how many virtual machines 111 are hosted on the NUMA node 110, and how much CPU and memory resources are allocated to the virtual machines 111. Then, based on the load of each NUMA node 110 within the source server, the scheduling system 200 determines the fragmented resources existing on the source server. These fragmented resources include CPU resources and / or memory resources. Based on these fragmented resources, the scheduling system 200 determines the CPU fragmentation rate and memory fragmentation rate of the source server, thereby determining the resource fragmentation rate of the source server. The CPU fragmentation rate is the proportion of CPU fragmentation to the total CPU resources of the source server, and the memory fragmentation rate is the proportion of memory fragmentation to the total memory resources of the source server. The resource fragmentation rate of the source server is a weighted average of the CPU fragmentation rate and the memory fragmentation rate.

[0104] More specifically, the scheduling system 200 determines the fragmented resources existing on the source server based on the load of each NUMA node 110 within the source server, including the following steps: For each NUMA node 110, the scheduling system 200 determines whether there are unallocated CPU resources and memory resources on the NUMA node 110 based on its load. When there are unallocated CPU resources on the NUMA node 110, the scheduling system 200 determines whether these unallocated CPU resources can be allocated to a new virtual machine. If not, the scheduling system 200 determines that these unallocated CPU resources are fragmented resources; otherwise, it determines that these unallocated CPU resources are not fragmented resources. Similarly, when there are unallocated memory resources on the NUMA node 110, the scheduling system 200 determines whether these unallocated memory resources can be allocated to a new virtual machine. If not, the scheduling system 200 determines that these unallocated memory resources are fragmented resources; otherwise, it determines that these unallocated memory resources are not fragmented resources. Therefore, the scheduling system 200 can determine the CPU fragments and memory fragments existing on all NUMA nodes 110 within the source server, and thus determine the CPU fragments and memory fragments existing on the source server.

[0105] In another implementation, the scheduling system 200 can determine whether the source server meets migration scenario ③ based on the sum of the unallocated resources of all NUMA nodes 110 within the source server and the specifications of the virtual machine to be deployed to the source server. Specifically, if the sum of the unallocated resources of all NUMA nodes 110 within the source server is greater than the specifications of the virtual machine to be deployed, but none of the NUMA nodes 110 within the source server can support the virtual machine to be deployed, then the source server is determined to meet migration scenario ③. Conversely, if the sum of the unallocated resources of all NUMA nodes 110 within the source server is greater than the specifications of the virtual machine to be deployed, and there are NUMA nodes 110 within the source server capable of supporting the virtual machine to be deployed, then the source server is determined not to meet migration scenario ③.

[0106] (4) Regarding migration scenarios ④ and ⑤, the scheduling system 200 can determine whether the above migration scenarios are met in the following ways:

[0107] The scheduling system 200 obtains the I / O read / write rate and power of the source server based on its operating data, and then compares the source server's I / O read / write rate and power with preset read / write rate and preset power, respectively. Based on the comparison results, it determines whether migration scenario ④ and migration scenario ⑤ are met. The preset read / write rate and preset power can be set by the user or by the scheduling system 200 according to the specifications of the source server and business requirements; this application does not limit this setting.

[0108] Step 103: The scheduling system 200 determines the first source virtual machine on the source server.

[0109] The scheduling system 200 determines the first source virtual machine based on the migration scenario satisfied by the source server. Specifically, when the source server satisfies migration scenario ①, the first source virtual machine is virtual machine 111 on NUMA node 110 where resource hotspots occur; when the source server satisfies migration scenario ②, the first source virtual machine is virtual machine 111 on NUMA node 110 where the virtual machine with degraded QoS resides; when the source server satisfies migration scenario ③, the first source virtual machine is virtual machine 111 on NUMA node 110 where fragmented resources exist; when the source server satisfies migration scenario ④ or migration scenario ⑤, the first source virtual machine is virtual machine 111 on NUMA node 110 with high load and a priority lower than a preset priority. The preset priority can be set by the user or by the scheduling system 200 in combination with the application scenario and business requirements.

[0110] More specifically, the scheduling system 200 can determine the first source virtual machine through the following sub-steps:

[0111] Step 103a: The scheduling system 200 selects one NUMA node 110 (hereinafter referred to as "first source NUMA node") from the multiple NUMA nodes 110 contained in the source server according to the migration scenario satisfied by the source server.

[0112] When the source server meets migration scenario ①, the scheduling system 200 identifies one or more NUMA nodes 110 in the source server that exhibit resource hotspots. If there is only one NUMA node 110 in the source server exhibiting a resource hotspot, the scheduling system 200 determines that NUMA node as the first source NUMA node. If there are multiple NUMA nodes 110 in the source server exhibiting resource hotspots, the scheduling system 200 can randomly select one NUMA node 110 from these multiple NUMA nodes 110 as the first source NUMA node, or determine the first source NUMA node based on the load of the multiple NUMA nodes 110, wherein the load of the first source NUMA node is greater than the load of the other NUMA nodes 110.

[0113] When the source server meets migration scenario ②, the scheduling system 200 determines one or more NUMA nodes 110 in the source server that host QoS-degraded virtual machines 111. If only one NUMA node 110 in the source server hosts QoS-degraded virtual machines 111, the scheduling system 200 determines that NUMA node as the first source NUMA node. If multiple NUMA nodes 110 in the source server host QoS-degraded virtual machines 111, the scheduling system 200 can randomly select one NUMA node 110 from the multiple NUMA nodes 110 as the first source NUMA node, or determine the first source NUMA node based on the load of the multiple NUMA nodes 110, wherein the load of the first source NUMA node is greater than the load of the other NUMA nodes 110.

[0114] When the source server meets migration scenario ③, the scheduling system 200 determines one or more NUMA nodes 110 in the source server that have fragmented resources. If only one NUMA node 110 in the source server has fragmented resources, the scheduling system 200 determines that NUMA node as the first source NUMA node. If multiple NUMA nodes 110 in the server 100 have fragmented resources, the scheduling system 200 can randomly select one NUMA node 110 from the multiple NUMA nodes 110 as the first source NUMA node, or determine the first source NUMA node based on the load of the multiple NUMA nodes 110, wherein the load of the first source NUMA node is greater than the load of the other NUMA nodes 110.

[0115] Step 103b: The scheduling system 200 determines the first source virtual machine from one or more virtual machines 111 contained in the first source NUMA node.

[0116] In its specific implementation, the scheduling system 200 can determine the first source virtual machine through one or more of the following methods:

[0117] Method ①: The scheduling system 200 randomly selects one virtual machine 111 from one or more virtual machines 111 contained in the first source NUMA node as the first source virtual machine.

[0118] Method ②: The scheduling system 200 determines the first source virtual machine based on the priority of one or more virtual machines 111 contained in the first source NUMA node, wherein the first source virtual machine is a virtual machine 111 with a priority lower than a preset priority. In this way, the operation of the higher-priority virtual machines 111 on the first source NUMA node can be guaranteed.

[0119] Method ③: When the resources of the first source NUMA node are hot, the scheduling system 200 can determine the first source virtual machine from one or more virtual machines 111 contained in the first source NUMA node based on the usage of hot resources on the first source NUMA node, wherein the first source virtual machine uses more hot resources than other virtual machines 111. In this way, the resource hotspot situation of the first source NUMA node can be eliminated to a greater extent.

[0120] For example, assuming the first source NUMA node is a CPU hotspot, the first source virtual machine could be virtual machine 111 with the highest CPU utilization on that node. Migrating this virtual machine from the first source NUMA node can reduce the CPU resource usage of that node, thus eliminating the CPU hotspot. Similarly, assuming the first source NUMA node is a memory hotspot, the first source virtual machine could be virtual machine 111 with the highest memory bandwidth usage on that node. Migrating this virtual machine from the first source NUMA node can reduce the memory resource usage, thus eliminating the memory hotspot.

[0121] Method 4: The scheduling system 200 determines the first source virtual machine from one or more virtual machines 111 contained in the first source NUMA node based on the load of each NUMA node 110 in the source server. The resource fragmentation rate of the source server corresponding to the first source virtual machine after migrating it from the first source NUMA node is less than the resource fragmentation rate of the source server corresponding to the other virtual machines after migrating them from the first source NUMA node.

[0122] Method 5: The scheduling system 200 determines the first source virtual machine based on whether one or more virtual machines 111 contained in the first source NUMA node have virtual passthrough devices mounted. The first source virtual machine is the virtual machine 111 that has not mounted a virtual passthrough device. In this way, it can be ensured that the first source virtual machine can be migrated to other servers 100.

[0123] It is understood that the methods ① to ⑤ mentioned above are only a few ways that the scheduling system 200 may use to determine the first source virtual machine from one or more virtual machines 111 contained in the first source NUMA node. In practical applications, the scheduling system 200 may also use other methods to determine the first source virtual machine in combination with specific application scenarios and business requirements. This application does not limit this.

[0124] It's worth noting that the first source virtual machine in this step can be a complete virtual machine 111, or a part of virtual machine 111 (i.e., the part deployed on the first source NUMA node). Specifically, a virtual machine 111 can be deployed independently on a NUMA node 110. For example, virtual machines 1, 2, 3, and 4 in the previous example are all deployed independently on a single NUMA node 110. In this case, the first source virtual machine is a complete virtual machine 111. Alternatively, a virtual machine 111 can be deployed on multiple NUMA nodes, meaning that multiple NUMA nodes 110 allocate resources to the virtual machine. In this case, the first source virtual machine is a part of virtual machine 111. Continuing with the example from the previous text, suppose server 100 also includes NUMA node 3 and NUMA node 4, with NUMA node 3 and NUMA node 4 having 16 physical CPU cores and 32GB of local memory, respectively. If virtual machine 5 needs to be deployed on server 100, with virtual machine 5 having 10 cores and 10GB of memory, then NUMA node 3 and NUMA node 4 can each provide virtual machine 5 with 5 cores and 5GB of memory. In other words, virtual machine 5 is deployed on NUMA node 3 and NUMA node 4, and the first source virtual machine can be the part deployed on NUMA node 3 or the part deployed on NUMA node 4.

[0125] Step 104: The scheduling system 200 determines whether the local migration conditions are met. If yes, proceed to step 105; otherwise, proceed to step 108.

[0126] Local migration conditions include one or more of the following: ① The source server has NUMA nodes 110 that meet the specifications of the first source virtual machine. ② The source server does not meet migration scenario ④ and migration scenario ⑤.

[0127] In its implementation, the scheduling system 200 can determine whether the local migration conditions are met through the following steps:

[0128] Regarding local migration condition ①, the scheduling system 200 can obtain the specifications of the first source virtual machine and the load of each NUMA node 110 within the source server based on the source server's operating data. The specifications of the first source virtual machine include the number of CPU cores and memory size allocated to it by the first source NUMA nodes. Then, based on the specifications of the first source virtual machine and the load of each NUMA node 110 within the source server, it determines whether there are NUMA nodes within the source server that meet the specifications of the first source virtual machine. Specifically, if the unallocated resources on one or more NUMA nodes 110 within the source server are greater than the specifications of the first source virtual machine, then it is determined whether there are NUMA nodes 110 within the source server that meet the specifications of the first source virtual machine; conversely, if the unallocated resources on all NUMA nodes 110 within the source server are less than the specifications of the first source virtual machine, then it is determined that there are no NUMA nodes 110 within the source server that meet the specifications of the first source virtual machine.

[0129] Regarding local migration condition ②, the scheduling system 200 can obtain the I / O read / write rate and power of the source server based on the source server's operating data, and then compare the source server's I / O read / write rate and power with the preset read / write rate and preset power respectively, and determine whether migration scenario ④ and migration scenario ⑤ are met based on the comparison results.

[0130] It is understandable that when the local migration conditions are met, it means that the source server supports the migration of the first source virtual machine between local NUMA nodes 110. Therefore, the scheduling system 200 can then execute step 105 to achieve the local migration of the first source virtual machine. Conversely, when the local migration conditions are not met, it means that the source server does not support the migration of the first source virtual machine between local NUMA nodes 110. Therefore, the scheduling system 200 needs to execute step 108 to migrate the first source virtual machine to the NUMA node 110 of another server 100.

[0131] Step 105: The scheduling system 200 determines whether the local migration requirement is met. If yes, proceed to step 106; otherwise, proceed to either step 106 or step 108.

[0132] The local migration requirements include one or more of the following: ① The source server has a NUMA node 110 capable of supporting the normal operation of the first source virtual machine. Normal operation here means that the QoS of the first source virtual machine will not degrade after migration. ② The first source NUMA node hosts one or more important virtual machines. Important virtual machines here refer to virtual machines 111 with a priority higher than the preset priority. ③ QoS degradation of one or more important virtual machines on the first source NUMA node.

[0133] In its implementation, the scheduling system 200 can determine whether the local migration requirement is met through the following steps:

[0134] Regarding the local migration requirement①, the scheduling system 200 can determine the specifications of the first source virtual machine and the load of each NUMA node 110 in the source server based on the operating data of the source server. Then, based on the specifications of the first source virtual machine and the load of each NUMA node 110 in the source server, it can determine whether there are NUMA nodes 110 in the source server that can meet the normal operation of the first source virtual machine.

[0135] As one possible implementation, the scheduling system 200 is configured with a prediction model. This prediction model can be an existing industry model that can predict whether a NUMA node can still support a new virtual machine and ensure its normal operation based on the NUMA node's load. Examples include long short-term memory (LSTM) models, reinforcement learning models, and regression models. Therefore, the scheduling system 200 can input the specifications of the first source virtual machine and the load of each NUMA node 110 in the source server into the prediction model. Then, the prediction model will predict which NUMA nodes can support the normal operation of the first source virtual machine based on the specifications of the first source virtual machine and the load of each NUMA node 110 in the source server.

[0136] Regarding local migration requirements ② and ③, the scheduling system 200 can determine the priority of one or more virtual machines 111 contained in the first source NUMA node based on the source server's operating data, and then determine whether the first source NUMA node carries an important virtual machine, and whether the priority of the first virtual machine is greater than the preset priority.

[0137] It is understandable that when the local migration requirement is met, it indicates that the QoS of important virtual machines on the first source NUMA node has deteriorated or is about to deteriorate. In this case, the scheduling system 200 can execute step 106 below to migrate the first source virtual machine to another local NUMA node 110. Because this process has high migration efficiency, the normal operation of important virtual machines on the first source NUMA node can be quickly restored. Conversely, when the local migration requirement is not met, it indicates that the migration of the first source virtual machine has little impact on other virtual machines 111 on the first source NUMA node. In this case, the scheduling system 200 can execute step 106 below to migrate the first source virtual machine to another local NUMA node 110, or it can execute step 108 below to migrate the first source virtual machine to the NUMA node 110 of another server 100.

[0138] Step 106: The scheduling system 200 migrates the first source virtual machine to the first target NUMA node of the source server.

[0139] Specifically, the scheduling system 200 can achieve the local migration of the first source virtual machine through the following sub-steps:

[0140] Step 106a: The scheduling system 200 determines the first target NUMA node.

[0141] The scheduling system 200 determines the first target virtual machine that meets the specification requirements of the first source virtual machine from the multiple NUMA nodes 110 contained in the source server, based on the specifications of the first source virtual machine.

[0142] When there exists a NUMA node 110 (hereinafter referred to as a "candidate NUMA node") that meets the specifications of the first source virtual machine but is different from the first source NUMA node, the scheduling system 200 can further determine the first target NUMA node from the multiple candidate NUMA nodes based on the migration scenario satisfied by the source server. Specifically,

[0143] If the source server 100 meets migration scenario ① or migration scenario ②, the scheduling system 200 can determine the first target NUMA node in any of the following ways:

[0144] Method ①: The scheduling system 200 determines the first target NUMA node based on the load of multiple candidate NUMA nodes, wherein the load of the first target NUMA node is less than the load of the other candidate NUMA nodes.

[0145] Method ②: The scheduling system 200 determines the first target NUMA node based on the priority of the virtual machines 111 carried on multiple candidate NUMA nodes, wherein the first target NUMA node does not carry any important virtual machines.

[0146] Method 3: The scheduling system 200 determines the first target NUMA node based on the load of multiple candidate NUMA nodes and the priority of the virtual machines 111 carried on the multiple candidate NUMA nodes. Specifically, the scheduling system 200 first determines the candidate NUMA node with the lowest load based on the load of multiple candidate NUMA nodes. Then, it predicts whether the resources of the candidate NUMA node will be hotspots after the first source virtual machine is migrated to the candidate NUMA node. If not, the first target NUMA node is determined to be the aforementioned candidate NUMA node with the lowest load. If so, the scheduling system 200 determines the first target NUMA node based on the priority of the virtual machines 111 carried on the aforementioned multiple candidate NUMA nodes. The first target NUMA node does not carry any important virtual machines.

[0147] If the source server meets migration scenario ③, the scheduling system 200 predicts the resource fragmentation rate of the source server after the first source virtual machine is migrated to different candidate NUMA nodes, and then determines the first target NUMA node based on the prediction results. The resource fragmentation rate corresponding to migrating the first source virtual machine to the first target NUMA node is less than the resource fragmentation rate corresponding to migrating the first source virtual machine to other candidate NUMA nodes. It should be noted that the method for calculating the resource fragmentation rate of the source server here can refer to the method for calculating the resource fragmentation rate of the source server in step 102 above; for simplicity, it will not be elaborated further here.

[0148] Step 106b: The scheduling system 200 creates the first target virtual machine on the first target NUMA node.

[0149] The scheduling system 200 sends a virtual machine creation instruction to the virtual machine manager 140 in the source server. The virtual machine creation instruction includes the specifications of the first source virtual machine. Then, the virtual machine manager 140 in the source server creates a new virtual machine 111 (i.e., the "first target virtual machine") on the first target NUMA node according to the specifications of the first source virtual machine carried in the virtual machine creation instruction. The specifications of the first target virtual machine are consistent with the specifications of the first source virtual machine.

[0150] Step 106c: The scheduling system 200 establishes a communication channel between the first source virtual machine and the first target virtual machine.

[0151] Specifically, the scheduling system 200 sends a channel establishment command to the virtual machine manager 140 of the source server. The channel establishment command includes the identifier of the first source virtual machine and the identifier of the first target virtual machine. The identifier of the first source virtual machine may include one or more unique identifiers, such as its name and ID. Similarly, the identifier of the first target virtual machine may include one or more unique identifiers, such as its name and ID. Subsequently, the virtual machine manager 140 of the source server establishes a communication channel between the first source virtual machine and the first target virtual machine based on the identifiers of the first source virtual machine and the first target virtual machine carried in the channel establishment command.

[0152] For example, the communication channel between the first source virtual machine and the first target virtual machine can be a UNIX socket channel. A UNIX socket channel is a communication channel based on an inter-process communication mechanism, enabling communication between different processes on the same server. Specifically, the virtual machine manager 140 of the source server can establish a UNIX socket channel between the first source virtual machine and the first target virtual machine through the following steps: creating a UNIX socket file and configuring its access permissions and path so that both the first source virtual machine and the first target virtual machine can access the aforementioned UNIX socket file; configuring the management process (i.e., qemu) of the first source virtual machine so that it connects to the aforementioned UNIX socket file; and configuring the management process of the first target virtual machine so that it connects to the aforementioned UNIX socket file. Thus, the management processes of the first source virtual machine and the first target virtual machine connect to the same UNIX socket file, thereby establishing a UNIX socket channel between the first source virtual machine and the first target virtual machine.

[0153] It is worth noting that the communication channel between the first source virtual machine and the first target virtual machine can be a UNIX socket channel, or other types of channels, such as a remote direct memory access (RDMA) channel. This application does not make any specific restrictions on this.

[0154] In step 106e, the scheduling system 200 migrates the state and data of the first source virtual machine to the first target virtual machine through the communication channel between the first source virtual machine and the first target virtual machine.

[0155] The scheduling system 200 sends a virtual machine migration instruction to the virtual machine manager 140 of the source server. The virtual machine migration instruction includes the identifier of the first source virtual machine and the identifier of the first target virtual machine. Then, the virtual machine manager 140 of the source server determines the first source virtual machine and the first target virtual machine based on the identifier of the first source virtual machine and the identifier of the first target virtual machine carried in the virtual machine migration instruction, and then sends the status and data of the first source virtual machine to the first target virtual machine through the communication channel between the first source virtual machine and the first target virtual machine.

[0156] More specifically, the virtual machine manager 140 of the source server can migrate the state and data of the first source virtual machine to the first target virtual machine using any of the following methods:

[0157] Method ①: Pre-copy method

[0158] The virtual machine manager 140 on the source server first iteratively copies the memory of the first source virtual machine to the first target virtual machine through the channel between the first source virtual machine and the first target virtual machine. Specifically, in the first round of copying, all current memory pages of the first source virtual machine are copied to the first target virtual machine. Since the first source virtual machine is still running during the memory copying process, some memory pages are modified (hereinafter referred to as "dirty pages"). Therefore, in each subsequent round of copying, the dirty pages generated in the previous round of memory copying are copied to the first target virtual machine until the number of newly generated dirty pages is lower than a set threshold. After that, the first source virtual machine is paused, and the remaining dirty pages and the virtual CPU state of the first source virtual machine are copied to the first target virtual machine all at once.

[0159] Next, if a virtual passthrough device is mounted on the first source virtual machine, the virtual machine manager 140 of the source server will unmount the virtual passthrough device from the first source virtual machine (i.e., remove the mapping relationship between the virtual passthrough device and the first source virtual machine) and remount the virtual passthrough device onto the first target virtual machine (i.e., establish the mapping relationship between the virtual passthrough device and the first target virtual machine so that the first target virtual machine can use the virtual passthrough device). Similarly, if a virtual remote device is mounted on the first source virtual machine, the virtual machine manager 140 of the source server will unmount the virtual remote device from the first source virtual machine (i.e., remove the mapping relationship between the virtual remote device and the first source virtual machine) and remount the virtual remote device onto the first target virtual machine (i.e., establish the mapping relationship between the virtual remote device and the first target virtual machine so that the first target virtual machine can use the virtual remote device).

[0160] Subsequently, the virtual machine manager 140 of the source server exchanges management data between the first source virtual machine and the first target virtual machine through the channel between the first source virtual machine and the first target virtual machine. The management data of the virtual machine 111 refers to information used to configure, run and maintain the virtual machine 111, such as one or more of the virtual machine 111's ID, name and so on.

[0161] Finally, the source server's virtual machine manager 140 starts the first target virtual machine.

[0162] Method ②: Post-copy method

[0163] The virtual machine manager 140 of the source server first suspends the first source virtual machine, and then copies the current virtual CPU state and some memory pages (such as the memory pages required before suspending) of the first source virtual machine to the first target virtual machine through the communication channel between the first source virtual machine and the first target virtual machine, and exchanges management data between the first source virtual machine and the first target virtual machine. In addition, if a virtual passthrough device and / or a virtual remote device are mounted on the first source virtual machine, the virtual machine manager 140 of the source server will also unmount the aforementioned virtual passthrough device and / or virtual remote device from the first source virtual machine and remount it to the first target virtual machine.

[0164] Next, the virtual machine manager 140 of the source server starts the first target virtual machine. Since the memory pages of the first source virtual machine have not yet been completely copied to the target virtual machine, the first target virtual machine may be interrupted during its operation due to a page fault (i.e., accessing an uncopied memory page). In this case, the first target virtual machine will request the missing memory page from the first source virtual machine through the communication channel between the two virtual machines. Correspondingly, the first source virtual machine will also send the memory page requested by the target virtual machine to the first target virtual machine through the same communication channel. This process continues until all memory pages of the first source virtual machine have been copied to the first target virtual machine.

[0165] It is understood that the above methods ① and ② are only two possible ways for the virtual machine manager 140 of the source server to migrate the state and data of the first source virtual machine to the first target virtual machine. In practical applications, the virtual machine manager 140 of the source server may also adopt other methods to migrate the state and data of the first source virtual machine to the first target virtual machine in combination with specific application scenarios and business requirements. This application does not make specific limitations on this.

[0166] Step 106f: The scheduling system 200 deletes the first source virtual machine and releases the resources occupied by the first source virtual machine.

[0167] The scheduling system 200 sends a notification message to the virtual machine manager 140 of the source server, indicating that the migration of the first source virtual machine is complete. Afterwards, the virtual machine manager 140 of the source server, based on the notification message, deletes the first source virtual machine on the first source NUMA node and releases the resources occupied by the first source virtual machine.

[0168] Step 107: The scheduling system 200 determines whether the load of each NUMA node 110 in the source server is balanced with the load of NUMA nodes 110 in other servers 100 after the migration is completed. If yes, the virtual machine migration task ends; otherwise, step 109 is executed.

[0169] After the migration of the first source virtual machine is completed, the scheduling system 200 determines the load of each NUMA node 110 within the source server. The load of the first source NUMA node after migration does not include the first source virtual machine, while the load of the first target NUMA node after migration includes the first source virtual machine. Then, the scheduling system 200 compares the load of each NUMA node 110 within the source server with the load of NUMA nodes 110 within other servers 100. If the difference between the load of each NUMA node 110 within the source server and the load of NUMA nodes 110 within other servers 100 is less than a preset difference, then the load of each NUMA node 110 within the source server is determined to be balanced with the load of NUMA nodes 110 within other servers 100. Conversely, if the difference between the load of one or more NUMA nodes 110 within the source server and the load of NUMA nodes 110 within other servers 100 is greater than a preset difference, then the load of each NUMA node 110 within the source server is determined to be unbalanced with the load of NUMA nodes 110 within other servers 100. The aforementioned preset difference can be set by the user or by the scheduling system 200 in combination with specific application scenarios and business needs; this application does not limit this.

[0170] The "other servers 100" in the above steps include one or more servers 100 other than the source server among the m servers 100. Similar to the source server, the scheduling system 200 can also obtain the running data of the other servers 100, and determine the load of each NUMA node 110 within the other servers 100 based on the obtained running data. For the sake of brevity, it will not be described in detail here.

[0171] Step 108: The scheduling system 200 migrates the first source virtual machine to the second target NUMA node of the first target server.

[0172] Since virtual passthrough devices cannot be migrated across hosts, no virtual passthrough device is mounted on the first source virtual machine in this step. Specifically, the scheduling system 200 can achieve cross-host migration of the first source virtual machine through the following sub-steps:

[0173] Step 108a: The scheduling system 200 determines the second target NUMA node.

[0174] The scheduling system 200 determines the first target server based on the operating data of one or more servers 100 (excluding the source server) among the m servers 100. Then, based on the operating data of the first target server, it determines the load of each NUMA node 110 within the first target server. Subsequently, based on the load of each NUMA node 110 within the first target server, it determines the second target NUMA node. The second target NUMA node meets the specification requirements of the first source virtual machine, and the load of the second target NUMA node is less than the load of other NUMA nodes 110 within the first target server.

[0175] More specifically, the scheduling system 200 can determine the first target server through one or more of the following methods:

[0176] Method ①: The scheduling system 200 determines the load of each NUMA node 110 within the m servers 100 (excluding the source server) based on the operating data of one or more servers 100. Then, based on the determined load of the multiple NUMA nodes 110, it determines the first target server. The first target server includes the NUMA node 110 with the lowest load.

[0177] Method ②: When the source server meets the migration scenario ④, the scheduling system 200 can determine the I / O read / write rate of one or more servers 100 (excluding the source server) based on their operating data, and then determine the first target server based on the I / O read / write rate of these servers 100. The first target server is the NUMA node 110 with the highest I / O read / write rate.

[0178] Method ③: When the source server meets the migration scenario ⑤, the scheduling system 200 can determine the power of one or more servers 100 (excluding the source server) based on their operating data, and then determine the first target server based on the power of these servers 100. The first target server is the NUMA node 110 with the lowest power.

[0179] Step 108b: The scheduling system 200 creates a second target virtual machine on the second target NUMA node.

[0180] The scheduling system 200 sends a virtual machine creation instruction to the virtual machine manager 140 of the first target server. The virtual machine creation instruction includes the specifications of the first source virtual machine. Subsequently, the virtual machine manager 140 of the first target server creates a new virtual machine 111 (hereinafter referred to as the "second target virtual machine") on the second target NUMA node according to the specifications of the first source virtual machine carried in the virtual machine creation instruction. The specifications of the second target virtual machine are consistent with those of the first source virtual machine.

[0181] In step 108c, the scheduling system 200 migrates the state and data of the first source virtual machine to the second target virtual machine.

[0182] The scheduling system 200 sends a virtual machine migration instruction to the virtual machine manager 140 of the source server. This instruction includes the identifier of the first source virtual machine and the identifier of the second target virtual machine. Subsequently, the virtual machine manager 140 of the source server, based on the aforementioned migration instruction, migrates the state and data of the first source virtual machine to the second target virtual machine via the network between the source server and the first target server.

[0183] More specifically, the virtual machine manager 140 of the source server can migrate the state and data of the first source virtual machine to the second target virtual machine using any of the following methods:

[0184] Method ①: Pre-copy method

[0185] The virtual machine manager 140 of the source server first sends the memory of the first source virtual machine to the virtual machine manager 140 of the first target server iteratively through the network between the source server and the first target server, and then iteratively copies it to the second target virtual machine. Specifically, in the first round of copying, all the current memory pages of the first source virtual machine are copied to the second target virtual machine. In each subsequent round of copying, the dirty pages generated in the previous round of memory copying are copied to the second target virtual machine until the number of newly generated dirty pages is lower than a set threshold. Then, the first source virtual machine is paused, and the remaining dirty pages and the virtual CPU state of the first source virtual machine are copied to the second target virtual machine all at once.

[0186] Next, if a virtual remote device is mounted on the first source virtual machine, the virtual machine manager 140 of the source server will send the relevant information of the virtual remote device to the virtual machine manager 140 of the first target server through the network between the source server and the first target server 100. After that, the virtual machine manager of the source server will unmount the virtual remote device from the first source virtual machine, and the virtual machine manager of the first target server will mount the virtual remote device to the second target virtual machine according to the relevant information of the virtual remote device.

[0187] Subsequently, the virtual machine manager 140 of the source server and the virtual machine manager 140 of the first target server exchange management data of the first source virtual machine and the second target virtual machine through the network between the source server and the first target server.

[0188] Finally, the virtual machine manager 140 of the first target server starts the second target virtual machine.

[0189] Method ②, Post-copy method

[0190] The virtual machine manager 140 of the source server first pauses the first source virtual machine, and then sends the current virtual CPU state and some memory pages (such as the memory pages required before the pause) of the first source virtual machine to the virtual machine manager 140 of the first target server through the network between the source server and the first target server, and then copies them to the second target virtual machine.

[0191] The virtual machine manager 140 of the source server also exchanges management data of the first source virtual machine and the second target virtual machine with the virtual machine manager 140 of the first target server through the network between the source server and the first target server.

[0192] In addition, if a virtual remote device is mounted on the first source virtual machine, the virtual machine manager 140 of the source server will send the relevant information of the virtual remote device to the virtual machine manager 140 of the first target server through the network between the source server and the first target server 100. After that, the virtual machine manager of the source server will unload the virtual remote device from the first source virtual machine, and the virtual machine manager of the first target server will mount the virtual remote device to the second target virtual machine according to the relevant information of the virtual remote device.

[0193] Next, the virtual machine manager 140 of the first target server starts the second target virtual machine. Since the memory pages of the first source virtual machine have not yet been completely copied to the target virtual machine, an interruption may be triggered during the operation of the second target virtual machine due to a page fault. In this case, the second target virtual machine will request the missing memory pages from the first source virtual machine through the network between the source server and the first target server. Correspondingly, the first source virtual machine will also send the memory pages requested by the target virtual machine to the second target virtual machine through the network between the source server and the first target server, and so on, until all the memory pages of the first source virtual machine have been copied to the second target virtual machine.

[0194] It is understood that the above methods ① and ② are only two possible ways for the virtual machine manager 140 of the source server to migrate the state and data of the first source virtual machine to the second target virtual machine. In practical applications, the virtual machine manager 140 of the source server may also adopt other methods to migrate the state and data of the first source virtual machine to the second target virtual machine in combination with specific application scenarios and business requirements. This application does not make specific limitations on this.

[0195] In step 108d, the scheduling system 200 deletes the first source virtual machine and releases the resources occupied by the first source virtual machine.

[0196] Specifically, the scheduling system 200 sends a notification message to the virtual machine manager 140A, which indicates that the migration of the first source virtual machine is complete. After receiving the notification message from the scheduling system 200, the virtual machine manager 140A will delete the first source virtual machine on the first source NUMA node and release the resources occupied by the first source virtual machine.

[0197] Step 109: The scheduling system 200 migrates the second source virtual machine on the source server to the third target NUMA node on the second target server.

[0198] Specifically, the scheduling system 200 can achieve cross-host migration of the second source virtual machine through the following steps:

[0199] Step 109a: The scheduling system 200 determines the second source virtual machine.

[0200] The scheduling system 200 determines the load of each NUMA node within the source server based on the source server's operating data. Then, based on the load of each NUMA node within the source server, it determines a second source NUMA node, which is the NUMA node with the highest load among the multiple NUMA nodes 110 contained in the source server. Next, the scheduling system 200 determines the priority of the virtual machines 111 carried on the second source NUMA node based on the source server's operating data. Then, based on the priority of the virtual machines 111 carried on the second source NUMA node, it determines a second source virtual machine, which is a virtual machine 111 with a priority lower than a preset priority.

[0201] Step 109b: The scheduling system 200 determines the third target NUMA node.

[0202] The scheduling system 200 determines the load of each NUMA node 110 within the m servers 100 based on the operating data of one or more servers 100 excluding the source server. Then, based on the determined load of the multiple NUMA nodes 110, it determines the second target server, and then determines the third target NUMA node. The second target server includes the third target NUMA node, which is the NUMA node 110 with the lowest load among the multiple NUMA nodes 110 determined above.

[0203] Step 109c: The scheduling system 200 creates a third target virtual machine on the third target NUMA node.

[0204] In step 109d, the scheduling system 200 migrates the state and data of the second source virtual machine to the third target virtual machine.

[0205] In step 109e, the scheduling system 200 deletes the second source virtual machine and releases the resources occupied by the second source virtual machine.

[0206] It should be noted that the implementation process of steps 109c to 109e can refer to the implementation process of steps 108b to 108d above. For the sake of brevity, it will not be described in detail here.

[0207] As described in steps 101 to 109, for a single server 100, the scheduling system 200 can perform virtual machine migration operations when resource hotspots occur on the NUMA nodes 110 within that server, when the CPU and memory resources of the NUMA nodes 110 are unevenly distributed, or when the QoS of the virtual machine 111 deteriorates, thereby resolving the aforementioned problems. Furthermore, during the migration process, virtual machine migration can be prioritized between the NUMA nodes 110 within the server 100. Since local migration is faster, the resources occupied by the source virtual machine in the source NUMA node can be quickly released, thus rapidly restoring the normal operation of important virtual machines in the source NUMA node. For a cluster consisting of m servers 100, the scheduling system 200 can achieve load balancing for the entire cluster through cross-host migration of virtual machines.

[0208] The above text combined Figures 1 to 6 This application details the virtual machine migration method provided, and then, in conjunction with... Figure 7 The structure of the scheduling system 200 that executes the above method is described.

[0209] See Figure 7 , Figure 7 A schematic diagram of a scheduling system 200 is shown. Figure 7 As shown, the scheduling system 200 includes a first scheduling module 210, and optionally, the scheduling system 200 also includes a second scheduling module 220.

[0210] The first scheduling module 210 is used to execute steps 101 to 107, and the second scheduling module 220 is used to execute steps 108 and 109. It is understood that the specific details of any possible implementation of the relevant steps by each module of the scheduling system 200 can be found in the relevant content of the foregoing method embodiments, and will not be repeated here.

[0211] In specific implementations, the first scheduling module 210 and the second scheduling module 220 can be implemented in software or in hardware. For example, the implementation of the first scheduling module 210 will be described below. Similarly, the implementation of the second scheduling module 220 can refer to the implementation of the first scheduling module 210.

[0212] As an example of a software functional unit, the first scheduling module 210 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the first scheduling module 210 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0213] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0214] As an example of a hardware functional unit, the first scheduling module 210 may include at least one computing device, such as a server. The first scheduling module 210 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD can be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0215] The multiple computing devices included in the first scheduling module 210 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the first scheduling module 210 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the first scheduling module 210 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0216] It should be noted that, in other embodiments, the first scheduling module 210 and the second scheduling module 220 can be used to execute any step in the virtual machine migration method described above. The steps implemented by the first scheduling module 210 and the second scheduling module 220 can be specified based on specific application scenarios and business requirements. The scheduling system 200 achieves all its functions by implementing different steps in the virtual machine migration method through the first scheduling module 210 and the second scheduling module 220 respectively. For example, suppose the first scheduling module 210 is deployed on each of m servers 100, and the second scheduling module 220 is deployed on other computing devices outside of the m servers 100. In this case, if the first scheduling module 210 executes steps 101 to 106, rapid virtual machine migration can be achieved within the server 100. Furthermore, when the first scheduling module 210 determines through steps 104 and 107 that the server 100 does not support local migration of virtual machines, or that local migration of virtual machines would cause an imbalance in the load of the NUMA nodes 110 within the server 100, the first scheduling module 210 can report this situation to the second scheduling module 220. Next, the second scheduling module 220 will execute steps 108 or 109 to achieve global scheduling of virtual machine migration, thereby resolving the aforementioned problem and achieving load balancing across the entire server cluster. Alternatively, the second scheduling module 220 can execute steps 101 to 109, while the first scheduling module 210 only receives scheduling instructions sent by the second scheduling module 220 and communicates with the local virtual machine manager 140 according to these instructions to achieve virtual machine migration. This saves resources occupied by the first scheduling module 210 within the server 100.

[0217] This application also provides a computing device. The computing device can be a server, such as a server in a cloud data center, an edge server, or a local server in a local data center, or a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0218] See Figure 8 , Figure 8An exemplary schematic diagram of a computing device is shown. For example... Figure 8 As shown, the computing device 400 includes a bus 410, multiple processors 420, a memory 430, and a communication interface 440, wherein the processors 420, the memory 430, and the communication interface 440 communicate with each other via the bus 410. It should be understood that this application does not limit the number of memories 430 in the computing device 400; that is, the computing device 400 may include one or more memories 430, but for ease of description... Figure 8 Only one memory 430 is shown. The various parts of the computing device 400 will be described in detail below.

[0219] Bus 410 can be a PCI bus or an EISA bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 8 The bus 410 is represented by only one line, but this does not mean that the computing device 400 has only one bus or one type of bus. The bus 410 may include a path for transmitting information between various components of the computing device 400 (such as the processor 420, memory 430 and communication interface 440).

[0220] Processor 420 may include any one or more processors such as CPU, GPU, NPU, microprocessor (MP), or digital signal processor (DSP).

[0221] Memory 430 may include volatile memory, such as random access memory (RAM). Memory 430 may also include non-volatile memory, such as SSD, flash memory, read-only memory (ROM), or hard disk drive (HDD).

[0222] The memory 430 stores executable code. The processor 420 executes the code stored in the memory 430 to implement the functions of the first scheduling module 210 and the second scheduling module 220, respectively, thereby implementing the virtual machine migration method described above. That is to say, the memory 430 stores instructions for executing the virtual machine migration method described above.

[0223] The communication interface 440 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 400 and other devices or communication networks. For example, the computing device 400 can communicate with the server 100 through the communication interface 440 to enable the migration of virtual machines on the server 100.

[0224] It should be understood that the computing device 400 provided according to this application can correspond to the scheduling system 200 described above, and the functions of each component in the computing device 400 are respectively for implementing... Figure 6 For the sake of brevity, the corresponding processes for each step in the method shown will not be elaborated here.

[0225] This application also provides a computing device cluster. The computing device cluster includes at least one computing device, which may be a server, such as a central server in a cloud data center, an edge server, or a local server in a local data center. In some embodiments, the computing device in the computing device cluster may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0226] See Figure 9 , Figure 9 An exemplary schematic diagram of a computing device cluster is shown. For example... Figure 9 As shown, the computing device cluster includes at least one computing device 400. The memory 430 of one or more computing devices 400 in the computing device cluster may store the same instructions for executing the virtual machine migration method described above.

[0227] In some possible implementations, the memory 430 of one or more computing devices 400 in the computing device cluster may also store partial instructions for executing the virtual machine migration method described above. In other words, a combination of one or more computing devices 400 can jointly execute the instructions of the virtual machine migration method described above.

[0228] It should be noted that the memories 430 of different computing devices 400 in the computing device cluster can store different instructions, which are used to execute some functions of the scheduling system 200. That is, the instructions stored in the memories 430 of different computing devices 400 can implement the functions of at least one of the first scheduling module 210 and the second scheduling module 220.

[0229] In some possible implementations, one or more computing devices 400 in a computing device cluster can be connected via a network, which can be a wide area network or a local area network, etc. Figure 10 One possible implementation is shown, such as Figure 10As shown, computing devices 400A, 400B, 400C, and 400D are connected via a network. Specifically, they are connected to the network through communication interfaces in computing devices 400A, 400B, 400C, and 400D. In this implementation, the memory 430 in computing devices 400A, 400B, and 400C stores instructions for implementing the functions of the first scheduling module 210, and the memory 430 in computing device 400D stores instructions for implementing the functions of the second scheduling module 220.

[0230] Figure 10 The connection method between computing devices 400 in the computing device cluster shown can be based on the fact that the first scheduling module 210 is deployed on computing devices 400A, 400B, and 400C to achieve local fast decision-making (i.e., fast migration of virtual machines between NUMA nodes of local computing devices), and the second scheduling module 220 is deployed on computing device 400D to be aware of global resources (i.e., the resources of NUMA nodes on computing devices 400A, 400B, and 400C), thereby achieving global scheduling to achieve load balancing of global resources.

[0231] It should be understood that Figure 10 The functions of one or more of the computing devices 400A, 400B, 400C and 400D shown can also be performed by multiple computing devices 400.

[0232] It should also be understood that the computing device cluster provided according to this application can correspond to the scheduling system 200 described above, and the functions of each component in the computing device cluster are respectively for implementing Figure 6 For the sake of brevity, the corresponding processes for each step in the method shown will not be elaborated here.

[0233] This application also provides a computer program product containing instructions. This computer program product may be a software or program product containing instructions capable of running on a computing device or stored on any available medium. When the computer program product is run on a computing device, it causes the computing device to perform the virtual machine migration method described above.

[0234] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives). The computer-readable storage medium includes instructions that instruct the computing device to perform the virtual machine migration method described above.

[0235] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A virtual machine migration method, characterized in that, The first server consists of multiple non-uniform memory access NUMA nodes, each of which hosts one or more virtual machines; The method includes: It was detected that the first NUMA node on the first server could not meet the normal operation of the first virtual machine, or that the CPU and memory resources of the first NUMA node were unevenly allocated. Migrate one or more virtual machines on the first NUMA node to the second NUMA node on the first server.

2. The method according to claim 1, characterized in that, The detection that the first NUMA node on the first server cannot meet the normal operation requirements of the first virtual machine includes: When the CPU and / or memory resources of the first NUMA node become hotspots, it is determined that the first NUMA node cannot meet the normal operation requirements of the first virtual machine.

3. The method according to claim 1, characterized in that, The detection that the first NUMA node on the first server cannot meet the normal operation requirements of the first virtual machine includes: When the QoS of the application running on the first virtual machine deteriorates, it is determined that the first NUMA node cannot meet the normal operation requirements of the first virtual machine.

4. The method according to any one of claims 1 to 3, characterized in that, Migrating one or more virtual machines on the first NUMA node to a second NUMA node on the first server includes: Migrate the second virtual machine on the first NUMA node to the second NUMA node, wherein the priority of the second virtual machine is lower than the preset priority.

5. The method according to claim 4, characterized in that, The migration of the second virtual machine on the first NUMA node to the second NUMA node includes: Create a third virtual machine on the second NUMA node; Establish a local communication channel between the second virtual machine and the third virtual machine; The state and data of the second virtual machine are migrated to the third virtual machine through the local communication channel.

6. The method according to claim 4 or 5, characterized in that, The second virtual machine is bound to one or more pass-through devices plugged into the first server.

7. The method according to any one of claims 1 to 6, characterized in that, Prior to migrating one or more virtual machines on the first NUMA node to the second NUMA node, the method further includes: The second NUMA node is selected from the multiple NUMA nodes included in the first server based on one or more of the following: the load of each NUMA node on the first server, the priority of the virtual machines carried on each NUMA node, and the fragmented resources existing on the first server after migration.

8. The method according to any one of claims 1 to 7, characterized in that, Prior to migrating one or more virtual machines on the first NUMA node to the second NUMA node, the method further includes: Determine if the local migration requirements are met; wherein, the local migration requirements include one or more of the following: The first server includes NUMA nodes capable of supporting the normal operation of the one or more virtual machines; or, The first NUMA node hosts one or more important virtual machines, wherein the important virtual machines are virtual machines with a priority higher than the preset priority; or, QoS degradation of one or more critical virtual machines.

9. The method according to claim 8, characterized in that, The method further includes: If the local migration requirement is not met, migrate one or more virtual machines on the first NUMA node to the NUMA node of the second server.

10. A scheduling system, characterized in that, The first server consists of multiple non-uniform memory access NUMA nodes, each of which hosts one or more virtual machines; The system includes: The first scheduling module is used to detect that the first NUMA node on the first server cannot meet the normal operation of the first virtual machine, or that the CPU and memory resources of the first NUMA node are unevenly allocated. Migrate one or more virtual machines on the first NUMA node to the second NUMA node on the first server.

11. The system according to claim 10, characterized in that, The first scheduling module is used to determine that the first NUMA node cannot meet the normal operation of the first virtual machine when the CPU resources and / or memory resources of the first NUMA node are hot.

12. The system according to claim 10, characterized in that, The first scheduling module is used to determine that the first NUMA node cannot meet the normal operation of the first virtual machine when the QoS of the application running on the first virtual machine deteriorates.

13. The system according to any one of claims 10 to 12, characterized in that, The first scheduling module is used to migrate the second virtual machine on the first NUMA node to the second NUMA node, wherein the priority of the second virtual machine is lower than the preset priority.

14. The system according to claim 13, characterized in that, The first scheduling module is used to create a third virtual machine on the second NUMA node; establish a local communication channel between the second virtual machine and the third virtual machine; and migrate the state and data of the second virtual machine to the third virtual machine through the local communication channel.

15. The method according to claim 13 or 14, characterized in that, The second virtual machine is bound to one or more pass-through devices plugged into the first server.

16. The system according to any one of claims 10 to 15, characterized in that, The first scheduling module is further configured to select the second NUMA node from the plurality of NUMA nodes included in the first server based on one or more of the following: the load of each NUMA node on the first server, the priority of the virtual machines carried on each NUMA node, and the fragmented resources existing on the first server after migration.

17. The system according to any one of claims 10 to 16, characterized in that, The first scheduling module is further configured to determine those meeting local migration requirements; wherein, the local migration requirements include one or more of the following: The first server includes NUMA nodes capable of supporting the normal operation of the one or more virtual machines; or, The first NUMA node hosts one or more important virtual machines, wherein the important virtual machines are virtual machines with a priority higher than the preset priority; or, QoS degradation of one or more critical virtual machines.

18. The system according to claim 17, characterized in that, The system also includes: The second scheduling module is used to migrate one or more virtual machines on the first NUMA node to the NUMA node of the second server when the local migration requirement is not met.

19. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including memory and a processor; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 9.

20. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 9.

21. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 9.

Images