Virtual machine live migration method, system, device and storage medium

Abstract

Embodiments of the present application provide a virtual machine live migration method, system, device and storage medium. The method comprises the following steps: obtaining a queue state of a source data queue sent by a source host; the source data queue is created by a source virtual device according to a queue creation command generated by a source virtual operating system; replaying the queue creation command executed by the source virtual device through the destination virtual device to create a destination data queue and determine the queue state of the destination data queue according to the queue state of the source data queue. According to the present application, the queue creation command of the source host can be replayed, the existing creation process can be reused, and the difficulty of live migration of the virtual device in the scenario of queue management by the virtual operating system is reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, system, device and storage medium for live migration of virtual machines. Background Technology

[0002] With the continuous development of virtualization technology, the application of virtual machines (VMs) is becoming increasingly widespread. In practical applications, it is often necessary to migrate virtual machines from one physical machine to another through hot migration to achieve dynamic scheduling of computing resources, proactive maintenance of physical faults, and so on.

[0003] Typically, a virtual machine not only has a virtual operating system but also virtual devices (or virtual hardware), such as virtual network adapters, virtual storage devices, and so on. Therefore, the process of hot migration of a virtual machine involves hot migration of virtual devices.

[0004] In existing technologies, in order to achieve hot migration of virtual devices, a lot of additional processing logic needs to be designed to ensure that the state is consistent before and after the migration, which is complex and difficult. Summary of the Invention

[0005] This application proposes a virtual machine hot migration method, system, device, and storage medium to reduce the difficulty of hot migration of virtual devices in scenarios where the virtual operating system manages the queue.

[0006] Therefore, in one embodiment of this application, a virtual machine hot migration method is provided, applicable to a target host machine, wherein the target host machine is provided with a target virtual device; wherein, the method includes:

[0007] Obtain the queue status of the source data queue sent by the source host machine; the source data queue is created by the source virtual device according to the queue creation command generated by the source virtual operating system;

[0008] The destination data queue is created by replaying the queue creation command executed by the source virtual device through the destination virtual device, and the queue status of the destination data queue is determined according to the queue status of the source data queue.

[0009] In another embodiment of this application, a virtual machine hot migration method is provided, applicable to the source host machine, comprising:

[0010] Determine the queue status of the source data queue; the source data queue is created by the source virtual device according to the queue creation command generated by the source virtual operating system;

[0011] Send the queue status of the source data queue to the destination host machine;

[0012] The destination host machine is equipped with a destination virtual device; the destination host machine is used to: replay the queue creation command executed by the source virtual device through the destination virtual device to create a destination data queue and determine the queue status of the destination data queue according to the queue status of the source data queue.

[0013] In another embodiment of this application, an electronic device is provided. The electronic device includes: a memory and a processor, wherein,

[0014] The memory is used to store programs;

[0015] The processor, coupled to the memory, is configured to execute the program stored in the memory to implement the method described in any of the preceding embodiments.

[0016] In another embodiment of this application, a computer-readable storage medium storing a computer program is provided, which, when executed by a computer, can implement the methods described in any of the above-described embodiments.

[0017] In the technical solution provided in this application embodiment, the destination virtual device replays the queue creation command generated by the source virtual operating system that was executed by the source virtual device to create a destination data queue and determine the queue status of the destination data queue based on the queue status of the source data queue, so that the status of the virtual device is synchronized before and after migration. By replaying the queue creation command of the source host, the existing creation process can be reused, reducing the difficulty of hot migration of virtual devices in scenarios where the virtual operating system manages the queue. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A structural block diagram of a virtual machine hot migration system provided in an embodiment of this application;

[0020] Figure 2 A flowchart illustrating a virtual machine hot migration method provided in an embodiment of this application;

[0021] Figure 3 A structural block diagram of a virtual machine hot migration system provided in another embodiment of this application;

[0022] Figure 4 This is an example diagram of virtual machine hot migration provided in an embodiment of this application;

[0023] Figure 5 A flowchart illustrating a virtual machine hot migration method provided in an embodiment of this application;

[0024] Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] Currently, many virtual machines use Virtio paravirtualization drivers to improve performance. The Virtio framework includes front-end drivers and back-end drivers. The front-end driver is the driver module in the virtual operating system of the virtual machine; the back-end driver is implemented in virtual machine software, such as QEMU (a virtual operating system emulator). A circular queue is used between these front-end and back-end drivers to store execution information, and it can store multiple I / O requests from the front-end driver at once, allowing the back-end driver to proactively process them in batches. This avoids processing every I / O request from the virtual operating system, reducing the number of VMEntry operations, VMExit operations, and context switches through batch sending and receiving.

[0026] To further improve performance, the vhost-user framework has been proposed. In the vhost-user framework, the backend driver is offloaded to the user-space host service process (i.e., the process of the host machine's physical operating system). The virtual machine software and the user-space host service process interact through the vhost-user protocol. NVMe (Non-Volatile Memory Express) virtual devices based on the vhost-user protocol exhibit good read and write performance. Notably, the vhost-user-based NVMe virtual device is a solution based on the KVM model, which is a pure software emulation solution.

[0027] If existing virtual device hot migration schemes are used to achieve hot migration of NVMe virtual devices based on the vhost-user protocol, a lot of additional processing logic needs to be designed, making the scheme very complex. Therefore, this application provides a simple hot migration scheme to achieve hot migration of NVMe virtual devices based on the vhost-user protocol.

[0028] Terminology Explanation:

[0029] A virtual machine (VM) is a virtual environment created on top of a physical hardware system (located externally or internally) that acts as a virtual computer system. It simulates its own complete hardware, including the CPU, memory, network interface, and storage. Using software called a hypervisor, users can decouple and properly provision the machine's resources for use by the VM.

[0030] Virtual machine migration refers to the process of moving a running virtual machine from a source host machine to a destination host machine.

[0031] QEMU is an open-source emulator and virtual machine monitor (VMM), also known as virtual machine software, which can simulate a virtual machine that can run an operating system independently.

[0032] KVM (Kernel-based Virtual Machine): This refers to a kernel-based virtual machine, an open-source virtualization technology built into Linux. KVM transforms Linux into a hypervisor, enabling a host computer to run multiple isolated virtual environments, i.e., virtual guests or virtual machine (VM) hosts. A virtual machine (VM) host is the platform on which the virtual machine runs; it can only run on a host machine and acts as the server for virtualization software.

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0034] Furthermore, some processes described in the specification, claims, and accompanying drawings of this application include multiple operations that appear in a specific order. These operations may be performed out of order or in parallel. Operation numbers such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be performed sequentially or in parallel. It should be noted that the terms "first," "second," etc., used herein are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0035] Before introducing the virtual machine hot migration method provided in the embodiments of this application, the system architecture involved in the virtual machine hot migration method provided in the embodiments of this application will be introduced. For example Figure 1 As shown, the virtual machine hot migration system includes: source host 1 and destination host 2;

[0036] The source host machine 1 is used to: determine the queue status of the source data queue; the source data queue is created by the source virtual device 11 according to the queue creation command generated by the source virtual operating system 12; the source host machine 1 is equipped with the source virtual operating system 12 and the source virtual device 11; and send the queue status of the source data queue to the destination host machine 2.

[0037] The destination host machine 2 is used to: replay the queue creation command executed by the source virtual device 11 through the destination virtual device 21 to create a destination data queue and determine the queue status of the destination data queue according to the queue status of the source data queue;

[0038] The target host machine is equipped with the target virtual device 21.

[0039] In the technical solution provided in this application embodiment, the destination virtual device replays the queue creation command generated by the source virtual operating system that was executed by the source virtual device to create a destination data queue and determine the queue status of the destination data queue based on the queue status of the source data queue, so that the status of the virtual device is synchronized before and after migration. By replaying the queue creation command of the source host, the existing creation process can be reused, reducing the difficulty of hot migration of virtual devices in scenarios where the virtual operating system manages the queue.

[0040] In practical applications, cloud computing architecture includes a computing cluster and a storage cluster. A cluster is a collection of multiple physical machines, logically managed as a whole to provide computing or storage services (e.g., cloud disks). The computing cluster provides cloud server resources. Cloud servers are provided by cloud service providers based on cloud computing technology, allowing users to operate and manage them remotely. In terms of user experience, they are the same as ordinary remote physical servers. The cloud disk functionality required by cloud servers is provided by the cloud hosts in the computing cluster along with the backend storage cluster. A computing cluster contains multiple computing nodes, each of which is a physical machine. Multiple virtual machine software programs can run on a single physical machine (each virtual machine software virtualizes a virtual machine, and each virtual machine is equivalent to a cloud server). Each virtual machine software can correspond to multiple virtual devices, such as cloud disks. A cloud disk is a disk instance built on a distributed storage system and can be read and written to in a cloud server just like a computer disk. Each compute node has a cloud disk hosting service (i.e., the backend driver mentioned later). The cloud disk hosting service is responsible for handling all cloud disk services of all cloud servers on the current compute node, mainly forwarding cloud disk IO read and write requests to the backend storage cluster. Therefore, in one example, the source host 1 mentioned above is specifically the source host compute node; the destination host 2 mentioned above is specifically the destination host compute node. The source host compute node and the destination host compute node are different compute nodes in the compute cluster.

[0041] The processing procedures of the target host and the source host in the above system, as well as the interaction between them, will be described in detail in the following embodiments.

[0042] Figure 2 A flowchart illustrating the virtual machine hot migration method provided in this application embodiment is shown. This method is applicable to the target host machine; that is, the execution subject of the method provided in this application embodiment is the target host machine, which hosts the target virtual device. For example... Figure 2 As shown, the method includes:

[0043] 201. Obtain the queue status of the source data queue sent by the source host machine.

[0044] The source data queue is created by the source virtual device according to the queue creation command generated by the source virtual operating system; the source virtual operating system and the source virtual device run on the source host machine.

[0045] 202. Replay the queue creation command executed by the source virtual device through the destination virtual device to create a destination data queue and determine the queue status of the destination data queue based on the queue status of the source data queue.

[0046] A target virtual operating system can also be virtualized on the target host machine. In one example, target virtual machine software, such as QEMU, can be run on the target host machine to virtualize the aforementioned target virtual operating system and target virtual device.

[0047] In section 201 above, the source data queue may include a source data commit queue and a source data completion queue. The source data commit queue is used to store I / O commands generated by the source virtual operating system; the source data completion queue is used to store response information from the source virtual device to I / O commands read from the source data commit queue. Correspondingly, the destination data queue may include a destination data commit queue and a destination data completion queue.

[0048] The queue state of the source data queue is used to indicate the positions of the head and tail elements of the queue. In one instance, the queue state of the source data queue may include: queue pointers, which include a head pointer and a tail pointer. The head pointer indicates the position of the head element of the source data queue, and the tail pointer indicates the position of the tail element.

[0049] In this system, the tail pointer of the source data queue is maintained by the producer of the source data queue, while the head pointer is maintained by the consumer of the source data queue. When the source data queue is a source data submission queue, its producer is the source virtual operating system, and its consumer is the source virtual device. When the source data queue is a source data completion queue, its producer is the source virtual device, and its consumer is the source virtual operating system.

[0050] In one example, a queue creation command may include: queue number, queue size, queue's guest physical address (GPA), etc. The queue number may include: queue ID. A queue creation command for creating a source data submission queue may also include: the queue number of the corresponding source data completion queue; a queue creation command for creating a source data completion queue may also include: interrupt indication information, which indicates whether an interrupt should be inserted when writing response information to the source data completion queue.

[0051] In step 202 above, the destination virtual device replays the queue creation command generated by the source virtual operating system and executed by the source virtual device to create a destination data queue. The destination virtual device modifies the queue state of the destination data queue according to the queue state of the source data queue. Specifically, the destination virtual device modifies the queue state of the destination data queue maintained by the destination virtual device according to the queue state of the source data queue.

[0052] Creating a destination data queue on the destination virtual device essentially involves creating a queue structure for the destination data queue in the memory corresponding to the destination virtual device. This queue structure is also known as queue metadata. Note: The memory corresponding to the destination virtual operating system and the memory corresponding to the destination virtual device do not overlap. The queue structure contains the queue attribute information and queue status of the destination data queue. Queue attribute information may include: queue number, queue length, virtual address of the queue on the destination virtual device side, etc. The virtual address of the queue on the destination virtual device side is obtained by mapping the queue's client physical address carried in the queue creation command. The destination virtual device accesses the queue through its virtual address on the destination virtual device side.

[0053] The queue state contained in the queue structure may include: a queue pointer to the destination data queue maintained by the destination virtual device. When the destination data queue is a destination data submission queue, the queue state contained in the queue structure may include: a head pointer to the destination data submission queue; when the destination data queue is a destination data completion queue, the queue state contained in the queue structure may include: a tail pointer to the destination data completion queue.

[0054] By replaying the queue creation command, the original data queue creation process can be reused, that is, the original queue creation code can be reused. This eliminates the need to migrate the queue state maintained by the destination virtual device, modify the virtual address of the queue on the virtual device side within the queue structure, or adapt the data format within the migrated queue structure to the differences between the virtual device versions before and after migration. This simplifies the virtual machine device hot migration solution. It should be noted that, in this embodiment, regardless of whether the virtual device version changes before and after migration, since the destination virtual device replays the queue creation command executed by the source virtual device based on its own logic code, the queue structure it creates is suitable for its own logic code, and there will be no incompatibility issues.

[0055] In this embodiment, before the destination backend driver takes over the services provided by the source backend driver, the source backend driver requests an external distributed storage system (such as the aforementioned storage cluster) to complete the corresponding read and write operations based on the IO commands of the source virtual operating system. After the destination backend driver takes over the services provided by the source backend driver, the destination backend driver requests the distributed storage system to complete the corresponding read and write operations based on the IO commands of the destination virtual operating system.

[0056] Furthermore, data stored in the source data queue can be migrated to the destination host machine by synchronizing the memory of the source virtual operating system. Specifically, the above method may further include: synchronizing the memory data of the source virtual operating system to the memory of the destination virtual operating system; the destination host machine is equipped with the destination virtual operating system; the memory data includes the data stored in the source data queue. The specific implementation of memory synchronization can be found in existing technologies and will not be elaborated here.

[0057] In the technical solution provided in this application embodiment, the destination virtual device replays the queue creation command generated by the source virtual operating system that was executed by the source virtual device to create a destination data queue and determine the queue status of the destination data queue based on the queue status of the source data queue, so that the status of the virtual device is synchronized before and after migration. By replaying the queue creation command of the source host, the existing creation process can be reused, reducing the difficulty of hot migration of virtual devices in scenarios where the virtual operating system manages the queue.

[0058] Taking an NVMe virtual device as an example, the control queue is created immediately after device initialization, and it includes a commit queue and a completion queue. Other data queues are generated by control commands in the control queue. NVMe defines a flexible relationship between data queues; either one commit queue corresponds to one completion queue, or several commit queues can share a single completion queue.

[0059] In one example, the above method may further include: receiving queue attribute information of the source data queue sent by the source host; the queue attribute information may include: queue number, queue length, virtual address of the queue on the destination virtual device side, etc. When the source data queue is a source data submission queue, the queue attribute information may further include: the queue number of the corresponding source data completion queue; when the source data queue is a source data completion queue, the queue attribute information may further include: the aforementioned interrupt indication information, which is used to indicate whether an interrupt needs to be inserted when writing response information to the source data completion queue; and reconstructing the queue creation command executed by the source virtual device based on the aforementioned queue attribute information.

[0060] In one feasible implementation, the above method may further include:

[0061] 203. Receive the queue creation command executed by the source virtual device sent by the source host machine.

[0062] The queue creation commands executed by the source virtual device are recorded by the source virtual device.

[0063] In this embodiment, playback can be achieved simply by recording, making it much simpler to implement.

[0064] In one instance, the source host can perform the following steps: when the source virtual device creates a source data queue based on the queue creation command generated by the source virtual operating system, the source virtual device records the currently executed queue creation command; when a virtual machine hot migration is required, the queue creation command recorded by the source virtual device is sent to the destination host.

[0065] In practical applications, sometimes it is necessary to delete queues after they are created. That is, in addition to recording queue creation commands, queue deletion commands also need to be recorded. Therefore, the source host can perform the following steps: When the source virtual device creates a source data queue based on the queue creation command generated by the source virtual operating system, the source virtual device records the currently executed queue creation command; when the source virtual device deletes a data queue based on the queue deletion command generated by the source virtual operating system, the source virtual device records the currently executed queue deletion command; when a virtual machine hot migration is required, it is determined whether the source virtual device has recorded a queue deletion command. If no queue deletion command is recorded, the recorded queue creation command is sent to the destination host; if a queue deletion command is recorded, the recorded queue creation command is reduced according to the recorded queue deletion command to obtain a reduced queue creation command; the reduced queue creation command is then sent to the destination host. The number of reduced queue creation commands is less than the number of recorded queue creation commands.

[0066] In the above embodiments, the deletion operation is performed by the source host. In another embodiment, the deletion operation can also be performed by the destination host, in which case the source host can directly send the recorded queue deletion command and queue creation command together to the destination host.

[0067] In practical applications, the virtual operating system transmits data to virtual devices through the aforementioned data queue; the virtual operating system also manages and controls virtual devices through a control queue. Therefore, the above method may further include:

[0068] 204. Obtain the queue status of the source control queue.

[0069] 205. The destination virtual device constructs a destination control queue and determines the queue status of the destination control queue based on the queue status of the source control queue.

[0070] In step 204 above, the source control queue is used to store the queue creation commands that the source virtual operating system needs to send to the source virtual device.

[0071] The source virtual operating system writes the queue creation command and queue deletion command it generates into the source control queue in sequence; the source virtual device reads the queue creation command and queue deletion command from the source control queue in sequence; the source virtual device creates the source data queue based on the read queue creation command and deletes the source data queue based on the read queue deletion command in sequence.

[0072] The aforementioned source control queue may include a source control commit queue and a source control completion queue. The source control commit queue stores queue creation commands, queue deletion commands, etc., from the source virtual operating system; the source control completion queue stores response information from the source virtual device to queue creation or queue deletion commands read from the source control commit queue. Correspondingly, the destination control queue may include a destination control commit queue and a destination control completion queue.

[0073] The queue state of the source control queue is used to indicate the positions of the head and tail elements of the queue. In one instance, the queue state of the source control queue may include: queue pointers, which include a head pointer and a tail pointer. The head pointer indicates the position of the head element of the source control queue, and the tail pointer indicates the position of the tail element.

[0074] In this system, the tail pointer of the source control queue is maintained by the producer of the source control queue, while the head pointer is maintained by the consumer of the source control queue. When the source control queue is a source control commit queue, its producer is the source virtual operating system, and its consumer is the source virtual device. When the source control queue is a source control completion queue, its producer is the source virtual device, and its consumer is the source virtual operating system.

[0075] The source control queue resides in the memory of the source virtual operating system; that is, the memory data of the source virtual operating system includes the data stored in the source control queue. Therefore, the data stored in the source control queue can also be migrated to the destination host machine by synchronizing the memory of the source virtual operating system.

[0076] In step 205 above, the destination virtual device can automatically construct a queue structure for the destination control queue. The queue structure includes the queue state of the destination control queue maintained by the destination virtual device. The destination virtual device modifies the queue state of the destination control queue maintained by the destination virtual device within this queue structure based on the queue state of the source control queue.

[0077] The aforementioned virtual devices can specifically be virtual storage devices. The following will describe the embodiments of this application using NVMe virtual devices based on the vhost-user protocol as an example, with both the source and destination virtual devices being NVMe virtual devices:

[0078] In one instance, such as Figure 3As shown, the destination virtual device 21 includes: a destination virtual device controller 211 (or destination virtual device interface) and a destination backend driver 212; the destination virtual device controller 211 and the destination virtual operating system 22 are virtualized by the destination virtual machine software 200 running on the destination host machine 2; the destination virtual machine software 200 and the destination backend driver 212 run in different user-mode processes of the physical operating system of the destination host machine 2; the destination virtual operating system 22 and the destination virtual device controller 211 together constitute the destination virtual machine.

[0079] The target virtual operating system 22 may have a target front-end driver 221 installed in it; the target virtual operating system 22 sends queue creation commands and I / O commands to the target virtual device controller 211 through the target front-end driver 221.

[0080] The virtual device controller in this application embodiment, also known as the virtual device controller, is used to control virtual devices to realize data exchange between virtual devices and virtual operating systems.

[0081] The phrase "replaying the queue creation command executed by the source virtual device through the destination virtual device to create a destination data queue and determining the queue status of the destination data queue based on the queue status of the source data queue" in section 202 above may include:

[0082] 2021. The destination virtual device controller replays the queue creation command executed by the source virtual device to request the destination backend driver to create a destination data queue and determine the queue status of the destination data queue based on the queue status of the source data queue.

[0083] In this embodiment, the destination control queue is used for interaction between the destination virtual operating system and the destination virtual device controller; the destination data queue is used for interaction between the destination virtual operating system and the destination backend driver.

[0084] The destination virtual device controller sends the queue creation command executed by the source virtual device to the destination backend driver. The destination backend driver creates the queue structure for the destination data queue based on this queue creation command. The destination backend driver modifies the queue state (e.g., queue pointer) of the destination data queue within this queue structure based on the queue state of the source data queue. The destination data queue includes a destination data submission queue and a destination data completion queue. When the destination data queue is a destination data submission queue, the queue state maintained by the destination backend driver is the head pointer of the destination data submission queue; when the destination data queue is a destination data completion queue, the queue state maintained by the destination backend driver is the tail pointer of the destination data completion queue.

[0085] In addition, the source virtual device 11 includes: a source virtual device controller 111 (or source virtual device interface) and a source backend driver 112; the source virtual device controller 111 and the source virtual operating system 12 are virtualized from the source virtual machine software 100 running on the source host machine 1; the source virtual machine software 100 and the source backend driver 112 run in different user-mode processes of the physical operating system of the source host machine 1; the source virtual operating system 12 and the source virtual device controller 111 together constitute the source virtual machine.

[0086] A source virtual operating system 12 may have a source front-end driver 121 installed in it; the source virtual operating system 12 sends queue creation commands and I / O commands to the source virtual device controller 111 through the source front-end driver 121.

[0087] In this embodiment, the source data queue is created by the source virtual device controller 111 upon request from the source backend driver 112 based on a queue creation command generated by the source virtual operating system 12. The source virtual device controller 111 reads the queue creation command from the source control submission queue, sends the queue creation command to the source backend driver 112 and records the queue creation command. The source backend driver 112 then creates the queue structure of the source data queue in its memory based on the queue creation command. The queue state of the source data queue maintained by the source virtual device specifically refers to the queue state of the source data queue maintained by the source backend driver. The queue state of the source control queue maintained by the source virtual device specifically refers to the queue state of the source data queue maintained by the source virtual device controller.

[0088] In this embodiment of the application, the virtual device is implemented based on a user-mode offloading scheme, that is, some functions of the virtual device are offloaded to another user-mode host process, that is, the backend driver running in another user-mode host process implements some functions of the virtual device.

[0089] Optionally, the queue state of the source data queue includes: a queue pointer of the source data queue maintained by the source backend driver.

[0090] The phrase "the destination backend driver creates the destination data queue" in the aforementioned 2021 includes:

[0091] S11. The target backend driver creates a queue structure for the target data queue.

[0092] The queue structure includes a queue pointer to the destination data queue, maintained by the destination backend driver. This queue structure resides in the memory of the destination backend driver.

[0093] The phrase "the destination backend driver determines the queue status of the destination data queue based on the queue status of the source data queue" in the aforementioned 2021 includes:

[0094] The destination backend driver modifies the queue pointer of the destination data queue maintained by the destination backend driver in the queue structure according to the queue pointer of the source data queue maintained by the source backend driver.

[0095] The queue structure contains queue attribute information and queue status of the destination data queue. Queue attribute information may include: queue number, queue length, virtual address of the queue on the destination backend driver, etc. The virtual address of the queue on the destination backend driver is obtained by mapping the queue's client physical address carried in the queue creation command. The destination backend driver accesses the queue through this virtual address.

[0096] It's worth noting that address translation is more complex in vhost-user. From the perspective of the QEMU process, the virtual operating system's memory uses two types of addresses: GPA (Guest Physical Address) and QVA (QEMU Virtual Address). From the perspective of the backend driver's process, the virtual operating system's memory also uses two types of addresses: GPA (Guest Physical Address) and VVA (Vhost Virtual Address).

[0097] GPA is the most important address in virtio. In the virtio standard, each item in the data queue used to store packet addresses is represented by GPA. However, for virtual operating systems, processes accessing the data queue actually use virtual addresses; the physical addresses are masked. In other words, a process can only access memory if it has the virtual address corresponding to the physical address.

[0098] The QEMU process is easy to implement, since the mapping relationship between QVA and GPA is established when the main process of the virtual operating system pre-allocates memory to the virtual operating system.

[0099] For the process hosting the backend driver, it's necessary to determine the starting address of the virtual address space mapped from the virtual operating system's memory to that of the backend driver's virtual address space. This assigns a GPA to the backend driver's process, which can then use the address offset to calculate the corresponding VVA (virtual address of the queue on the destination backend driver's side) and perform memory access.

[0100] Optionally, the source data queue includes a source data submission queue and a source data completion queue. The queue pointer of the source data queue maintained by the source backend driver is determined by the source host after the source backend driver stops reading I / O commands from the source data submission queue and executes the read I / O commands. Read I / O commands refer to I / O commands that have been read from the source data submission queue. Specifically, after receiving a virtual machine migration request, the source virtual device controller sends a queue status acquisition request to the source backend driver. After receiving the queue status acquisition request, the source backend driver stops reading I / O commands from the source data submission queue and continues to execute the I / O commands already read from the source data submission queue. When the I / O commands read from the source data submission queue are completed, the source virtual device controller sends the queue status of the source data queue maintained by the source backend driver to the source virtual device controller, which then sends the queue status of the source data queue maintained by the source backend driver to the destination virtual device controller. Typically, there are multiple source data queues. The source backend driver can send the queue states of multiple source data queues, maintained by the source backend driver, to the source virtual device controller in a single interaction. Similarly, the destination virtual device controller can send the queue states of multiple source data queues, maintained by the source backend driver, to the destination backend driver in a single interaction. Specifically, a new set of VHOST messages can be defined: VHOST_USER_NVME_GET_QUEUE and VHOST_USER_NVME_SET_QUEUE. The source backend driver can use VHOST_USER_NVME_GET_QUEUE to report the head pointers of all created source data submission queues and the tail pointers of source data completion queues to the source virtual device controller at once. The destination virtual device controller uses VHOST_USER_NVME_SET_QUEUE to send the head pointers of all created source data submission queues and the tail pointers of source data completion queues to the destination backend driver. This eliminates the need for the virtio-blk protocol definition, which relies on Qemu to obtain queue pointers by sending VHOST messages queue by queue based on a predefined number of queues.

[0101] The source virtual device controller can send its status, the queue status of the source data queue (maintained by the source backend driver), and the queue status of the source control queue to the destination virtual machine software. The source virtual device controller's status may include whether the source virtual operating system has the source virtual device enabled.

[0102] It should be noted that, in addition to being migrated through the source virtual machine software, the queue state can also be transmitted through an independent new host process, a virtual machine manager, a centralized control process that calls the virtual machine manager, or communication between the source backend driver and the destination backend driver. This application embodiment does not impose specific limitations on this.

[0103] In this embodiment, the destination virtual device and the destination backend driver constitute the destination virtual device, and the source virtual device and the source backend driver constitute the source virtual device. The technical solution provided in this embodiment is not only applicable to the hot migration of NVMe virtual devices based on the vhost-user protocol mentioned above, but also applicable to the hot migration of other virtual devices in any scenario where the virtual operating system manages the queue. This embodiment does not specifically limit this application.

[0104] In one instance, after receiving the queue status from the source backend driver, the source virtual device controller can disconnect the interaction link with the source backend driver; after detecting that the interaction link between the source backend driver and the source virtual device controller is disconnected, the source backend driver can shut down the relevant services for the source virtual device controller, such as cloud disk services, and reclaim resources.

[0105] In another instance, after receiving the queue status from the source backend driver, the source virtual device controller does not need to actively disconnect the interaction link with the source backend driver. The source backend driver can check at preset time intervals whether the services it provides (such as cloud disk services) have been preempted by the destination backend driver. If preempted, the source backend driver disconnects the interaction link with the source virtual device controller, shuts down the relevant services for that source virtual device controller, and reclaims resources.

[0106] The following will combine Figure 3 and Figure 4 The technical solutions provided in the embodiments of this application are illustrated with examples:

[0107] Before relocation:

[0108] On the source host machine 1, there is a source virtual machine manager 13, a source virtual machine software 100, and a source backend driver 112. The source virtual machine software 100 virtualizes a source virtual operating system 12 and a source virtual device controller 111. The aforementioned source backend driver 112 can provide cloud disk services for one or more source virtual machine software on the source host machine 1. The source virtual device controller 111 of the source virtual machine software 100 records the queue creation commands and queue deletion commands it has executed. When the virtual device is an NVMe virtual device, the aforementioned queue creation commands and queue deletion commands are both NVMe Admin commands.

[0109] When the hardware resources of the source host machine 1 are insufficient to support the multiple virtual machine software running on it, it is necessary to perform hot migration of some of the virtual machine software on the source host machine 1 to reduce the burden on the source host machine. In practical applications, the process of hot migration of virtual machine software (i.e., virtual machine hot migration) can be triggered automatically by the system or manually.

[0110] like Figure 3 and Figure 4 As shown, the migration process includes:

[0111] 1. The source virtual machine manager 13 sends a virtual machine migration request to the source virtual machine software 100.

[0112] A virtual machine migration request can be sent to the source virtual machine software 100 via an external call to the source virtual machine manager 13. The source virtual machine manager 13 is deployed on the source host machine 1 and is used to create, shut down, and migrate virtual machines. A destination virtual machine manager can also run on the destination host machine 2. The source virtual machine manager 13 on the source host machine 1 instructs the destination virtual machine manager on the destination host machine 2 to start the destination virtual machine software 200 to virtualize the destination virtual machine, thereby enabling interaction between the source virtual machine software 100 and the destination virtual machine software 200. Specifically, the source virtual machine manager 13 can send the configuration information of the source virtual machine to the destination virtual machine manager, so that the destination virtual machine manager can create the destination virtual machine.

[0113] 2. The source virtual device controller 111 of the source virtual machine software 100 sends a queue status acquisition request to the source backend driver 112.

[0114] 3. The source virtual device controller backend in the source cloud disk host 112 (i.e., the source backend driver mentioned above) stops reading I / O commands from the source data submission queue and continues to flush the read I / O commands.

[0115] 4. The source virtual device controller backend in the source cloud disk host 112 uploads the queue status of the source data queue maintained by the source cloud disk host 112 to the source virtual device controller 111.

[0116] 5. The source virtual machine software 100 disconnects from the source cloud disk host 112.

[0117] 6. The local state manager in the source cloud disk host 112 destroys the local state and shuts down the cloud disk service.

[0118] 7. The source virtual machine software 100 sends the queue creation command executed by the source virtual device controller 111, the queue status of the source data queue maintained by the source backend driver 112, the status of the source virtual device controller 111, and the queue status of the source control queue to the destination virtual machine software, which is called data migration.

[0119] 8. The destination virtual device controller 211 establishes a connection with the destination backend driver 212 so that the destination backend driver 212 can open the cloud disk service.

[0120] 9. The destination virtual device controller 212 sends the queue status of the source data queue, which is maintained by the source backend driver 112, to the destination backend driver.

[0121] 10. Command to create replay queue for the target virtual device controller 211.

[0122] 11. The destination cloud disk host 212 (i.e. the destination backend driver mentioned above) reconstructs the destination data queue according to the replayed queue creation command, and reconstructs the destination data queue's queue state maintained by the destination backend driver 212 according to the queue state of the source data queue maintained by the source backend driver 112.

[0123] After the destination cloud disk host 212 rebuilt its local state, it began to take over the services normally.

[0124] In addition, the virtual operating system generates a queue creation command, writes the queue creation command into the source control submission queue in the virtual operating system's memory, and maintains the write pointer of the source control submission queue; the source virtual device reads the queue creation command from the source control submission queue, maintains the read pointer of the source control submission queue and records the read queue creation command, and sends the queue creation command to the source cloud disk host; the source cloud disk host creates a source data queue in the virtual operating system's memory according to the queue creation command; the data queue may include a data plane submission queue and a data queue completion queue corresponding to the data plane submission queue; the virtual operating system writes IO commands into the data plane submission queue through the front-end driver, maintains the write pointer of the data plane submission queue, and writes the write pointer of the data plane submission queue into the doorbell register of the source virtual device controller; the source cloud disk host reads IO requests from the data plane submission queue according to the value in the doorbell register and the read pointer of the data plane submission queue it maintains, processes the IO commands, then writes the response result of the IO commands into the data queue completion queue, maintains the write pointer of the data plane completion queue, and sends an interrupt to the virtual operating system. Upon receiving an interrupt, the virtual operating system reads the response information from the data plane completion queue and maintains the read pointer for the data plane completion queue. The virtual operating system's memory is shared memory, shared with the source cloud disk host.

[0125] In practical applications, the value in the doorbell register of the source virtual device controller can also be sent to the destination virtual device controller.

[0126] In summary, by recording the queue creation commands executed at the source end and replaying them at the destination end, the state before and after migration is synchronized. Furthermore, by extending the vhost messages between the virtual device controller and the cloud disk host, the cloud disk host can send the queue status information of multiple queues to the virtual device controller through a one-time interaction.

[0127] Figure 5 A flowchart illustrating a virtual machine hot migration method according to an embodiment of this application is shown. The execution subject of this method is the source host machine.

[0128] 501. Determine the queue status of the source data queue.

[0129] The source data queue is created by the source virtual device according to the queue creation command generated by the source virtual operating system; the source host machine is equipped with the source virtual operating system and the source virtual device.

[0130] 502. Send the queue status of the source data queue to the destination host machine;

[0131] The destination host machine is equipped with a destination virtual device; the destination host machine is used to: replay the queue creation command executed by the source virtual device through the destination virtual device to create a destination data queue and determine the queue status of the destination data queue according to the queue status of the source data queue.

[0132] The specific implementation process of steps 501 and 502 above can be found in the corresponding content of the above embodiments, and will not be repeated here.

[0133] In the technical solution provided in this application embodiment, the destination virtual device replays the queue creation command generated by the source virtual operating system that was executed by the source virtual device to create a destination data queue and determine the queue status of the destination data queue based on the queue status of the source data queue, so that the status of the virtual device is synchronized before and after migration. By replaying the queue creation command of the source host, the existing creation process can be reused, reducing the difficulty of hot migration of virtual devices in scenarios where the virtual operating system manages the queue.

[0134] Optionally, the above method may further include:

[0135] 503. When the source virtual device creates a source data queue according to the queue creation command generated by the source virtual operating system, the currently executed queue creation command is recorded.

[0136] 504. When a virtual machine hot migration is required, send the queue creation command that the source virtual device has executed, which is recorded, to the destination host.

[0137] The specific implementation of steps 503 and 504 above can be found in the corresponding content of the above embodiments, and will not be repeated here.

[0138] It should be noted that any steps in the method provided in this application that are not described in detail can be found in the corresponding content of the above embodiments, and will not be repeated here. Furthermore, the method provided in this application may include other parts or all of the steps in the above embodiments in addition to the steps described above; for details, please refer to the corresponding content of the above embodiments, and will not be repeated here.

[0139] Figure 6 A schematic diagram of the structure of an electronic device according to an embodiment of this application is shown. Figure 6 As shown, the electronic device includes a memory 1101 and a processor 1102. The memory 1101 can be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device. The memory 1101 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0140] The memory 1101 is used to store programs;

[0141] The processor 1102 is coupled to the memory 1101 and is used to execute the program stored in the memory 1101 to implement the methods provided in the above-described method embodiments.

[0142] Furthermore, such as Figure 6 As shown, the electronic device also includes: communication component 1103, display 1104, power supply component 1105, audio component 1106, and other components. Figure 6 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 6 The components shown.

[0143] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the steps or functions of the methods provided in the above-described method embodiments.

[0144] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A virtual machine hot migration method, applicable to a destination host machine, wherein the destination host machine is equipped with a destination virtual device controller and a destination backend driver; wherein, The method includes: Obtain the queue status of the source data queue sent by the source host; the source data queue is a queue created by the source backend driver; wherein, the source virtual operating system of the source virtual machine sends a queue creation command to the source virtual device controller located on the source host through its source frontend driver, and the source virtual device controller requests the source backend driver to create the source data queue according to the queue creation command; The destination virtual device controller replays the queue creation command used by the source virtual device controller when requesting the source backend driver to create the source data queue, in order to request the destination backend driver to create the destination data queue and determine the queue status of the destination data queue based on the queue status of the source data queue.

2. The method according to claim 1, wherein, Also includes: Receive the queue creation command executed by the source virtual device sent by the source host machine; The queue creation commands executed by the source virtual device are recorded by the source virtual device.

3. The method according to claim 1, wherein, Also includes: Get the queue status of the source control queue; The source control queue is used to store the queue creation commands that the source virtual operating system needs to send to the source virtual device; The destination virtual device constructs a destination control queue, and the destination virtual device determines the queue status of the destination control queue based on the queue status of the source control queue.

4. The method according to any one of claims 1 to 3, wherein, The target virtual device controller and the target virtual operating system are obtained by virtualizing the target virtual machine software running on the target host machine; the target virtual machine software and the target backend driver run in different user-mode processes of the physical operating system of the target host machine.

5. The method according to claim 4, wherein, The source virtual device controller and the source virtual operating system are obtained by virtualizing the source virtual machine software running on the source host machine; the source virtual machine software and the source backend driver run in different user-mode processes of the physical operating system of the source host machine.

6. The method according to claim 5, wherein, The queue state of the source data queue includes: the queue pointer of the source data queue maintained by the source backend driver; The destination backend driver creates the destination data queue, including: The target backend driver creates a queue structure for the target data queue; wherein, the queue structure includes a queue pointer for the target data queue maintained by the target backend driver; The destination backend driver determines the queue status of the destination data queue based on the queue status of the source data queue, including: The destination backend driver modifies the queue pointer of the destination data queue maintained by the destination backend driver in the queue structure according to the queue pointer of the source data queue maintained by the source backend driver.

7. The method according to claim 5, wherein, The source data queue includes: a source data submission queue and a source data completion queue; The queue pointer of the source data queue, maintained by the source backend driver, is determined by the source host after the source backend driver stops reading I / O commands from the source data submission queue and has finished executing the read I / O commands.

8. The method according to any one of claims 1 to 3, wherein, Also includes: Synchronize the memory data of the source virtual operating system to the memory of the destination virtual operating system; The target host machine is equipped with the target virtual operating system; The memory data includes the data stored in the source data queue.

9. A virtual machine hot migration method, applicable to the source host machine, wherein, include: Determine the queue status of the source data queue; The source data queue is a queue created by the source backend driver; wherein, the source virtual operating system of the source virtual machine sends a queue creation command to the source virtual device controller located on the source host through its source frontend driver, and the source virtual device controller requests the source backend driver to create the source data queue according to the queue creation command; Send the queue status of the source data queue to the destination host machine; The destination host machine is equipped with a destination virtual device controller and a destination backend driver. The destination host machine is used to: replay the queue creation command used by the source virtual device controller when requesting the source backend driver to create the source data queue, so as to request the destination backend driver to create the destination data queue and determine the queue status of the destination data queue according to the queue status of the source data queue.

10. The method according to claim 9, wherein, Also includes: When the source virtual device creates a source data queue according to the queue creation command generated by the source virtual operating system, the currently executed queue creation command is recorded; When a virtual machine hot migration is required, the queue creation command that the source virtual device has executed, which is recorded, is sent to the destination host machine.

11. A virtual machine hot migration system, wherein, include: Source host and destination host; The source host machine is used to: determine the queue status of the source data queue; The source data queue is a queue created by the source backend driver; wherein, the source virtual operating system of the source virtual machine sends a queue creation command to the source virtual device controller located on the source host through its source frontend driver, and the source virtual device controller requests the source backend driver to create the source data queue according to the queue creation command; and sends the queue status of the source data queue to the destination host. The destination host is used to: replay the queue creation command used by the source virtual device controller when requesting the source backend driver to create the source data queue, so as to request the destination backend driver to create the destination data queue and determine the queue status of the destination data queue according to the queue status of the source data queue; the destination host is equipped with the destination virtual device controller and the destination backend driver.

12. The system according to claim 11, wherein, The source host machine is also used for: When the source virtual device creates a source data queue according to the queue creation command generated by the source virtual operating system, the currently executed queue creation command is recorded; When a virtual machine hot migration is required, the queue creation command that the source virtual device has executed, which is recorded, is sent to the destination host machine.

13. An electronic device, wherein, include: Memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program stored in the memory to implement the method of any one of claims 1 to 10.

14. A computer-readable storage medium storing a computer program, wherein, When the computer program is executed by a computer, it can implement the method of any one of claims 1 to 10.

Images