Virtual machine migration method and device, computer program product and storage medium

Abstract

The embodiment of the invention provides a virtual machine migration method and device, a computer program product and a storage medium. For a target virtual machine which performs rendering acceleration by using a hardware GPU, after a source host receives a live migration trigger instruction, the target virtual machine can be controlled to stop sending an instruction to the used virtual GPU besides starting to execute a conventional live migration process, and the target virtual machine can be controlled to stop sending the instruction to the used virtual GPU after each rendering instruction which is still being executed is executed. And migrating the equipment state data corresponding to the virtual GPU and the rendering data and the state data generated by the hardware rendering process corresponding to the virtual GPU to a target host, so as to support state restoration for the virtual GPU and the hardware rendering process involved in the target virtual machine in the target host. Therefore, instruction playback does not need to be carried out on the target host, so that the migration efficiency of the virtual machine can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular to a virtual machine migration method, device, computer program product, and storage medium. Background Technology

[0002] Virtual machine hot migration refers to the process of migrating a running virtual machine from a source host to a destination host.

[0003] Currently, technologies such as Virtual I / O GPU (Virtio-GPU) for Graphics Processing Units (GPUs) enable the use of hardware GPUs in virtual machines to handle some complex rendering tasks, thereby accelerating rendering. For these types of virtual machines, instruction replay is typically used to ensure that all components related to rendering acceleration can be restored to their pre-migration state.

[0004] However, this method requires re-running all the historical rendering instructions that the virtual machine had performed on the source host after starting the virtual machine on the destination host. This results in a long playback time, which in turn leads to an excessively long migration time for this type of virtual machine. Summary of the Invention

[0005] This application provides a virtual machine migration method, apparatus, computer program product, and storage medium to improve migration efficiency for virtual machines using hardware GPUs for rendering acceleration.

[0006] This application provides a virtual machine migration method, applicable to the source host where the target virtual machine resides, the method comprising:

[0007] In response to a hot migration trigger command for the target virtual machine, control the target virtual machine to stop sending rendering commands to the virtual GPU used by the target virtual machine;

[0008] During the hot migration of the target virtual machine, if it is detected that the rendering instructions received by the virtual GPU have been executed, the device status data corresponding to the virtual GPU is migrated to the destination host to support the destination host in restoring the state of the virtual GPU.

[0009] The rendering data and state data generated by the hardware rendering process corresponding to the virtual GPU are migrated to the destination host to support the destination host in restoring the state of the hardware rendering process.

[0010] This application also provides a virtual machine migration method, adapted to the destination host to which the target virtual machine is migrated, the method comprising:

[0011] In response to a recovery command for the target virtual machine, the target virtual machine is initialized;

[0012] After initialization is complete, the virtual GPU to be used by the target virtual machine is restored according to the device status data corresponding to the virtual GPU migrated from the source host of the target virtual machine.

[0013] Based on the rendering data and status data generated by the hardware rendering process corresponding to the virtual GPU migrated from the source host, the state of the hardware rendering process is restored.

[0014] This application also provides a computing device, including a memory, a processor, and a communication component;

[0015] The memory is used to store one or more computer instructions;

[0016] The processor is coupled to the memory and the communication components and is used to execute one or more computer instructions for performing the aforementioned virtual machine migration method as a source host or destination host.

[0017] This application also provides a computer-readable storage medium for storing a computer program, which, when executed by one or more processors, causes the one or more processors to perform the aforementioned virtual machine migration method.

[0018] This application also provides a computer program product, including a computer program that, when executed by one or more processors, causes the one or more processors to perform the aforementioned virtual machine migration method.

[0019] In this embodiment, a virtual machine migration method is proposed for virtual machines using hardware GPUs for rendering acceleration. Upon receiving a hot migration trigger command, the source host, in addition to starting the regular hot migration process, can control the target virtual machine to stop sending commands to the virtual GPU used by the target virtual machine. After all currently executing rendering commands are completed, the source host can begin migrating the device state data corresponding to the virtual GPU used by the target virtual machine, as well as the rendering data and state data generated by the hardware rendering process corresponding to the virtual GPU, to the destination host. This supports state restoration of the virtual GPU and hardware rendering process involved in the target virtual machine on the destination host. Accordingly, in this embodiment, command replay is no longer required on the destination host. Instead, a detailed migration of the aforementioned data is interspersed within the original hot migration process of the virtual machine. This ensures that all entities related to rendering acceleration (such as the virtual GPU and hardware rendering process) can be restored to their pre-migration state on the destination host side, effectively improving the migration efficiency for this type of virtual machine. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 A flowchart illustrating a virtual machine migration method provided for an exemplary embodiment of this application;

[0022] Figure 2 A schematic diagram illustrating the principle of a target virtual machine performing rendering using a hardware GPU, provided as an exemplary embodiment of this application;

[0023] Figure 3 A flowchart illustrating another virtual machine migration method provided as an exemplary embodiment of this application;

[0024] Figure 4 A schematic diagram illustrating the rendering acceleration principle based on VirtIO and Virglrenderer technologies, provided for embodiments of this application;

[0025] Figure 5 A flowchart illustrating another virtual machine migration method provided as an exemplary embodiment of this application;

[0026] Figure 6 A flowchart illustrating a virtual machine migration method provided as another exemplary embodiment of this application;

[0027] Figure 7A flowchart illustrating another virtual machine migration method provided as another exemplary embodiment of this application;

[0028] Figure 8 This is a schematic diagram of the structure of a computing device provided as another exemplary embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] Before proceeding with a detailed description of the technical solutions provided in the various embodiments of this application, the following is a brief explanation of several technical concepts involved in this application.

[0031] Virtual machine hot migration refers to the process of migrating a running virtual machine from a source host to a destination host. Virtual machine hot migration mainly involves the migration of memory data and can be broadly divided into two migration mechanisms: pre-copy and post-copy. During hot migration, the virtual machine's downtime is very short; therefore, users are usually unaware of the hot migration process.

[0032] VirtIO, short for Virtual I / O, is a paravirtualization I / O technology that employs a front-end and back-end architecture. It includes device drivers, virtualized devices, and virtualized queues (Vrings) for communication between the device drivers and virtualized devices. The device driver resides within the virtual machine (acting as the front-end driver) and is the corresponding driver for the virtualized device within the virtual machine. The virtualized device is created by the Virtual Machine Manager (VMM) through virtualization of external devices (acting as the back-end device). Both the device driver and the virtualized device can read and write data from the virtualized queue. The device driver can save I / O instructions to the virtualized queue and then notify the virtualized device, which can then retrieve and process the I / O instructions from the queue.

[0033] VirtIO-GPU, short for Virtual I / O GPU, is a GPU virtualization solution based on the VirtIO technology mentioned above. It can be understood as virtualizing the hardware GPU using VirtIO technology to create a virtual GPU (as a backend device), and creating a device driver for the virtual GPU within the virtual machine (as a frontend driver). Applications within the virtual machine can send rendering commands to this device driver, which then passes the rendering commands to the virtual GPU, which can then invoke the hardware GPU to perform rendering.

[0034] Software rendering refers to rendering using the CPU. For virtual machines, this can be understood as using the CPU to accelerate rendering.

[0035] Hardware rendering, on the other hand, uses the hardware GPU to perform rendering. From the perspective of a virtual machine, it can be understood as using the hardware GPU to accelerate rendering. Correspondingly, the hardware rendering process is a rendering process that runs on the hardware GPU.

[0036] As described in the background section, currently, if a virtual machine needs to use a hardware GPU for rendering acceleration, after migrating such a virtual machine, all historical rendering instructions that occurred on the source host need to be rerun on the destination host to ensure that all components related to rendering can be restored to their pre-migration state. This requires a long replay time, resulting in excessively long migration times for such virtual machines.

[0037] In response, this application proposes a virtual machine migration scheme based on a state transition mechanism for this type of virtual machine. This eliminates the need for instruction replay on the destination host, ensuring that all entities related to rendering acceleration can be restored to their pre-migration state, thereby improving the migration efficiency of this type of virtual machine.

[0038] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0039] Figure 1 This is a flowchart illustrating a virtual machine migration method provided as an exemplary embodiment of this application. Figure 1 As shown, this method can be executed by the source host where the target virtual machine to be migrated resides, specifically by a virtual machine migration device integrated in the source host. This virtual machine migration device can be implemented as software, hardware, or a combination of software and hardware.

[0040] In this embodiment, the target virtual machine can be any virtual machine that requires the use of a hardware GPU for rendering acceleration. It should be understood that, in this embodiment, by default, both the source and destination hosts corresponding to the target virtual machine are equipped with hardware GPUs and can support the aforementioned VirtIO-GPU and other technologies capable of virtualizing hardware GPUs.

[0041] Figure 2 This is a schematic diagram illustrating the principle of a target virtual machine performing rendering using a hardware GPU, as provided in an exemplary embodiment of this application. (Reference) Figure 2 The host (source host or destination host) used to host the target virtual machine is equipped with a hardware GPU. In this embodiment, the hardware GPU can be virtualized to generate a virtual GPU that can be used by the target virtual machine.

[0042] During their research, the inventors discovered that the target virtual machine can send rendering instructions to the virtual GPU it uses during operation, and the virtual GPU can drive the corresponding hardware GPU to execute the rendering instructions in the host through the hardware rendering process mentioned above.

[0043] Based on this, this embodiment proposes, with reference to Figure 1 The virtual machine migration method executed by the source host where the target virtual machine resides may include:

[0044] Step 100: In response to the hot migration trigger command for the target virtual machine, control the target virtual machine to stop sending rendering commands to the virtual GPU used by the target virtual machine;

[0045] Step 101: During the hot migration of the target virtual machine, if it is detected that the rendering instructions received by the virtual GPU have been executed, the device status data corresponding to the virtual GPU will be migrated to the destination host to support the destination host in restoring the virtual GPU's status.

[0046] Step 102: Migrate the rendering data and state data generated by the hardware rendering process corresponding to the virtual GPU to the destination host to support the destination host in restoring the state of the hardware rendering process.

[0047] refer to Figure 1 In step 100, after receiving the hot migration trigger command for the target virtual machine, the source host can begin executing the hot migration process. In this embodiment, it can be understood that a logical modification has been made to the virtual machine hot migration technology. Based on the existing migration logic in the virtual machine hot migration technology, the migration logic provided in steps 100-102 of this embodiment is added. The existing migration logic in the virtual machine hot migration technology may include migration logic for memory data and the virtual machine's own device state, etc. Thus, it can be understood that in this embodiment, one triggering function of the hot migration trigger command is to trigger the source host to begin executing the existing migration logic in the virtual machine hot migration technology for the target virtual machine.

[0048] refer to Figure 1In step 100, another triggering function is designed for the hot migration trigger command: that is, to trigger the source host to control the target virtual machine to stop sending rendering commands to the virtual GPU used by the target virtual machine. In other words, after the hot migration trigger command for the target virtual machine occurs, through the control of the source host, the target virtual machine will stop sending rendering commands to the virtual GPU it uses, and correspondingly, the virtual GPU used by the target virtual machine will stop receiving new rendering commands.

[0049] In this embodiment, when receiving new rendering instructions stops, the rendering instructions previously received by the virtual GPU may not have been completed, causing the virtual GPU's device state data to remain in flux. To address this, this embodiment proposes waiting for the rendering instructions already received by the virtual GPU to complete. The duration of this waiting process is uncertain and depends on factors such as the execution complexity of these rendering instructions. However, this waiting process does not affect the hot migration process of the target virtual machine; for example, during this waiting process, the migration of memory data for the target virtual machine proceeds normally.

[0050] refer to Figure 1 In step 101, the source host can monitor whether the rendering instructions received by the virtual GPU have been executed completely during the hot migration of the target virtual machine. In practical applications, the virtual GPU usually receives a completion notification after the rendering instructions have been executed. Based on this, in this embodiment, the source host can initiate a query operation on the instruction execution status to the virtual GPU to determine whether all the rendering instructions received by the virtual GPU have been executed completely. Of course, the virtual GPU can also proactively send a completion notification to the source host after confirming that all the rendering instructions it has received have been executed, so that the source host knows that the rendering instructions received by the virtual GPU have been executed completely.

[0051] Once all rendering instructions received by the virtual GPU have been executed, the virtual GPU's device state will no longer change. Although the corresponding hardware rendering process of the virtual GPU in the source host is still running, it will also remain unchanged because there are no more rendering instructions to execute.

[0052] Therefore, steps 101 and 102 in this embodiment propose that state transitions for the entities related to rendering acceleration can begin after the rendering instructions received by the virtual GPU have been executed. This embodiment proposes that the entities related to rendering acceleration may include at least: the virtual GPU used by the target virtual machine and the corresponding hardware rendering process of the virtual GPU in the source host.

[0053] refer to Figure 1In step 101, it is proposed that the source host can migrate the device state data corresponding to the virtual GPU to the destination host to support the destination host in restoring the state of the virtual GPU.

[0054] During their research, the inventors discovered that current virtualization technologies for hardware CPUs are typically implemented by a Virtual Machine Manager (VMM) on the host machine (source or destination host), and the VMM is usually responsible for maintaining device state data for the virtual GPU. A VMM, also known as a Hypervisor, is a software layer running on the host machine that provides resource isolation, monitoring, and management functions for virtualization. Common Hypervisors include Qemu and VirtualBox. Device state data describes the working state of the virtual GPU and may include static attribute information and the values ​​of various state variables involved. The specific device state data is not limited here. In this embodiment, the migration of device state data aims to support the destination host in correctly restoring the state of the virtual GPU. Therefore, in practical applications, ensuring the virtual GPU can be restored to its pre-migration state can be the goal for defining the required data items in the device state data in step 101. In this way, in step 101, the source host can collect these required data items and migrate them to the destination host.

[0055] Based on this, the device state data maintained by the virtual machine manager (VMM) for the virtual GPU to be migrated can be read from the VMM (which can be understood as the physical memory used by the VMM in the source host) and migrated to the destination host.

[0056] It is worth noting that in this embodiment, the source host can save the device state data corresponding to the virtual GPU to a storage space outside the destination host, and the destination host can then read the required device state data from that storage space when state restoration is needed to complete the migration of device state data. Of course, the source host can also directly send the device state data corresponding to the virtual GPU to the destination host to complete the migration of device state data; this embodiment does not limit this approach.

[0057] As mentioned earlier, once the rendering instructions received by the virtual GPU have been executed, the corresponding hardware rendering process of the virtual GPU on the source host will no longer change. Based on this, this embodiment proposes in step 102 that the rendering data and state data generated by the hardware rendering process corresponding to the virtual GPU can be migrated to the destination host to support the destination host in restoring the state of the hardware rendering process.

[0058] Rendering data refers to the data used or generated by the hardware GPU when running the hardware rendering process. Rendering data may include, but is not limited to, pixel data and coordinate data, etc., without further examples here. Status data describes the running status of the hardware rendering process, including but not limited to the identification information of the rendering resources used, whether lighting calculations are used, and a series of data such as the light sources that are turned on, etc., without further examples of the data items in status data.

[0059] Based on this, in step 102, a lookup logic for the rendering data and status data generated by the hardware rendering process can be designed for the source host. This allows the source host to meticulously locate the rendering data and status data generated by the hardware rendering process corresponding to the virtual GPU and migrate it to the destination host. During their research, the inventors discovered that the hardware rendering process runs based on the hardware GPU, but it is usually the source host that is responsible for managing and scheduling the hardware rendering process. Therefore, in this embodiment, based on the lookup logic designed for the source host, the source host can comprehensively locate the rendering data and status data generated by the hardware rendering process corresponding to the virtual GPU and migrate it to the destination host.

[0060] It is worth noting that, for reference Figure 2 This embodiment describes a technical concept for state migration of the virtual GPU and hardware rendering processes related to rendering acceleration by the source host. This provides a comprehensive and accurate data foundation for the target host to restore the state of the virtual GPU and hardware rendering processes after the target virtual machine is migrated to the destination host, thereby ensuring that the target host can restore the virtual GPU and hardware rendering processes to their pre-migration state. Of course, in practical applications, the entities related to rendering acceleration may not only be the virtual GPU and hardware rendering processes, but also other entities, such as the target virtual machine itself, the virtual GPU device driver, etc. However, as mentioned earlier in this embodiment, during the hot migration process, in addition to executing the state migration technical logic provided in this embodiment, the source host also executes the existing migration logic in the hot migration technology, which includes the migration logic for virtual machine memory data. Therefore, the memory data corresponding to the target virtual machine can be fully migrated to the destination host, while the data related to the state of the aforementioned other entities is basically already included in the memory data of the target virtual machine. In this way, the memory data migrated to the destination host can provide a comprehensive and accurate data foundation for the destination host to restore the state of other entities, thereby ensuring that the target host can also restore other entities related to rendering acceleration to their pre-migration state.

[0061] Therefore, the technical concept provided in this embodiment does not limit the GPU virtualization technology used in the source host. For example, GPU virtualization can be performed using the aforementioned VirtIO technology, or software simulation technology, pass-through technology, or pass-through sharing technology SR-IOV, etc. Different GPU virtualization technologies will generate virtual GPUs for use by the target virtual machine; however, the names of these virtual GPUs may differ depending on the technology. For example, under VirtIO technology, the virtual GPU is usually called VirtIO-GPU; and under SR-IOV technology, it is usually called vGPU. In the technical concept provided in this embodiment, the source host can perform state migration for the virtual GPU generated by any GPU virtualization technology.

[0062] Furthermore, the technical concept provided in this embodiment does not limit the driver technology framework used to support virtual GPU drivers of hardware GPUs. Regardless of the driver technology framework, it essentially needs to run the hardware rendering process based on the hardware GPU, although there may be differences in the management and scheduling details of the hardware rendering process. This embodiment proposes to take the hardware rendering process as the main body for state transition. This low-level and fundamental perspective ensures that the technical concept provided in this embodiment can be adapted to various driver technology frameworks. For different driver technology frameworks, the technical logic can be adapted in the engineering implementation.

[0063] Preferably, in this embodiment, virglrenderer can be used as the driver technology framework. Virglrenderer is a driver technology framework used to pass OpenGL rendering instructions to the hardware GPU. Its basic principle is to provide a virtual renderer (called virgl) in the host as a bridge between the virtual GPU and the hardware GPU, which forwards the rendering instructions provided by the virtual GPU to the hardware GPU and drives the hardware GPU to perform rendering. Under this driver technology framework, the source host can manage and schedule the hardware rendering process through the virtual renderer virgl. Moreover, the Virglrenderer driver technology framework also provides support for OpenGL. OpenGL stands for Open Graphics Library, which is a cross-language, cross-platform application programming interface for rendering vector graphics. Based on this, the aforementioned virtual renderer virgl can provide a set of general OpenGL interfaces to the virtual GPU, and the virtual GPU can call these OpenGL interfaces to drive the hardware GPU to perform rendering.

[0064] Furthermore, based on the cross-language and cross-platform capabilities of the OpenGL interface, this embodiment can support the target virtual machine to use various types of hardware GPUs for rendering acceleration, without being limited by the hardware GPU's proprietary driver. As mentioned earlier, the technical concept provided in this embodiment takes the hardware rendering process as the main body. Therefore, when using various types of hardware GPUs for rendering acceleration, the technical concept provided in this embodiment can be used to complete the state transition of the hardware rendering process.

[0065] In summary, this embodiment proposes a virtual machine migration method for virtual machines using hardware GPUs for rendering acceleration. Upon receiving a hot migration trigger command, the source host, in addition to starting the regular hot migration process, can control the target virtual machine to stop sending commands to the virtual GPU it uses and wait for all currently executing rendering commands to complete. Then, it can begin migrating the device state data corresponding to the virtual GPU used by the target virtual machine, as well as the rendering data and state data generated by the hardware rendering process corresponding to the virtual GPU, to the destination host. This supports state restoration of the virtual GPU and hardware rendering process involved in the target virtual machine on the destination host. Accordingly, in this embodiment, it is no longer necessary to replay commands on the destination host. Instead, the detailed migration of the aforementioned data is interspersed within the original hot migration process of the virtual machine. Therefore, based on the aforementioned data, the destination host can ensure that all entities related to rendering acceleration of the target virtual machine can be restored to their pre-migration state, which effectively improves the migration efficiency for this type of virtual machine.

[0066] Figure 3 This is a flowchart illustrating another virtual machine migration method provided as an exemplary embodiment of this application. (Reference) Figure 3 This method is still executed by the source host. (See reference) Figure 3 The virtual machine migration method performed by the source host may include:

[0067] Step 300: In response to the hot migration trigger command for the target virtual machine, control the target virtual machine to stop sending rendering commands to the virtual GPU used by the target virtual machine;

[0068] Step 301: During the hot migration of the target virtual machine, if it is detected that the rendering instructions received by the virtual GPU have been executed, the device status data corresponding to the virtual GPU is migrated to the destination host to support the destination host in restoring the state of the virtual GPU.

[0069] Step 302: If the virtual GPU is generated by virtualizing the hardware GPU using VirtIO technology, then migrate the driver state data generated by the device driver of the virtual GPU in the target virtual machine to the target host to support the target host in restoring the state of the device driver.

[0070] Step 303: Migrate the rendering data and state data generated by the hardware rendering process corresponding to the virtual GPU to the destination host to support the destination host in restoring the state of the hardware rendering process.

[0071] Step 300 can be referred to the above. Figure 1 The description of step 100 in the illustrated embodiment is similar to that of step 301, which can be found in the foregoing. Figure 1 The description of step 101 in the illustrated embodiment is as follows; step 303 can be referred to the foregoing. Figure 1 The description of step 102 in the illustrated embodiment will not be repeated here. In this embodiment, compared to Figure 1 The virtual machine migration method provided in the illustrated embodiment adds step 302. Step 302 proposes that, in this embodiment, VirtIO technology can be used to virtualize the hardware GPU to generate a virtual GPU used by the target virtual machine. Referring to the preceding explanation of the VirtIO technology concept, VirtIO technology adopts a front-end / back-end architecture. In this embodiment, the virtual GPU used by the target virtual machine is the back-end device in this front-end / back-end architecture. Furthermore, VirtIO technology sets up a device driver for the generated virtual GPU and deploys the device driver in the target virtual machine.

[0072] Figure 4 This diagram illustrates the rendering acceleration principle based on VirtIO and Virglrenderer technologies, as provided in an embodiment of this application. (Reference) Figure 4 VirtIO technology provides a device driver for the virtual GPU, which is deployed in the target virtual machine. The application in the target virtual machine can send rendering instructions to the device driver, which can then pass the rendering instructions to the virtual GPU. The virtual GPU can then pass the rendering instructions to the virtual renderer virgl provided in Virglrenderer technology. The virtual renderer virgl then drives the hardware GPU to perform rendering according to the rendering instructions.

[0073] Based on this, in step 302, it is proposed that the source host can migrate the driver state data generated by the virtual GPU device driver in the target virtual machine to the destination host after the rendering instructions received by the virtual GPU have been executed, so as to support the target host to restore the state of the device driver.

[0074] In practical applications, as mentioned in the previous embodiments, the driver state data to be migrated in step 302 is a part of the memory data corresponding to the target virtual machine. Therefore, the migration of driver state data in step 302 can be achieved based on the migration logic of memory data.

[0075] In one exemplary implementation: after the virtual GPU has completed executing the received rendering instructions, the driver state data can be migrated to the destination host as dirty page data corresponding to the target virtual machine. Dirty page data refers to memory data that has changed. By marking the driver state data as dirty page data after the virtual GPU has completed executing the received rendering instructions, migration can be performed only after the driver state data has stabilized. This eliminates the need for data migration while the driver state data is still changing, thus reducing the workload. Of course, this embodiment is not limited to this; existing memory data migration logic can also ensure that the data migrated to the destination host is the driver state data after the virtual GPU has completed executing the received rendering instructions.

[0076] In summary, this embodiment proposes that when using VirtIO technology to virtualize a hardware GPU, the driver state data generated by the virtual GPU's device driver within the target virtual machine should be used as the data to be migrated, thereby supporting the destination host in restoring the state of the virtual GPU's device driver. The driver state data, as part of the memory data corresponding to the target virtual machine, can be fully migrated to the destination host.

[0077] Figure 5 This is a flowchart illustrating another virtual machine migration method provided as an exemplary embodiment of this application. (Reference) Figure 5 This method still applies to the source host, and may include:

[0078] Step 500: In response to the hot migration trigger command for the target virtual machine, control the target virtual machine to stop sending rendering commands to the virtual GPU used by the target virtual machine;

[0079] Step 501: During the hot migration of the target virtual machine, if it is detected that the rendering instructions received by the virtual GPU have been executed, the device status data corresponding to the virtual GPU is migrated to the destination host to support the destination host in restoring the virtual GPU's status.

[0080] Step 502: If the virtual GPU is generated by virtualizing the hardware GPU using VirtIO technology, then migrate the driver state data generated by the device driver of the virtual GPU in the target virtual machine to the target host to support the target host in restoring the state of the device driver.

[0081] Step 503: Determine whether the target virtual machine and virtual GPU use shared memory in the source host. If yes, proceed to step 504; otherwise, step 504 is not required.

[0082] Step 504: Read the address mapping relationship between the target virtual machine and the virtual GPU from the virtual machine manager in the source host; generate address binding information for the rendering resources used in the hardware rendering process, the address binding information is used to describe the guest physical memory address occupied by the rendering resources in shared memory; carry the address mapping relationship and address binding information to the destination host to support the destination host to update the address mapping relationship and the rendering resources;

[0083] Step 505: Migrate the rendering data and state data generated by the hardware rendering process corresponding to the virtual GPU to the destination host to support the destination host in restoring the state of the hardware rendering process.

[0084] It is worth noting that, for ease of illustration, Figure 5 The specific details of steps 500-505 have been abbreviated, but this should not cause ambiguity or loss of the scope of protection. Compared to Figure 3 In this embodiment, the provided virtual machine migration method includes steps 503 and 504.

[0085] In this embodiment, a shared memory mechanism is supported between the virtual GPU and the target virtual machine to accelerate data transfer and improve rendering acceleration performance. For example, in the aforementioned VirtIO technology, a shared memory mechanism is typically used between the VirtIO-GPU and the virtual machine. (See reference...) Figure 2 and Figure 4 The target virtual machine resides in the guest operating system on the source host, while the Virtual Machine Manager (VMM) resides in the host operating system. Under the shared memory mechanism, the physical memory allocated to the virtual GPU can be shared by both the target virtual machine and the virtual GPU, and both can access this shared memory via Direct Memory Access (DMA). Under the shared memory mechanism, the VMM maintains an address mapping relationship between the target virtual machine and the virtual GPU for this shared memory. In practical applications, this address mapping relationship records the mapping between the Guest Physical Address (GPA) used by the target virtual machine and the Host Virtual Address (HVA) used by the virtual GPU. Based on the shared memory mechanism, the virtual machine can access the virtual GPU's memory space via DMA without the need for the source host's CPU.

[0086] Therefore, this embodiment proposes that if the target virtual machine and the virtual GPU use shared memory on the source host, the source host can also migrate the aforementioned address mapping relationship maintained for the shared memory to the destination host.

[0087] In addition to proposing the migration of address mapping relationships, this embodiment also proposes that the source host be the rendering resource used in the hardware rendering process, and that address binding information be generated. The address binding information records the client physical memory address (GPA) occupied by the rendering resource in shared memory.

[0088] Rendering resources can be understood as resource objects used during the rendering process (i.e., the process of running the hardware rendering process). Common rendering resources include, but are not limited to, textures, buffers, and shaders, which will not be further exemplified here. Rendering resources typically define various attributes. Under a shared memory mechanism, rendering resources define attributes for recording DMA addresses. Here, the DMA address specifically refers to the host virtual memory address (HVA) in shared memory, not the guest physical memory address (GPA). This is mainly because rendering resources are created within the hardware rendering process, and the management and scheduling of the hardware rendering process also reside on the host system. Therefore, the address recorded in the rendering resource is the HVA, not the GPA.

[0089] During their research, the inventors discovered that after a target virtual machine is migrated to a destination host, the destination host retains the guest physical memory address (GPA) for shared memory in the target virtual machine. However, it reallocates a host virtual memory address (HVA) for the shared memory. This leads to inconsistencies between the HVA recorded in the rendering resources and the HVA reallocated on the destination host for the same memory location in shared memory, potentially causing the rendering resources to become unusable.

[0090] Therefore, this embodiment proposes adding processing logic to the source host to generate address binding information for rendering resources. An exemplary generation logic could be:

[0091] The process involves querying the target host virtual memory address (HVA) in shared memory corresponding to the rendering data used by the target rendering resource; based on the address mapping relationship, determining the target guest physical memory address (GPA) mapped to the target host virtual memory address HVA, and recording it as the guest physical memory address (GPA) occupied by the target rendering resource in the shared memory; based on this, the aforementioned address binding information can be generated for the target rendering resource during the migration of the target virtual machine. Here, the target rendering resource is any rendering resource used in the hardware rendering process, and the aforementioned target host virtual memory address HVA is recorded in the target rendering resource.

[0092] In this exemplary generation logic, the source host can record the GPA (Gross Address Mapping) occupied in shared memory for the rendering resources used in the hardware rendering process (determined by querying the address mapping table). When virtual machine migration is required, the source host can generate address binding information for the rendering resources in this embodiment based on this information recorded for the rendering resources. It should be understood that in this embodiment, the source host does not interfere with the addresses recorded internally by the rendering resources, but generates additional address binding information for the rendering resources outside of the rendering resources themselves.

[0093] Accordingly, this embodiment proposes that when shared memory is used between the target virtual machine and the virtual GPU, during the hot migration of the target virtual machine, the source host will also migrate the address mapping relationship maintained for the shared memory; the source host will also generate address binding information for the rendering resources used by the hardware rendering process to be migrated, and migrate it. Based on these two aspects of data migration proposed in this embodiment, the target host can update the migrated address mapping information according to the HVA reallocated for the shared memory after restoring the shared memory, and update the HVA recorded in the migrated rendering resources, so as to ensure that the migrated shared memory and rendering resources can be used normally.

[0094] In the above or following embodiments, various implementation methods can be used to migrate the rendering data and state data generated by the hardware rendering process corresponding to the virtual GPU to the destination host. One optional implementation method is provided below.

[0095] First, the logic for processing state data in this implementation will be explained.

[0096] In this optional implementation, regarding state data, it is proposed that if the virtual GPU calls the hardware GPU through the virtual rendering component provided by the source host, the source host can migrate the context information maintained by the hardware rendering process corresponding to the virtual GPU in the virtual rendering component to the destination host.

[0097] Here, the virtual rendering component is a functional component in the source host used to manage and schedule the hardware rendering process. The implementation of the virtual rendering component may differ across different driver technology frameworks. As mentioned earlier, the Virglrenderer driver technology architecture is preferred to support virtual GPU calls to the hardware GPU. In this driver technology architecture, the virtual renderer (virgl) is the virtual rendering component provided by the source host.

[0098] The virtual rendering component maintains context information for each hardware rendering process. For example, the virtual renderer (virgl) maintains a virgl context for each hardware rendering process, which is essentially an OpenGL context. The virtual renderer (virgl) encapsulates the OpenGL context to create a virgl context. The virtual renderer (virgl) is essentially a state machine. A typical interpretation of the virgl context states that it records all the information and states required to perform rendering. In other words, the virgl context records all the information and states needed for the corresponding hardware rendering process to run.

[0099] Based on this, this alternative implementation proposes using the context information maintained by the virtual rendering component for the hardware rendering process as the state data to be migrated. Clearly, the content recorded in the context information is sufficient to support the destination host in restoring the state of the hardware rendering process.

[0100] In addition, this optional implementation proposes the following regarding state data: the source host can also migrate the global state information maintained for the virtual GPU in the virtual rendering component to the destination host. The global state information records the hardware rendering process being executed by the virtual GPU when the migration trigger command is received.

[0101] In this optional implementation, the global state information maintained for the virtual GPU is also used as the state data to be migrated. Thus, by migrating the global state information, the destination host can restore the hardware rendering process currently executing in the virtual rendering component to the hardware rendering process recorded in the global state information, thereby restoring the progress of the virtual rendering component to its state before migration.

[0102] Based on this, taking the aforementioned Virglrenderer driver architecture as an example, in this optional implementation, the source host can use the following two types of data as state data that needs to be migrated for the hardware rendering process:

[0103] 1) virgl context;

[0104] 2) Global status information.

[0105] Here's a brief explanation of the Virgl context. This embodiment supports creating one or more hardware rendering processes for the virtual GPU. The virtual rendering component maintains context information for each hardware rendering process. This context information can contain two types of information: main context information and sub-context information. The main context information typically records some attributes and states shared by the process, and multiple sub-context information can share the main context information. A hardware rendering process can contain multiple rendering tasks, and each rendering task can be considered a rendering thread within the hardware rendering process. The virtual rendering component maintains sub-context information for each rendering thread. Thus, multiple sub-context information sharing the main context information can also be interpreted as multiple rendering threads sharing rendering resources within the rendering process.

[0106] Based on this, in this implementation, a Virgl context can include at least the following migration aspects:

[0107] 1) Identifier of the virgl context;

[0108] 2) The main context in the virgl context;

[0109] 3) Sub-contexts within the virgl context.

[0110] By migrating the two types of state data, virgl context and global state information, to the destination host, the destination host can restore the state of the hardware rendering process based on the virgl context, and the destination host can restore the working progress of the virtual renderer (virgl) to the hardware rendering process recorded in the global state information.

[0111] The following section explains the processing logic for rendering data in this implementation.

[0112] In this implementation, the following steps are proposed for rendering data: determining the target rendering resources used in the hardware rendering process; collecting the rendering data of the target rendering resources stored in the hardware GPU, and / or, the rendering data stored in the physical memory of the source host; and migrating the collected rendering data to the destination host.

[0113] In this implementation, the collected context information can be traversed to parse the rendering resources used by the hardware rendering process from the context information.

[0114] For a rendering resource, during its use by the hardware rendering process, the generated rendering data may be stored in the hardware GPU or in the physical memory of the source host (e.g., the aforementioned shared memory). For rendering data stored in the physical memory of the source host, the source host can perform read operations on the physical memory to migrate this rendering data to the destination host.

[0115] As for the rendering data stored in the hardware GPU, due to the isolation of memory address spaces, the source host does not know the storage location of the rendering data in the hardware GPU. Therefore, the source host cannot directly collect this part of the rendering data.

[0116] Therefore, in this implementation, for rendering data stored in the hardware GPU, it is further proposed that: if the first rendering resource in the target rendering resource stores rendering data in the hardware graphics processor, the source host can select the required access interface for the first rendering resource based on the mapping relationship between the type of rendering resource and the access interface. The access interface defines access logic for accessing preset data from the hardware GPU; using the selected access interface, the rendering data stored in the hardware GPU by the first rendering resource is read.

[0117] The access interface in this embodiment is custom-defined. A pre-defined communication protocol can be established between the access interface and the hardware GPU to ensure that access commands initiated by the access interface are correctly responded to by the hardware GPU. For different types of rendering resources, the types of rendering data to be accessed may differ, and the required access logic may also differ. Therefore, this embodiment proposes designing separate access interfaces for different types of rendering resources. Specifically, one or more access interfaces can be designed for a single type of rendering resource to support accessing various types of required rendering data from the hardware GPU for that type of rendering resource. It is understood that, given the type of rendering resource and the type of rendering data to be accessed, the user manuals of components such as the virtual renderer (virgl) can be consulted to discover the memory access logic of the hardware GPU during the hardware rendering process. From this memory access logic, access paths can be found, and the access logic within the corresponding access interface can be designed based on these access paths. The design of the access logic within the access interface is an engineering implementation issue and can be flexibly designed in practical applications; therefore, it will not be elaborated upon here.

[0118] Of course, the above optional implementation methods are merely examples. In this embodiment, the source host can also use other implementation methods to migrate the rendering data and state data generated by the hardware rendering process corresponding to the virtual GPU to the destination host. For example, a portion of the data can be selected from the context information of the hardware rendering process for migration, without having to migrate all the context information. Further examples are not provided here. Additionally, in some driver technology frameworks, the virtual GPU can also autonomously manage and schedule the hardware rendering process without relying on the aforementioned virtual rendering component. In this case, the virtual GPU needs to have the ability to drive different models of hardware GPUs. As mentioned earlier, this is only different from the driver technology framework that drives the hardware GPU through the virtual rendering component; the only difference is the method of managing and scheduling the hardware rendering process. In this case, the state data corresponding to the hardware rendering process is still maintained by the source host (specifically, the Virtual Machine Manager (VMM) in the source host). Therefore, the source host can still obtain the state data corresponding to the hardware rendering process without difficulty, and the same applies to the rendering data. Further details are not provided here.

[0119] In summary, in this embodiment, the source host can meticulously collect rendering data and state data for the hardware rendering process that needs to be migrated and migrate them to the destination host. This provides the destination host with sufficient data foundation to support the destination host in restoring the state of the hardware rendering process and the virtual rendering components.

[0120] The above describes the virtual machine migration method on the source host side. It should be understood that referring to... Figure 5 Once the data migration described above is completed on the source host, and the source host has reached the shutdown time according to the existing migration logic (for example, when migrating memory data using the pre-copy method, the amount of dirty pages in the latest round of memory has fallen below a preset threshold), the target virtual machine can be shut down on the source host. The virtual machine migration method on the destination host is adapted to the virtual machine migration method on the source host and can be understood as the reverse process of the virtual machine migration method on the source host. The following describes the virtual machine recovery process on the destination host.

[0121] Figure 6 This is a flowchart illustrating a virtual machine migration method as provided in another exemplary embodiment of this application. This virtual machine migration method is applicable to the destination host to which the target virtual machine needs to be migrated. Similarly, the destination host may also integrate a virtual machine migration device, which executes the virtual machine migration method. This virtual machine migration device can be implemented as software, hardware, or a combination of software and hardware. (Reference) Figure 6 The virtual machine migration method may include:

[0122] Step 600: In response to the recovery command for the target virtual machine, initialize the target virtual machine;

[0123] Step 601: After initialization, restore the state of the virtual GPU required by the target virtual machine based on the device status data of the virtual GPU migrated from the source host of the target virtual machine.

[0124] Step 602: Restore the state of the hardware rendering process based on the rendering data and state data generated by the hardware rendering process corresponding to the virtual GPU migrated from the source host.

[0125] In this embodiment, the destination host will also use the existing migration logic in virtual machine hot migration technology, such as initializing the virtual machine and allocating physical memory to the virtual machine. These routine tasks will not be described in detail.

[0126] refer to Figure 6 In step 601, it is proposed that after the initialization of the target virtual machine is completed, the state restoration of each subject related to rendering acceleration can begin.

[0127] Echoing the source host side Figure 1 In this embodiment, step 601 proposes that the target host can restore the state of the virtual GPU required by the target virtual machine based on the device status data corresponding to the virtual GPU migrated from the source host of the target virtual machine.

[0128] Echoing the source host side Figure 1 The rendering data and state data corresponding to the hardware rendering process proposed in the document are described in step 602. The destination host can restore the state of the hardware rendering process based on the rendering data and state data generated by the hardware rendering process corresponding to the virtual GPU migrated from the source host.

[0129] It should be understood that state restoration in this embodiment can be understood as storing the data migrated from the source host in the destination host, so as to ensure that all entities related to rendering acceleration can use this data in the destination host as before the migration, thereby ensuring that the state of each entity is the same as before the migration. It is clear that in this embodiment, instruction replay is no longer required in the destination host; instead, the state restoration problem is transformed into a data migration problem. The destination host simply stores the migrated virtual GPU device state data and the rendering data and state data corresponding to the hardware rendering process in the same way as in the source host.

[0130] Taking device status data as an example, since the device status data migrated from the source host describes the working state of the virtual GPU at the time of migration, the destination host can restore the virtual GPU's state to its working state at the time of migration by saving the migrated device status data to a storage location recognized by the virtual GPU. The same principle applies to the state restoration of the hardware rendering process.

[0131] In summary, this embodiment proposes a virtual machine migration method for virtual machines using hardware GPUs for rendering acceleration. Upon receiving a hot migration trigger command, the source host, in addition to starting the regular hot migration process, can control the target virtual machine to stop sending commands to the virtual GPU it uses. After all currently executing rendering commands are completed, the source host can begin migrating the device state data corresponding to the virtual GPU used by the target virtual machine, as well as the rendering data and state data generated by the hardware rendering process corresponding to the virtual GPU, to the destination host. This supports state restoration of the virtual GPU and hardware rendering process involved in the target virtual machine on the destination host. Therefore, in this embodiment, it is no longer necessary to replay commands on the destination host. Instead, the detailed migration of the aforementioned data is interspersed within the original hot migration process of the virtual machine. Thus, based on the aforementioned data, the destination host can ensure that all entities related to rendering acceleration of the target virtual machine can be restored to their pre-migration state, effectively improving the migration efficiency for this type of virtual machine. As can be seen, in this embodiment, the source host and the destination host of the target virtual machine can cooperate with each other, and through meticulous data migration operations, they can ensure that the relevant entities involved in the process of the target virtual machine using hardware GPU for rendering acceleration can be restored to the state before migration.

[0132] In one optional embodiment, the same GPU virtualization technology is used in both the destination host and the source host. If the virtual GPU is generated by virtualizing a hardware GPU using VirtIO technology, the destination host can restore the state of the device driver corresponding to the virtual GPU used by the target virtual machine based on the driver state data migrated from the source host.

[0133] Since the device driver corresponding to the virtual GPU is still deployed in the target virtual machine on the destination host, the driver state data corresponding to the device driver is stored in the destination host as part of the memory data according to the existing memory data storage scheme. This ensures that the device driver can access the required driver state data without obstacles, thereby ensuring that the device driver can be restored to the state before migration.

[0134] Figure 7 A flowchart illustrating another virtual machine migration method provided as an exemplary embodiment of this application. (Reference) Figure 7 This method is applicable to the destination host and may include:

[0135] Step 700: In response to the recovery command for the target virtual machine, initialize the target virtual machine;

[0136] Step 701: After initialization, restore the state of the virtual GPU required by the target virtual machine based on the device status data of the virtual GPU migrated from the source host of the target virtual machine.

[0137] Step 702: If the virtual GPU is generated by virtualizing the hardware GPU using VirtIO technology, then restore the state of the device driver corresponding to the virtual GPU used by the target virtual machine based on the driver state data migrated from the source host.

[0138] Step 703: Determine whether the target virtual machine and virtual GPU need to use shared memory. If yes, proceed to step 703; otherwise, do not proceed to step 703.

[0139] Step 704: Allocate physical memory from the destination host as shared memory; In the host virtual memory address space of the destination host, reallocate the host virtual memory address HVA for the shared memory, while the guest physical memory address GPA used by the target virtual machine on the shared memory remains unchanged; Update the address mapping relationship for the shared memory migrated from the source host according to the reallocated host virtual memory address HVA.

[0140] Step 705: Restore the state of the hardware rendering process based on the rendering data and state data generated by the hardware rendering process corresponding to the virtual GPU migrated from the source host.

[0141] For ease of illustration, Figure 7 The specific content of steps 700-704 has been simplified, but this should not cause ambiguity or loss of the scope of protection.

[0142] As mentioned earlier, the address mapping relationship mentioned in step 703 is used to record the mapping relationship between the guest physical memory address (GPA) used by the target virtual machine and the host virtual memory address (HVA) used by the virtual GPU in the shared memory. In the destination host, the HVA is reallocated for this shared memory; therefore, the destination host can replace the HVA in the address mapping relationship migrated from the source host. This ensures that the migrated target virtual machine and virtual GPU can use the shared memory normally, and will not be unable to use it due to an incorrect address mapping relationship.

[0143] In addition, refer to Figure 7In response to the address binding information corresponding to the rendering resources migrated from the source host, step 704 further includes: the destination host can also, for the target rendering resources used by the hardware rendering process, parse the target client physical memory address occupied by the target rendering resource in shared memory from the address binding information corresponding to the target rendering resource migrated from the source host; based on the updated address mapping relationship, determine the target host virtual memory address mapped to the target client physical memory address; and update the host virtual memory address recorded in the target rendering resource to the target host virtual memory address. Here, the target rendering resource is any rendering resource used by the hardware rendering process.

[0144] To this end, the destination host can traverse all rendering resources used by the migrated hardware rendering process and reassign the host virtual memory address (HVA) occupied in shared memory for each rendering resource, thereby updating the attribute used to record the DMA address in each rendering resource. This ensures that the migrated hardware rendering process can access the required rendering data normally when using rendering resources, thus avoiding the inability to use rendering resources normally due to address issues.

[0145] Optionally, if the virtual GPU on the source host calls the hardware GPU through a virtual rendering component, the destination host also provides a virtual rendering component for the migrated virtual GPU to use. Based on this, if the state data migrated from the source host contains context information corresponding to the hardware rendering process, the destination host can restore the context information for the hardware rendering process corresponding to the virtual GPU within the virtual rendering component provided by the destination host. Context information is typically stored in the physical memory allocated to the virtual rendering component on the source host. Here, the context information can be stored in the physical memory allocated to the virtual rendering component on the destination host to ensure that the virtual rendering component provided by the destination host can use this context information correctly. Since this context information describes the working state of the hardware rendering process at the time of migration, correctly storing this context information on the destination host can restore the hardware rendering process to its state before migration.

[0146] Optionally, if the state data also includes global state information, the destination host can restore the virtual rendering component to execute the target hardware rendering process recorded in the global state information. The target hardware rendering process is the hardware rendering process that the virtual GPU was executing when the source host received the migration trigger instruction. In this way, the virtual rendering component's progress can be restored to its state before migration on the destination host, and the target hardware rendering process will continue to run on the destination host after the target virtual machine restarts.

[0147] In the above or following embodiments, the state recovery process of the hardware rendering process may include:

[0148] Iterate through the collected context information to determine the target rendering resources used by the hardware rendering process in the virtual rendering component;

[0149] Save the rendering data used by the target rendering resource to the hardware GPU and / or the physical memory of the destination host.

[0150] In other words, for the rendering data corresponding to the hardware rendering process, according to the storage method in the source host, part of the rendering data will be saved to the physical memory of the destination host (such as the aforementioned reallocated shared memory); the other part of the rendering data can be saved to the hardware GPU.

[0151] Optionally, for target rendering data that needs to be saved to the hardware GPU, if the second rendering resource in the target rendering resource needs to save rendering data to the hardware graphics processor, the destination host can select the required access interface for the second rendering resource based on the mapping relationship between the type of rendering resource and the access interface. The access interface defines access logic for accessing preset data from the hardware GPU. Using the selected access interface, the rendering data that the second rendering resource needs to save in the hardware GPU is written to the hardware GPU.

[0152] It is worth noting that the technical details of the virtual machine migration method for the destination host described above can be found by referring to the reverse process of the virtual machine migration method for the source host, and will not be elaborated upon here. However, this should not result in any loss of the scope of protection of this application.

[0153] It should be noted that some processes described in the above embodiments and accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel.

[0154] Figure 8 This is a schematic diagram of the structure of a computing device provided as another exemplary embodiment of this application. For example... Figure 8 As shown, the computing device may include: a memory 80, a processor 81, and a communication component 82.

[0155] This computing device can serve as a source host during virtual machine migration. In this case, processor 81, coupled with memory and communication components 82, executes computer programs stored in memory 80 for:

[0156] In response to a hot migration trigger command for the target virtual machine, control the target virtual machine to stop sending rendering commands to the virtual GPU used by the target virtual machine;

[0157] During the hot migration of the target virtual machine, if it is detected that the rendering instructions received by the virtual GPU have been executed, the device status data corresponding to the virtual GPU is migrated to the destination host to support the destination host in restoring the state of the virtual GPU.

[0158] The rendering data and state data generated by the hardware rendering process corresponding to the virtual GPU are migrated to the destination host to support the destination host in restoring the state of the hardware rendering process.

[0159] In an alternative embodiment, processor 81 may also be used for:

[0160] If the virtual GPU is generated by virtualizing a hardware GPU using VirtIO technology, then after the rendering instructions received by the virtual GPU have been executed, the driver state data generated by the device driver of the virtual GPU in the target virtual machine is migrated to the target host to support the target host in restoring the state of the device driver.

[0161] In an optional embodiment, when the processor 81 migrates the driver state data generated by the device driver of the virtual GPU in the target virtual machine to the destination host, it can specifically be used for:

[0162] The driver state data is used as the dirty page data of the target virtual machine and migrated to the destination host.

[0163] In an alternative embodiment, processor 81 may also be used for:

[0164] If the target virtual machine and the virtual GPU use shared memory in the source host, then the address mapping relationship between the target virtual machine and the virtual GPU is read from the virtual machine manager in the source host;

[0165] For the rendering resources used in the hardware rendering process, address binding information is generated. The address binding information is used to describe the client physical memory address occupied by the rendering resources in the shared memory.

[0166] The address mapping relationship and the address binding information are migrated to the destination host to support the destination host in updating the address mapping relationship and the rendering resources.

[0167] In an optional embodiment, the address mapping relationship is used to record the mapping relationship between the guest physical memory address used by the target virtual machine and the host virtual memory address used by the virtual GPU; the processor 81 can also be used for:

[0168] Query the target host virtual memory address occupied by the rendering data used by the target rendering resource in the shared memory; based on the address mapping relationship, determine the target client physical memory address mapped to the target host virtual memory address, and record it as the client physical memory address occupied by the target rendering resource in the shared memory;

[0169] The target rendering resource is any rendering resource used in the hardware rendering process.

[0170] In an optional embodiment, when the processor 81 migrates the state data generated by the hardware rendering process corresponding to the virtual GPU to the destination host, it can specifically be used for:

[0171] If the virtual GPU calls the hardware GPU through the virtual rendering component, then the context information maintained by the virtual rendering component for the hardware rendering process corresponding to the virtual GPU is migrated to the destination host; and / or,

[0172] The global state information maintained by the virtual rendering component for the virtual GPU is migrated to the destination host. The global state information records the hardware rendering process that is being executed by the virtual GPU when the migration trigger instruction is received.

[0173] In an optional embodiment, when the processor 81 migrates the rendering data generated by the hardware rendering process corresponding to the virtual GPU to the destination host, it can specifically be used for:

[0174] Determine the target rendering resources used in the hardware rendering process;

[0175] Collect rendering data of the target rendering resources stored in the hardware GPU, and / or rendering data stored in the physical memory of the source host;

[0176] The collected rendering data will be migrated to the target host.

[0177] In an optional embodiment, when the processor 81 collects rendering data of the target rendering resource stored in the hardware GPU, it may specifically be used for:

[0178] If the first rendering resource in the target rendering resources stores rendering data in the hardware graphics processor, then based on the mapping relationship between the type of rendering resource and the access interface, the access interface to be called is selected for the first rendering resource. The access interface defines access logic for accessing preset data from the hardware GPU.

[0179] Using the selected access interface, read the rendering data stored in the hardware GPU of the first rendering resource.

[0180] In one optional embodiment, virtual I / O technology is used to virtualize the hardware GPU on the source host to generate the virtual GPU; the rendering instructions use OpenGL instructions; virglrenderer technology is used to support the virtual GPU in calling the corresponding hardware GPU to execute the hardware rendering process, and the virglrenderer component in the virglrenderer technology serves as the virtual rendering component, forwarding the rendering instructions received by the virtual GPU to the hardware GPU to drive the hardware GPU to execute rendering according to the OpenGL instructions.

[0181] It is worth noting that the technical details of the above-mentioned embodiments of the computing device can be referred to the relevant descriptions in the aforementioned source host side method embodiments. To save space, they will not be repeated here, but this should not cause any loss to the scope of protection of this application.

[0182] This computing device can serve as a source host during virtual machine migration. In this case, processor 81, coupled with memory and communication components 82, executes computer programs stored in memory 80 for:

[0183] In response to a recovery command for the target virtual machine, the target virtual machine is initialized;

[0184] After initialization is complete, the virtual GPU to be used by the target virtual machine is restored according to the device status data corresponding to the virtual GPU migrated from the source host of the target virtual machine.

[0185] Based on the rendering data and status data generated by the hardware rendering process corresponding to the virtual GPU migrated from the source host, the state of the hardware rendering process is restored.

[0186] In an alternative embodiment, processor 81 may also be used for:

[0187] If the virtual GPU is generated by virtualizing a hardware GPU using VirtIO technology, then the device driver corresponding to the virtual GPU in the target virtual machine is restored based on the driver state data migrated from the source host.

[0188] In an alternative embodiment, processor 81 may also be used for:

[0189] If the target virtual machine and the virtual GPU need to use shared memory, then physical memory is allocated from the destination host as the shared memory;

[0190] In the host virtual memory address space of the destination host, a host virtual memory address is reallocated for the shared memory, while the guest physical memory address used by the target virtual machine on the shared memory remains unchanged;

[0191] Update the address mapping relationship for the shared memory migrated from the source host according to the reallocated host virtual memory address.

[0192] In an alternative embodiment, processor 81 may also be used for:

[0193] For the target rendering resources used by the hardware rendering process, the target client physical memory address occupied by the target rendering resources in the shared memory is parsed from the address binding information corresponding to the target rendering resources migrated from the source host.

[0194] Based on the updated address mapping relationship, determine the target host virtual memory address mapped to the target client's physical memory address;

[0195] Update the host virtual memory address recorded in the target rendering resource to the target host virtual memory address;

[0196] The target rendering resource is any rendering resource used by the hardware rendering process.

[0197] In an optional embodiment, when the processor 81 restores the state of the hardware rendering process based on the state data generated by the hardware rendering process corresponding to the virtual GPU migrated from the source host, it can specifically be used for:

[0198] If the virtual GPU invokes the hardware GPU through the virtual rendering component provided by the destination host, then based on the context information corresponding to the hardware rendering process contained in the state data, the context information for the hardware rendering process corresponding to the virtual GPU is restored in the virtual rendering component provided by the destination host; and / or,

[0199] Based on the global state information contained in the state data, the virtual rendering component is restored to execute the target hardware rendering process recorded in the global state information. The target hardware rendering process is the hardware rendering process that is being executed by the virtual GPU when the source host receives the migration trigger instruction.

[0200] In an optional embodiment, when the processor 81 restores the state of the hardware rendering process based on the rendering data generated by the hardware rendering process corresponding to the virtual GPU migrated from the source host, it can specifically be used for:

[0201] The collected context information is traversed to determine the target rendering resources used by the hardware rendering process in the virtual rendering component;

[0202] The rendering data used by the target rendering resource is saved to the hardware GPU and / or the physical memory of the target host.

[0203] In an optional embodiment, when the processor 81 saves the rendering data used by the target rendering resource to the hardware GPU, it can specifically be used for:

[0204] If the second rendering resource in the target rendering resource needs to save rendering data to the hardware graphics processor, then based on the mapping relationship between the type of rendering resource and the access interface, the access interface to be called is selected for the second rendering resource. The access interface defines access logic for accessing preset data from the hardware GPU.

[0205] Using the selected access interface, the rendering data that the second rendering resource needs to be stored in the hardware GPU is written to the hardware GPU.

[0206] It is worth noting that the technical details of the above-mentioned embodiments of the computing device can be referred to the relevant descriptions in the aforementioned method embodiments on the target host side. To save space, they will not be repeated here, but this should not cause any loss to the scope of protection of this application.

[0207] Furthermore, such as Figure 8 As shown, the computing device may also include other components such as a power supply component 83. Figure 8 The diagram only shows some components and does not mean that the computing device includes only these components. Figure 8 The components shown.

[0208] This application also provides a virtual machine migration system, which includes the source host where the target virtual machine is located and the destination host to which it needs to be migrated.

[0209] The source host can be used to respond to the hot migration trigger command for the target virtual machine, control the target virtual machine to stop sending rendering commands to the virtual GPU used by the target virtual machine; during the hot migration of the target virtual machine, after detecting that the rendering commands received by the virtual GPU have been executed, the source host migrates the device state data corresponding to the virtual GPU to the destination host to support the destination host in restoring the state of the virtual GPU; and migrates the rendering data and state data generated by the hardware rendering process corresponding to the virtual GPU to the destination host to support the destination host in restoring the state of the hardware rendering process.

[0210] The destination host can be used to respond to recovery commands for the target virtual machine, initialize the target virtual machine; after initialization, restore the state of the virtual GPU required by the target virtual machine based on the device state data of the virtual GPU migrated from the source host of the target virtual machine; and restore the state of the hardware rendering process based on the rendering data and state data generated by the hardware rendering process corresponding to the virtual GPU migrated from the source host.

[0211] For technical details regarding the source host and destination host, please refer to the aforementioned method embodiments on the source host side and method embodiments on the destination host side. To save space, these will not be elaborated here, but this should not cause any loss to the scope of protection of this application.

[0212] Accordingly, in this virtual machine migration system, a virtual machine migration method is proposed for virtual machines using hardware GPUs for rendering acceleration. After receiving the hot migration trigger command, the source host, in addition to starting the regular hot migration process, can also control the target virtual machine to stop sending commands to the virtual GPU used by the target virtual machine. After the execution of each rendering command still in progress is completed, the device state data corresponding to the virtual GPU used by the target virtual machine, as well as the rendering data and state data generated by the hardware rendering process corresponding to the virtual GPU, can be migrated to the destination host. This supports state restoration of the virtual GPU and hardware rendering process involved in the target virtual machine on the destination host. Accordingly, in this embodiment, it is no longer necessary to perform command replay on the destination host. Instead, the detailed migration of the above-mentioned data is interspersed during the original hot migration process of the virtual machine. This ensures that all entities related to rendering acceleration (such as virtual GPUs and hardware rendering processes) of the target virtual machine can be restored to their pre-migration state on the destination host side, which can effectively improve the migration efficiency for this type of virtual machine.

[0213] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed, can implement the steps in the above method embodiments.

[0214] Accordingly, this application also provides a computer program product, which, when executed, can implement the steps in the above method embodiments.

[0215] The above Figure 8 The memory in a computer is used to store computer programs and can be configured to store various other data to support operation on a computing platform. Examples of this data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc. The memory can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disks, or optical disks.

[0216] The above Figure 8 The communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component processor 81 can also be used in a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0217] The above Figure 8 The power supply component provides power to the various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0218] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0219] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0220] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0221] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0222] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements, but also that processor 81 can be used for other elements not expressly listed, or for elements inherent to such process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0223] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0224] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A virtual machine migration method, characterized in that, Applicable to the source host where the target virtual machine resides, the method includes: In response to a live migration trigger command for the target virtual machine, control the target virtual machine to stop sending rendering commands to the virtual graphics processor used by the target virtual machine; During the hot migration of the target virtual machine, if it is detected that the rendering instructions received by the virtual graphics processor have been executed, the device status data corresponding to the virtual graphics processor is migrated to the destination host to support the destination host in restoring the status of the virtual graphics processor. The rendering data and state data generated by the hardware rendering process corresponding to the virtual graphics processor are migrated to the destination host to support the destination host in restoring the state of the hardware rendering process.

2. The method according to claim 1, characterized in that, Also includes: If the virtual graphics processor is generated by virtualizing the hardware graphics processor using VirtIO technology, then after the rendering instructions received by the virtual graphics processor have been executed, the driver state data generated by the device driver of the virtual graphics processor in the target virtual machine is migrated to the target host to support the target host in restoring the state of the device driver.

3. The method according to claim 2, characterized in that, Migrating the driver state data generated by the device driver of the virtual graphics processor in the target virtual machine to the destination host includes: The driver state data is used as the dirty page data of the target virtual machine and migrated to the destination host.

4. The method according to claim 1, characterized in that, Also includes: If the target virtual machine and the virtual graphics processor use shared memory in the source host, then the address mapping relationship between the target virtual machine and the virtual graphics processor is read from the virtual machine manager in the source host; For the rendering resources used in the hardware rendering process, address binding information is generated. The address binding information is used to describe the client physical memory address occupied by the rendering resources in the shared memory. The address mapping relationship and the address binding information are migrated to the destination host to support the destination host in updating the address mapping relationship and the rendering resources.

5. The method according to claim 4, characterized in that, The address mapping relationship is used to record the mapping relationship between the guest physical memory address used by the target virtual machine and the host virtual memory address used by the virtual graphics processor; The method further includes: Query the target host virtual memory address occupied by the rendering data used by the target rendering resource in the shared memory; Based on the address mapping relationship, the target client physical memory address mapped to the target host virtual memory address is determined and recorded as the client physical memory address occupied by the target rendering resource in the shared memory; The target rendering resource is any rendering resource used in the hardware rendering process.

6. The method according to claim 1, characterized in that, Migrating the state data generated by the hardware rendering process corresponding to the virtual graphics processor to the destination host includes: If the virtual graphics processor calls the hardware graphics processor through the virtual rendering component, then the context information maintained by the virtual rendering component for the hardware rendering process corresponding to the virtual graphics processor is migrated to the destination host; and / or, The global state information maintained by the virtual rendering component for the virtual graphics processor is migrated to the destination host. The global state information records the hardware rendering process that is being executed by the virtual graphics processor when the migration trigger instruction is received.

7. The method according to claim 6, characterized in that, Migrating rendering data generated by the hardware rendering process corresponding to the virtual graphics processor to the destination host includes: Determine the target rendering resources used in the hardware rendering process; Collect the rendering data of the target rendering resources stored in the hardware graphics processor, and / or the rendering data stored in the physical memory of the source host; The collected rendering data will be migrated to the target host.

8. The method according to claim 7, characterized in that, Collect the rendering data of the target rendering resource stored in the hardware graphics processor, including: If the first rendering resource in the target rendering resource stores rendering data in the hardware graphics processor, then based on the mapping relationship between the type of rendering resource and the access interface, the access interface to be called is selected for the first rendering resource. The access interface defines access logic for accessing preset data from the hardware graphics processor. Using the selected access interface, read the rendering data of the first rendering resource stored in the hardware graphics processor.

9. The method according to claim 1, characterized in that, Virtual I / O technology is used to virtualize the hardware graphics processor on the source host to create the virtual graphics processor; the rendering instructions use OpenGL instructions; virglrenderer technology is used to support the virtual graphics processor in calling its corresponding hardware graphics processor to execute the hardware rendering process. The virglrenderer component in the virglrenderer technology serves as the virtual rendering component, forwarding the rendering instructions received by the virtual graphics processor to the hardware graphics processor to drive the hardware graphics processor to execute rendering according to the OpenGL instructions.

10. A virtual machine migration method, characterized in that, The method, adapted to the destination host to which the target virtual machine is migrated, includes: In response to a recovery command for the target virtual machine, the target virtual machine is initialized; After initialization is completed, the virtual graphics processor to be used by the target virtual machine is restored according to the device status data corresponding to the virtual graphics processor migrated from the source host of the target virtual machine. Based on the rendering data and status data generated by the hardware rendering process corresponding to the virtual graphics processor migrated from the source host, the state of the hardware rendering process is restored.

11. The method according to claim 10, characterized in that, Also includes: If the virtual graphics processor is generated by virtualizing a hardware graphics processor using VirtIO technology, then the state of the device driver corresponding to the virtual graphics processor in the target virtual machine is restored based on the driver state data migrated from the source host.

12. The method according to claim 10, characterized in that, Also includes: If the target virtual machine and the virtual graphics processor need to use shared memory, then physical memory is allocated from the destination host as the shared memory; In the host virtual memory address space of the destination host, a host virtual memory address is reallocated for the shared memory, while the guest physical memory address used by the target virtual machine on the shared memory remains unchanged; Update the address mapping relationship for the shared memory migrated from the source host according to the reallocated host virtual memory address.

13. The method according to claim 12, characterized in that, Also includes: For the target rendering resources used by the hardware rendering process, the target client physical memory address occupied by the target rendering resources in the shared memory is parsed from the address binding information corresponding to the target rendering resources migrated from the source host. Based on the updated address mapping relationship, determine the target host virtual memory address mapped to the target client's physical memory address; Update the host virtual memory address recorded in the target rendering resource to the target host virtual memory address; The target rendering resource is any rendering resource used by the hardware rendering process.

14. The method according to claim 10, characterized in that, Based on the state data generated by the hardware rendering process corresponding to the virtual graphics processor migrated from the source host, the state of the hardware rendering process is restored, including: If the virtual graphics processor invokes the hardware graphics processor through the virtual rendering component provided by the destination host, then according to the context information corresponding to the hardware rendering process contained in the status data, the context information for the hardware rendering process corresponding to the virtual graphics processor is restored in the virtual rendering component; and / or, Based on the global state information contained in the state data, the virtual rendering component is restored to execute the target hardware rendering process recorded in the global state information. The target hardware rendering process is the hardware rendering process that is being executed by the virtual graphics processor when the source host receives the migration trigger instruction.

15. The method according to claim 14, characterized in that, Based on the rendering data generated by the hardware rendering process corresponding to the virtual graphics processor migrated from the source host, the state of the hardware rendering process is restored, including: The collected context information is traversed to determine the target rendering resources used by the hardware rendering process in the virtual rendering component; The rendering data used by the target rendering resource is saved to the hardware graphics processor and / or the physical memory of the target host.

16. The method according to claim 15, characterized in that, Saving the rendering data used by the target rendering resource to the hardware graphics processor includes: If the second rendering resource in the target rendering resource needs to save rendering data to the hardware graphics processor, then based on the mapping relationship between the type of rendering resource and the access interface, the access interface to be called is selected for the second rendering resource. The access interface defines access logic for accessing preset data from the hardware graphics processor. Using the selected access interface, the rendering data that the second rendering resource needs to be stored in the hardware graphics processor is written to the hardware graphics processor.

17. A virtual machine migration system, characterized in that, This includes the source host where the target virtual machine resides and the destination host to which it needs to be migrated: The source host is used to execute the virtual machine migration method according to any one of claims 1-9; The destination host is used to execute the virtual machine migration method according to any one of claims 10-16.

18. A computing device, characterized in that, Includes memory, processor, and communication components; The memory is used to store one or more computer instructions; The processor is coupled to the memory and the communication component and is used to execute one or more computer instructions to perform the virtual machine migration method of any one of claims 1-9 as a source host, or to perform the virtual machine migration method of any one of claims 10-16 as a destination host.

19. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is executed by one or more processors, the one or more processors perform the virtual machine migration method according to any one of claims 1-16.

20. A computer program product, characterized in that, Includes a computer program that, when executed by one or more processors, causes the one or more processors to perform the virtual machine migration method according to any one of claims 1-16.

Images