Virtual machine migration method, chip system and host

Abstract

The embodiment of the invention provides a virtual machine migration method, a chip system and a host. Application to the computer field. The chip system comprises a host system and a network card system. The method comprises the following steps: migrating first dirty page data from a first storage area of a host system to a destination host by a network card system; the first storage area is used for storing memory data of the virtual machine. And in the process of migrating the first dirty page data, the network card system stops initiating new service information with the at least one service host according to the first indication information. The business information indicates business processing between the virtual machine and at least one business host. And after the first dirty page data is migrated, the virtual machine management system controls the virtual machine to shut down, and migrates state data of the virtual machine to the destination host based on the first transmission connection. The state data comprises a service interaction state of the virtual machine and at least one service host. According to the embodiment of the invention, the initiation of the new service is stopped before the virtual machine is shut down, so that the virtual machine live migration under virtualization direct connection is realized.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a virtual machine migration method, a chip system, and a host. Background Art

[0002] Virtualization technology can be implemented based on the chip system on the host. The chip system includes a host system and a network card system. The host system includes a processor. A virtual machine and a virtual machine management system run on the processor. After being controlled and managed by the virtual machine management system, the virtual machine interacts with other service hosts through the network card chip. Usually, in the process of service interaction, the functions of some network protocol layers (such as the transport layer and the network layer) in the network model are set in the processor, and the processor encapsulates and parses the messages in the process of service interaction. However, the encapsulation and parsing of the messages will greatly reduce the processing performance of the processor and the virtual machine.

[0003] In order to improve the processing and computing performance of the processor, the SR-IOV technology can be used to make the virtual machine without the control and management of the virtual machine management system, and the virtual machine can communicate directly with the network card system. At the same time, the functions of some network protocol layers (such as the transport layer and the network layer, and even the network protocol layer above the transport layer) or all network protocol layers are unloaded from the processor of the host system to the network card system. Through the network card system, business interaction is carried out based on some state information. These state information is used to help the network card system to encapsulate and parse the interactive messages. This business interaction mode can be called stateful business interaction. In stateful business interaction, after the network card system completes the processing of relevant business information, it needs to output the completion signal corresponding to the relevant business to the host system. When the virtual machine in the host system obtains the completion signal, it can be confirmed that the relevant business information has been processed. For the solution of the virtual machine directly passing through the network card system, because during the hot migration of the virtual machine, the virtual machine continues to interact with the business of the business host. Therefore, it cannot be guaranteed that the completion signals of all initiated business information have been transmitted to the virtual machine before the state information of the virtual machine is migrated to the destination host. Therefore, when a virtual machine is directly connected to a network card system for stateful business interaction, how to hot migrate the virtual machine has always been a difficult problem in the industry. Even in many network models, there is no solution for hot migration of the virtual machines corresponding to the direct network card system. Summary of the invention

[0004] The embodiments of the present application provide a virtual machine migration method, a chip system, and a host, which realize hot migration of virtual machines performing stateful business interactions when the virtual machines are directly connected to a network card system.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a virtual machine migration method, which is based on a chip system. The chip system includes a network card system and a host system. A virtual machine and a virtual machine management system are running on the host system. The method includes: the network card system migrates first dirty page data from a first storage area of ​​the host system to a destination host. The first storage area is used to store memory data of the virtual machine. The first dirty page data is dirty page data in the memory data. In the process of migrating the first dirty page data, the network card system stops interacting with at least one business host for subsequent initiated business information according to the first indication information, and interacts with at least one business host based on the initiated business information. The business information is used to indicate the business processing between the virtual machine and at least one business host. After migrating the first dirty page data, the virtual machine management system controls the virtual machine to shut down, and migrates the state data of the virtual machine to the destination host based on the first transmission connection. The state data includes the business interaction state of the virtual machine and at least one business host.

[0007] In the process of iteratively copying the first dirty page data, the embodiment of the present application stops interacting with at least one business host for the subsequently initiated business information, and interacts with at least one business host based on the business information that has been initiated. On the one hand, it can ensure that no new business information is initiated within a period of time before the virtual machine is shut down, so as to avoid the situation where the completion information corresponding to the newly initiated business information has not been transmitted to the virtual machine when the virtual machine is shut down. On the other hand, the business information that has been initiated previously can be processed during the period from the cessation of initiating new business information to the shutdown of the virtual machine, so as to ensure that the completion signal corresponding to this part of the business information has been delivered to the virtual machine when the virtual machine is shut down. The embodiment of the present application realizes direct connection of the virtual machine based on the above-mentioned scheme, and the virtual machine successfully performs hot migration of the virtual machine during the process of stateful business interaction with the business host.

[0008] In a possible implementation, after the migration of the first dirty page data, the virtual machine management system controls the virtual machine to shut down, including: the network card system verifies whether all the initiated business information has been processed. When all the initiated business information has been processed, a shutdown confirmation signal is output to the virtual machine management system of the host system. The shutdown confirmation signal is used to indicate that the initiated business information has been processed. The virtual machine management system controls the virtual machine to shut down according to the shutdown confirmation signal. In an embodiment of the present application, in order to ensure that all the initiated business information has been processed before the virtual machine is shut down. After the migration of the first dirty page data, the completion of the processing of the business information can be verified. When it is verified that all the initiated business information has been completed, the virtual machine is shut down to ensure the success of the hot migration of the virtual machine.

[0009] In a possible implementation, the business information includes first business information. The first business information is information initiated by at least one business host to the virtual machine through the network card system. The above verification of whether all initiated business information has been processed includes: the network card system generates a first write instruction according to the first business information. The first write instruction is used to write new first dirty page data to the first storage area. According to the first write instruction, a corresponding first verification instruction is generated. The first verification instruction is used to verify the reading of new first dirty page data from the first storage area. If the new first dirty page data is verified to be read from the first storage area, the network card system outputs the corresponding first business completion notification to the host system. In an embodiment of the present application, the first business information is business information executed on the virtual machine side. The first business information may include control indication information and data processing indication information. When the first business information includes data processing indication information, and the data processing indication information is to write data processing to the first storage area of ​​the virtual machine, a first write instruction is generated to perform the write data processing operation. The first write instruction causes the first storage area to generate new memory data, i.e., the first dirty page data. In order to verify that the first business information has been processed, after executing the first write instruction, a first verification instruction may be generated to read the first storage area to verify whether the first dirty page data has been processed.

[0010] In a possible implementation, the business information includes second business information. The second business information is information initiated by the virtual machine to at least one business host through the network card system. The above verification of whether all initiated business information has been processed includes: the network card system: when sending a business message to at least one business host according to the second business information, the first count value is increased by the first value; when receiving the first response information, the first count value is reduced by the first value; the first response information is used to indicate that at least one business host has processed the corresponding second business information; when the size of the first count value is the initial value, the second verification completion signal is output to the virtual machine management system; the second verification completion signal is used to indicate that the corresponding second business information has been processed. In an embodiment of the present application, the network card system sends a business message to the business host side according to the second business information. The initiation and completion of the second business information can be recorded in a counting manner. Subsequently, it is verified based on the count value whether all second business information has been completed.

[0011] In a possible implementation, after the first dirty page data is migrated, the virtual machine management system controls the virtual machine to shut down, and further includes: after the first dirty page data is migrated, the host system outputs a shutdown preparation signal to the network card system based on the virtual machine management system. The shutdown preparation signal is used to indicate that the first dirty page data has been migrated. The network card system, based on the shutdown preparation signal, verifies whether all initiated business information has been processed. In an embodiment of the present application, a shutdown preparation signal can be generated by the virtual machine management system to instruct the network card system to start verifying the completion of the initiated business information.

[0012] In a possible implementation, the method further includes: when the data volume of the first dirty page data is less than the first threshold value, the virtual machine management system outputs a first indication signal to the network card system. In an embodiment of the present application, when the data volume of the first dirty page data is less than the first threshold value, it means that the business interaction volume is less than a certain level. At this time, shutting down the interactive business data has a small impact on the business interaction, and the data volume of the first dirty page data can be gradually reduced to quickly shut down and complete the hot migration of the virtual machine. Avoid long-term business interaction of the virtual machine, and fail to shut down the virtual machine for a long time, which leads to the failure of hot migration.

[0013] In some possible implementations, the method further includes: after the state data is transmitted based on the first transmission connection, the virtual machine management system sends an activation signal to the destination host based on the first transmission connection, and restarts the virtual machine on the destination host through the activation signal.

[0014] In a possible implementation, the method further includes: when the initiation rate of the second business information is greater than the second threshold, or when the initiation amount of the second business information is greater than the third threshold, the virtual machine management system outputs a first indication signal to the network card system. In an embodiment of the present application, when the initiation rate or initiation amount of the second business information reaches a certain level, it means that there is a lot of interaction of business information. In order to avoid the increase of more business information, resulting in a long-term inability to shut down the virtual machine, the interactive business information can be stopped in this case. Compared with the method of stopping business interaction when the amount of data of the first dirty page data is less than the first threshold, the hot migration speed of the virtual machine can be accelerated through the embodiment of the present application.

[0015] In a possible implementation, the virtual machine management system is also used to implement network protocol processing at the transport layer. The method also includes: before the virtual machine management system controls the virtual machine to shut down, the virtual machine management system sends a first connection request to the destination host through the network card system. The first connection request is used to instruct the destination host to establish a first transmission connection with the host system. In an embodiment of the present application, when the transport layer is presented on the virtual machine management system, in order to avoid establishing the first transmission connection after the shutdown, resulting in a long time for the shutdown migration status data and other processes, the first transmission connection can be established before the virtual machine is shut down to ensure that the shutdown phase is completed as quickly as possible to avoid the impact of too long shutdown on business interactions.

[0016] In a possible implementation, the virtual machine management system is also used to implement network protocol processing for the transport layer. The method also includes: the virtual machine and at least one service host interact with each other regarding service information based on the second transmission connection. The second transmission connection is a transport layer connection channel exclusively used by the virtual machine. In an embodiment of the present application, when the transport layer is set on the virtual machine management system, multiple transport layer connection channels can be established on the virtual machine management system. Multiple virtual machines on the host system can use a transport layer connection channel together for service interaction. At this time, an exclusive transport layer connection channel can be set for the virtual machine that needs to be hot migrated. The virtual machine that is hot migrated uses the exclusive transport layer connection channel as the first transmission channel, and the service interaction based on the first transmission channel and the service host can avoid the service blocking problem caused by the service information of other virtual machines sharing the transport layer connection channel, thereby speeding up the service interaction progress of the hot migrated virtual machine.

[0017] In a second aspect, an embodiment of the present application further provides a chip system, which includes a host system and a network card system. A virtual machine and a virtual machine management system are running on the host system. Among them: the network card system is used to: migrate the first dirty page data from the first storage area of ​​the host system to the destination host. The first storage area is used to store the memory data of the virtual machine. The first dirty page data is the dirty page data in the memory data. In the process of migrating the first dirty page data, according to the first indication information, stop the interaction of the subsequent initiated business information with at least one business host, and interact with at least one business host based on the initiated business information. The business information is used to indicate the business processing between the virtual machine and at least one business host. The host system is used to: after migrating the first dirty page data, control the virtual machine to shut down based on the virtual machine management system. The network card system is also used to: after the virtual machine is shut down, migrate the state data of the virtual machine to the destination host based on the first transmission connection. The state data includes the business interaction state of the virtual machine and at least one business host.

[0018] In a possible implementation, the network card system is further used to: verify whether all initiated service information has been processed. When all initiated service information has been processed, a shutdown confirmation signal is output to the virtual machine management system of the host system. The shutdown confirmation signal is used to indicate that the initiated service information has been processed. The host system is further used to: control the virtual machine to shut down according to the shutdown confirmation signal based on the virtual machine management system.

[0019] In a possible implementation, the business information includes first business information. The first business information is information initiated by at least one business host to the virtual machine through the network card system. The network card system is also used to: generate a first write instruction based on the first business information. The first write instruction is used to write new first dirty page data to the first storage area. According to the first write instruction, a corresponding first verification instruction is generated. The first verification instruction is used to verify and read new first dirty page data from the first storage area. If the new first dirty page data is verified and read from the first storage area, a first verification completion signal is output to the virtual machine management system. The first verification completion signal is used to indicate that the corresponding first business information has been processed.

[0020] In a possible implementation, the business information includes second business information. The second business information is information initiated by the virtual machine to at least one business host through the network card system. The network card system is also used to: when sending a business message to at least one business host according to the second business information, increase the first count value by a first value. When receiving the first response information, reduce the first count value by a first value. The first response information is used to indicate that at least one business host has processed the corresponding second business information. When the size of the first count value is the initial value, a second verification completion signal is output to the virtual machine management system. The second verification completion signal is used to indicate that the corresponding second business information has been processed.

[0021] In a possible implementation, the host system is further used to: after migrating the first dirty page data, based on the virtual machine management system, output a shutdown preparation signal to the network card system. The shutdown preparation signal is used to indicate that the first dirty page data has been migrated. The network card system is also used to: according to the shutdown preparation signal, execute verification whether all initiated business information has been processed.

[0022] In a possible implementation, the host system is further configured to: when the data volume of the first dirty page data is less than a first threshold, output a first indication signal to the network card system based on the virtual machine management system.

[0023] In a possible implementation, the host system is further configured to: output a first indication signal to the network card system based on the virtual machine management system when the initiation rate of the second service information is greater than a second threshold, or when the initiation amount of the second service information is greater than a third threshold.

[0024] In a possible implementation, the virtual machine management system is further used to implement network protocol processing at the transport layer. The host system is further used to: before controlling the virtual machine to shut down based on the virtual machine management system, send a first connection request to the destination host through the network card system based on the virtual machine management system. The first connection request is used to instruct the destination host to establish a first transport connection.

[0025] In a possible implementation, the virtual machine management system is further used to implement network protocol processing at the transport layer. The host system is further used to control the virtual machine and at least one service host to exchange service information based on the second transport connection. The second transport connection is a transport layer connection channel exclusively used by the virtual machine.

[0026] In a third aspect, an embodiment of the present application further provides a host, which includes a circuit board and a chip system as described in the second aspect.

[0027] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, the computer-readable storage medium comprising instructions. When the instructions are executed on the chip system described in the second aspect, the chip system executes the virtual machine migration method described in the first aspect. When the instructions are executed on the host of the third aspect, the host executes the virtual machine migration method described in the first aspect.

[0028] For the description of the technical principles and beneficial effects of the second, third and fourth aspects, reference can be made to the relevant description of the first aspect above, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of the structure of a host;

[0030] Figure 2 The schematic diagram is a schematic diagram of an architecture of a virtualized network card system directly connected to a virtual machine;

[0031] Figure 3 A schematic diagram of a structure of an interactive architecture including multiple hosts provided in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of a migration process of a first virtual machine hot migration failure provided in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of an interaction process for business interaction between multiple hosts when a transport layer and a transaction layer are separated provided in an embodiment of the present application;

[0034] Figure 6 A flow chart of a virtual machine migration method provided in an embodiment of the present application Figure 1 ;

[0035] Figure 7Another virtual machine migration method provided in this application embodiment Figure 2 ;

[0036] Figure 8 A schematic diagram of an interaction process for business interaction between multiple hosts provided in an embodiment of the present application;

[0037] Fig. 9 A flow chart of another virtual machine migration method provided in the embodiment of the present application Figure 3 ;

[0038] Fig.10 A flow chart of another virtual machine migration method provided in the embodiment of the present application Figure 4 ;

[0039] Fig.11 A flow chart of another virtual machine migration method provided in the embodiment of the present application Figure 5 . DETAILED DESCRIPTION

[0040] It should be noted that the terms "first", "second", etc. involved in the embodiments of the present application are only used to distinguish features of the same type and cannot be understood as indicating relative importance, quantity, order, etc.

[0041] The terms "exemplary" or "for example" and the like in the embodiments of the present application are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the terms "exemplary" or "for example" is intended to present the related concepts in a specific way.

[0042] The terms "coupling" and "connection" involved in the embodiments of the present application should be understood in a broad sense. For example, they may refer to a direct physical connection, or an indirect connection achieved through electronic devices, such as a connection achieved through resistors, inductors, capacitors or other electronic devices.

[0043] First, some basic concepts involved in the embodiments of the present application are explained:

[0044] The service interaction between network nodes implements message encapsulation and parsing based on the network model. Depending on the network model, the corresponding message encapsulation and parsing are also different. The three common network models are the OSI seven-layer network model, the TCP five-layer network model, and the TCP / IP four-layer network model. The OSI seven-layer network model includes the physical layer, data link layer, network layer, transport layer, session layer, presentation layer, and application layer from bottom to top. The TCP five-layer network model includes the physical layer, data link layer, network layer, transport layer, and application layer from bottom to top, among which the application layer of the TCP five-layer network model corresponds to the session layer, presentation layer, and application layer of the OSI seven-layer network model. The lower four layers of the TCP five-layer network model are consistent with the OSI seven-layer network model. The TCP / IP four-layer network model includes the network access layer, network interconnection layer, transport layer, and application layer from bottom to top. Among them, the network access layer of the TCP / IP four-layer network model corresponds to the physical layer and data link layer of the OSI seven-layer network model; the internetworking layer of the TCP / IP four-layer network model corresponds to the network layer of the OSI seven-layer network model; the application layer of the TCP / IP four-layer network model corresponds to the session layer, presentation layer and application layer of the OSI seven-layer network model. In the three network models, the network model layers above the transport layer are used for transaction processing by APP users. We can call the network model layers above the transport layer in each network model the transaction layer.

[0045] The host includes a circuit board and a chip system. The chip system is arranged on the circuit board. The chip system includes a host system 1 and a network card system 2. Figure 1 As shown, the host system 1 includes a processor 11 and a host memory 12. The network card system 2 can access the host memory 12 of the host system 1 through a data link based on direct memory access (DMA) technology, and exchange control signals with the processor 11 of the host system 1 through a control link. Each host can serve as a network node. Business interaction can be performed between multiple network nodes. In some possible implementations, the transaction layer is presented in the user state of the processor 11 of the host system 1, and the transport layer and the network layer are presented in the kernel state of the processor 11 of the host system 1. The network model layer below the network layer is presented in the network card system 2. At this time, the interaction of message messages between the two network nodes still relies heavily on the processor 11 of the host system 1 to encapsulate and parse the message messages based on the network protocol layer (such as the transport layer and the network layer, and the network protocol layer above the transport layer). In order to improve the processing and computing performance of the processor 11, the industry has proposed TCP Offload Engine (TOE), remote direct memory access (RDMA) and unified bus (UB) protocols.

[0046] TOE unloads protocols such as the transport layer and the network layer from the processor network of the host system to the network card system, and presents the transport layer and the network layer on the network card system, and encapsulates and parses the message based on the network card system to improve the work efficiency of the processor of the host system. TOE technology can be applied to network nodes on one side. RDMA also unloads protocols such as the transport layer and the network layer from the processor network of the host system to the network card system. The kernel bypass mechanism of RDMA enables the APP and the network card system to directly read and write memory data on the data link. The zero copy mechanism of RDMA enables direct memory data reading between two network nodes, which can reduce the operating burden of the processor and improve processing efficiency. However, in the control link, a small number of control signals still need to be processed by the kernel state of the processor. RDMA is a technology that allows the local network node to directly access the remote network node, which requires the application of RDMA technology on both sides of the network nodes to achieve. In RDMA, the APP can access the memory of the remote network node, and the remote network node can be read without the participation of the process or processor on the remote network node. Common RDMA technologies include RDMA over converged ethernet (RoCE) protocol based on Ethernet. UB and RDMA have similar implementation functions. One difference between UB and RDMA is that the transaction layer and the transport layer are separated.

[0047] Network nodes can perform a variety of communication interaction operations based on RDMA protocol, UB protocol, etc. Taking RDMA protocol as an example, two basic communication interaction operations can be performed: memory semantics (memory verbs) operation and channel semantics (messaging verbs) operation. Memory verbs operation belongs to unilateral communication interaction operation, which only requires the local network node to clarify the source address and destination address of the message, and the processor 11 of the remote network node does not need to perceive this unilateral communication interaction operation. The data reading operation and data writing operation of the local network node on the remote network node are completed by the network card system 2 of the remote network node based on DMA operation, and then the network card system 2 of the remote network node encapsulates and returns the relevant message to the network card system 2 of the local network node, and the network card system 2 of the local network node writes data to the host memory 12 of the local end and / or feedbacks the relevant transaction completion information to the processor 11 of the local end. Memory verbs operation includes RDMA read (read) operation, RDMA write (write) operation and RDMA atomic (atomic) operation. Atomic operation refers to an operation that is not interrupted, that is, it is the smallest execution unit. The simplest atomic operation is a series of assembly instructions. Specifically, RDMA atomic operations can include atomic fetch operations, atomic addition operations, atomic comparison operations, atomic swap operations, etc. Messaging verbs operations are bilateral operations, that is, the communication interaction operations can only be completed when both the local network node and the remote network node's APPs are aware of and participate. Messaging verbs operations include RDMA send operations and RDMA receive operations. The send operation allows the local network node to send data to the receive queue of the remote network node. The remote network node as the receiving end needs to have pre-registered the buffer for receiving data. The local network node as the sending end cannot control the placement of data in the remote network node. In the RDMA send operation, you can choose whether to use an immediate number, transmit the immediate number and data to the remote network node together, and send the immediate number to the receiving end as a notification of receipt. The RDMA receive operation is the operation corresponding to the RDMA send operation. The functions of the relevant operations of the UB protocol are similar to those of the aforementioned RDMA protocol, and will not be repeated here.

[0048] Virtualization is the foundation of cloud computing technology. In cloud computing technology, the physical hardware resources (including computing resources, storage resources, and network resources, etc.) of the host (such as a server device) are virtualized through virtualization technology to obtain multiple virtual machines (VMs) that run independently and do not interfere with each other. VM is an independent virtual processing device built using the computing resources, storage resources, and network resources of the host. Before the host is virtualized, its computer system architecture can be divided from top to bottom into the application layer, the operating system (OS) layer, and the hardware layer. The operating system layer is also called the host OS layer. Among them, the hardware layer is based on the electronic devices of the host (such as the host system 1 including the processor 11 and the host memory 12, and the network card system 2), providing computing resources, storage resources, and network resources for the upper architecture. The operating system layer can run in the kernel state of the host. The application layer can run in the user state of the host. After the host is virtualized, the resource management layer and the virtualization layer are virtualized on the hardware layer. There is at least one VM in the virtualization layer. Each VM includes a client operating system (guest OS) and an APP. A virtual machine manager (VMM) is set in the resource management layer to manage each VM. In some scenarios, VMM can also be called Hypervisor. Different types of operating systems correspond to different Hypervisors.

[0049] Virtualization pass-through means that the VM can communicate directly with the network card system 2. One virtualization pass-through method is to allocate the actual network card system 2 to the VM on the physical architecture. Another virtualization pass-through method is single root I / O virtualization (SR-IOV). SR-IOV is a standard for sharing PCIe interface functions with virtual machines. SR-IOV bypasses the VMM to achieve data access by providing independent memory space, interrupts and DMA streams for VMs. The SR-IOV protocol introduces the concept of two types of functions: physical function (PF) and virtual function (VF). PF contains complete PCIe functions, including the expansion capability of SR-IOV, and PF functions can be used for the configuration and management of SR-IOV. VF contains lightweight PCIe functions. Each VF has its own exclusive PCI configuration area and may share the same physical resources with other VFs. The PF driver varies with different hypervisors, and generally requires higher permissions than ordinary VMs to operate the PF driver. The PF driver includes all the functions of traditional data interface drivers, allowing the hypervisor to access the host's I / O resources. You can call the PF driver to perform related operations that affect the entire host. Figure 2 As shown, the network card system 2 can provide multiple PCIe interfaces to the host system 1 side through the SR-IOV technology. The PCIe interface in the network card system 2 that directly interacts with the PF driver of the processor 11 is the PF interface. The PCIe interface in the network card system that directly connects to the VM in the host system 1 is the VF interface. The SR-IOV technology can be regarded as the virtualization of the network card system 2. The direct connection between the VM and the network card system 2 can maximize the performance of the VM's related data communication and processing.

[0050] Stateless service interaction means that when two network nodes exchange messages, the network nodes' processing of the messages does not depend on specific state information and can process the messages independently without being affected.

[0051] Stateful service interaction means that when two network nodes exchange messages, there is a dependency relationship between the processing of different messages. The processing of these messages needs to rely on a specific working state. In actual processing, it is necessary to rely on relevant state information to process the message. And in the process of processing the message, the content of the state information may change accordingly. In the process of a VM of a certain network node performing service interaction with other network nodes Q1 based on the network card system, these state information may be address mapping relationships, related network protocol control information, and network state control information. The address mapping relationship may include mapping relationships of addresses such as guest virtual address (GVA), guest physical address (GPA), actual virtual address (HVA), and actual physical address (HPA). Generally, these state information can be stored in the form of a mapping state table.

[0052] Virtual machine migration, virtual machine migration technology is to move the VM running on the source host to the destination host, and enable the moved VM to run normally on the destination host. A complete virtual machine computer system consists of a virtual processor, virtual memory, virtual disk, virtual network card, etc., among which virtual disk and virtual network card are both virtual I / O devices. Virtual machine migration technology can migrate the virtual processor state information, virtual memory data information and virtual I / O device state information in the VM from a physical electronic device (such as the source host) to another physical host (such as the destination host). Virtual machine migration technology is widely used in the server cluster management of cloud computing data centers, and can achieve load balancing, disaster recovery and online upgrade of server clusters. Through virtual machine migration, the operating cost of cloud data centers can be reduced and the management efficiency of server clusters can be improved. Virtual machine migration is divided into static migration and dynamic migration. Static migration, also known as cold migration, refers to shutting down the VM on the source host and then migrating the memory data and state data of the VM on the source host to the destination host. Among them, the state data includes business interaction state information, virtual processor state information and virtual I / O device state information. Then, the VM is started and run on the destination electronic device. Cold migration usually requires interrupting the VM service for several minutes or even longer. Live migration, also known as hot migration, means that only a short downtime is required to migrate part of the VM data, and other data is dynamically copied when the VM is running normally.

[0053] The present application embodiment proposes an interactive architecture, such as Figure 3As shown, the interactive architecture includes a source host A, a destination host B and at least one service host C. The source host A includes a first chip system. The destination host B includes a second chip system. The service host C includes a third chip system. Among them:

[0054] The first chip system of the source host A includes a first host system A1 and a first network card system A2. The first host system A1 includes a first processor A11 and a first host memory A12. At least one VM and a first virtual machine management system VMM1 are running on the first processor A11. At least one VM includes a first virtual machine VM1. The first host memory A12 is used to store memory data of the first virtual machine VM1. According to different actual application scenarios, the state information of the first virtual machine VM1 can be stored in the first host memory A12, the cache of the first processor A11 and at least one device in the network card system A2. The devices in the source host A are coupled through the first bus BUS1.

[0055] The second chip system of the destination host B includes a second host system B1 and a second network card system B2. The second host system B1 includes a second processor B11 and a second host memory B12. The components in the destination host B are coupled via a second bus BUS2.

[0056] The third chip system of the service host C includes a third host system C1 and a third network card system C2. The third host system C1 includes a third processor C11 and a third host memory C12. The components in each service host C are coupled via a third bus BUS3.

[0057] In some possible implementations, when the first virtual machine VM1 is not directly connected to the first network card system A2, the functions of the transport layer and the network layer are presented in the first processor A11. The first network card system A2 only performs stateless information processing. The first virtual machine VM1 can be hot migrated completely based on the software level. During the hot migration process, the memory data is usually copied and migrated in the dynamic copy stage. The processor state information and I / O device state information are copied and migrated in a short time during the shutdown stage. The hot migration process will not cause the interruption of the business running on the virtual machine, and compared with cold migration, the availability of the service can be improved. For the case where the first virtual machine VM1 is not directly connected to the first network card system A2, there are many ways to implement hot migration. For example, in the virtualization technology based on KVM-QEMU, the pre-copy method is mainly used to implement hot migration. Pre-copy can be divided into: migration initialization stage, iterative copy stage, shutdown copy stage and restart stage in chronological order.

[0058] First, in the migration initialization phase, hot migration related initialization is performed, and all memory pages of the first virtual machine VM1 are marked dirty. The migration initialization phase usually lasts for a short time, and sometimes this phase can be ignored.

[0059] Next, in the iterative copy phase, the first network card system A2 starts to copy the memory data of the first virtual machine VM1 and migrate it to the destination host B. In this process, the source host will also interact with at least one service host C. These business interactions may modify the memory data of the first virtual machine VM1, so new memory data will be iteratively generated during the migration process. The first network card system A2 will mark the new memory data dirty during the interaction process to obtain the first dirty page data, and iteratively migrate these first dirty page data. When the data volume of the first dirty page data is less than the shutdown threshold, the iterative copy phase ends.

[0060] Then, in the shutdown copy phase, the first virtual machine VM1 is shut down. The first network card system A2 performs shutdown migration on the remaining first dirty page data and the state data of the first virtual machine VM1. After the shutdown migration of the remaining first dirty page data and the state data is completed, the shutdown copy phase ends.

[0061] Finally, in the restart phase, the memory data and state information of the first virtual machine VM1 are run on the destination host B to restart the first virtual machine VM1.

[0062] In the pre-copy hot migration, the first virtual machine VM1 is in the running stage during the migration initialization stage, the shutdown copy stage and the restart stage. It will only be shut down for a period of time during the iterative copy stage. The length of this period is determined by the amount of remaining memory data and status information migrated during the shutdown copy stage. It is generally a few milliseconds. However, when the first virtual machine VM1 is directly connected to the first network card system A2, some network layer protocol functions are unloaded from the first processor A11 to the first network card system A2. At this time, it is necessary to perform relevant message encapsulation and parsing processing on the first network card system A2. In the process of the first virtual machine VM1 performing stateful business interaction with the third virtual machine VM3 in the business host C based on the second transmission connection, when the first virtual machine VM1 located in the first host system A1 obtains the completion signal, it can be confirmed that the relevant business information has been processed. However, for the solution of the first virtual machine VM1 directly connected to the first network card system A2, in the process of hot migration of the first virtual machine VM1, it cannot be guaranteed that the completion signals of all initiated business information are transmitted to the first virtual machine VM1 before the first virtual machine VM1 is shut down. Therefore, hot migration of virtual machines in the virtual machine direct network card system solution has always been a difficult problem in the industry, and even many network models do not have hot migration solutions. Figure 3When the first virtual machine VM1 shown is based on RDMA technology (such as RoCE protocol), the hardware based on SR-IOV technology (such as the first network card system A2) accelerates the internal processing matters of the first virtual machine VM1, but this makes the business interaction status information of the first virtual machine VM1 and at least one business host C exist not only in the software layer of the first host system A1, but also in the hardware of the first network card system A2, such as the address mapping relationship between the first virtual machine VM1, the first processor A11 and the first host memory A12 (such as the mapping relationship of GVA, GPA, HVA and HPA addresses) and the ongoing protocol control information and network status control information of the first network card system A2. Because in the process of business interaction between the first virtual machine VM1 and at least one business host C, many transactions cannot be initiated repeatedly, such as RDMA send operations and RDMA atomic operations, etc., and the repeated initiation of these operations will have a great impact on the operation of the first virtual machine VM1. When the aforementioned pre-copy hot migration method is adopted, such as Figure 4 As shown, there may be a situation where the hot migration shown in steps S1' to S4' fails as follows:

[0063] S1', initiating an atomic add operation: the first virtual machine VM1, in the iterative copy phase, initiates an atomic add operation to a third virtual machine VM3 of at least one service host C.

[0064] S2', the migration of the first virtual machine VM1 is completed: the iterative copy phase ends, and the first virtual machine VM1 is shut down. In the shutdown copy phase, the first network card system A2 begins to migrate the remaining memory data and status information of the first virtual machine VM1. In the restart phase, after migrating to the destination host B, the first virtual machine VM1 will think that the atomic plus operation is not completed. After the first virtual machine VM1 is shut down, the message of successful atomic plus operation processing reaches the transport layer of the first network card system A2, or reaches the kernel state of the processor A11 but does not reach the APP of the first virtual machine VM1. At this time, the first virtual machine VM1 is hot migrated to the destination host B.

[0065] S3', atomic plus operation completion notification: When the first virtual machine VM1 is shut down, the message of the successful atomic plus operation processing reaches the transport layer of the first network card system A2, or reaches the kernel state of the processor A11 but does not reach the APP of the first virtual machine VM1. After the first virtual machine VM1 is shut down, the message of the successful atomic plus operation processing reaches the source host A, but at this time, the first virtual machine VM1 on the source host A has been shut down, and the memory data and status data are no longer updated.

[0066] S4', repeatedly initiate the atomic add operation: restart the running first virtual machine VM1 on the destination host B. The first virtual machine VM1 on the destination host B has not obtained the message of successful processing of the atomic add operation from the first network card system A2. The first virtual machine VM1 on the destination host B will re-initiate the atomic add operation to at least one business host C. For at least one business host C, the atomic add operation is executed twice, while for the first virtual machine VM1, the atomic add operation is only executed once. This will cause abnormal operation of the first virtual machine VM1.

[0067] Another example: When Figure 3 The first virtual machine VM1 shown is based on the UB protocol. Figure 5 As shown, at least one service host C includes a third host system C1 and a third network card system C2. A third virtual machine VM3 runs on the third host system C1. In the UB protocol, the transport layer and the transaction layer are separated. The transaction layer is presented to the first virtual machine VM1. The transport layer of the source host A is set on the first virtual machine management system VMM1. The transport layer of at least one service host C is set on the second source host A. The transport layer is set on the third virtual machine management system VMM3. When hot migration is performed on the first virtual machine VM1, only the transaction layer of the first virtual machine VM1 needs to be hot migrated, and the transport layer located in the first virtual machine management system VMM1 is not hot migrated. The first virtual machine management system VMM1 establishes a first transmission connection with the third virtual machine management system VMM3 based on the first network card system A2 and the third network card system C2. The first transmission connection can be established on the first virtual machine management system VMM1 and the third virtual machine management system VMM3, or the first transmission connection can be established on the first network card system A2 and the third network card system C2. When the aforementioned pre-copy hot migration method is adopted, during the process of business interaction between the first virtual machine VM1 and the third virtual machine VM3, the first virtual machine VM1 sends relevant business information, such as work queue element (WQE), to the first virtual machine management system VMM1. The relevant business information is used to indicate the atomic add operation for the third virtual machine VM3. If the first network card system A2 completes the task of the relevant business information and outputs the corresponding completion information to the first virtual machine management system VMM1, but the relevant completion information is not transmitted to the first virtual machine VM1, the first virtual machine VM1 enters the shutdown copy stage. After the first virtual machine VM1 is restarted on the destination host B, it also believes that the atomic add operation has not been successfully executed, and will re-send the relevant business information to perform the atomic add operation.

[0068] In some possible implementations, Figure 3In the architecture shown, when the first virtual machine VM1 is directly connected to the first network card system A2:

[0069] The first network card system A2 is used to migrate the first dirty page data from the first storage area of ​​the first host system A1 to the destination host C. The first storage area is used to store the memory data of the first virtual machine VM1. The first dirty page data is the dirty page data in the memory data. In the process of migrating the first dirty page data, according to the first indication information, the interaction of the subsequent initiated business information with at least one business host C is stopped, and the interaction with at least one business host C is performed based on the initiated business information. The business information is used to indicate the business processing between the first virtual machine VM1 and at least one business host C.

[0070] The first host system A1 is used for controlling the first virtual machine VM1 to shut down based on the first virtual machine VM1 management system VMM1 after migrating the first dirty page data.

[0071] The first network card system A2 is also used to: after the first virtual machine VM1 is shut down, migrate the state data of the first virtual machine VM1 to the destination host C based on the first transmission connection; the state data includes the business interaction state of the first virtual machine VM1 and at least one business host C.

[0072] In an embodiment of the present application, during the iterative copying of the first dirty page data in the memory data of the first virtual machine VM1, the first network card system A2 is controlled to stop interacting with at least one service host C for service information. After the iterative copying of all the first dirty page data is completed, the first virtual machine VM1 is shut down, and the status information of the first virtual machine VM1 is migrated. In the embodiment of the present application, during the iterative copying of the first dirty page data, new service information is stopped from being initiated. On the one hand, it can be ensured that no new service information is initiated within a period of time before the first virtual machine VM1 is shut down, so as to avoid the situation where the completion information corresponding to the newly initiated service information before the shutdown is not transmitted to the first virtual machine VM1 when the first virtual machine VM1 is shut down. On the other hand, the service information that has been initiated previously can be processed during the period from the cessation of initiating new service information to the shutdown of the first virtual machine VM1, so as to ensure that the completion signal corresponding to this part of the service information has been delivered to the first virtual machine VM1 when the first virtual machine VM1 is shut down.

[0073] In some possible implementations, a fixed time parameter may be set according to a business scenario of actual application, and the time parameter is used to confirm the moment to stop business interaction during the process of iteratively copying the first dirty page data.

[0074] In some possible implementations, changes in relevant parameters in hot migration can be used as judgment conditions to confirm the moment to stop business interaction during the iterative copying of the first dirty page data. In some examples, when the amount of data of the first dirty page data is less than a first threshold, it can be confirmed that it is necessary to stop initiating new business information. At this time, the first host system A1 outputs a first indication signal to the first network card system A2 based on the virtual machine management system VMM1. In some examples, when the initiation rate of business information is greater than the second threshold, or when the initiation amount of business information is greater than the third threshold, it is confirmed that it is necessary to stop initiating new business information. At this time, the first host system A1 outputs a first indication signal to the first network card system A2 based on the virtual machine management system VMM1.

[0075] In some possible implementations, the first network card system A2 is also used to verify whether all initiated business information has been processed. When all initiated business information has been processed, a shutdown confirmation signal is output to the first virtual machine VM1 management system VMM1 of the first host system A1. The shutdown confirmation signal is used to indicate that the initiated business information has been processed. The first host system A1 is also used to control the shutdown of the first virtual machine VM1 according to the shutdown confirmation signal based on the first virtual machine VM1 management system VMM1. In an embodiment of the present application, the completion of the processing of the initiated business information can be verified after all the first dirty page data has been iteratively copied. After verifying that all the initiated business information has been processed, the first virtual machine VM1 is shut down. The embodiment of the present application can further ensure the accuracy of hot migration by verifying the processing of business information.

[0076] The present application also provides a virtual machine migration method for Figure 3 The interactive architecture shown in the figure is to hot migrate the first virtual machine VM1 from the source host A to the destination host B based on the second transmission connection. During the hot migration process, the source host A may perform business interaction with at least one business host C based on the first transmission connection. The virtual machine migration method includes the following steps: Figure 6 The operations of step S100 to step S400 shown are:

[0077] S100 , start migrating the first dirty page data of the first virtual machine VM1 , and stop initiating new service information with the service host C.

[0078] In some possible implementations, after the first virtual machine VM1 in the first processor A11 initiates the hot migration process, a data transmission connection is established with the destination host B. In the first period, the source host A starts to perform iterative copy migration on the memory data of the first virtual machine VMM1. When the iterative copy migration reaches the first moment in the first period, the source host A stops the business interaction with the business host C. The first moment is a certain time earlier than the end moment of the first period. In the time after the first moment in the first period, the first network card system A2 only needs to process the initiated business information and make the relevant completion information reach the first virtual machine VM1 before the first virtual machine VM1 is shut down. In the embodiment of the present application, a time point is set in the first period of the iterative copy phase, that is, the first moment. After reaching the first moment, the first network card system A2 is controlled not to accept new interaction information, and the interaction information obtained from the first host system A1 and / or the business host C before the first moment. In the first period, all the acquired interaction information is processed and the corresponding completion information is fed back to the first virtual machine VM1, and then the iterative copy phase is stopped. At this time, there is no unobtained completion information before the first virtual machine VM1 is shut down. It can be shut down normally.

[0079] Exemplarily, the first moment may be a fixed time parameter set according to a business scenario of actual application, and the time parameter is used to confirm the moment of stopping business interaction during the process of iteratively copying the first dirty page data.

[0080] Exemplarily, the first moment may be a moment determined as a judgment condition according to a change in a relevant parameter in hot migration. In some examples, the first moment may be a moment when the amount of data of the first dirty page data is less than a first threshold. In some examples, the first moment may be a moment when the initiation rate of the business information is greater than a second threshold.

[0081] In some possible implementations, such as Figure 7 As shown, step S100 may include the following operations of step S110 to step S130:

[0082] S110. The first virtual machine management system VMM1 outputs a migration instruction signal to the first network card system A2.

[0083] Exemplarily, after the first virtual machine VM1 initiates the hot migration process. The hot migration management program running in the first virtual machine management system VMM1 will output a migration indication signal to the first network card system A2. The migration indication signal indicates to the first network card system A2 that the VF channel (the transmission channel between the VF driver and the VF interface) corresponding to the first virtual machine VM1 has entered the hot migration state. After inputting the migration indication signal, the first network card system A2 sets a hot migration label for the VF interface corresponding to the first virtual machine VM1. Step S110 can be regarded as the pre-copy migration initialization stage.

[0084] Exemplarily, the first storage area of ​​the first host memory A12 is used to store memory data of the first virtual machine VM1, etc. Because at the beginning of hot migration, all memory data of the first virtual machine VM1 has not been migrated to the destination host B. Therefore, the first virtual machine management system VMM1 will mark all memory pages in the first storage area of ​​the first host memory A12 dirty during the initialization phase. At this time, all memory data of the first virtual machine VM1 are first dirty page data.

[0085] Exemplarily, after the hot migration tag is set, the first network card system A2 may mark the related memory pages after writing dirty during the subsequent DMA write operation based on the VF interface.

[0086] S120 . The first network card system A2 migrates first dirty page data of the first virtual machine VM1 .

[0087] In some possible implementations, the first network card system A2 hot migrates the first dirty page data in the memory data of the first virtual machine VM1 from the first storage area of ​​the first host memory A12 to the destination host B based on the VF interface corresponding to the first virtual machine VM1 within the first time period according to the migration indication signal.

[0088] Exemplarily, after the first network card system A2 inputs a migration indication signal, the first time period begins. During the first time period, during the first round of iterative copying, the first network card system A2 migrates all memory data of the first virtual machine VM1 as first dirty page data to the destination host B at the destination host B. Because during the first time period, the first virtual machine VM1 was not shut down. All first virtual machines VM1 will also perform business interactions with at least one business host C at a business host C. The business interaction of the first virtual machine VM1 is essentially a DMA operation on the memory data of the first virtual machine VM1. Therefore, in the process of iterative migration of memory data, new memory data will be generated. The memory pages where these new memory data are located are marked dirty to obtain new first dirty page data. In the subsequent iterative migration process, the first dirty page data generated in real time is continuously migrated.

[0089] S130: The first network card system A2 stops initiating new service information.

[0090] In some possible implementations, during the first time period, the first virtual machine VM1 and at least one service host CC continuously initiate service information. At the first moment of the first time period, the first virtual machine management system VMM1 in the first host system A1 can output a first instruction signal to the first network card system A2, and instruct the transmission subnode to stop initiating new service information through the first instruction signal. During the time after the first moment of the first time period, the first network card system A2 processes the service that has been initiated before the first moment.

[0091] In some possible implementations, after receiving the first indication signal, the first network card system A2 sets a first flag for the VF interface corresponding to the first virtual machine VM1. After the first flag is set, service interaction through the VF interface is stopped. For example, after the first moment, for a new first service signal obtained by the first network card system A2 from at least one service host C, the first network card system A2 refuses to receive the signal and sends a rejection signal (such as an RNR status signal) to the corresponding service host C to inform the service host C that the first virtual machine VM1 of the source host A on the local side has not received the relevant service information. At the same time, after the first moment, no service message about the second service signal is sent to at least one service host C.

[0092] S200: Confirm whether all initiated business information has been processed.

[0093] In some possible implementations, at the second moment, the first dirty page data of the first virtual machine VM1 is iteratively migrated, and the first dirty page data does not exist in the first storage area. The first network card system A2 verifies whether all initiated service information has been processed to ensure that the first virtual machine VM1 obtains corresponding completion information for all processed service information.

[0094] In some possible implementations, in some network protocols (such as UB protocol), the transport layer of the source host A (i.e., source host A) is set on the first virtual machine management system VMM1. Because the first period is the period in which the first virtual machine VM1 runs, the first virtual machine VM1 will shut down after the first period. It is necessary to establish a first transmission connection during the first period to quickly migrate the state data of the first virtual machine VM1 after the first virtual machine VM1 shuts down, so as to avoid the problem of extending the shutdown migration time due to establishing the first transmission connection after the shutdown. Exemplarily, during the first period, the first virtual machine management system VMM1 sends a first connection request to the destination host B through the first network card system A2; the first connection request is used to instruct the destination host B to establish a first transmission connection with the first host system A1. The establishment of the first transmission connection can be in the iterative copy stage (i.e., the stage of step S100) within the first period, or in the verification and confirmation stage (i.e., the stage of step S200) within the first period.

[0095] S300 . After the end of the first time period, the first virtual machine VM1 is shut down, and the state data of the first virtual machine VM1 is migrated.

[0096] In some possible implementations, after the first virtual machine management system VMM1 receives a shutdown confirmation signal, the first virtual machine VM is shut down, and all state information of the first virtual machine VM is migrated to the destination host through the first network card system A2 based on the second transmission connection. The state data includes the service interaction state of the first virtual machine VM1 and at least one service host CC.

[0097] For example, according to different actual application scenarios, the state data of the first virtual machine VM1 may be stored in at least one device among the first host memory A12, the cache of the first processor A11, and the first network card system A2.

[0098] S400: Restart and run the first virtual machine VM1 on the destination host B.

[0099] In some possible implementations, after the memory data and state data of the first virtual machine VM1 are migrated on the destination host B, the first virtual machine management system VMM1 sends an activation signal to the destination host B through the first network card system A2 based on the first transmission connection; the activation signal is used to instruct the destination host B to start and run the first virtual machine VM1. Exemplarily, the activation signal can be a doorbell signal. Alternatively, the activation signal can instruct the second virtual machine management system VMM2 on the destination host B to restart the relevant timer of the first virtual machine VM1 to restart the first virtual machine VM1.

[0100] The embodiment of the present application implements hot migration of the first virtual machine VM1 to the destination host B through the above steps S100 to S400 when the first virtual machine VM1 is directly connected to the first network card system A2.

[0101] In some possible implementations, such as Figure 8As shown, the message interaction between the source host A and the service host C needs to be based on the work queue (WQ) and completion queue (CQ) in the first processor A11 and the third processor C11. In the WQ, there is a transaction request issued by the APP, that is, a work queue element (WQE). The first network card system A2 and the third network card system C3 obtain the relevant WQE from the WQ and perform the relevant interaction tasks according to the content indicated by the WQE. After the first network card system A2 and the third network card system C3 complete the relevant WQE, a completion queue element (CQE) will be sent to the CQ of the corresponding processor. CQE is used to indicate that the relevant WQE has been completed. Taking RDMA as an example, two types of WQ can be supported, namely send queue (SQ) and receive queue (RQ). A pair of SQ and RQ can be used as a queue group (queue pair, QP). Each network node (i.e., source host A, destination host B, and service host C, etc.) may include one or more QPs, and the SQ and RQ between multiple QPs may be independent; or, multiple QPs may also be based on a shared receive queue (SRQ) as the RQ. Each time the network card system 2 completes a WQE, it will generate a corresponding CQE to the CQ of the host system 1 at the local end, and indicate that the WQE has been completed through the CQE. Similarly, when an exception occurs in the execution of a WQE, the network card system 2 will also generate a CQE and write it to the host system 1 at the local end to record information such as the execution status of the WQE. Between two network nodes, communication interaction is established in units of QP. At the same time, communication interaction can be connected or unconnected. Connection refers to the establishment of a connection between a pair of QPs at the local end and the remote end. In unconnected communication interaction, a QP in a network node at one end can communicate and interact with any number of QPs in a network node at the other end. In addition, communication interactions (including connection and disconnection) can be either reliable transmission or unreliable transmission. Compared with unreliable transmission, reliable transmission adds a verification link to ensure the transmission reliability of communication interactions. The network layer provides unreliable transmission services. For reliable transmission needs, it is necessary to rely on the network model layer above the network layer (i.e., the transport layer) to implement handshake verification between network nodes based on the transport layer to achieve reliable transmission. When reliable transmission is performed between two network nodes, such as Figure 8 As shown, taking the local source host A sending a data packet message to the remote service host C based on the RDMA send operation as an example, the following steps are included:

[0102] S1. The APP of the service host C sends a WQE1 including an RDMA receive task to the RQ in QP3. WQE1 is used to instruct the third network card system C2 of the service host C to prepare to receive relevant data and put it in the corresponding storage area.

[0103] S2. The APP of the source host A sends a WQE2 including an RDMA send task to the SQ of QP1. WQE2 is used to instruct the first network card system A2 of the source host A to prepare to obtain relevant data from the memory of the source host A and send it to the service host C.

[0104] S3. The first host network card system A2 of the source host A obtains WQE2 from the SQ of QP1.

[0105] S4. The first host network card system A2 of the source host A obtains relevant data from the memory space of the source host A based on WQE2 and a DMA read operation, and performs network protocol encapsulation to obtain a data message.

[0106] S5. The first host network card system A2 of the source host A sends the data packet message to the third network card system C2 of the service host C through the physical link of the network model (eg, the physical layer and data link layer of the OSI model).

[0107] S6. The third network card system C2 of the service host C takes out WQE1 from the RQ of QP3.

[0108] S7. The third network card system C2 of the service host C stores the parsed data in the memory space indicated by WQE1 based on a DMA write operation.

[0109] S8. After receiving the data message, the third network card system C2 of the service host C performs verification and sends a response (ACK) signal to the first network card system A2 of the source host A.

[0110] S9. The third network card system C2 of the service host C generates CQE1 and places CQE1 in CQ3 of the service host C.

[0111] S10. The APP of the service host C learns, based on CQE1, that the task of WQE1 has been completed.

[0112] S11, after obtaining the ACK signal of step S5, the first network card system A2 of the source host A generates CQE2 and puts CQE2 in CQ1 of the source host AQP1. CQE2 is used to indicate that the task of WQE2 has been completed.

[0113] S12. The APP in the first host system A1 of the source host A learns, based on CQE2, that WQE2 has been completed.

[0114] The above steps S1 to S12 are examples of interactions under reliable transmission. In reliable transmission, the sender sends a data packet message to the receiver according to a certain WQE. After receiving the relevant data packet message, the receiver feeds back an ACK signal to the sender. After the sender obtains the ACK signal, it confirms that the WQE has been completed, and writes the corresponding CQE to the CQ at the sender. When it is an unreliable transmission, after the sender's first network card system A2 sends a data packet message according to the WQE of the SQ of the source host A, it does not need to obtain an ACK signal, and directly generates a corresponding CQE to the first host system A1 of the source host A. The APP of the source host A learns that the WQE has been completed based on the CQE. Corresponding to Figure 8 Reliable transmission, unreliable transmission omits the operation of step S8. After executing step S5, the operations of step S11 and step S12 can be directly executed. In addition, Figure 8 The processing order in the above is only an example, and the execution order of some steps may be changed in different implementation scenarios, such as the execution order of step S8, step S9, step S10, step S11 and step S12.

[0115] In some possible implementations, in some network protocols (such as UB protocols), the transport layer of the source host A (i.e., source host A) is set on the first virtual machine management system VMM1. At this time, multiple VMs in the source host A can share a transport connection at the transport layer. In order to ensure the timeliness of business interaction between the first virtual machine VM1 and at least one business host C, thereby improving the efficiency of hot migration. When the first virtual machine VM1 on the source host A interacts with at least one business host C, the second transport connection can be used as the exclusive transport connection of the first virtual machine VM1. By monopolizing the second transport connection by the first virtual machine VM1, the interactive processing time of the business information of the first virtual machine VM1 is accelerated.

[0116] In some possible implementations, the business information initiated by the first virtual machine VM1 and at least one business host C may include first business information and second business information. The first business information is information initiated by at least one business host C to the first virtual machine VM1 through the first network card system A2 (for example, WQE in SQ). The first business information is used to indicate DMA data processing of the first storage area. The second business information is information initiated by the first virtual machine VM1 to at least one business host C through the first network card system A2 (for example, WQE in RQ); the second business information is used to indicate DMA data processing of at least one business host C. In the first time period, the time after the first moment, the first network card system A2 needs to process all initiated business signals (including the first business information and the second business information) and transmit the completion information to the first virtual machine VM1.

[0117] In some possible implementations, the first service signal and the second service signal may be complete transactions or transaction segments. For example, in RDMA technology under the RoCE protocol, the first service signal and the second service signal may interact based on complete transactions. For example, under the UB protocol, the first service signal and the second service signal may interact based on transaction segments.

[0118] In some possible implementations, when hot migrating the first virtual machine VM1 on the source host A to the destination host B, the first virtual machine VM1 on the source host A is performing a service interaction operation related to the aforementioned embodiment with at least one service host C (for example, Figure 8 When the business interaction operation shown in Figure 7 In the virtual machine migration method shown, in step S130, after each processing of the business information, the business information is subjected to relevant verification processing:

[0119] In some examples, the first network card system A2 verifies the first service information obtained before the first moment: Fig. 9 As shown, step S130 may include the following operations S131-S135:

[0120] S131. Input first business information; the first business information is used to indicate a DMA write operation to a first storage area.

[0121] Exemplarily, the first service information may be Figure 8 WQE related transactions in RQ described in the embodiment.

[0122] S132. The first network card system A2 outputs a first write instruction to the first host memory A12 according to the first service information.

[0123] Exemplarily, the first write instruction is a related instruction for processing and executing the first business information during the business interaction process. In an embodiment of the present application, a DMA write operation is performed through the first write instruction to write new memory data to the first storage area. In some examples, after the first write instruction is executed, the memory page where the new memory data is located is marked dirty to obtain the first dirty page data.

[0124] S133. The first network card system A2 outputs a first verification instruction to the first host memory A12 according to the first write instruction.

[0125] Exemplarily, in order to ensure that the first write instruction has been executed. The first network card system A2 also generates a corresponding first verification instruction after outputting the first write instruction. A DMA read operation is performed on the first storage area of ​​the first host memory A12 through the first verification instruction, and whether the first write instruction is correctly executed is determined based on the DMA read operation. The first verification instruction here is a newly added instruction based on the interactive service, which is used to verify and confirm the execution completion status of the first service information during the hot migration process.

[0126] S134. The first network card system A2 outputs a first verification completion signal to the first virtual machine management system VMM1 according to the first verification instruction.

[0127] Exemplarily, after determining that the first write instruction is correctly executed, the first network card system A2 outputs a first verification completion signal to the first virtual machine management system VMM1. Exemplarily, the first verification completion signal here can be Figure 8 The CQE described in the embodiment shown.

[0128] S135 . The first virtual machine management system VMM1 outputs a first service completion notification to the first virtual machine VM1 according to the first verification completion signal.

[0129] In some examples, after performing a DMA read operation through a first verification instruction to verify whether the first business information is executed, the verified memory page may be marked dirty to obtain first dirty page data.

[0130] In some examples, the first network card system A2 verifies the second service information initiated by the first virtual machine VM1 before the first moment: Fig.10 As shown, step S130 may include the following operations S131'-S134':

[0131] S131', the first network card system A2 initiates a service message to at least one service host C.

[0132] Exemplarily, there is a counter related to the VF interface in the first network card system A2. When the first virtual machine VM1 initiates the second business information to the first network card system A2 through its VF interface, the first network card system A2 increases the first count value of the counter by the first value each time after sending a business message to the business host C based on the second business information (or after inputting the second business information initiated by the first virtual machine VM1). The number of initiated but unfinished second business information is verified by the size of the first count value of the counter. When the first value is at a certain initial value, it means that there is no unprocessed second business information. For example, the initial value of the first count value is 0, and the first value is 1. Then, when the first count value is 0, the first network card system A2 can determine that there is currently no unprocessed second business information.

[0133] S132': The first network card system A2 receives the first response information.

[0134] Exemplarily, after the first network card system A2 sends a service message to at least one service host C based on the second service signal, the corresponding service host C will send back a first response message (e.g., ACK message). The first response message is used to indicate that the corresponding service host C has obtained or processed the corresponding second service information. After the first network card system A2 receives the first response signal, it indicates that the corresponding second service information has been processed. The first count value is reduced by a first value.

[0135] S133': the second network card system A2 outputs a second verification completion signal to the first virtual machine management system VMM1 according to the first response information.

[0136] In the embodiment of the present application, the second verification completion signal is used to indicate that the corresponding second service information has been processed.

[0137] S134′: The first virtual machine management system VMM1 outputs a second service completion notification to the first virtual machine VM1 according to the second verification completion signal, to inform the first virtual machine VM1 that the corresponding second indication information has been processed.

[0138] In some possible implementations, the first moment may be the moment when the data volume of the first dirty page data is less than the first threshold. In an embodiment of the present application, when the data volume of the first dirty page data remains greater than or equal to the first threshold, it represents that the business interaction volume between the first virtual machine VM1 and at least one business host C is too large, and the memory data of the first virtual machine VM1 is continuously updated. At this time, when the data volume of the first dirty page data is less than the first threshold, it represents that the business interaction volume of the first virtual machine VM1 is reduced, and this moment can be used as the first moment in the first time period. When the first virtual machine management system VMM1 detects that the data volume of the first dirty page data is less than the first threshold, a first indication signal can be generated. The first indication signal is used to instruct the first network card system A2 to stop interacting with new business signals.

[0139] In some possible implementations, the first moment may be the moment when the initiation rate of the second business information is greater than the second threshold, or the moment when the initiation amount of the second business information is greater than the third threshold. In an embodiment of the present application, both the initiation rate and the production amount of the second business information can reflect the size of the interaction amount of the first virtual machine VM1. When the interaction amount of the first virtual machine VM1 is too large, a first indication signal may also be generated to control the first network card system A2 to stop interacting with new business signals. The embodiment of the present application can avoid the problem of too long iterative copy time of hot migration due to excessive business interaction amount of the first virtual machine VM1, and can also avoid the problem of hot migration failure due to excessive iterative copy time, thereby improving the success rate of hot migration.

[0140] In some possible implementations, based on the above Fig.10 and Fig.11 According to the operation described in the embodiment of the present invention, the first business information and the second business information can be verified respectively in step S200:

[0141] In some possible implementations, such as Fig.11 As shown, step S200 may include the following operations S210-S220:

[0142] S210. The first virtual machine management system VMM1 outputs a shutdown preparation signal to the first network card system A2.

[0143] Exemplarily, at the second moment, the first network card system A2 has completed the hot migration of all memory data of the first virtual machine VM1. The first virtual machine management system VMM1 outputs a shutdown preparation signal to the first network card system A2, indicating that the iterative migration is completed.

[0144] S220 . The first network card system A2 outputs a shutdown confirmation signal to the first virtual machine association system VMM1 according to the shutdown preparation signal.

[0145] In some possible implementations, after receiving the shutdown confirmation signal, the first network card system A2 verifies whether there is any unprocessed business information. Fig. 9 In the embodiment shown in step S130, the first verification instruction is used to perform real-time confirmation. Fig.10 The second business information processed in the embodiment shown is confirmed. Exemplarily, the first network card system A2 can read the first count value. When the first count value is the initial value (for example, 0), it can be confirmed that all initiated business information has been processed and a shutdown confirmation signal is output to the first virtual machine association system VMM1. The time of outputting the shutdown confirmation signal is the end time of the first period.

[0146] The embodiment of the present application provides a virtual machine migration method, a chip system and a host. The chip system is arranged on the host, and the chip system includes a network card system and a host system. A virtual machine and a virtual machine management system are running on the host system. The virtual machine can interact directly with the network card system without the control and management of the virtual machine management system. The virtual machine on the host can perform business interaction with the business host based on the network card system. In the process of business interaction, the virtual machine can be hot migrated from the host to the destination host based on the virtual machine migration method. The virtual machine migration method includes: in the process of iteratively copying the first dirty page data in the memory data of the virtual machine, controlling the network card system to stop initiating new business information with at least one business host, and processing the initiated business information. After the iterative copying of all the first dirty page data is completed, the virtual machine is shut down, and the state information of the virtual machine is migrated. In the process of iteratively copying the first dirty page data, the embodiment of the present application shuts down to initiate new business information, which can ensure that there is no new business information that needs to be processed within a period of time before the virtual machine is shut down, so as to avoid that the completion information corresponding to the new business information has not been transmitted to the virtual machine when the virtual machine is shut down. On the other hand, the previously initiated business information can be processed during the period from the cessation of initiating new business information to the shutdown of the virtual machine, so as to ensure that the completion signal corresponding to this part of the business information has been delivered to the virtual machine when the virtual machine is shut down. The embodiment of the present application realizes hot migration of the virtual machine in the case of direct connection of the virtual machine based on the above solution.

[0147] The embodiment of the present application further provides a computer-readable storage medium, which includes instructions; when the instructions are executed on a host as described in the above embodiment, the host executes the above including Figure 6 , Figure 7 , Figure 8 , Fig. 9 , Fig.10 and Fig.11 When running on a chip system (such as the first chip system) as described in the above embodiment, the chip system executes the above-mentioned steps including Figure 6 , Figure 7 , Figure 8 , Fig. 9 , Fig.10 and Fig.11 The method shown.

[0148] The host system and the network card system involved in the embodiments of the present application may be a chip. For example, it may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0149] The memory involved in the embodiments of the present application may be a volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DRRAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0150] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0151] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0152] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0153] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0154] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one device or distributed on multiple devices. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0155] In addition, each functional module in each embodiment of the present application may be integrated into one device, or each module may exist physically separately, or two or more modules may be integrated into one device.

[0156] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing a computer program instruction on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or may contain one or more servers, data centers and other data storage devices that can be integrated with a medium. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0157] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A virtual machine migration method, characterized in that: The virtual machine migration method is based on a chip system; The chip system includes a network card system and a host system; a virtual machine and a virtual machine management system are running on the host system; and the method includes: The network card system migrates first dirty page data from a first storage area of ​​the host system to a destination host; the first storage area is used to store memory data of the virtual machine; the first dirty page data is dirty page data in the memory data; in the process of migrating the first dirty page data, the network card system stops interacting with at least one business host for subsequent initiated business information according to first indication information, and interacts with the at least one business host based on the initiated business information; the business information is used to indicate business processing between the virtual machine and the at least one business host; After migrating the first dirty page data, the virtual machine management system controls the virtual machine to shut down, and migrates the state data of the virtual machine to the destination host based on the first transmission connection; the state data includes the business interaction state of the virtual machine and the at least one business host.

2. The method according to claim 1, characterized in that After the first dirty page data is migrated, the virtual machine management system controls the virtual machine to shut down, including: The network card system verifies whether all the initiated service information has been processed; when all the initiated service information has been processed, outputs a shutdown confirmation signal to the virtual machine management system of the host system; the shutdown confirmation signal is used to indicate that the initiated service information has been processed; The virtual machine management system controls the virtual machine to shut down according to the shutdown confirmation signal.

3. The method according to claim 2, characterized in that The service information includes first service information; the first service information is information initiated by the at least one service host to the virtual machine through the network card system; The verification of whether all initiated business information has been processed includes: The network card system generates a first write instruction according to the first business information; the first write instruction is used to write new first dirty page data to the first storage area; according to the first write instruction, a corresponding first verification instruction is generated; the first verification instruction is used to verify and read the new first dirty page data from the first storage area; if the new first dirty page data is verified and read from the first storage area, the network card system outputs a corresponding first business completion notification to the host system.

4. The method according to claim 2 or 3, characterized in that: The business information includes second business information; the second business information is information initiated by the virtual machine to the at least one business host through the network card system; the verification of whether all the initiated business information has been processed includes The network card system, when sending a service message to the at least one service host according to the second service information, increases the first count value by the first value; when receiving the first response information, reduces the first count value by the first value; the first response information is used to indicate that the at least one service host has processed the corresponding second service information; when the size of the first count value is the initial value, outputs a second verification completion signal to the virtual machine management system; the second verification completion signal is used to indicate that the corresponding second service information has been processed.

5. The method according to any one of claims 2 to 4, characterized in that: After the migration of the first dirty page data, the virtual machine management system controls the virtual machine to shut down, further comprising: The host system, after migrating the first dirty page data, outputs a shutdown preparation signal to the network card system based on the virtual machine management system; the shutdown preparation signal is used to indicate that the migration of the first dirty page data has been completed; The network card system, according to the shutdown preparation signal, performs the verification to determine whether all the initiated service information has been processed.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: When the data volume of the first dirty page data is less than a first threshold, the virtual machine management system outputs the first indication signal to the network card system.

7. The method according to claim 4 or 5, characterized in that: The method further comprises: When the initiation rate of the second service information is greater than a second threshold, or when the initiation amount of the second service information is greater than a third threshold, the virtual machine management system outputs the first indication signal to the network card system.

8. The method according to any one of claims 1 to 7, characterized in that: The virtual machine management system is also used to implement network protocol processing at the transport layer; the method also includes: Before the virtual machine management system controls the virtual machine to shut down, the virtual machine management system sends a first connection request to the destination host through the network card system; the first connection request is used to instruct the destination host to establish the first transmission connection with the host system.

9. The method according to any one of claims 1 to 8, characterized in that: The virtual machine management system is also used to implement network protocol processing at the transport layer; the method also includes: The virtual machine and the at least one service host exchange the service information based on a second transmission connection; the second transmission connection is a transport layer connection channel exclusively used by the virtual machine.

10. A chip system, characterized in that: The chip system includes a host system and a network card system; a virtual machine and a virtual machine management system are running on the host system; wherein: The network card system is used to: migrate first dirty page data from a first storage area of ​​the host system to a destination host; the first storage area is used to store memory data of the virtual machine; the first dirty page data is dirty page data in the memory data; in the process of migrating the first dirty page data, according to first indication information, stop interacting with at least one business host for subsequent initiated business information, and interact with the at least one business host based on the initiated business information; the business information is used to indicate business processing between the virtual machine and the at least one business host; The host system is used to: after migrating the first dirty page data, control the virtual machine to shut down based on the virtual machine management system; The network card system is also used to: after the virtual machine is shut down, migrate the state data of the virtual machine to the destination host based on the first transmission connection; the state data includes the service interaction state of the virtual machine and the at least one service host.

11. The chip system according to claim 10, characterized in that: The network card system is also used to: verify whether all the initiated service information has been processed; when all the initiated service information has been processed, output a shutdown confirmation signal to the virtual machine management system of the host system; the shutdown confirmation signal is used to indicate that the initiated service information has been processed; The host system is further configured to: control the virtual machine to shut down according to the shutdown confirmation signal based on the virtual machine management system.

12. The chip system according to claim 11, characterized in that: The service information includes first service information; the first service information is information initiated by the at least one service host to the virtual machine through the network card system; The network card system is further used to: generate a first write instruction according to the first service information; the first write instruction is used to write new first dirty page data to the first storage area; and generate a corresponding first verification instruction according to the first write instruction; The first verification instruction is used to verify and read the new first dirty page data from the first storage area; if the new first dirty page data is verified and read from the first storage area, a first verification completion signal is output to the virtual machine management system; the first verification completion signal is used to indicate that the corresponding first business information has been processed.

13. The chip system according to claim 11 or 12, characterized in that: The service information includes second service information; the second service information is information initiated by the virtual machine to the at least one service host through the network card system; The network card system is also used to: increase the first count value by the first value when sending a service message to the at least one service host according to the second service information; reduce the first count value by the first value when receiving the first response information; the first response information is used to indicate that the at least one service host has processed the corresponding second service information; when the size of the first count value is the initial value, output a second verification completion signal to the virtual machine management system; the second verification completion signal is used to indicate that the corresponding second service information has been processed.

14. The chip system according to any one of claims 11 to 13, characterized in that: The host system is further used to: after migrating the first dirty page data, based on the virtual machine management system, output a shutdown preparation signal to the network card system; the shutdown preparation signal is used to indicate that the migration of the first dirty page data has been completed; The network card system is also used for: executing the verification whether all the initiated service information has been processed according to the shutdown preparation signal.

15. The chip system according to any one of claims 10 to 14, characterized in that: The host system is also used to: When the data volume of the first dirty page data is less than a first threshold, the first indication signal is output to the network card system based on the virtual machine management system.

16. The chip system according to claim 13 or 14, characterized in that: The host system is also used to: When the initiation rate of the second service information is greater than a second threshold, or when the initiation amount of the second service information is greater than a third threshold, the first indication signal is output to the network card system based on the virtual machine management system.

17. The chip system according to any one of claims 10 to 16, characterized in that: The virtual machine management system is also used to implement network protocol processing at the transport layer; the host system is also used to: Before controlling the virtual machine to shut down based on the virtual machine management system, sending a first connection request to the destination host through the network card system based on the virtual machine management system; The first connection request is used to instruct the destination host to establish the first transmission connection.

18. The chip system according to any one of claims 10 to 17, characterized in that: The virtual machine management system is also used to implement network protocol processing at the transport layer; the host system is also used to: The virtual machine and the at least one service host are controlled to exchange the service information based on a second transmission connection; the second transmission connection is a transport layer connection channel exclusively used by the virtual machine.

19. A host, characterized in that: The host comprises a circuit board and a chip system as described in any one of claims 10-18; the chip system is arranged on the circuit board.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes instructions; when the instructions are executed on the chip system as described in any one of claims 10-18, the chip system executes the virtual machine migration method as described in any one of claims 1-9; when the instructions are executed on the host as described in claim 19, the host executes the virtual machine migration method as described in any one of claims 1-9.

Images