Virtual machine migration method, chip system, and host

Abstract

The present embodiment is applied to the field of computers, and provides a virtual machine migration method, a chip system, and a host. The chip system comprises a host system and a network card system. The method comprises: the network card system migrates first dirty page data from a first storage area of the host system to a destination host, wherein the first storage area is used for storing memory data of a virtual machine; in the process of migrating the first dirty page data, on the basis of first indication information, the network card system stops initiating new service information with at least one service host, wherein the service information indicates the service processing between the virtual machine and at least one service host; and after the first dirty page data is migrated, a virtual machine management system controls the virtual machine to shut down, and on the basis of a first transmission connection, migrates state data of the virtual machine to the destination host, wherein the state data comprises a service interaction state between the virtual machine and at least one service host. According to the present embodiment, initiating a new service is stopped before a virtual machine is shut down, thereby achieving virtual machine live migration under virtualization pass-through.

Description

A virtual machine migration method, chip system and host

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on July 21, 2023, with application number 202310911870.3 and application name “A virtual machine migration method, chip system and host”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computers, and in particular to a virtual machine migration method, a chip system, and a host. Background Art

[0003] Virtualization technology can be implemented based on a host's chip system. This chip system includes a host system and a network interface card (NIC) system. The host system includes a processor. A virtual machine and a virtual machine management system run on the processor. After being controlled and managed by the virtual machine management system, the virtual machine interacts with other service hosts through the NIC chip. Typically, during service interaction, the functions of some network protocol layers (such as the transport layer and network layer) in the network model are set up in the processor, which then encapsulates and parses the messages during the service interaction. However, this packet encapsulation and parsing can significantly reduce the processing performance of the processor and virtual machine.

[0004] To improve processor processing performance, SR-IOV technology can be used to enable virtual machines to communicate directly with the network interface card (NIC) system without the control and management of a virtual machine management system. This technology also offloads some or all network protocol layer functions (such as the transport layer and network layer, or even the network protocol layer above the transport layer) from the host system's processor to the NIC system. Service interaction is performed through the NIC system based on certain state information. This state information helps the NIC system encapsulate and parse interactive messages. This type of service interaction is called stateful service interaction. In stateful service interaction, after processing relevant service information, the NIC system must output a completion signal corresponding to the relevant service to the host system. When the virtual machine in the host system receives the completion signal, it confirms that the relevant service information has been processed. For virtual machines with direct access to the NIC system, because the virtual machine continues to interact with the service host during live migration, there is no guarantee that the completion signals for all initiated service information have been transmitted to the virtual machine before its state information is migrated to the destination host. Therefore, when a virtual machine is directly connected to a network card system with stateful business interactions, how to hot migrate the virtual machine has always been a difficult problem in the industry. Even in many network models, there is no solution for hot migration of the virtual machines corresponding to the direct network card system.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a virtual machine migration method, a chip system, and a host, which realize hot migration of virtual machines performing stateful business interactions when the virtual machines are directly connected to the network card system.

[0007] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0008] In a first aspect, an embodiment of the present application provides a virtual machine migration method based on a chip system. The chip system includes a network card system and a host system. A virtual machine and a virtual machine management system are running on the host system. The method includes: the network card system migrates first dirty page data from a first storage area of ​​the host system to a destination host. The first storage area is used to store the memory data of the virtual machine. The first dirty page data is dirty page data in the memory data. During the migration of the first dirty page data, the network card system stops interacting with at least one business host for subsequent initiated business information based on the first indication information, and interacts with at least one business host based on the business information that has been initiated. The business information is used to indicate the business processing between the virtual machine and at least one business host. After the first dirty page data is migrated, the virtual machine management system controls the virtual machine to shut down and migrates the state data of the virtual machine to the destination host based on the first transmission connection. The state data includes the business interaction state of the virtual machine and at least one business host.

[0009] In the process of iteratively copying the first dirty page data, the embodiment of the present application stops interacting with at least one business host for the subsequently initiated business information, and interacts with at least one business host based on the business information that has been initiated. On the one hand, it can ensure that no new business information is initiated within a period of time before the virtual machine is shut down, so as to avoid the situation where the completion information corresponding to the newly initiated business information has not yet been transmitted to the virtual machine when the virtual machine is shut down. On the other hand, the business information that has been initiated previously can be processed during the period from the cessation of initiating new business information to the shutdown of the virtual machine, so as to ensure that the completion signal corresponding to this part of the business information has been delivered to the virtual machine when the virtual machine is shut down. The embodiment of the present application realizes direct connection of the virtual machine based on the above-mentioned scheme, and the virtual machine successfully performs hot migration of the virtual machine during the process of stateful business interaction with the business host.

[0010] In one possible implementation, after the migration of the first dirty page data, the virtual machine management system controls the virtual machine to shut down, including: the network card system verifies whether all the initiated business information has been processed. When all the initiated business information has been processed, a shutdown confirmation signal is output to the virtual machine management system of the host system. The shutdown confirmation signal is used to indicate that the initiated business information has been processed. The virtual machine management system controls the virtual machine to shut down according to the shutdown confirmation signal. In an embodiment of the present application, in order to ensure that all the initiated business information has been processed before the virtual machine is shut down. After the migration of the first dirty page data, the completion of the processing of the business information can be verified. When it is verified that all the initiated business information has been completed, the virtual machine is shut down to ensure the success of the hot migration of the virtual machine.

[0011] In one possible embodiment, the business information includes first business information. The first business information is information initiated by at least one business host to the virtual machine through the network card system. The above-mentioned verification of whether all the initiated business information has been processed includes: the network card system generates a first write instruction based on the first business information. The first write instruction is used to write new first dirty page data to the first storage area. According to the first write instruction, a corresponding first verification instruction is generated. The first verification instruction is used to verify the reading of new first dirty page data from the first storage area. If the new first dirty page data is verified to be read from the first storage area, the network card system outputs the corresponding first business completion notification to the host system. In an embodiment of the present application, the first business information is business information executed on the virtual machine side. The first business information may include control indication information and data processing indication information. When the first business information includes data processing indication information, and the data processing indication information is to write data processing to the first storage area of ​​the virtual machine, a first write instruction is generated to perform the write data processing operation. The first write instruction will cause the first storage area to generate new memory data, i.e., first dirty page data. In order to verify that the first business information has been processed, a first verification instruction may be generated after executing the first write instruction to read the first storage area to verify whether the first dirty page data has been processed.

[0012] In one possible implementation, the business information includes second business information. The second business information is information initiated by the virtual machine to at least one business host through the network card system. The above verification of whether all initiated business information has been processed includes: the network card system: when sending a business message to at least one business host according to the second business information, the first count value is increased by the first value; when receiving the first response information, the first count value is reduced by the first value; the first response information is used to indicate that at least one business host has processed the corresponding second business information; when the size of the first count value is the initial value, the second verification completion signal is output to the virtual machine management system; the second verification completion signal is used to indicate that the corresponding second business information has been processed. In an embodiment of the present application, the network card system sends a business message to the business host side according to the second business information. The initiation and completion status of the second business information can be recorded in a counting manner. Subsequently, verification is performed based on the count value to determine whether all second business information has been completed.

[0013] In one possible implementation, after the first dirty page data is migrated, the virtual machine management system controls the virtual machine to shut down, and further includes: after the first dirty page data is migrated, the host system outputs a shutdown preparation signal to the network card system based on the virtual machine management system. The shutdown preparation signal is used to indicate that the first dirty page data has been migrated. The network card system, based on the shutdown preparation signal, verifies whether all initiated business information has been processed. In an embodiment of the present application, a shutdown preparation signal can be generated by the virtual machine management system to instruct the network card system to start verifying the completion of the initiated business information.

[0014] In a possible implementation, the method further includes: when the data volume of the first dirty page data is less than a first threshold value, the virtual machine management system outputs a first indication signal to the network card system. In an embodiment of the present application, when the data volume of the first dirty page data is less than the first threshold value, it represents that the business interaction volume is less than a certain level. At this time, shutting down the interactive business data has a small impact on the business interaction, and the data volume of the first dirty page data can be gradually reduced to quickly shut down and complete the hot migration of the virtual machine. This avoids the virtual machine from engaging in business interaction for a long time, and the virtual machine cannot be shut down for a long time, which leads to the failure of hot migration.

[0015] In some possible implementations, the method further includes: after the state data is transmitted via the first transmission connection, the virtual machine management system sends an activation signal to the destination host via the first transmission connection, and restarts the virtual machine on the destination host via the activation signal.

[0016] In a possible implementation, the method further includes: when the initiation rate of the second business information is greater than a second threshold, or when the initiation amount of the second business information is greater than a third threshold, the virtual machine management system outputs a first indication signal to the network card system. In an embodiment of the present application, when the initiation rate or initiation amount of the second business information reaches a certain level, it represents that there is a lot of interaction of business information. In order to avoid the increase of business information, which leads to the inability to shut down the virtual machine for a long time, the interactive business information can be stopped in this case. Compared to the method of stopping business interaction when the amount of data of the first dirty page data is less than the first threshold, the hot migration speed of the virtual machine can be accelerated through the embodiment of the present application.

[0017] In one possible implementation, the virtual machine management system is also used to implement network protocol processing for the transport layer. The method also includes: before the virtual machine management system controls the virtual machine to shut down, the virtual machine management system sends a first connection request to the destination host through the network card system. The first connection request is used to instruct the destination host to establish a first transmission connection with the host system. In an embodiment of the present application, when the transport layer is presented on the virtual machine management system, in order to avoid establishing the first transmission connection after the shutdown, resulting in a long time for the shutdown migration status data and other processes, the first transmission connection can be established before the virtual machine is shut down to ensure that the shutdown phase is completed as quickly as possible to avoid the impact of a long shutdown on business interactions.

[0018] In one possible implementation, the virtual machine management system is also used to implement network protocol processing for the transport layer. The method also includes: the virtual machine and at least one business host interact with each other regarding business information based on the second transport connection. The second transport connection is a transport layer connection channel used exclusively by the virtual machine. In an embodiment of the present application, when the transport layer is set on the virtual machine management system, multiple transport layer connection channels can be established on the virtual machine management system. Multiple virtual machines on the host system can use a transport layer connection channel together for business interaction. At this time, an exclusive transport layer connection channel can be set up for the virtual machine that needs to be hot migrated. The virtual machine that is hot migrated uses the exclusive transport layer connection channel as the first transmission channel, and business interaction with the business host based on the first transmission channel can avoid the business congestion problem caused by the business information of other virtual machines sharing the transport layer connection channel, thereby speeding up the business interaction progress of the hot migrated virtual machine.

[0019] In a second aspect, an embodiment of the present application further provides a chip system, which includes a host system and a network card system. A virtual machine and a virtual machine management system are running on the host system. Among them: the network card system is used to: migrate first dirty page data from the first storage area of ​​the host system to the destination host. The first storage area is used to store the memory data of the virtual machine. The first dirty page data is dirty page data in the memory data. In the process of migrating the first dirty page data, according to the first indication information, stop interacting with at least one business host for subsequent initiated business information, and interact with at least one business host based on the initiated business information. The business information is used to indicate the business processing between the virtual machine and at least one business host. The host system is used to: after migrating the first dirty page data, control the virtual machine to shut down based on the virtual machine management system. The network card system is also used to: after the virtual machine is shut down, migrate the status data of the virtual machine to the destination host based on the first transmission connection. The status data includes the business interaction status of the virtual machine and at least one business host.

[0020] In one possible implementation, the network card system is further configured to verify whether all initiated service information has been processed. When all initiated service information has been processed, the network card system outputs a shutdown confirmation signal to the virtual machine management system of the host system. The shutdown confirmation signal indicates that the initiated service information has been processed. The host system is further configured to control the virtual machine to shut down based on the shutdown confirmation signal, based on the virtual machine management system.

[0021] In one possible embodiment, the business information includes first business information. The first business information is information initiated by at least one business host to the virtual machine through the network card system. The network card system is also used to: generate a first write instruction based on the first business information. The first write instruction is used to write new first dirty page data to the first storage area. According to the first write instruction, a corresponding first verification instruction is generated. The first verification instruction is used to verify and read the new first dirty page data from the first storage area. If the new first dirty page data is verified and read from the first storage area, a first verification completion signal is output to the virtual machine management system. The first verification completion signal is used to indicate that the corresponding first business information has been processed.

[0022] In one possible embodiment, the business information includes second business information. The second business information is information initiated by the virtual machine to at least one business host through the network card system. The network card system is further configured to: when sending a business message to the at least one business host based on the second business information, increase the first count value by a first value. Upon receiving the first response information, decrease the first count value by the first value. The first response information is used to indicate that the at least one business host has processed the corresponding second business information. When the first count value reaches the initial value, a second verification completion signal is output to the virtual machine management system. The second verification completion signal is used to indicate that the corresponding second business information has been processed.

[0023] In one possible implementation, the host system is further configured to, after migrating the first dirty page data, output a shutdown preparation signal to the network card system based on the virtual machine management system. The shutdown preparation signal is configured to indicate that the migration of the first dirty page data has been completed. The network card system is further configured to, based on the shutdown preparation signal, verify whether all initiated service information has been processed.

[0024] In a possible implementation, the host system is further configured to: when the amount of the first dirty page data is less than a first threshold, output a first indication signal to the network card system based on the virtual machine management system.

[0025] In a possible implementation, the host system is further configured to output a first indication signal to the network card system based on the virtual machine management system when the initiation rate of the second business information is greater than a second threshold, or when the initiation amount of the second business information is greater than a third threshold.

[0026] In one possible implementation, the virtual machine management system is further configured to perform transport layer network protocol processing. The host system is further configured to, before the virtual machine management system controls the virtual machine to shut down, send a first connection request to the destination host via the network card system based on the virtual machine management system. The first connection request is used to instruct the destination host to establish a first transport connection.

[0027] In one possible implementation, the virtual machine management system is further configured to implement transport layer network protocol processing. The host system is further configured to control the virtual machine and at least one service host to exchange service information based on a second transport connection. The second transport connection is a transport layer connection channel used exclusively by the virtual machine.

[0028] In a third aspect, an embodiment of the present application further provides a host comprising a circuit board and a chip system as described in the second aspect.

[0029] In a fourth aspect, embodiments of the present application further provide a computer-readable storage medium comprising instructions. When the instructions are executed on the chip system described in the second aspect, the chip system executes the virtual machine migration method described in the first aspect. When the instructions are executed on the host of the third aspect, the host executes the virtual machine migration method described in the first aspect.

[0030] For the description of the technical principles and beneficial effects of the second, third and fourth aspects, reference can be made to the relevant description of the first aspect above, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] FIG1 is a schematic structural diagram of a host;

[0032] FIG2 is a schematic diagram of the architecture of a virtualized network card system directly connected to a virtual machine;

[0033] FIG3 is a schematic diagram of a structure of an interactive architecture including multiple hosts provided in an embodiment of the present application;

[0034] FIG4 is a schematic diagram of a migration process in which a first virtual machine hot migration fails according to an embodiment of the present application;

[0035] FIG5 is a schematic diagram of an interaction process for business interactions between multiple hosts when the transport layer and the transaction layer are separated, provided by an embodiment of the present application;

[0036] FIG6 is a flowchart of a virtual machine migration method according to an embodiment of the present application;

[0037] FIG7 is a second flow diagram of another virtual machine migration method provided in an embodiment of the present application;

[0038] FIG8 is a schematic diagram of an interaction process for business interaction between multiple hosts provided in an embodiment of the present application;

[0039] FIG9 is a third flow chart of another virtual machine migration method provided in an embodiment of the present application;

[0040] FIG10 is a fourth flow chart of another virtual machine migration method provided in an embodiment of the present application;

[0041] FIG11 is a fifth flow chart of another virtual machine migration method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] It should be noted that the terms "first", "second", etc. involved in the embodiments of the present application are only used to distinguish features of the same type and cannot be understood as indicating relative importance, quantity, order, etc.

[0043] The terms "exemplary" or "for example" in the embodiments of this application are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0044] The terms "coupling" and "connection" involved in the embodiments of this application should be understood in a broad sense. For example, they may refer to a physical direct connection, or an indirect connection achieved through electronic devices, such as a connection achieved through resistors, inductors, capacitors or other electronic devices.

[0045] First, some basic concepts involved in the embodiments of this application are explained:

[0046] Business interactions between network nodes implement message encapsulation and parsing based on the network model. Depending on the network model, the corresponding message encapsulation and parsing are also different. Three common network models are the OSI seven-layer network model, the TCP five-layer network model, and the TCP / IP four-layer network model. The OSI seven-layer network model includes, from bottom to top, the physical layer, data link layer, network layer, transport layer, session layer, presentation layer, and application layer. The TCP five-layer network model includes, from bottom to top, the physical layer, data link layer, network layer, transport layer, and application layer. The application layer of the TCP five-layer network model corresponds to the session layer, presentation layer, and application layer of the OSI seven-layer network model. The lower four layers of the TCP five-layer network model are consistent with the OSI seven-layer network model. The TCP / IP four-layer network model includes, from bottom to top, the network access layer, network interconnection layer, transport layer, and application layer. The network access layer of the TCP / IP four-layer network model corresponds to the physical layer and data link layer of the OSI seven-layer network model; the internetworking layer of the TCP / IP four-layer network model corresponds to the network layer of the OSI seven-layer network model; and the application layer of the TCP / IP four-layer network model corresponds to the session layer, presentation layer, and application layer of the OSI seven-layer network model. In these three network models, the network model layers above the transport layer are used for transaction processing by app users. We can refer to the network model layers above the transport layer in each network model as the transaction layer.

[0047] The host includes a circuit board and a chip system. The chip system is arranged on the circuit board. The chip system includes a host system 1 and a network card system 2. As shown in Figure 1, the host system 1 includes a processor 11 and a host memory 12. The network card system 2 can access the host memory 12 of the host system 1 through a data link based on direct memory access (DMA) technology, and exchange control signals with the processor 11 of the host system 1 through a control link. Each host can serve as a network node. Business interactions can be carried out between multiple network nodes. In some possible implementations, the transaction layer is presented in the user state of the processor 11 of the host system 1, and the transport layer and network layer are presented in the kernel state of the processor 11 of the host system 1. The network model layer below the network layer is presented in the network card system 2. At this time, the interaction of message messages between the two network nodes still relies heavily on the processor 11 of the host system 1 to encapsulate and parse the message messages based on the network protocol layer (such as the transport layer and the network layer, and the network protocol layer above the transport layer). In order to improve the processing performance of the processor 11, the industry has proposed TCP Offload Engine (TOE), remote direct memory access (RDMA) and unified bus (UB) protocols.

[0048] TOE offloads protocols such as the transport and network layers from the host system's processor network to the network interface card (NIC) system, presenting them there. The NIC system then performs packet encapsulation and parsing, improving the efficiency of the host system's processor. TOE technology can be applied to single-side network nodes. RDMA similarly offloads protocols such as the transport and network layers from the host system's processor network to the NIC system. RDMA's kernel bypass mechanism enables direct memory reads and writes between apps and the NIC system over the data link. RDMA's zero-copy memory mechanism allows direct memory reads and other operations between two network nodes, reducing the processor's workload and improving processing efficiency. However, a small number of control signals still need to be processed in the processor's kernel state on the control link. RDMA is a technology that allows a local network node to directly access a remote network node. This technology requires the implementation of RDMA technology on both network nodes. In RDMA, apps can access the memory of a remote network node, and remote network nodes can be read without the involvement of processes or processors on the remote node. Common RDMA technologies include the RDMA over converged Ethernet (RoCE) protocol. UB and RDMA have similar implementation functions, but differ from RDMA in that UB separates the transaction layer from the transport layer.

[0049] Network nodes can perform various communication interactions based on protocols such as RDMA and UB. Taking the RDMA protocol as an example, two basic communication interactions can be performed: memory verbs and messaging verbs. Memory verbs are unilateral communication interactions that only require the local network node to clearly identify the source and destination addresses of the message. The processor 11 of the remote network node does not need to be aware of this unilateral communication interaction. Data reads and writes from the local network node to the remote network node are performed by the remote network node's network card system 2 based on DMA operations. The remote network node's network card system 2 then encapsulates and returns the relevant message to the local network node's network card system 2. The local network node's network card system 2 then writes the data to the local host memory 12 and / or feedbacks transaction completion information to the local processor 11. Memory verbs include RDMA read operations, RDMA write operations, and RDMA atomic operations. Atomic operations are uninterrupted operations, i.e., they are the smallest execution units. The simplest atomic operation is a series of assembly instructions. Specifically, RDMA atomic operations can include atomic fetch, atomic add, atomic compare, and atomic swap. Messaging verbs are bilateral operations, requiring the participation of both the local and remote network node's apps for communication interaction. Messaging verbs include RDMA send and RDMA receive operations. The send operation allows the local network node to send data to the remote network node's receive queue. The remote network node, acting as the receiver, must have pre-registered a buffer for receiving data. The local network node, acting as the sender, cannot control the placement of data in the remote network node. In RDMA send operations, you can choose whether to use an immediate number. The immediate number is transmitted to the remote network node along with the data, and the immediate number is sent to the receiver as a notification of receipt. The RDMA receive operation corresponds to the RDMA send operation. The functions of the relevant operations in the UB protocol are similar to those in the aforementioned RDMA protocol and are not detailed here.

[0050] Virtualization is the foundation of cloud computing technology. In cloud computing, virtualization is used to virtualize the physical hardware resources (including computing, storage, and network resources) of a host (e.g., a server device) to create multiple independently running virtual machines (VMs). A VM is an independent virtual processing device built by utilizing the host's computing, storage, and network resources. Before virtualization, the host's computer system architecture can be divided from top to bottom into the application layer, the operating system (OS) layer, and the hardware layer. The OS layer is also known as the host OS layer. The hardware layer, based on the host's electronic components (e.g., the host system 1 including the processor 11 and host memory 12, and the network interface card system 2), provides computing, storage, and network resources for the upper layers. The OS layer can run in the host's kernel state, while the application layer can run in the host's user state. After virtualization, the resource management layer and the virtualization layer are virtualized above the hardware layer. The virtualization layer contains at least one VM. Each VM includes a guest operating system (guest OS) and an application. The resource management layer includes a virtual machine manager (VMM) to manage each VM. In some scenarios, the VMM is also called a hypervisor. Different operating systems have different corresponding hypervisors.

[0051] Virtualization passthrough allows a VM to communicate directly with the network card system 2. One virtualization passthrough method is to assign the actual network card system 2 to the VM on the physical architecture. Another virtualization passthrough method is single root I / O virtualization (SR-IOV). SR-IOV is a standard for sharing PCIe interface functions with virtual machines. SR-IOV provides VMs with independent memory spaces, interrupts, and DMA streams, bypassing the VMM for data access. The SR-IOV protocol introduces two types of functions: physical functions (PFs) and virtual functions (VFs). PFs contain full PCIe functionality, including SR-IOV expansion capabilities, and can be used for SR-IOV configuration and management. VFs contain lightweight PCIe functionality. Each VF has its own dedicated PCI configuration area and may share the same physical resources with other VFs. PF drivers vary depending on the hypervisor and generally require higher privileges than ordinary VMs to operate the PF driver. The PF driver includes all the functions of traditional data interface drivers, allowing the hypervisor to access the host's I / O resources. The entire host can be affected by calling the PF driver to perform related operations. As shown in Figure 2, the network card system 2 can provide multiple PCIe interfaces to the host system 1 side through SR-IOV technology. The PCIe interface in the network card system 2 that directly interacts with the PF driver of the processor 11 is the PF interface. The PCIe interface in the network card system that directly connects to the VM in the host system 1 is the VF interface. SR-IOV technology can be regarded as the virtualization of the network card system 2. The direct connection between the VM and the network card system 2 can maximize the performance of the VM's related data communication and processing.

[0052] Stateless service interaction means that when two network nodes exchange messages, the network nodes' processing of the messages does not depend on specific state information and can independently process the messages without being affected.

[0053] Stateful service interaction refers to the interaction of messages between two network nodes, where the processing of different messages depends on dependencies. The processing of these messages requires a specific working state. In actual processing, messages are processed based on relevant state information. Furthermore, the content of this state information may change during the message processing process. When a VM on a network node interacts with another network node Q1 based on the network card system, this state information may include address mappings, relevant network protocol control information, and network state control information. Address mappings can include mappings between guest virtual addresses (GVAs), guest physical addresses (GPAs), host virtual addresses (HVAs), and host physical addresses (HPAs). Generally, this state information can be stored in a mapping state table.

[0054] Virtual machine migration is a technology that moves a VM running on a source host to a destination host, ensuring that the moved VM can operate normally on the destination host. A complete virtual machine computer system consists of virtual processors, virtual memory, virtual disks, and virtual network cards, with virtual disks and virtual network cards both being virtual I / O devices. Virtual machine migration technology migrates the virtual processor state information, virtual memory data, and virtual I / O device state information within a VM from one physical electronic device (e.g., the source host) to another physical host (e.g., the destination host). Virtual machine migration technology is widely used in server cluster management in cloud computing data centers, enabling load balancing, disaster recovery, and online upgrades. Virtual machine migration can reduce cloud data center operating costs and improve server cluster management efficiency. Virtual machine migration can be divided into static migration and dynamic migration. Static migration, also known as cold migration, involves shutting down the VM on the source host and then migrating the VM's memory data and state data from the source host to the destination host. State data includes service interaction status, virtual processor status, and virtual I / O device status. The VM is then started on the destination electronic device. Cold migration typically requires interrupting the VM's service for several minutes or even longer. Live migration, also known as hot migration, involves migrating only a portion of the VM's data during a short downtime, while the remaining data is dynamically copied while the VM is operating normally.

[0055] The embodiment of the present application proposes an interactive architecture, as shown in FIG3 , which includes a source host A, a destination host B, and at least one service host C. The source host A includes a first chip system. The destination host B includes a second chip system. The service host C includes a third chip system.

[0056] The first chip system of source host A includes a first host system A1 and a first network card system A2. First host system A1 includes a first processor A11 and a first host memory A12. At least one VM and a first virtual machine management system VMM1 run on the first processor A11. The at least one VM includes a first virtual machine VM1. First host memory A12 is used to store memory data of the first virtual machine VM1. Depending on the actual application scenario, the status information of the first virtual machine VM1 can be stored in the first host memory A12, the cache of the first processor A11, and at least one device in the network card system A2. The various devices in source host A are coupled via a first bus BUS1.

[0057] The second chip system of the destination host B includes a second host system B1 and a second network card system B2. The second host system B1 includes a second processor B11 and a second host memory B12. The components in the destination host B are coupled via a second bus BUS2.

[0058] The third chip system of the service host C includes a third host system C1 and a third network card system C2. The third host system C1 includes a third processor C11 and a third host memory C12. The components in each service host C are coupled via a third bus BUS3.

[0059] In some possible implementations, when the first virtual machine VM1 is not directly connected to the first network card system A2, the functions of the transport layer and the network layer are presented in the first processor A11. The first network card system A2 only performs stateless information processing. The first virtual machine VM1 can be hot migrated entirely at the software level. During the hot migration process, memory data is usually copied and migrated during the dynamic copy stage. Processor status information and I / O device status information are copied and migrated within a short period of time during the shutdown stage. The hot migration process will not cause interruption of the business running on the virtual machine, and can improve service availability compared to cold migration. For the case where the first virtual machine VM1 is not directly connected to the first network card system A2, there are multiple ways to implement hot migration. For example, in the virtualization technology based on KVM-QEMU, hot migration is mainly implemented by pre-copying. Pre-copying can be divided into the following chronological order: migration initialization stage, iterative copying stage, shutdown copying stage and restarting stage.

[0060] First, in the migration initialization phase, hot migration related initialization is performed, and all memory pages of the first virtual machine VM1 are marked dirty. The migration initialization phase is usually short-lived and can sometimes be ignored.

[0061] Next, in the iterative copy phase, the first network card system A2 begins copying the memory data of the first virtual machine VM1 and migrating it to the destination host B. During this process, the source host also interacts with at least one service host C. These interactions may modify the memory data of the first virtual machine VM1, resulting in the iterative generation of new memory data during the migration. During this interaction, the first network card system A2 marks the new memory data as dirty to obtain first dirty page data, and iteratively migrates this first dirty page data. The iterative copy phase ends when the amount of first dirty page data falls below the shutdown threshold.

[0062] Then, in the shutdown copy phase, the first virtual machine VM1 is shut down. The first network card system A2 performs shutdown migration on the remaining first dirty page data and the state data of the first virtual machine VM1. After the shutdown migration of the remaining first dirty page data and the state data is completed, the shutdown copy phase ends.

[0063] Finally, in the restart phase, the memory data and state information of the first virtual machine VM1 are run on the destination host B to restart the first virtual machine VM1.

[0064] During pre-copy live migration, the first virtual machine VM1 is in operation during the migration initialization phase, the downtime copy phase, and the restart phase. It only experiences a downtime period during the iterative copy phase, the length of which is determined by the amount of remaining memory data and state information migrated during the downtime copy phase. This downtime typically lasts several milliseconds. However, when the first virtual machine VM1 is directly connected to the first network card system A2, some network layer protocol functions are offloaded from the first processor A11 to the first network card system A2. At this point, the relevant message encapsulation and parsing processing must be performed on the first network card system A2. During stateful service interaction between the first virtual machine VM1 and the third virtual machine VM3 in the service host C over the second transport connection, when the first virtual machine VM1 on the first host system A1 receives a completion signal, it confirms that the relevant service information has been processed. However, for the scenario where the first virtual machine VM1 is directly connected to the first network card system A2, during the live migration of the first virtual machine VM1, there is no guarantee that the completion signals for all initiated service information will be transmitted to the first virtual machine VM1 before the first virtual machine VM1 is shut down. Therefore, live migration of virtual machines under a virtual machine direct network card system solution has always been a difficult problem in the industry, and even many network models do not yet have a solution for live migration. For example, when the first virtual machine VM1 shown in Figure 3 is based on RDMA technology (such as the RoCE protocol), the hardware based on SR-IOV technology (such as the first network card system A2) accelerates the internal processing of the first virtual machine VM1. However, this means that the service interaction status information during the service interaction between the first virtual machine VM1 and at least one service host C not only exists at the software level of the first host system A1, but also exists within the hardware of the first network card system A2, such as the address mapping relationship between the first virtual machine VM1, the first processor A11, and the first host memory A12 (such as the mapping relationship of GVA, GPA, HVA, and HPA addresses), as well as the protocol control information and network status control information being processed by the first network card system A2. Because during the service interaction between the first virtual machine VM1 and at least one service host C, many transactions cannot be initiated repeatedly, such as RDMA send operations and RDMA atomic operations, the repeated initiation of these operations will have a significant impact on the operation of the first virtual machine VM1. When the aforementioned pre-copy hot migration method is used, as shown in FIG4 , there may be a situation where the hot migration shown in steps S1 ′ to S4 ′ fails:

[0065] S1′, initiating an atomic add operation: the first virtual machine VM1, in the iterative copy phase, initiates an atomic add operation to the third virtual machine VM3 of at least one service host C.

[0066] S2', the migration of the first virtual machine VM1 is completed: the iterative copy phase ends, and the first virtual machine VM1 is shut down. During the shutdown copy phase, the first network card system A2 begins to migrate the remaining memory data and status information of the first virtual machine VM1. During the restart phase, after migrating to the destination host B, the first virtual machine VM1 will think that the atomic add operation is not completed. After the first virtual machine VM1 is shut down, the message of successful atomic add operation processing reaches the transport layer of the first network card system A2, or reaches the kernel state of the processor A11 but does not reach the APP of the first virtual machine VM1. At this time, the first virtual machine VM1 is hot migrated to the destination host B.

[0067] S3', atomic add operation completion notification: When the first virtual machine VM1 is shut down, the message of the successful atomic add operation reaches the transport layer of the first network card system A2, or reaches the kernel state of the processor A11 but does not reach the APP of the first virtual machine VM1. After the first virtual machine VM1 is shut down, the message of the successful atomic add operation reaches the source host A. However, at this time, the first virtual machine VM1 on the source host A has been shut down, and its memory data and status data are no longer updated.

[0068] S4', repeatedly initiate the atomic add operation: restart the running first virtual machine VM1 on the destination host B. The first virtual machine VM1 on the destination host B has not received the message of successful atomic add operation from the first network card system A2. The first virtual machine VM1 on the destination host B will re-initiate the atomic add operation to at least one business host C. For at least one business host C, the atomic add operation is executed twice, while for the first virtual machine VM1, the atomic add operation is only executed once. This will cause abnormal operation of the first virtual machine VM1.

[0069] For another example, the first virtual machine VM1 shown in Figure 3 uses the UB protocol. As shown in Figure 5, at least one service host C includes a third host system C1 and a third network interface card system C2. A third virtual machine VM3 runs on the third host system C1. The UB protocol separates the transport layer and the transaction layer. The transaction layer is presented to the first virtual machine VM1. The transport layer of the source host A is set on the first virtual machine management system VMM1. The transport layer of at least one service host C is set on the second source host A, and the transport layer of the third virtual machine management system VMM3. When hot migrating the first virtual machine VM1, only the transaction layer of the first virtual machine VM1 needs to be hot migrated, while the transport layer of the first virtual machine management system VMM1 is not hot migrated. The first virtual machine management system VMM1 establishes a first transmission connection with the third virtual machine management system VMM3 based on the first network interface card system A2 and the third network interface card system C2. The first transmission connection can be established between the first virtual machine management system VMM1 and the third virtual machine management system VMM3, or the first transmission connection can be established between the first network interface card system A2 and the third network interface card system C2. When the aforementioned pre-copy hot migration method is adopted, during the process of business interaction between the first virtual machine VM1 and the third virtual machine VM3, the first virtual machine VM1 sends relevant business information, such as a work queue element (WQE), to the first virtual machine management system VMM1. The relevant business information is used to indicate the atomic add operation for the third virtual machine VM3. If the first network card system A2 completes the task of executing the relevant business information and outputs the corresponding completion information to the first virtual machine management system VMM1, but the relevant completion information is not transmitted to the first virtual machine VM1, the first virtual machine VM1 enters the shutdown copy stage. After the first virtual machine VM1 is restarted on the destination host B, it also believes that the atomic add operation has not been successfully executed, and will re-send the relevant business information to execute the atomic add operation.

[0070] In some possible implementations, in the architecture shown in FIG3 , when the first virtual machine VM1 is directly connected to the first network card system A2:

[0071] The first network card system A2 is configured to migrate first dirty page data from the first storage area of ​​the first host system A1 to the destination host C. The first storage area is configured to store memory data of the first virtual machine VM1. The first dirty page data is dirty page data within the memory data. During the migration of the first dirty page data, based on the first indication information, the interaction of subsequently initiated business information with at least one business host C is stopped, and interaction with the at least one business host C is continued based on the business information that has already been initiated. The business information is configured to indicate business processing between the first virtual machine VM1 and the at least one business host C.

[0072] The first host system A1 is configured to: after migrating the first dirty page data, control the first virtual machine VM1 to shut down based on the first virtual machine VM1 management system VMM1.

[0073] The first network card system A2 is also used to: after the first virtual machine VM1 is shut down, migrate the state data of the first virtual machine VM1 to the destination host C based on the first transmission connection; the state data includes the business interaction state of the first virtual machine VM1 and at least one business host C.

[0074] In an embodiment of the present application, during the iterative copying of the first dirty page data in the memory data of the first virtual machine VM1, the first network card system A2 is controlled to stop interacting with at least one business host C for business information. After the iterative copying of all the first dirty page data is completed, the first virtual machine VM1 is shut down, and the state information of the first virtual machine VM1 is migrated. In the embodiment of the present application, during the iterative copying of the first dirty page data, new business information is stopped from being initiated. On the one hand, it can ensure that no new business information is initiated within a period of time before the first virtual machine VM1 is shut down, so as to avoid the situation where the completion information corresponding to the newly initiated business information before the shutdown is not transmitted to the first virtual machine VM1 when the first virtual machine VM1 is shut down. On the other hand, the business information that has been initiated previously can be processed during the period from the cessation of initiating new business information to the shutdown of the first virtual machine VM1, so as to ensure that the completion signal corresponding to this part of the business information has been delivered to the first virtual machine VM1 when the first virtual machine VM1 is shut down.

[0075] In some possible implementations, a fixed time parameter may be set according to a business scenario of actual application, and the time parameter may be used to determine the moment to stop business interaction during the iterative copying of the first dirty page data.

[0076] In some possible implementations, changes in relevant parameters in hot migration can be used as judgment conditions to confirm the moment to stop business interaction during the iterative copying of the first dirty page data. In some examples, when the amount of data in the first dirty page data is less than a first threshold, it can be confirmed that it is necessary to stop initiating new business information. At this time, the first host system A1 outputs a first indication signal to the first network card system A2 based on the virtual machine management system VMM1. In some examples, when the initiation rate of business information is greater than a second threshold, or when the initiation amount of business information is greater than a third threshold, it is confirmed that it is necessary to stop initiating new business information. At this time, the first host system A1 outputs a first indication signal to the first network card system A2 based on the virtual machine management system VMM1.

[0077] In some possible implementations, the first network card system A2 is further used to verify whether all initiated business information has been processed. When all initiated business information has been processed, a shutdown confirmation signal is output to the first virtual machine VM1 management system VMM1 of the first host system A1. The shutdown confirmation signal is used to indicate that the initiated business information has been processed. The first host system A1 is further used to control the shutdown of the first virtual machine VM1 according to the shutdown confirmation signal based on the first virtual machine VM1 management system VMM1. In an embodiment of the present application, the completion of the processing of the initiated business information can be verified after all the first dirty page data has been iteratively copied. After verifying that all the initiated business information has been processed, the first virtual machine VM1 is shut down. The embodiment of the present application can further ensure the accuracy of hot migration by verifying the processing of business information.

[0078] The present application also provides a virtual machine migration method for hot migrating a first virtual machine VM1 from a source host A to a destination host B based on a second transmission connection in the interaction architecture shown in FIG3 . During the hot migration process, the source host A may interact with at least one service host C based on the first transmission connection. The virtual machine migration method includes the following operations, as shown in FIG6 , from steps S100 to S400 :

[0079] S100 : Start migrating the first dirty page data of the first virtual machine VM1 and stop initiating new service information with the service host C.

[0080] In some possible implementations, after initiating a live migration process, the first virtual machine VM1 within the first processor A11 establishes a data transmission connection with the destination host B. During a first period, the source host A begins iterative copy migration of the memory data of the first virtual machine VMM1. At the first moment in the first period, the source host A ceases service interaction with the service host C. The first moment is a certain amount of time prior to the end of the first period. During the time after the first moment in the first period, the first network card system A2 only needs to process the initiated service information and ensure that the relevant completion information reaches the first virtual machine VM1 before the first virtual machine VM1 shuts down. In this embodiment of the present application, a time point, namely the first moment, is set within the first period of the iterative copy phase. After reaching this first moment, the first network card system A2 is controlled to not accept new interaction information. The interaction information obtained from the first host system A1 and / or service host C before the first moment is processed. During the first period, all obtained interaction information is processed and the corresponding completion information is fed back to the first virtual machine VM1 before the iterative copy phase is terminated. At this point, the first virtual machine VM1 has no unacquired completion information before shutting down, allowing for a normal shutdown.

[0081] Illustratively, the first moment may be a fixed time parameter set according to an actual business scenario, and the time parameter is used to determine the moment when business interaction is stopped during the iterative copying of the first dirty page data.

[0082] For example, the first moment may be determined based on a change in a relevant parameter during hot migration as a judgment condition. In some examples, the first moment may be when the amount of the first dirty page data is less than a first threshold. In some examples, the first moment may be when the service information initiation rate is greater than a second threshold.

[0083] In some possible implementations, as shown in FIG7 , step S100 may include the following operations of steps S110 to S130:

[0084] S110 : The first virtual machine management system VMM1 outputs a migration instruction signal to the first network card system A2 .

[0085] For example, after the first virtual machine VM1 initiates the hot migration process. The hot migration management program running in the first virtual machine management system VMM1 will output a migration indication signal to the first network card system A2. The migration indication signal indicates to the first network card system A2: the VF channel (the transmission channel between the VF driver and the VF interface) corresponding to the first virtual machine VM1 has entered the hot migration state. After inputting the migration indication signal, the first network card system A2 sets a hot migration label for the VF interface corresponding to the first virtual machine VM1. Step S110 can be regarded as the pre-copy migration initialization stage.

[0086] For example, the first storage area of ​​the first host memory A12 is used to store memory data of the first virtual machine VM1. Because all memory data of the first virtual machine VM1 has not been migrated to the destination host B at the start of live migration, the first virtual machine management system VMM1 marks all memory pages in the first storage area of ​​the first host memory A12 as dirty during initialization. At this point, all memory data of the first virtual machine VM1 is first dirty page data.

[0087] For example, after the hot migration tag is set, the first network card system A2 may mark the related memory pages after the writing dirty during the subsequent DMA write operation based on the VF interface.

[0088] S120 : The first network card system A2 migrates first dirty page data of the first virtual machine VM1 .

[0089] In some possible implementations, the first network card system A2 hot migrates the first dirty page data in the memory data of the first virtual machine VM1 from the first storage area of ​​the first host memory A12 to the destination host B based on the VF interface corresponding to the first virtual machine VM1 within the first time period according to the migration indication signal.

[0090] Exemplarily, after the first network card system A2 inputs a migration indication signal, the first time period begins. During the first time period, during the first round of iterative copying, the first network card system A2 migrates all the memory data of the first virtual machine VM1 as the first dirty page data to the destination host B at the destination host B. Because during the first time period, the first virtual machine VM1 is not shut down. All first virtual machines VM1 will also perform business interactions with at least one business host C at a business host C. The business interaction with the first virtual machine VM1 is essentially a DMA operation on the memory data of the first virtual machine VM1. Therefore, in the process of iterative migration of memory data, new memory data will be generated. The memory pages where these new memory data are located are marked dirty to obtain new first dirty page data. In the subsequent iterative migration process, the first dirty page data generated in real time is continuously migrated.

[0091] S130: The first network card system A2 stops initiating new service information.

[0092] In some possible implementations, during a first period, the first virtual machine VM1 and at least one service host CC continuously initiate service information. At a first moment in the first period, the first virtual machine management system VMM1 within the first host system A1 may output a first instruction signal to the first network card system A2, instructing the transmission subnode to stop initiating new service information. During the time after the first moment in the first period, the first network card system A2 processes services initiated before the first moment.

[0093] In some possible implementations, after receiving the first indication signal, the first network card system A2 sets a first flag on the VF interface corresponding to the first virtual machine VM1. After the first flag is set, service interaction through the VF interface is stopped. For example, after the first moment, for a new first service signal obtained by the first network card system A2 from at least one service host C, the first network card system A2 rejects the reception process and sends a rejection signal (such as an RNR status signal) to the corresponding service host C to inform the service host C that the first virtual machine VM1 of the source host A on the local side has not received and processed the relevant service information. At the same time, after the first moment, service messages regarding the second service signal are no longer sent to at least one service host C.

[0094] S200: Confirm whether all initiated business information has been processed.

[0095] In some possible implementations, at the second moment, the iterative migration of the first dirty page data of the first virtual machine VM1 is complete, and the first dirty page data no longer exists in the first storage area. The first network card system A2 verifies whether all initiated service information has been processed to ensure that the first virtual machine VM1 obtains corresponding completion information for all processed service information.

[0096] In some possible implementations, in some network protocols (such as the UB protocol), the transport layer of the source host A (i.e., source host A) is set on the first virtual machine management system VMM1. Because the first period is the period during which the first virtual machine VM1 is running, the first virtual machine VM1 will be shut down after the first period. It is necessary to establish a first transmission connection during the first period to quickly migrate the state data of the first virtual machine VM1 after the first virtual machine VM1 is shut down, so as to avoid the problem of extending the shutdown migration time due to establishing the first transmission connection after the shutdown. Exemplarily, during the first period, the first virtual machine management system VMM1 sends a first connection request to the destination host B through the first network card system A2; the first connection request is used to instruct the destination host B to establish a first transmission connection with the first host system A1. The establishment of the first transmission connection can be in the iterative copy stage (i.e., the stage of step S100) within the first period, or in the verification and confirmation stage (i.e., the stage of step S200) within the first period.

[0097] S300 : After the end of the first time period, the first virtual machine VM1 is shut down, and state data of the first virtual machine VM1 is migrated.

[0098] In some possible implementations, after receiving a shutdown confirmation signal, the first virtual machine management system VMM1 shuts down the first virtual machine VM and migrates all state information of the first virtual machine VM to a destination host via the first network card system A2 over the second transmission connection. The state information includes the service interaction status of the first virtual machine VM1 and at least one service host CC.

[0099] For example, depending on different actual application scenarios, the state data of the first virtual machine VM1 may be stored in at least one of the first host memory A12, the cache of the first processor A11, and the first network card system A2.

[0100] S400: Restart and run the first virtual machine VM1 on the destination host B.

[0101] In some possible implementations, after the memory data and state data of the first virtual machine VM1 are migrated to the destination host B, the first virtual machine management system VMM1 sends an activation signal to the destination host B via the first network card system A2 based on the first transmission connection; the activation signal is used to instruct the destination host B to start and run the first virtual machine VM1. Exemplarily, the activation signal may be a doorbell signal. Alternatively, the activation signal may instruct the second virtual machine management system VMM2 on the destination host B to restart the timer associated with the first virtual machine VM1, thereby restarting the first virtual machine VM1.

[0102] In the embodiment of the present application, through the above steps S100 to S400, the first virtual machine VM1 is hot migrated to the destination host B when the first virtual machine VM1 is directly connected to the first network card system A2.

[0103] In some possible implementations, as shown in FIG8 , message interaction between the source host A and the service host C is based on the work queue (WQ) and completion queue (CQ) in the first processor A11 and the third processor C11. The WQ contains transaction requests issued by the APP, namely work queue elements (WQEs). The first network card system A2 and the third network card system C3 obtain relevant WQEs from the WQ and perform relevant interaction tasks according to the content indicated by the WQEs. After the first network card system A2 and the third network card system C3 complete the relevant WQEs, they will send completion queue elements (CQEs) to the CQs of the corresponding processors. The CQEs are used to indicate that the relevant WQEs have been completed. Taking RDMA as an example, two types of WQs can be supported: send queues (SQs) and receive queues (RQs). A pair of SQs and RQs can be used as a queue pair (QP). Each network node (i.e., source host A, destination host B, and service host C, etc.) can include one or more QPs. The SQs and RQs between multiple QPs can be independent; alternatively, multiple QPs can use a shared receive queue (SRQ) as the RQ. Each time the network card system 2 completes a WQE, it generates a corresponding CQE and writes it to the CQ of the local host system 1, indicating the completion of the WQE. Similarly, when an exception occurs during the execution of a WQE, the network card system 2 also generates a CQE and writes it to the local host system 1 to record information such as the execution status of the WQE. Communication between two network nodes is established in units of QPs. Communication can be either connected or disconnected. A connection refers to the establishment of a connection between a pair of QPs at the local and remote ends. In disconnected communication, a QP in one network node can communicate with any number of QPs in the other network node. In addition, communication interactions (including connection and non-connection) can be reliable transmission or unreliable transmission. Compared with unreliable transmission, reliable transmission adds a verification link to ensure the transmission reliability of communication interactions. The network layer provides unreliable transmission services. For the demand for reliable transmission, it is necessary to rely on the network model layer above the network layer (i.e., the transport layer), and implement handshake verification between network nodes based on the transport layer to achieve reliable transmission. When reliable transmission is performed between two network nodes, as shown in Figure 8, taking the local source host A sending a data packet message to the remote service host C based on the RDMA send operation as an example, the following steps are included:

[0104] S1: The APP of service host C sends WQE1 including RDMA receive task to the RQ in QP3. WQE1 is used to instruct the third network card system C2 of service host C to prepare to receive relevant data and store it in the corresponding storage area.

[0105] S2: The APP of source host A sends a WQE2 containing an RDMA send task to the SQ of QP1. WQE2 is used to instruct the first network card system A2 of source host A to prepare to obtain relevant data from the memory of source host A and send it to service host C.

[0106] S3. The first host network card system A2 of the source host A obtains WQE2 from the SQ of QP1.

[0107] S4. The first host network card system A2 of the source host A obtains relevant data from the memory space of the source host A based on WQE2 and a DMA read operation, and performs network protocol encapsulation to obtain a data message.

[0108] S5. The first host network card system A2 of the source host A sends the data packet message to the third network card system C2 of the service host C through the physical link of the network model (such as the physical layer and data link layer of the OSI model).

[0109] S6. The third network card system C2 of the service host C takes out WQE1 from the RQ of QP3.

[0110] S7. The third network card system C2 of the service host C stores the parsed data in the memory space indicated by WQE1 based on a DMA write operation.

[0111] S8. After receiving the data message, the third network card system C2 of the service host C performs verification and sends an ACK signal to the first network card system A2 of the source host A.

[0112] S9 . The third network card system C2 of the service host C generates CQE1 and places CQE1 in CQ3 of the service host C.

[0113] S10. The APP of the service host C learns that the task of WQE1 has been completed according to CQE1.

[0114] S11. After receiving the ACK signal of step S5, the first network card system A2 of the source host A generates CQE2 and places CQE2 in CQ1 of the source host AQP1. CQE2 is used to indicate that the task of WQE2 has been completed.

[0115] S12 : The APP in the first host system A1 of the source host A learns, based on CQE2 , that WQE2 has been completed.

[0116] The above steps S1 to S12 are examples of interactions under reliable transmission. In reliable transmission, the sender sends a data message to the receiver based on a certain WQE. After receiving the relevant data message, the receiver feeds back an ACK signal to the sender. After receiving the ACK signal, the sender confirms that the WQE has been completed and writes the corresponding CQE into the CQ at the sender. In unreliable transmission, after the sender's first network card system A2 sends a data message based on the WQE of the SQ of the source host A, it does not need to obtain an ACK signal and directly generates a corresponding CQE to the first host system A1 of the source host A. The APP of the source host A learns that the WQE has been completed based on the CQE. Corresponding to the reliable transmission shown in Figure 8, the unreliable transmission omits the operation of step S8. After executing step S5, the operations of steps S11 and S12 can be directly executed. In addition, the processing order in Figure 8 is only an example. In different implementation scenarios, the implementation order of some steps can be changed. For example, the execution order of step S8, step S9, step S10, step S11 and step S12.

[0117] In some possible implementations, in some network protocols (such as the UB protocol), the transport layer of the source host A (i.e., source host A) is set on the first virtual machine management system VMM1. At this time, multiple VMs in the source host A can share a transport connection at the transport layer. In order to ensure the timeliness of business interaction between the first virtual machine VM1 and at least one business host C, thereby improving the efficiency of hot migration. When the first virtual machine VM1 on the source host A interacts with at least one business host C for business, the second transport connection can be used as the exclusive transport connection for the first virtual machine VM1. By having the first virtual machine VM1 exclusively occupy the second transport connection, the interactive processing time of the business information of the first virtual machine VM1 is accelerated.

[0118] In some possible implementations, the business information initiated by the first virtual machine VM1 and at least one business host C may include first business information and second business information. The first business information is information initiated by at least one business host C to the first virtual machine VM1 through the first network card system A2 (for example, WQE in SQ). The first business information is used to indicate DMA data processing of the first storage area. The second business information is information initiated by the first virtual machine VM1 to at least one business host C through the first network card system A2 (for example, WQE in RQ); the second business information is used to indicate DMA data processing of at least one business host C. Within the first time period, within the time after the first moment, the first network card system A2 needs to process all initiated business signals (including the first business information and the second business information) and transmit the completion information to the first virtual machine VM1.

[0119] In some possible implementations, the first service signal and the second service signal can be complete transactions or transaction segments. For example, in RDMA technology under the RoCE protocol, the first service signal and the second service signal can interact based on complete transactions. For example, in the UB protocol, the first service signal and the second service signal can interact based on transaction segments.

[0120] In some possible implementations, when hot migrating the first virtual machine VM1 on the source host A to the destination host B, the first virtual machine VM1 on the source host A is performing a service interaction operation (e.g., the service interaction operation shown in FIG8 ) related to the aforementioned embodiment with at least one service host C. In the virtual machine migration method shown in FIG7 , in step S130, after each processing of the service information, the service information is subjected to relevant verification processing:

[0121] In some examples, the first network card system A2 verifies the first service information obtained before the first moment. As shown in FIG9 , step S130 may include the following operations: S131 to S135 :

[0122] S131. Input first business information; the first business information is used to instruct a DMA write operation to a first storage area.

[0123] Illustratively, the first service information may be a WQE-related transaction in the RQ recorded in the embodiment of FIG. 8 .

[0124] S132. The first network card system A2 outputs a first write instruction to the first host memory A12 according to the first service information.

[0125] Exemplarily, the first write instruction is an instruction for processing and executing the first business information during the business interaction process. In an embodiment of the present application, a DMA write operation is performed by the first write instruction to write new memory data to the first storage area. In some examples, after the first write instruction is executed, the memory page where the new memory data is located can be marked dirty to obtain first dirty page data.

[0126] S133. The first network card system A2 outputs a first verification instruction to the first host memory A12 according to the first write instruction.

[0127] For example, to ensure that the first write instruction has been executed, the first network card system A2 further generates a corresponding first verification instruction after outputting the first write instruction. Using the first verification instruction, a DMA read operation is performed on the first storage area of ​​the first host memory A12, and the correct execution of the first write instruction is determined based on the DMA read operation. The first verification instruction herein is a newly added instruction based on the interactive service, and is used to verify the completion of the execution of the first service information during the hot migration process.

[0128] S134 . The first network card system A2 outputs a first verification completion signal to the first virtual machine management system VMM1 according to the first verification instruction.

[0129] For example, after determining that the first write instruction is correctly executed, the first network card system A2 outputs a first verification completion signal to the first virtual machine management system VMM1. For example, the first verification completion signal here can be the CQE described in the embodiment shown in FIG8 .

[0130] S135 . The first virtual machine management system VMM1 outputs a first service completion notification to the first virtual machine VM1 according to the first verification completion signal.

[0131] In some examples, after performing a DMA read operation through a first verification instruction to verify whether the first business information is executed, the verified memory page may be marked dirty to obtain first dirty page data.

[0132] In some examples, the first network card system A2 verifies the second service information initiated by the first virtual machine VM1 before the first moment. As shown in FIG10 , step S130 may include the following operations S131′ to S134′:

[0133] S131′: The first network card system A2 initiates a service message to at least one service host C.

[0134] For example, there is a counter related to the VF interface in the first network card system A2. When the first virtual machine VM1 initiates the second business information to the first network card system A2 through its VF interface, the first network card system A2 increases the first count value of the counter by the first value each time it sends a business message to the business host C based on the second business information (or after inputting the second business information initiated by the first virtual machine VM1). The number of initiated but unfinished second business information is verified by the size of the first count value of the counter. When the first value is at a certain initial value, it means that there is no unprocessed second business information. For example, the initial value of the first count value is 0, and the first value is 1. Then, when the first count value is 0, the first network card system A2 can determine that there is no unprocessed second business information at present.

[0135] S132': The first network card system A2 receives the first response information.

[0136] For example, after the first network card system A2 sends a service message to at least one service host C based on the second service signal, the corresponding service host C will send back a first response message (e.g., an ACK message). The first response message is used to indicate that the corresponding service host C has obtained or processed the corresponding second service information. After the first network card system A2 receives the first response signal, it indicates that the corresponding second service information has been processed. The first count value is decremented by the first value.

[0137] S133': The second network card system A2 outputs a second verification completion signal to the first virtual machine management system VMM1 according to the first response information.

[0138] In the embodiment of the present application, the second verification completion signal is used to indicate that the corresponding second service information has been processed.

[0139] S134′: The first virtual machine management system VMM1 outputs a second service completion notification to the first virtual machine VM1 according to the second verification completion signal, to inform the first virtual machine VM1 that the corresponding second indication information has been processed.

[0140] In some possible implementations, the first moment may be the moment when the data volume of the first dirty page data is less than the first threshold. In an embodiment of the present application, when the data volume of the first dirty page data remains greater than or equal to the first threshold, it represents that the business interaction volume between the first virtual machine VM1 and at least one business host C is too large, and the memory data of the first virtual machine VM1 is continuously updated. At this time, when the data volume of the first dirty page data is less than the first threshold, it represents that the business interaction volume of the first virtual machine VM1 is reduced, and this moment can be used as the first moment in the first time period. When the first virtual machine management system VMM1 detects that the data volume of the first dirty page data is less than the first threshold, a first indication signal can be generated. The first indication signal is used to instruct the first network card system A2 to stop interacting with new business signals.

[0141] In some possible implementations, the first moment may be the moment when the initiation rate of the second business information is greater than the second threshold, or the moment when the initiation amount of the second business information is greater than the third threshold. In an embodiment of the present application, both the initiation rate and the production amount of the second business information can reflect the size of the interaction amount of the first virtual machine VM1. When the interaction amount of the first virtual machine VM1 is too large, a first indication signal can also be generated to control the first network card system A2 to stop interacting with new business signals. The embodiment of the present application can avoid the problem of the iterative copy time of hot migration being too long due to the excessive business interaction amount of the first virtual machine VM1, and can also avoid the problem of hot migration failure due to excessive iterative copy time, thereby improving the success rate of hot migration.

[0142] In some possible implementations, based on the operations described in the embodiments of FIG. 10 and FIG. 11 , the first service information and the second service information may be verified respectively in step S200 :

[0143] In some possible implementations, as shown in FIG11 , step S200 may include the following operations: step S210 to step S220:

[0144] S210 : The first virtual machine management system VMM1 outputs a shutdown preparation signal to the first network card system A2 .

[0145] For example, at the second moment, the first network card system A2 has completed the hot migration of all memory data of the first virtual machine VM1. The first virtual machine management system VMM1 outputs a shutdown preparation signal to the first network card system A2, indicating that the iterative migration is complete.

[0146] S220 : The first network card system A2 outputs a shutdown confirmation signal to the first virtual machine association system VMM1 according to the shutdown preparation signal.

[0147] In some possible implementations, after inputting the shutdown confirmation signal, the first network card system A2 verifies whether there is any unprocessed business information. Exemplarily, for the processed first business information, real-time confirmation is performed through the first verification instruction in step S130 of the embodiment shown in FIG9 above. In the verification stage after the second moment, the second business information processed in the embodiment shown in FIG10 is mainly confirmed. Exemplarily, the first network card system A2 can read the first count value, and when the first count value is the initial value (for example, 0), it can be confirmed that all initiated business information has been processed, and a shutdown confirmation signal is output to the first virtual machine association system VMM1. The moment of outputting the shutdown confirmation signal is the end moment of the first time period.

[0148] An embodiment of the present application provides a virtual machine migration method, a chip system, and a host. The chip system is set on the host, and the chip system includes a network card system and a host system. A virtual machine and a virtual machine management system are running on the host system. The virtual machine can interact directly with the network card system without the control and management of the virtual machine management system. The virtual machine on the host can perform business interactions with the business host based on the network card system. During the business interaction, the virtual machine can be hot migrated from the host to the destination host based on the virtual machine migration method. The virtual machine migration method includes: in the process of iteratively copying the first dirty page data in the memory data of the virtual machine, controlling the network card system to stop initiating new business information with at least one business host and processing the initiated business information. After the iterative copying of all the first dirty page data is completed, the virtual machine is shut down and the state information of the virtual machine is migrated. In the embodiment of the present application, the process of shutting down and initiating new business information during the iterative copying of the first dirty page data can ensure that there is no new business information that needs to be processed within a period of time before the virtual machine is shut down, thereby avoiding the situation where the completion information corresponding to the new business information has not yet been transmitted to the virtual machine when the virtual machine is shut down. On the other hand, the previously initiated service information can be processed during the period from the cessation of initiating new service information to the shutdown of the virtual machine, to ensure that the completion signal corresponding to this part of the service information has been delivered to the virtual machine when the virtual machine is shut down. Based on the above solution, the embodiment of the present application realizes the hot migration of the virtual machine in the case of virtual machine direct connection.

[0149] The present application also provides a computer-readable storage medium comprising instructions. When the instructions are executed on a host as described in the above embodiment, the host executes the above methods including those shown in Figures 6, 7, 8, 9, 10, and 11. When the instructions are executed on a chip system (e.g., the first chip system) as described in the above embodiment, the chip system executes the above methods including those shown in Figures 6, 7, 8, 9, 10, and 11.

[0150] The host system and network card system involved in the embodiments of the present application can be a single chip. For example, it can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0151] The memory involved in the embodiments of the present application may be a volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0152] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0153] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0154] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0156] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located on a single device or distributed across multiple devices. Some or all of the modules may be selected to achieve the purpose of this embodiment based on actual needs.

[0157] In addition, the functional modules in the various embodiments of the present application may be integrated into one device, or each module may exist physically separately, or two or more modules may be integrated into one device.

[0158] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more media that can be integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0159] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A virtual machine migration method, characterized in that: The virtual machine migration method is based on a chip system; The chip system includes a network card system and a host system; a virtual machine and a virtual machine management system are running on the host system; and the method includes: The network card system migrates first dirty page data from a first storage area of ​​the host system to a destination host; the first storage area is used to store memory data of the virtual machine; the first dirty page data is dirty page data in the memory data; in the process of migrating the first dirty page data, the network card system stops interacting with at least one business host for subsequent initiated business information according to first indication information, and interacts with the at least one business host based on the initiated business information; the business information is used to indicate business processing between the virtual machine and the at least one business host; After migrating the first dirty page data, the virtual machine management system controls the virtual machine to shut down, and migrates the state data of the virtual machine to the destination host based on the first transmission connection; the state data includes the business interaction state of the virtual machine and the at least one business host.

2. The method according to claim 1, characterized in that After the first dirty page data is migrated, the virtual machine management system controls the virtual machine to shut down, including: The network card system verifies whether all the initiated service information has been processed; when all the initiated service information has been processed, outputs a shutdown confirmation signal to the virtual machine management system of the host system; the shutdown confirmation signal is used to indicate that the initiated service information has been processed; The virtual machine management system controls the virtual machine to shut down according to the shutdown confirmation signal.

3. The method according to claim 2, characterized in that The service information includes first service information; the first service information is information initiated by the at least one service host to the virtual machine through the network card system; The verification of whether all initiated business information has been processed includes: The network card system generates a first write instruction according to the first business information; the first write instruction is used to write new first dirty page data to the first storage area; according to the first write instruction, a corresponding first verification instruction is generated; the first verification instruction is used to verify and read the new first dirty page data from the first storage area; if the new first dirty page data is verified and read from the first storage area, the network card system outputs a corresponding first business completion notification to the host system.

4. The method according to claim 2 or 3, characterized in that: The business information includes second business information; the second business information is information initiated by the virtual machine to the at least one business host through the network card system; the verification of whether all the initiated business information has been processed includes The network card system, when sending a service message to the at least one service host according to the second service information, increases the first count value by the first value; when receiving the first response information, reduces the first count value by the first value; the first response information is used to indicate that the at least one service host has processed the corresponding second service information; when the size of the first count value is the initial value, outputs a second verification completion signal to the virtual machine management system; the second verification completion signal is used to indicate that the corresponding second service information has been processed.

5. The method according to any one of claims 2 to 4, characterized in that: After the migration of the first dirty page data, the virtual machine management system controls the virtual machine to shut down, further comprising: The host system, after migrating the first dirty page data, outputs a shutdown preparation signal to the network card system based on the virtual machine management system; the shutdown preparation signal is used to indicate that the migration of the first dirty page data has been completed; The network card system executes the verification to determine whether all the initiated service information has been processed according to the shutdown preparation signal.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: When the data volume of the first dirty page data is less than a first threshold, the virtual machine management system outputs the first indication signal to the network card system.

7. The method according to claim 4 or 5, characterized in that: The method further comprises: When the initiation rate of the second service information is greater than a second threshold, or when the initiation amount of the second service information is greater than a third threshold, the virtual machine management system outputs the first indication signal to the network card system.

8. The method according to any one of claims 1 to 7, characterized in that: The virtual machine management system is also used to implement network protocol processing at the transport layer; the method also includes: Before the virtual machine management system controls the virtual machine to shut down, the virtual machine management system sends a first connection request to the destination host through the network card system; the first connection request is used to instruct the destination host to establish the first transmission connection with the host system.

9. The method according to any one of claims 1 to 8, characterized in that: The virtual machine management system is also used to implement network protocol processing at the transport layer; the method also includes: The virtual machine and the at least one service host exchange the service information based on a second transmission connection; the second transmission connection is a transport layer connection channel exclusively used by the virtual machine.

10. A chip system, characterized in that: The chip system includes a host system and a network card system; a virtual machine and a virtual machine management system are running on the host system; wherein: The network card system is used to: migrate first dirty page data from a first storage area of ​​the host system to a destination host; the first storage area is used to store memory data of the virtual machine; the first dirty page data is dirty page data in the memory data; in the process of migrating the first dirty page data, according to first indication information, stop interacting with at least one business host for subsequent initiated business information, and interact with the at least one business host based on the initiated business information; the business information is used to indicate business processing between the virtual machine and the at least one business host; The host system is used to: after migrating the first dirty page data, control the virtual machine to shut down based on the virtual machine management system; The network card system is also used to: after the virtual machine is shut down, migrate the state data of the virtual machine to the destination host based on the first transmission connection; the state data includes the service interaction state of the virtual machine and the at least one service host.

11. The chip system according to claim 10, characterized in that: The network card system is also used to: verify whether all the initiated service information has been processed; when all the initiated service information has been processed, output a shutdown confirmation signal to the virtual machine management system of the host system; the shutdown confirmation signal is used to indicate that the initiated service information has been processed; The host system is further configured to: control the virtual machine to shut down according to the shutdown confirmation signal based on the virtual machine management system.

12. The chip system according to claim 11, characterized in that: The service information includes first service information; the first service information is information initiated by the at least one service host to the virtual machine through the network card system; The network card system is further used to: generate a first write instruction according to the first service information; the first write instruction is used to write new first dirty page data to the first storage area; and generate a corresponding first verification instruction according to the first write instruction; The first verification instruction is used to verify and read the new first dirty page data from the first storage area; if the new first dirty page data is verified and read from the first storage area, a first verification completion signal is output to the virtual machine management system; the first verification completion signal is used to indicate that the corresponding first business information has been processed.

13. The chip system according to claim 11 or 12, characterized in that: The service information includes second service information; the second service information is information initiated by the virtual machine to the at least one service host through the network card system; The network card system is also used to: increase the first count value by the first value when sending a service message to the at least one service host according to the second service information; reduce the first count value by the first value when receiving the first response information; the first response information is used to indicate that the at least one service host has processed the corresponding second service information; when the size of the first count value is the initial value, output a second verification completion signal to the virtual machine management system; the second verification completion signal is used to indicate that the corresponding second service information has been processed.

14. The chip system according to any one of claims 11 to 13, characterized in that: The host system is further used to: after migrating the first dirty page data, based on the virtual machine management system, output a shutdown preparation signal to the network card system; the shutdown preparation signal is used to indicate that the migration of the first dirty page data has been completed; The network card system is also used for: executing the verification whether all the initiated service information has been processed according to the shutdown preparation signal.

15. The chip system according to any one of claims 10 to 14, characterized in that: The host system is also used to: When the data volume of the first dirty page data is less than a first threshold, the first indication signal is output to the network card system based on the virtual machine management system.

16. The chip system according to claim 13 or 14, characterized in that: The host system is also used to: When the initiation rate of the second service information is greater than the second threshold, or when the initiation amount of the second service information is greater than When the virtual machine management system reaches a third threshold, the first indication signal is output to the network card system.

17. The chip system according to any one of claims 10 to 16, characterized in that: The virtual machine management system is also used to implement network protocol processing at the transport layer; the host system is also used to: Before controlling the virtual machine to shut down based on the virtual machine management system, sending a first connection request to the destination host through the network card system based on the virtual machine management system; The first connection request is used to instruct the destination host to establish the first transmission connection.

18. The chip system according to any one of claims 10 to 17, characterized in that: The virtual machine management system is also used to implement network protocol processing at the transport layer; the host system is also used to: The virtual machine and the at least one service host are controlled to exchange the service information based on a second transmission connection; the second transmission connection is a transport layer connection channel exclusively used by the virtual machine.

19. A host, characterized in that: The host comprises a circuit board and a chip system as described in any one of claims 10-18; the chip system is arranged on the circuit board.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes instructions; when the instructions are executed on the chip system as described in any one of claims 10-18, the chip system executes the virtual machine migration method as described in any one of claims 1-9; when the instructions are executed on the host as described in claim 19, the host executes the virtual machine migration method as described in any one of claims 1-9.

Images