Real-time workload migration for hardware offload capable NIC using rdma

Abstract

The invention relates to real-time workload migration for a hardware offload capable NIC using RDMA. Some embodiments of the present invention provide a method of migrating a VM from a first host computer to a second host computer, the first host computer having a first PNIC that performs at least one of a network forwarding operation and an intermediate box service operation for the VM. At an RDMA client executing on a set of one or more processors of a first host computer, the method directs an RDMA server executing on a first PNIC to provide networking status data associated with at least one of a network forwarding operation and a middle box service operation performed by the first PNIC for a VM. The provided networking status data resides in a memory of a first PNIC accessible to the RDMA server. At the RDMA client, the method provides the obtained networking state data to the second host computer as part of a data migration performed to migrate the VM from the first host computer to the second host computer.

Description

Technical Field

[0001] The present disclosure relates generally to virtual machines and, in particular, to real-time workload migration for NICs with hardware offload capabilities using RDMA (Remote Direct Memory Access). Background Art

[0002] A Smart NIC is a physical NIC with an embedded arm core that allows virtualization software (e.g., a hypervisor) to offload networking and storage functions to the Smart NIC while saving CPU costs on the virtualization software side. However, during some machine migrations, the state from the Smart NIC associated with the migrating machine is copied from the device to the Smart NIC memory and then transferred to the host computer's memory, and similarly, on the destination host computer, the state is copied from the destination host computer's memory to the Smart NIC memory, all on the host computer's kernel thread. Such an approach is limited due to the limited memory of the Smart NIC and the potentially large size of the data being transferred, resulting in potential failures during data preservation and recovery, especially as the data scales up. Summary of the invention

[0003] Some embodiments of the present invention provide a method for migrating a virtual machine (VM) from a first host computer including a first physical network interface card (PNIC) to a second host computer, the first PNIC performing at least one operation, such as network forwarding or a middlebox service operation, for the VM. The first host computer establishes an event channel with the first PNIC for obtaining networking state data associated with the at least one operation performed by the first PNIC for the VM. The first host computer then uses the event channel to perform an RDMA (Remote Direct Memory Access) operation to obtain the networking state data from the first PNIC, and provides the obtained networking state data to the second host computer as part of a data migration performed to migrate the VM from the first host computer to the second host computer.

[0004] In some embodiments, an RDMA client is configured on the first host computer and an RDMA server is configured on the first PNIC for performing the RDMA operation to transfer the networking state data from the first PNIC to the first host computer. According to some embodiments, the configuration of the RDMA server includes a configuration that enables the RDMA server to access the networking state data of the PNIC. In some embodiments, the RDMA client on the first host computer initiates a connection with the RDMA server on the first PNIC to establish the event channel and request the networking state data from the RDMA server. In response to the request, the RDMA server transfers the networking state data to the RDMA client on the first host computer via the event channel.

[0005] In some embodiments, after the networking state data has been transmitted from the first PNIC to the first host computer, the event channel between the RDMA client and the RDMA server is terminated. In some embodiments, the RDMA client on the first host computer adds the received networking state data to a storage structure, the storage structure being accessible by a VM migration module configured on the first host computer to perform the data migration. In some embodiments, the storage structure is also used to store VM data. In some embodiments, the VM data includes configuration state data associated with a configuration of the VM.

[0006] In some embodiments, the VM migration module configured on the first host computer is a first VM migration module, and a second VM migration module is configured on the second host computer. In some embodiments, the first VM migration module retrieves the networking state data and the VM configuration state data from the storage structure on the first host computer to send to the second VM migration module on the second host computer. After all of the data has been migrated to the second host computer, the PNIC of some embodiments deletes the networking state data stored on the PNIC. In other embodiments, the PNIC deletes the networking state data after the networking state data has been transmitted to the first host computer.

[0007] At the second host computer, the VM state configuration data is used to configure the migrated VM, and the networking state data is used to configure the second PNIC of the second host computer to perform at least one of a network forwarding operation and a middlebox service operation. Examples of network forwarding operations of some embodiments include layer 2 (L2) forwarding operations and layer 3 (L3) forwarding operations. Examples of middlebox service operations of some embodiments include distributed firewall operations, load balancing operations, and network address translation (NAT) operations.

[0008] In some embodiments, an RDMA server is configured on the second host computer, and an RDMA client is configured on the second PNIC to perform an RDMA operation to transfer the networking state data received at the second host computer to the second PNIC as part of the data migration. In some such embodiments, the RDMA client on the second PNIC initiates a connection with the RDMA server on the second host computer to establish an event channel. The RDMA client on the second PNIC then requests the networking state data using the event channel, and the RDMA server on the second host computer performs the RDMA operation to provide the networking state data to the RDMA client on the second PNIC via the event channel. In other embodiments, the networking state data is written directly to the memory of the second PNIC.

[0009] As an alternative to performing RDMA operations between the first PNIC and the first host computer, some embodiments of the present invention perform the RDMA operations between the first PNIC and the second PNIC as part of the data migration for migrating the VM from the first host computer to the second host computer. For example, in some embodiments, once the data migration has started between the first VM migration module and the second VM migration module on the first computer and the second host computer, the second PNIC receives a notification from the second host computer that the data migration has started. Based on the notification, the second PNIC establishes an event channel with the first PNIC of the first host computer and uses the event channel to perform RDMA operations to obtain the networking state data associated with at least one of a network forwarding operation and a middlebox service operation performed by the first PNIC for the VM.

[0010] In some embodiments, the event channel is established by an RDMA client configured on the second PNIC and an RDMA server configured on the first PNIC. The RDMA client on the second PNIC of some embodiments establishes a connection with the RDMA server on the first PNIC to establish the event channel, and then uses the event channel to request the networking state data from the RDMA server. According to some embodiments, the RDMA server on the first PNIC that is configured to access the networking state data performs the RDMA operation in response to the request to transmit the networking state data to the RDMA client on the second PNIC via the event channel. After the networking state data has been transmitted, the event channel between the RDMA server on the first PNIC and the RDMA client on the second PNIC is terminated.

[0011] As the RDMA operation is performed between the first PNIC and the second PNIC, the first VM migration module configured on the first host computer migrates the VM configuration state data to the second VM migration module configured on the second host computer. According to some embodiments, the data migration between the first VM migration module and the second VM migration module and the RDMA operation between the first PNIC and the second PNIC are performed asynchronously. Then, the migrated VM is configured on the second host computer using the VM configuration state data.

[0012] In some embodiments, the first PNIC is associated with a set of physical ports that connect the first PNIC to a physical network, and each port in the set is associated with a specific identifier and a specific IP (Internet Protocol) address assigned to the first PNIC. Before the second PNIC receives notification from the second host computer that the data migration has begun, a compatibility test is performed to ensure that the specific identifier and specific IP address are available to the second PNIC at the second host computer. In some embodiments, when the identifier and IP address are not available, another host computer is identified for the migration of the VM.

[0013] In addition, the first PNIC and the second PNIC of some embodiments each include a virtual switch. In some embodiments, a set of identifiers associated with the physical port is also associated with the VNIC (virtual network interface card) of the VM. In some embodiments, the set of identifiers is added to the virtual switch of the second PNIC, and according to some embodiments, the virtual switch assigns the set of identifiers to a set of interfaces of the virtual switch, associates the set of interfaces with the VNIC of the VM, and uses the interfaces to forward data messages to and from the VNIC.

[0014] The foregoing summary is intended as a brief introduction to some embodiments of the present invention. It is not intended to be an introduction or overview of all creative subject matter disclosed in this document. The following detailed description and the drawings cited in the detailed description will further describe the embodiments described in the summary as well as other embodiments. Therefore, in order to understand all embodiments described by this document, a comprehensive review of the summary, detailed description, drawings and claims is required. In addition, the subject matter claimed is not limited by the illustrative details in the summary, detailed description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The novel features of the invention are set forth in the appended claims.For purposes of illustration, however, several embodiments of the invention are set forth in the following figures.

[0016] Figure 1 Diagram conceptually illustrating VM migration of some embodiments, where data from a smart NIC is transferred to a host computer using RDMA.

[0017] Figure 2A Conceptually illustrated is a process that, in some embodiments, is executed to utilize RDMA to transfer networking state data associated with one or more operations performed by a first PNIC of a first host computer for a particular VM of the first host computer as part of a data migration to migrate the particular VM to a second host computer.

[0018] Figure 2B A process is conceptually illustrated that is performed in some embodiments to utilize RDMA to transfer networking state data associated with one or more operations to be performed by a PNIC of a host computer to which a particular VM has been migrated.

[0019] Figure 3 The hardware of the smart NIC of some embodiments that may be configured to perform network forwarding and middlebox services offload for a host computer is illustrated.

[0020] Figure 4The NIC OS of the smart NIC of some embodiments is conceptually illustrated.

[0021] Figure 5 An example host computer is described that offloads network forwarding operations and middlebox service operations to a PNIC that executes as part of the host computer's physical hardware.

[0022] Figure 6 A more detailed example of a host computer is described that, in some embodiments, is used to establish a distributed architecture for offloading network forwarding operations and / or middlebox service operations to a PNIC (not shown).

[0023] Figure 7 A diagram conceptually illustrating some embodiments in which RDMA is used to transfer networking state data from a first Intelligent NIC of a first host computer to a second Intelligent NIC of a second host computer.

[0024] Figure 8 A process performed in some embodiments to transfer data from a first PNIC of a first host computer to a second PNIC of a second host computer using RDMA is conceptually illustrated.

[0025] Fig. 9 A computer system is conceptually described that implements some embodiments of the present invention. DETAILED DESCRIPTION

[0026] In the following detailed description of the invention, many details, examples and embodiments of the invention are set forth and described. However, it will be clear and apparent to those skilled in the art that the invention is not limited to the embodiments set forth, and the invention may be practiced without some of the specific details and examples discussed.

[0027] Some embodiments of the present invention provide a method for migrating a virtual machine (VM) from a first host computer including a first physical network interface card (PNIC) to a second host computer, the first PNIC performing at least one operation, such as network forwarding or a middlebox service operation, for the VM. The first host computer establishes an event channel with the first PNIC for obtaining networking state data associated with the at least one operation performed by the first PNIC for the VM. The first host computer then uses the event channel to perform an RDMA (remote direct memory access) operation to obtain the networking state data from the first PNIC, and provides the obtained networking state data to the second host computer as part of a data migration performed to migrate the VM from the first host computer to the second host computer.

[0028] In some embodiments, an RDMA client is configured on the first host computer and an RDMA server is configured on the first PNIC for performing the RDMA operation to transfer the networking state data from the first PNIC to the first host computer. According to some embodiments, the configuration of the RDMA server includes a configuration that enables the RDMA server to access the networking state data of the PNIC. In some embodiments, the RDMA client on the first host computer initiates a connection with the RDMA server on the first PNIC to establish the event channel and request the networking state data from the RDMA server. In response to the request, the RDMA server transfers the networking state data to the RDMA client on the first host computer via the event channel.

[0029] In some embodiments, after the networking state data has been transmitted from the first PNIC to the first host computer, the event channel between the RDMA client and the RDMA server is terminated. In some embodiments, the RDMA client on the first host computer adds the received networking state data to a storage structure, the storage structure being accessible by a VM migration module configured on the first host computer to perform the data migration. In some embodiments, the storage structure is also used to store VM data. In some embodiments, the VM data includes configuration state data associated with the configuration of the VM.

[0030] In some embodiments, the VM migration module configured on the first host computer is a first VM migration module, and a second VM migration module is configured on the second host computer. In some embodiments, the first VM migration module retrieves the networking state data and the VM configuration state data from the storage structure on the first host computer to send to the second VM migration module on the second host computer. After all of the data has been migrated to the second host computer, the PNIC of some embodiments deletes the networking state data from the PNIC. In other embodiments, the PNIC deletes the networking state data after the networking state data has been transmitted to the first host computer.

[0031] At the second host computer, the VM state configuration data is used to configure the migrated VM, and the networking state data is used to configure the second PNIC of the second host computer to perform at least one of a network forwarding operation and a middlebox service operation. Examples of network forwarding operations of some embodiments include layer 2 (L2) forwarding operations and layer 3 (L3) forwarding operations. Examples of middlebox service operations of some embodiments include distributed firewall operations, load balancing operations, and network address translation (NAT) operations.

[0032] In some embodiments, an RDMA server is configured on the second host computer, and an RDMA client is configured on the second PNIC to perform an RDMA operation to transfer the networking state data received at the second host computer to the second PNIC as part of the data migration. In some such embodiments, the RDMA client on the second PNIC initiates a connection with the RDMA server on the second host computer to establish an event channel. The RDMA client on the second PNIC then uses the event channel to request the networking state data, and the RDMA server on the second host computer performs the RDMA operation to provide the networking state data to the RDMA client on the second PNIC via the event channel. In other embodiments, the networking state data is written directly to the memory of the second PNIC.

[0033] As an alternative to performing RDMA operations between the first PNIC and the first host computer, some embodiments of the present invention perform the RDMA operations between the first PNIC and the second PNIC as part of the data migration for migrating the VM from the first host computer to the second host computer. For example, in some embodiments, once the data migration has started between the first VM migration module and the second VM migration module on the first computer and the second host computer, the second PNIC receives a notification from the second host computer that the data migration has started. Based on the notification, the second PNIC establishes an event channel with the first PNIC of the first host computer and uses the event channel to perform RDMA operations to obtain the networking state data associated with at least one of a network forwarding operation and a middlebox service operation performed by the first PNIC for the VM.

[0034] In some embodiments, the event channel is established by an RDMA client configured on the second PNIC and an RDMA server configured on the first PNIC. The RDMA client on the second PNIC of some embodiments establishes a connection with the RDMA server on the first PNIC to establish the event channel, and then uses the event channel to request the networking status data from the RDMA server. According to some embodiments, the RDMA server on the first PNIC that is configured to access the networking status data performs the RDMA operation in response to the request to transmit the networking status data to the RDMA client on the second PNIC via the event channel. After the networking status data has been transmitted, the event channel between the RDMA server on the first PNIC and the RDMA client on the second PNIC is terminated.

[0035] As the RDMA operation is performed between the first PNIC and the second PNIC, the first VM migration module configured on the first host computer migrates the VM configuration state data to the second VM migration module configured on the second host computer. According to some embodiments, the data migration between the first VM migration module and the second VM migration module and the RDMA operation between the first PNIC and the second PNIC are performed asynchronously. Then, the VM configuration state data is used to configure the migrated VM on the second host computer.

[0036] In some embodiments, the first PNIC is associated with a set of physical ports that connect the first PNIC to a physical network, and each port in the set is associated with a specific identifier and a specific IP (Internet Protocol) address assigned to the first PNIC. Before the second PNIC receives notification from the second host computer that the data migration has begun, a compatibility test is performed to ensure that the specific identifier and specific IP address are available to the second PNIC at the second host computer. In some embodiments, when the identifier and IP address are not available, another host computer is identified for the migration of the VM.

[0037] In addition, the first PNIC and the second PNIC of some embodiments each include a virtual switch. In some embodiments, a set of identifiers associated with the physical port is also associated with the VNIC (virtual network interface card) of the VM. In some embodiments, the set of identifiers is added to the virtual switch of the second PNIC, and according to some embodiments, the virtual switch assigns the set of identifiers to a set of interfaces of the virtual switch, associates the set of interfaces with the VNIC of the VM, and uses the interfaces to forward data messages to and from the VNIC.

[0038] Figure 1 Diagram 100 conceptually illustrates VM migration of some embodiments, where data from a smart NIC is transferred to a host computer using RDMA. As shown, diagram 100 includes a first host computer 110 and a second host computer 115. Host computer 110 includes a VM 140 to be migrated to host computer 115, a VM migration module (VMM) 150, a VM and networking state data storage device 170, an RDMA client 160, and a smart NIC 120. Host computer 115 includes a VMM 155, a storage device 175 for VM and networking state data to be populated during a data migration operation for migrating VM 140, an RDMA server 165, and a smart NIC 125.

[0039] VMM 150 performs VM migration to migrate VM 140 on host computer 110 to host computer 115. In some embodiments, VMM 150 instantiates a VM from another host computer onto host computer 110. VMM 150 of some embodiments may be implemented in different areas of host computer 110. For example, in some embodiments, VMM 150 is implemented in virtualization software (not shown) of host computer 110 (e.g., a hypervisor of host computer 110), while in other embodiments, VMM 150 is implemented in user space or VM 140.

[0040] Smart NIC 120 includes an RDMA server 130 and a networking element 134 that can access networking state data 132. The networking element 134 of some embodiments stores its state data in an RDMA accessible memory (e.g., networking state data storage device 132) that can be directly accessed by RDMA server 130. In this example, smart NIC memory 136 is shown in dashed outline and is crossed out to visually represent that RDMA server 130 directly accesses networking state data 132 without involving the smart NIC or the host computer's CPU.

[0041] In some embodiments, networking element 134 includes one or more forwarding elements and / or one or more middlebox service elements. Thus, networking state data 132 of some embodiments includes state data associated with L2 and / or L3 forwarding operations performed by networking element 134, and / or state data associated with one or more middlebox service operations (e.g., firewall operations, NAT operations, load balancing operations, IPS operations, IDS operations, etc.) performed by networking element 134.

[0042] In some embodiments, the networking state data includes logical forwarding state data and logical middlebox service state data stored by a logical networking element. For example, in some embodiments, the networking element 134 includes a logical forwarding element that is implemented by multiple management forwarding elements on multiple PNICs and / or multiple host computers. The networking element 134 of some embodiments also includes a logical middlebox service element implemented by multiple physical middlebox service elements.

[0043] As mentioned above, in some embodiments, networking state data store 132 is RDMA accessible memory that can be accessed by RDMA server 130 without involving the CPU. In some embodiments, the RDMA server is able to directly access the memory of smart NIC 120 to retrieve networking state data due to the dedicated hardware of the smart NIC being able to implement such retrieval without involving the CPU. In some embodiments, RDMA server 130 accesses networking state data store 132 as if networking state data store 132 were part of the RDMA server's own memory.

[0044] As part of the data transfer for migrating VM 140 from host computer 110 to host computer 115, RDMA client 160 establishes event channel 138 with RDMA server 130 on smart NIC 120. RDMA client 160 uses event channel 138 to request and receive networking state data 132 from RDMA server 130. During RDMA operations, RDMA server 130 retrieves the networking state data from storage device 132 and transfers the data to RDMA client 160. The networking state data is transferred directly between the buffers of RDMA server 130 and RDMA client 160 via event channel 138 as discrete messages (i.e., as opposed to data streams) using zero-copy networking and without involving the CPU.

[0045] When RDMA client 160 of some embodiments receives the networking state data, RDMA client 160 writes the received networking state data to VM and networking state data store 170 for retrieval by VMM 150. VMM 150 sends the data retrieved from VM and networking state data store 170 to VMM 155 on second host computer 115. On host computer 115, VMM 155 of some embodiments stores the received VM and networking state data in networking state data store 175. Additional details regarding the use of RDMA between a host computer and a PNIC as part of VM migration are described below in more detail. Figure 2A to 2B describe.

[0046] Figure 2A Conceptually illustrating process 201, in some embodiments process 201 is performed to utilize RDMA to transfer networking state data associated with one or more operations performed by a first PNIC of a first host computer for a specific VM of the first host computer as part of a data migration to migrate the specific VM to a second host computer. Process 201 is performed by an RDMA client configured on the first host computer. Process 201 will be described below with reference to Figure 100 described above.

[0047] Process 201 begins when an RDMA client establishes (at 210) an event channel with a first PNIC for use in an RDMA operation to obtain networking state data from the first PNIC. More specifically, the RDMA client establishes an event channel with an RDMA server configured on the first PNIC. The RDMA server is configured with access to the networking state data in order to provide the networking state data to the RDMA client on the first host computer. For example, the host computer 110 includes an RDMA client 160, and the smart NIC 120 includes an RDMA server 130, which can access the networking state data 132. The RDMA client 160 has an event channel 138 established with the RDMA server 130.

[0048] Process 201 uses (at 220) an event channel to send a request to a first PNIC to initiate a data transfer of networking state data from the first PNIC to a first host computer using RDMA. In Figure 100, for example, RDMA client 160 requests networking state data 132 from RDMA server 130 via event channel 138. In some embodiments, in response to the request, RDMA structures required to implement the request are allocated. For example, in some embodiments, after receiving the request from the RDMA client, the RDMA server obtains a context associated with the PNIC, allocates a protection domain for the session, creates a completion channel and completion queue for publishing completion events, generates a work request for completion queue notification, creates a queue pair, and allocates and registers a direct byte buffer for transmitting networking state data.

[0049] The process 201 receives (at 230) requested networking state data from a first PNIC. An RDMA client on a first host computer receives the networking state data from an RDMA server on the first PNIC via an event channel. The requested networking state data includes data associated with one or more network forwarding operations and / or one or more middlebox service operations performed by the first PNIC on a data message for a particular VM.

[0050] Examples of network forwarding operations of some embodiments include layer 2 (L2) forwarding operations and layer 3 (L3) forwarding operations. These L2 and L3 forwarding operations are operations performed by a logical forwarding element implemented on a PNIC in some embodiments. In some embodiments, header values ​​of a data message stream are used to determine where and how to forward data messages belonging to the data message stream. The logical forwarding element implemented by the PNIC of some embodiments performs the L2 and L3 forwarding operations based on the L2 and L3 network addresses.

[0051] In some embodiments, examples of middlebox service operations include firewall service operations, load balancing service operations, network address translation (NAT) service operations, intrusion detection service (IDS) operations, intrusion prevention service (IPS) operations, etc. According to some embodiments, the middlebox service operations are service operations that are offloaded from the host computer to the PNIC. As will be further described below, the operating system (OS) of the PNIC of some embodiments executes a virtualization program that is similar to a hypervisor and enables the resources of the PNIC (e.g., memory, CPU, etc.) to be shared between multiple machines (e.g., multiple VMs) executing on the same host computer as the PNIC. The virtualization program of some embodiments provides computing and / or network virtualization services. In some embodiments, the network virtualization service performs the middlebox service operations.

[0052] Returning to process 201, the process determines (at 240) whether all requested data has been received. When the RDMA client on the first host computer has not received all requested data, the RDMA client continues to wait until all requested data has been received. When the RDMA client on the first host computer has received all requested data, process 201 transitions to terminate (at 250) the event channel between the RDMA client on the first host computer and the RDMA server on the first PNIC. In some embodiments, the RDMA client on the first host computer also generates an event indicating that the event channel is terminated.

[0053] Then, process 201 provides (at 260) the networking state data to the VMM. For example, in diagram 100, RDMA client 160 on host computer 110 provides the networking state data received from RDMA server 130 of intelligent NIC 120 to VM and networking state data store 170, where it can be retrieved by VMM 150. VMM 150 sends the VM and networking state data it retrieved from VM and networking state data store 170 to VMM 155 on the host computer as part of migrating VM 140 from host computer 110 to host computer 115. After 260, process 201 ends.

[0054] In some embodiments, when the networking state data is received by the destination host computer, additional RDMA operations are performed between the destination host computer and the PNIC of the destination host computer. Figure 2B Conceptually, process 202 is performed by an RDMA client configured on a PNIC of a host computer to which a particular VM is migrated in some embodiments to obtain networking state data migrated to the host computer from an RDMA server configured on the host computer. Process 202 will be described below with reference to FIG.

[0055] Process 202 begins when an RDMA client on the PNIC establishes (at 270) an event channel with a host computer to obtain migrated networking state data from the host computer. As illustrated in diagram 100, for example, the smart NIC 125 includes an RDMA client 180, and the host computer 115 includes an RDMA server 165. The RDMA client 180 on the smart NIC 125 and the RDMA server 165 on the host computer 115 are connected by an event channel 188, as shown. In some embodiments, after receiving notification from the host computer that the networking state data has been migrated and is available for transfer to the PNIC, the RDMA client establishes an event channel with the RDMA server.

[0056] Process 202 uses (at 275) the event channel to send a request to the host computer to initiate a data transfer of networking state data from the host computer to the PNIC using RDMA. According to some embodiments, the networking state data on the host computer is stored in an RDMA accessible memory for retrieval by the RDMA server. In Figure 100, for example, RDMA server 165 can access VM and networking state data storage device 175. Although illustrated as a single storage device for VM configuration state data and networking state data, in other embodiments, VM configuration state data and networking state data are stored in separate storage structures.

[0057] The process 202 receives (at 280) the requested networking state data from the host computer. When the RDMA server 165 receives the request for networking state data from the RDMA client 180, the RDMA server 165 of some embodiments retrieves the networking data from the VM and the networking state data store 175 (or other RDMA accessible storage location for storing the networking state data) and transmits the networking state data to the RDMA client 180 via the event channel 188. In some embodiments, the networking state data is transmitted directly from the RDMA client to the memory of the PNIC without involving the CPU.

[0058] The process 202 determines (at 285) whether all requested data has been received. When the RDMA client on the PNIC has not received all requested data, the RDMA client continues to wait until all requested data has been received. When the RDMA client on the PNIC has received all requested data, the process 202 transitions to terminate (at 290) the event channel. In some embodiments, the RDMA client also generates an event indicating that the event channel has been terminated. After 290, the process 202 ends.

[0059] In some embodiments, the smart NIC includes additional elements beyond those illustrated by FIG. 100. For example, Figure 3The hardware of some embodiments of the intelligent NIC 300 that can be configured to perform network forwarding and middlebox service offload for a host computer is illustrated. As shown, the intelligent NIC 300 includes its own general purpose (x86) CPU 315, a set of application specific integrated circuits (ASICs) 310, memory 305, and a configurable peripheral component interconnect express (PCIe) interface 320. In some embodiments, the intelligent NIC 300 also includes a networking state data storage device 325. In some embodiments, the ASICs 310 include at least one I / O ASIC that handles the processing of data messages forwarded to and from the computer, and is controlled at least in part by the CPU 315. In some embodiments, in addition to or in lieu of the ASICs, the intelligent NIC may include a set of configurable field programmable gate arrays (FPGAs).

[0060] Configurable PCIe interface 320 enables Smart NIC 300 to connect to other physical components of a computer system (e.g., x86 CPU, memory, etc.) via the computer system's PCIe bus 330. Through this configurable PCIe interface, Smart NIC 300 can present itself to the computer system as multiple devices, including a data message processing NIC, a hard disk (using Non-Volatile Memory Express (NVMe) over PCIe), or other types of devices. In some embodiments, CPU 315 executes a NIC operating system (OS), which controls ASIC 310 and may also perform other operations. In some embodiments, Network Forwarding and Middlebox Services Offload ASIC 310 performs operations to offload network forwarding and middlebox services from the host computer.

[0061] Figure 4 The NIC OS 420 of the intelligent NIC 400 of some embodiments is conceptually illustrated. In addition to the NIC OS 420, the intelligent NIC 400 also includes an I / O ASIC 410, a physical port 405, and in some embodiments, other ASICs 415. In some embodiments, the NIC OS 420 is executed by the CPU of the intelligent NIC (e.g., CPU 315). This NIC OS 420 includes a PCIe driver 425, a virtual switch 430, and other functions 435.

[0062] The PCIe driver 425 includes multiple physical functions 440, each of which can instantiate multiple virtual functions 445. These different physical functions 440 enable the smart NIC to present itself as multiple different types of devices to the computer system to which it is attached via its PCIe bus 330. For example, in some embodiments, the smart NIC can present itself as a network adapter (for processing data messages to and from the computer system) as well as a non-volatile memory express (NVMe) disk.

[0063] The NIC OS 420 of some embodiments is capable of executing a virtualization program (similar to a hypervisor) that enables sharing of resources (e.g., memory, CPU resources) of the smart NIC between multiple machines (e.g., VMs), provided that those VMs are executed on a computer. In some embodiments, the virtualization program may provide computing virtualization services and / or network virtualization services similar to those of a managed hypervisor. In some embodiments, these network virtualization services include separating data messages into different private (e.g., overlay) networks defined on a physical network (shared between private networks), forwarding data messages for these private networks (e.g., performing switching and / or routing operations), and / or performing middlebox services for private networks.

[0064] To implement these network virtualization services, the NIC OS 420 of some embodiments executes a virtual switch 430. In some embodiments, the virtual switch 430 enables the intelligent NIC to perform software defined networking and provides a set of flow entries (e.g., cache entries described herein) to the I / O ASIC 410 of the intelligent NIC 400 so that the I / O ASIC 410 can perform flow processing offload (FPO) for the computer system. In some embodiments, the I / O ASIC 410 receives data messages from the network and transmits data messages to the network via one or more physical network ports 405.

[0065] Other functions 435 performed by the NIC operating system 400 of some embodiments may include various other operations, including operations not directly related to data message processing (e.g., operations for a machine learning system). In addition, the NIC operating system 400 (the virtual switch 430 or other functions 435 of the operating system) may perform various cache entry validation and invalidation operations and maintain a rule update table for performing cache entry validation.

[0066] As mentioned, the smart NICs of some embodiments process data messages using cache entries (e.g., cache entries installed by a software forwarding element (SFE) executing on a computer for which the smart NIC is an interface) so that at least a subset of data messages received at the smart NIC can be processed without providing the data messages to the SE. Data message processing by the smart NIC ASIC tends to be faster than processing by the SE, even before taking into account the savings realized by avoiding passing data messages to and from the computer (e.g., via a PCIe interface).

[0067] Figure 5 An example host computer 500 is illustrated that offloads network forwarding operations and middlebox service operations to a PNIC 525 that executes as part of physical hardware 520 of the host computer 500. The host computer 500 of some embodiments is linked to other devices (e.g., other host computers) through a physical network 540. The host computer 500 may perform processing of network traffic (e.g., data messages).

[0068] The host computer 500 may include a hypervisor 505 (e.g., virtualization software) and VMs 510-1 to 510-N. As used herein, the term system hypervisor may generally refer to a software layer or component that supports the execution of multiple VMs, including system-level software that supports namespace containers. The host computer 500 provides resources to allow the VMs 510 to send and receive data messages to and from the physical network 540. In some embodiments, the hypervisor 505 includes a network stack 530 for each VM 510 that processes data messages. The network stack 530 may perform operations on the data messages in software using the hypervisor 505. The operations performed may be defined by protocols such as the transmission control protocol (TCP / Internet Protocol (IP)). The network stack 530 may include an input / output (I / O) chain that may perform network operations related to the delivery and reception of data messages and other operations.

[0069] The host computer 500 also includes physical hardware 520, which includes hardware devices for the host computer 500, such as a CPU, storage devices, etc. The physical hardware 520 also includes a PNIC 525, which in some embodiments is a physical interface for the host computer 500 to the physical network 540. For example, the PNIC 525 is an interface for sending data messages from the VM 510 to the physical network 540 and also for receiving data messages from the physical network 540 destined for the VM 510. In a receiving scenario, the PNIC 525 includes a queue (not shown) that stores data messages received by the PNIC 525. The queue buffers incoming data messages received from the physical network 510, but the queue can also perform other operations. The driver 515 then reads the data message corresponding to the queue for further processing by the network stack 530.

[0070] Note that the above explanation and other explanations herein may reflect a common explanation or abstraction of the actual processing mechanism. For example, a data message may not actually be stored in a queue, but instead a descriptor to the data message or a pointer to the data message may be stored in the queue, while the data message itself is actually stored in the host memory. It is conventional to refer to these descriptors / pointers that identify the data message as the data message itself to abstract away the complexity and explain the higher-level operations without burdening the reader with unnecessary technical details of the well-known mechanism. Such abstractions in the description herein should not be interpreted as limiting in any way.

[0071] The VMM 535 may perform VM migration to migrate any of the VMs 510 on the host computer 500 to another host computer, or to instantiate a VM from another host computer onto the host computer 500. However, the VMM 535 may be implemented in other areas, such as in user space or in the VMs 510. When network forwarding and / or middlebox service operations are offloaded from the hypervisor 505 to the PNIC 525, the VMM 535 of some embodiments requests state information about the VM from the PNIC 525 in the event of VM migration.

[0072] Figure 6 A more detailed example of a host computer 600 is illustrated, which in some embodiments is used to establish a distributed architecture for offloading network forwarding operations and / or middlebox service operations to a PNIC (not shown). This host computer 600 includes a service engine 620, a service rule storage 650, a service state information storage 655, a guest VM (GVM) 610, and an SVM 630. The service engine 620 in this example includes a firewall engine 645, but other embodiments may include any or more types of service engines (e.g., discovery, process control, encryption, load balancer, etc.).

[0073] On the host computer 600, the GVM 610 executes on a hypervisor. In addition, the host computer 600 includes an SFE 605. In some embodiments, the SFE 605, the service engine 620, the service rule storage 650, and the service state information storage 655 operate in the kernel space of the hypervisor, while the GVM 610 and the SVM 630 operate in the user space of the hypervisor. In other embodiments, one or more service engines are user space modules (e.g., are SVMs). The SVMs of some embodiments execute in the virtualization software of the PNIC. In some embodiments, during VM migration, in addition to the networking state data stored by the PNIC, the service rule storage 650 and the service state information storage 655 are also migrated to the destination host computer.

[0074] In some embodiments, GVMs 610 are used as data endpoints in a data center. Examples of such machines include network servers, application servers, database servers, etc. In some cases, all VMs belong to one entity, such as an enterprise operating the host. In other cases, the host computer 600 operates in a multi-tenant environment (e.g., in a multi-tenant data center), and different GVMs 610 may belong to one tenant or to multiple tenants.

[0075] As shown, in some embodiments, each GVM 610 also includes a virtual network interface card (VNIC) 615. Each VNIC is responsible for exchanging messages between its VM and the SFE 605. Each VNIC is connected to a specific port 660 of the SFE 605. The SFE 605 is also connected to a PNIC (not shown) of the host computer 600. In some embodiments, a VNIC is a software abstraction created by the hypervisor of one or more PNICs of the host computer 600. In addition, each of the GVMs 610 may include a guest introspection (GI) agent 620 that performs GI operations for the GVM 610.

[0076] In some embodiments, SFE 605 maintains a single port 660 for each VNIC of each GVM 610 and for SVM 630. SFE 605 connects to the host PNIC (through a NIC driver (not shown)) to send outgoing messages and receive incoming messages. In some embodiments, SFE 605 is defined to include port 665, which connects to the driver of the PNIC to send messages to and receive messages from the PNIC. The host PNIC of some embodiments also performs network forwarding and middlebox service operations for the host computer 600.

[0077] SFE 605 performs message processing operations to forward a message received on one of its ports to another of its ports. For example, in some embodiments, the SFE attempts to match the message to a flow-based rule using data in the message (e.g., data in the message header), and when a match is found, performs the action specified by the matching rule (e.g., handing the message to one of its ports 660 or 665, which directs the message to be served to a destination VM or to a PNIC).

[0078] In some embodiments, SFE 605 is a software switch, while in other embodiments, it is a software router or a combined software switch / router. In some embodiments, SFE 605 implements one or more logical forwarding elements (e.g., logical switches or logical routers) where the SFEs are executed on other hosts in a multi-host environment. In some embodiments, the logical forwarding elements may span multiple hosts to connect VMs that are executed on different hosts but belong to one logical network.

[0079] Different logical forwarding elements may be defined to specify different logical networks for different users, and each logical forwarding element may be defined by multiple SFEs on multiple hosts. Each logical forwarding element isolates the traffic of a VM of one logical network from the VM of another logical network served by another logical forwarding element. Logical forwarding elements may connect VMs executed on the same host and / or different hosts. In some embodiments, the SFE extracts a logical network identifier (e.g., VNI) and a media access control (MAC) address from a data message. The SFE in these embodiments uses the extracted VNI to identify a logical port group, and then uses the MAC address to identify a port within the port group.

[0080] A software switch (e.g., a software switch of a hypervisor) is sometimes referred to as a virtual switch because it operates in software and provides shared access to the host's PNIC for VMs. However, in this document, software switches are referred to as physical switches because they are items in the physical world. This term also distinguishes software switches from logical switches, which are abstractions of the types of connections provided by software switches. There are various mechanisms for creating logical switches from software switches. Virtual Extensible Local Area Network (VXLAN) provides a way to create such logical switches. The VXLAN standard is described in the following document: Mahaligam, Mallik; Dutt, Dinesh G.; et al. (2013-05-08), VXLAN: A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks, IETF.

[0081] In some embodiments, port 660 of SFE 605 includes one or more function calls to one or more modules that implement special input / output (I / O) operations on incoming and outgoing messages received at the port. Examples of I / O operations implemented by port 660 include address resolution protocol (ARP) broadcast suppression operations and dynamic host configuration protocol (DHCP) broadcast suppression operations, as described in U.S. Pat. No. 9,548,965. In some embodiments of the present invention, other I / O operations (e.g., firewall operations, load balancing operations, network address translation operations, etc.) may be implemented in this way. In some embodiments, by implementing a stack of such function calls, the port may implement a chain of I / O operations on incoming and / or outgoing messages. In addition, in some embodiments, other modules in the data path (e.g., VNIC 615, port 660, etc.) implement I / O function call operations instead of port 660 or in conjunction with port 660. In some embodiments, SFE 605 uses port 660 to communicate with SVM 630 as it communicates with GVM 610. In other embodiments, to communicate with SVM 630, SFE 605 uses port 660 and SVM interface (SVMI) 625.

[0082] In some embodiments, one or more of the function calls of SFE port 660 may be to one or more service engines 620 that process service rules in service rule storage 650. In some embodiments, each service engine 650 has its own service rule storage 650 and service state information storage 655. The host computer 600 includes only one service rule storage 650 and service state information storage 655 for all service engines so as not to obscure the presentation in this figure with unnecessary detail. In addition, in some embodiments, each GVM 610 has its own instance of each service engine 640 (e.g., its own firewall engine 645). In other embodiments, one service engine may service data message flows for multiple VMs on a host (e.g., VMs for the same logical network).

[0083] The service engine 640 of some embodiments maintains middlebox service operation state information in the storage device 655 to provide to another host computer in the event of migration of one or more of the GVMs 610. In such embodiments, the PNIC of the host computer 600 does not perform middlebox service operations for the host. The service engine 640 can retrieve state information about a particular VM from the storage device 655 to provide to other host computers so that the other host computers will receive service state information for the particular GVM. In other embodiments, when the PNIC performs middlebox service operations for the host computer 600, the PNIC maintains this state information and transmits the state information to the host computer using RDMA, also as described above, or to the PNIC of the destination host computer using RDMA, as will be further described below.

[0084] SVM 630 is a host forwarding module of host computer 600 in some embodiments, which maintains network forwarding state information of GVM 610. In such embodiments, the PNIC of host computer 600 does not perform network forwarding operations for the host. This forwarding operation state information can be provided to SVM 630 by GI agent 620 together with configuration state information. SVM 630 of some embodiments includes a forwarding state information storage device 635, which is used to store state information of each VM (e.g., configuration state information, forwarding operation state information, etc.) to provide to another host computer when one or more VMs are migrated to other hosts. SVM 630 can retrieve state information about a specific VM from storage device 635 to provide to other host computers, so that the other host computers to which the VM is being migrated will receive the forwarding state information of the specific VM. In other embodiments, when the PNIC performs network forwarding operations for host computer 600, the PNIC maintains this state information, and during VM migration, the state information is transmitted from the PNIC to the host computer using RDMA, or from the PNIC to the PNIC of the destination host using RDMA. SVM 630 in these embodiments may still maintain configuration state information for GVM 610 in storage device 635 and, in some embodiments, migrate this configuration state information during data migration from a source host to a destination host.

[0085] As mentioned above, as an alternative to using RDMA between a host computer and a PNIC of a host computer to obtain networking state data for a VM migration operation, such as migrating a VM from a first host computer to a second host computer, some embodiments use RDMA between a first PNIC of a first host computer from which the VM is migrating and a second PNIC of a second host computer to which the VM is migrating.

[0086] Figure 7 Diagram 700 conceptually illustrates some embodiments in which RDMA is used to transfer networking state data from a first intelligent NIC of a first host computer to a second intelligent NIC of a second host computer. As shown, diagram 700 includes host computer 710 and host computer 715. Host computer 710 includes VM 740 to be migrated to host computer 715, VM configuration state data storage 770, VMM 750, and intelligent NIC 720. Host computer 715 includes VMM 755 and intelligent NIC 725.

[0087] Smart NIC 720 includes an RDMA server 730, a networking element 734, and a memory 736 that can access networking state data 732. Smart NIC 725 includes an RDMA client 780, a networking element 784, and a memory 786. Similar to FIG. 100, smart NIC memory 736 is illustrated with a dashed outline and is crossed out to visually represent that the RDMA server 730 directly accesses the networking state data 732 without involving the smart NIC or the OS (e.g., CPU) of the host computer.

[0088] When VMM 750 on the first host computer and VMM 755 on the second host computer begin data transfer as part of migrating VM 140 from first host computer 710 to second host computer 715, VMM 755 on second host computer 715 notifies RDMA client 780 on smart NIC 725 that the data transfer has begun so that RDMA client 780 can establish event channel 738 with RDMA server 730 on smart NIC 720 for requesting and receiving networking status data associated with operations performed by networking element 734 on data messages of VM 740.

[0089] On first host computer 710, VMM 750 retrieves VM configuration state data associated with the configuration of VM 740 and transmits this data to VMM 755, which writes the data to VM configuration state data storage device 775 on host computer 715. VM 740 on host computer 715 is then configured using the VM configuration state data in storage device 775. On smart NIC 725, networking element 784 is configured using the networking state data received from smart NIC 720. Additional details regarding the use of RDMA to transmit networking state data between smart NICs will be described below in detail. Figure 8 Further description.

[0090] Figure 8 Conceptually illustrate a process 800 performed in some embodiments to use RDMA to transfer data from a first PNIC of a first host computer to a second PNIC of a second host computer. In some embodiments, process 800 is performed by an RDMA client configured on the second PNIC. Process 800 will be described below with reference to Figure 700.

[0091] Process 800 begins when the RDMA client on the second PNIC receives (at 810) a notification from the second host computer that data migration for migrating a particular VM from the first host computer to the second host computer has begun. In some embodiments, the RDMA client on the second PNIC receives the notification from the VMM on the second host computer. For example, in diagram 700, RDMA client 780 receives a notification from VMM 755 on host computer 715 once VMM 755 begins data migration with VMM 755 on host computer 710.

[0092] Based on the notification, process 800 establishes (at 820) an event channel with an RDMA server configured on a first PNIC of a first host computer for use in an RDMA operation to obtain networking state data from the first PNIC. This event channel allows networking state data to be transferred from the first PNIC to the second PNIC without involving the CPU of either host. For example, in diagram 700, RDMA client 780 establishes event channel 738 with RDMA server 730.

[0093] Process 800 uses (at 830) an event channel to send a request to an RDMA server of a first PNIC to initiate a data transfer of networking state data from the first PNIC to a second PNIC using RDMA. According to some embodiments, one or more forwarding elements of the first PNIC and one or more middlebox elements of the first PNIC store their networking state data in an RDMA accessible memory. The RDMA server on the first PNIC retrieves the networking state data from the accessible memory and transmits the retrieved networking state data to an RDMA client on the second PNIC via the event channel using RDMA. In diagram 700, for example, RDMA server 730 can access networking state data 732 and transmit the data to RMA client 780 via event channel 788 using RDMA.

[0094] Process 800 receives (at 840) the requested networking state data from the RDMA server of the first PNIC. In some embodiments, as the RDMA client on the second PNIC receives the networking state data, the RDMA client stores the received networking state data in the networking state data storage of the second PNIC. For example, the RDMA client 780 on the host computer 715 can access the networking state data 782. In other embodiments, the networking state data is written directly to the memory 786 of the second PNIC 780.

[0095] Process 800 determines (at 850) whether all requested data has been received. When the RDMA client on the first host computer has not received all requested data, the RDMA client continues to wait until all requested data has been received. When all requested data has been received, process 800 transitions to terminate (at 860) the event channel. In some embodiments, the RDMA client on the second PNIC also generates an event indicating that the event channel has been terminated. After 860, process 800 ends.

[0096] Many of the features and applications described above are implemented as software processes specified as a set of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more processing units (e.g., one or more processors, processor cores, or other processing units), they cause the processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. Computer-readable media do not include carrier waves and electronic signals transmitted wirelessly or through wired connections.

[0097] In this specification, the term "software" means firmware residing in a read-only memory or an application stored in a magnetic storage device, which can be read into a memory for processing by a processor. In addition, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while maintaining different software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Ultimately, any combination of separate programs that implement the software inventions described herein together are within the scope of the present invention. In some embodiments, when the software program is installed to operate on one or more electronic systems, the software program defines one or more specific machine implementations that execute and perform the operations of the software program.

[0098] Fig. 9 A computer system 900 for implementing some embodiments of the present invention is conceptually illustrated. The computer system 900 can be used to implement any of the hosts, controllers, gateways, and edge forwarding elements described above. Thus, it can be used to perform any of the processes described above. This computer system 900 includes various types of non-transitory machine-readable media and interfaces for various other types of machine-readable media. The computer system 900 includes a bus 905, a processing unit 910, a system memory 925, a read-only memory 930, a permanent storage device 935, an input device 940, and an output device 945.

[0099] Bus 905 collectively represents all system, peripheral, and chipset buses that communicatively connect the many internal devices of computer system 900. For example, bus 905 communicatively connects processing unit 910 with read-only memory 930, system memory 925, and permanent storage 935.

[0100] From these various memory units, the processing unit 910 retrieves instructions to be executed and data to be processed in order to perform the processes of the present invention. In different embodiments, the processing unit 910 may be a single processor or a multi-core processor. The read-only memory (ROM) 930 stores static data and instructions required by the processing unit 910 and other modules of the computer system 900. On the other hand, the permanent storage device 935 is a read-write memory device. This device 935 is a non-volatile memory unit that stores instructions and data even when the computer system 900 is turned off. Some embodiments of the present invention use a large-capacity storage device (such as a magnetic disk or optical disk and its corresponding disk drive) as the permanent storage device 935.

[0101] Other embodiments use removable storage devices (e.g., floppy disks, flash drives, etc.) as permanent storage devices. Like permanent storage device 935, system memory 925 is a read-write memory device. However, unlike storage device 935, system memory 925 is a volatile read-write memory, such as random access memory. System memory 925 stores some of the instructions and data that the processor needs at run time. In some embodiments, the processes of the present invention are stored in system memory 925, permanent storage device 935, and / or read-only memory 930. From these various memory units, processing unit 910 retrieves instructions to be executed and data to be processed in order to perform the processes of some embodiments.

[0102] The bus 905 is also connected to input devices 940 and output devices 945. The input devices 940 enable a user to communicate information and select commands to the computer system 900. The input devices 940 include an alphanumeric keyboard and a pointer device (also called a "cursor control device"). The output devices 945 display images generated by the computer system 900. The output devices 945 include printers and display devices, such as cathode ray tubes (CRTs) or liquid crystal displays (LCDs). Some embodiments include devices that function as both input devices 940 and output devices 945 (e.g., a touch screen).

[0103] Finally, if Fig. 9As shown in FIG. 9 , bus 905 also couples computer system 900 to a network 965 through a network adapter (not shown). In this way, computer 900 may be part of a network of computers, such as a local area network ("LAN"), a wide area network ("WAN"), or an intranet, or a network of networks, such as the Internet. Any or all components of computer system 900 may be used in conjunction with the present invention.

[0104] Some embodiments include electronic components such as microprocessors, storage devices, and memories that store computer program instructions in machine-readable or computer-readable media (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, compact disk-read only (CD-ROM), compact disk-recordable (CD-R), compact disk-rewritable (CD-RW), read-only digital versatile disks (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable Blu-ray disks, and the like. CD-ROM, ultra-density CD-ROM, any other optical or magnetic media and floppy disk. Computer readable media can store a computer program executable by at least one processing unit and contain sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as produced by a compiler, and files containing high-level code that are executed by a computer, electronic component or microprocessor using an interpreter.

[0105] Although the above discussion primarily involves microprocessors or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuits themselves.

[0106] As used in this specification, the terms "computer", "server", "processor" and "memory" all refer to electronic or other technical devices. These terms exclude people or groups of people. For the purposes of this specification, the terms "display" or "display" mean displaying on an electronic device. As used in this specification, the terms "computer-readable medium", "computer-readable medium" and "machine-readable medium" are entirely limited to tangible physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other short-lived or transient signals.

[0107] Although the present invention has been described with reference to many specific details, those skilled in the art will recognize that the present invention can be embodied in other specific forms without departing from the spirit of the present invention. Therefore, those skilled in the art will understand that the present invention is not limited by the foregoing illustrative details, but is defined by the appended claims.

Claims

1. A method for migrating a specific virtual machine VM from a first host computer to a second host computer, wherein the first host computer has a first physical network interface card PNIC, and the PNIC performs at least one of a network forwarding operation and a middlebox service operation for the specific VM, the method comprising: At an RDMA remote direct memory access client executing on a set of one or more processors of the first host computer: directing an RDMA server executing on the first PNIC to provide networking state data associated with at least one of a network forwarding operation and a middlebox service operation performed by the first PNIC for the particular VM, wherein the provided networking state data resides in a memory of the first PNIC accessible to the RDMA server; and The obtained networking state data is provided to the second host computer as part of a data migration performed to migrate the particular VM from the first host computer to the second host computer.

2. The method according to claim 1, wherein the guiding comprises: establishing communication with the RDMA server executing on the first PNIC; and An RDMA operation is performed using the established communications to obtain the networking state data from the RDMA server on the first PNIC. 3 . The method of claim 2 , wherein establishing communication with the RDMA server comprises establishing an event channel with the RDMA server.

4. The method of claim 3, wherein the at least one of a network forwarding operation and a middlebox service operation is performed by at least one forwarding element of the first PNIC and at least one middlebox element of the first PNIC, wherein the PNIC forwarding element and the middlebox element store their state data in an RDMA accessible memory.

5. The method of claim 3, wherein after the RDMA client on the first host computer obtains the networking status data from the RDMA server on the first PNIC, the method further comprises terminating the event channel.

6. The method of claim 1, wherein the second host computer uses the networking state data to configure a second PNIC of the second host computer to perform at least one of a network forwarding operation and a middlebox service operation for the particular VM.

7. The method of claim 6, wherein the second host computer configures the second PNIC by directing an RDMA client configured on the second PNIC to (i) establish an event channel with the RDMA server configured on the second host computer, and (ii) perform RDMA operations using the event channel to obtain the networking status data.

8. The method of claim 1, wherein the first PNIC deletes the networking state data from a memory of the first PNIC after the particular VM is migrated from the first host computer to the second host computer.

9. The method of claim 1 , wherein providing the obtained networking state data to the second host computer comprises providing the networking state data to a first VM migration module executing on the first host computer, the first VM migration module providing the networking state data to a second VM migration module executing on the second host computer as part of the data migration of migrating the particular VM from the first host computer to the second host computer.

10. The method of claim 1, wherein the network forwarding operation comprises at least one of a layer 2 (L2) forwarding operation and a layer 3 (L3) forwarding operation.

11. The method of claim 1, wherein the middlebox service operation comprises at least one of a distributed firewall operation, a load balancing operation, and a network address translation (NAT) operation.

12. The method of claim 1, wherein a set of VM data is provided from the first host computer to the second host computer along with the networking state data as part of the data migration.

13. The method of claim 12, wherein the VM data comprises configuration state data associated with a configuration of the particular VM.

14. The method of claim 1, wherein the first PNIC comprises a first Smart NIC.

15. A non-transitory machine-readable medium storing a program for execution by a set of one or more processing units of a first host computer, the program for migrating a specific virtual machine (VM) from the first host computer to a second host computer, the first host computer having a first physical network interface card (PNIC) that performs at least one of a network forwarding operation and a middlebox service operation for the specific VM, the program comprising an instruction set for: At an RDMA remote direct memory access client executing on a set of one or more processors of the first host computer: directing an RDMA server executing on the first PNIC to provide networking state data associated with at least one of a network forwarding operation and a middlebox service operation performed by the first PNIC for the particular VM, wherein the provided networking state data resides in a memory of the first PNIC accessible to the RDMA server; and The obtained networking state data is provided to the second host computer as part of a data migration performed to migrate the particular VM from the first host computer to the second host computer.

16. The non-transitory machine-readable medium of claim 15, wherein the set of instructions for the booting comprises a set of instructions for: establishing communications with the RDMA server executing on the first PNIC; and An RDMA operation is performed using the established communications to obtain the networking state data from the RDMA server on the first PNIC.

17. The non-transitory machine-readable medium of claim 16, wherein the set of instructions for establishing communications with the RDMA server comprises a set of instructions for establishing an event channel with the RDMA server.

18. The non-transitory machine-readable medium of claim 17, wherein the at least one of a network forwarding operation and a middlebox service operation is performed by at least one forwarding element of the first PNIC and at least one middlebox element of the first PNIC, wherein the PNIC forwarding element and the middlebox element store their state data in an RDMA accessible memory.

19. The non-transitory machine-readable medium of claim 17, wherein after the RDMA client on the first host computer obtains the networking status data from the RDMA server on the first PNIC, the program further comprises a set of instructions for terminating the event channel.

20. The non-transitory machine-readable medium of claim 15, wherein the second host computer uses the networking state data to configure a second PNIC of the second host computer to perform at least one of a network forwarding operation and a middlebox service operation for the particular VM by directing an RDMA client configured on the second PNIC to (i) establish an event channel with the RDMA server configured on the second host computer, and (ii) perform an RDMA operation using the event channel to obtain the networking state data.

Images