Stream processing offload using virtual port identifiers

Abstract

Some embodiments of the present invention provide a method for providing flow processing offload (FPO) for a host computer at a physical network interface card (pNIC) connected to the host computer. A set of compute nodes executing on the host computer are each associated with a set of interfaces, wherein each interface is assigned a locally unique virtual port identifier (VPID) by a flow processing and action generator. The pNIC includes a set of interfaces assigned physical port identifiers (PPIDs) by the pNIC. The method includes receiving a data message at an interface of the pNIC and matching the data message with a stored flow entry specifying a destination using the VPID. The method also includes identifying the PPID as the destination of the received data message using the VPID by performing a lookup in a mapping table storing a set of VPIDs and a corresponding set of PPIDs, and forwarding the data message to the interface of the pNIC associated with the identified PPID.

Description

Background Art

[0001] More operations typically associated with servers are being pushed to programmable network interface controllers (NICs). Some of the operations pushed to programmable NICs include stream processing for virtualized compute nodes. As these programmable NICs become more common and perform more stream processing on behalf of virtualized networks, optimizations in stream processing will enhance the capabilities of programmable NICs. Therefore, it is desirable to optimize the stream processing offloaded to programmable NICs. Summary of the Invention

[0002] Some embodiments of the present invention provide a method for configuring a physical network card or physical network controller (pNIC) to provide flow processing offload (FPO) for a host computer connected to the pNIC. The host computer hosts a set of compute nodes (e.g., virtual machines, pods, containers, etc.) in a virtual network. Each compute node in the set of compute nodes is associated with a set of interfaces (virtual NICs, ports, etc.), each of which is assigned a locally unique virtual port identifier (VPID) by the virtual network controller. The pNIC includes a set of interfaces (physical ports connected to a physical network, Peripheral Component Interconnect Express (PCIe) ports, physical functions (PFs), virtual functions (VFs), etc.) assigned physical port identifiers (PPIDs) by the pNIC. The method includes providing a set of mappings between VPIDs and PPIDs to the pNIC. The method also includes sending updates to the mappings when a compute node migrates, connects to a different interface of the pNIC, is assigned a different VPID, etc. The method of some embodiments is performed by a flow processing and action generator. In some embodiments, the flow processing and action generator executes on a processing unit of the host computer, while in other embodiments, the flow processing and action generator executes on a set of processing units of the pNIC that includes flow processing hardware and a set of programmable processing units.

[0003] The method also includes providing a set of flow entries for a set of data message flows associated with a set of compute nodes to the pNIC. In some embodiments, the set of flow entries defines one or both of a set of matching criteria and an action using a VPID. In some embodiments, the action specifies a destination. In some embodiments, each destination is specified based on a VPID, and the pNIC uses a set of mappings to resolve the VPID to a PPID (i.e., an egress interface). In some embodiments, each flow entry is for a specific data message flow and is generated based on a first data message received in the data message flow. In some embodiments, the flow entry is generated based on the results of data message processing performed by a virtual (e.g., software) switch and provided to the pNIC to allow the pNIC to process subsequent data messages in the data message flow.

[0004] In some embodiments, the pNIC stores a set of flow entries and mappings in network processing hardware to perform flow processing for a set of compute nodes executing on a connected host computer. In some embodiments, the flow entries and mappings are stored in a separate memory cache (e.g., a content-addressable memory (CAM), a ternary CAM (TCAM), etc.) to perform fast lookups. In some embodiments, the pNIC receives a data message at an interface of the pNIC and performs a lookup in a set of flow entries stored by the network processing hardware to identify an action for the data message based on matching criteria associated with the data message. In some embodiments, the flow entry includes a set of criteria for identifying a data message flow and an action specifying forwarding the data message to an interface identified by a VPID. If a flow entry exists that specifies a VPID as the destination of the received data message, the pNIC performs a lookup in the VPID to PPID mapping to identify the pNIC's interface associated with the VPID. The pNIC then forwards the data message to the pNIC's interface identified by the PPID mapped to the specified destination VPID.

[0005] In some embodiments, the network processing hardware is further programmed with a default flow entry that identifies an interface of the pNIC as the destination for data messages that do not match other flow entries. In some embodiments, the identified interface is an interface of a virtual (e.g., software) switch that forwards the data message to a flow processing and action generator. In some embodiments, the virtual switch performs first data message processing (e.g., slow path processing) and, based on the results of the processing, returns the flow entry to the network processing hardware for processing subsequent data messages in the data message flow to which the data message belongs.

[0006] Some embodiments provide a method for updating a VPID to PPID mapping when a compute node is connected to a different interface of a pNIC. In some embodiments, the connection to the different interface of the pNIC occurs because the compute node is migrated to a different interface of the pNIC, or even because different host computers are connected to different interfaces of the pNIC when the pNIC provides FPO for multiple host computers. In some embodiments, the connection to the different interface of the pNIC is based on a VM transitioning from passthrough mode (e.g., connected to a VF) to emulated mode (e.g., connected to a PF) or vice versa. In this case, even though the compute node interface is now connected to a different pNIC interface (i.e., has a different PPID), flow entries identifying the VPID of the compute node interface as the destination remain valid. Data messages matching those flow entries are directed to the pNIC interface currently connected to the compute node interface based on a lookup in a mapping table that identifies the updated mapping of VPIDs to the PPID of the currently connected pNIC interface.

[0007] In some embodiments, the method also addresses situations where the pNIC includes multiple physical ports (PPs) connected to a physical network that enables link aggregation (e.g., LACP, trunking, bundling, grouping, etc.) In some embodiments, a mapping of a first VPID to a first PPID of a first PP connected to the physical network is updated to map the first VPID to a second PPID of a second PP connected to the physical network in the event that (1) the first PP fails or (2) a load balancing decision is updated to direct traffic associated with the VPID to the second PP instead of the first PP.

[0008] In some embodiments, even if the vNIC is still connected to the same interface of the pNIC, an updated VPID-to-PPID mapping is required for a compute node interface that is assigned a new VPID after the configuration of the compute node interface has changed. In some embodiments, for any updated VPID-to-PPID mapping, the flow processing and action generator sends an instruction set (e.g., two separate instructions or a single instruction that performs two actions) to remove the invalid VPID-to-PPID mapping and create a new VPID-to-PPID mapping for the updated association between the VPID and PPID. Because the configuration of the compute node interface has changed, some previous data message flows are no longer valid, and any data messages that match the flow entries of those data message flows are redirected to the virtual switch of the flow processing and action generator for evaluation based on the new configuration of the compute node interface. In some embodiments, the redirection to the virtual switch is based on a lookup in the VPID-to-PPID mapping table returning an 'error' (e.g., a null result or other result indicating that there is no entry for the VPID in the mapping table). In some embodiments, data messages that match a flow entry but do not match the VPID to PPID mapping are forwarded to the flow processing and action generator along with an identifier of the flow entry that the data message matches, so as to allow the flow processing and action generator to instruct the pNIC to remove invalid flow entries (i.e., flow entries that point to VPIDs that no longer exist) from the set of flow entries stored by the network processing hardware.

[0009] In some embodiments, the flow processing and action generator stores information about flow entries generated for each VPID identified as a source destination VPID. When the VPID of a particular compute node interface is invalid (e.g., as described above) and a new configuration has taken effect, the flow processing and action generator can identify the flow entry associated with the invalid VPID and instruct the pNIC to remove the identified flow entry from the set of flow entries stored by the network processing hardware. This process does not need to be performed before the configuration change can take effect and can be performed by the flow processing and action generator and the pNIC as a background process when processing power is available.

[0010] Removing flow entries that specify an invalid VPID as a destination allows the VPID to be reused without concern that old flows associated with a compute node interface previously associated with the invalid VPID will be directed to the compute node interface currently associated with the reused VPID. Furthermore, in some embodiments, the network processing hardware performs a process for aging flow entries that have not been used (i.e., no data messages matching the flow entry have been received) for a specific amount of time. Thus, in such embodiments, the VPID can be safely reused even without the flow processing and action generator instructing the pNIC to remove the invalid flow entry after an amount of time based on a specific amount of time (e.g., the specific amount of time plus a timeout for previously active flows directed to the invalid VPID). In some embodiments, the VPID is configured to have more bits than the PPID so that the VPID to PPID mapping is sparse (i.e., there are at least as many unused VPIDs as the number of possible PPIDs).

[0011] In some embodiments, the mapping table is also used to identify the VPID associated with the PPID on which the data message was received. The data message received with the PPID is associated with the VPID to which the PPID is mapped, and a lookup in a set of flow entries is performed based on the VPID and a set of other matching criteria. For a PPID associated with multiple VPIDs, for example, a physical function (PF) of a pNIC connected to an interface of a virtual switch, the interface of the virtual switch being connected to multiple compute node interfaces each having a different VPID, data messages received at the PF have been associated with the VPID to distinguish between traffic from different sources. In addition, for a VPID mapped to a PPID identifying the PF connected to the virtual switch, some embodiments include an indication in the mapping table (e.g., a flag bit associated with the mapping entry) that the VPID should be included with forwarded data messages that match the mapping entry.

[0012] In some embodiments, the mapping table is not programmed with a mapping of the VPIDs connected to the virtual switch, and the network processing hardware is programmed to send any data message that matches a flow entry but does not match an entry in the mapping table to the pNIC interface connected to the virtual switch (i.e., the virtual switch of the flow processing and action generator) and the destination VPID specified in the matching flow entry. The virtual switch can then forward the data message based on the destination VPID of the data message or other matching criteria. In some embodiments, the virtual switch includes a fast path processing pipeline based on stored flow entries and a slow path processing pipeline based on the configuration of the virtual network and the characteristics of the received data message.

[0013] The foregoing summary is intended to serve as a brief introduction to some embodiments of the present invention. It is not an introduction or overview of all inventive subject matter disclosed in this document. The following detailed description and the drawings referenced in the detailed description will further describe the embodiments described in the summary as well as other embodiments. Therefore, in order to understand all the embodiments described in this document, a comprehensive review of the summary, detailed description, drawings, and claims is required. In addition, the claimed subject matter is not limited by the illustrative details in the summary, detailed description, and drawings, but is defined by the appended claims, as the claimed subject matter may be embodied in other specific forms without departing from the spirit of the subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The novel features of the invention are set forth in the appended claims.For purposes of illustration, however, several embodiments of the invention are set forth in the following drawings.

[0015] Figure 1 An exemplary system configured to provide FPO for a host computer at a physical NIC is shown.

[0016] Figure 2 Shown Figure 1 pNIC and fill Figure 1 Flow Processing and Action Generator (FPAG) for the mapping table and flow entry table.

[0017] Figure 3 A more detailed view of the FPAG is shown, which includes a mapping generator for generating VPID to PPID mapping entries and a local controller interacting with a set of SDN controllers and a set of SDN managers.

[0018] Figure 4 A system is shown in which FPAG executes on a set of general-purpose processors of a pNIC.

[0019] Figure 5 A system is shown in which FPAG executes on one of a plurality of servers representing all servers connected to the pNIC.

[0020] Figure 6 The process performed in some embodiments to provide VPID to PPID mappings to be stored in a mapping table of a pNIC to perform flow processing is conceptually illustrated.

[0021] Figure 7 The process for providing flow entries from the flow processing and action generator to the FPO hardware is conceptually illustrated.

[0022] Figure 8 The process for processing a data message received at a pNIC is conceptually illustrated.

[0023] Figure 9 The flow processing hardware of the pNIC is shown storing the mapping table and the flow processing table.

[0024] Figure 10 A VM migration from one virtual function of a pNIC to another virtual function of a pNIC is shown.

[0025] Figure 11 A VM migration from one host computer connected to a virtual function of a pNIC to another host computer connected to a virtual function of a pNIC is shown.

[0026] Figure 12 A system including a VM at time "T1" transitioning from pass-through mode to emulation mode is shown.

[0027] Figure 13 Shown are selected elements of a system in which a change to the configuration of a VM's vNIC results in a change to the vNIC's VPID.

[0028] Figure 14 The process for removing invalid flow entries is conceptually shown as a background process that can be executed when resources are available.

[0029] Figure 15 The process performed by the FPO hardware to remove flow entries that specify invalid VPIDs is conceptually illustrated.

[0030] Figure 16 A system is shown in which link aggregation of physical network ports is enabled.

[0031] Figure 17 A computer system is conceptually illustrated with which some embodiments of the present invention are implemented. DETAILED DESCRIPTION

[0032] In the following detailed description of the invention, many details, examples and embodiments of the present invention are set forth and described. However, it will be clear and apparent to those skilled in the art that the invention is not limited to the embodiments set forth, and that the invention can be implemented without some of the specific details and examples discussed.

[0033] Some embodiments of the present invention provide a method for configuring a physical network card or physical network controller (pNIC) to provide flow processing offload (FPO) for a host computer connected to the pNIC. The host computer hosts a set of computing nodes (e.g., virtual machines (VMs), pods, containers, etc.) in a virtual or logical network. The set of computing nodes are each associated with a set of interfaces (virtual NICs, ports, etc.), each of which is assigned a locally unique virtual port identifier (VPID) by a flow processing and action generator. The pNIC includes a set of interfaces (physical ports connected to a physical network, peripheral component interconnect express (PCIe) ports including physical functions (PFs) and virtual functions (VFs), etc.) that are assigned physical port identifiers (PPIDs) by the pNIC.

[0034] As used herein, physical functions (PFS) and virtual functions (VFs) refer to ports exposed by a pNIC using a PCIe interface. A PF refers to an interface of a pNIC that is identified as a unique resource with a separately configurable PCIe interface (e.g., separate from other PFs on the same pNIC). A VF refers to a virtual interface that cannot be separately configured and is not identified as a unique PCIe resource. In some embodiments, a VF is provided to provide a pass-through mechanism that allows a compute node executing on a host computer to receive data messages from the pNIC without traversing the host computer's virtual switch. In some embodiments, the VF is provided by virtualization software executing on the pNIC.

[0035] In some embodiments, a virtual network includes one or more logical networks, which include one or more logical forwarding elements, such as logical switches, routers, gateways, etc. In some embodiments, a logical forwarding element (LFE) is defined by configuring several physical forwarding elements (PFEs), some or all of which are executed on a host computer along with the deployed compute nodes (e.g., VMs, Pods, containers, etc.). In some embodiments, a PFE is configured to implement two or more LFEs to connect two or more different subsets of the deployed compute nodes. In some embodiments, a virtual network is a virtual network such as one deployed by NSX-T. TM A software-defined network (SDN) is deployed and includes a set of SDN managers and SDN controllers. In some embodiments, the set of SDN managers manages network elements and instructs a set of SDN controllers to configure the network elements to implement the desired forwarding behavior of the SDN. In some embodiments, the set of SDN controllers interacts with a local controller on a host computer to configure the network elements. In some embodiments, these managers and controllers are NSX-T managers and controllers licensed from VMware.

[0036] As used herein, a data message refers to a set of bits in a specific format that is sent across a network. One of ordinary skill in the art will recognize that the term "data message" is used in this document to refer to various formatted sets of bits that are sent across a network. The formatting of these bits can be specified by standardized protocols or non-standardized protocols. Examples of data messages that follow standardized protocols include Ethernet frames, IP packets, TCP segments, UDP datagrams, etc. In addition, as used herein, references to L2, L3, L4, and L7 layers (or Layer 2, Layer 3, Layer 4, and Layer 7) are references to the second data link layer, the third network layer, the fourth transport layer, and the seventh application layer, respectively, of the OSI (Open Systems Interconnection) layer model.

[0037] Figure 1 An exemplary system 100 is shown configured to provide FPO for a host computer 110 at a physical NIC 120. The host computer 110 includes a first set of hosted virtual machines (VMs) 111a-n connected to the pNIC 120 in pass-through mode. Figure 1 In the illustrated embodiment, each VM 111a-n has a virtual NIC (e.g., vNIC 112a-n) that is a virtual function (VF) 133a-n connected to a physical function (PF) 134a of the pNIC 120 via a PCIe bus 131. The virtual machines (111a-n and 113a-m) and virtual switch 115 are shown as executing within virtualization software 114. The VFs 133a-n are provided by virtualization software 135. In some embodiments, the virtualization software 135 is manufacturer virtualization software that provides single root I / O virtualization (SR-IOV), which enables efficient sharing of resources of PCIe-connected devices between compute nodes (e.g., VMs 111a-n). In other embodiments, the virtualization software 135 is a hypervisor (e.g., ESX Server 1000) that is specifically designed to virtualize the resources of intelligent NICs. TM or ESXio TM ).

[0038] In some embodiments, the connection between vNICs 112a-n and VFs 133a-n is enabled by VF drivers 118a-n on host computer 110. Host computer 110 also includes a second set of VMs 113a-m connected to a virtual switch 115 of host computer 110. Virtual switch 115 is connected to pNIC 120 via PF 134m via PCIe bus 131. In some embodiments, PFs 134a-m are also virtualized by virtualization software 135 to appear as separate PCIe-connected devices to host computer 110 or a group of connected host devices. VMs and vNICs are merely one example of compute nodes and interfaces that may be implemented in embodiments of the present invention.

[0039] The pNIC 120 also includes a physical network port 121 that connects the pNIC 120 and VMs 111a-n and vNICs 112a-n to a physical network. The PCIe bus 131 and physical network port 121 are connected to flow processing offload (FPO) hardware 140 to perform flow processing for the VMs 111a-n and vNICs 112a-n. The FPO hardware 140 includes a flow entry table 143 that stores a set of flow entries for performing flow processing. In some embodiments, a flow entry specifies a set of matching criteria and an action to be taken for data messages that match the matching criteria. One or both of the set of matching criteria and the action use a VPID to identify a compute node interface. In some embodiments, additional matching criteria include header values of the data message (e.g., header values associated with L2, L3, L4, etc.). In some embodiments, possible actions include dropping the data message or forwarding the data message to the VPID.

[0040] FPO hardware 140 also includes a mapping table 142. Mapping table 142 includes a set of VPID-to-PPID mappings for resolving a VPID specified in a flow entry to an interface of pNIC 120. Mapping table 142 maps a VPID to a PPID, and the PPID identifies an interface of pNIC 120. In some embodiments, the PPID is assigned by pNIC 120, and the VPID is assigned by a flow processing and action generator (not shown) and associated with a specific interface of pNIC 120. As will be discussed in the following examples, specifying a destination based on a VPID and using a mapping table to identify an interface of a pNIC allows a flow entry to remain valid even if an interface of a compute node changes its association from one interface of a pNIC to another interface of a pNIC.

[0041] Figure 2 Shown Figure 1 The pNIC 120 and the flow processing and action generator (FPAG) 260 that populates the mapping table 142 and the flow entry table 143. In some embodiments, the FPAG 260 replaces Figure 1 FPAG 260 includes a virtual switch 115 (e.g., virtual switch 261 and local cache 262) for forwarding data messages in host computer 110. FPAG 260 includes a local cache 262 that stores all generated flow entries and, for some data messages received at FPAG 260, provides actions to be performed on the received data messages to pNIC 120. In some embodiments, FPAG 260 executes on a host computer (e.g., host computer 110), and local cache 262 serves as a fast path for data message processing that is not offloaded to pNIC 120.

[0042] FPAG 260 also includes a virtual switch 261, which in turn includes a slow path processor 263 and a flow generator 264. Slow path processor 263 performs slow path processing on data messages for which FPO hardware 140 does not store a valid flow entry. Flow generator 264 then uses the results of the slow path processing to generate flow entries to offload flow processing to FPO hardware 140. For example, the slow path processing may indicate that a particular forwarding rule applies to a data message flow and provide a set of criteria that uniquely identifies the flow to which the data message belongs and the action to be taken for future data messages belonging to the flow. In some embodiments, for a particular forwarding rule that uses a reduced set of criteria, the generated flow entry includes a set of wildcard values specified by the flow entry in the matching criteria for those data message characteristics that are not used by the particular forwarding rule to determine an action. Figure 9 The types of criteria and actions that may be specified in a flow entry generated by flow generator 264 are described in more detail.

[0043] In some embodiments, the virtual network is a software defined network (SDN) that includes a set of SDN managers and a set of SDN controllers. Figure 3 A more detailed view of FPAG 260 is shown, which includes a mapping generator 368 for generating VPID to PPID mapping entries and a local controller 365 that interacts with a set of SDN controllers 366 and a set of SDN managers 367. In some embodiments, local controller 365 receives configuration information for locally hosted compute nodes and managed switching elements (e.g., virtual switches 261). In some embodiments, local controller 365 receives VPIDs for compute node interfaces from the set of SDN controllers 366 or locally assigns VPIDs to compute node interfaces. Furthermore, in some embodiments, local controller 365 interacts with pNIC 120 to identify PPIDs for interfaces of pNIC 120 and configure connections between compute node interfaces and interfaces of pNIC 120.

[0044] In some embodiments, the local controller 365 configures the slow path processor 263 with the forwarding rules and additional policies (e.g., firewall policies, encryption policies, etc.) required to implement the data message processing pipeline defined for the SDN (or a set of logical forwarding elements of the SDN). In some embodiments, the local controller 365 also provides information received from the pNIC 120 and the SDN controller 366 to the mapping generator 368 to identify the VPIDs and PPIDs of the different interfaces and the connections between the interfaces to generate the VPID to PPID mappings. In addition, when configuration changes affect the VPID to PPID mappings, the local controller 365 notifies the mapping generator 368 to allow the mapping generator 368 to generate new or updated VPID to PPID mappings and, when applicable, identify mappings that must be deleted. Although the FPAG 260 is Figure 2 and Figure 3 120, but in some embodiments discussed below, FPAG 260 is implemented on the processing unit of pNIC 120.

[0045] Figure 4 A system 400 is shown in which a FPAG 460 executes on a set of general-purpose processors 450 of a pNIC 420 . Figure 4 Also shown is an embodiment in which the pNIC 420 is connected to multiple servers 410a-n at a set of physical functions 434a-i via a PCIe bus 432, each server hosting a set of compute nodes (e.g., VMs 411a-x). In some embodiments, the PCIe bus 432 is a set of separate PCIe buses for connecting to a set of host computers or peripheral devices, and the PFs 434a-i are physically separate interfaces that may or may not be implemented as PFs for separate PCIe buses. The FPAG 460 generates flow entries for each of the servers 410a-n and communicates with other elements of the pNIC 420 using a separate internal PCIe bus 431 (in some embodiments, via physical functions, not shown).

[0046] Figure 5 The system 500 is shown in which the FPAG 560 executes on one server 510a of a plurality of servers 510a-n representing all of the servers 510a-n connected to the pNIC 520. Figure 5Also shown is that in some embodiments, a server (e.g., server 510n) that does not execute FPAG 560 executes virtual switch 515. In some embodiments, virtual switch 515 is a lightweight virtual switch that implements forwarding decisions made by FPAG 560 and does not require a full network stack. In some embodiments, virtual switch 515 connects to a set of emulated VMs 513a-m (e.g., VMs with vNICs that are not configured in pass-through mode).

[0047] The method includes providing a set of mappings between VPIDs and PPIDs to the pNIC. Figure 6 A process 600 is conceptually illustrated that is performed in some embodiments to provide VPID to PPID mappings to be stored in a mapping table on a pNIC for performing flow processing. In some embodiments, process 600 is performed by a flow processing and action generator (e.g., via mapping generator 368) and flow processing offload (FPO) hardware 140. In some embodiments, the flow processing and action generator is implemented on the pNIC, while in other embodiments, the flow processing and action generator is implemented on a host computer connected to the pNIC. Process 600 begins (at 605) by identifying a set of VPIDs associated with compute nodes (e.g., VMs, Pods, containers, etc.) of a device connected to the pNIC. In some embodiments, the flow processing and action generator communicates with a set of network management computers that manage the virtual network to identify a set of compute nodes and the VPIDs associated with the set of compute nodes.

[0048] Process 600 also identifies (at 610) the interfaces of the pNICs connected to the identified compute node interfaces and the PPIDs associated with those pNIC interfaces. In some embodiments, the flow processing and action generator identifies the PPIDs by querying the pNICs for the PPIDs. In some embodiments, the flow processing and action generator is aware of all interfaces of the pNICs and their PPIDs and determines the interface of the pNIC to which each compute node interface is connected.

[0049] Based on the identified VPIDs of the compute node interfaces and the PPIDs of the interfaces of the pNICs to which they are connected, the flow processing and action generator generates (at 615) a set of mappings between the VPIDs and the PPIDs. The generated set of mappings is sent (at 620) to the FPO hardware of the pNIC. In some embodiments, the generated set of mappings is sent to the FPO hardware using a PF connected by a PCIe connection between the processing unit performing the flow processing and action generator and the FPO hardware. As described above, the processing unit performing the flow processing and action generator is a processing unit of the host computer, while in other embodiments, the pNIC is an integrated NIC (e.g., a programmable NIC, a smart NIC, etc.) that includes the processing unit and the FPO hardware.

[0050] The FPO hardware receives (at 625) the VPID-to-PPID mapping sent from the stream processing and action generator. The received VPID-to-PPID mapping is stored (at 630) in a mapping table in the FPO hardware. In some embodiments, the mapping table is stored in a memory cache (e.g., a content addressable memory (CAM), a ternary CAM (TCAM), etc.) that can be used to identify the PPID based on the VPID or identify the VPID based on the PPID. One of ordinary skill in the art will appreciate that process 600 describes the initial mapping of VPIDs to PPIDs, and that in different embodiments, certain operations may represent multiple operations or be performed in a different order (e.g., operation 610 may precede operation 605), and that the description of process 600 is not meant to exclude equivalent processes for achieving the same result.

[0051] The method also includes sending updates to the mapping when a compute node migrates, connects to a different interface of a pNIC, is assigned a different VPID, or the like. Those skilled in the art will appreciate that, in some embodiments, each time the flow processing and action generator detects a change to a VPID or an association between a VPID and a PPID, the revised process 600 for a particular VPID-to-PPID mapping is performed. For example, operation 605 identifies a particular set of VPIDs that were added, moved, or invalidated by a particular configuration change of the virtual network, and operation 610 identifies the current association of the added or moved set of VPIDs with a set of PPIDs of the pNIC. Generating mapping entries (at 615) is performed only for the added or moved set of VPIDs that are mapped to the identified set of PPIDs. Furthermore, in some embodiments, sending the mapping generated for the updated VPID-to-PPID mapping (at 620) includes sending an instruction to remove a previously sent VPID-to-PPID mapping that was invalidated based on the detected configuration change (invalidating a VPID or moving a VPID to connect to an interface identified by a different PPID).

[0052] The method also includes providing a set of flow entries for a set of data message flows associated with a set of compute nodes to the pNIC. In some embodiments, the set of flow entries uses a VPID to define one or both of a set of matching criteria and an action. In some embodiments, the action specifies a destination. In some embodiments, each destination is specified based on a VPID, and the pNIC uses a set of mappings to resolve the VPID to a PPID (i.e., an egress interface). In some embodiments, each flow entry is for a specific data message flow and is generated based on a first data message received in the data message flow. In some embodiments, the flow entry is generated based on the results of data message processing performed by a virtual (e.g., software) switch and provided to the pNIC to allow the pNIC to process subsequent data messages in the data message flow.

[0053] Figure 7 A process 700 for providing flow entries from a flow processing and action generator to FPO hardware is conceptually illustrated. Process 700 begins by receiving (at 705) at the FPO hardware a data message that does not match both: (1) a flow entry for the data message flow to which the data message belongs and (2) a VPID to PPID mapping stored by the FPO hardware. Alternatively, the data message may match only a default rule that identifies an interface connected to the flow processing and action generator as the destination for data messages that match the default rule. In some embodiments, the received data message is the first data message in the data message flow. The FPO hardware forwards (at 710) the data message to the flow processing and action generator (e.g., for slow path processing).

[0054] The stream processing and action generator processes the data message through a processing pipeline (at 715) to determine an action to be taken on a subsequent data message in the same data message stream. For example, in some embodiments, the processing pipeline includes a set of logical forwarding operations and a set of other operations (e.g., firewall, middlebox services, etc.) that result in a decision to either discard a data message of the data message stream or to identify a destination for the data message of the data message stream (possibly encapsulating or decapsulating it before forwarding). In some embodiments, identifying the destination of the data message of the data message stream includes identifying a VPID of a compute node interface that is the destination of the data message of the data message stream.

[0055] Based on (1) the characteristics of the received data message that identify the data message flow to which it belongs and (2) the action to be taken based on processing the data message, the flow processing and action generator (at 720) generates a flow entry for the FPO hardware to use in processing subsequent data messages of the data message flow. The flow processing and action generator (at 725) sends the generated flow entry to the FPO hardware. As described above, in some embodiments, the PF uses a PCIe connection between the processing unit that performs the flow processing and action generator and the FPO hardware to send the generated flow entry to the FPO hardware.

[0056] The FPO hardware receives (at 730) the flow entry sent from the flow processing and action generator. The received flow entry is stored (at 735) in a set of flow entries (e.g., a flow entry table) of the FPO hardware. In some embodiments, the set of flow entries is stored in a memory cache (e.g., a content addressable memory (CAM), a ternary CAM (TCAM), etc.) that can be used to identify flow entries that specify a set of matching criteria associated with the received data message.

[0057] In some embodiments, the pNIC stores the set of flow entries and mappings in network processing hardware to perform flow processing for a set of compute nodes executing on a connected host computer. In some embodiments, the flow entries and mappings are stored in a separate memory cache (e.g., a content addressable memory (CAM), a ternary CAM (TCAM), etc.) to perform fast lookups. Figure 8 A process 800 for processing a data message received at a pNIC is conceptually illustrated. In some embodiments, the process 800 is performed by the FPO hardware of the pNIC. The process 800 begins (at 805) by receiving a data message at an interface of the pNIC to be processed by the FPO hardware. In some embodiments, the data message is one of a data message received at a physical port of the pNIC connected to a physical network and a data message received at an interface of the pNIC connected to a host computer.

[0058] Process 800 determines (at 810) whether a received data message matches a flow entry stored by the FPO hardware. In some embodiments, the determination of whether the FPO hardware has stored a flow entry that matches the received data message is based on a lookup in the stored flow entries according to characteristics of the received data message (e.g., a 5-tuple, header values at different layers of the OSI model, metadata, etc.). If it is determined (at 810) that the received data message does not match the flow entry, the process 800 continues (at 815) with forwarding the data message to a flow processing and action generator for slow path processing, (at 820) with receiving a flow entry for a data message flow to which the received data message belongs, and (at 825) with storing a flow entry for processing subsequent data messages of the data message flow. The above is in conjunction with the steps corresponding to operations 815-825. Figure 7 The discussion of operations 710, 730, and 735 of FIG. 5 describes operations 815-825 in more detail.

[0059] If it is determined that the received data message matches the flow entry, process 800 continues (at 830) by determining whether the matching flow entry specifies that the data message matching the flow entry is to be forwarded to the destination VPID. If process 800 determines that the flow entry specifies that the data message is to be forwarded to the destination VPID, process 800 determines (at 835) whether a mapping exists in the mapping table for the VPID. In some embodiments, determining whether a mapping exists in the mapping table for the VPID includes searching a content addressable memory (CAM) based on the VPID. If process 800 determines (at 830) that the flow entry does not specify a destination VPID (e.g., the flow entry specifies that the data message should be discarded), or if process 800 determines (at 835) that a mapping exists in the mapping table for the VPID, then the action specified in the flow entry is performed (at 800) and the process ends.

[0060] If the process 800 determines (at 835) that the VPID is not in the mapping table, the process 800 returns to operations 815-825. In some embodiments, determining that the VPID is not in the mapping table 142 is based on the VPID lookup returning a default result (associated with operations 815-825) that the data message is directed to the interface associated with slow path processing. In other embodiments, rather than including a default entry in the mapping table 142, some embodiments determine that the VPID is not in the mapping table based on the VPID lookup returning an 'error' (e.g., a null result or other result indicating that there is no entry in the mapping table for the VPID). In some embodiments where a default entry does not exist in the mapping table 142, the FPO hardware 140 is configured to direct all data messages for which an error is returned to the virtual switch. As will be described below with reference to Figure 9 and 13 As described, a flow entry may identify a VPID that is no longer valid if the compute node interface associated with the VPID is reconfigured and assigned a new VPID.

[0061] Figure 9 Flow processing offload hardware 940 of a pNIC is shown storing a mapping table 942 and a flow processing table 943. In some embodiments, the flow processing table 943 is stored in a CAM and includes a set of flow entries 951-956 that specify a set of match criteria 950 and an action 960. In the illustrated embodiment, the set of match criteria 950 includes a source IP address (SIP), a source MAC address (SMAC), a source port (SPort), a destination IP address (DIP), a destination MAC (DMAC), a destination port (DPort), and metadata. In some embodiments, the metadata may be user-configurable or may identify a type of metadata and a matching value for that type of metadata in the set of match criteria.

[0062] For example, flow entries 951 and 952 specify VLAN identifiers in the set of match criteria 950, while flow entry 954 specifies VXLAN identifiers in the set of match criteria 950. In some embodiments, additional types of metadata added internally are also specified, for example, in flow entry 955. Flow entry 955 specifies a set of VPIDs (i.e., VPIDs 0001-0003) as metadata criteria (characteristics) associated with data messages after the PPID identifying the interface of the pNIC that received the data message is converted to a VPID. In some embodiments, VPIDs 0001-0003 are associated with pNIC interfaces connected to a physical network, such that flow entry 955 applies only to data messages received from the physical network.

[0063] In some embodiments, the IP address is specified as a classless inter-domain routing symbol to identify an IP prefix that represents an IP address range (e.g., a range of IP addresses assigned to access that should or should not be granted to a particular other application or user group). For example, flow entry 953 specifies a source IP range IP4 / 28, which indicates an IP address "IP4" and a mask length of 28 bits, such that any IP address that matches the first 28 bits will be a match. Similarly, flow entry 953 specifies a destination IP range IP5 / 30, which indicates an IP address "IP5" and a mask length of 30 bits, such that any IP address that matches the first 30 bits will be a match. In addition, in some embodiments, the flow entry includes at least one criterion using a wildcard value (identified by "*") that is considered to match any value of an associated characteristic of a received data message. For example, rules 952-956 all specify at least one criterion (e.g., a data message characteristic) using a wildcard value.

[0064] In some embodiments, flow entries are prioritized such that for a data message that matches multiple flow entries, the action specified in the flow entry with the highest priority is taken for the data message. In some embodiments, the priority is determined by the specificity of the flow entry's matching criteria when the flow entry is generated during slow path processing and is included in the generated flow entry. In some embodiments, a default rule 956 is specified that directs data messages that do not match any higher priority rule to the VPID associated with slow path processing (e.g., VPID 5000) (e.g., to the virtual switch of the flow processing and action generator).

[0065] In some embodiments, each flow entry includes actions associated with data messages that match the flow entry. In some embodiments, these actions include: forwarding operation (FWD), discarding (DROP) of packets that are not forwarded, modifying the header of a packet and a set of modified headers, copying a packet (along with a set of associated destinations), decapsulating (DECAP) an encapsulated packet that needs to be decapsulated before forwarding to its destination, and encapsulating (ENCAP) a packet that needs to be encapsulated before forwarding to its destination. In some embodiments, some actions specify a series of actions. For example, flow entry 954 specifies that a data message with source IP address "IP6", any source MAC address, source port "Port6", destination IP address "IP7", destination MAC address "MAC7", source port "4789" and metadata indicating that the data message is associated with VXLAN "VXLAN2" is decapsulated and forwarded to VPID "3189". In some embodiments, the identified VPID is a VPID associated with a specific interface of a compute node executed on a host computer. The VPID identified by some flow entries specifying a DECAP action is the VPID for the physical function that is connected to the virtual switch of the flow processing and action generator for processing the decapsulated data message through slow path processing. For other flow entries specifying a DECAP action, the interface identifier (e.g., VPID or PPID) is the identifier of the loopback interface of the FPO hardware to allow the FPO hardware to process the internal data message (the decapsulated data message). In some embodiments, the flow entry specifying the DECAP action also explicitly specifies further processing of the decapsulated data message by the FPO hardware.

[0066] In some embodiments, the mapping table 942 is stored in the CAM and includes a set of VPID to PPID mappings 971-975 that specify a VPID in a "VPID" field 970, a corresponding PPID in a "PPID" field 980, and a flag bit in an "Append VPID" field 990 that indicates whether the VPID associated with the data message should be appended to the forwarded data message. Figure 1 、 6 The mapping tables described in , 7 and 8 are used to resolve the VPID specified in a flow entry to a PPID associated with an interface of the pNIC, and for certain data messages received at the interface of the pNIC, to resolve the PPID to a VPID. Figure 9 It is shown that not every VPID specified in a set of flow entries has an entry in the mapping table. For example, VPID 5472 (specified in flow entry 955) does not have an entry in the VPID field 970.

[0067] For VPIDs not found in the mapping table 942, some embodiments define a default entry 975 that specifies a wildcard 976 in the VPID field 970. Figure 9 In the illustrated embodiment, a default entry 975 is included in the mapping table 942 to direct data messages associated with an invalid VPID to the interface of the pNIC associated with a particular PPID (in this example, PPID 986 ("1111") associated with the interface of the virtual switch connected to the flow processing and action generator). In some embodiments, a data message that matches a default mapping table entry 975 that matches a non-default flow entry (e.g., 951-955) indicates an invalid VPID and flow entry. In other embodiments, rather than including a default entry in the mapping table 942, some embodiments define an action for when a VPID lookup returns an 'error' (e.g., a null result or other result indicating that there is no entry in the mapping table for the VPID). For example, the action specifies that all data messages that match the flow entry but return an error from the VPID lookup will be forwarded to the virtual switch, and in some embodiments, an identifier of the matching flow entry that specifies the invalid VPID is included (e.g., in metadata).

[0068] In this case, some embodiments include a flow entry identifier when forwarding the data message to the virtual switch of the flow processing and action generator. The flow entry identifier is stored in a metadata field or is appended to the data message in a manner that allows the flow processing and action generator to identify that the identified flow entry should be removed from the set of flow entries stored by the FPO hardware. The VPID may be invalid because the associated compute node interface has changed configuration and been assigned a new VPID, or the associated compute node has been shut down. If the compute node interface has been assigned a new VPID, a mapping entry is provided to the mapping table that maps the newly assigned VPID to the PPID of the associated interface of the pNIC, and the flow entry associated with the invalid VPID will eventually be removed, as described above and as referenced in Figure 14 and 15 As further described.

[0069] In some embodiments, multiple VPIDs are associated with a single PPID. For example, mapping table entries 972, 974, and 975 are all associated with PPID 1111. In some embodiments, the appended VPID field 990 is used to identify the destination VPID of the data message that should be forwarded along with the data message. As described above, PPID 1111 is associated with the interface of the pNIC of the virtual switch connected to the flow processing and action generator. In some embodiments, the virtual switch provides a single connection to the pNIC for multiple simulated compute nodes, and appending a VPID (e.g., VPID 2225) allows the virtual switch to use local fast path processing or other forms of minimal processing to forward the data message associated with the VPID to its destination. Furthermore, on the return path, in some embodiments, the data message is associated with the VPID, and the appended VPID flag indicates that the VPID should not be removed before the data message is provided to the FPO hardware 940. In other embodiments, the VPID associated with the data message is retained by default (e.g., stored in a metadata field of the data message). Appending (or reserving) a VPID on the return path allows the FPO hardware 940 to distinguish between different compute nodes connected to the pNIC using the same interface.

[0070] Figure 10-13 Each shows a different type of VM configuration change and an update to a mapping table associated with the VM configuration change. Like numbered elements (eg, 1010, 1110, and 1210) represent similar functional elements. Figure 10 VM 1011a is shown migrating from one virtual function 1033a of pNIC 1020 to another virtual function 1033n of pNIC 1020 at time "T1." In some embodiments, this migration occurs due to a failure of virtual function 1033a or due to other reasons determined by the virtual network's controller. Virtual function 1033a is identified by PPID 9123, and virtual function 1033n is identified by PPID 9234. At time T1, vNIC 1012a of VM 1011a disconnects from virtual function 1033a and connects to virtual function 1033n. Also at (or about) time T1, FPAG 1060 sends an updated VPID-to-PPID mapping for VPID 1000, associating it with PPID 9234 instead of PPID 9123. In other embodiments, the previous association between VPID 1000 and PPID 9123 is deleted, and a new mapping between VPID 1000 and PPID 9234 is added by FPAG 1060. As shown, flow entry table 1043 is the same at time T0 (before T1) and T1, while mapping table 1042 is updated between time T0 and T1.

[0071] Figure 11 The migration of VM 1111a from one host computer 1110a connected to virtual function 1133a of pNIC 1120 to another host computer 1110n connected to virtual function 1133n of pNIC 1120 at time "T1" is shown. Virtual function 1133a is identified by PPID 9123, and virtual function 1133n is identified by PPID 9234. At time T1, VM 1111a is shut down and disconnected from virtual function 1133a, and migrated to host computer 1110n and connected to virtual function 1133n. Also at time T1 (or approximately time T1), FPAG 1160 sends a set of instructions for (1) deleting the previous VPID to PPID mapping and (2) adding a new VPID to PPID mapping for the new connection. As shown, the flow entry table 1143 is the same at time T0 (before T1) and at T1, while the mapping table 1142 is updated between time T0 and T1.

[0072] Figure 12A system 1200 is shown that includes VM 1211a, which transitions from pass-through mode to emulation mode at time "T1." In some embodiments, pass-through mode is a mode in which vNIC 1212a is connected to a virtual function 1233a that allows direct communication between pNIC 1220 and VM 1211a, while emulation mode is a mode in which communication between pNIC 1220 and VM 1211a passes through virtual switch 1215. In some embodiments, virtual switch 1215 is a lightweight virtual switch that does not perform any slow path processing but instead relies on flow processing provided by FPO hardware or FPAG 1260. In the illustrated embodiment, virtual switch 1215 is connected to physical function 1234. In some embodiments, the virtual switch is connected to multiple compute nodes and is connected to the pNIC through a physical function that (1) has greater bandwidth than the virtual function and (2) has greater configurability than the virtual function. Thus, the VPID to PPID mapping associates PPID 1111 of physical function 1234 with multiple VPIDs. At time T1, vNIC 1212a of VM 1211a is disconnected from virtual function 1233a and connected to virtual switch 1215. Also at (or about) time T1, FPAG 1260 sends an updated VPID-to-PPID mapping for VPID 1000 to associate it with PPID 1111 instead of PPID 9123. In other embodiments, the previous association between VPID 1000 and PPID 9123 is deleted, and a new mapping between VPID 1000 and PPID 1111 is added by FPAG 1260. Additionally, the VPID-to-PPID mapping is updated or replaced to change the value in the associated "Append VPID Field" from "0" at time T0 to "1" at time T1 to indicate that the VPID associated with vNIC 1212a should be maintained when forwarding data messages to PF 1234 identified by PPID 1111. As shown, the flow entry table 1243 is the same at time T0 (before T1) and at T1, while the mapping table 1242 is updated between time T0 and T1.

[0073] Figure 131 . Selected elements of system 1300 are shown, wherein a change in the configuration of vNIC 1312a of VM 1311a at time "T1" causes a change in the VPID of vNIC 1312a. VNIC 1312a is connected to virtual function 1333a, which is identified by PPID 9123 both before and after the configuration change. At time T1, vNIC 1312a is reconfigured so that it is actually a different vNIC and is assigned a new VPID 3000. Also at time T1 (or approximately time T1), FPAG 1360 sends a set of instructions to (1) delete the previous VPID to PPID mapping and (2) add a new VPID to PPID mapping for the new VPID. As shown, mapping table 1342 is updated between times T0 and T1 to account for the newly assigned VPID. A mapping table lookup for a flow entry specifying the previous VPID (ie, VPID 1000) as the destination will now result in an error (or hit the default mapping) and be directed to FPAG 1360, as described above with reference to Figure 9 described.

[0074] Figure 10-12 Both illustrate a situation where the VPID identifying a particular compute node interface (i.e., vNICs 1012a, 1112a, and 1212a) remains unchanged throughout the conversion or migration process. In this case, the flow entries provided to the FPO hardware remain valid, and by updating the VPID to PPID mapping table, existing data message flows are directed to the current PPID (and destination compute node interface) without having to update individual flow entries or invalidate existing flow entries before the change takes effect. However, in Figure 13 In the scenario shown, the flow entries for existing flows are invalid (specify a destination VPID that no longer exists). The system considers all flow entries invalid because, in some embodiments and for certain data message flows, configuration changes do not allow or support certain existing data message flows and each data message flow must be revalidated. However, Figure 10-12 As in the scenario above, changes to the compute node interfaces and VPIDs take effect without updating or deleting flow entries.

[0075] Figure 14Conceptually, a process 1400 for removing invalid flow entries is shown, which can be executed as a background process when resources are available. In some embodiments, process 1400 is performed by the FPAG. In some embodiments, process 1400 is performed based on information stored at the FPAG regarding flow entries generated by the FPAG for each VPID, and additionally or alternatively based on information received from the FPO hardware. Process 1400 begins by identifying a VPID that has been invalidated (i.e., no longer associated with a compute node interface). In some embodiments, identifying the invalidated VPID is based on notification from a local controller that the VPID is no longer associated with a compute node interface (e.g., the compute node interface previously associated with the VPID is now associated with a different VPID). In some embodiments, identifying the invalidated VPID includes receiving a data message from the FPO hardware that matches the flow entry but fails to match the VPID-to-PPID mapping. In some embodiments, the data message received from the FPO hardware includes the invalidated VPID in metadata, or a control message is sent along with the data message to identify the invalidated VPID.

[0076] Then, process 1400 identifies (at 1410) a set of flow entries associated with the invalidated VPID. In some embodiments, the FPAG stores each flow entry generated by specifying the VPID as a source or destination. Based on the identified invalidated VPID, the FPAG can identify each entry that specifies the invalidated VPID as a source or destination. In some embodiments, the FPAG does not identify all flow entries associated with the invalidated VPID, but rather identifies flow entries associated with the invalidated VPID based on a data message received from the FPO hardware. In some embodiments, the data message received from the FPO hardware (e.g., in metadata or as content of a control message) includes a flow entry identifier for a flow entry that matches a data message received at the FPO hardware that produced an error (or hit a default rule) from a lookup in the mapping table. One of ordinary skill in the art will appreciate that in some embodiments, ( Figure 7 ) operations 715-725 to generate a new flow entry for the received data message that generated an error from the VPID lookup.

[0077] Then, process 1400 generates (at 1415) an instruction set to remove the identified flow entries from the FPO hardware. In some embodiments, the instruction set is generated as a single instruction to remove multiple flow entries, while in other embodiments, the instruction set includes separate instructions to remove each identified flow entry. In some embodiments, the instruction set is generated as a background process when resources are available.

[0078] The instruction set is sent (at 1420) to the FPO hardware to cause the FPO hardware to remove the flow entries from its memory. The FPO hardware then removes the invalidated flow entries and process 1400 ends. In some embodiments, the FPO hardware also processes the instructions only as a background process that does not consume resources required by other higher priority processes. In some embodiments, the FPO hardware sends an acknowledgment that the identified set of flow entries has been removed to allow the FPAG to reuse the invalidated VPIDs. Process 1400 and the instruction processing at the FPO hardware can be performed as a background process because configuration changes can take effect based on the updated VPID to PPID mapping before the invalidated flow entries are removed. The flow entries are removed to save resources of the FPO hardware and to enable the invalidated VPID to be reused after the flow entries previously generated for the VPID are removed.

[0079] Figure 15 A process 1500 is conceptually illustrated of a flow entry executed by FPO hardware to remove a flow entry that specifies a deprecated VPID. The process 1500 begins (at 1505) by receiving a data message that matches a flow entry that specifies a deprecated VPID as a destination. The data message can be a data message for an existing flow or a data message for a new flow that matches the criteria of a flow entry that specifies a wildcard value or a value range as the matching criteria.

[0080] Then, process 1500 determines (at 1510) that no VPID-to-PPID mapping exists for the VPID specified as the destination in the matching flow entry. In some embodiments, this determination is based on a lookup in the mapping table producing an error or a default mapping being the only match returned. In some embodiments, the identifier of the flow entry that matched the data message is retained (e.g., forwarded with the data message) until a non-default destination is identified.

[0081] Then, process 1500 removes the flow entry from the FPO hardware (at 1515). In some embodiments, if no non-default match is found in the mapping table, the FPO hardware stores the flow entry along with a bit indicating whether the flow entry should be automatically invalidated (e.g., deleted). In some embodiments, the FPO hardware automatically invalidates the flow entry that matches the data message based on the bit stored with the flow entry or as a default behavior that is not based on storing a flag bit with the flow entry, and process 1500 ends. In some embodiments, invalidating the flow entry (at 1515) includes sending a data message to the FPAG identifying the flow entry as a flow entry that does not resolve to the destination PPID (i.e., no non-default match was generated from the lookup in the mapping table). Then, the FPGA executes process 1400 to generate an instruction, which is received by the FPO hardware to invalidate (or remove) the flow entry. Based on the received instruction, the FPO hardware invalidates (or removes) the flow entry, and process 1500 ends.

[0082] In some embodiments, FPO also has an internal process for invalidating (e.g., aging) flow entries based on them not being used for a specified amount of time. In some such embodiments, the FPO hardware stores data regarding the last time a flow entry matched a data message. If the time elapsed since the last time a flow entry matched a data message is greater than an aging threshold time, the flow entry is removed (or invalidated). Thus, after a reuse threshold time at least as long as the aging threshold time, a VPID that was invalidated can be reused. In some embodiments, the reuse threshold time is set to be equal to or greater than the time an average data message flow would time out plus the aging time to ensure that the aging threshold is met in the FPO hardware. To further facilitate VPID reuse, in some embodiments, a VPID is defined to have more bits than a PPID. In some embodiments, the number of bits in a PPID is based on the number of PFs a pNIC has and the number of VFs each PF supports. Assuming a 16-bit PPID, in some embodiments, the VPID is 18 or 20 bits, depending on the desired sparsity of the VPID-to-PPID mapping.

[0083] In some embodiments, the mapping table includes a set of reverse mappings to identify the VPIDs associated with the PPIDs on which data messages were received. In some embodiments, the reverse mappings are generated using a process similar to process 600, but with the PPID-to-VPID and VPID-to-PPID mappings generated (at 615). In some embodiments, the reverse mappings are stored in a separate reverse mapping table. As described above, a particular PPID can be associated with multiple VPIDs. For data messages received from a compute node executing on a host computer, the VPID is appended (or retained) when the data message is provided to the FPO hardware.

[0084] Figure 16 A system 1600 is shown in which link aggregation of physical network ports 1621a-n is enabled. In some embodiments, each physical network port 1621a-n is associated with a different VPID. As shown, all physical ports 1621a-n are included in a link aggregation group 1690. In the event of a physical port failure, the VPID to PPID mapping can be updated so that the VPID associated with the failed physical port is associated with a functioning physical port. Figure 16 The mapping table 1642 is shown before and after a failure of physical port 1621n. The original mapping table 1642 includes a mapping between VPID 00000n and PPID 000n at time T0. After the failure of physical port 1621n, the mapping table 1642 is updated at time T1 to include a new mapping of VPID 0000n to PPID 0001. This allows data messages directed from pNIC 1620 to be sent out of physical port 1621a without invalidating and overwriting any flow entries that specify VPID 0000n as the destination. Figure 16 As shown in T1, at least one physical port is associated with multiple VPIDs. To resolve a PPID to a specific VPID, some embodiments associate a priority with a set of VPID-to-PPID mappings so that the reverse mapping (from PPID to VPID) produces consistent results.

[0085] In addition to fast failover in the event of a link failure without rewriting the flow entries associated with the failed link, the use of a mapping table also allows for updating load balancing decisions made to distribute data messages across multiple physical ports without rewriting the associated flow entries. For example, if the bandwidth of a particular physical port in a link aggregation group changes, then in some embodiments, a group of data messages previously sent to the particular physical port are redirected to a different physical port by updating the VPID to PPID mapping such that the VPID associated with the particular physical port is now mapped to the PPID of the different physical port. In some embodiments, each physical port is assigned multiple VPIDs (e.g., Figure 16(The physical port 1621a is mapped to at least two ports.) In some embodiments, each specific physical port is assigned a primary VPID for reverse lookup and a set of secondary VPIDs, each of which is used to distribute a portion of the data message traffic for egress from the pNIC (e.g., via load balancing via a link aggregation protocol). In some embodiments, the assigned VPIDs are used in a round-robin manner (or some other selection mechanism) when generating flow entries for egress through a specific physical port. In some embodiments, using multiple VPIDs for each port allows updated load balancing decisions to be made with finer granularity. For example, if 10 VPIDs are associated with a single physical port, each VPID can be remapped separately, allowing 10% of the data message load to be rebalanced across the physical port, rather than taking an all-or-nothing approach. Those skilled in the art will understand that the number 10 is provided as an example only, and that more or fewer VPIDs can be assigned to balance the granularity of rebalancing and the complexity of generating flow entries specifying multiple VPIDs for the same destination physical port with the complexity of updating multiple VPID-to-PPID mappings.

[0086] Many of the above features and applications are implemented as software processes, which are specified as sets of instructions recorded on computer-readable storage media (also referred to as computer-readable media). When these instructions are executed by one or more processing units (e.g., one or more processors, processor cores, or other processing units), they cause the (multiple) processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, and the like. Computer-readable media do not include carrier waves and electronic signals transmitted over wireless or wired connections.

[0087] In this specification, the term "software" is intended to include firmware residing in a read-only memory or an application stored in a magnetic storage device, which can be read into a memory for processing by a processor. In addition, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while maintaining different software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that together implement the software inventions described herein is within the scope of the present invention. In some embodiments, the software program, when installed to operate on one or more electronic systems, defines one or more specific machine implementations that run and execute the operations of the software program.

[0088] Figure 17A computer system 1700 is conceptually illustrated, with which some embodiments of the present invention may be implemented. Computer system 1700 may be used to implement any of the aforementioned hosts, controllers, and managers. Thus, it may be used to perform any of the aforementioned processes. The computer system includes various types of non-transitory machine-readable media and interfaces for various other types of machine-readable media. Computer system 1700 includes a bus 1705, processing unit(s) 1710, system memory 1725, read-only memory 1730, permanent storage device 1735, input device 1740, and output device 1745.

[0089] Bus 1705 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of computer system 1700. For example, bus 1705 communicatively connects processing unit(s) 1710 with read-only memory 1730, system memory 1725, and permanent storage 1735.

[0090] Processing unit(s) 1710 retrieve instructions to execute and data to process from these various memory units in order to perform the processes of the present invention. In various embodiments, the processing unit(s) may be a single processor or a multi-core processor. Read-only memory (ROM) 1730 stores static data and instructions needed by processing unit 1710 and other modules of the computer system. Persistent storage device 1735, on the other hand, is a read-write memory device. This device is a non-volatile memory unit that stores instructions and data even when computer system 1700 is powered off. Some embodiments of the present invention use a mass storage device (such as a magnetic or optical disk and its corresponding disk drive) as permanent storage device 1735.

[0091] Other embodiments use removable storage devices (e.g., floppy disks, flash drives, etc.) as permanent storage devices. Like permanent storage device 1735, system memory 1725 is a read-write storage device. However, unlike storage device 1735, system memory is a volatile read-write memory, such as random access memory. System memory stores some instructions and data needed by the processor at runtime. In some embodiments, the processes of the present invention are stored in system memory 1725, permanent storage device 1735, and / or read-only memory 1730. Processing unit(s) 1710 retrieves instructions to execute and data to process from these various memory units in order to perform the processes of some embodiments.

[0092] Bus 1705 also connects to input and output devices 1740 and 1745. Input device 1740 enables a user to communicate information and selection requests to the computer system. Input device 1740 includes an alphanumeric keyboard and a pointing device (also known as a "cursor control device"). Output device 1745 displays images generated by computer system 1700. Output device 1745 includes a printer and a display device, such as a cathode ray tube (CRT) or a liquid crystal display (LCD). Some embodiments include devices such as a touch screen that function as both an input and output device.

[0093] Finally, if Figure 17 As shown, bus 1705 also couples computer system 1700 to a network 1765 via a network adapter (not shown). In this manner, computer system 1700 may be part of a network of computers, such as a local area network ("LAN"), a wide area network ("WAN"), or an intranet, or a network of networks, such as the Internet. Any or all components of computer system 1700 may be used in conjunction with the present invention.

[0094] Some embodiments include electronic components, such as microprocessors, that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as a computer-readable storage medium, machine-readable medium, or machine-readable storage medium). Some examples of such computer-readable media include RAM, ROM, compact disc read-only disk (CD-ROM), compact disc recordable (CD-R), compact disc rewritable (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable Compact discs, ultra-high-density compact discs, any other optical or magnetic media, and floppy disks. Computer-readable media can store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as produced by a compiler, and files containing higher-level code that are executed by a computer, electronic component, or microprocessor using an interpreter.

[0095] While the above discussion primarily relates to microprocessors or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuits themselves.

[0096] As used in this specification, the terms "computer," "server," "processor," and "memory" refer to electronic or other technological devices. These terms do not include people or groups of people. For the purposes of this specification, the term "display" refers to displaying on an electronic device. As used in this specification, the terms "computer-readable medium," "computer-readable media," and "machine-readable medium" are entirely limited to tangible, physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals, or any other short-lived or transient signals.

[0097] While the present invention has been described with reference to numerous specific details, those skilled in the art will recognize that the invention may be embodied in other specific forms without departing from the spirit of the invention. Furthermore, while several of the examples above refer to Pod containers, other embodiments utilize containers other than Pods. Therefore, those skilled in the art will understand that the present invention is not limited by the foregoing illustrative details, but rather is defined by the appended claims.

Claims

1. A method for providing Stream Processing Offload (FPO) to a host computer at a physical network interface card (pNIC) connected to the host computer, the method comprising: receiving, at a first interface of the pNIC, a data message for a particular data message flow, wherein the pNIC stores a flow entry that specifies a particular virtual port identifier (VPID) as a destination for the data message for the particular data message flow; using the particular VPID, identifying a physical port identifier (PPID) as a destination of the received data message by performing a lookup in a mapping table storing a set of VPIDs and a corresponding set of PPIDs; as well as The data message is forwarded to a second interface of the pNIC associated with the identified PPID.

2. The method of claim 1, wherein The flow entry is received from a flow processing and action generator that processes a first data message in the particular data message flow to determine a destination of the first data message and provides the flow entry to the pNIC based on a result of processing the first data message, and The mapping table stores mappings between VPIDs and PPIDs received at the pNIC from the flow processing and action generator, which generates a set of mappings based on configuration data regarding VPIDs and PPIDs received by the flow processing and action generator.

3. The method according to claim 2, wherein: Following a change in data message processing that affects a set of data message flows, the pNIC updates the mapping table based on updates received from the flow processing and action generator, and The changes affecting data message processing include changes related to at least one of (i) an interface of the pNIC and (ii) an interface of a machine executing on the host computer.

4. The method according to claim 3, wherein: The change is a change from an association between a first vNIC of a machine executing on the host computer and a first interface of the pNIC to an association between the first vNIC and a second interface of the pNIC, wherein the first vNIC is associated with a first VPID, the first interface of the pNIC is associated with a first PPID, and the second interface of the pNIC is associated with a second PPID, the updating of the mapping table updates the mapping from the first VPID to the first PPID to a mapping from the first VPID to the second PPID, the received data message is the first data message, and the identified PPID is the first PPID, the method further comprising: After the mapping table is updated, receiving a second data message of the specific data message stream; identifying the second PPID as the destination for the received data message based on (i) the flow entry that specifies the first VPID as the destination for the data message for the particular data message flow and (ii) a lookup in the updated mapping table for the PPID associated with the first VPID; and The data message is forwarded to a second interface of the pNIC associated with a second PPID.

5. The method of claim 4, wherein The second PPID is associated with multiple VPIDs, The mapping of the first VPID to the second PPID includes an indication that data messages destined for the first VPID should be forwarded along with the first VPID to the second interface of the pNIC associated with the second PPID, and Forwarding the second data message to the second interface of the pNIC includes forwarding the data message with a first VPID that is used by a software switch executing on the host computer to direct the second data message to the first vNIC.

6. The method according to claim 5, wherein: Before updating the mapping table, The first PPID is associated only with the first VPID, Data messages destined for the first VPID are delivered to the first vNIC without traversing the software switch, and The VPID is not forwarded with the data message to the second interface of the pNIC associated with the first PPID.

7. The method of claim 3, wherein: The host computer is a first host computer of a plurality of host computers connected to the pNIC, the pNIC provides FPO for each of the plurality of host computers, and each host computer is associated with at least one set of PPIDs that are not associated with different hosts in the plurality of host computers.

8. The method of claim 7, wherein: The change affecting data message processing comprises migration of a machine from a first host computer to a second host computer from the plurality of host computers, and the update received from the stream processing and action generator comprises a new mapping of a VPID associated with the machine to a PPID associated with the second host computer from the set of PPIDs, the new mapping replacing a previous mapping of the VPID associated with the machine to a PPID associated with the first host computer from the set of PPIDs.

9. The method of claim 7, wherein: The flow processing and action generator executes on at least one of a set of processing units of the pNIC and a set of processing units of one of the plurality of host computers.

10. The method of claim 3, wherein said change affecting the processing of data messages is a change in the selection of a physical uplink port for forwarding a particular set of data messages to an external destination, The pNIC includes a first physical uplink port associated with a first PPID and a second physical uplink port associated with a second PPID, In the flow entry associated with the particular set of data messages, the first VPID is designated as the destination VPID, and The updating of the mapping table updates the mapping from the first VPID to the first PPID to the mapping from the first VPID to the second PPID.

11. The method according to claim 10, wherein: The change in selection of the physical uplink port is based on at least one of a failure of the first physical uplink port and a load balancing operation performed to select the physical uplink port for the particular set of data messages.

12. The method of claim 1, wherein: The specific VPID is a first VPID, the specific data message flow is a first data message flow, the data message is a first data message, and the flow entry is a first flow entry, the method further comprising: receiving, at an interface of the pNIC, a second data message of a second data message flow, wherein the pNIC stores a second flow entry specifying a second VPID as a destination for the data message of the second data message flow; determining that a mapping for the second VPID does not appear in the mapping table; forwarding the data message to an interface of the pNIC associated with a flow processing and action generator; receiving a third flow entry that specifies a third VPID as a destination for data messages of a second data message flow, wherein the third VPID appears in the mapping table along with a corresponding PPID; and Remove the second stream entry.

13. The method of claim 12, wherein: Forwarding the data message to the interface of the pNIC includes sending an identifier of a second flow entry along with the forwarded data message to the flow processing and action generator; and Removing the second flow entry includes removing the flow entry in response to an instruction to remove the second flow entry received from the flow processing and action generator.

14. The method of claim 1, wherein: The received data message is received at a third interface of the pNIC associated with a particular PPID, the method further comprising: Converting the specific PPID into a corresponding converted VPID using the mapping table; and A flow entry for the particular data message flow is identified based on the converted VPID.

15. The method of claim 1, wherein: The particular VPID is a first VPID, the particular data message flow is a first data message flow, the data message is a first data message, the flow entry is a first flow entry, the identified PPID is a first PPID, and the pNIC includes a third interface associated with a plurality of VPIDs, the method further comprising: receiving, at a third interface of the pNIC, a second data message of a second data message flow and a second VPID associated with the second data message; identifying, based on the second VPID, a second flow entry that specifies a third VPID as a destination for data messages of a second data message flow; using the third VPID, identifying the second PPID as a destination for the second data message by performing a lookup in the mapping table; and The data message is forwarded to a fourth interface of the pNIC associated with a second PPID.

16. A machine-readable medium storing a program, which, when executed by at least one processor unit, implements the method according to any one of claims 1 to 15.

17. An electronic device comprising: a set of processing units; as well as A machine-readable medium storing a program which, when executed by at least one of the set of processing units, implements the method according to any one of claims 1 to 15.

18. An electronic system comprising means for implementing the method according to any one of claims 1 to 15.

19. A physical network interface card (pNIC) connected to a host computer for performing stream processing offload (FPO) for a group of machines executing on the host computer, the pNIC comprising: a first set of memory elements storing a set of mappings between virtual port identifiers (VPIDs) of interfaces of the set of machines executing on the host computer and physical port identifiers (PPIDs) of interfaces of the pNIC; a second set of memory units storing a set of flow entries for a set of data message flows, each flow entry specifying a destination using a VPID; and The FPO hardware, for a received data message, (1) identifies a flow entry that specifies a VPID as a destination for the data message and (2) uses the VPID and the set of mappings to identify an interface of the pNIC as an egress interface for the received data message.

20. The pNIC of claim 19, wherein The set of mappings between VPIDs and PPIDs is received at the pNIC from a flow processing and action generator, the flow processing and action generator generating the set of mappings based on configuration data received by the flow processing and action generator regarding the VPIDs and PPIDs, and The set of flow entries is received from the flow processing and action generator, which processes a first data message in each data message in the set of data message flows to determine a destination of the first data message, and provides a flow entry for the data message flow in the set of data message flows to the pNIC based on a result of processing the first data message in the data message flow.

21. The pNIC of claim 20, wherein: The flow processing and action generator executes on at least one of a set of processing units of the pNIC and a set of processing units of the host computer.

22. The pNIC of claim 19, wherein: At least one of the first group of memory cells and the second group of memory cells is a content addressable memory cell.

23. The pNIC of claim 19, wherein: The pNIC includes an interface configured as at least one of (1) a physical function and (2) a virtual function connected to a software switch of the host computer.

Images