NUMA-aware output interface selection

Abstract

The invention relates to NUMA aware output interface selection. Some embodiments provide a method for a data message processing apparatus that includes a plurality of network interfaces associated with at least two different non-uniform memory access (NUMA) nodes. The method receives a data message at a first network interface associated with a particular one of the NUMA nodes. Based on the processing of the data message, the method identifies a plurality of output options for the data message. Each of the output options has an equal forwarding cost, and each output option is associated with a respective one of the NUMA nodes. The method selects an output option associated with the particular NUMA node for the data message to avoid cross-NUMA node processing of the data message.

Description

Technical Field

[0001] The present disclosure relates to a NUMA-aware output interface selection. Background Art

[0002] As data center networking becomes more advanced, it is becoming more common for network devices that perform logical network operations to have multiple network interfaces. These different network interfaces may be associated with different NUMA (non-uniform memory access) nodes of the network device. Logical networking operations that bundle interfaces together (e.g., for ECMP purposes) may generate cross-NUMA traffic, which is very resource-intensive. As such, ensuring that data traffic is not sent between NUMA nodes is critical to ensuring that the network device performs efficiently. Summary of the invention

[0003] Some embodiments provide a method for a data message processing device having multiple network interfaces associated with multiple different non-uniform memory access (NUMA) nodes to select between otherwise equivalent output options for a data message in a NUMA-aware manner. When the data message processing device receives a data message at a first network interface associated with a first NUMA node, and identifies multiple output options for the data message with equal forwarding costs based on processing of the data message, the device selects one of the output options associated with the first NUMA node. That is, the device applies a preference for using an output option associated with the same NUMA node at which the data message is received to avoid cross-NUMA node processing of the data message.

[0004] In some embodiments, the data message processing device is an edge device that processes data messages between a logical network implemented in a data center and an external network (i.e., a network external to the logical network). In some embodiments, the edge device implements a logical router having multiple uplink interfaces connected to an external router and a set of logical switches connected to the logical router (or an intermediate logical router). Each of the uplinks is associated with one or more of the network interfaces of the edge device. Logical network endpoints executed on a host computer in the data center are (logically) connected to the logical switches, and for each logical switch, the edge device uses multiple tunnel endpoints (e.g., virtual tunnel endpoints (VTEPs)) to encapsulate data traffic directed to these logical network endpoints.

[0005] When the edge device receives a data message sent from a logical network endpoint and directed to an external destination, the edge device performs logical router processing on the data message and identifies an uplink for outputting an outgoing data message. In some embodiments, the logical router processing identifies an equal cost multipath (ECMP) rule having multiple routes with the same forwarding cost. Each of these routes specifies a corresponding external router next hop address and a corresponding output network interface (or a logical router uplink corresponding to an output network interface). In some embodiments, the edge device stores various metadata (e.g., logical context information) with data messages, including the NUMA node associated with the network interface at which the data message is received. If the ECMP rule includes an indicator of the NUMA node associated with each route (i.e., based on the output uplink specified for the route), the edge device can narrow the potential list of ECMP routes to only routes associated with the NUMA node on which the data message is received. This may be a single route or multiple routes, depending on the number of uplinks associated with each NUMA node. In the latter case, the edge device may perform a load balancing operation (eg, using a hashing operation) among the ECMP routes associated with the particular NUMA node.

[0006] A similar principle is applied to data messages received from the external network at the edge device and directed to a logical network endpoint. In this case, the edge device applies a logical router process that identifies a logical switch (e.g., based on the destination network address of the data message). The edge device then applies the process for this logical switch and determines the destination host computer (corresponding to the destination tunnel endpoint) on which the logical network endpoint operates based on the destination data link address. In some cases, the edge device has the option of using multiple different source tunnel endpoints (e.g., VTEPs) to send the data message to the destination host computer through the physical network via the source tunnel endpoints, and the source tunnel endpoints have equal forwarding costs. These source tunnel endpoints may correspond to different network interfaces of the edge device, and therefore to different NUMA nodes. In this way, the edge device again uses the metadata indicating the NUMA node associated with the network interface at which the data message is received as a preference to narrow the list of potential tunnel endpoints, thereby avoiding cross-NUMA node processing (and then load balancing between the tunnel endpoints associated with the NUMA nodes).

[0007] Additionally, for incoming or outgoing data messages, in some cases, the edge device data path (performing logical network processing) will need to send the data message to another application executing on the edge device (and then, possibly receive the data message back from this application). Specifically, in some embodiments, the data path performs only Layer 2 to Layer 4 (L2-L4) operations (e.g., switching, routing, network address and port translation, L4 load balancing), while other applications executing on the edge device perform Layer 5 to Layer 7 (L5-L7) operations (e.g., L7 load balancing, TLS proxy, URL filtering, etc.). To send data messages to such applications, in some embodiments, the data path outputs the data message to a transport interface (e.g., a kernel network interface (KNI) that transmits the data message from user space to a network stack in the kernel space of the device). Like the network interface, the transport interfaces are each associated with a respective NUMA node. Thus, when selecting a transport interface via which to send a data message to an L7 application, the data path again prefers the interface associated with the NUMA node on which the data message was received (and if multiple transport interfaces are associated with this NUMA node, then load balancing is performed across these interfaces).

[0008] The foregoing summary is intended as a brief introduction to some embodiments of the present invention. It is not meant to be an introduction or overview of all inventive subject matter disclosed herein. The following detailed description and the drawings referenced in the detailed description will further describe the embodiments described in the detailed description as well as other embodiments. Therefore, in order to understand all the embodiments described in this document, a comprehensive review of the summary, the detailed description, and the drawings is required. Moreover, the claimed subject matter is not limited by the illustrative details of the summary, the detailed description, and the drawings, but is defined by the appended claims, because the claimed subject matter may be embodied in other specific forms without departing from the spirit of the subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The novel features of the invention are set forth in the appended claims. However, for purposes of illustration, several embodiments of the invention are set forth in the following figures.

[0010] Figure 1 The process of some embodiments for processing data messages in a NUMA-aware manner is conceptually illustrated.

[0011] Figure 2 An example of a logical network of some embodiments is conceptually illustrated.

[0012] Figure 3 Conceptual map illustrating implementation Figure 2An edge device of some embodiments of one of the service routers of a logical network.

[0013] Figures 4A to 4B Conceptual illustration by Figure 3 The edge device processes northbound data messages.

[0014] Figures 5A to 5B Conceptual illustration by Figure 3 The edge device processes southbound data messages.

[0015] Figure 6 Conceptually illustrates the path of sending a data message from a data path to an L7 application on an edge device according to some embodiments.

[0016] Figures 7A to 7B Conceptual illustration by Figure 3 The edge device processes northbound data messages sent to L7 applications.

[0017] Figures 8A to 8B Conceptual diagram illustrating the processing of northbound data messages by an edge device.

[0018] Fig. 9 The diagram conceptually illustrates an electronic system in which some embodiments of the invention are implemented. DETAILED DESCRIPTION

[0019] In the following detailed description of the invention, numerous details, examples and embodiments of the invention are stated and described. However, it will be clear and apparent to those skilled in the art that the invention is not limited to the embodiments stated, and that the invention can be practiced without some of the specific details and examples discussed.

[0020] Some embodiments provide a method for a data message processing device having multiple network interfaces associated with multiple different non-uniform memory access (NUMA) nodes to select between otherwise equivalent output options for a data message in a NUMA-aware manner. When the data message processing device receives a data message at a first network interface associated with a first NUMA node, and identifies multiple output options for the data message with equal forwarding costs based on processing of the data message, the device selects one of the output options associated with the first NUMA node. That is, the device applies a preference for using an output option associated with the same NUMA node at which the data message is received to avoid cross-NUMA node processing of the data message.

[0021] Figure 1Conceptually illustrates a process 100 for some embodiments of processing data messages in a NUMA-aware manner. In some embodiments, process 100 is performed by an edge device that processes data messages between a logical network implemented in a data center and an external network (i.e., a network external to the logical network), although the process may be performed by other data message processing devices having multiple NUMA nodes. The device may be a bare metal device having multiple physical NUMA nodes, a virtual machine (VM) having multiple virtual NUMA nodes (i.e., backed by multiple physical NUMA nodes), or another type of device that processes network data traffic and has multiple processing cores with multiple NUMA nodes.

[0022] As shown, process 100 begins by receiving (at 105) a data message at a particular interface of a data message processing device. In some embodiments, this interface is a physical or virtual network interface controller (NIC) associated with a particular one of the NUMA nodes of the data message processing device. Additionally, although not shown, in some embodiments, the device stores the data message in a memory associated with the same NUMA node as the particular interface. Some embodiments store the data message as a set of fields that include various data message headers (e.g., source and destination addresses, protocol, etc.) as well as additional metadata (e.g., logical networking context). In some such embodiments, the metadata also includes an indicator that specifies the NUMA node at which the data message was received.

[0023] The process 100 then processes (at 110) the data message to identify a set (one or more) of possible output options with equal forwarding costs. In some embodiments, the processing is performed by a thread executing on a specific core of the edge device. This core is associated with the same NUMA node as the memory in which the data message is stored (and the NIC at which the data message is received), and therefore the processing of the data message does not involve any cross-NUMA processing.

[0024] It should be noted that in some cases, the device will actually block or drop the data message (e.g., based on a firewall rule) rather than identifying an output option for the data message. In various different embodiments, the identified output options may be different types of outputs. For example, the output options may be equal cost multi-path (ECMP) routing that specifies (or maps to) different NICs of the device or tunnel endpoints (e.g., virtual tunnel endpoints (VTEPs)) corresponding to different NICs of the device. In other embodiments, the output option is a transport interface (e.g., a kernel network interface (KNI)) that enables the data message to be transported from a data path that performs data message processing on the device to another application executing on the device. Since the different identified options all have equal forwarding costs, in the absence of NUMA-aware data message processing, the edge device will select from all of these identified options (e.g., using a load balancing algorithm), regardless of whether the selected output option will result in cross-NUMA processing.

[0025] Having identified a set of potential output options, process 100 determines (at 115) whether any of the output options are associated with the same NUMA node as the particular interface at which the data message is received. It should be noted that process 100 is a conceptual process, and the data message processing device may not perform all of these operations in the particular manner shown in the figure. For example, while some embodiments first identify all potential output options, then narrow down these options in a NUMA-aware manner, and then select one of the remaining options, other embodiments combine all or some of these operations into a single operation that identifies and selects an output option in a NUMA-aware manner.

[0026] If none of the identified output options are associated with the same NUMA node as the interface at which the data message is received, then process 100 selects (at 120) one of the output options from all of the identified output options. In some embodiments, this selection involves the device performing a load balancing operation among the various output options. The load balancing operation may involve a deterministic hash function (e.g., based on a set of data message header fields), a round robin or similar load balancing operation, a load balancing operation that evaluates the current load on different interfaces, and the like.

[0027] In this case, because none of the identified output options are associated with the NUMA node on which the data message was received and stored in memory, the data message processing device needs to perform more resource (and time) intensive cross-NUMA processing. This may occur, for example, if the data message is routed to a particular next hop that needs to be output through a NIC associated with a different NUMA node, and if the different NICs of the device do not have equivalent connectivity.

[0028] On the other hand, if at least one of the output options is associated with the same NUMA node as the interface at which the data message is received, then process 100 determines (at 125) whether there is more than one such output option. If only one output option is associated with a NUMA node, then process 100 selects (at 130) that option in order to avoid cross-NUMA processing of the data message.

[0029] If more than one of the identified output options is associated with the same NUMA node as the interface at which the data message is received, then process 100 performs (at 135) a balancing operation to select one of the options associated with that NUMA node. The balancing operation may involve a deterministic hash function (e.g., based on a set of data message header fields), a round-robin or similar load balancing operation, a load balancing operation that evaluates current loads on different interfaces, etc. Some embodiments use deterministic operations based on connection parameters associated with the data message (e.g., source and / or destination network addresses, source and / or destination transport layer ports, transport layer protocol) so that all data messages for a given connection are output via the same interface (or other type of output option).

[0030] Finally, process 100 outputs (at 140) the data message via the selected option. In some embodiments, this entails sending the data message onto a network (e.g., a physical network) via the selected interface. If the output interface is to a logical overlay network, then in some embodiments, the data message is output with an encapsulation header. If the output interface is to an external network, then in some embodiments, the output data message is an internal data message, wherein the encapsulation header is removed during processing by the device. In other embodiments, the output interface is a software interface (e.g., a KNI or a VNIC).

[0031] As described above, in some embodiments, the data message processing device is an edge device that processes data messages between a logical network implemented in a data center and an external network (i.e., a network external to the logical network). In some embodiments, the edge device implements a logical router having a plurality of uplink interfaces connected to an external router and a set of logical switches connected to the logical router (or an intermediate logical router).

[0032] Figure 2 An example of such a logical network 200 of some embodiments is conceptually illustrated. The logical network 200 includes a logical router 205 and two logical switches 210 and 215 connected to the logical router 205. Various logical network endpoints (implemented as, for example, virtual machines, containers, etc.) that are sources and destinations of data messages processed by the logical network elements 205 to 215 are connected to these logical switches 210 and 215. It should be noted that in some embodiments, the logical network includes a first-level logical router that connects the logical network to an external network and a plurality of second-level logical routers interposed between the logical switch groups and the first-level logical routers. In the latter case, the second-level logical routers can isolate different groups of logical switches and provide different services for data traffic to and from these logical switches.

[0033] In some embodiments, a network administrator (or other user) defines a logical router with one or more uplinks as a gateway between a logical network and an external network through a network management system, and the network management system internally defines several routing components of the logical router. Figure 2 As shown in FIG. 1 , the logical router 205 is defined as including two service routers 220 and 225 (also referred to as centralized routing components) and a distributed router 230 (also referred to as a distributed routing component), wherein a transit logical switch 235 connects the internal interfaces of these routing components 220 to 230.

[0034] Service routers 220 and 225 are defined as having two uplinks each, with equivalent connectivity between the two service routers. In some embodiments, a network administrator defines uplink interface groups, with one member of each group assigned to each of service routers 220 and 225. In some embodiments, service routers 220 and 225 can be implemented in an active-standby mode (e.g., as a pair of service routers) or an active-active mode (e.g., as any number of service routers). As shown, the uplinks of service routers 220 and 225 are connected to two different external routers 240 and 245, which provide connectivity to external networks (e.g., to the public Internet, different data centers via virtual private networks, etc.).

[0035] Figure 3An edge device 300 is conceptually illustrated that implements some embodiments of one of the service routers 220 (or 225) of the logical router 205. The edge device 300 is a physical computing device (e.g., a bare metal computing device), but it should be understood that the discussion herein is equally applicable to virtual edge devices (e.g., implemented in a VM). In this case, the illustrated physical resources would correspond to virtual resources (e.g., physical processing cores corresponding to virtual processors, physical memory corresponding to virtual memory mapped to physical memory in a different NUMA node, NICs corresponding to VNICs mapped to physical NICs of a physical device implementing the virtual edge device, etc.).

[0036] In this figure, the hardware elements of the edge device are shown with solid lines, while the software executed on the edge device is shown with dashed lines. The edge device 300 includes two sets of processing cores 315 and 320 (each of the sets may include one or more cores) and associated memories 325 and 330 (each of the memories may include any number of memory cells), which operate as two independent NUMA nodes 305 and 310. In addition, the edge device 300 includes a cross-NUMA bus 365 for handling processing of cross-NUMA nodes (e.g., when one of the processing cores 315 needs to access data in the memory 330 or one of the processing cores 320 needs to access data in the memory 325).

[0037] The edge device also includes two NICs 335 and 345 connected to external routers 240 and 245, respectively, and two NICs 340 and 350 connected to a data center underlying network 355 (i.e., a physical network of a data center in which the logical network is implemented and to which the edge device 300 belongs). NICs 335 and 340 are each associated with a first NUMA node 305, while NICs 345 and 350 are each associated with a second NUMA node 310. It should be noted that in some embodiments, NICs 335 and 340 are actually the same NIC, while NICs 345 and 350 are also the same NIC. In some such embodiments, each of these NICs is connected to a top of rack (TOR) switch or other network element capable of acting as both (i) a router connecting the logical network to an external network and (ii) a connection to the data center underlying network 355. It should also be understood that in some embodiments, the edge device 300 may include multiple NICs associated with each of its (two or more) NUMA nodes.

[0038] The edge device 300 may execute the data path 360 among other software (e.g., KNI, network stack, other applications for performing layer 7 (L7) data message processing). In some embodiments, the data path 360 is a data plane development kit (DPDK) based data path that executes (i) a set of data message processing threads and (ii) a set of control processing threads (for handling control plane processing, such as per-logical router QoS tracking, routing protocol (e.g., BGP) processing, etc.). In some embodiments, each data message processing thread (which is a run-to-completion thread) is pinned to a respective processing core, with the control threads being scheduled among the other cores. Because each data message processing thread is pinned to a specific core, each of these threads is associated with a specific NUMA node (i.e., the NUMA node to which its core is associated).

[0039] When processing data messages, data path 360 implements at least a subset of logical network elements of logical network 200. Edge device 300 is assigned one of service routers 220 and 225, and thus data path 360 is configured to implement this service router as well as distributed router 230 (and transit logical switch 235) and logical switches 210 and 215. In some embodiments, data path 360 stores configuration data for each of these logical network elements. In some embodiments, northbound (outgoing) data messages typically apply logical switch and distributed router processing at the source of the data message (per the first hop processing principle), and thus data path 360 only applies service router processing (and transit logical switch to identify that service router processing is required) to these data messages. On the other hand, for southbound (incoming) data messages, data path 360 applies service router, distributed router, and logical switch processing.

[0040] Each of the uplinks of the service router 220 (or 225) is associated with one or more of the northbound NICs 335 and 345 of the edge device 300. Because these NICs 335 and 345 are associated with two different NUMA nodes 305 and 310, the data path 360 can choose to output some northbound data messages via the first NUMA node 305 and output other northbound data messages via the second NUMA node 310.

[0041] For southbound data messages, once the data path 360 applies the logical switch processing (for logical switch 210 or 215), this processing typically identifies the logical egress port (corresponding to the destination logical network endpoint MAC address in the data message) and specifies that the data message be output via a VTEP or a group of VTEPs (the data message will be encapsulated using the network address associated with the selected VTEP). In some embodiments, each of the VTEPs corresponds to one of the southbound NICs 340 and 350, and is therefore associated with corresponding NUMA nodes 305 and 310. The VTEPs typically have equivalent connectivity into the data center underlay 355, and therefore the data path 360 can select either of the NICs 340 and 350 for a given southbound data message (to avoid cross-NUMA traffic).

[0042] Figures 4A to 4B The diagram conceptually illustrates the processing of a northbound data message 400 by an edge device 300. Figure 4A , data message 400 is received at south-facing NIC 340. Because this NIC 340 is associated with NUMA node 305, the data message properties are stored in one of the memories 325. In some embodiments, data message 400 is stored in memory as a data message object 405 that can be manipulated by data path 360. As shown, data message 405 includes a set of fields, such as source and destination IP addresses, source and destination MAC addresses, and various other header fields (Ethertype, transport protocol, L7 payload, etc.). In addition, data message object 405 includes various metadata fields that are not part of the data message 400 on the wire, but provide additional data used by data path 360 to process the data message. These metadata fields include indicators about the NUMA node on which the data message was received (and therefore the NUMA node at which the data message object is stored) and other fields (e.g., logical ingress or egress port, logical forwarding element identifier, etc.).

[0043] Figure 4B The data path 360 is shown processing this data message 400 (e.g., once the data message is popped from the data message queue to be processed). In some embodiments, one of the data path threads pinned to the core 315 associated with the first NUMA node 305 is assigned to process the data message 400 and, therefore, access the data message object 405 in the memory 325.

[0044] As described above, for northbound data messages sent from a logical network endpoint and directed to an external destination, data path 360 performs service router processing on the data message and identifies an uplink via which to output the outgoing data message. In this case, based on the destination IP address of data message 400, data path 360 identifies a set of ECMP routes that specify that data message 400 be output to next-hop IP address IP1 (i.e., the IP address of external router 240) via a first uplink of service router 220, or to next-hop IP address IP2 (i.e., the IP address of external router 245) via a second uplink of service router 220. In the absence of NUMA-aware processing, data path 360 would select between these two uplinks, assuming equal forwarding costs (e.g., by using a load balancing mechanism). However, the data path logical router configuration also specifies the associated NUMA nodes for each of the possible routes. In this case, the first uplink is mapped to the first NIC 335 associated with the first NUMA node 305, and the second uplink is mapped to the third NIC 345 associated with the second NUMA node 310. Based on these mappings and metadata indicating that the data message is associated with the first NUMA node 305, the data path 360 selects the first route (via the first uplink mapping to the NIC 335) and outputs the data message via the first NIC 335 (rather than performing a load balancing operation that may result in the data message having to be output through the NIC 345 via the cross-NUMA bus 365).

[0045] Similar principles apply where data messages are received from an external network at an edge device and directed to a logical network endpoint. Figures 5A to 5B The diagram conceptually illustrates the processing of southbound data message 500 by edge device 300. Figure 5A, a data message 500 is received at a north-facing NIC 335. Because this NIC 335 is associated with a NUMA node 305, the data message properties are stored in one of the memories 325. In some embodiments, the data message 500 is stored in the memory as a data message object 505 that can be manipulated by the data path 360. As shown, the data message 505 includes a set of fields, such as source and destination IP addresses, source and destination MAC addresses, and various other header fields (Ethertype, transport protocol, L7 payload, etc.). In addition, the data message object 505 includes various metadata fields that are not part of the data message 500 on the wire, but provide additional data used by the data path 360 to process the data message. These metadata fields include indicators about the NUMA node at which the data message is received (and therefore the NUMA node at which the data message object is stored) as well as other fields (e.g., logical ingress or egress port, logical forwarding element identifier, etc.).

[0046] Figure 5B The data path 360 is shown processing this data message 500 (e.g., once the data message is popped from the data message queue to be processed). In some embodiments, one of the data path threads pinned to the core 315 associated with the first NUMA node 305 is assigned to process the data message 500 and, therefore, access the data message object 505 in the memory 325.

[0047] As described above, for southbound data messages sent from an external destination to a logical network endpoint, the data path 360 initially applies a service router process that identifies the distributed router 230 as the next hop. The distributed router process applied by the data path 360 identifies one of the logical switches (in this case, the second logical switch 215) based on the destination IP address of the data message 500 (it should be noted that in some cases, the service router and distributed router processes are combined into a single routing table for southbound data messages). The data path 360 then applies the process for this logical switch 215 and, based on the destination MAC address, determines the destination host computer (corresponding to the destination tunnel endpoint) on which the logical network endpoint with the MAC address operates.

[0048] In some cases, the edge device has the option of using multiple different source tunnel endpoints (e.g., VTEPs) via which to send data messages to a destination host computer through the physical underlying network 355. In this example, the data path 360 identifies the logical egress port of the logical switch 215 based on the destination MAC address (after the logical router processing performs any required ARP and replaces the destination MAC address of the data message object 505). This logical egress port maps to a pair of possible source VTEPs that can be used to reach the destination with equal forwarding cost. In the absence of NUMA-aware processing, the data path 360 would choose between these two source VTEPs, assuming that this forwarding cost is equal (e.g., by using a load balancing mechanism). However, the data path logical switch configuration also specifies an associated NUMA node for each of these VTEPs. In this case, the first VTEP is mapped to the second NIC 340 associated with the first NUMA node 305, and the second VTEP is mapped to the fourth NIC 350 associated with the second NUMA node 310. Based on these mappings and metadata indicating that the data message is associated with the first NUMA node 305, the data path 360 selects the first VTEP and thus encapsulates the data message using the first VTEP as the source IP address of the encapsulation header. The data path 360 outputs the data message via the second NIC 340, rather than performing a load balancing operation that may cause the data message to have to be output through the NIC 350 via the cross-NUMA bus 365.

[0049] Additionally, for incoming or outgoing data messages, in some cases, the edge device data path (which performs logical network processing) will need to send a data message to another application executing on the edge device (and then, possibly receive a data message back from this application). Specifically, in some embodiments, the data path performs only Layer 2 to Layer 4 (L2-L4) operations (e.g., switching, routing, network address and port translation, L4 load balancing), while other applications executing on the edge device perform Layer 5 to Layer 7 (L5-L7) operations (e.g., L7 load balancing, TLS proxy, URL filtering, etc.).

[0050] Figure 6The path through which a data message is sent from a data path to an L7 application on an edge device 600 is conceptually illustrated in accordance with some embodiments. As shown, a data path 605 and a set of L7 applications 610 execute within a user space 615 of an edge device, while a kernel network stack (e.g., a TCP / IP stack) 620 executes in a device kernel 625. To send a data message from the data path 605 to one of the L7 applications 610, in some embodiments, the data path 605 outputs the data message to a kernel network interface (KNI) 630, which communicates the data message from the user space 615 to the network stack 620 in the kernel space 625 of the edge device 600. In some embodiments, the L7 applications 610 are designed so that data messages are received from the kernel network stack 620, so the kernel network stack 620 provides the data message to the appropriate L7 application 610. In some embodiments, when one of the L7 applications 610 transmits a data message back to the data path 605 (i.e., after performing its processing on the data message), the data message uses the return path (from the L7 application 610 to the kernel network stack 620 to the data path 605 (via the KNI 630)).

[0051] In some embodiments, the edge device includes multiple KNIs, each of which can be used to reach the network stack in the kernel, each KNI (or other transport interface) is associated with a respective NUMA node like a NIC. In other embodiments, the device includes a single KNI with multiple queues, each of which is associated with a different NUMA node. In either case, when selecting between KNIs (or between KNI queues), the data path is configured to avoid cross-NUMA traffic.

[0052] Figures 7A to 7B The diagram conceptually illustrates the processing of a northbound data message 700 sent by an edge device 300 to an L7 application 710. Fig. 7A , a data message 700 is received at a south-facing NIC 340. Because this NIC 340 is associated with a NUMA node 305, the data message properties are stored in one of the memories 325. In some embodiments, the data message 700 is stored in the memory as a data message object 705 that can be manipulated by the data path 360. This figure also illustrates that the device executes a first KNI 715 on the first NUMA node 305 and executes a second KNI 720 on the second NUMA node 310. In some embodiments, each of these KNIs can be used to send data traffic to a set of L7 applications.

[0053] As shown, the data message 705 includes a set of fields such as source and destination IP addresses, source and destination MAC addresses, and various other header fields (Ethertype, transport protocol, L7 payload, etc.). In addition, the data message object 705 includes various metadata fields that are not part of the data message 700 on the wire, but provide additional data used by the data path 360 to process the data message. These metadata fields include indicators about the NUMA node on which the data message was received (and therefore the NUMA node at which the data message object is stored), as well as other fields (e.g., logical ingress or egress port, logical forwarding element identifier, etc.).

[0054] Figure 7B The data path 360 is shown processing this data message 700 (e.g., once the data message is popped from the data message queue to be processed). In some embodiments, one of the data path threads pinned to the core 315 associated with the first NUMA node 305 is assigned to process the data message 700 and, therefore, access the data message object 705 in the memory 325.

[0055] As discussed (and as Figure 4B ), for northbound data messages sent from a logical network endpoint and directed to an external destination, the data path 360 performs service router processing on the data message. In this case, the data path 360 identifies a set of KNIs to which the data path can output data messages, rather than initially identifying an uplink via which the data message is output. Here, the policy-based routing rules specify that the data message 700 is transmitted to the TLS proxy application 710 via either of the two KNIs 715 and 720 based on the source IP and destination port of the data message 700. In the absence of NUMA-aware processing, the data path 360 would select between the two KNIs 715 and 720, assuming equal forwarding costs (e.g., by using a load balancing mechanism). However, the data path logical router configuration also specifies the associated NUMA nodes for each of the KNIs 715 and 720. In this case, the first KNI 715 is associated with the first NUMA node 305, while the second KNI 720 is associated with the second NUMA node 310. Based on these mappings and metadata indicating that the data message is associated with the first NUMA node 305, the data path 360 selects a first KNI 715 to which to output the data message 700. The data path 360 sends the data message 700 to the TLS proxy application 710 via the selected KNI 715.

[0056] As mentioned above, in some cases, multiple output options may be associated with the same NUMA node, in which case the data message processing apparatus of some embodiments load balances between the options associated with the NUMA node at which the data message is received. Figures 8A to 8B The processing of a northbound data message 875 by edge device 800 is conceptually illustrated. Edge device 800 is similar to edge device 300, except that device 800 includes three north-facing NICs 835, 837, and 845, two of which are associated with first NUMA node 805.

[0057] like Fig. 8A , data message 875 is received at the south-facing NIC 840. Because this NIC 840 is associated with a NUMA node 805, the data message properties are stored as a data message object 880 in one of the memories 825 associated with that NUMA node 805.

[0058] Figure 8B The data path 860 is shown processing this data message 875 (e.g., once the data message is popped from the data message queue to be processed). In some embodiments, one of the data path threads pinned to the core 815 associated with the first NUMA node 805 is assigned to process the data message 875 and, therefore, access the data message object 880 in the memory 825.

[0059] As in the example described above for northbound data messages, data path 860 performs service router processing on data message 875 and identifies the uplink through which the outgoing data message is output. In this case, based on the destination IP address of data message 875, data path 860 identifies a set of ECMP routes, thereby specifying that data message 875 is output to next-hop IP address IP1 via the first uplink of the service router, output to next-hop IP address IP2 via the second uplink of the service router, or output to next-hop IP address IP3 via the third uplink of the service router. In the absence of NUMA-aware processing, data path 860 will select between these three uplinks, assuming that the forwarding cost is equal (e.g., by using a load balancing mechanism). However, the data path logical router configuration also specifies the associated NUMA nodes for each of the possible routes. In this case, the first uplink is mapped to the first NIC 835, the second uplink is mapped to the third NIC 837, and the third uplink is mapped to the fifth NIC 845. Thus, the first uplink and the second uplink are mapped to the NIC associated with the first NUMA node 805, while the third uplink is mapped to the NIC associated with the second NUMA node 810. Based on these mappings and the metadata indicating that the data message is associated with the first NUMA node 805, the data path 860 performs a load balancing operation to select between the first two next-hop IP addresses (e.g., using a hashing algorithm). In this example, the load balancing operation selects the second uplink, so the data path 860 outputs the data message 875 via the third NIC 837.

[0060] Fig. 9 An electronic system 900 is conceptually illustrated to implement some embodiments of the present invention. The electronic system 900 may be a computer (e.g., a desktop computer, a personal computer, a tablet computer, a server computer, a mainframe, a blade computer, etc.), a phone, a PDA, or any other kind of electronic device. Such an electronic system includes various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 900 includes a bus 905, a processing unit 910, a system memory 925, a read-only memory 930, a permanent storage device 935, an input device 940, and an output device 945.

[0061] Bus 905 collectively represents all system, peripheral, and chipset buses that communicatively connect the various internal devices of electronic system 900. For example, bus 905 communicatively connects processing unit 910 with read-only memory 930, system memory 925, and permanent storage 935.

[0062] Processing unit 910 retrieves instructions to execute and data to process from these various memory units in order to perform the processes of the present invention.In various embodiments, the processing unit may be a single processor or a multi-core processor.

[0063] Read-only memory (ROM) 930 stores static data and instructions required by processing unit 910 and other modules of the electronic system. On the other hand, permanent storage device 935 is a read-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the electronic system 900 is disconnected. Some embodiments of the present invention use a mass storage device (such as a magnetic disk or optical disk and its corresponding magnetic disk drive) as permanent storage device 935.

[0064] Other embodiments use removable storage devices (e.g., floppy disks, flash drives, etc.) as permanent storage devices. Like permanent storage device 935, system memory 925 is a read-write memory device. However, unlike storage device 935, system memory is a volatile read-write memory, such as random access memory. System memory stores some instructions and data that the processor needs at run time. In some embodiments, the processes of the present invention are stored in system memory 925, permanent storage device 935, and / or read-only memory 930. Processing unit 910 retrieves instructions to be executed and data to be processed from these various memory units in order to perform the processes of some embodiments.

[0065] The bus 905 is also connected to an input device 940 and an output device 945. The input device enables a user to communicate information and select commands to the electronic system. The input device 940 includes an alphanumeric keyboard and a pointing device (also called a "cursor control device"). The output device 945 displays images generated by the electronic system. The output device includes a printer and a display device, such as a cathode ray tube (CRT) or a liquid crystal display (LCD). Some embodiments include devices such as a touch screen that function as both an input device and an output device.

[0066] Finally, if Fig. 9 As shown in FIG. 9 , bus 905 also couples electronic system 900 to a network 965 via a network adapter (not shown). In this manner, the computer may be part of a computer network (e.g., a local area network ("LAN"), a wide area network ("WAN"), or an intranet, or a network of networks such as the Internet). Any or all components of electronic system 900 may be used in conjunction with the present invention.

[0067] Some embodiments include electronic components such as microprocessors, storage devices, and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as a computer-readable storage medium, a machine-readable medium, or a machine-readable storage medium). Some examples of such computer-readable media include RAM, ROM, compact disk-read only (CD-ROM), compact disk-recordable (CD-R), compact disk-rewritable (CD-RW), digital versatile disk-read only (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable Blu-ray discs. A computer readable medium may store a computer program executable by at least one processing unit and contain multiple sets of instructions for performing various operations. Examples of computer programs or computer code include machine code (such as that produced by a compiler) and files containing high-level code that are executed by a computer, electronic component, or microprocessor using an interpreter.

[0068] Although the above discussion mainly refers to microprocessors or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuit itself.

[0069] As used in this specification, the terms "computer," "server," "processor," and "memory" all refer to electronic or other technical devices. These terms do not include people or groups of people. For the purposes of this specification, the terms display or displaying mean displaying on an electronic device. As used in this specification, the terms "computer-readable medium," "computer-readable media," and "machine-readable medium" are entirely limited to tangible physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals, and any other transient signals.

[0070] This specification refers throughout to computing and networking environments that include virtual machines (VMs). However, a virtual machine is only one example of a data computing node (DCN) or data computing end node (also called an addressable node). A DCN can include non-virtualized physical hosts, virtual machines, containers that run on a host operating system without a hypervisor or separate operating system, and hypervisor kernel network interface modules.

[0071] In some embodiments, the VM uses the resources of the host virtualized by virtualization software (e.g., a hypervisor, a virtual machine monitor, etc.) to operate on the host with its own client operating system. The tenant (i.e., the owner of the VM) can choose which applications to operate on the client operating system. On the other hand, some containers are structures that run on top of the host operating system without the need for a hypervisor or a separate client operating system. In some embodiments, the host operating system uses namespaces to isolate containers from each other, and thus provides operating system-level isolation of different application groups running in different containers. This isolation is similar to the VM isolation provided in a hypervisor virtualization environment that virtualizes the system hardware, and can therefore be viewed as a form of virtualization that isolates different application groups operating in different containers. Such containers are more lightweight than VMs.

[0072] In some embodiments, the hypervisor kernel network interface module is a non-VM DCN that includes a network stack with a hypervisor kernel network interface and receive / transmit threads. An example of a hypervisor kernel network interface module is the vmknic module, which is a component of VMware's ESXi TM part of the management program.

[0073] It should be understood that although the specification mentions VMs, the examples given may be any type of DCN, including physical hosts, VMs, non-VM containers, and hypervisor kernel network interface modules. In fact, in some embodiments, the exemplary network may include a combination of different types of DCNs.

[0074] Although the present invention has been described with reference to many specific details, those skilled in the art will recognize that the present invention may be embodied in other specific forms without departing from the spirit of the present invention. Figure 1 ) conceptually illustrates the processes. The specific operations of these processes may not be performed in the exact order shown and described. Specific operations may not be performed in a continuous series of operations, and different specific operations may be performed in different embodiments. In addition, the processes may be implemented using several sub-processes or as part of a larger macro process. Therefore, it will be understood by those skilled in the art that the present invention is not limited by the foregoing illustrative details, but is defined by the appended claims.

Claims

1. A method comprising: At a data message processing apparatus including a plurality of network interfaces associated with at least two different non-uniform memory access NUMA nodes: receiving a data message at a first network interface associated with a particular one of the NUMA nodes; Based on processing the data message, identifying a plurality of output options for the data message, wherein (i) each of the output options has an equal forwarding cost, and (ii) each output option is associated with a respective one of the NUMA nodes; and An output option associated with the particular NUMA node for the data message is selected to avoid cross-NUMA node processing of the data message. 2 . The method according to claim 1 , wherein the data message processing device is an edge device that processes data messages between a logical first network implemented in a data center and an external second network.

3. The method of claim 2, wherein the plurality of output options are equal cost multi-path ECMP routes for sending the data message to a router in the external second network, each respective ECMP route specifying (i) a respective external router next-hop network address and (ii) a respective output network interface. 4 . The method of claim 3 , wherein the ECMP rule used by the edge device to identify the plurality of ECMP routes indicates, for each respective ECMP route, the NUMA node to which the respective output network interface is associated.

5. The method of claim 2, wherein the plurality of output options are virtual tunnel endpoints VTEPs for tunneling the data message to a specific host computer in the data center based on processing of a specific logical switch to which the data message is routed, each respective VTEP being associated with a respective network interface.

6. The method of claim 5, wherein the VTEP selection rule associated with the particular logical switch indicates, for each respective VTEP, the NUMA node to which the respective network interface is associated.

7. The method of claim 2, wherein the edge device executes: (i) a data path that performs the processing on the data message; (ii) a set of applications that perform layer 7 processing on the data message and return the data message to the data path; and (iii) a plurality of transport interfaces that enable data messages to be transported between the data path and the applications, each transport interface being associated with a respective one of the NUMA nodes.

8. The method of claim 7, wherein the plurality of output options comprises a set of the delivery interfaces for delivering the data message to a specific application based on a logical router process, the logical router process indicating that the data message requires a specific type of layer 7 processing to be performed by the specific application.

9. The method of claim 1, wherein the data message processing device stores metadata for use in processing the data message, the metadata comprising an indicator of the particular NUMA node associated with the first network interface at which the data message was received.

10. The method of claim 1, wherein the selected output option is associated with the first network interface.

11. The method of claim 1, wherein the plurality of output options comprises at least two different output options associated with the particular NUMA node having equal forwarding costs.

12. The method of claim 11, wherein selecting the output option comprises using a load balancing algorithm that selects between the at least two different output options associated with the particular NUMA node.

13. A non-transitory machine-readable medium storing a program for execution by at least one processing unit of a message processing apparatus, the message processing apparatus comprising a plurality of network interfaces associated with at least two different non-uniform memory access (NUMA) nodes, the program comprising a plurality of sets of instructions for: receiving a data message at a first network interface associated with a particular one of the NUMA nodes; Based on processing the data message, identifying a plurality of output options for the data message, wherein (i) each of the output options has an equal forwarding cost, and (ii) each output option is associated with a respective one of the NUMA nodes; and An output option associated with the particular NUMA node for the data message is selected to avoid cross-NUMA node processing of the data message. 14 . The non-transitory machine-readable medium of claim 13 , wherein the data message processing device is an edge device that processes data messages between a logical first network implemented in a data center and an external second network.

15. The non-transitory machine-readable medium of claim 14, wherein: The plurality of output options are equal cost multi-path ECMP routes for sending the data message to a router in the external second network, each respective ECMP route specifying (i) a respective external router next hop network address and (ii) a respective output network interface; and The ECMP rules used by the edge device to identify the plurality of ECMP routes indicate, for each respective ECMP route, the NUMA node to which the respective output network interface is associated.

16. The non-transitory machine-readable medium of claim 14, wherein: The plurality of output options are virtual tunnel endpoints VTEPs for tunneling the data message to a particular host computer in the data center based on processing of a particular logical switch to which the data message is routed, each respective VTEP being associated with a respective network interface; and The VTEP selection rule associated with the particular logical switch indicates, for each respective VTEP, the NUMA node to which the respective network interface is associated.

17. The non-transitory machine-readable medium of claim 14, wherein: said program being a data path program for performing said processing of said data message; The edge device also executes: (i) a set of applications that perform layer 7 processing on data messages and return the data messages to the data path; and (ii) a plurality of transport interfaces that enable data messages to be transported between the data path and the applications, each transport interface being associated with a respective one of the NUMA nodes; and The plurality of output options includes a set of the delivery interfaces for delivering the data message to a specific application based on a logical router process indicating that the data message requires a specific type of layer 7 processing to be performed by the specific application.

18. The non-transitory machine-readable medium of claim 13, wherein the data message processing device stores metadata for use in processing the data message, the metadata comprising an indicator of the particular NUMA node associated with the first network interface at which the data message was received.

19. The non-transitory machine-readable medium of claim 13, wherein the selected output option is associated with the first network interface.

20. The non-transitory machine-readable medium of claim 13, wherein: The plurality of output options include at least two different output options associated with the particular NUMA node having equal forwarding costs; and The set of instructions for selecting the output option includes a set of instructions for using a load balancing algorithm that selects between the at least two different output options associated with the particular NUMA node.

Images