Multi-tenancy of cloud scale for roce (RDMA over converged ethernet)

Abstract

To provide a method and a device for data networking.SOLUTION: A method includes: a step of receiving a first layer 2 remote direct memory access (RDMA) packet containing a virtual local area network (VLAN) tag and a quality of service (QoS) data field; a step of converting the first layer 2RDMA packet into a first layer 3 capsulated packet; and a step of transferring the first layer 3 capsulated packet to a switch fabric. The step for the conversion includes a step of adding at least one header to the first layer 2RDMA packet, and at least the one header contains a virtual network identifier based on information from a VLAN tag and a QoS value based on the information from the QoS data field.SELECTED DRAWING: Figure 7A

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is filed under the title "CLOUD SCALE MULTI-TENANCY FOR RDMA OVER CONVERGED ETHERNET (RoCE) U.S. Provisional Application No. 63 / 132,417, filed December 30, 2020, entitled "CLOUD SCALE MULTI-TENANCY FOR RDMA OVER CONVERGED ETHERNET (RoCE)" No. 17 / 165,877, filed on February 2, 2021, entitled “CLASS-BASED QUEUEING FOR SCALABLE MULTI-TENANT RDMA TRAFFIC” No. 17 / 166,922, filed on February 3, 2021, and entitled “CLASS-BASED QUEUEING FOR SCALABLE MULTI-TENANT RDMA TRAFFIC,” which is hereby incorporated by reference in its entirety. This application claims priority to PCT Application No. PCT / US2021 / 025459, filed April 1, 2021, which is incorporated by reference in its entirety for all purposes. [Background technology]

[0002] background RoCE (RDMA over Converged Ethernet) is a network protocol that enables remote direct memory access (RDMA) over lossless Ethernet networks. RoCE makes this possible by encapsulating InfiniBand (IB) transport packets over RDMA. Generally speaking, RoCE involves a layer 2 network with dedicated RDMA queues and dedicated VLANs. However, layer 2 networks are not scalable and do not offer very high performance because they lack important properties and characteristics present in the more scalable and high performance layer 3 networks. Therefore, existing public cloud implementations are not able to provide data transfer using the RoCE protocol. Summary of the Invention [Means for solving the problem]

[0003] Quick Overview The present disclosure relates generally to data networking. More specifically, techniques are described that enable communication of layer 2 traffic over a layer 3 network using a layer 3 protocol. In a particular embodiment, the techniques described in the present disclosure enable communication of remote direct memory access (RDMA) traffic (e.g., RDMA over Converged Ethernet (RoCE) traffic) from a compute instance on a multitenant host machine (i.e., a host machine that hosts compute instances belonging to various tenants or customers) to a compute instance on another multitenant host machine over a shared layer 3 physical network or switch fabric using a layer 3 routing protocol. Such communication may optionally include other traffic (e.g., TCP and / or UDP traffic). The customer or tenant experiences the communication as occurring over a dedicated layer 2 network, but in reality the communication occurs over a shared (i.e., shared among multiple customers or tenants) layer 3 network using a layer 3 routing protocol. Methods, systems, programs, codes or instructions executable by one or more processors. Various embodiments are described in this disclosure, including a non-transitory computer-readable storage medium for storing instructions.

[0004] In a particular embodiment, a method of data networking includes transmitting a first Layer 2 RDMA packet for a first tenant among a plurality of tenants from a host machine executing a plurality of compute instances of the plurality of tenants to an ingress switch. receiving a first Layer 2 RDMA packet from an intermediate switch of the switch fabric, translating the first Layer 2 RDMA packet into a first Layer 3 encapsulated packet having at least one header, and forwarding the first Layer 3 encapsulated packet to a switch fabric, the first Layer 2 RDMA packet including a Virtual Local Area Network (VLAN) tag and a Quality-of-Service (QoS) data field, the translating including adding the at least one header to the first Layer 2 RDMA packet, the at least one header including a virtual network identifier based on information from the VLAN tag and a QoS value based on information from the QoS data field. The method may further include modifying a congestion notification data field of the at least one header of the first Layer 3 encapsulated packet at an intermediate switch of the switch fabric in response to an indication of congestion. Alternatively or additionally, the method may further include receiving a second Layer 2 RDMA packet including a VLAN tag and a QoS data field, converting the second Layer 2 RDMA packet into a second Layer 3 encapsulated packet having at least one header, and forwarding the second Layer 3 encapsulated packet to the switch fabric, where the VLAN tag of the second Layer 2 RDMA packet indicates a different VLAN than the VLAN tag of the first Layer 2 RDMA packet. Such a method may further include, at an intermediate switch of the switch fabric, queuing the first Layer 3 encapsulated packet in a first queue of the intermediate switch based on the QoS value of the at least one header of the first Layer 3 encapsulated packet, and queuing the second Layer 3 encapsulated packet in a second queue of the intermediate switch different from the first queue based on the QoS value of the at least one header of the second Layer 3 encapsulated packet.

[0005] In yet another embodiment, a method of data networking includes receiving, at an egress switch, a first Layer 3 encapsulated packet, decapsulating the first Layer 3 encapsulated packet to obtain a first Layer 2 RDMA packet, setting a value of a congestion notification data field of the first Layer 2 RDMA packet based on information in a congestion notification data field of the at least one header of the first Layer 3 encapsulated packet, and after the setting, forwarding the first Layer 2 RDMA packet to a first compute instance based on a VLAN tag of the first Layer 2 RDMA packet. The method may further include receiving, at the egress switch, a second Layer 3 encapsulated packet, decapsulating the second Layer 3 encapsulated packet to obtain a second Layer 2 RDMA packet, and forwarding the second Layer 2 RDMA packet to a second compute instance different from the first compute instance based on a VLAN tag of the second Layer 2 RDMA packet. Such a method may further include, at the egress switch, queuing the first Layer 3 encapsulated packet in a first queue of the egress switch based on a quality of service (QoS) value of an outer header of the first Layer 3 encapsulated packet, and queuing the second Layer 3 encapsulated packet in a second queue of the egress switch, different from the first queue, based on a QoS value of an outer header of the second Layer 3 encapsulated packet.

[0006] In yet other embodiments, techniques are described for class-based queuing of RDMA traffic (e.g., in Layer 3 networks) that can be used to maintain class-based isolation across a network fabric at cloud scale such that RDMA traffic in a particular queue does not affect RDMA traffic in other queues. According to certain embodiments, a system can be implemented to include a shared fabric for transport of different classes of RDMA traffic and for transport of RDMA traffic from different tenants, where each device in a path across the shared fabric from one RDMA Network Interface Controller (NIC) to another NIC includes multiple queues dedicated to different classes of RDMA traffic.

[0007] According to a particular embodiment, a method of queuing RDMA packets includes a networking device receiving a plurality of RDMA packets, each RDMA packet in the plurality of RDMA packets including a quality of service (QoS) data field, and for each RDMA packet in the plurality of RDMA packets, the QoS data field indicates a class of service for the RDMA packet and has a QoS value among a plurality of QoS values. The method also includes the networking device distributing the plurality of RDMA packets among a plurality of RDMA queues, the distributing being performed according to a first mapping of the plurality of QoS values ​​to the plurality of RDMA queues. The method further includes the networking device removing the plurality of RDMA packets from the plurality of RDMA queues according to a first weighting among the plurality of RDMA queues. The removed plurality of RDMA packets may include a plurality of packet flows, in which case an example may further include routing the plurality of packet flows of the removed plurality of RDMA packets according to a per-flow equal-cost multipath scheme. Each RDMA packet in the plurality of RDMA packets may be a RoCEv2 packet, or each RDMA packet in the plurality of RDMA packets may be a Layer 3 encapsulated packet formatted according to an overlay encapsulation protocol (e.g., VxLAN, NVGRE, GENEVE, STT or MPLS).

[0008] In a further example, the distributing step includes, in response to determining that the QoS data field of a first RDMA packet in the plurality of RDMA packets has a first QoS value, storing the first RDMA packet in a first RDMA queue in the plurality of RDMA queues, and, in response to determining that the QoS data field of a second RDMA packet in the plurality of RDMA packets has a second QoS value, storing the second RDMA packet in a second RDMA queue in the plurality of RDMA queues, wherein the second QoS value is different from the first QoS value.

[0009] According to a particular embodiment, a further method of queuing RDMA packets also includes the networking device removing a plurality of control packets from a control queue, the removing the plurality of control packets having stricter priority than the removing the plurality of RDMA packets. In this case, the control queue may be configured to have a lower bandwidth than any of the plurality of RDMA queues. Alternatively or additionally, the plurality of control packets may include at least one network control protocol packet (e.g., a BGP packet) and / or at least one congestion notification packet (CNP packet).

[0010] According to certain embodiments, a networking device (e.g., a leaf switch or a spa a networking device (in-switch) may be configured to include a plurality of RDMA queues and a processing circuit coupled to the plurality of RDMA queues and configured to receive a plurality of RDMA packets, each RDMA packet in the plurality of RDMA packets including a quality of service (QoS) data field, and the networking device may be further configured to distribute the plurality of RDMA packets among the plurality of RDMA queues according to a first mapping of a plurality of QoS values ​​to the plurality of RDMA queues and to retrieve the plurality of RDMA packets from the plurality of RDMA queues according to a first weighting among the plurality of RDMA queues. For each RDMA packet in the plurality of RDMA packets, the QoS data field indicates a class of service for the RDMA packet and has a value among the plurality of QoS values.

[0011] In yet another embodiment, a technique is described for class-based marking of encapsulated remote direct memory access (RDMA) traffic that can be used to maintain consistent class-based isolation across a network fabric at cloud scale (e.g., during Layer 3 transport) such that RDMA traffic in a particular queue does not affect RDMA traffic in other queues. In accordance with a particular embodiment, a system may be implemented to include a shared fabric for transport of various classes of RDMA traffic and RDMA traffic from various tenants, with each device in a path across the shared fabric from one RDMA network interface controller (NIC) to another NIC including multiple queues dedicated to various classes of RDMA traffic. Various embodiments of the invention are described in this disclosure, including methods, systems, non-transitory computer-readable storage media storing programs, codes, instructions executable by one or more processors, and the like.

[0012] In accordance with a particular embodiment, a method of data networking includes a networking device receiving a plurality of RDMA packets, each RDMA packet in the plurality of RDMA packets including a quality of service (QoS) data field having a QoS value indicating a class of service for the RDMA packet. The plurality of RDMA packets includes an RDMA packet in which the QoS data field has a first QoS value and an RDMA packet in which the QoS data field has a second QoS value different from the first QoS value. The method also includes, for each of the plurality of RDMA packets, encapsulating the RDMA packet to generate a corresponding one of a plurality of Layer 3 encapsulation packets, the corresponding Layer 3 encapsulation packet having at least one outer header. For each of the plurality of RDMA packets, encapsulating the RDMA packet includes adding at least one outer header of the corresponding Layer 3 encapsulation packet to the RDMA packet. For each of the plurality of Layer 3 encapsulation packets, a QoS data field of the at least one outer header of the Layer 3 encapsulation packet takes a QoS value based on the QoS value of the QoS data field of the corresponding RDMA packet. For each Layer 3 encapsulation packet in the plurality of Layer 3 encapsulation packets, the at least one outer header may include a virtual network identification field based on a VLAN ID of the corresponding RDMA packet. In such a case, the plurality of RDMA packets may include RDMA packets each having a first VLAN ID (some packets possibly having different QoS values ​​than other packets) and RDMA packets each having a second VLAN ID different from the first VLAN ID. Alternatively or additionally, at least one Layer 3 encapsulation packet in the plurality of Layer 3 encapsulation packets may include a first VLAN tag and a second VLAN tag different from the first VLAN tag.

[0013] For each of the plurality of Layer 3 encapsulation packets, the at least one outer header of the encapsulation packet may include a User Datagram Protocol (UDP) header having a destination port number of 4791 (e.g., a RoCEv2 reserved UDP port). Alternatively or additionally, the at least one outer header of the Layer 3 encapsulation packet may include an Internet Protocol (IP) header having a destination IP address associated with a destination Media Access Control (MAC) address of the corresponding RDMA packet. It's fine.

[0014] For each RDMA packet in the plurality of RDMA packets, the QoS data field of the RDMA packet may be a DSCP data field of an IP header of the RDMA packet. In this case, for each of the plurality of Layer 3 encapsulation packets, the QoS value in the QoS data field of the at least one outer header of the Layer 3 encapsulation packet may be equal to the QoS value in the QoS data field of the corresponding RDMA packet. Alternatively, for each RDMA packet in the plurality of RDMA packets, the QoS data field of the RDMA packet may be an IEEE 802.1p data field of a VLAN tag. In this case, encapsulating the RDMA packet may include obtaining a QoS value for the QoS data field of the at least one outer header of the corresponding Layer 3 encapsulation packet from the QoS value of the QoS data field of the RDMA packet and a QoS value mapping, and storing the obtained QoS value in the QoS data field of the at least one outer header of the Layer 3 encapsulation packet.

[0015] According to a particular embodiment, the further method of data networking also includes, for each of at least one Layer 3 encapsulation packet in the plurality of Layer 3 encapsulation packets, copying congestion indication information from the corresponding RDMA packet to the at least one outer header of the Layer 3 encapsulation packet. Alternatively or additionally, the method of data networking may further include decapsulating each of a second plurality of Layer 3 encapsulation packets to obtain a corresponding one of a plurality of decapsulated RDMA packets. For at least one of the plurality of decapsulated RDMA packets, the decapsulating may include copying congestion indication information from the at least one outer header of the corresponding Layer 3 encapsulation packet to the decapsulated RDMA packet.

[0016] According to certain embodiments, the non-transitory computer-readable memory may store a plurality of instructions executable by one or more processors, the plurality of instructions including instructions that, when executed by the one or more processors, cause the one or more processors to perform any one of the methods.

[0017] According to certain embodiments, a system may include one or more processors and a memory coupled to the one or more processors, the memory may store a plurality of instructions executable by the one or more processors, the plurality of instructions including instructions that, when executed by the one or more processors, cause the one or more processors to perform any one of the methods.

[0018] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims and accompanying drawings. [Brief description of the drawings]

[0019] [Figure 1]FIG. 1 illustrates a high-level diagram of a distributed environment showing a virtual or overlay cloud network hosted by a cloud service provider infrastructure according to certain embodiments. [Diagram 2] 1 is an architectural schematic diagram illustrating physical elements of a physical network in a CSPI according to a particular embodiment. [Diagram 3] FIG. 2 illustrates an exemplary arrangement within CSPI in which a host machine is connected to multiple Network Virtualization Devices (NVDs), in accordance with certain embodiments. [Figure 4] A diagram illustrating connections between a host machine and a NVD that provides I / O virtualization to support multi-tenancy, according to certain embodiments. [Diagram 5] 1 is a schematic block diagram illustrating a physical network provided by CSPI in accordance with a particular embodiment. [Figure 6] FIG. 1 illustrates an example of a distributed cloud environment for data networking, in accordance with certain embodiments. [Figure 7A] 1 is a schematic flowchart illustrating a process for performing an RDMA data transfer from a source compute instance on a multi-tenant source host machine to a destination compute instance on a multi-tenant destination host machine via a shared Layer 3 switch fabric using a Layer 3 routing protocol, according to a particular embodiment. [Figure 7B] 1 is a schematic flowchart illustrating a process for performing an RDMA data transfer from a source compute instance on a multi-tenant source host machine to a destination compute instance on a multi-tenant destination host machine via a shared Layer 3 switch fabric using a Layer 3 routing protocol, according to a particular embodiment. [Figure 7C]1 is a schematic flowchart illustrating a process for performing an RDMA data transfer from a source compute instance on a multi-tenant source host machine to a destination compute instance on a multi-tenant destination host machine via a shared Layer 3 switch fabric using a Layer 3 routing protocol, according to a particular embodiment. [Figure 8A] 1 is a diagram showing an RDMA packet format according to the RoCEv2 (version 2 of the RDMA over Converged Ethernet) protocol. [Figure 8B] FIG. 2 illustrates a format of a VLAN-tagged RoCEv2 packet, according to a particular embodiment. [Figure 8C] FIG. 2 illustrates a format of a Q-in-Q tagged RoCEv2 packet, according to a particular embodiment. [Figure 9A] FIG. 1 is a diagram illustrating the format of an Internet Protocol (IP) header. [Figure 9B] FIG. 2 illustrates an implementation of multiple queues in accordance with certain embodiments. [Figure 9C] FIG. 2 illustrates an implementation of multiple queues in accordance with certain embodiments. [Figure 10] FIG. 2 illustrates a VxLAN packet format in accordance with certain embodiments. [Figure 11] FIG. 1 is a block diagram illustrating one pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 12] FIG. 2 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 13] FIG. 2 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 14]FIG. 2 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 15] FIG. 1 is a block diagram illustrating an example computer system in accordance with at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Detailed Description In the following description, for purposes of explanation, specific details are set forth to provide a thorough understanding of certain embodiments. However, it will be apparent that various embodiments may be practiced without these specific details. The drawings and description are not intended to be limiting. The term "exemplary" is used in this disclosure to mean "serving as an example, instance, or illustration." Any embodiment or design described in this disclosure as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0021] The present disclosure relates generally to networking, and more particularly to techniques that enable communication of Layer 2 traffic over a Layer 3 network using a Layer 3 protocol. In a particular embodiment, the techniques described in this disclosure enable communication of RDMA over Converged Ethernet (RoCE) traffic from a compute instance on a multi-tenant host machine (i.e., a host machine that hosts compute instances belonging to various tenants or customers) to a compute instance on another multi-tenant host machine over a shared Layer 3 physical network or switch fabric using a Layer 3 routing protocol. The customer or tenant experiences the communication as occurring over a dedicated Layer 2 network, but in reality, the communication occurs over a shared (i.e., shared among multiple customers or tenants) Layer 3 network using a Layer 3 routing protocol.

[0022] Also disclosed is a technique that allows a VLAN identification (e.g., a VLAN ID) that may identify a tenant to be specified in the Layer 2 header of a RoCE packet (e.g., the VLAN ID is included in an 802.1Q tag added to the RoCE packet) and also allows the VLAN identification to be mapped to information included in a Layer 3 overlay encapsulation protocol wrapper that is added to the 802.1Q tagged RoCE Layer 2 packet as the packet traverses the switch fabric. By mapping the VLAN identification (or tenancy information) to a field in the Layer 3 encapsulation wrapper, distinctions between traffic from different tenants become visible to networking devices in the Layer 3 switch fabric. The networking devices can use this information to segregate traffic belonging to different customers or tenants.

[0023] Techniques are disclosed that allow QoS information associated with a Layer 2 RDMA packet (e.g., a RoCE packet) to be preserved end-to-end, from a source host machine where the data is forwarded, all the way through a switch fabric to a destination host machine where the data is forwarded. The QoS information encoded in a Layer 2 RoCE packet is forwarded to a traffic source by a host (e.g., an ingress Top-of-Rack (TOR) switch) when the packet enters the switch fabric. The QoS information is made visible to networking devices in the switch fabric by encoding it into a layer 3 overlay encapsulation protocol wrapper that is added to 802.1Q tagged RoCE packets by the initial switch processing the packet. By mapping (e.g., copying) the QoS information into the encapsulation wrapper, networking devices in the switch fabric can use layer 3 routing protocols to route RoCE traffic through the switch fabric according to the QoS information associated with each packet.

[0024] Also disclosed is a technique that allows any of the networking devices in the switch fabric to indicate congestion on a per-packet basis. This congestion information is provided when a packet is switched from a TOR switch connected to a source host machine (an "ingress TOR switch") to a TOR switch connected to a destination host machine (an "egress TOR switch"). The congestion information is stored in the packet as it passes through the switch fabric. At the TOR switch connected to the destination host machine, the congestion information from the Layer 3 encapsulation wrapper is translated (e.g., copied) into the RoCE packet header (e.g., into the ECN bits in the IP header of the RoCE packet) so that it is stored and made available to the destination host machine. The destination host machine can then respond to the congestion information by sending a congestion notification packet (e.g., to indicate the congestion to the source host machine so that the source host machine can, e.g., reduce its sending rate accordingly).

[0025] In a typical computing environment, when data is being transferred between two computers, the data being transferred is copied multiple times by the network protocol stack software executed by the computers. This is referred to as the multi-copy problem. Furthermore, the computer's OS kernel and CPU are involved in these communications because the network stack (e.g., the TCP stack) is inherent to the kernel. This can result in significant latency in data transfer, latency that some applications cannot tolerate.

[0026] Remote Direct Memory Access (RDMA) is a direct memory access mechanism that allows the movement of data between a computer's or server's application memory without involving the computer's CPU (CPU bypass) or operating system (OS kernel bypass). This mechanism allows for high throughput, fast data transfer rates, and low latency networking. RDMA supports zero-copy networking by allowing a network adapter or network interface card (NIC) on a computer to transfer data directly from the wire to the computer's application memory, or from the computer's application memory to the wire, eliminating the need to copy data between application memory and data buffers in the computer's operating system. Such transfers require little work to be done by the CPU or caches, avoid computer context switches, and allow transfers to continue in parallel with other system operations. RDMA is very useful for High-Performance Computing (HPC) and applications that require low latency.

[0027] RoCE (RDMA over Converged Ethernet) is a network protocol that enables remote direct memory access (RDMA) over lossless Ethernet networks. RoCE enables this by encapsulating InfiniBand (IB) transport packets over Ethernet. Generally speaking, RoCE involves the use of dedicated RDMA queues and dedicated VLANs, as well as layer 2 networks. However, layer 2 networks are not scalable and are not very high performance because they lack important properties and characteristics present in the more scalable and high performance layer 3 networks. For example, Layer 2 networks do not support multiple paths between data producers (e.g., sources) and data consumers (e.g., destinations) within the network fabric, have problems with Layer 2 loops, have problems with flooding of Layer 2 frames, have no support for hierarchy in address schemes (e.g., Layer 2 does not have the concept of CIDR, prefixes and subnets), have problems with large amounts of broadcast traffic, have no control protocols that allow advertising of network connections (e.g., Layer 2 does not have protocols similar to BGP, RIP or IS-IS), have no troubleshooting protocols and tools (e.g., Layer 2 does not have tools such as ICMP or Traceroute), etc.

[0028] Currently, there are two versions of the RoCE protocol: RoCEv1 and RoCEv2. RoCEv2, also known as "routable RoCE", is defined in the document "InfiniBand® Architecture Specification Release 1.2.1 Annex A17: RoCEv2" (InfiniBand Trade Association, Beaverton, Oregon, September 2, 2014). RoCEv2 uses User Datagram Protocol (UDP) as the transport protocol. Unfortunately, UDP lacks the advanced congestion control and congestion control mechanisms that TCP provides. As a result, RoCEv2 has the following problems: These include issues like network livelock (e.g., a process is changing state and frames are moving, but frames are not moving forward), network deadlock (e.g., a process is stuck in a waiting state due to periodic resource dependencies), head-of-line (HOL) blocking (e.g., packets at the head of the queue cannot be forwarded, causing packets behind it to stall), victim flows (e.g., flows between non-congested nodes through a congested switch), unfairness (e.g., a high-bandwidth flow increases the latency of other flows), and the negative impact on lossy traffic (e.g., TCP) due to buffer consumption by lossless traffic (e.g., RDMA).

[0029] Also, generally speaking, a successful RoCEv2 implementation requires dedicated network paths and VLANs for RDMA traffic. Furthermore, RoCEv2 as a protocol relies on layer 2 Priority Flow Control (PFC), Explicit Congestion Notification (ECN), or a combination of PFC and ECN to achieve some aspect of congestion management, but these schemes are not yet fully implemented in practice. is often insufficient. PFC supports up to eight independent classes of traffic and allows a receiver to request a sender to pause the flow of a given class of traffic by sending a PAUSE frame to the sender. Unfortunately, PFC is susceptible to PAUSE frame storms (e.g., an excessive amount of PAUSE frames affecting all traffic of a given class along the entire path to the traffic source), which can lead to a complete deadlock of the network. Furthermore, PFC PAUSE frames do not allow multi-tenant operation because a PAUSE frame forces a sender to pause transmission of all traffic of a given class, and although PFC provides for up to eight traffic classes, the number of tenants can be many times greater than eight.

[0030] The disclosed embodiments include systems, methods, and apparatus for implementing multi-tenancy RoCE (RDMA over Converged Ethernet) at public cloud scale. Such disclosed embodiments include techniques for supporting multi-tenant RoCE traffic in public clouds while avoiding head-of-line blocking and maintaining high performance, low latency, lossless operation for RDMA applications. At the same time, the disclosed techniques can be implemented to support regular non-RDMA applications that use TCP / IP or UDP as transport protocols. These techniques can be applied to RoCE-enabled Ethernet network interfaces at all standard speeds, including, for example, 25G, 40G, 100G, 400G, and 800G.

[0031] Techniques for extending RoCE in the cloud, as disclosed in this disclosure, may include one or more of the following aspects: providing each customer with a VLAN or set of VLANs for their traffic; enabling hosts to use 802.1q-in-802.1q to separate traffic between customers and across different applications for a given customer (e.g., a packet may be split into two 802.1Q tags, The three steps are: mapping each VLAN to a VxLAN VNI on the ToR, assigning a unique VxLAN VNI to each customer and each of their VLANs; using a VxLAN overlay to carry customer Layer 2 traffic over the Layer 3 network; and using EVPN (Ethernet VPN) to carry MAC address information across the underlying Layer 3 network (substrate).

[0032] The embodiments described in this disclosure may be implemented to support multiple RDMA applications (e.g., cloud services, high performance computing (HPC) and / or database applications), each having multiple traffic classes. Such support may be provided by isolating traffic using the concept of network QoS traffic classes, where separate traffic classes having mission-critical traffic are assigned to a dedicated set of RDMA queues. This isolation using RDMA queues can ensure that a particular queue (e.g., congestion of a particular queue) does not affect another queue. Using such techniques, multiple RDMA tenants (also known as "public cloud customers") can be supported such that queue configuration in the Clos fabric is transparent to the end customer hosts (cloud customers). The network may be configured to map DSCP markings received from customer hosts to the correct settings of network queues, thereby decoupling host QoS policies (configurations) from fabric QoS policies (configurations). Customers may signal performance expectations using DSCP traffic classes (also known as DSCP code points) and / or 802.1q traffic classes. These DSCP and 802.1q classes are mapped to QoS queues in the Clos network providing decoupling of host QoS configuration from the Clos fabric QoS configuration.

[0033] To convey QoS queue information and ECN markings across a Clos fabric, it may be desirable to ensure that the QoS queue information is carried across multiple network domains, for example, from a Layer 2 port to a host, from a Layer 3 port to another switch, or from a VxLAN virtual Layer 2 port to another VxLAN interface on another switch. Such cross-domain transport of QoS queue information may include carrying and performing QoS markings and ECN bit markings across these various network domains as described in this disclosure.

[0034] Example Virtual Network Architecture The term cloud services generally refers to the systems and infrastructure (cloud infrastructure) provided by a cloud service provider (CSP). Cloud services refer to services that can be consumed by users or customers on demand (e.g., via a subscription model) using a CSP's infrastructure (a common infrastructure structure). Typically, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premise servers and systems. Thus, customers can use cloud services provided by CSPs without having to separately purchase hardware and software resources for the services. Cloud services are designed to provide subscribing customers with easy and scalable access to applications and computing resources without the customer having to invest in procuring the infrastructure used to deliver the services.

[0035] Some cloud service providers offer different types of cloud services, including a variety of different types or models, such as Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), and Infrastructure-as-a-Service (IaaS).

[0036] A customer can subscribe to one or more cloud services offered by a CSP. A customer can be any entity, such as an individual, an organization, a business, etc. When a customer subscribes or registers for a service offered by a CSP, a tenant or account is created for that customer. The customer can then access one or more subscribed cloud resources associated with the account through this account.

[0037] As mentioned above, IaaS (Infrastructure as a Service) is a service that provides services to a specific type of client. Cloud Computing Services. In the IaaS model, the CSP provides infrastructure (called Cloud Service Provider Infrastructure or CSPI) that customers can use to build their own customizable networks and deploy customer resources. Thus, the customer's resources and networks are hosted in a distributed environment by the infrastructure provided by the CSP. This differs from traditional computing, where the infrastructure provided by the customer hosts the customer's resources and networks.

[0038] CSPI may include interconnected high performance computing resources including various host machines, memory resources, and network resources that form a physical network, also called a substrate network or underlay network. CSPI resources may be distributed across one or more data centers that are geographically distributed across one or more geographic regions. Virtualization software may run on these physical resources to provide a virtualized distributed environment. Virtualization creates an overlay network (also called a software-based network, software-defined network, or virtual network) on the physical network. The CSPI physical network provides the basis for creating one or more overlay or virtual networks on top of the physical network. The virtual or overlay network may include one or more Virtual Cloud Networks (VCNs). Virtual networks are implemented using software virtualization techniques (e.g., hypervisors, functions performed by a network virtualization device (NVD) (e.g., smart NICs), top-of-rack (TOR) switches, smart TORs that implement one or more functions performed by an NVD, and other mechanisms) to create a network abstraction layer that can run on top of a physical network. Virtual networks can take various forms, such as peer-to-peer networks, IP networks, etc. Virtual networks are typically either layer 3 IP networks or layer 2 VLANs. Such virtual or overlay networks are often referred to as virtual layer 3 networks or overlay layer 3 networks. Examples of protocols developed for virtual networks are IP-in-IP (or Generic Routing Encapsulation (GRE)), Virtual Extensible LAN (VXLAN - IETF RFC7348), Virtual Layer 3 Layer 3 Networks (VXLAN) and Virtual Layer 3 ... Private network (VPN: Virtual Private Network) (e.g. MPLS layer Layer 3 Virtual Private Network (RFC4364), VMware NSX, GENEVE (Generic Network Virtualization Encapsulation), etc.

[0039] In the case of IaaS, the infrastructure provided by the CSP (CSPI) may be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing service provider may host the infrastructure elements (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider may provide various services (e.g., billing, monitoring, logging, security, load balancing, and clustering, etc.) that are attached to those infrastructure elements. These services are policy-driven, allowing IaaS users to maintain application availability and performance by implementing policies to drive load balancing. CSPI provides infrastructure and a set of complementary cloud services. It enables customers to build and run a wide range of applications and services in a highly available, hosted, distributed environment. CSPI provides high performance compute resources and power as well as storage capacity on flexible virtual networks that can be securely accessed from various network locations, including customer on-premise networks. When a customer subscribes or registers for an IaaS service offered by a CSP, the tenancy created for that customer is a securely isolated partition from CSPI, where the customer can create, organize and manage their cloud resources.

[0040] Customers can build their own virtual networks using the compute, memory, and networking resources provided by CSPI. On these virtual networks, they can deploy one or more customer resources or workloads, such as compute instances. For example, customers can build one or more customizable private virtual networks, called Virtual Cloud Networks (VCNs), using resources provided by CSPI. Customers can deploy one or more customer resources, e.g., compute instances, on the customer VCN. The compute instances may be virtual machines, bare metal instances, etc. Thus, CSPI provides an infrastructure and a set of complementary cloud services that enable customers to build and run various applications and services in a highly available virtual host environment. Customers do not manage or control the underlying physical resources provided by CSPI, but they control the operating systems, storage, and deployed applications, and in some cases have limited control over some networking components (e.g., firewalls).

[0041] The CSP may provide a console that allows customers and network administrators to configure, access, and manage resources deployed in the cloud using CSPI resources. In certain embodiments, the console provides a web-based user interface that can be used to utilize and manage the CSPI. In some embodiments, the console is a web-based application provided by the CSP.

[0042] CSPI can support single-tenancy or multi-tenancy architectures. In a single-tenancy architecture, software (e.g., applications, databases) or hardware elements (e.g., host machines or servers) serve a single customer or tenant. In a multi-tenancy architecture, software or hardware elements serve multiple customers or tenants. Thus, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenancy environment, CSPI has precautions and safeguards in place to ensure that each tenant's data is isolated and not visible to other tenants.

[0043] In a physical network, a network endpoint (endpoint) refers to a computing device or system that is connected to the physical network and communicates bidirectionally with the connected network. A network endpoint of a physical network may be connected to a local area network (LAN), a wide area network (WAN), or another type of physical network. Examples of traditional endpoints in a physical network are modems, hubs, bridges, switches, routers, and other networking devices; physical computers (or host machines); Each physical device in a physical network has a fixed network address that can be used to communicate with that device. This fixed network address may be a Layer 2 address (e.g., a MAC address), a fixed Layer 3 address (e.g., an IP address), and the like. In a virtualized environment or virtual network, the endpoints may include various virtual endpoints, such as virtual machines hosted by elements of the physical network (e.g., hosted by a physical host machine). These endpoints in the virtual network are addressed by overlay addresses, such as overlay Layer 2 addresses (e.g., overlay MAC addresses) and overlay Layer 3 addresses (e.g., overlay IP addresses). Network overlays achieve flexibility by allowing network administrators to move overlay addresses associated with network endpoints using software management (e.g., via software implementing the control plane of the virtual network). Thus, unlike physical networks, in a virtual network, overlay addresses (e.g., overlay IP addresses) can be moved from one endpoint to another using network management software. Because virtual networks are built on physical networks, both the virtual network and the underlying physical network are involved in communications between elements of the virtual network. To facilitate such communications, each element of the CSPI is configured to learn and store mappings that map overlay addresses of the virtual network to real physical addresses of the substrate network, or real physical addresses of the substrate network to overlay addresses of the virtual network. These mappings are used to facilitate communications. To facilitate virtual network routing, customer traffic is encapsulated.

[0044] Thus, physical addresses (e.g., physical IP addresses) are associated with elements of a physical network, and overlay addresses (e.g., overlay IP addresses) are associated with entities of a virtual network. Both physical and overlay IP addresses are real IP addresses. They are distinct from virtual IP addresses, which are mapped to multiple real IP addresses. Virtual IP addresses provide a one-to-many mapping between a virtual IP address and multiple real IP addresses.

[0045] A cloud infrastructure or CSPI is physically hosted in one or more data centers in one or more regions around the world. A CSPI may include elements of a physical or substrate network and virtualized elements of a virtual network (e.g., virtual networks, compute instances, virtual machines) built on the physical network elements. In certain embodiments, a CSPI is organized and hosted in realms, regions, and availability domains. A realm is a localized geographic area that typically includes one or more data centers. The regions are generally independent of one another and may be separated by vast distances, e.g., across countries or continents. For example, a first region may be in Australia, another region may be in Japan, and yet another region may be in India. CSPI resources are divided among these regions such that each region has an independent subset of CSPI resources. Each region may provide a set of core infrastructure services and resources, such as compute resources (e.g., bare metal servers, virtual machines, containers and related infrastructure), storage resources (e.g., block volume storage, file storage, object storage, archive storage), networking resources (e.g., virtual cloud networks (VCNs), load balancing resources, connections to on-premise networks), database resources, edge networking resources (e.g., DNS), access management, and monitoring resources. Each region typically has multiple routes to connect the region to other regions in the realm.

[0046] Typically, applications are deployed in the region where they are used the most (i.e., on infrastructure associated with that region) because using nearby resources is faster than using resources that are farther away. Applications may also be deployed in different regions for a variety of reasons, such as redundancy to reduce risk from region-wide events such as large weather systems or earthquakes, redundancy to meet various requirements for legal jurisdictions, tax domains, and other business or societal standards.

[0047] Datacenters within a region may be further organized and subdivided into availability domains (ADs). An availability domain is one or more datacenters located in a region. A region may correspond to a single datacenter or multiple datacenters. A region may be composed of one or more availability domains. In such a distributed environment, CSPI resources are specific to a region, such as a Virtual Cloud Network (VCN), or specific to an availability domain, such as a compute instance.

[0048] ADs within a region are isolated from each other to be fault tolerant and configured to have a very low probability of failing simultaneously. This is achieved by configuring the ADs to not share critical infrastructure resources such as networking, physical cables, cable paths, and cable entrances, so that a failure of one AD in a region will have little impact on the availability of other ADs in the same region. ADs within the same region can be connected to each other with low-latency, high-bandwidth networks to provide highly available connections to other networks (e.g., the Internet, customer on-premise networks) and to build replicated systems for both high availability and disaster recovery in multiple ADs. Cloud services utilize multiple ADs to ensure high availability as well as to protect against resource failures. As the infrastructure provided by the IaaS provider grows, more regions and ADs may be added along with additional capacity. Traffic between available domains is typically encrypted.

[0049] In certain embodiments, regions are grouped into realms. A realm is a logical collection of regions. Realms are isolated from each other and do not share any data. Regions in the same realm can communicate with each other, but regions in different realms cannot. A CSP's customer's tenancy or account exists in a single realm and can span one or more regions that belong to that single realm. Typically, when a customer subscribes to an IaaS service, a tenancy or account for that customer is created in a customer-specified region (called a "home" region) within a realm. The customer can extend the customer's tenancy to one or more other regions within the realm. The customer cannot access regions that do not exist within the realm in which the customer's tenancy resides.

[0050] An IaaS provider may offer multiple realms, each corresponding to a particular set of customers or users. For example, a commercial realm may be offered for commercial customers. As another example, a realm may be offered for a particular country or for customers in that country. As yet another example, a government realm may be offered, for example, for a government. For example, a government realm may be created for a particular government and may have a higher security level than a commercial realm. For example, Oracle® Cloud Infrastructure (OCI) currently offers a realm for the commercial domain and two realms for the government cloud domain (e.g., FedRAMP authorized and IL5 authorized).

[0051] In certain embodiments, an AD can be subdivided into one or more fault domains. A fault domain is a grouping of infrastructure resources within an AD to provide anti-affinity. can distribute compute instances so that compute instances are not placed on the same physical hardware within one AD. This is known as anti-affinity. A failure domain refers to a collection of hardware elements (computers, switches, etc.) that share a single point of failure. A compute pool is logically divided into failure domains. Thus, a hardware failure or compute hardware maintenance event that affects one failure domain does not affect instances in other failure domains. In some embodiments, the number of failure domains in each AD may be different. For example, in a particular embodiment, each AD includes three failure domains. The failure domains act as logical data centers within an AD.

[0052] When a customer subscribes to an IaaS service, resources from CSPI are provisioned to the customer and associated with the customer's tenancy. Customers can use these provisioned resources to build private networks and deploy resources on these networks. A customer network hosted on the cloud by CSPI is called a Virtual Cloud Network (VCN). Customers can configure one or more Virtual Cloud Networks (VCNs) using CSPI resources allocated for the customer. A VCN is a virtual or software-defined private network. Customer resources deployed on a customer's VCN can include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances may represent various customer workloads such as applications, load balancers, databases, etc. Compute instances deployed on a VCN can communicate with publicly accessible endpoints (public endpoints) over a public network such as the Internet, can communicate with other instances in the same VCN or other VCNs (e.g., other VCNs of the customer, or VCNs not belonging to the customer), can communicate with customer on-premise data centers or networks, can communicate with service endpoints, and can communicate with other types of endpoints.

[0053] CSPs can offer various services using CSPI. In some cases, customers of CSPI themselves can act as service providers and offer services using CSPI resources. Service providers can expose service endpoints that are characterized by identifying information (e.g., IP addresses, DNS names and ports). Customer resources (e.g., compute instances) can consume a particular service by accessing the service endpoint for that particular service exposed by the service. These service endpoints are generally publicly accessible endpoints over a public communications network, such as the Internet, where users use a public IP address associated with the endpoint. Publicly accessible network endpoints are sometimes referred to as public endpoints.

[0054] In certain embodiments, a service provider may expose a service through a service endpoint (sometimes referred to as a service endpoint). A customer of the service may access the service using this service endpoint. In certain embodiments, a service endpoint provided for a service may be accessed by multiple customers wishing to consume the service. In other implementations, a dedicated service endpoint may be provided to a customer. Thus, only that customer may access the service using that dedicated service endpoint.

[0055] In certain embodiments, when a VCN is created, it has a private key assigned to it. Private overlay Classless Inter-Domain Routing: Private overlay IP address range (e.g., 10.0 / 16) A VCN is associated with a CIDR (common domain name, shared domain, and local area network) address space. A VCN contains associated subnets, route tables, and gateways. A VCN exists within a single region, but can extend to one or more or all available domains in the region. A gateway is a virtual interface configured for a VCN that enables traffic communication between the VCN and one or more endpoints outside the VCN. You can configure one or more different types of gateways for a VCN to enable communication between the different types of endpoints.

[0056] A VCN may be subdivided into one or more subnetworks, such as one or more subnets. A subnet is thus a unit or division that may be created within a VCN. A VCN may have one or more subnets. Each subnet in a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that does not overlap with other subnets in the VCN and that represents an address space subset of the VCN's address space.

[0057] Each compute instance is associated with a Virtual Network Interface Card (VNIC), which allows each compute instance to participate in a subnet of a VCN. A VNIC is a logical representation of a physical network interface card (NIC). In general, a VNIC is an interface between an entity (e.g., compute instance, service) and a virtual network. A VNIC resides in a subnet and has one or more associated IP addresses and associated security rules or policies. A VNIC corresponds to a layer 2 port on a switch. A VNIC is connected to a compute instance and a subnet in a VCN. A VNIC associated with a compute instance allows the compute instance to be part of a subnet of a VCN, and allows the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, endpoints in a different subnet in the VCN, or endpoints outside the VCN. Thus, a VNIC associated with a compute instance determines how the compute instance connects with endpoints inside and outside the VCN. A VNIC for a compute instance is created and associated with the compute instance when the compute instance is created and added to a subnet in a VCN. When a subnet consists of a set of compute instances, it includes VNICs corresponding to the set of compute instances, where each VNIC is connected to a compute instance in the set of compute instances.

[0058] Each compute instance is assigned a private overlay IP address via the VNIC associated with the compute instance. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route traffic for the compute instance. All VNICs in a particular subnet use the same route table, security lists, and DHCP options. As described above, each subnet in a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets in the VCN and that represent an address space subset of the VCN's address space. For a VNIC on a particular subnet of a VCN, the overlay IP address assigned to the VNIC is an address from the contiguous range of overlay IP addresses assigned to the subnet.

[0059] In certain embodiments, if desired, the compute instances may include private overlays. In addition to the overlay IP address, an additional overlay IP address can be assigned, e.g., one or more public IP addresses in case of a public subnet. These multiple addresses are assigned to the same VNIC or to multiple VNICs associated with a compute instance. However, each instance has a primary VNIC associated with an overlay private IP address that is created at instance launch and assigned to the instance. This primary VNIC cannot be deleted. Additional VNICs, called secondary VNICs, can be added to an existing instance in the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. Secondary VNICs can be in the same subnetwork as the primary VNIC, or in different subnetworks in the same or different VCNs.

[0060] Compute instances can optionally be assigned a public IP address if they are in a public subnet. When creating a subnet, you can specify the subnet as either a public or private subnet. A private subnet means that resources in the subnet (e.g., compute instances) and associated VNICs cannot have public overlay IP addresses. A public subnet means that resources in the subnet and associated VNICs can have public IP addresses. Customers can specify subnets that exist across a single available domain or multiple available domains within a region or realm.

[0061] As mentioned above, a VCN may be subdivided into one or more subnets. In certain embodiments, a virtual router (referred to as a VCN VR or simply VR) configured for a VCN allows communication between subnets of the VCN. For a subnet in a VCN, the VR represents a logical gateway for that subnet that allows the subnet (i.e., the compute instances on that subnet) to communicate with endpoints on other subnets inside the VCN and with other endpoints outside the VCN. A VCN VR is a logical entity configured to route traffic between VNICs in a VCN and a virtual gateway (gateway) associated with the VCN. Gateways are further described below with respect to FIG. 1. A VCN VR is a layer 3 / IP layer concept. In one embodiment, there is one VCN VR for a VCN. This VCN VR potentially has an unlimited number of ports addressed by IP addresses, one port for each subnet of the VCN. In this way, the VCN VR has a different IP address for each subnet of the VCN to which the VCN VR is connected. The VRs are also connected to various gateways configured for the VCN. In certain embodiments, a particular overlay IP address from a subnet's overlay IP address range is reserved in a port of the subnet's VCN VR. For example, consider a VCN with two subnets, each with associated address ranges 10.0 / 16 and 10.1 / 16. For a first subnet of the VCN with address range 10.0 / 16, addresses from this range are reserved in a port of the subnet's VCN VR. In some cases, a first IP address from this range may be reserved in a VCN VR. For example, for a subnet with overlay IP address range 10.0 / 16, IP address 10.0.0.1 may be reserved in a port of the subnet's VCN VR. For a second subnet in the same VCN with address range 10.1 / 16, the VCN VR may have a port of the second subnet with IP address 10.1.0.1.A VCN VR has a different IP address for each subnet in the VCN.

[0062] In some other embodiments, each subnet in a VCN is associated with a VR. A VCN may have a VR associated with it that is addressable by the subnet using a reserved or default IP address. The reserved or default IP address may be, for example, the first IP address from a range of IP addresses associated with the subnet. VNICs in the subnet may use this default or reserved IP address to communicate (e.g., send and receive packets) with the VR associated with the subnet. In such an embodiment, the VR is the ingress / egress point for the subnet. VRs associated with subnets in a VCN may communicate with other VRs associated with other subnets in the VCN. The VRs may also communicate with gateways associated with the VCN. The VR functions of a subnet are performed on or by one or more NVDs that perform the VNIC functions of the VNICs in the subnet.

[0063] Route tables, security rules, and DHCP options may be configured for a VCN. A route table is a virtual route table for a VCN and contains rules for routing traffic from subnets inside the VCN to destinations outside the VCN through gateways or specially configured instances. You can customize a VCN's route table to control the forwarding / routing of packets into and out of the VCN. DHCP options refer to configuration information that is automatically provided to an instance when it is launched.

[0064] Security rules configured for a VCN represent the overlay firewall rules for the VCN. Security rules can include ingress and egress rules and can specify the types of traffic allowed to flow in and out of instances in the VCN (e.g., based on protocol and port). Customers can choose whether certain rules are stateful or stateless. For example, a customer can allow incoming SSH traffic from anywhere to a set of instances by configuring a stateful ingress rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules may be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to resources in that group. A security list, on the other hand, contains rules that apply to all resources in a subnet that uses that security list. A VCN may include default security rules and a default security list. DHCP options configured for a VCN provide configuration information that is automatically provided when instances in the VCN are launched.

[0065] In certain embodiments, configuration information for a VCN is determined and stored by a VCN control plane. The configuration information for a VCN may include, for example, address ranges associated with the VCN, subnets and associated information within the VCN, one or more VRs associated with the VCN, compute instances and associated VNICs within the VCN, NVDs performing various virtualized network functions (e.g., VNICs, VRs, gateways) associated with the VCN, state information for the VCN, and other VCN-related information. In certain embodiments, the VCN distribution service publishes the configuration information stored by the VCN control plane, or a portion thereof, to the NVD. The distributed information can be used to forward packets to and from compute instances in the VCN by updating information (e.g., forwarding tables, routing tables, etc.) stored and used by the NVD.

[0066] In certain embodiments, the creation of VCNs and subnets is handled by a VCN Control Plane (CP), and the launch of compute instances is handled by a compute Control Plane (CP). The compute control plane allocates the physical resources of the compute instances. , which is configured to then invoke the VCN control plane to create and attach VNICs to the compute instances. The VCN CP also sends the VCN data mappings to a VCN data plane, which is configured to perform packet forwarding and routing functions. In certain embodiments, the VCN CP provides a distribution service configured to provide updates to the VCN data plane. Examples of the VCN control plane are shown in Figures 11, 12, 13, and 14 (see reference numbers 1116, 1216, 1316, and 1416) and described below.

[0067] A customer can create one or more VCNs with resources hosted by CSPI. Compute instances deployed on the customer VCN can communicate with different endpoints. These endpoints can include endpoints hosted by CSPI and endpoints external to CSPL.

[0068] Various different architectures for implementing cloud-based services using CSPI are shown in Figures 1, 2, 3, 4, 5, 11, 12, 13, and 14 and described below. Figure 1 is a high-level diagram of a distributed environment 100 showing an overlay VCN or customer VCN hosted by CSPI, according to certain embodiments. The distributed environment shown in Figure 1 includes multiple elements in an overlay network. The distributed environment 100 shown in Figure 1 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some implementations, the distributed environment shown in Figure 1 may have more or fewer systems or elements than those shown in Figure 1, may combine two or more systems, or may have a different system configuration or arrangement.

[0069] As shown in the example of FIG. 1, a distributed environment 100 includes a CSPI 101 that provides services and resources that customers can subscribe to and use to build a virtual cloud network (VCN). In a particular embodiment, the CSPI 101 provides IaaS services to subscribing customers. Data centers in the CSPI 101 may be organized into one or more regions. FIG. 1 shows an example of a region, “US Region” 102. A customer configures a customer VCN 104 for the region 102. A customer can deploy various compute instances on the VCN 104, which may include virtual machines or bare metal instances. Examples of instances include applications, databases, load balancers, etc.

[0070] In the embodiment shown in Figure 1, customer VCN 104 includes two subnets, "Subnet-1" and "Subnet-2," each with its own CIDR IP address range. In Figure 1, the overlay IP address range of Subnet-1 is 10.0 / 16, and the address range of Subnet-2 is 10.1 / 16. VCN virtual router 105 represents the logical gateway of the VCN, enabling communication between subnets of VCN 104 and with other endpoints outside the VCN. VCN VR 105 is configured to route traffic between VNICs in VCN 104 and gateways associated with VCN 104. VCN VR 105 provides a port to each subnet of VCN 104. For example, VR 105 may provide a port with IP address 10.0.0.1 to subnet-1 and a port with IP address 10.1.0.1 to subnet-2.

[0071] Multiple compute instances may be deployed on each subnet. In this case, the compute instances may be virtual machine instances and / or bare metal instances. The compute instances in a subnet may be hosted by one or more host machines in CSPI 101. The compute instances may be configured to communicate with each other. Each compute instance joins a subnet via an associated VNIC. For example, as shown in FIG. 1, compute instance C1 is part of subnet-1 via a VNIC associated with the compute instance. Similarly, compute instance C2 is part of subnet-1 via a VNIC associated with C2. Similarly, multiple compute instances, which may be virtual machine instances or bare metal instances, may be part of subnet-1. Each compute instance is assigned a private overlay IP address and a MAC address via an associated VNIC. For example, in FIG. 1, compute instance C1 has an overlay IP address of 10.0.0.2 and a MAC address of M1, and compute instance C2 has a private overlay IP address of 10.0.0.3 and a MAC address of M2. Each compute instance in subnet-1, including compute instances C1 and C2, has a default route to VCN VR105 using IP address 10.0.0.1, which is the IP address of a port of VCN VR105 in subnet-1.

[0072] Subnet-2 may have multiple compute instances deployed thereon, including virtual machine instances and / or bare metal instances. For example, as shown in FIG. 1, compute instances D1 and D2 are part of subnet-2 via VNICs associated with the respective compute instances. In the embodiment shown in FIG. 1, compute instance D1 has an overlay IP address 10.1.0.2 and a MAC address MM1, and compute instance D2 has a private overlay IP address 10.1.0.3 and a MAC address MM2. Each compute instance in subnet-2, including compute instances D1 and D2, has a default route to VCN VR105 using IP address 10.1.0.1, which is the IP address of a port in VCN VR105 in subnet-2.

[0073] VCN A 104 may also include one or more load balancers. For example, a load balancer may be provided for a subnet and configured to load balance traffic among multiple compute instances on the subnet. A load balancer may also be provided to load balance traffic among subnets in the VCN.

[0074] A particular compute instance deployed on VCN 104 can communicate with a variety of different endpoints. These endpoints may include endpoints hosted by CSPI 200 and endpoints external to CSPI 200. Endpoints hosted by CSPI 101 may include endpoints on the same subnet as the particular compute instance (e.g., communication between two compute instances in subnet-1), endpoints in a different subnet but in the same VCN (e.g., communication between a compute instance in subnet-1 and a compute instance in subnet-2), endpoints in a different VCN in the same region (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN in the same region 106 or 110, communication between a compute instance in subnet-1 and an endpoint in a service network 110 in the same region), or endpoints in a VCN in a different region (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN in a different region 108). Also, compute instances in a subnet hosted by CSPI 101 can communicate with endpoints not hosted by CSPI 101 (i.e., external to CSPI 101). These external endpoints include endpoints in customer on-premise networks 116, endpoints in other remote cloud host networks 118, public endpoints 114 accessible via a public network such as the Internet, and other endpoints.

[0075] Communication between compute instances on the same subnet is facilitated using VNICs associated with the source and destination compute instances. For example, compute instance C1 in subnet-1 may want to send a packet to compute instance C2 in subnet-1. For a packet sent from a source compute instance whose destination is another compute instance in the same subnet, the packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the packet's destination information from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the packet's next hop, performing any packet encapsulation / decapsulation functions as necessary, and forwarding / routing the packet to the next hop to facilitate communication to the packet's intended destination. If the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing. The VNIC associated with the destination compute instance then executes and forwards the packet to the destination compute instance.

[0076] When communicating a packet from a compute instance in a subnet to an endpoint in a different subnet of the same VCN, the communication is facilitated by the VNICs associated with the source and destination compute instances and the VCN VRs. For example, if compute instance C1 in subnet-1 of FIG. 1 wants to send a packet to compute instance D1 in subnet-2, the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR105 using the VCN VR's default route or port 10.0.0.1. VCN VR105 is configured to route the packet to subnet-2 using port 10.1.0.1. The packet is then received and processed by the VNIC associated with D1, which forwards the packet to compute instance D1.

[0077] To communicate a packet from a compute instance in VCN 104 to an endpoint outside VCN 104, the communication is facilitated by a VNIC associated with the source compute instance, a VCN VR 105, and a gateway associated with VCN 104. One or more types of gateways may be associated with VCN 104. A gateway is an interface between a VCN and another endpoint, the other endpoint being outside VCN. A gateway is a layer 3 / IP layer concept that enables a VCN to communicate with an endpoint outside VCN. Thus, a gateway facilitates traffic flow between a VCN and other VCNs or networks. A variety of different types of gateways may be configured in a VCN to facilitate different types of communication with different types of endpoints. With a gateway, the communication may occur over a public network (e.g., the Internet) or a private network. These communications may use a variety of communication protocols.

[0078] For example, compute instance C1 may wish to communicate with an endpoint outside of VCN104. The packet may first be processed by a VNIC associated with the source compute instance C1. The VNIC processing determines that the destination of the packet is outside of Cl's subnet-1. The VNIC associated with C1 may forward the packet to VCN VR105 in VCN104. VCN VR105 then processes the packet and, as part of the processing, determines a particular gateway associated with VCN104 as the next hop for the packet based on the destination of the packet. VCN VR105 then forwards the packet For example, if the destination is an endpoint in a customer's operating premises network, the packet may be forwarded by VCN VR 105 to a Dynamic Routing Gateway (DRG) 122 configured for VCN 104. The packet is then forwarded from the gateway to the next The packet may then be forwarded to the next hop to facilitate communication of the packet to its intended final destination.

[0079] Various different types of gateways may be configured for a VCN. Examples of gateways that may be configured for a VCN are shown in FIG. 1 and described below. Examples of gateways associated with a VCN are also shown in FIGS. 11, 12, 13, and 14 (e.g., gateways indicated by reference numbers 1134, 1136, 1138, 1234, 1236, 1238, 1334, 1336, 1338, 1434, 1436, and 1438) and described below. As shown in the embodiment shown in FIG. 1, a dynamic routing gateway (DRG) 122 may be added to or associated with a customer VCN 104. The DRG 122 provides a path for private network traffic communication between the customer VCN 104 and another endpoint. The other endpoint may be a customer on-premise network 116, a VCN 108 in a different region of the CSPI 101, or another remote cloud network 118 not hosted by the CSPI 101. The customer on-premise network 116 may be a customer network or customer data center built with the customer's resources. Access to the customer on-premise network 116 is generally highly restricted. For a customer that has both the customer on-premise network 116 and one or more VCNs 104 deployed or hosted in the cloud by the CSPI 101, the customer may want the on-premise network 116 and the cloud-based VCNs 104 to be able to communicate with each other. This allows the customer to build an extended hybrid environment that includes the customer's VCNs 104 and the on-premise network 116 hosted by the CSPI 101. The DRG 122 enables such communication. To enable such communication, a communication channel 124 is set up. In this case, one endpoint of the communication channel is located in the customer on-premise network 116, and the other endpoint is located in the CSPI 101 and connected to the customer VCN 104. The communication channel 124 may be via a public communication network, such as the Internet, or a private communication network.Using a variety of different communication protocols, such as IPsec VPN technology over a public communication network such as the Internet, or Oracle's FastConnect technology, which uses a private network instead of a public network. The device or equipment in the customer premises network 116 that forms one endpoint of the communication channel 124 is called Customer Premise Equipment (CPE), such as the CPE 126 shown in Figure 1. The endpoint on the CSPI 101 side may be a host machine running the DRG 122.

[0080] In certain embodiments, a Remote Peering Connection (RPC) can be added to a DRG, allowing a customer to connect one VCN to another Using such RPCs, a customer VCN 104 can use a DRG 122 to connect to a VCN 108 in another region. The DRG 122 can also connect to other remote cloud networks 118 not hosted by CSPI 101, such as Microsoft® Azure cloud, Azure, and other cloud services. Amazon® may be used to communicate with the AWS Cloud.

[0081] As shown in FIG. 1, an Internet Gateway (IGW) is configured in the customer VCN 104 to allow compute instances on the customer VCN 104 to communicate with public endpoints 114 accessible over a public network, such as the Internet. In one embodiment, an IGW 1120 may be configured to connect a VCN to a public network, such as the Internet. The IGW 120 allows public subnets in a VCN, such as VCN 104 (resources in the public subnet have public overlay IP addresses) to directly access public endpoints 112 on a public network 114, such as the Internet. The IGW 120 may be used to initiate connections from subnets in VCN 104 or from the Internet.

[0082] A Network Address Translation (NAT) gateway 128 can be configured in the customer VCN 104. It allows cloud resources without dedicated public overlay IP addresses in the CN to access the Internet without exposing them to direct incoming Internet connections (e.g., L4-L7 connections). This allows private subnets in a VCN, such as private subnet-1 in VCN 104, to privately access public endpoints on the Internet. With a NAT gateway, connections can be initiated from a private subnet to the public Internet, but connections cannot be initiated from the Internet to a private subnet.

[0083] In certain embodiments, a service gateway (SGW) 126 can be configured in the customer VCN 104. The SGW 126 is a gateway that connects the VCN 104 and the service It provides a conduit for private network traffic to and from service endpoints supported by the network 110. In certain embodiments, the service network 110 may be provided by a CSP and may offer a variety of services. An example of such a service network is the Oracle® Service Network, which offers a variety of services that customers can use. For example, a compute instance (e.g., a database system) in a private subnet of the customer VCN 104 may back up data to a service endpoint (e.g., an object store) without needing a public IP address or access to the Internet. In some embodiments, a VCN may have only one SGW, and connections may be initiated only from subnets within the VCN, not from the service network 110. When peering a VCN to another VCN, resources in the other VCN typically do not have access to the SGW. FastConnect or VPN Resources in an on-premises network connected to a VCN with Connect can also use a service gateway configured in that VCN.

[0084] In some implementations, the SGW 126 uses service Classless Inter-Domain Routing (CIDR) labels. A CIDR label is a string that represents all the regional public IP address ranges for a service or group of services of interest. A customer uses the service CIDR label to control traffic to a service when configuring the SGW and associated routing rules. A customer can use the service CIDR label as an option when configuring security rules without having to adjust the security rules if the service's public IP address changes in the future.

[0085] Local Peering Gateway (LPG) 132 is a peering gateway that allows customers to A gateway that can be added to a VCN 104 that allows the VCN 104 to peer with another VCN in the same region. Peering means that the VCNs communicate using private IP addresses without traffic going over a public network such as the Internet or routing traffic through the customer on-premises network 116. In a preferred embodiment, a VCN N has a separate LPG for each peering it establishes. Local peering or VCN peering is a common practice used to establish network connectivity between different applications or infrastructure management functions.

[0086] A service provider, such as a provider of services in service network 110, can provide access to a service using different access models. According to a public access model, a service may be exposed as a public endpoint publicly accessible by a compute instance in the customer VCN over a public network such as the Internet, or may be accessed privately through SGW 126. According to a specific private access model, a service may be accessed as a private IP endpoint in a private subnet in the customer VCN. This is called Private Endpoint (PE) access and allows a service provider to expose its services as instances in the customer's private network. A private endpoint resource represents a service in a customer VCN. Each PE appears as a VNIC (called a PE-VNIC and has one or more private IPs) that the customer selects from a subnet in the customer VCN. Thus, the PE provides a way to provide a service in the customer's private VCN subnet using a VNIC. Because the endpoints are exposed as VNICs, the PE VNIC can utilize all the functionality associated with a VNIC, such as routing rules and security lists.

[0087] A service provider enables access through a PE by registering a service. The provider can associate policies with the service that regulate the visibility of the service to customer tenants. A provider can register multiple services under a single virtual IP address (VIP), especially for multi-tenant services. There can also be multiple private endpoints representing the same service (in multiple VCNs).

[0088] Compute instances in the private subnet can then access the service using the private IP address or service DNS name of the PE VNIC. Compute instances in the customer VCN can access the service by sending traffic to the private IP address of the PE in the customer VCN. A Private Access Gateway (PAGW) 130 is a gateway resource that can be connected to a service provider VCN (e.g., a VCN in the service network 110) that serves as an ingress / egress point for all traffic from / to the customer subnet private endpoints. The PAGW 130 allows the provider to scale the number of PE connections without utilizing internal IP address resources. The provider only needs to configure one PAGW for any number of services registered in a single VCN. The provider can present the services as private endpoints in multiple VCNs of one or more customers. From the customer's perspective, the PE VNIC appears to be connected not to the customer's instance but to the service the customer wants to interact with. Traffic destined for the private endpoint is routed to the service through the PAGW 130. These are called customer-to-service private connections (C2S connections).

[0089] Also, the PE concept is used to ensure that traffic is routed across FastConnect / IPsec links and customer VCNs. It also extends private access of services to customer on-premises networks and data centers by allowing traffic to flow through private endpoints in the customer VCN. It also extends private access of services to customer peering VCNs by allowing traffic to flow between LPG132 and PEs in the customer VCN. It can also be expanded.

[0090] Customers can control VCN routing at the subnet level, so they can specify which subnets in a customer VCN, such as VCN 104, use each gateway. The VCN's route tables can be used to determine whether traffic can be routed outside of the VCN through a particular gateway. For example, in a particular case, the route table for a public subnet in customer VCN 104 can send non-local traffic through IGW 120. The route table for a private subnet in the same customer VCN 104 can send traffic to CSP services through SGW 126. All remaining traffic may be sent through NAT gateway 128. The route tables control only traffic that leaves the VCN.

[0091] Security lists associated with a VCN are used to control traffic coming into the VCN through inbound connections and gateways. All resources in a subnet use the same mute table and security lists. Security lists may be used to control certain types of traffic entering and leaving instances in a subnet of a VCN. Security list rules may include ingress (inbound) rules and egress (outbound) rules. For example, ingress rules may specify allowed source address ranges and egress rules may specify allowed destination address ranges. Security rules may specify specific protocols (e.g., TCP, ICMP), specific ports (e.g., port 22 for SSH, port 3389 for Windows RDP), etc. In certain implementations, the operating system of the instance may enforce its own firewall rules that match the security list rules. Rules may be stateful (e.g., connections are tracked and responses are automatically allowed without explicit security list rules for the response traffic) or stateless.

[0092] Access from a customer VCN (i.e., resources or compute instances deployed on VCN 104) may be categorized as public access, private access, or dedicated access. Public access refers to an access model for accessing public endpoints using public IP addresses or NAT. Private access allows customer workloads in VCN 104 with private IP addresses (e.g., resources in a private subnet) to access a service without traversing a public network such as the Internet. In certain embodiments, CSPI 101 allows customer VCN workloads with private IP addresses to access public service endpoints of a service using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer VCN and the public endpoints of the service that reside outside the customer private network.

[0093] In addition, CSPI is working to develop dedicated peering services using technologies such as FastConnect public peering. FastConnect can provide brick-and-mortar access, where a customer on-premises instance can access the FastConnect connection without going through a public network such as the Internet. CSPI can also provide dedicated private access using FastConnect private peering. In this case, customer on-premises instances with private IP addresses can access customer VCN workloads using a FastConnect connection. FastConnect can connect to a customer's on-premises network using the public Internet. FastConnect is a network connection used instead to connect your network to CSPI and its services. FastConnect offers higher bandwidth options compared to internet-based connections. It provides an easy, flexible and economical way to create dedicated, private connections with the flexibility and reliability of a consistent networking experience.

[0094] FIG. 1 and the accompanying description above describe various virtualized elements in an exemplary virtual network. As mentioned above, a virtual network is built on an underlying physical network or substrate network. FIG. 2 is a simplified architecture diagram illustrating physical elements in a physical network in CSPI 200 that provides the foundation for a virtual network, according to a particular embodiment. As shown, CSPI 200 provides a distributed environment including elements and resources (e.g., compute resources, memory resources, and networking resources) provided by a cloud service provider (CSP). These elements and resources are used to provide cloud services (e.g., IaaS services) to subscribing customers, i.e., customers who subscribe to one or more services provided by the CSP. Based on the service to which the customer subscribes, CSPI 200 provides some resources (e.g., compute resources, memory resources, and networking resources) to the customer. The customer can then build their own cloud-based (i.e., CSPI-hosted) customizable private virtual network using the physical compute resources, memory resources, and networking resources provided by CSPI 200. As mentioned above, these customer networks are called virtual cloud networks (VCNs). Customers can deploy one or more customer resources, such as compute instances, into these Customer VCNs. The compute instances may be virtual machines, bare metal instances, etc. CSPI 200 provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available hosted environment.

[0095] In the exemplary embodiment shown in FIG. 2, the physical elements of CSPI 200 include one or more physical host machines or servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), a physical network (e.g., 218), and switches within physical network 218. The physical host machines or servers can host and execute various compute instances participating in one or more subnets of a VCN. The compute instances may include virtual machine instances and bare metal instances. For example, the various compute instances shown in FIG. 1 may be hosted by the physical host machines shown in FIG. 2. The virtual machine compute instances in a VCN may be executed by one host machine or may be executed by multiple different host machines. Also, the physical host machines may host virtual host machines, container-based hosts or functions, etc. The VICs and VCN VRs shown in FIG. 1 may be executed by the FTVDs shown in FIG. 2. The gateway shown in FIG. 1 may be implemented by the host machine and / or NVD shown in FIG.

[0096] A host machine or server may run a hypervisor (also called a virtual machine monitor or VMM) that creates and enables a virtualized environment on the host machine. Virtualization or a virtualized environment facilitates cloud-based computing. One or more compute instances may be created, executed, and managed on the host machine by the hypervisor on the host machine. The hypervisor on the host machine manages the physical compute resources of the host machine (e.g., compute resources, memory resources, and networking resources can be shared among various computational instances running on the host machine.

[0097] For example, as shown in FIG. 2, host machines 202 and 208 run hypervisors 260 and 266, respectively. These hypervisors may be implemented using software, firmware, hardware, or a combination thereof. Typically, a hypervisor is a process or software layer that resides in the host machine's Operating System (OS), which runs on the host machine's hardware processor. The hypervisor provides a virtualization environment that allows the host machine's physical computing resources (e.g., processing resources such as processors / cores, memory resources, networking resources, etc.) to be shared among various virtual machine computing instances executed by the host machine. For example, in FIG. 2, hypervisor 260 resides in the OS of host machine 202 and allows the host machine's computing resources (e.g., processing resources, memory resources, and networking resources) to be shared among computing instances (e.g., virtual machines) executed by host machine 202. A virtual machine can have its own OS (called a guest OS). This guest OS may be the same as or different from the host machine's OS. The OS of a virtual machine executed by a host machine may be the same as or different from the OS of other virtual machines executed by the same host machine. Thus, the hypervisor can run multiple OSs in parallel while sharing the same computing resources of the host machine. The host machines shown in FIG. 2 may have the same type of hypervisor or different types of hypervisors.

[0098] A compute instance may be a virtual machine instance or a bare metal instance. In Figure 2, compute instance 268 on host machine 202 and compute instance 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0099] In certain examples, an entire host machine may be provided to a single customer, and one or more compute instances (either virtual machines or bare metal instances) hosted by that host machine may all belong to the same customer. In other examples, a host machine may be shared among multiple customers (i.e., multiple tenants). In such a multi-tenant scenario, a host machine may host virtual machine compute instances that belong to different customers. These compute instances may be members of different VCNs of different customers. In certain embodiments, bare metal compute instances are hosted by bare metal servers that do not have a hypervisor. When bare metal compute instances are provided, a single customer or tenant maintains control of the physical CPU, memory, and network interfaces of the host machine that hosts the bare metal instance, and the host machine is not shared with other customers or tenants.

[0100] As previously discussed, each compute instance that is part of a VCN is associated with a VNIC that enables the compute instance to be a member of a subnet of the VCN. The VNIC associated with a compute instance facilitates communication of packets or frames to and from the compute instance. The VNIC is associated with the compute instance when the compute instance is created. In certain embodiments, for a compute instance executed by a host machine, the VNIC associated with the compute instance is executed by an NVD connected to the host machine. For example, in FIG. 2, host machine 202 executes virtual machine compute instance 268 associated with VNIC 276, which is executed by an NVD 210 connected to host machine 202. In another embodiment, the VNIC associated with a compute instance is executed by an NVD 210 connected to host machine 202. As an example, bare metal instance 272 hosted by host machine 206 is associated with VNIC 280 executed by NVD 212 connected to host machine 206. As yet another example, VNIC 284 is associated with compute instance 274 executed by host machine 208, VNIC 284 executed by NVD 212 connected to host machine 208.

[0101] For a compute instance hosted by a host machine, the NVD connected to that host machine executes a VCN VR corresponding to the VCN of which the compute instance is a member. For example, in the embodiment shown in Figure 2, NVD 210 executes VCN VR 277 corresponding to the VCN of which compute instance 268 is a member. NVD 212 may also execute one or more VCN VRs 283 corresponding to the VCNs corresponding to the compute instances hosted by host machines 206 and 208.

[0102] A host machine may include one or more network interface cards (NICs) for connecting the host machine to other devices. The NICs on a host machine may provide one or more ports (or interfaces) for communicatively connecting the host machine to another device. For example, the host machine may be connected to the NVD using one or more ports (or interfaces) provided on the host machine and the NVD. The host machine may also be connected to other devices, such as other host machines.

[0103] 2, host machine 202 is connected to NVD 210 using link 220 extending between port 234 provided by NIC 232 of host machine 202 and port 236 of NVD 210. Host machine 206 is connected to NVD 212 using link 224 extending between port 246 provided by NIC 244 of host machine 206 and port 248 of NVD 212. Host machine 208 is connected to NVD 212 using link 226 extending between port 252 provided by NIC 250 of host machine 208 and port 254 of NVD 212.

[0104] Similarly, the NVDs are connected via communication links to top-of-rack (TOR) switches that are connected to a physical network 218 (also referred to as a switch fabric). In certain embodiments, the links between the host machines and the NVDs and between the NVDs and the TOR switches are Ethernet links. For example, in FIG. 2, NVDs 210 and 212 are connected to TOR switches 214 and 216, respectively, via links 228 and 230. In certain embodiments, links 220, 224, 226, 228, and 230 are Ethernet links. The collection of host machines and NVDs connected to a TOR may be referred to as a rack.

[0105] The physical network 218 provides a communications fabric that allows the TOR switches to communicate with each other. The physical network 218 may be a multi-tier network. In a particular implementation, the physical network 218 is a multi-tier Clos network of switches, with the TOR switches 214 and 216 representing leaf-level nodes of the multi-tier and multi-node physical switching network 218. Different Clos network configurations are possible, including, but not limited to, 2-tier networks, 3-tier networks, 4-tier networks, 5-tier networks, and generally "n"-tier networks. An example of a Clos network is shown in FIG. 5 and described below.

[0106] Between a host machine and N VDs, there are one-to-one configurations, many-to-one configurations, and one-to-many configurations. A variety of different connection configurations are possible. In a one-to-one implementation, each host machine is connected to its own separate NVD. For example, in FIG. 2, host machine 202 is connected to NVD 210 via NIC 232 of host machine 202. In a many-to-one configuration, multiple host machines are connected to one NVD. For example, in FIG. 2, host machines 206 and 208 are connected to the same NVD 212 via NICs 244 and 250, respectively.

[0107] In a one-to-many configuration, one host machine is connected to multiple NVDs. Figure 3 shows an example in CSPI 300 where a host machine is connected to multiple NVDs. As shown in Figure 3, a host machine 302 includes a network interface card (NIC) 304 that includes multiple ports 306 and 30S. The host machine 300 is connected to a first NVD 310 via port 306 and link 320, and to a second NVD 312 via port 308 and link 322. The ports 306 and 308 may be Ethernet ports, and the links 320 and 322 between the host machine 302 and the NVDs 310 and 312 may be Ethernet links. The NVD 310 is connected to a first TOR switch 314, and the NVD 312 is connected to a second TOR switch 316. The links between the NVDs 310 and 312 and the TOR switches 314 and 316 may be Ethernet links. TOR switches 314 and 316 represent Tier-0 switching devices within a multi-tier physical network 318 .

[0108] 3 provides two separate physical network paths from the physical switch network 318 to the host machine 302: a first path from the TOR switch 314 to the host machine 302 via the NVD 310, and a second path from the TOR switch 316 to the host machine 302 via the NVD 312. The separate paths provide enhanced availability (called high availability) of the host machine 302. If there is a problem with one of the paths (e.g., a link on one of the paths fails) or if there is a problem with a device (e.g., a particular NVD is not functioning), the other path can be used for communication to and from the host machine 302.

[0109] In the configuration shown in Figure 3, the host machine is connected to two different NVDs using two different ports provided by the host machine's NIC. In other embodiments, the host machine may include multiple NICs allowing the host machine to connect to multiple NVDs.

[0110] Referring again to FIG. 2, an NVD is a physical device or element that performs one or more network virtualization functions and / or storage virtualization functions. An NVD may include one or more processing units (e.g., CPU, Network Processing Unit (NPU), FPGA, packet processing pipeline), memory including cache, and It may be any device with a port. Various virtualization functions may be performed by software / firmware executed by one or more processing units of the NVD.

[0111] The NVD may be implemented in a variety of different ways. For example, in a particular embodiment, the NVD is implemented as an interface card with an embedded processor, called a smart NIC or intelligent NIC. The smart NIC is a separate device from the NIC on the host machine. In FIG. 2, the NVD 210 may be implemented as a smart NIC connected to the host machine 202, and the NVD 212 may be implemented as a smart NIC connected to the host machines 206 and 208.

[0112] However, smart NICs are only one example of an NVD implementation. Various other implementations are possible. For example, in some other implementations, the NVD or one or more functions performed by the NVD may be incorporated into or performed by one or more host machines, one or more TOR switches, and other elements of the CSPI200. For example, the NVD may be integrated into a host machine. In this case, the functions performed by the NVD are performed by the host machine. As another example, the NVD may be part of a TOR switch, or the TOR switch may be configured to perform the functions performed by the NVD that allow the TOR switch to perform various complex packet transformations used for public clouds. A TOR that performs the functions of an NVD may be referred to as a smart TOR. A TOR that provides virtual machine (VM) instances rather than bare metal (BM) instances to customers may be configured to provide virtual machines (VMs) to customers. In yet another implementation provided, the functionality provided by the NVD may be implemented within a hypervisor of a host machine, hi some other implementations, some of the functionality of the NVD may be offloaded to a centralized service running on a set of host machines.

[0113] In certain embodiments, such as when implemented as a smart NIC as shown in FIG. 2, an NVD may comprise multiple physical ports that allow the NVD to be connected to one or more host machines and one or more TOR switches. Ports on an NVD can be classified as host-facing ports (also called "south ports") or network-facing or TOR-facing ports (also called "north ports"). A host-facing port of an NVD is a port used to connect the NVD to a host machine. Examples of host-facing ports in FIG. 2 include port 236 of NVD 210, and ports 248 and 254 of NVD 212. A network-facing port of an NVD is a port used to connect the NVD to a TOR switch. Examples of network-facing ports in FIG. 2 include port 256 of NVD 210, and port 258 of NVD 212. As shown in FIG. 2, the NVD 210 is connected to the TOR switch 214 via a link 228 that extends from port 256 of NVD 210 to the TOR switch 214. Similarly, the NVD 212 is connected to the TOR switch 216 via a link 230 that extends from a port 258 of the NVD 212 to the TOR switch 216 .

[0114] The NVD can receive packets and frames (e.g., packets and frames generated by compute instances hosted by the host machine) from the host machine via the host-facing port, perform any necessary packet processing, and then forward the packets and frames to the TOR switch via the NVD's network-facing port. The NVD can receive packets and frames from the TOR switch via the NVD's network-facing port, and perform any necessary packet processing, and then forward the packets and frames to the host machine via the NVD's host-facing port.

[0115] In certain embodiments, multiple ports and associated links may be provided between the NVD and the TOR switch. These ports and links may be aggregated to form a link aggregator group (referred to as a LAG) of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and the TOR switch) to be treated as a single logical link. All physical links in a given LAG can operate at the same speed and in full-duplex mode. LAGs help increase the bandwidth and reliability of the connection between the two endpoints. If one of the physical links in the LAG fails, traffic is dynamically and transparently reassigned to another physical link in the LAG. The aggregated physical links provide higher bandwidth than individual links. Multiple ports associated with a LAG are treated as a single logical port. Traffic can be load balanced across multiple physical links in a LAG. One or more LAGs can be configured between two endpoints. The two endpoints can be, for example, between an NVD and a TOR switch, or between a host machine and an NVD.

[0116] The NVD implements or performs network virtualization functions. These functions are performed by software / firmware executed by the NVD. Examples of network virtualization functions include, but are not limited to, packet encapsulation and decapsulation functions, functions for creating VCN networks, functions for implementing network policies such as VCN security list (firewall) functions, functions for facilitating routing and forwarding of packets to and from compute instances in the VCN, etc. In a particular embodiment, upon receiving a packet, the NVD is configured to execute a packet processing pipeline that processes the packet and determines how to forward or route the packet. As part of this packet processing pipeline, the NVD provides the execution of one or more virtual functions associated with the overlay network, such as executing VNICs associated with compute instances in the VCN, executing Virtual Routers (VRs) associated with the VCN, encapsulating and decapsulating packets to facilitate forwarding or routing within the virtual network, executing specific gateways (e.g., local peering gateways), implementing security lists, network security groups, network address translation (NAT) functions (e.g., public IP to private IP translation per host), throttling functions, and other functions.

[0117] In some embodiments, the packet processing data path in the NVD may include multiple packet pipelines. Each packet pipeline is composed of a series of packet transformation stages. In some implementations, upon receiving a packet, the packet is parsed and sorted into a single pipeline. The packet is then processed stage by stage in a linear fashion until it is either discarded or sent out through an interface of the NVD. These stages provide packet processing building blocks of basic functions (e.g., validate headers, perform throttling, insert new layer 2 headers, perform L4 firewalling, VCN encapsulation / decapsulation), such that new pipelines can be constructed by assembling existing stages, and new functionality can be added by creating and inserting new stages into existing pipelines.

[0118] The NVD can perform both control plane and data plane functions corresponding to the control and data planes of the VCN. Examples of the VCN control plane are shown in Figures 11, 12, 13, and 14 (see reference numbers 1116, 1216, 1316, and 1416) and described below. Examples of the VCN data plane are shown in Figures 11, 12, 13, and 14 (see reference numbers 1118, 1218, 1318, and 1418) and described below. The control plane functions include functions used to configure the network (e.g., setting routes and route tables, configuring VNICs) to control how data is forwarded. In certain embodiments, a VCN control plane is provided that centrally computes and exposes all overlay-to-substrate mappings to the NVD and virtual network edge devices (e.g., various gateways such as DRGs, SGWs, IGWs, etc.). The same mechanism can also be used to expose firewall rules. In certain embodiments, the NVD retrieves only mappings that are relevant to the NVD. The data plane functions include those that do the actual routing / forwarding of packets based on the configurations set using the control plane. The VCN data plane is implemented by encapsulating customer network packets before they traverse the backbone network. The encapsulation / decapsulation functions are implemented in the NVD. In certain embodiments, the NVD manages all network traffic entering and leaving the host machines. It is configured to intercept packets and perform network virtualization functions.

[0119] As described above, the NVD performs various virtualization functions including VNICs and VCN VRs. The NVD may perform VNICs associated with compute instances hosted by one or more host machines connected to the VNICs. For example, as shown in FIG. 2, the NVD 210 performs functions of a VNIC 276 associated with a compute instance 268 hosted by a host machine 202 connected to the NVD 210. As another example, the NVD 212 performs VNICs 280 associated with a bare metal compute instance 272 hosted by a host machine 206 and VNICs 284 associated with a compute instance 274 hosted by a host machine 208. The host machines may host compute instances belonging to different VCNs that belong to different customers. The NVDs connected to the host machines may perform VNICs (i.e., perform functions related to the VNICs) corresponding to the compute instances.

[0120] NVDs also run VCN virtual routers corresponding to the VCNs of the compute instances. For example, in the embodiment shown in FIG. 2, NVD 210 runs VCN VR 277 corresponding to the VCN to which compute instance 268 belongs. NVD 212 runs one or more VCN VRs 283 corresponding to one or more VCNs to which compute instances hosted on host machines 206 and 208 belong. In certain embodiments, a VCN VR corresponding to a VCN is run by all NVDs connected to a host machine that hosts at least one compute instance that belongs to that VCN. If a host machine hosts compute instances that belong to different VCNs, the NVDs connected to that host machine may run VCN VRs corresponding to the different VCNs.

[0121] In addition to the VNICs and VCN VRs, the NVD may include one or more hardware elements that run various software (e.g., daemons) and facilitate various network virtualization functions performed by the NVD. For simplicity, these various elements are grouped as "packet processing elements" as shown in FIG. 2. For example, NVD 210 includes packet processing element 286, and NVD 212 includes packet processing element 288. For example, the packet processing element of the NVD may include a packet processor configured to monitor all packets received and communicated using the NVD by interacting with the ports and hardware working interfaces of the NVD and store network information. The network information may include, for example, network flow information for identifying different network flows processed by the NVD and information for each flow (e.g., statistics for each flow). In certain embodiments, the network flow information may be stored on a per-VNIC basis. As another example, the packet processing element may include a replication agent configured to replicate information stored by the NVD to one or more different replication target stores. Yet another example As such, the packet processing element may include a logging agent configured to perform logging functions of the NVD. The NVD may include software for monitoring the performance and health of the NVD and, in some cases, for monitoring the status and health of other elements connected to the NVD.

[0122] FIG. 1 illustrates elements of an exemplary virtual or overlay network, including a VCN, a subnet in the VCN, a compute instance deployed on the subnet, a VNIC associated with the compute instance, a VR in the VCN, and a set of gateways configured for the VCN. The overlay elements illustrated in FIG. 1 may be executed or hosted by one or more of the physical elements illustrated in FIG. 2. For example, a compute instance in a VCN may be executed or hosted by one or more host machines illustrated in FIG. 2. For a compute instance hosted by a host machine, The VNICs associated with a VCN are typically executed by an NVD connected to its host machine (i.e., the VNIC functionality is provided by an NVD connected to that host machine). The VCN VR functionality of a VCN is executed by all NVDs connected to the host machines that host or run the compute instances that are part of that VCN. Gateways associated with a VCN may be executed by one or more different types of NVDs. For example, some gateways may be executed by smart NICs and other gateways may be executed by one or more host machines or other implementations of NVDs.

[0123] As described above, a compute instance in a customer VCN can communicate with a variety of different endpoints. These endpoints may be in the same subnet as the source compute instance, in a different subnet but in the same VCN as the source compute instance, or may include endpoints outside the VCN of the source compute instance. These communications are facilitated using VNICs associated with the compute instance, VCN VRs, and gateways associated with the VCN.

[0124] Communication between two compute instances on the same subnet in a VCN is facilitated using VNICs associated with the source compute instance and the destination compute instance. The source compute instance and the destination compute instance may be hosted by the same host machine or different host machines. A packet originating from a source compute instance may be forwarded from the host machine hosting the source compute instance to an NVD connected to that host machine. In the NVD, the packet is processed using a packet processing pipeline, which may include execution of a VNIC associated with the source compute instance. Because the destination endpoint of the packet is in the same subnet, execution of the VNIC associated with the source compute instance forwards the packet to an NVD running a VNIC associated with the destination compute instance, which processes the packet and forwards it to the destination compute instance. The VNICs associated with the source compute instance and the destination compute instance may be executed on the same NVD (e.g., if both the source compute instance and the destination compute instance are hosted by the same host machine) or may be executed on different NVDs (e.g., if the source compute instance and the destination compute instance are hosted by different host machines connected to different NVDs). The VNIC can use the routing / forwarding tables stored by the NVD to determine the next hop for a packet.

[0125] When communicating a packet from a compute instance in a subnet to an endpoint in a different subnet in the same VCN, the packet originating from the source compute instance is communicated from the host machine hosting the source compute instance to the NVD connected to that host machine. In the NVD, the packet is processed using a packet processing pipeline that may include running one or more VNICs and VRs associated with the VCN. For example, the NVD executes or invokes a function corresponding to a VNIC associated with the source compute instance (also referred to as executing a VNIC) as part of the packet processing pipeline. The function executed by the VNIC may include searching for a VLAN tag on the packet. Because the destination of the packet is outside the subnet, a VCN VR function is invoked and executed by the NVD. The VCN VR then routes the packet to the NVD that executes the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards the packet to the destination compute instance. The VNICs associated with the source compute instance and the destination compute instance may be executed on the same NVD (e.g., when both the source compute instance and the destination compute instance are hosted by the same host machine). , may be executed on different NVDs (e.g., when the source and destination computation instances are hosted by different host machines connected to different NVDs).

[0126] If the destination of the packet is outside the VCN of the source compute instance, the packet originating from the source compute instance is communicated from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD runs the VNIC associated with the source compute instance. Because the destination endpoint of the packet is outside the VCN, the packet is processed by the VCN VR for that VCN. The NVD invokes a VCN VR function, which may result in the packet being forwarded to an NVD running an appropriate gateway associated with the VCN. For example, if the destination is an endpoint in a customer on-premises network, the packet may be forwarded by the VCN VR to an NVD running a DRG gateway configured for the VCN. The VCN VR may run on the same NVD as the NVD running the VNIC associated with the source compute instance, or may be run by a different NVD. The gateway may be run by an NVD that is a smart NIC, a host machine, or another NVD implementation. The packet is then processed by the gateway and forwarded to a next hop to facilitate communication of the packet to the intended destination endpoint. 2, a packet originating from compute instance 268 may be communicated from host machine 202 to NVD 210 over link 220 (using NIC 232). VNIC 276 on NVD 210 is called out because it is the VNIC associated with the source compute instance 268. VNIC 276 is configured to inspect encapsulation information in the packet, determine a next hop for forwarding the packet in order to facilitate communication of the packet to an intended destination endpoint, and forward the packet to the determined next hop.

[0127] Compute instances deployed on a VCN can communicate with a variety of different endpoints. These endpoints may include endpoints hosted by CSPI200 and endpoints external to CSPI200. Endpoints hosted by CSPI200 may include instances in the same VCN or other VCNs (which may be customer VCNs or VCNs not belonging to the customer). Communications between endpoints hosted by CSPI200 may be performed over physical network 218. Compute instances may also communicate with endpoints not hosted by or external to CSPI200. Examples of these endpoints include endpoints in a customer on-premise network or datacenter, or public endpoints accessible over a public network such as the Internet. Communications with endpoints external to CSPI200 may be performed over a public network (e.g., the Internet) (not shown in FIG. 2) or a private network (not shown in FIG. 2) using a variety of communication protocols.

[0128] The architecture of CSPI 200 shown in FIG. 2 is merely an example and is not intended to be limiting. Variations, substitutions, and modifications are possible in alternative embodiments. For example, in some implementations, CSPI 200 may have more or fewer systems or elements than those shown in FIG. 2, may combine two or more systems, or may have a different system configuration or arrangement. The systems, subsystems, and other elements shown in FIG. 2 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device).

[0129] FIG. 4 illustrates a connection between a host machine and an NVD to provide I / O virtualization to support multitenancy, according to certain embodiments. As shown in FIG. 4, a host machine 402 runs a hypervisor 404 that provides a virtualization environment. The host machine 402 runs two virtual machine instances: VM1 406 that belongs to customer / tenant #1 and VM2 408 that belongs to customer / tenant #2. The host machine 402 includes a physical NIC 410 that is connected to an NVD 412 via link 414. Each of the compute instances is connected to a VNIC that is executed by the NVD 412. In the embodiment of FIG. 4, VM1 406 is connected to VNIC-VM1 420, and VM2 408 is connected to VNIC-VM2 422.

[0130] 4, NIC 410 includes two logical NICs: logical NIC A 416 and logical NIC B 418. Each virtual machine is connected to and configured to operate with its own logical NIC. For example, VM1 406 is connected to logical NIC A 416, and VM2 408 is connected to logical NIC B 418. The logical NICs allow each tenant's virtual machine to believe it owns its own host machine and NIC, even though host machine 402 consists of only one physical NIC 410 shared by multiple tenants.

[0131] In a particular embodiment, each logical NIC is assigned its own VLAN ID. Thus, logical NIC A 416 for tenant #1 is assigned a particular VLAN ID, and logical NIC B 418 for tenant #2 is assigned another VLAN ID. When a packet is communicated from VM1 406, the hypervisor attaches a tag assigned to tenant #1 to the packet before communicating the packet from host machine 402 to NVD 412 over link 414. Similarly, when a packet is communicated from VM2 408, the hypervisor attaches a tag assigned to tenant #2 to the packet before communicating the packet from host machine 402 to NVD 412 over link 414. Thus, a packet 424 communicated from host machine 402 to NVD 412 has an associated tag 426 that identifies the particular tenant and associated VM. When a packet 424 is received on the NVD from host machine 402, a tag 426 associated with the packet is used to determine whether the packet should be processed by VNIC-VM1 420 or VNIC-VM2 422. The packet is then processed by the corresponding VNIC. The configuration shown in Figure 4 allows each tenant's compute instance to believe that it owns its own host machine and NIC. The configuration shown in Figure 4 provides I / O virtualization to support multi-tenancy.

[0132] FIG. 5 is a schematic block diagram illustrating a physical network 500 according to a particular embodiment. The embodiment illustrated in FIG. 5 is constructed as a Clos network. A Clos network is a particular type of network topology designed to provide connection redundancy while maintaining high bisection bandwidth and maximum resource utilization. A Clos network is a kind of non-blocking, multi-stage or multi-layer switching network, where the number of stages or layers may be 2, 3, 4, 5, etc. The embodiment illustrated in FIG. 5 is a three-layer network including layers 1, 2, and 3. A TOR switch 504 represents a layer-0 switch of the Clos network. One or more NVDs are connected to the TOR switch. The layer-0 switch is also called an edge device of the physical network. The layer-0 switch is connected to a layer-1 switch, also called a leaf switch. In the embodiment illustrated in FIG. 5, "m" layer-0 TOR switches are connected to "r" layer-1 switches (the integers m and r may have the same value or different values) to form a pod. Each layer-0 switch in the pod is connected to all layer-1 switches in the pod. The pods are interconnected, but the switches between the pods are not. In a particular implementation, the two pods are referred to as a block. Each block is served by or connected to "q" layer-2 switches (also referred to as spine switches). The physical network topology may include multiple blocks. Similarly, the layer-2 switches are connected to "p" layer-3 switches (also referred to as super-spine switches), where the integers p and q may have the same or different values. Communication of packets through the physical network 500 is typically performed using one or more layer-3 communication protocols. Typically, all layers of the physical network except the TOR layer are redundant (e.g., p-way redundant, q-way redundant, or r-way redundant), thus enabling high availability. The physical network can be scaled by specifying policies on the pods and blocks to control the mutual visibility of the switches of the physical network.

[0133] A feature of Clos networks is that the maximum hop count from one tier-0 switch to another tier-0 switch (or from an NVD connected to a tier-0 switch to another NVD connected to a tier-0 switch) is constant. For example, in a three-tier Clos network, a maximum of seven hops are required for a packet to reach from one NVD to another NVD. In this case, the source NVD and the target NVD are connected to the leaf tier of the Clos network. Similarly, in a four-tier Clos network, a maximum of nine hops are required for a packet to reach from one NVD to another NVD. In this case, the source NVD and the target NVD are connected to the leaf tier of the Clos network. Thus, the Clos network architecture keeps the overall network latency constant, which is important for intra- and inter-datacenter communications. Clos topologies are horizontally scalable and cost-effective. The bandwidth / throughput capacity of the network can be easily increased by adding more switches (e.g., more leaf switches and spine switches) at each tier and by increasing the number of links between switches at adjacent tiers.

[0134] In certain embodiments, each resource in the CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource's information. This identifier can be used to manage the resource, for example, via a console or API. An exemplary syntax for a CID is as follows:

[0135] ocid1.<RESOURCE TYPE> . <realm>.[REGION].[FUTURE USE].<UNIQUE ID> It is. During the ceremony, "ocid1" is a string indicating the version of the CID.

[0136] "RESOURCE TYPE" is the type of resource (e.g., instance, volume, VCN , subnet, user, or group).

[0137] "REALM" represents the region in which the resource resides. An example value is "c1" for commercial "c1" represents the government cloud region, "c2" represents the government cloud region, or "c3" represents the federal government cloud region. Each region can have its own domain name.

[0138] "REGION" represents the region that the resource belongs to. If a region does not apply to the resource, this part may be blank.

[0139] "FUTURE USE" indicates that the item is reserved for future use. "UNIQUE ID" is the unique ID part. This format is used for resources or servers. This may vary depending on the type of bis.

[0140] RDMA / RoCE Technology FIG. 6 illustrates an example of a distributed multi-tenant cloud environment 600 that may be hosted by a Cloud Service Provide Infrastructure (CSPI). As shown in FIG. 6, multiple host machines (e.g., 602 and 622) are communicatively connected via a physical network or switch fabric 640 that includes multiple switches or, more broadly, networking devices. In a particular implementation, the switch fabric 640 may be an n-tier Clos network as shown in FIG. 5 and described above, where the design may optimize performance through the placement of the Clos fabric and physical switches 642, 644, and 646. Depending on the implementation, the value of "n" may be 1, 2, 3, etc. However, it should be noted that each additional tier is expected to increase the latency of packet forwarding across the fabric, which may not be desirable in certain applications. Top-of-rack (TOR) switches 642 and 644 represent leaf or tier-0 devices in the switch fabric 640. Depending on the value of "n," the switch fabric 640 may include one or more spine switches, super-spine switches, etc. In FIG. 6, the switches between TOR switch 642 and TOR switch 644 (e.g., the tier-1, tier-2, and tier-3 switches in FIG. 5) are represented by intermediate switches 646. Intermediate switches 646 may generally include one or more switches or networking devices. Switch fabric 640 may also be implemented to include switch substrate IP addresses that are not reachable by customer compute instances (e.g., for administrative purposes). It may be desirable to implement Spanning Tree Protocol (STP) on the TOR switches of switch fabric 640 (e.g., to avoid loops that may occur due to bugs, etc.).In some configurations, each TOR switch on the switch fabric 640 is dedicated to a service (e.g., database cloud services, HPC cloud services, GPU cloud services, etc.), and traffic of different services is only mixed at a higher tier (e.g., intermediate switch 646).

[0141] Host machines 602 and 622 may be referred to as multi-tenant host machines because they may host compute instances for multiple customers or tenants. For example, as shown in FIG. 6, host machine 602 hosts compute instance A-1 604 for customer A and compute instance B-1 for customer B. Host machine 604 hosts compute instance A-2 624 for customer A and compute instance B-2 626 for customer B. In certain embodiments, compute instances 604, 606, 624, and 626 are virtual machines. In this manner, virtual machines belonging to different customers may be hosted on the same host machine. However, each of these compute instances experiences ownership of the entire host machine. In certain embodiments, a customer's compute instances may also include bare metal hosts. The teachings of the present disclosure may be applied to compute instances in the form of virtual machines or bare metal hosts. For ease of illustration, the example of FIG. 1 shows only two multi-tenant host machines 602 and 622, but this is not intended to be limiting. The principles disclosed by this disclosure are not limited to any particular number of multi-tenant hosts, and particular examples including a greater number of multi-tenant hosts and / or including one or more compute instances that are bare metal hosts (e.g., single-tenant hosts) are also possible.

[0142] In the multi-tenant environment illustrated in FIG. 6, it is desirable to properly isolate traffic originating from and destined for different customers' compute instances from one another. In certain embodiments, this traffic isolation is achieved by configuring separate network domains for different customers. For example, Customer A's compute instances may be assigned to a particular network domain that is separate and distinct from the network domain to which customer B's compute instances are assigned. In a particular implementation, these network domains may be configured in the form of virtual LANs (VLANs), with each VLAN identified by a unique VLAN identifier. For example, in FIG. 6, customer A's compute instances A-1 604 and A-2 Customer B’s compute instances B-1 606 and B-2 626 are assigned to VLAN 1002, with “1002” representing the unique VLAN identifier. For ease of illustration, the example in FIG. 1 shows only two members of each VLAN 1001 and 1002, but this is not intended to be limiting. The principles disclosed by this disclosure are not limited to any particular number of VLAN members. Also, there may be multiple VLANs rather than just the two shown in Figure 6. The IEEE 802.1Q standard, for example, supports the identification of up to 4096 different VLANs.

[0143] In certain implementations, computing instances belonging to the same customer may have different quality of service expectations. For example, a customer may have computing instances belonging to two or more different services (or applications or departments), such as a first set of one or more computing instances corresponding to service A (e.g., a simulation service) and a second set of computing instances corresponding to service B (e.g., a backup service), where the two services have very different quality of service expectations (e.g., in terms of latency, packet loss, bandwidth requirements, etc.). For example, service A may be more latency sensitive than service B, and as a result, the customer may want traffic associated with service A to be given a different traffic class (e.g., higher priority) than traffic associated with service B. In such a situation, different computing instances belonging to the same customer will have different quality of service requirements.

[0144] Compute instances on the same VLAN or on peer VLANs (e.g., VLANs on different layer 2 domains that belong to the same tenant but may have different VLAN IDs) may want to communicate with each other. In certain implementations, compute instances on a VLAN (or peer VLANs) may exchange data using RDMA and RoCE protocols. In such implementations, the host machines that host these compute instances are equipped with special hardware and software that enable RDMA and RoCE-based communication. For example, as shown in FIG. 6, host machines 602 and 622 include RDMA network interface cards (NICs) (e.g., RoCE NICs) 608 and 628, respectively, that enable the compute instances hosted by host machine 602 to exchange data with the compute instances hosted by host machine 622 on the same VLAN (or peer VLAN) using RDMA and RoCE protocols. The RoCE NIC may be implemented, for example, as a hardware assembly (e.g., an interface card) installed in a host machine (e.g., RoCE NIC 608 installed in multitenant host machine 602 and RoCE NIC 628 installed in host machine 622). Compute instances A-1 604 and A-2 624 belonging to the same VLAN 1001 can use RoCE NICs 608 and 628 in host machines 602 and 622, respectively, to exchange data using RDMA and RoCE protocols. In certain implementations, these RoCE NICs are separate from the NICs shown in Figures 2 and 3 and described above. In other implementations, the NICs shown in Figures 2 and 3 may be configured to operate as RoCE NICs as well.

[0145] As shown in the example shown in FIG. 6, a RoCE NIC includes a RoCE engine and implements virtual functions (e.g., SR-IOV functions), each of which may be configured for a different corresponding one of the virtual machines supported by the host machine. In this example, the RoCE NIC is implemented to support multitenancy through a technology called SR-IOV (Single Root Input / Output Virtualization), which allows a physical device to appear as multiple different virtual instances (also called "virtual functions" or VFs) on a Peripheral Component Interconnect Express (PCI Express or PCIe) bus, with each VF assigned to a respective VM and having resources separate from those of the other VFs. For example, in FIG. 6, a RoCE NIC 608 on a host machine 602 includes a RoCE engine 610, a virtual function VF-A-1 612 for virtual machine compute instance A-1 604, and a virtual function VF-B-1 614 for virtual machine compute instance B-1 606. The RoCE NIC 628 on the host machine 622 includes a RoCE engine 630, a virtual function VF-A-2 632 for virtual machine compute instance A-2 624, and a virtual function VF-B-2 634 for virtual machine compute instance B-2 626. For ease of explanation, the example in FIG. 6 shows only two virtual functions per host machine, but this is in no way intended to be limiting, and the principles described in this disclosure are not limited to any particular number of virtual functions. In one example, SR-IOV may support up to 16 VFs per physical NIC port, and the host machine may also have multiple RDMA NICs (e.g., multiple RoCE NICs).

[0146] In particular embodiments, the virtual functions of the RoCE NIC are programmed by a hypervisor on a host machine for a particular virtual machine compute instance and are configured to enforce that packets from the virtual machine communicated over a network, such as switch fabric 640, are tagged with a VLAN tag (e.g., an 802.1Q VLAN tag) corresponding to the VLAN to which the virtual machine belongs. In the example shown in Figure 6, virtual function VF-A-1 612 may be configured to add a VLAN tag indicating VLAN1001 (e.g., a VLAN tag having a VLAN ID with value 1001) to packets carrying data from virtual machine compute instance A-1 604, and virtual function VF-B-1 614 may be configured to add a VLAN tag indicating VLAN1002 (e.g., a VLAN tag having a VLAN ID with value 1002) to packets carrying data from virtual machine compute instance B-1 606. Similarly, virtual function VF-A-2 632 may be configured to add a VLAN tag indicating VLAN 1001 (e.g., a VLAN tag having a VLAN ID with value 1001) to packets carrying data from virtual machine compute instance A-2 624, and virtual function VF-B-2 634 may be configured to add a VLAN tag indicating VLAN 1002 (e.g., a VLAN tag having a VLAN ID with value 1002) to packets carrying data from virtual machine compute instance B-2 626. These VLAN tags may be used by downstream network elements to segregate or separate traffic belonging to different VLANs (e.g., in FIG. 6, traffic belonging to customer A's compute instances from traffic belonging to customer B's compute instances).

[0147] In a particular implementation, the virtual function assigned to each compute instance is configured (e.g., in cooperation with a RoCE engine on a RoCE NIC) to perform direct memory access (DMA) read operations from and DMA write operations to the memory space of the corresponding compute instance for RDMA data transfers. In the example of FIG. 6, virtual function VF-A-1 612 is configured to cooperate with RoCE engine 608 to perform direct memory access read and write operations to compute instance A-1 604 as part of an RDMA operation. Similarly, virtual function VF-B-1 614 is configured to cooperate with RoCE engine 608 to perform direct memory access read and write operations to compute instance A-1 604 as part of an RDMA operation. It is configured to perform direct memory access read and write operations to compute instance B-1 606.

[0148] The RoCE engine in the RoCE NIC is configured to facilitate sending RDMA / RoCE traffic from a host machine and receiving RDMA / RoCE traffic sent by another host machine. In a particular embodiment, the RoCE engine receives instructions (e.g., metadata) that identify an address range in the application memory of a compute instance, which represents a block of data to be transferred to the application memory of a target compute instance using RDMA and RoCE. For example, the RoCE engine 610 may receive information identifying an RDMA channel set up for data transfer and an address range that represents a block of data to be transferred from the application memory of compute instance A-1 604 (i.e., from A-1's application memory provided by host machine 602) to the application memory of compute instance A-2 624 on host machine 622 (i.e., to A-2's application memory provided by host machine 622) using RDMA. The RoCE engine is configured to access data from the application memory of a source compute instance, packetize the data (i.e., generate and assemble a layer 2 frame for the data) in an appropriate packet format that enables communication of the data to a target or destination compute instance, and then communicate the packets to a TOR switch (e.g., a leaf switch in a switch fabric used to forward the data to the destination compute instance). Thus, the RoCE engine is an offload engine, and the host machine's CPU or OS need not be involved in the data transfer. Such offloading reduces the latency associated with the data transfer.

[0149] For example, the RoCE engine 610 may be configured to append headers (e.g., UDP and IP headers) and VLAN tags (e.g., as enforced by virtual functions 612 and 614) to the data payload to create a VLAN-tagged RoCEv2 formatted packet and send the RoCEv2 packet over a wire (e.g., Ethernet cable) to a leaf switch (e.g., TOR switch 642) of the switch fabric 640. For traffic coming from the switch fabric to the RoCE engine, the RoCE engine 610 may be configured to receive RoCEv2 packets from the TOR switch 642, remove the UDP and IP headers, strip the VLAN tag, and forward each resulting frame (e.g., as the IB payload sent by the source host) to an SR-IOV virtual function that is mapped to the VLAN ID on which the packet was received. The virtual function may be configured to store the data payload of the packet in the memory space of the destination compute instance on the corresponding VLAN.

[0150] The layer 2 frames assembled by the RoCE NIC are then communicated using layer 3 routing protocols through multiple networking devices in the layer 3 switch fabric to the RoCE NIC of the host machine hosting the destination or target compute instance. For example, if data is transferred from compute instance A-1 604 in FIG. 6 to destination compute instance A-2 624 on host machine 622 using RDMA and RoCE, the path taken by the packet with the data payload is as follows: source compute instance A-1 604 on host machine 602 → RoCE NIC 608 on host machine 602 → TOR switch 642 → one or more intermediate switches 646 → TOR switch 644 → RoCE NIC 628 on host machine 622 → compute instance A-2 624 on host machine 622. As part of this communication, the TOR switch 642, which represents an ingress edge device of the switch fabric 640, routes the packets through the switch fabric 640 to communicate the packets. The TOR switch 642 is configured to convert Layer 2 frames received from the RoCE NIC into Layer 3 packets by encapsulating the packets in a wrapper (e.g., including one or more headers) corresponding to the Layer 3 tunneling protocol used for the TOR switch 642. A variety of different tunneling protocols may be used, such as VxLAN, NVGRE, STT, GENEVE, MPLS, etc. The Layer 3 packets then travel from the TOR switch 642 through one or more intermediate switches 646 to a TOR switch 644, which represents an egress edge device of the switch fabric 640. The TOR switch 644 is configured to decapsulate the packets and convert them into Layer 2 frames, which are then communicated to a RoCE NIC 628 on a host machine 622 hosting a destination or target compute instance A-2 624. The RoCE NIC 628 on the host machine 622 then forwards the data to the destination compute instance A-2 624. The packets may be forwarded to compute instance A-2 by writing the packet data to an application memory of compute instance A-2 624. Further details relating to the processing performed by various network elements to facilitate the transfer of data from one multitenant host machine to another multitenant host machine using RDMA and RoCE are provided below.

[0151] As another example, if compute instance B-1 606 wants to forward data to compute instance B-2 626, the path the packet takes is as follows: source compute instance B-1 606 on host machine 602 → RoCE NIC 608 on host machine 602 → TOR switch 642 → one or more intermediate switches 646 → TOR switch 644 → RoCE NIC 628 on host machine 622 → compute instance B-2 626 on host machine 622. In this way, the switch fabric 640 is shared by customers or tenants for communication of layer 2 RoCE traffic. The same switch fabric is used to communicate RoCE packets of different tenants. RoCE packets (and optionally, normal non-RoCE IP traffic) from different tenants flow in the same common network fabric. Segregation of traffic within this common network fabric is done using tags associated with the packets. Although each customer (e.g., a customer's compute instances on a VLAN or peer VLANs) experiences that it has a dedicated Layer 2 network for transporting RoCE traffic, in reality, the traffic is carried over a shared cloud-based Layer 3 switched fabric network. The host machines that generate RoCE traffic according to the customer's applications generate Layer 2 Ethernet frames (also called Layer 2 packets) instead of Layer 3 packets.

[0152] 7A, 7B, and 7C are schematic flow charts 700 illustrating a process for performing an RDMA data transfer from a source compute instance on a multi-tenant source host machine to a destination compute instance on a multi-tenant destination host machine over a shared layer 3 switch fabric using a layer 3 routing protocol, according to certain embodiments. The processes illustrated in FIGS. 7A-7C may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., memory device). The methods illustrated in FIGS. 7A-7C and described below are intended to be exemplary and non-limiting. Although FIGS. 7A-7C depict various processing steps occurring in a particular sequence or order, this depiction is not intended to be limiting. In certain alternative embodiments, the processes may be performed in some different order and / or some steps may be performed in parallel. In certain embodiments, such as the embodiment shown in FIG. 6, the process shown in FIGS. 7A-7C includes RoCE NICs 608 and 628, TOR switches 642 and 644 of switch fabric 640, and 7A-7C may be performed cooperatively by one or more intermediate switches 646 of the switch fabric 640. The methods shown in Figures 7A-7C and described below may be used with Layer 2 RDMA packets according to various versions of RoCE, such as RoCEv2 and other future versions, as well as other RDMA protocols that support VLAN tagging.

[0153] For purposes of illustrating the method, and as an example using the embodiment shown in FIG. 6, data may be transferred using RDMA and RoCE from compute instance A-1 604 hosted by host machine 602 to compute instance A-2 hosted by host machine 622. 624, where A-1 and A-2 belong to the same customer A and are on the same VLAN 1001. The compute instance (e.g., A-1 604) that originates the data to be transferred may be referred to as a source compute instance, and the host machine (e.g., host machine 602) that hosts the source compute instance may be referred to as a source host machine. The compute instance (e.g., A-2 624) to which the data is transferred may be referred to as a destination or target compute instance, and the host machine (e.g., host machine 622) that hosts the destination compute instance may be referred to as a destination or target host machine. The source and destination compute instances may be virtual machines or bare metal instances. The source and destination host machines may be in the same Ethernet domain or in different Ethernet domains.

[0154] At 702, a RoCE NIC on a source host machine hosting a source compute instance receives information (e.g., from a virtual function) identifying data to be transferred from the source compute instance to a destination compute instance using RDMA and RoCE. For example, the RoCE NIC may receive information identifying the source compute instance and a memory address range identifying a block of data to be transferred from the source compute instance to the destination compute instance. In the embodiment shown in FIG. 6, the RoCE NIC 608 may receive information that a block of data is to be transferred from compute instance A-1 604 hosted by host machine 602 to compute instance A-2 624 hosted by host machine 622.

[0155] At 704, the RoCE NIC accesses the data to be transferred from the application memory of the source compute instance on the source host machine to generate layer 2 802.1Q tagged RoCE packets for the data to be transferred, with VLAN information encoded into an 802.1Q tag attached to each packet and QoS information encoded into one or more headers of the packets. Each layer 2 802.1Q tagged RoCE packet may have one 802.1Q tag or may have two or more 802.1Q tags (e.g., 802.1ad or "Q-in-Q tagging" as described in this disclosure). The data may be accessed from the memory using a direct memory access (DMA) controller on the RoCE NIC, and the accessed data is then packetized by breaking it into RDMA payload chunks for the packets. For example, 1 megabyte (MB) of data may be marked for transfer from the source compute instance to the destination compute instance. The RoCE NIC may access this data from the application memory of the source compute instance and divide the data into 2 kilobyte (KB) chunks, with each chunk representing an RDMA payload to be transferred to the destination compute instance. Each payload is then packetized by the RoCE NIC to generate a RoCE Layer 2 packet (or "frame"). The packets are formatted according to an appropriate version of the RoCE protocol, such as RoCEv2 (or other RoCE protocol, or another RDMA protocol that supports VLAN tagging).

[0156] The RoCE NIC adds an 802.1Q tag to each RoCE packet, which encodes information that identifies the VLAN (e.g., a VLAN identifier) ​​to which the source compute instance belongs. The 802.1Q protocol covers the use of VLANs and their support on Ethernet networks. The 802.1Q protocol uses tags (referred to as 802.1Q tags) to traverse trunks and demarcate traffic belonging to various VLANs.

[0157] In a particular embodiment, the process at 704 is performed cooperatively by a RoCE engine on the RoCE NIC and a virtual function on the RoCE NIC corresponding to the source compute instance. The RoCE engine is responsible for generating RoCE packets. The RoCE engine may not have information identifying a VLAN identifier or QoS information for the packets to be encoded into the 802.1Q tag added to the packets. In a particular embodiment, the virtual function corresponding to the source compute instance provides a particular VLAN identifier indicating the VLAN of the source compute instance. This VLAN identifier is then encoded into the 802.1Q tag added to each packet. In a particular implementation, the virtual function on the host machine is programmed by the hypervisor to enforce that packets egressing from the host machine's RoCE NIC towards the switch fabric network have an 802.1Q VLAN tag. For example, in the embodiment shown in FIG. 6, VF VF-A-1 612 may be implemented to enforce marking of RoCE packets from source compute instance A-1 604 with a VLAN ID indicating VLAN1001, and VF VF-B-1 614 may be implemented to enforce marking of RoCE packets from source compute instance B-1 606 with a VLAN ID indicating VLAN1002.

[0158] The QoS information may be encoded into one or more different parts of the 802.1Q tagged RoCE packet. In a particular implementation, the QoS information is encoded into the DSCP bits of the IP header of each packet. In certain other embodiments, the QoS information may be encoded into the 802.1p bits in the Ethernet header of each packet. In general, the QoS information includes information indicating the priority (or class) of various traffic flows. The QoS information of a packet may specify a particular priority class of that packet that is used to forward / route the packet to its destination. For example, the QoS information may include information identifying a priority (e.g., high, low, etc.) assigned to the RoCE packet. The QoS information may also include other information, such as information specifying various parameters of each priority class related to flow control, buffer allocation, queuing, scheduling, etc. In one example, the QoS information is provided by the compute instance initiating the RDMA transfer.

[0159] Figure 8A is a diagram illustrating a RoCE packet format according to version 2 (RoCEv2) of the RoCE protocol. As shown in Figure 8A, a RoCEv2 packet 800 includes a 22-byte Ethernet header 801, a 20-byte IP header 803, and an 8-byte UDP header 804. The Ethernet header 801 includes an 8-byte preamble field 816, a 6-byte destination MAC address field 809, a 6-byte source MAC address field 810, and a 2-byte Ethertype field 811 (whose value indicates that the header is attached to an IP packet). The UDP header 804 includes a value indicating a destination port number of 4791, which specifies RoCEv2. The RoCEv2 packet 800 also includes a 12-byte InfiniBand (IB) based transport header 805, an RDMA data payload 806 (which may have a length of up to approximately 1400 bytes), a 32-bit RoCE end-to-end Invariant Cyclic Redundancy Check (ICRC) field 807, and a 4-byte Hop-by-Interface (HBI) field 808. Ethernet hop Frame Check Sequence (FCS) filter and field 808. RoCEv2 is shown merely as an example, and it is contemplated that the systems, methods, and apparatus described in this disclosure may similarly be implemented using one or more other protocols that support, for example, VLAN tagging of RDMA traffic.

[0160] As part of the process at 704 to generate a RoCEv2 Layer 2 packet shown in Figure 8A, a RoCE NIC on the host machine is configured to access the data to be transferred and prepare an RDMA payload 806. The RoCE NIC is then configured to add headers 805, 804, 803 and 801 (and check fields 807 and 808) to generate a RoCEv2 Layer 2 packet.

[0161] As mentioned above, the virtual functions are programmed (e.g., by the host's hypervisor) to enforce that RDMA packets going out to the fabric are tagged with an IEEE 802.1Q header (sometimes referred to as a "VLAN tag") that identifies the VLAN in which the packet's source compute instance is found. In certain embodiments, VLANs are associated with separate customers or tenants, and the associated VLAN identifiers are used within the fabric to enforce traffic isolation between different tenants at layer 2. FIG. 8B is a diagram illustrating the format of an 802.1Q VLAN tagged RoCEv2 packet 820, which includes a 4-byte VLAN tag 802 inserted between a source MAC address field 810 and an Ethertype field 811. The VLAN tag 802 is a 16-bit Tag Protocol ID data field 812 having a value of 0x8100, a 3-bit User Priority Code Point (PCP) data field 813, and a 4-bit User Priority Code Point (PCP) data field 814. The VLAN tag 802 includes a 1-bit priority discard indicator data field 813, a 1-bit priority discard indicator data field 814, and a 12-bit VLAN identifier data field 815 that identifies a VLAN. In a particular implementation, the packet's VLAN identifier is encoded in the VLAN identifier field 815. In a particular embodiment, the PCP data field 813 (also referred to as the "IEEE 802.1p" or "802.1p" data field) in the VLAN tag 802 can be used to encode QoS information (e.g., traffic class priority information) for the packet.

[0162] In some other embodiments, the QoS information may be encoded into the DSCP bits of each packet's IP header 803. Figure 9A shows the format of the IP header 803, which includes an 8-bit version and header length data field 901 and a 6-bit Differentiated Services Code Point (DSCP) field. The RoCEv2 packet includes a Distributed Services Code Point (DSCP) data field 902, a 2-bit Explicit Congestion Notification (ECN) data field 903, a 16-bit length data field 904, a 16-bit identification data field 905, a 16-bit fragment flag and offset data field 906, an 8-bit Time-To-Live (TTL) data field 907, an 8-bit protocol data field 908, a 16-bit header checksum data field 909, a 4-byte source IP address data field 910, and a 4-byte destination IP address data field 911. In a RoCEv2 packet, the protocol data field 908 has a value indicating that the header is applied to a UDP packet. The DSCP data field 902 can be used to carry QoS information of the packet. As described in more detail below, the ECN data field 903 can be used to indicate congestion information indicating that the packet encountered congestion in the path from the source compute instance to the destination compute instance.

[0163] As described above, the QoS information can be used to indicate the traffic class of a packet. For example, the QoS information can indicate various levels of traffic class for a packet. A packet may be used to specify a packet priority, and the priority associated with the packet is used to determine the priority of forwarding the packet on the network path from the source host machine to the destination host machine. The priority specified for a packet originating from one application may be different from the priority of a packet originating from a different application. In general, packets associated with an application that is sensitive to network latency have a higher priority than packets associated with an application that is not sensitive to network latency. The QoS information of a packet may be specified by a compute instance that initiates an RDMA transfer (e.g., by an application running on the compute instance). In one such example, initiating a compute instance may instruct a corresponding virtual function to perform the RDMA transfer according to a predetermined quality of service (QoS) (e.g., a predetermined traffic class), and the virtual function then causes an RDMA engine on the RoCE NIC to generate a packet that includes a VLAN tag that identifies the VLAN of the source compute instance and encodes the specified QoS information into a data field (e.g., an 802.1p field or a DSCP field) of the 802.1Q tagged RoCE packet.

[0164] Customers can indicate performance expectations using QoS values. For example, a customer may specify a low QoS priority for RoCE packets that carry large amounts of data for latency-tolerant applications and / or a higher QoS priority value for RoCE packets that carry small amounts of data for highly latency-sensitive applications.

[0165] In another example, the value of the QoS data field is indicated by the type of RoCE transfer being performed (e.g., according to a predefined mapping of QoS priority values ​​to RDMA transfer types). For example, RoCE packets carrying large amounts of data transfers may be tagged with a different QoS than RoCE packets carrying small amounts of data transfers that are highly latency sensitive. Examples of large transfers may include backup, reporting or batch messages, and examples of low latency critical transfers may include congestion information notification, cluster heartbeats, transaction commits, cache fusion operations, etc.

[0166] 6, if packets represent data to be transferred by RDMA from compute instance A-1 604 to compute instance A-2 624 on VLAN 1001, the RoCE NIC 608 generates layer 2 802.1Q tagged RoCE packets for the transferred data, with each packet having an 802.1Q tag in which the VLAN ID field 815 of each packet encodes information identifying VLAN 1001. Additionally, QoS information for each packet may be encoded in the packet's DSCP and / or PCP fields.

[0167] The 12-bit VLAN tag field defined in the IEEE 802.1Q standard can identify up to 4096 VLANs. The 802.1ad Q-in-Q standard (also called "802.1q-in-802.1q" or "Q-in-Q tagging" or "Q-in-Q standard") was developed to extend the number of VLANs beyond 4096. According to the Q-in-Q standard, two (or more) 802.1Q VLAN tags can be attached to a packet. These two tags, referred to as the inner tag and the outer tag, can be used for a variety of different purposes (e.g., the inner tag can represent additional security rules). For example, in some embodiments, the inner tag and the outer tag can be used to support application-specific network enforcement. It may be desirable to differentiate packets based on tenant and application, with one tag corresponding to the tenant and the other tag corresponding to the tenant. In one such example, a host machine configured to run multiple compute instances for the same customer on the same service VLAN may use a customer VLAN tag to separate traffic across the multiple compute instances on the service VLAN. Alternatively or additionally, a host machine configured to run multiple applications for the same customer on the same service VLAN may use a customer VLAN tag to separate traffic across the multiple applications on the service VLAN. Thus, in certain situations, as part of the processing performed at 704, the RoCE NIC may add two tags to each RoCE packet according to the Q-in-Q standard. In situations where tenants have different applications, for example, Tenant A has a simulation application and a backup application, two separate 802.1Q tags may be added to the RoCE packet, one tag with a VLAN ID that identifies the tenancy (e.g., Tenant A) and a second tag with a VLAN ID that identifies the application (e.g., simulation, backup).

[0168] An example of a Q-in-Q tagged RoCEv2 packet 830 is shown in Figure 8C. In such a case, each packet includes a second VLAN tag 822 (also referred to as an "inner" VLAN tag, a "private" VLAN tag, or a customer-VLAN (C-VLAN) tag) in addition to a first VLAN tag 802 (also referred to as an "outer" VLAN tag, a "public" VLAN tag, or a service-VLAN (S-VLAN) tag). One or more additional VLAN tags may be added to the Q-in-Q tagged RoCEv2 packet as well.

[0169] Returning to FIG. 7A, at 706, the 802.1Q tagged RoCE packet is forwarded from the RoCE NIC on the source host machine to a TOR switch connected to the source host machine. The TOR switch receiving the 802.1Q tagged RoCE packet may also be referred to as an ingress TOR switch, since it represents an ingress edge device of the switch fabric. The TOR switch represents the lowest layer (or leaf level) networking device of the switch network or switch fabric used to forward packets from a source compute instance on the source host machine to a destination compute instance on the destination host machine. For example, in FIG. 6, the 802.1Q tagged RoCE packet generated by the RoCE NIC 608 is communicated to the TOR switch 642, which is a tier-0 switch in the switch fabric 640.

[0170] Although the embodiment in FIG. 6 and the flow charts in FIGS. 7A-7C describe processing performed by a TOR switch, this example is not intended to be limiting. In general, RoCE packets may be communicated from a source host machine to a networking device that provides layer 2 functionality (e.g., a switch), provides layer 3 functionality (e.g., a router), or provides both layer 2 and layer 3 functionality. For example, the TOR switches 642 and 644 shown in FIG. 6 may provide both layer 2 and layer 3 functionality. Generally speaking, the networking device that receives the RoCE packets from the source host machine is an edge device of a switch fabric used to forward data from the source host machine to the destination host machine.

[0171] The source host machine 602 may be connected to the TOR switch 642 via an Ethernet cable (e.g., copper, fiber optic, etc.). In a particular embodiment, the packet arrives at a trunk port of the TOR switch 642. The trunk port may allow packets belonging to multiple VLANs to come in while traffic separation is performed using VLAN information encoded in the packet. For example, compute instance A-1 The RoCE packet representing data forwarded from 604 identifies VLAN 1001. RoCE packets representing data tagged with a tag and forwarded from compute instance B-1 606 are tagged with a tag that identifies VLAN1002.

[0172] At 708, the ingress TOR switch (e.g., TOR switch 642) that received the packet converts each Layer 2 802.1Q tagged RoCE packet forwarded through the intermediate switch 646 into a Layer 3 packet whose format is based on the particular Overlay Encapsulation Protocol (OEP) used to communicate the packet through the switch fabric. For example, various protocols such as Virtual Extensible LAN (VxLAN), Network Virtualization using Generic Routing Encapsulation (NVGRE), Generic Networking Virtualization Encapsulation (GENEVE), MPLS, Stateless Transport Tunneling (STT), etc. A variety of different overlay encapsulation protocols may be used to communicate RoCE packets through the switch fabric. For example, in the embodiment of FIG. 6, the ingress TOR switch 642 receives 802.1Q tagged RoCE packets from the RoCE NIC 608 and transforms each packet as described at 708 by encapsulating the packet. The encapsulation is performed by adding a wrapper to the packet that corresponds to the overlay encapsulation protocol used to communicate the packet through the layer 3 switch fabric using a layer 3 routing protocol, the wrapper including one or more headers. (For completeness, it is noted that in some environments implementing method 700, the ingress TOR switch may receive 802.1Q tagged RoCE packets from a source compute instance that are destined for a destination compute instance in the same rack as the source compute instance. In such a case, the ingress TOR switch may forward the packet to the destination compute instance by layer 2 transport (e.g., via the respective RoCE NIC) without processing the packet at 708 or below.) As part of the processing at 708, the ingress TOR switch 642 generates the appropriate header or headers corresponding to the overlay encapsulation protocol in use and adds a wrapper containing the header to each received Layer 2 802.1Q tagged RoCE packet, converting the packet to a Layer 3 packet, with the overlay encapsulation protocol added to the packet being visible to the networking devices in the switch fabric 640. The Layer 2 frames are converted to Layer 3 packets to enable these packets to be routed from the TOR switch 642 connected to the source host machine to the TOR switch 644 connected to the destination host machine 622 via the Layer 3 switch fabric 640 using a Layer 3 routing protocol that is more robust and scalable than the Layer 2 forwarding protocol.

[0173] As part of the processing performed at 708, for each 802.1Q tagged RoCE packet received at 708-1, the TOR switch 642 determines VLAN information from the received packet and maps or translates that information to a field (or fields) in a wrapper that is added to the packet at 708. In this manner, VLAN identifier information is added to the packet at 708 and mapped to a Layer 3 header that is visible to the various networking devices in the switch fabric 640. The Layer 3 packet includes at least one outer header that is added to the Layer 2 packet at 708.

[0174] For example, if the VxLAN protocol is used as the layer 3 encapsulation protocol for communicating packets across switch fabric 640, then at 708 the layer 2 802.1Q packet is encapsulated as a VxLAN packet by adding a VxLAN header (among other fields of the VxLAN wrapper) to the received packet at 708. As part of this operation, at 708-1, the TOR switch 642 determines the VLAN identifier information encoded in the packet's 802.1Q tag and maps (or encodes) that information into a field in the VxLAN header that is added to the packet at 708. In a particular implementation, the VLAN information in the 802.1Q tag of the RoCE packet is mapped (e.g., according to a VNI-to-VLAN map) to a corresponding unique VNI that is copied into a VNI field in the VxLAN header that is added to the packet. In this manner, the VLAN identifier information in the 802.1Q tag, which may also identify a particular tenant, is included in or carried forward to a corresponding identifier in the overlay encapsulation protocol header of the packet.

[0175] If the RoCE packet has more than one VLAN tag (e.g., Q-in-Q tagged RoCE packet 830), the TOR switch 642 maps the VLAN ID in the outer tag of the RoCE packet to a corresponding VNI according to a VNI-VLAN map. The VNI-VLAN map, which may be stored in the memory of the TOR switch 642 (e.g., as a table), is a one-to-one correspondence between the VNIs assigned to the TOR switch 642 and the VLANs to which the VNIs are assigned. Because VLANs only have local significance to a switch, the same VLAN ID may be mapped to different VNIs elsewhere in the fabric (for convenience, it may be desirable to use the same VNI-VLAN mapping in multiple switches in the fabric), and / or the same VNI may be mapped to different VLAN IDs elsewhere in the fabric (e.g., different VLAN IDs assigned to the same tenant). The VNI is copied into a corresponding data field of at least one outer header (e.g., an encapsulation protocol header) of the corresponding Layer 3 encapsulated packet, thereby extending multitenancy across L3 network boundaries. If the overlay encapsulation protocol is VxLAN (or GENEVE), the VNI is held in the 24-bit VNI field of the VxLAN (or GENEVE) header of the Layer 3 encapsulation packet. If the overlay encapsulation protocol is NVGRE, the VNI is held in the 24-bit Virtual Subnet ID (VSID) field of the NVGRE header of the Layer 3 encapsulation packet. In the STT case, the VNI is carried in the 64-bit Context ID field of the STT header of the Layer 3 encapsulated packet.

[0176] Additionally, as part of the processing at 708, at 708-2, for each 802.1Q tagged RoCE packet received, the TOR switch 642 determines QoS information from the received packet and maps or translates that information to a field (or fields) in a header that is added to the packet at 708. In this manner, the QoS information is mapped to part of a wrapper (e.g., outer header) that is added to the packet at 708 and that is visible to various networking devices in the switch fabric 640.

[0177] For example, if the VxLAN protocol is used as the Layer 3 encapsulation protocol for communicating packets over the switch fabric 640, then at 708 the Layer 2 802.1Q packet is converted to a VxLAN packet by adding at 708 a VxLAN envelope or wrapper (including a VxLAN header and some outer headers) to the received packet. As part of this operation, at 708-2 the TOR switch 642 determines the QoS information encoded in the received packet and maps (or encodes) that information into fields in the VxLAN wrapper that is added to the packet at 708. As discussed above, depending on the implementation, the QoS information may be encoded into one or more different parts of the received Layer 2 packet. For example, the QoS information may be encoded into the PCP or 802.1p bits of the 802.1Q tag. The QoS information may be included in the Layer 2 packet and / or may be encoded in the DSCP field of the Ethernet header of the received Layer 2 packet. As part of 708-2, the TOR switch 642 determines and maps or translates this QoS information to a field (or fields) in the VxLAN wrapper that is added to the packet at 708. In a particular implementation, the QoS information from the RoCE packet is mapped to the DSCP field in the outer IP header of the VxLAN wrapper. In this manner, the QoS information in the Layer 2 802.1Q tagged RoCE packet is included in or carried forward in the Layer 3 wrapper of the VxLAN packet in a manner that makes it visible to various networking devices in the switch fabric 640.

[0178] 10 is a diagram showing the format of a Layer 3 encapsulated packet 1000 (also called a VxLAN packet) generated by an ingress TOR switch that applies VxLAN as an OEP. As shown in FIG. 10, the VxLAN packet 1000 includes an outer Ethernet header 1010, an outer IP header 1020, an outer UDP header 1040, a VxLAN header 1050, an original packet (e.g., RoCEv2 packet) 1060, and a Frame Check Sequence (FCS) 1070. In the process performed at 708, if VxLAN is the overlay encapsulation protocol, then as part of encapsulating the 802.1Q tagged RoCE packet, the TOR switch places a VxLAN header 1050 on the outside of the "original" 802.1Q tagged RoCE packet, then places an outer UDP header 1040 on the outside of the VxLAN header, then places an outer IP header 1020 on the outside of the outer UDP header, and then adds an outer Ethernet header 1010 on top of the outer IP header.

[0179] The outer Ethernet header 1010 includes a destination MAC address field 1011, a source MAC address field 1012, (optionally) a VLAN type field 1013, (optionally) a VLAN ID tag 1014, and an Ethertype field 1015 which holds the value 0x0800. The outer IP header 1020 includes an 8-bit version and header length data field 1021, a 6-bit DSCP data field 1022, a 2-bit ECN data field 1024, a 16-bit length data field 1024, a 16-bit identification data field 1025, a 16-bit fragment flag and offset data field 1026, an 8-bit time to live (TTL) data field 1027, an 8-bit protocol data field 1028 which holds a value 17 (indicating UDP), a 16-bit header checksum data field 1029, a 4-byte source IP address data field 1030 indicating the IP address of the ingress TOR switch, and a 4-byte destination IP address data field 1031 indicating the IP address of the egress TOR switch. The outer UDP header 1040 includes a source port field 1041, which may hold a value that is a hash of information from the original RDMA packet, a destination (VxLAN) port field 1042, which holds the value 4789, a UDP length field 1043, and a checksum field 1044. The VxLAN header 1050 includes an 8-bit flags field 1051, a 24-bit VNI field 1053 that carries the VNI, and two reserved fields 1052 and 1054.

[0180] As part of creating a Layer 3 VxLAN packet by encapsulating the “original” 802.1Q tagged RoCE packet at 708, the TOR switch encodes VLAN ID information (e.g., tenancy information), QoS information (e.g., traffic class), and congestion information from the original packet into various fields of one or more of the headers added to the original 802.1Q tagged RoCE packet. For example, in a particular implementation, the VLAN ID field is mapped to a VNI and retained in the VNI field 1053, and the QoS information from the RoCE packet is copied (or mapped to a value) into the DSCP field 1022 in the IP header 1020. QoS information in the DSCP data field of the RoCE packet's IP header (e.g., from the DSCP field 902 in FIG. 9 ) can be copied or otherwise mapped to the DSCP data field 1022 of the outer IP header 1020 of the encapsulating VxLAN packet. In embodiments where the QoS information in the RoCE packet is encoded in the PCP data field 813 of the RoCE packet's 802.1Q tag, that information may also be mapped to the DSCP data field 1022 of the outer IP header 1020 of the VxLAN packet.

[0181] Thus, at 708, an overlay encapsulation protocol wrapper is added to the RoCE packet, and the VLAN ID and QoS information from the RoCE packet are mapped and encoded into the overlay encapsulation protocol wrapper in a manner that is visible to devices in the switch fabric. With reference to FIG. 7B, at 710, the encapsulated Layer 3 packet generated at 708 is routed through the switch fabric from the TOR switch that receives the packet from the source host machine to the TOR switch connected to the destination host machine. In a particular embodiment, the packet is forwarded and sent through a tunnel (e.g., a VxLAN tunnel if the VxLAN overlay encapsulation protocol is used) that carries the packet through the switch fabric from the TOR switch connected to the source host machine to the TOR switch connected to the destination host machine. The path taken by the packet in the switch fabric may traverse multiple networking devices, each configured to receive the packet via an ingress port of the networking device and forward the packet to a next-hop networking device via an egress port of the networking device to facilitate communication of the packet to the TOR switch connected to the destination host machine. For example, in the embodiment shown in FIG. 6, the encapsulated Layer 3 packets are forwarded from TOR switch 642 to TOR switch 644 via one or more intermediate switches 646 in switch fabric 640 .

[0182] At 708, by translating the VLAN ID and QoS information from the layer 2 802.1Q RoCE packet to information carried in a layer 3 encapsulated packet wrapper, the VLAN information (e.g., tenancy information) and QoS information become visible to the switches and networking devices in the switch fabric 640 for the purpose of routing the packet using a layer 3 routing protocol. As part of the process at 710, each networking device in the switch fabric 640 that receives and forwards the layer 3 encapsulated RoCE packet, including the TOR switches 642 and 644, is configured to forward the packet based on layer 3 forwarding according to the QoS information specified in the packet's encapsulation wrapper at 710-1. In a particular implementation, each networking device that receives a packet may have multiple RDMA data queues corresponding to different QoS priority levels. Upon receiving a packet, the networking device is configured to determine the QoS information of the packet from one or more fields of the encapsulation wrapper (or, in the first TOR switch that receives the packet, from the layer 2 header) and queue the packet in the RDMA data queue that corresponds to the priority level specified by the QoS information. The packets are then dequeued and forwarded from the networking device according to the queue's particular priority. The multiple RDMA data queues may be implemented, for example, using one or more buffers, and the networking device may include enqueuing logic configured to distribute incoming packets among the multiple queues according to a predetermined priority level (e.g., traffic class), and an enqueuing logic configured to service the multiple queues according to a desired scheduling scheme (e.g., weighted round robin scheduling and / or strict priority scheduling). The networking device may also include configured dequeue logic. When implementing a networking device using merchant silicon, it may be desirable to make the most of a relatively small amount of buffers. For example, it may be desirable to repurpose unused buffers (e.g., from queues that are not in use) for use as at least a portion of multiple queues. For example, it may be desirable to implement at least a portion of the environment 600 (e.g., TOR switches 642 and 644, switch fabric 640) to exclude support for multicast traffic, in which case buffers previously allocated to store multicast traffic may instead be reprogrammed as storage for multiple queues.

[0183] As part of 710, at 710-2, each networking device in the switch fabric that receives and forwards a Layer 3 encapsulated RoCE packet may, when it experiences congestion, signal the congestion by marking a field in the outer header of the packet's Layer 3 encapsulation wrapper. In a particular embodiment, a mechanism called Explicit Congestion Notification (ECN), an IP protocol concept, is used by the RoCE protocol to specify congestion information. In accordance with this mechanism, ECN bits in the packet's IP header are used to specify or encode congestion information. Thus, as part of the processing at 710-2, when a networking device that receives and forwards a packet detects congestion (e.g., detects that a buffer occupancy exceeds a threshold), the networking device may signal the congestion by setting an ECN bit in the outer IP header of the packet's overlay encapsulation protocol wrapper. For example, if VxLAN is the overlay encapsulation protocol, a bit in the ECN field 1023 of the outer IP header 1020 may be set by the networking device to signal congestion (e.g., if the bit is not already set).

[0184] In this manner, as the packet travels through the switch fabric, congestion information is updated in the packet and carried by the packet to the packet's destination. For example, in the embodiment shown in Figure 6, as the packet travels through the series of networking devices in switch fabric 640 in moving from source host machine 602 to destination host machine 622, any of the networking devices traversed by the packet may indicate congestion by setting a congestion bit in the packet's Layer 3 encapsulation wrapper (e.g., setting a bit in the ECN field of the outer IP header). The series of networking devices includes TOR switch 642, TOR switch 644, and any intermediate switches 646 in the path traversed by the packet.

[0185] The indication or marking of congestion in a packet can be done by any of the networking devices in the switch fabric that routes the packet from the source host machine to the destination host machine. The egress TOR copies the ECN bits from the layer 3 wrapper to the IP header of the inner packet, and the inner packet is sent as layer 2 to the destination host machine. In this way, the congestion information is carried in the packet all the way to the destination host machine. The congestion information is carried across the boundaries of the layer 2 network, which includes the source and destination compute instances and host machines, and the boundaries of the layer 3 network, which includes the TOR switches and intermediate switches in the switch fabric.

[0186] Traditionally, RoCE relies on Layer 2 Priority Flow Control (PFC), or ECN, or a combination of PFC and ECN, for congestion control. It may be desirable to implement TOR switches (e.g., TOR switches 642 and 644) to perform priority flow control (PFC). When the receive buffer of a PFC-enabled switch fills to a threshold level, the switch returns a PAUSE frame of the corresponding priority class to the sender. PFC provides up to eight priority classes and It may be desirable to implement the switch to use the PCP value of the packet at the head of the buffer as the priority class indicated by the PAUSE frame (or, alternatively, to use a mapping of the packet's DSCP value to the priority class indicated by the PAUSE frame).

[0187] It may be desirable to implement congestion control on a per-application basis so that congestion control to pause one customer's RDMA traffic does not pause another customer's RDMA traffic. Because a PAUSE frame can affect multiple customers because it forces the sender to pause all traffic of the indicated priority class, it may be desirable to configure each TOR switch to prevent any PAUSE frames from traveling beyond the fabric side of the TOR switch. For example, it may be desirable to restrict PFC to the host-facing interfaces of each TOR switch 642 and 644 (e.g., by configuring each input port of the TOR switch to only pass PAUSE frames to the host or to another input port of the TOR switch). Localized containment of PAUSE frames in this manner can help avoid congestion spreading within a large fabric and / or can help avoid live lockups.

[0188] At 712, a TOR switch (also referred to as an egress TOR switch, since it represents an egress edge device of the switch fabric) connected to the destination host machine receives the Layer 3 encapsulated RoCE packet. For example, in the embodiment shown in Figure 6, the egress TOR switch 644 receives the packet.

[0189] At 714, for each received Layer 3 encapsulated packet, the TOR switch determines congestion information from the packet's Layer 3 overlay encapsulation protocol wrapper and maps or translates the congestion information to the Layer 2 802.1Q tagged RoCE frame encapsulated by the Layer 3 wrapper. Thus, for each Layer 3 encapsulated RoCE packet received by the egress TOR switch, congestion information that may have been signaled by one or more networking devices (e.g., the TOR switch 642, one or more intermediate switches 646) along the path traversed by the packet in the switch fabric and that was encoded in a field (e.g., the ECN field) of the header of the Layer 3 overlay encapsulation protocol wrapper is mapped and preserved in the header of the 802.1Q tagged RoCE Layer 2 frame. In a particular implementation, congestion information determined from the Layer 3 wrapper (e.g., from the ECN field 1023 in the IP header 1020 of the VxLAN packet) is copied to the ECN field of the IP header of the Layer 2 802.1Q tagged RoCE frame (e.g., the ECN field 903 shown in FIG. 9).

[0190] At 716, for each received Layer 3 encapsulated RoCE packet, the egress TOR switch decapsulates the packet by removing the encapsulation wrappers (e.g., outer Ethernet header 1010, outer IP header 1020, outer UDP header 1040, VxLAN header 1050 and FCS 1070) added to the packet at 708 to recover an inner Layer 2 802.1Q tagged RoCE packet. For example, if a VxLAN wrapper was added to the packet, it is removed at 714 to leave an 802.1Q tagged RoCE packet. Because the congestion information is mapped to the 802.1Q tagged RoCE packet header at 714, the congestion information in the Layer 3 overlay encapsulation protocol wrapper is not lost as a result of decapsulation of the Layer 3 encapsulated packet. The mapping (e.g., copying) of the congestion information at 714 may be performed before, during or after the decapsulation at 716.

[0191] As part of the processing at 716, in addition to translating congestion information from the layer 3 overlay encapsulation protocol wrapper into a header in the layer 2 802.1Q tagged RoCE packet, the egress TOR switch 644 may itself set an ECN congestion bit to signal congestion if it experiences congestion.

[0192] Referring to Figure 7C, at 718, the decapsulated layer 2 802.1Q tagged RoCE packets are forwarded by the egress TOR switch to the destination host machine. For example, in the embodiment of Figure 6, the RoCE packets decapsulated by the TOR switch 644 are forwarded to the destination host machine 622. At the destination host machine, these packets are received and processed by the RoCE NIC on the destination host machine.

[0193] At 720, for each 802.1Q tagged RoCE packet received, the RoCE NIC on the destination host machine checks whether congestion is indicated in the header of the received packet. For example, congestion can be indicated when the ECN bits in the IP header of the packet are set. If the packet is determined to indicate congestion at 720, the RoCE NIC sends a response to the sender of the packet (e.g., the RoCE NIC on the source host machine) at 722 indicating congestion and requesting the sender to slow down the data transfer rate. In a particular implementation using the ECN protocol, the response is in the form of a Congestion Notification packet (CNP packet) sent from the RoCE NIC on the destination host machine to the RoCE NIC on the source host machine. For example, a Data Center Quantized Congestion Notification (DCQCN) packet may be sent to the RoCE NIC (e.g., an RDMA NIC card and corresponding software driver) to notify the RoCE NIC of the congestion. ) may be implemented to use the ECN information to perform flow control by sending CNP packets to inform the sender about the congestion. These CNP packets inform the source host machine that there is congestion in the network and request the source host machine to slow down the rate at which it is sending RoCE packets. The CNP packets are sent to the appropriate sender, which is identified from information in the received Layer 2 RoCE packet (e.g., source MAC address and / or source IP address). Upon receiving such a notification, the sender (e.g., the RoCE The NIC can then throttle back the transmission of RoCE packets accordingly. Further details related to CNP packets and how they are communicated from the destination host machine to the source host machine are as follows.

[0194] In a particular implementation, the sender may use an algorithm to calculate a percentage decrease in data transmission rate. For example, upon receiving a first CNP packet, the sender (e.g., a RoCE NIC on a source host machine) may decrease its transmission rate by a certain percentage. Upon receiving another CNP packet, the sender may further decrease its transmission rate by an additional percentage amount, and so on. In this manner, the sender may perform adaptive rate control in response to receiving a CNP packet.

[0195] At 724, for each received 802.1Q tagged RoCE packet, the RoCE NIC on the destination host machine retrieves the RDMA data payload from the packet and transfers the data to an application memory on the destination host machine of the destination compute instance via a corresponding virtual function (VF). In a particular embodiment, the virtual function on the RoCE NIC corresponding to the destination compute instance is configured to control the RoCE engine of the RoCE NIC to transfer the RDMA data payload to the memory space of the destination compute instance via a DMA transfer to the application memory of the destination host machine. This operation is called an RDMA transfer from the source compute instance to the destination compute instance. Complete the data transfer.

[0196] As described above, a VLAN identifier that can identify a tenant is included in an 802.1Q tag that is added to a RoCE packet, for example in the VLAN ID field of the 802.1Q tag. The VLAN ID or tenancy information is also mapped to a VNI that is included in a Layer 3 overlay encapsulation protocol wrapper that is added to the 802.1Q tagged RoCE packet by a TOR switch connected to the source host machine. By mapping the VLAN identifier (or tenancy information) to an identifier in a field of the Layer 3 encapsulation wrapper, the tenancy information becomes visible to networking devices in the Layer 3 switch fabric. These networking devices use this information to separate traffic belonging to different customers or tenancies.

[0197] The QoS information associated with a packet is also preserved from the RoCE NIC on the source host machine all the way to the RoCE NIC on the destination host machine. The QoS information encoded in the layer 2 RoCE packet is made visible to the networking devices in the switch fabric by encoding the information in a layer 3 overlay encapsulation protocol wrapper that is added to the 802.1Q tagged RoCE packet by the ingress TOR switch. The networking devices in the switch fabric can then use layer 3 routing protocols to route the RoCE traffic through the switch fabric according to the QoS information associated with each packet.

[0198] Any networking device in the switch fabric can signal congestion on a per-packet basis. This congestion information is preserved in the packet as it traverses the switch fabric from the TOR connected to the source host machine to the TOR connected to the destination host machine. At the TOR switch connected to the destination host machine, the congestion information from the layer 3 encapsulation wrapper is converted to the RoCE packet header (e.g., ECN bits in the IP header of the RoCE packet) and is thus preserved and made available to the destination host machine. The destination host machine can then respond to the congestion information by sending a CNP packet.

[0199] Routing of congestion notification information In general, RDMA data transfers are highly susceptible to network latency that may be caused by congestion in the switch fabric network. RDMA congestion notification packets (CNP packets) are very important in helping inform RDMA congestion management (flow control). As such, CNP packets do not require large network bandwidth, but are highly susceptible to packet loss and network latency. Thus, CNP packets sent by a destination host machine in response to a congestion notification by a received packet can be given high priority to reach the source host machine with minimal latency and notify the sender to slow down the transmission of data to minimize or avoid loss of data packets due to congestion. Furthermore, the priority queuing for CNP packets is such that it minimizes the possibility that the CNP packet itself is discarded due to congestion.

[0200] To accomplish this, congestion notification packet traffic (CNP traffic) is assigned a high priority such that at each networking device in the switch fabric, the CNP packets are assigned to a queue with a very high priority as they travel from the destination host machine to the source host machine. In a specific implementation, the CNP traffic is assigned to the second highest queue on each networking device in the switch fabric (e.g., the network control queue has only the second highest queue). assigned).

[0201] Furthermore, a strict priority queuing technique is used for CNP packets. With strict priority queuing, traffic assigned to a strict priority queue will starve other traffic. For example, if a networking device in a switch fabric has a packet from tenant #1, a packet from tenant #2, and a CNP packet, and the networking device can only send one packet out, the tenant #1 packet and the tenant #2 packet are queued and the CNP packet is sent instead. In a specific implementation, the CNP packet is configured with QoS information indicating a special class of the CNP packet and indicating that a strict priority queue is used to queue the packet for transmission.

[0202] However, strict queuing implementations should be careful that strict priority queuing does not starve other traffic indefinitely. Thus, a limit can be applied to how much CNP traffic is allowed to pass through the switch fabric. In a particular implementation, this limit is a small fraction of the total bandwidth of the link. For example, a dedicated strict priority queue can be assigned to CNP traffic with a low bandwidth guarantee so as not to starve the actual RDMA traffic class. As a result, if a situation arises where a rogue or misconfigured application starts generating a large number of CNP packets such that the CNP traffic can starve other traffic (e.g., RDMA data traffic), the limit threshold minimizes the impact of this problem on other traffic.

[0203] There is a feedback delay between the time an ECN-capable device marks a packet and the time the sender receives the resulting CNP packet. In the worst case, such a delay can lead to congestion collapse, a long-standing problem in high-performance networks. To avoid redundant feedback loops between ECN marking by switches, CNP reflection by receiving hosts, and RDMA congestion management actions by sending hosts, it may be desirable to configure the networking devices of the switch fabric 640 with low and deterministic ECN marking thresholds. For example, it may be desirable to configure each of the TOR switches and intermediate switches to mark any packets upon detection of congestion. An aggressive ECN marking policy provides a tight loop for congestion management to help protect the network from congestion collapse by ensuring that switches initiate ECN marking at the slightest hint of network congestion.

[0204] Queue-based routing to avoid head-of-line blocking As described above, QoS information associated with a packet is used by networking devices in the switch fabric to route the packet using a layer 3 routing protocol. The QoS information may identify a particular priority or class. A networking device in the switch fabric (e.g., any one or more (or all) of the TOR switch 642, the TOR switch 644, and the intermediate switch 646) may use this priority information to identify a particular prioritized queue to use for forwarding the packet from among multiple queues used by the networking device. For example, the networking device may maintain a set of queues with individual queues for each different priority class. Packets corresponding to different customers or tenants may have different assigned priorities or classes, such that the differently classified packets may be assigned to different queues on the networking devices in the switch fabric. Packets of different classes (e.g., different tenants, different applications) may be assigned to different queues on the networking devices in the switch fabric. In particular embodiments, RDMA packet flows from different tenants may be assigned to the same queue of a networking device (e.g., according to the same assigned priority class) and / or RDMA packet flows from the same tenant (e.g., packet flows from different applications of a tenant) may be assigned to different queues of a networking device (e.g., according to different respective assigned priority classes).

[0205] In certain embodiments, multiple queues are used on the networking device to handle RDMA / RoCE traffic to avoid head-of-line blocking issues. Providing multiple queues for RDMA data traffic avoids congestion that may result from queuing all RDMA / RoCE traffic in a single queue. Multiple (e.g., four) RDMA queues also enable multiple different applications that require different levels of performance, and all of these classes require lossless networking. In that way, the environment can provide dedicated network paths for latency-sensitive RDMA applications at cloud scale across the fabric, and can do so while avoiding HOL blocking issues. Each of the core network queues can support configurable weighted bandwidth distribution.

[0206] In some cases, a customer or tenant may control which prioritized queue is used to route traffic by QoS information set in the packet. In a particular implementation, on a networking device in a switch fabric having a large number of queues (e.g., multiple queues) for sending packets, a percentage of the queues are set aside for RDMA traffic. For example, in one implementation, if a switch in the switch fabric has eight queues, six of the queues may be set aside for RDMA traffic. These RDMA queues may be weighted round robin queues, where each of the weighted round robin queues gets its share of the network bandwidth but should not starve each other (e.g., to provide fairness across RDMA applications). In one such scheme, each of the RDMA queues is weighted equally, so that each of the RDMA queues is served once every dequeue cycle. For example, 95% of the capacity of the link (shared by traffic assigned to different queues) may be assigned to six RDMA queues, with each queue getting one-sixth of the 95% (e.g., by a weighted round robin scheme with equal weighting). Additionally, it may be desirable to ensure that the switch fabric is not oversubscribed so that there is sufficient bandwidth to handle the traffic being communicated through the switch fabric. Traffic from different customers or tenants may be assigned to the same RDMA queue, but differentiated based on the VLAN ID and / or VNI encoded in the packets.

[0207] The switch fabric 640 can use network control traffic classes in basic IP routing protocol functions (e.g., between TOR switches). In one example, the queues of the networking device include a network control queue for carrying Ethernet VPN (EVPN) traffic, which can be used for distribution of MAC address information and / or Virtual Tunnel End Point (VTEP) flood list notification throughout the underlying substrate network. These network control protocols (e.g., Border Gateway Protocol (BGP)) include the highest traffic class, which is used to route traffic to and from the TOR. For example, a network control queue may be assigned a traffic class. A network control traffic class may be dedicated to the network control traffic class and implemented as a strict priority queue so that the queue is emptied before any of the queues are served. Because network control traffic does not consume much network bandwidth, the network control traffic class may be allocated a small amount of the total bandwidth of the multiple queues.

[0208] FIG. 9B illustrates an example of multiple queues 950 of a networking device (e.g., TOR switch 642 or 644, intermediate switch 646, etc.) that includes four RDMA queues 960-1 through 960-4 and a network control (NC) queue 964. As shown in FIG. 9B, multiple queues 950 may also include a dedicated congestion notification (CN) queue 962 for carrying CNP packets. Prompt delivery of CNP packets to the sending host is critical to successful RDMA congestion management because the flow control that CNP packets signal is lost if the CNP packets are lost. Thus, CNP traffic is particularly susceptible to packet loss and network latency, but does not require a large amount of network bandwidth. These requirements can be balanced by configuring CN queue 962 to be a strict priority queue so that it empties before any of the RDMA queues (e.g., queues 960-1 through 960-4) are served, but have only a low bandwidth guarantee so as not to starve the actual RDMA traffic class. To prevent the congestion notification queue 962 from starving the network control queue 964 (e.g., in the event of a misconfiguration or other problem that results in an excessive amount of CNP traffic), it may be desirable to configure the congestion notification queue 962 to have a lower priority than the network control queue 964.

[0209] Additionally or alternatively, the multiple queues of the network device may also include a scavenger queue for non-RDMA traffic (e.g., other protocols such as TCP). The "scavenger" traffic class uses unused network bandwidth without adversely affecting the RDMA traffic class. The dequeue logic of the networking device may be configured to serve the scavenger queue with lower priority than the RDMA queue, for example, by assigning a low weight (e.g., the weight is guaranteed bandwidth) in a weighted round robin scheme to the scavenger queue. FIG. 9C illustrates an example 952 of multiple queues 950 of the above networking device (e.g., TOR switch 642 or 644, intermediate switch 646, etc.) that also includes a scavenger queue 966.

[0210] As described in this disclosure, RDMA packets (either Layer 2 RDMA packets or Layer 3 encapsulated packets carrying Layer 2 RDMA packets) carry a QoS value (e.g., in a PCP data field and / or in a DSCP data field) indicating the priority (e.g., traffic class) of the packet, and the enqueue logic of the networking device may be configured to distribute the incoming packets among the multiple RDMA queues of the networking device according to the QoS value. In an example where the PCP data field is used to carry the QoS value and the RDMA packets are distributed among RDMA queues 960-1 to 960-4 as shown in FIG. 9B, the following mapping may be used: RDMA packets with PCP values ​​of 6 or 7 are stored in RDMA queue 960-1, RDMA packets with PCP values ​​of 4 or 5 are stored in RDMA queue 960-2, RDMA packets with PCP values ​​of 2 or 3 are stored in RDMA queue 960-3, and RDMA packets with PCP values ​​of 0 or 1 are stored in RDMA queue 960-4. In the example shown in Figure 9B where DSCP data is used to carry QoS values ​​and RDMA packets are distributed among RDMA queues 960-1 through 960-4, the following mapping may be used: RDMA packets with DSCP values ​​in the range of 48 through 64 are stored in RDMA queue 960-1, RDMA packets with DSCP values ​​in the range of 32 through 47 are stored in RDMA queue 960-2, and so on. RDMA packets having DSCP values ​​in the range of 16-32 are stored in RDMA queue 960-3, and RDMA packets having DSCP values ​​in the range of 0-15 are stored in RDMA queue 960-4. Those skilled in the art will recognize that the above two mappings are non-limiting examples only, and that distribution of RDMA packets among multiple queues of a networking device may be performed (e.g., by enqueue logic) according to such mappings or other mappings of QoS values ​​to RDMA queues.

[0211] Deterministic congestion (e.g., ECN bits) marking In certain embodiments, a deterministic congestion marking scheme is used in which congestion marking is performed on a packet-by-packet basis. Thus, for each packet, when a networking device in the switch fabric experiences or detects congestion, the networking device signals the congestion by marking a field in the packet, such as by marking the ECN bits in the IP header of the RoCE packet's Layer 3 encapsulation wrapper. As a result, when there is congestion, multiple packets arriving at the destination host machine will have the congestion bit set. In response to each such packet, the destination host machine may send a CNP packet. In response to the CNP packet, the sender may decrease its transmission rate. The goal is to detect congestion early, thereby allowing the sender to slow down transmission early, thereby decreasing the probability of packet drops or loss.

[0212] The switch fabric architecture also plays a role in reducing the latency of RoCE packets and reducing packet loss. As mentioned above, the switch fabric can be constructed as a Clos network, such as the Clos network shown in Figure 5 and described above. For example, in a Clos network with a two-tier topology with only tier-0 switches (TOR switches) and tier-1 switches (spine switches), a RoCE packet can go from any source host machine to any destination host machine in three hops. Minimizing the number of hops means that the latency is very low, which is suitable for RoCE traffic.

[0213] In a particular embodiment, RDMA traffic belonging to the same flow follows the same path from the source host machine to the destination host machine because RDMA traffic is susceptible to packet reordering. This flow-based routing avoids the situation where packets arrive at the destination host machine out of order. For example, an ingress TOR switch can be configured to distribute Layer 3 encapsulated packets of multiple flows to spine switches by a per-flow equal-cost multipath (ECMP) scheme (e.g., an n-way ECMP scheme, where "n" should not be confused with the number of layers "n" in a Clos network) to maintain packet order within each flow. In general, the flow to which a packet belongs is defined by the combination of the packet's source IP address, destination IP address, source port, destination port, and protocol identifier (also called the 5-tuple).

[0214] Exemplary Infrastructure-as-a-Service (IaaS) Architecture As mentioned above, IaaS (Infrastructure as a Service) is a specific type of Cloud computing is an emerging technology that provides infrastructure services (IaaS) that can be deployed on a cloud-based platform, such as a cloud computing service (cloud computing). IaaS may be configured to provide virtualized computing resources over a public network (e.g., the Internet). In an IaaS model, a cloud computing provider may host infrastructure elements (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, an IaaS provider may provide additional services (e.g., a service provider) that are attached to the infrastructure elements. A variety of associated services (e.g., billing, monitoring, logging, security, load balancing, and clustering) can be provided, and since these services can be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance.

[0215] In some examples, an IaaS customer can access resources and services over a wide area network (WAN), such as the Internet, and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log into an IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and install enterprise software on the VMs. The customer can use the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting applications, monitoring performance, managing disaster recovery, etc.

[0216] In most cases, the cloud computing model requires the participation of a cloud provider, which may be, but does not have to be, a third-party service dedicated to providing (e.g., offering, renting, selling) IaaS. Enterprises can also deploy private clouds and become providers of infrastructure services.

[0217] In some examples, IaaS deployment is the process of deploying a new application or a new version of an application onto provisioned application servers, etc. IaaS deployment may include the process of provisioning a server (e.g., installing libraries, daemons, etc.). IaaS deployment is often managed below the hypervisor layer (e.g., servers, storage, network hardware, and virtualization) by the cloud provider. Thus, the customer may be responsible for OS, middleware, and / or application deployment (e.g., self-service virtual machines (e.g., that can be spun up on demand), etc.

[0218] In some instances, IaaS provisioning may include obtaining a computer or virtual host to be used and installing the necessary libraries or services on the computer or virtual host. In most cases, deployment does not include provisioning, which must be performed first.

[0219] In some cases, there are two different challenges in IaaS provisioning. First, there is the challenge of provisioning an initial set of infrastructure before anything can be done. Second, there is the challenge of evolving the existing infrastructure (e.g., adding new services, modifying services, removing services) after everything has been provisioned. In some cases, these two challenges can be addressed by allowing the configuration of the infrastructure to be defined declaratively. In other words, the infrastructure (e.g., which elements are required and how these elements interact) may be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which and how they work together) can be described declaratively. In some instances, once the topology is defined, workflows can be generated to create and / or manage the different elements described in the configuration files.

[0220] In some instances, the infrastructure may include many interconnected elements. For example, there may be one or more Virtual Private Clouds (VPCs) (e.g., potentially on-demand pools of configurable and / or shared computing resources), also known as a core network. In some examples, there may be one or more security group rules and one or more virtual machines (VMs) that are provisioned to define how the security of the network is configured. Other infrastructure elements such as load balancers, databases, etc. may also be provisioned. The infrastructure can evolve incrementally as more and more infrastructure elements are desired and / or added.

[0221] In some examples, continuous deployment techniques may be employed to enable deployment of infrastructure code across various virtual computing environments. The described techniques may also enable infrastructure management within these environments. In some examples, a service team may write code that is desired to be deployed to one or more, typically many, different production environments (e.g., across a variety of different geographic locations, sometimes spanning the globe). However, in some examples, the infrastructure for deploying the code must first be set up. In some examples, provisioning may be done manually, resources may be provisioned using a provisioning tool, and / or the code may be deployed using a deployment tool after the infrastructure has been provisioned.

[0222] 11 is a block diagram 1100 illustrating an example pattern of an IaaS architecture in accordance with at least one embodiment. A service operator 1102 may be communicatively connected to a secure host tenancy 1104, which may include a virtual cloud network (VCN) 1106 and a secure host subnet 1108. In some examples, the service operator 1102 may use one or more client computing devices. The one or more client computing devices may be, for example, a mobile device running software such as Microsoft Windows Mobile®, and / or iOS, Windows Phone, Android®, BlackBerry 8, and Palm OS. and may be an Internet, email, short message service (SMS), Blackberry® or other communication protocol enabled handheld mobile device (e.g., iPhone®, mobile phone, iPad®, tablet, personal digital assistant (PDA) or wearable device (e.g., Google® Glass® head mounted display). The client computing device may illustratively be a Microsoft Windows (registered (trademark) operating system, Apple Macintosh (trademark) operating system various versions of the operating system and / or Linux The client computing devices may be general-purpose personal computers, including personal computers and / or laptop computers running the VCN 1106. Alternatively, the client computing devices may be workstation computers running various commercially available UNIX or UNIX-like operating systems, including, but not limited to, various GNU / Linux operating systems, e.g., Google Chrome OS. Alternatively or additionally, the client computing devices may be other electronic devices capable of communicating over the VCN 1106 and / or a network with access to the Internet, such as thin-client computers, Internet-enabled gaming systems (e.g., Microsoft Xbox gaming consoles with or without Kinect gesture input devices), and / or personal messaging devices.

[0223] VCN1106 is connected to a secure SSH server via LPG1110 included in VCN1112. The IaaS provider may include a local peering gateway (LPG) 1110 that may be communicatively connected to a private shell (SSH) VCN 1112. The SSH VCN 1112 may include an SSH subnet 1114, and the SSH VCN 1112 may be communicatively connected to a control plane VCN 1116 via the LPG 1110 that is included in the control plane VCN 1116. The SSH VCN 1112 may also be communicatively connected to a data plane VCN 1118 via the LPG 1110. The control plane VCN 1116 and the data plane VCN 1118 may be included in a service tenancy 1119, which may be owned and / or operated by the IaaS provider.

[0224] The control plane VCN 1116 may include a control plane demilitarized zone (DMZ) tier 1120 that serves as a perimeter network (e.g., a portion of an enterprise network between the enterprise intranet and an external network). DMZ-based servers may have a certain reliability and contain security breaches. Additionally, the DMZ tier 1120 may include one or more load balancer (LB) subnets 1122, a control plane app tier 1124 that may include an app subnet 1126, and a control plane data tier 1128 that may include a database (DB) subnet 1130 (e.g., a front-end DB subnet and / or a back-end DB subnet). The LB subnet 1122 included in the control plane DMZ tier 1120 may be communicatively connected to an app subnet 1126 included in the control plane app tier 1124 and an Internet gateway 1134 that may be included in the control plane VCN 1116, which may be communicatively connected to a DB subnet 1130, a service gateway 1136, and a network address translation (NAT) gateway 1138 included in the control plane data tier 1128. The control plane VCN 1116 may include the service gateway 1136 and the NAT gateway 1138.

[0225] The control plane VCN 1116 can include a data plane mirror app layer 1140, which can include an app subnet 1126. The app subnet 1126 included in the data plane mirror app layer 1140 can include a virtual network interface controller (VNIC) 1142 on which a compute instance 1144 can run. The compute instance 1144 can communicatively connect the app subnet 1126 of the data plane mirror app layer 1140 to the app subnet 1126, which can be included in the data plane app layer 1146.

[0226] The data plane VCN 1118 may include a data plane app layer 1146, a data plane DMZ layer 1148, and a data plane data layer 1150. The data plane DMZ layer 1148 may include a LB subnet 1122 that may be communicatively connected to an app subnet 1126 of the data plane app layer 1146 and an Internet gateway 1134 of the data plane VCN 1118. The app subnet 1126 may be communicatively connected to a service gateway 1136 of the data plane VCN 1118 and a NAT gateway 1138 of the data plane VCN 1118. The data plane data layer 1150 may also include a DB subnet 1130 that may be communicatively connected to the app subnet 1126 of the data plane app layer 1146.

[0227] The Internet gateway 1134 of the control plane VCN 1116 and the Internet gateway 1134 of the data plane VCN 1118 may be communicatively connected to a metadata management service 1152, which may be communicatively connected to the public Internet 1154. The public Internet 1154 may be communicatively connected to a NAT gateway 1138 of the control plane VCN 1116 and the NAT gateway 1138 of the data plane VCN 1118. The service gateway 1136 of the VCN 1118 and the data plane VCN 1118 may be communicatively connected to cloud services 1156.

[0228] In some examples, a service gateway 1136 of the control plane VCN 1116 or the data plane VCN 1118 can make application programming interface (API) calls to cloud services 1156 without going through the public Internet 1154. The API calls from the service gateway 1136 to the cloud services 1156 can be one-way. The service gateway 1136 can make API calls to the cloud services 1156, and the cloud services 1156 can send request data to the service gateway 1136. However, the cloud services 1156 may not initiate the API calls to the service gateway 1136.

[0229] In some examples, the secure host tenancy 1104 may be directly connected to a service tenancy 1119, which may be an orphan. The secure host subnet 1108 may communicate with an SSH subnet 1114 via an LPG 1110, which allows bidirectional communication with the orphan system. By connecting the secure host subnet 1108 to the SSH subnet 1114, the secure host subnet 1108 may access other entities in the service tenancy 1119.

[0230] The control plane VCN 1116 allows users of the service tenancy 1119 to configure or provision desired resources. The desired resources provisioned in the control plane VCN 1116 may be deployed or used in the data plane VCN 1118. In some examples, the control plane VCN 1116 may be isolated from the data plane VCN 1118, and the data plane mirror app layer 1140 of the control plane VCN 1116 can communicate with the data plane app layer 1146 of the data plane VCN 1118 via a VNIC 1142, which may be included in the data plane mirror app layer 1140 and the data plane app layer 1146.

[0231] In some examples, a user or customer of the system can make a request, such as, for example, a create, read, update, or delete (CRUD) operation, via the public Internet 1154, which can communicate the request to a metadata management service 1152. The metadata management service 1152 can communicate the request to the control plane VCN 1116 via an Internet gateway 1134. The request can be received by a LB subnet 1122 included in the control plane DMZ layer 1120. The LB subnet 1122 can determine that the request is valid, and in response to this determination, the LB subnet 1122 can send the request to an app subnet 1126 included in the control plane app layer 1124. If the request is validated and requires a call to the public Internet 1154, the call to the public Internet 1154 can be sent to a NAT gateway 1138, which can make the call to the public Internet 1154. A memory for storing the request can be stored in the DB subnet 1130.

[0232] In some examples, the data plane mirror app layer 1140 can facilitate direct communication between the control plane VCN 1116 and the data plane VCN 1118. For example, it may be desirable for a change, update, or other suitable modification to a configuration to be applied to resources included in the data plane VCN 1118. The control plane VCN 1116 can communicate directly with resources included in the data plane VCN 1118 via the VNIC 1142 and thus can effect the change, update, or other suitable modification to the configuration.

[0233] In some embodiments, the control plane VCN 1116 and the data plane VCN 1118 may be included in the service tenancy 1119. In this case, a user or customer of the system may not own or operate either the control plane VCN 1116 or the data plane VCN 1118. Instead, an IaaS provider may own or operate the control plane VCN 1116 and the data plane VCN 1118, both of which may be included in the service tenancy 1119. This embodiment may prevent a user or customer from interacting with other users' resources or other customers' resources by enabling network isolation. This embodiment may also enable a user or customer of the system to store databases privately without having to rely on the public Internet 1154, which may not have the desired level of security for storage.

[0234] In another embodiment, the LB subnet 1122 included in the control plane VCN 1116 may be configured to receive signals from the service gateway 1136. In this embodiment, the control plane VCN 1116 and the data plane VCN 1118 may be configured to be called by the IaaS provider's customers without calling the public Internet 1154. The IaaS provider's customers may desire this embodiment because databases used by the customers may be stored in the service tenancy 1119, which may be controlled by the IaaS provider and isolated from the public Internet 1154.

[0235] 12 is a block diagram 1200 illustrating another example pattern of an IaaS architecture in accordance with at least one embodiment. A service operator 1202 (e.g., service operator 1102 of FIG. 11 ) may be communicatively connected to a secure host tenancy 1204 (e.g., secure host tenancy 1104 of FIG. 11 ), which may include a virtual cloud network (VCN) 1206 (e.g., VCN 1106 of FIG. 11 ) and a secure host subnet 1208 (e.g., secure host subnet 1108 of FIG. 11 ). The VCN 1206 may include a local peering gateway (LPG) 1210 (e.g., LPG 1110 of FIG. 11 ), which may be communicatively connected to a secure shell (SSH) VCN 1212 (e.g., SSH VCN 1112 of FIG. 11 ) via an LPG 1110 included in the SSH VCN 1212. The SSH VCN 2512 may include an SSH subnet 2514 (e.g., the SSH subnet 2414 in FIG. 24 ), VCN 1212 can be communicatively connected to a control plane VCN 1216 (e.g., control plane VCN 1116 in FIG. 11 ) via an LPG 1210 that is included in control plane VCN 1216. Control plane VCN 1216 may be included in a service tenancy 1219 (e.g., service tenancy 1119 in FIG. 11 ), and data plane VCN 1218 (e.g., data plane VCN 1118 in FIG. 11 ) may be included in a customer tenancy 1221, which may be owned or operated by a user or customer of the system.

[0236] The control plane VCN 1216 may include a control plane DMZ layer 1220 (e.g., the control plane DMZ layer 1120 of FIG. 11 ) that may include a LB subnet 1222 (e.g., the LB subnet 1122 of FIG. 11 ), a control plane app layer 1224 (e.g., the control plane app layer 1124 of FIG. 11 ) that may include an app subnet 1226 (e.g., the app subnet 1126 of FIG. 11 ), and a control plane data layer 1228 (e.g., the control plane data layer 1128 of FIG. 11 ) that may include a database (DB) subnet 1230 (e.g., similar to the DB subnet 1130 of FIG. 11 ). The LB subnet 1222 included in the control plane DMZ layer 1220 may include the app subnet 1226 included in the control plane app layer 1224 and an Internet gateway 1234 (e.g., the Internet gateway 1234 of FIG. 11 ) that may be included in the control plane VCN 1216. 11 ) and a network address translation (NAT) gateway 1238 (e.g., NAT gateway 1138 in FIG. 11 ) included in the control plane data layer 1228. The control plane VCN 1216 may include the service gateway 1236 and the NAT gateway 1238.

[0237] The control plane VCN 1216 can include a data plane mirror app layer 1240 (e.g., data plane mirror app layer 1140 of FIG. 11 ), which can include an app subnet 1226. The app subnet 1226 included in the data plane mirror app layer 1240 can include a virtual network interface controller (VNIC) 1242 (e.g., VNIC 1142) on which a compute instance 1244 (e.g., similar to compute instance 1144 of FIG. 11 ) can run. The compute instance 1244 can facilitate communication between the app subnet 1226 of the data plane mirror app layer 1240 and the app subnet 1226 that can be included in the data plane app layer 1246 (e.g., data plane app layer 1146 of FIG. 11 ) via the VNIC 1242 included in the data plane mirror app layer 1240 and the VNIC 1242 included in the data plane app layer 1246.

[0238] An Internet gateway 1234 included in the control plane VCN 1216 may be communicatively connected to a metadata management service 1252 (e.g., metadata management service 1152 of FIG. 11 ), which may be communicatively connected to a public Internet 1254 (e.g., public Internet 1154 of FIG. 11 ). The public Internet 1254 may be communicatively connected to a NAT gateway 1238 included in the control plane VCN 1216. A service gateway 1236 included in the control plane VCN 1216 may be communicatively connected to cloud services 1256 (e.g., cloud services 1156 of FIG. 11 ).

[0239] In some examples, the data plane VCN 1218 may be included in the customer tenancy 1221. In this case, the IaaS provider may provide a control plane VCN 1216 for each customer, and the IaaS provider may configure a unique compute instance 1244 for each customer, which is included in the service tenancy 1219. Each compute instance 1244 may allow communication between the control plane VCN 1216 included in the service tenancy 1219 and the data plane VCN 1218 included in the customer tenancy 1221. The compute instance 1244 may allow resources provisioned in the control plane VCN 1216 included in the service tenancy 1219 to be deployed or used in the data plane VCN 1218 included in the customer tenancy 1221.

[0240] In another example, an IaaS provider customer may have a database that resides in customer tenancy 1221. In this example, control plane VCN 1216 may include a data plane mirror app layer 1240, which may include app subnet 1226. The data plane mirror app layer 1240 may reside in data plane VCN 1218, but the data plane mirror app layer 1240 may not reside in data plane VCN 1218. That is, the data plane mirror app layer 1240 may have access to customer tenancy 1221, but the data plane mirror app layer 1240 may not reside in data plane VCN 1218 and may not be owned or operated by the IaaS provider customer. The data plane mirror app layer 1240 may be configured to make calls to the data plane VCN 1218, but may not be configured to make calls to any entity included in control plane VCN 1216. A customer may desire to deploy or use resources in a data plane VCN 1218 that are provisioned to a control plane VCN 1216, and the data plane mirror app layer 1240 may facilitate the desired deployment or other use of the customer's resources.

[0241] In some embodiments, the IaaS provider's customer can apply filters to the data plane VCN 1218. In this embodiment, the customer can determine what the data plane VCN 1218 can access, and the customer can restrict access from the data plane VCN 1218 to the public Internet 1254. The IaaS provider may not be able to apply filters or control access from the data plane VCN 1218 to any external networks or databases. Applying filters and controls to the data plane VCN 1218 that the customer includes in the customer tenancy 1221 can help isolate the data plane VCN 1218 from other customers and the public Internet 1254.

[0242] In some embodiments, cloud services 1256 can be called by service gateway 1236 to access services that may not be present on the public Internet 1254, on the control plane VCN 1216, or on the data plane VCN 1218. The connection between cloud services 1256 and the control plane VCN 1216 or the data plane VCN 1218 may not be live or continuous. Cloud services 1256 may be present on another network owned or operated by the IaaS provider. Cloud services 1256 may be configured to receive calls from service gateway 1236 and may be configured not to receive calls from the public Internet 1254. Some cloud services 1256 may be isolated from other cloud services 1256, and control plane VCN 1216 may be isolated from cloud services 1256 that may not be located in the same region as control plane VCN 1216. For example, control plane VCN 1216 may be located in “Region 1” and cloud service “Deployment 11” may be located in “Region 1” and “Region 2”. If a call to deployment 11 is made by a service gateway 1236 included in a control plane VCN 1216 located in region 1, the call may be sent to deployment 11 in region 1. In this example, control plane VCN 1216 or deployment 11 in region 1 may not be communicatively connected to deployment 11 in region 2 or may not communicate with deployment 11 in region 2.

[0243] 13 is a block diagram 1300 illustrating another example pattern of an IaaS architecture in accordance with at least one embodiment. A service operator 1302 (e.g., service operator 1102 of FIG. 11 ) may be communicatively connected to a secure host tenancy 1304 (e.g., secure host tenancy 1104 of FIG. 11 ), which may include a virtual cloud network (VCN) 1306 (e.g., VCN 1106 of FIG. 11 ) and a secure host subnet 1308 (e.g., secure host subnet 1108 of FIG. 11 ). The VCN 1306 may include an LPG 1310 (e.g., LPG 1110 of FIG. 11 ), which may be communicatively connected to an SSH VCN 1312 (e.g., SSH VCN 1112 of FIG. 11 ) via an LPG 1310 included in the SSH VCN 1312. SSH VCN 1312 can include an SSH subnet 1314 (e.g., SSH subnet 1114 in FIG. 11 ), which may be communicatively connected to a control plane VCN 1316 (e.g., control plane VCN 1116 in FIG. 11 ) via an LPG 1310 included in control plane VCN 1316, and may be communicatively connected to a data plane VCN 1318 (e.g., data plane 1118 in FIG. 11 ) via an LPG 1310 included in data plane VCN 1318. The control plane VCN 1316 and the data plane VCN 1318 may be associated with a service tenancy 1319 (e.g., service tenancy 1114 in FIG. 11 ). may be included in the service provider 1119).

[0244] The control plane VCN 1316 may include a control plane DMZ layer 1320 (e.g., the control plane DMZ layer 1120 of FIG. 11 ) that may include a load balancer (LB) subnet 1322 (e.g., the LB subnet 1122 of FIG. 11 ), a control plane app layer 1324 (e.g., the control plane app layer 1124 of FIG. 11 ) that may include an app subnet 1326 (e.g., similar to the app subnet 1126 of FIG. 11 ), and a control plane data layer 1328 (e.g., the control plane data layer 1128 of FIG. 11 ) that may include a DB subnet 1330. The LB subnet 1322 included in the control plane DMZ layer 1320 may be communicatively connected to the app subnet 1326 included in the control plane app layer 1324 and to an Internet gateway 1334 (e.g., the Internet gateway 1134 of FIG. 11 ) that may be included in the control plane VCN 1316. The app subnet 1326 may be communicatively connected to a DB subnet 1330 included in the control plane data layer 1328, and to a service gateway 1336 (e.g., the service gateway in FIG. 11 ) and a network address translation (NAT) gateway 1338 (e.g., the NAT gateway 1138 in FIG. 11 ). The control plane VCN 1316 may include the service gateway 1336 and the NAT gateway 1338.

[0245] The data plane VCN 1318 may include a data plane app layer 1346 (e.g., data plane app layer 1146 in FIG. 11 ), a data plane DMZ layer 1348 (e.g., data plane DMZ layer 1148 in FIG. 11 ), and a data plane data layer 1350 (e.g., data plane data layer 1150 in FIG. 11 ). The data plane DMZ layer 1348 may include a LB subnetwork 1322 that may be communicatively connected to the data plane app layer 1346 included in the data plane VCN 1318 and a trusted app subnetwork 1360 and an untrusted app subnetwork 1362 of an Internet gateway 1334. The trusted app subnetwork 1360 may be communicatively connected to a service gateway 1336 included in the data plane VCN 1318, a NAT gateway 1338 included in the data plane VCN 1318, and a DB subnetwork 1330 included in the data plane data layer 1350. The untrusted app subnet 1362 may be communicatively connected to a service gateway 1336 included in the data plane VCN 1318 and to a DB subnet 1330 included in the data plane data layer 1350. The data plane data layer 1350 may include a DB subnet 1330, which may be communicatively connected to a service gateway 1336 included in the data plane VCN 1318.

[0246] The untrusted app subnet 1362 may include one or more primary VNICs 1364(1)-(N) that may be communicatively connected to tenant virtual machines (VMs) 1366(1)-(N). Each tenant VM 1366(1)-(N) may be communicatively connected to a respective app subnet 1367(1)-(N) that may be included in a respective container egress VCN 1368(1)-(N) that may be included in a respective customer tenancy 1370(1)-(N). Each secondary VNIC 1372(1)-(N) may facilitate communication between the untrusted app subnet 1362 included in the data plane VCN 1318 and the app subnet included in the container egress VCN 1368(1)-(N). Each container egress VCN 1368(1)-(N) may include a NAT gateway 1338 that may be communicatively connected to the public Internet 1354 (e.g., public Internet 1154 of FIG. 11 ).

[0247] The Internet Gateway 1334 in the control plane VCN 1316 and the Internet Gateway 1334 in the data plane VCN 1318 are public 11 ), which may be communicatively connected to the public Internet 1354. The public Internet 1354 may be communicatively connected to a NAT gateway 1338 included in the control plane VCN 1316 and a NAT gateway 1338 included in the data plane VCN 1318. The service gateway 1336 included in the control plane VCN 1316 and the service gateway 1336 included in the data plane VCN 1318 may be communicatively connected to cloud services 1356.

[0248] In some embodiments, data plane VCN 1318 may be consolidated into customer tenancies 1370. This consolidation may be useful or desirable for an IaaS provider's customers in some cases, such as when they may want support when executing code. Customers may provide code that, when executed, may be disruptive, may communicate with other customer resources, or may cause undesirable effects. Thus, the IaaS provider may determine whether or not to run code that a customer has provided to the IaaS provider.

[0249] In some examples, an IaaS provider's customer can grant temporary network access to the IaaS provider and request a function to be added to the data plane app layer 1346. The code to perform the function may be executed in the VMs 1366(1)-(N) but cannot be configured to run elsewhere on the data plane VCN 1318. Each VM 1366(1)-(N) may be connected to one customer tenancy 1370. Each container 1371(1)-(N) included in the VMs 1366(1)-(N) may be configured to execute code. In this case, there may be double isolation (e.g., the containers 1371(1)-(N) execute code, and the containers 1371(1)-(N) may be included in at least the VMs 1366(1)-(N) included in the untrusted app subnet 1362), which can help prevent erroneous or unwanted code from damaging the IaaS provider's network or from damaging a different customer's network. Containers 1371(1)-(N) may be communicatively connected to customer tenancy 1370 and may be configured to send or receive data from customer tenancy 1370. Containers 1371(1)-(N) may not be configured to send or receive data from any other entity in data plane VCN 1318. Once code execution is complete, the IaaS provider can kill or discard containers 1371(I)-(N).

[0250] In some embodiments, trusted app subnet 1360 can execute code that may be owned or operated by the IaaS provider. In this embodiment, trusted app subnet 1360 may be communicatively connected to DB subnet 1330 and configured to perform CRUD operations in DB subnet 1330. Untrusted app subnet 1362 may be communicatively connected to DB subnet 1330, but in this embodiment, untrusted app subnet 1362 may be configured to perform read operations in DB subnet 1330. Containers 1371(1)-(N) included in each customer's VMs 1366(1)-(N) that can execute code from the customer may not be communicatively connected to DB subnet 1330.

[0251] In other embodiments, the control plane VCN 1316 and the data plane VCN 1318 may not be directly communicatively coupled. In this embodiment, there may be no direct communication between the control plane VCN 1316 and the data plane VCN 1318. However, there may be indirect communication by at least one method. An LPG 1310 may be established by an IaaS provider that can communicate with the control plane VCN 1316 or the data plane VCN 1318. In another example, the control plane VCN 1316 or the data plane VCN 1318 can make a call to a cloud service 1356 through a service gateway 1336. For example, a call from the control plane VCN 1316 to the cloud service 1356 can include a request for a service that can communicate with the data plane VCN 1318.

[0252] 14 is a block diagram 1400 illustrating another example pattern of an IaaS architecture in accordance with at least one embodiment. A service operator 1402 (e.g., service operator 1102 of FIG. 11 ) may be communicatively connected to a secure host tenancy 1404 (e.g., secure host tenancy 1104 of FIG. 11 ), which may include a virtual cloud network (VCN) 1406 (e.g., VCN 1106 of FIG. 11 ) and a secure host subnet 1408 (e.g., secure host subnet 1108 of FIG. 11 ). The VCN 1406 may include an LPG 1410 (e.g., LPG 1110 of FIG. 11 ), which may be communicatively connected to an SSH VCN 1412 (e.g., SSH VCN 1112 of FIG. 11 ) via an LPG 1410 included in the SSH VCN 1412. SSH VCN 1412 can include an SSH subnet 1414 (e.g., SSH subnet 1114 in FIG. 11 ), which may be communicatively connected to a control plane VCN 1416 (e.g., control plane VCN 1116 in FIG. 11 ) via an LPG 1410 included in control plane VCN 1416, and may be communicatively connected to a data plane VCN 1418 (e.g., data plane 1118 in FIG. 11 ) via an LPG 1410 included in data plane VCN 1418. Control plane VCN 1416 and data plane VCN 1418 may be included in service tenancy 1419 (e.g., service tenancy 1119 in FIG. 11 ).

[0253] The control plane VCN 1416 may include a control plane DMZ layer 1420 (e.g., the control plane DMZ layer 1120 of FIG. 11 ) that may include a LB subnet 1422 (e.g., the LB subnet 1122 of FIG. 11 ), a control plane app layer 1424 (e.g., the control plane app layer 1124 of FIG. 11 ) that may include an app subnet 1426 (e.g., the app subnet 1126 of FIG. 11 ), and a control plane data layer 1428 (e.g., the control plane data layer 1128 of FIG. 11 ) that may include a DB subnet 1430 (e.g., the DB subnet 1330 of FIG. 13 ). The LB subnet 1422 included in the control plane DMZ layer 1420 may be communicatively connected to the app subnet 1426 included in the control plane app layer 1424 and to an Internet gateway 1434 (e.g., the Internet gateway 1134 of FIG. 11 ) that may be included in the control plane VCN 1416. The app subnet 1426 may be communicatively connected to a DB subnet 1430 included in the control plane data layer 1428, and to a service gateway 1436 (e.g., the service gateway in FIG. 11) and a network address translation (NAT) gateway 1438 (e.g., the NAT gateway 1138 in FIG. 11). The control plane VCN 1416 may include the service gateway 1436 and the NAT gateway 1438.

[0254] The data plane VCN 1418 can include a data plane app layer 1446 (e.g., data plane app layer 1146 in FIG. 11 ), a data plane DMZ layer 1448 (e.g., data plane DMZ layer 1148 in FIG. 11 ), and a data plane data layer 1450 (e.g., data plane data layer 1150 in FIG. 11 ). The data plane DMZ layer 1448 communicates with the trusted app subnets 1460 (e.g., trusted app subnet 1360 in FIG. 13 ) and untrusted app subnets 1462 (e.g., untrusted app subnet 1362 in FIG. 13 ) of the data plane app layer 1446 and the Internet gateway 1434 included in the data plane VCN 1418. The trusted app subnet 1460 may include a LB subnet 1422 that may be communicatively connected to a service gateway 1436 included in the data plane VCN 1418, a NAT gateway 1438 included in the data plane VCN 1418, and a DB subnet 1430 included in the data plane data layer 1450. The untrusted app subnet 1462 may be communicatively connected to a service gateway 1436 included in the data plane VCN 1418, and a DB subnet 1430 included in the data plane data layer 1450. The data plane data layer 1450 may include a DB subnet 1430 that may be communicatively connected to a service gateway 1436 included in the data plane VCN 1418.

[0255] The untrusted app subnet 1462 may include primary VNICs 1464(1)-(N) that may be communicatively connected to tenant virtual machines (VMs) 1466(1)-(N) that reside in the untrusted app subnet 1462. Each tenant VM 1466(1)-(N) may execute code in a respective container 1467(1)-(N) and may be communicatively connected to an app subnet 1426 that may be included in a data plane app layer 1446 that may be included in a container egress VCN 1468. Each secondary VNIC 1472(1)-(N) may facilitate communication between the untrusted app subnet 1462 included in the data plane VCN 1418 and the app subnet included in the container egress VCN 1468. The container egress VCN may include a NAT gateway 1438 that may be communicatively connected to the public Internet 1454 (e.g., public Internet 1154 of FIG. 11 ).

[0256] The internet gateway 1434 included in the control plane VCN 1416 and the internet gateway 1434 included in the data plane VCN 1418 may be communicatively connected to a metadata management service 1452 (e.g., metadata management system 1152 of FIG. 11 ), which may be communicatively connected to the public internet 1454. The public internet 1454 may be communicatively connected to a NAT gateway 1438 included in the control plane VCN 1416 and the NAT gateway 1438 included in the data plane VCN 1418. The service gateway 1436 included in the control plane VCN 1416 and the service gateway 1436 included in the data plane VCN 1418 may be communicatively connected to cloud services 1456.

[0257] In some examples, the pattern illustrated by the architecture of block diagram 1400 of FIG. 14 may be considered an exception to the pattern illustrated by the architecture of block diagram 1300 of FIG. 13 and may be desirable for an IaaS provider's customer when the IaaS provider cannot directly communicate with the customer (e.g., in a non-connected region). The customer may have real-time access to each container 1467(1)-(N) contained in each customer's VM 1466(1)-(N). The containers 1467(1)-(N) may be configured to call each secondary VNIC 1472(1)-(N) contained in the app subnet 1426 of the data plane app tier 1446 that may be contained in the container egress VCN 1468. The secondary VNIC 1472(1)-(N) may send the call to a NAT gateway 1438 that may send the call to the public Internet 1454. In this example, containers 1467(1)-(N) that a customer can access in real time may be isolated from control plane VCN 1416 and may be isolated from other entities included in data plane VCN 1418. Containers 1467(1)-(N) may also be isolated from resources of other customers.

[0258] In another example, a customer can use containers 1467(1)-(N) to invoke cloud services 1456. In this example, the customer can execute code in containers 1467(1)-(N) that requests services from cloud services 1456. The container 1467(1)-(N) may send the request to a secondary VNIC 1472(1)-(N), which may send the request to a NAT gateway, which may send the request to the public Internet 1454. The public Internet 1454 may send the request to a LB subnet 1422 included in the control plane VCN 1416 via an Internet gateway 1434. In response to determining that the request is valid, the LB subnet 1422 may send the request to an app subnet 1426, which may send the request to a cloud service 1456 via a service gateway 1436.

[0259] It should be noted that the illustrated IaaS architectures 1100, 1200, 1300, and 1400 may include elements other than those illustrated, and the illustrated embodiments are only some examples of cloud infrastructure systems that may incorporate embodiments of the present disclosure. In other embodiments, the IaaS system may have more or fewer elements than those illustrated, may combine two or more elements, or may have a different configuration or arrangement of elements.

[0260] In certain embodiments, the IaaS system described in this disclosure may include a suite of application, middleware, and database services that are delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. One example of such an IaaS system is the Oracle® Cloud Infrastructure (OCI) offered by the present assignee.

[0261] 15 illustrates an exemplary computer system 1500 upon which various embodiments may be implemented. System 1500 may be used to implement any of the computer systems described above. As illustrated, computer system 1500 includes a processing unit 1504 that communicates with a number of peripheral subsystems via a bus subsystem 1502. These peripheral subsystems may include a processing acceleration unit 1506, an I / O subsystem 1508, a storage subsystem 1518, and a communication subsystem 1524. The storage subsystem 1518 includes a tangible computer-readable storage medium 1522 and a system memory 1510.

[0262] Bus subsystem 1502 provides a mechanism for allowing the various components and subsystems of computer system 1500 to communicate with each other as intended. Although bus subsystem 1502 is shown diagrammatically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1502 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard, and the like.

[0263] The processing unit 1504, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 1500. The processing unit 1504 may include one or more processors. These processors may include single-core or multi-core processors. In particular embodiments, the processing unit 1504 may be implemented as one or more independent processing units 1532 and / or 1534 with single-core or multi-core processors included in each processing unit. In other embodiments, the processing unit 1504 may be implemented as a quad-core processing unit formed by integrating two dual-core processors onto a single chip.

[0264] In various embodiments, the processing unit 1504 can execute various programs in response to program code and can maintain multiple programs or processes executing simultaneously. At any given time, some or all of the program code being executed may reside in the processor 1504 and / or in the storage subsystem 1518. The processor 1504 can provide various functionality as discussed above through appropriate programming. The computer system 1500 may further include a processing acceleration unit 1506, which may include a digital signal processor (DSP), a special purpose processor, and / or the like.

[0265] The I / O subsystem 1508 can include user interface input devices and user interface output devices. The user interface input devices can include a pointing device such as a keyboard, a mouse or a trackball, a touchpad or touch screen integrated into a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a voice input device with a voice command recognition system, a microphone, and other types of input devices. The user interface input devices can also include a motion sensing and / or gesture recognition device, such as a Microsoft Kinect® motion sensor. The Microsoft Kinect® motion sensor can control and interact with input devices, such as a Microsoft Xbox® 360 game controller, through a natural user interface that utilizes gestures and voice commands. The user interface input devices can also include an eye gesture recognition device, such as a Google Glass® blink detector. The Google Glass® blink detector detects the user's eye activity (e.g., "blinking" when taking a picture and / or selecting a menu) and translates the eye activity into input that is entered into an input device (e.g., Google Glass®). Additionally, the user interface input devices may include a voice recognition detection device that allows a user to interact with a voice recognition system (eg, the Siri® navigator) via voice commands.

[0266] User interface input devices may also include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, game pads, graphics tablets, audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser range finders, and eye-tracking devices. User interface input devices may also include medical imaging input devices such as, for example, computed tomography devices, magnetic resonance imaging devices, ultrasound emission tomography devices, and medical ultrasound devices. User interface input devices may also include audio input devices such as, for example, MIDI keyboards and electronic musical instruments.

[0267] User interface output devices may include non-visual displays such as a display subsystem, indicator lights, or audio output devices. The display subsystem may be a flat panel device using a cathode ray tube (CRT), liquid crystal display (LCD) or plasma display, a projection device, a touch screen, etc. In general, when the term "output device" is used, it is intended to include all possible types of devices and mechanisms for outputting information from computer system 1500 to a user or to another computer. For example, user interface output devices include, but are not limited to, various display devices that visually convey text, images, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0268] Computer system 1500 may include a storage subsystem 1518. The storage subsystem 1518 comprises software elements, which are illustratively located in system memory 1510. The system memory 1510 may store program instructions loadable and executable by the processing unit 1504, as well as data generated by the execution of these programs.

[0269] Depending on the configuration and type of computer system 1500, the system memory 1510 may be volatile memory (e.g., Random Access Memory (RAM)) and / or non-volatile memory (e.g., Read-Only Memory (ROM), flash memory). Typically, RAM contains data and / or program modules that are immediately accessible to the processing unit 1504 and / or that are currently being operated on and executed by the processing unit 1504. In some implementations, the system memory 1510 may be static random access memory (SRAM) or non-volatile memory (e.g., Read-Only Memory (ROM), flash memory). In some implementations, the memory may include multiple different types of memory, such as a dynamic random access memory (DRAM) or a dynamic random access memory (DRAM). A Basic Input / Output System (BIOS), which contains the basic routines that help to transfer information between elements within computer system 1500, such as during startup; Typically stored in ROM. By way of example and not limitation, system memory 1510 stores application programs 1512, program data 1514, which may include client applications, web browsers, mid-tier applications, relational database management systems (RDBMS), and the like. Also shown is an operating system 1516. By way of example, the operating system 1516 may be any of the following: Microsoft Windows®, Apple Macintosh® ) and / or various versions of the Linux operating system, various commercially available UNIX or UNIX-like operating systems (including various GNU / Linux operating systems, Google Chrome (including, but not limited to, iOS, Windows Phone, Android OS, BlackBerry 15 OS and mobile operating systems such as Palm OS A rating system may be included.

[0270] The storage subsystem 1518 may also provide a tangible computer-readable storage medium for storing the basic programming and data constructs that provide the functionality of some embodiments. Software (programs, code modules, instructions) that, when executed by a processor, provide the above-described functionality may be stored in the storage subsystem 1518. These software modules or instructions may be executed by the processing unit 1504. The storage subsystem 1518 may also provide a repository for storing data used in accordance with the present disclosure.

[0271] Storage subsystem 1500 may also include a computer readable storage medium reader 1520 that may be further connected to a computer readable storage medium 1522. Computer readable storage medium 1522 may comprehensively represent remote, local, fixed and / or removable storage devices, in addition to storage media for temporarily and / or permanently containing, storing, transmitting and retrieving computer readable information together with, or in combination with, system memory 1510 as appropriate.

[0272] Also, the computer readable storage medium 1522 containing the code or portions of the code may be used in the art. The information may include any suitable medium known or used in the art. Such media includes storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing and / or transmitting information. This may include tangible computer readable storage media, such as RAM, ROM, Electronically Erasable Programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, Digital Versatile Disk (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage, or other tangible computer readable media. This may also include intangible computer readable media, such as data signals, data transmissions, or other media that can be used to transmit the desired information and that can be accessed by computer system 1500.

[0273] By way of example, the computer readable storage medium 1522 may be a hard disk drive that reads from or writes to non-removable, non-volatile magnetic media, a magnetic disk drive that reads from or writes to a removable, non-volatile magnetic disk, and a CD The computer-readable storage medium 1522 may include an optical disk drive that reads from or writes to removable non-volatile optical disks such as ROM, DVD and Blu-ray disks or other optical media. The computer-readable storage medium 1522 may include, but is not limited to, Zip drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD disks, digital video tapes, and the like. The computer-readable storage medium 1522 may also include flash memory-based SSDs, enterprise flash drives, solid-state drives (SSDs) based on non-volatile memory such as solid-state ROM, solid-state RAM, dynamic RAM, and the like. SSDs based on volatile memory such as dynamic RAM, static RAM, DRAM-based SSDs, Magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. The disk drives and their associated computer-readable media can provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1500.

[0274] The communications subsystem 1524 provides an interface with other computer systems and networks. The communications subsystem 1524 serves as an interface for receiving data from other systems and transmitting data from the computer system 1500 to other systems. For example, the communications subsystem 1524 may enable the computer system 1500 to connect to one or more devices via the Internet. In some embodiments, the communications subsystem 1524 includes a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., using cellular technology, advanced data network technologies such as 3G, 4G, or EDGE (enhanced data rates for global evolution)), a WiFi (Internet of Things) transceiver component for accessing wireless voice and / or data networks (e.g., using cellular technology, advanced data network technologies such as 3G, 4G, or EDGE (enhanced data rates for global evolution)), and / or a wireless network (e.g., WiFi (Internet of Things) transceiver component for accessing wireless voice and / or data networks (e.g., using cellular technology, advanced data network technologies such as 3G, 4G, or EDGE (enhanced data rates for global evolution)). EE802.11 family of standards or other mobile communications technologies or any combination thereof), Global Positioning System (GPS) receiver components , and / or other components. In some embodiments, the communications subsystem 1524 can provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0275] In some embodiments, the communications subsystem 1524 also communicates structured and / or unstructured data feeds 1526, event streams 1528, event updates, and the like on behalf of one or more users that may be using the computer system 1500. An incoming communication may be received in the form of, for example, 1530 .

[0276] As an example, the communications subsystem 1524 may provide Twitter feeds, Facebook updates, Rich Site Summary (RSS) feeds, and other The social network may be configured to receive data feeds 1526, such as web feeds, in real time from users of social networks and / or other communication services, and / or to receive real-time updates from one or more third party information sources.

[0277] The communications subsystem 1524 may also be configured to receive data in the form of a continuous data stream, which may include an event stream 1528 of real-time events that may be continuous or may be essentially unbounded with no clear ends and / or event updates 1530. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like.

[0278] The communications subsystem 1524 may also be configured to output structured and / or unstructured data feeds 1526, event streams 1528, event updates 1530, etc. to one or more databases that may be in communication with one or more streaming data source computers coupled to the computer system 1500.

[0279] Computer system 1500 may be one of a variety of types, including a handheld mobile device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head mounted display), a PC, a workstation, a mainframe, a kiosk, a server rack, or other data processing system.

[0280] Because computers and networks are constantly evolving, the illustrated computer system 1500 is intended as a specific example only. Many other configurations having more or fewer components than the illustrated system are possible. For example, customized hardware may also be used and / or specific elements may be implemented in hardware, firmware, software (including applets), or a combination. Furthermore, connections to other computing devices, such as network input / output devices, may be utilized. Based on the disclosure and teachings provided in this disclosure, one of ordinary skill in the art will appreciate other means and / or methods for implementing various embodiments.

[0281] Although specific embodiments have been described, various changes, modifications, alternative configurations, and equivalents are encompassed within the scope of the disclosure. The embodiments are not limited to operating in a particular data processing environment, but can freely operate in multiple data processing environments. Furthermore, while the embodiments have been described using a particular sequence of actions and steps, it will be apparent to those skilled in the art that the scope of the disclosure is not limited to the sequence of actions and steps described. Various features and aspects of the above-described embodiments can be used individually or jointly.

[0282] Additionally, while embodiments have been described using a particular combination of hardware and software, it should be appreciated that other combinations of hardware and software are within the scope of the present disclosure. Embodiments may be implemented using only hardware, only software, or a combination thereof. The various processes described in this disclosure may be implemented using a single process, such as a single process for performing a single process, or a single process for performing a single process. Processes may run on the same processor or on any combination of different processors. Thus, when a component or module is described as being configured to perform a particular process, that configuration may be achieved, for example, by designing electronic circuitry to perform the process, by programming a programmable electronic circuit (such as a microprocessor) to perform the process, or by a combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for communicating between processes. Different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0283] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will be apparent, however, that additions, subtractions, deletions and other modifications and changes may be made without departing from the broad spirit and scope defined by the appended claims. Thus, although specific embodiments of the disclosure have been described, these embodiments are not intended to be limiting. Various modifications and equivalents thereof are intended to be encompassed within the scope of the appended claims.

[0284] The indefinite article "a" / used in the context of describing this disclosure (especially in the context of the claims) "An," the definite article "the" and similar references are used herein unless otherwise stated in this disclosure or by context. Unless clearly indicated otherwise, the terms "comprising," "having," "including," and "including" are to be construed as including both the singular and the plural. and "containing" should be construed as open ended terms (i.e., meaning "including, but not limited to") unless otherwise noted. The term "connected" should be construed as partly or wholly contained within, attached to, or joined together, even if there is something intervening. In this disclosure, the recitation of ranges of values ​​is intended merely as a shorthand method of referring to each individual value contained within the range, and each individual value is incorporated into this disclosure as if it were set forth individually in this disclosure, unless otherwise noted in this disclosure. All methods described in this disclosure can be performed in any suitable order unless otherwise noted in this disclosure or the context clearly indicates otherwise. In this disclosure, the use of any and all examples or exemplary language (e.g., "such as") is intended to make the embodiments more clear and does not limit the scope of the disclosure, unless otherwise noted. No language in the specification should be construed as indicating any non-claimed element essential to the practice of the disclosure.

[0285] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is intended to be understood in context as being generally used to indicate that an item, term, etc. may be either X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z), unless otherwise noted. Thus, such disjunctive language is not generally intended to, and does not imply, that a particular embodiment requires that at least one of X, at least one of Y, or at least one of Z be present.

[0286] Preferred embodiments of the present disclosure, including the best mode known for carrying out the present disclosure, are described herein. Variations of these preferred embodiments will become apparent to those skilled in the art upon reading the foregoing description. Those skilled in the art may adopt such variations as appropriate, and the present disclosure may be carried out in ways other than those specifically described herein. Accordingly, this disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by this disclosure unless otherwise indicated herein.

[0287] All references cited in this specification, including publications, patent applications, and patents, are incorporated by reference to the same extent as if each individual reference was individually and specifically indicated to be incorporated by reference and was set forth in its entirety herein.

[0288] In the foregoing specification, aspects of the disclosure have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the above-mentioned disclosure may be used individually or jointly. Moreover, the embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Thus, the specification and drawings should be regarded as illustrative rather than restrictive.< / realm>

Claims

1. 1. A method of data networking, comprising: receiving at an ingress switch a first Layer 2 RDMA packet for a first tenant among a plurality of tenants from a host machine running a plurality of computing instances of the plurality of tenants; converting the first Layer 2 RDMA packet into a first Layer 3 encapsulated packet having at least one header; and forwarding the first Layer 3 encapsulated packet to a switch fabric; the first Layer 2 RDMA packet includes a Virtual Local Area Network (VLAN) tag and a Quality-of-Service (QoS) data field; The converting step includes adding the at least one header to the first Layer 2 RDMA packet, the at least one header comprising: a virtual network identifier based on information from the VLAN tag; a QoS value based on information from the QoS data field.

2. 2. The method of claim 1, further comprising the step of modifying a congestion notification data field of the at least one header of the first Layer 3 encapsulated packet at an intermediate switch of the switch fabric in response to an indication of congestion.

3. receiving a second Layer 2 RDMA packet at the ingress switch, the second Layer 2 RDMA packet including a VLAN tag and a QoS data field; converting the second Layer 2 RDMA packet into a second Layer 3 encapsulated packet having at least one header; and forwarding the second Layer 3 encapsulated packet to the switch fabric; 2. The method of claim 1, wherein the VLAN tag of the second Layer 2 RDMA packet indicates a different VLAN than the VLAN tag of the first Layer 2 RDMA packet.

4. In the intermediate switch of the switch fabric, queuing the first Layer 3 encapsulated packet in a first queue of the intermediate switch based on the QoS value of the at least one header of the first Layer 3 encapsulated packet; and queuing the second Layer 3 encapsulated packet in a second queue of the intermediate switch, distinct from the first queue, based on the QoS value of the at least one header of the second Layer 3 encapsulated packet.

5. receiving the first Layer 3 encapsulated packet at an egress switch; decapsulating the first Layer 3 encapsulated packet to obtain the first Layer 2 RDMA packet; forwarding the first Layer 2 RDMA packet to a first computing instance based on the VLAN tag of the first Layer 2 RDMA packet; receiving the second Layer 3 encapsulated packet at the egress switch; decapsulating the second Layer 3 encapsulated packet to obtain the second Layer 2 RDMA packet; and forwarding the second Layer 2 RDMA packet to a second compute instance different from the first compute instance based on the VLAN tag of the second Layer 2 RDMA packet.

6. receiving the first Layer 3 encapsulated packet at an egress switch; decapsulating the first Layer 3 encapsulated packet to obtain the first Layer 2 RDMA packet; and forwarding the first Layer 2 RDMA packet to a first computing instance based on the VLAN tag of the first Layer 2 RDMA packet.

7. 7. The method of claim 6, further comprising setting a value of a congestion notification data field of the first Layer 2 RDMA packet based on information in a congestion notification data field of the at least one header of the first Layer 3 encapsulated packet.

8. the QoS value is a Differentiated Services Code Point (DSCP) field of an outer IP header of the first Layer 3 encapsulated packet; 8. The method of claim 1, wherein the converting step comprises copying a DSCP field of an IP header of the first Layer 2 RDMA packet to the DSCP field of the outer IP header of the first Layer 3 encapsulated packet.

9. the first Layer 3 encapsulated packet is a Virtual Extensible Local Area Network (VxLAN) packet; The method according to any one of claims 1 to 7, wherein the virtual network identifier is a Virtual Network Identifier (VNI) in a VxLAN header of the first Layer 3 encapsulated packet.

10. 1. A method of data networking, comprising: receiving a first Layer 3 encapsulated packet at an egress switch; decapsulating the first Layer 3 encapsulated packet to obtain a first Layer 2 RDMA packet; setting a value of a congestion notification data field of the first Layer 2 RDMA packet based on information from a congestion notification data field of the at least one header of the first Layer 3 encapsulated packet; and after the setting step, forwarding the first Layer 2 RDMA packet to a first compute instance executing on a host machine based on a VLAN tag of the first Layer 2 RDMA packet, the first compute instance being among a plurality of compute instances executing on the host machine.

11. receiving a second Layer 3 encapsulated packet at the egress switch; decapsulating the second Layer 3 encapsulated packet to obtain a second Layer 2 RDMA packet; and forwarding the second Layer 2 RDMA packet to a second compute instance different from the first compute instance based on a VLAN tag of the second Layer 2 RDMA packet.

12. In the egress switch, queuing the first Layer 3 encapsulated packet in a first queue of the egress switch based on a quality of service (QoS) value of an outer header of the first Layer 3 encapsulated packet; and queuing the second Layer 3 encapsulated packet in a second queue of the egress switch, different from the first queue, based on a QoS value in an outer header of the second Layer 3 encapsulated packet.

13. The method of any one of claims 10 to 12, further comprising receiving from the first computing instance a congestion notification packet directed to a source address of the first Layer 2 RDMA packet.

14. 1. A system for data networking, comprising: A switch fabric; an ingress switch, the ingress switch comprising: receiving a first Layer 2 RDMA packet for a first tenant among a plurality of tenants from a host machine executing a plurality of computing instances of the plurality of tenants; converting the first Layer 2 RDMA packet into a first Layer 3 encapsulated packet having at least one header; configured to forward the first Layer 3 encapsulated packet to the switch fabric; the first Layer 2 RDMA packet includes a virtual local area network (VLAN) tag and a quality of service (QoS) data field; The ingress switch configured to convert the first Layer 2 RDMA packet is configured to add the at least one header to the first Layer 2 RDMA packet, the at least one header comprising: a virtual network identifier based on information from the VLAN tag; and a QoS value based on information from the QoS data field.

15. 15. The system of claim 14, wherein the switch fabric comprises an intermediate switch configured to modify a congestion notification data field of the at least one header of the first Layer 3 encapsulated packet in response to an indication of congestion.

16. The ingress switch receiving a second Layer 2 RDMA packet including a VLAN tag and a QoS data field; converting the second Layer 2 RDMA packet into a second Layer 3 encapsulated packet having at least one header; configured to forward the second Layer 3 encapsulated packet to the switch fabric; 15. The system of claim 14, wherein the VLAN tag of the second Layer 2 RDMA packet indicates a different VLAN than the VLAN tag of the first Layer 2 RDMA packet.

17. The switch fabric includes a middle switch having a first queue and a second queue different from the first queue, the middle switch comprising: queuing the first Layer 3 encapsulated packet in the first queue based on the QoS value of the at least one header of the first Layer 3 encapsulated packet; 17. The system of claim 16, configured to queue the second Layer 3 encapsulated packet in the second queue based on the QoS value of the at least one header of the second Layer 3 encapsulated packet.

18. The egress switch further includes: receiving the first Layer 3 encapsulated packet; decapsulating the first Layer 3 encapsulated packet to obtain the first Layer 2 RDMA packet; setting a value of a congestion notification data field of the first Layer 2 RDMA packet based on information in a congestion notification data field of the at least one header of the first Layer 3 encapsulated packet; 18. The system of claim 14, configured to forward the first Layer 2 RDMA packet to a first computing instance based on the VLAN tag of the first Layer 2 RDMA packet.

19. 1. A non-transitory computer-readable medium storing a plurality of instructions executable by one or more processors, the plurality of instructions including instructions that, when executed by the one or more processors, cause the one or more processors to: receiving a first layer 2 RDMA packet from a host machine running multiple compute instances for multiple tenants; converting the first Layer 2 RDMA packet into a first Layer 3 encapsulated packet having at least one header; forwarding the first Layer 3 encapsulated packet to a switch fabric; the first Layer 2 RDMA packet includes a virtual local area network (VLAN) tag and a quality of service (QoS) data field; The instructions that, when executed by the one or more processors, cause the one or more processors to convert, cause the one or more processors to add the at least one header to the first Layer 2 RDMA packet, the at least one header comprising: a virtual network identifier based on information from the VLAN tag; and a QoS value based on information from the QoS data field.

20. the QoS value is a Differentiated Services Code Point (DSCP) field of an outer IP header of the first Layer 3 encapsulation packet; 20. The non-transitory computer-readable medium of claim 19, wherein the instructions that, when executed by the one or more processors, cause the one or more processors to convert, cause the one or more processors to copy a DSCP field of an IP header of the first Layer 2 RDMA packet to the DSCP field of the outer IP header of the first Layer 3 encapsulated packet.

21. 21. The non-transitory computer-readable medium of claim 19, wherein the first Layer 2 RDMA packet is a RoCEv2 packet.

Images