Adaptive equal cost multipath packet routing

Abstract

The invention discloses adaptive equal cost multipath packet routing. A device, a communication system and a method are provided. In one example, a system for routing traffic is described, the system including circuitry to receive a plurality of packets associated with a destination, determine a plurality of containers based on the destination of the packets, where each container of the plurality of containers is associated with a respective port of a plurality of ports, and for each of the plurality of packets, selecting one of the plurality of containers and forwarding the packets through the port associated with the selected container. The circuitry further receives the adaptive route notification packet through a first port associated with the first container, and in response, associates the first container with a second port and disassociates the first container from the first port.

Description

Technical Field

[0001] The present disclosure is generally directed to networking, and more particularly to networking devices, switches, and methods of operating the same. Background Art

[0002] Switches and similar network devices represent core components of many communications, security, and computing networks. Switches are often used to connect multiple devices, device types, networks, and network types.

[0003] Devices (including but not limited to personal computers, servers, or other types of computing devices) can be interconnected using network devices such as switches. These interconnected entities form a network that enables data communication and resource sharing between nodes. Typically, there may be multiple potential paths for data flow between any pair of devices. This feature is often referred to as multipath routing, allowing data (usually encapsulated in packets) to traverse different routes from the source device to the destination device. This network design enhances the robustness and flexibility of data communications because it provides alternatives in the event of path failures, congestion, or other adverse conditions. In addition, it facilitates load balancing across the network, thereby optimizing overall network performance and efficiency. However, managing multipath routing and ensuring optimal path selection can pose significant challenges, requiring advanced mechanisms and algorithms for network control and data routing, and power consumption can be unnecessarily high, especially during low traffic periods. Summary of the invention

[0004] According to one or more embodiments described herein, a computing system (e.g., a switch) can enable various systems (e.g., switches, servers, personal computers, and other computing devices) to communicate across a network. The ports of the computing system can be used as communication endpoints, allowing the computing system to manage multiple simultaneous network connections with one or more nodes.

[0005] Each port of a computing system can be used as an independent channel for data communication with the computing system. Each port of a computing system can be connected to one or more ports of one or more other computing systems. Ports allow concurrent network communications, enabling the computing system to perform multiple data exchanges with different network nodes at the same time. Each port of a computing system can be viewed as a lane and can be associated with a container (bin) of data (e.g., data in the form of packets) waiting to be sent through the port. When packets are ready to be sent from the computing system, each packet can be assigned to a specific container.

[0006] Each container can be associated with an egress port. When a packet is assigned to a specific container, the packet can be sent from the port associated with the container. The port associated with each container can be changed in response to receiving an adaptive routing notification (ARN) packet (which can be called a remote congestion notification packet), or in response to local reasons (e.g., by using a local adaptive routing mechanism to reroute traffic from a congested port). In fact, the port used to send a specific packet may depend on the container allocation to the packet and the association of the container with the port. Using the system described herein, data flows through the system can be provided in an efficient manner, and network traffic load balancing between multiple paths can be provided. The systems and methods described herein support equal-cost multi-path (ECMP) data routing for traffic that can accept out-of-order packet arrivals, such as Ethernet remote direct memory access (RDMA) streams and InfiniBand RDMA.

[0007] Load balancing network traffic across multiple paths is often a computationally difficult task. Consider a network switch that receives packets from one or more sources. Each packet that flows through the switch is associated with a specific destination. In simple topologies, the switch might have only one port from which the packet must be sent to reach the destination. However, in modern network topologies, such as a cluster of graphics processing units (GPUs) used for artificial intelligence (AI) related tasks, there may be many possible ports from which packets can be transmitted to the relevant destination. Therefore, since there are multiple paths in the network, it must be decided which of the many possible ports should transmit each packet.

[0008] In many applications, the goal of the switch in this case is to route packets to their destinations in a way that provides maximum overall throughput and avoids congestion. For example, consider switch S1 receiving two packets, A and B. Packet A is destined for destination node X, and packet B is destined for destination node Y. Switch S1 is connected to two other switches, S2 and S3. Switch S2 is connected to destination nodes X and Y, while switch S3 is connected to destination node X but not to destination node Y.

[0009] To reach destination node X, packet A can be sent from S1 to S2 or S3. To reach destination node Y, packet B must be sent from S1 to S2. If both packets A and B are sent from S1 to S2 to their respective destinations, S2 may become congested while S3 may be underutilized. Furthermore, if there is only one port connecting S1 to S2, that port may be overutilized while the other ports remain unused. In this case, one of packets A and B may be delayed in reaching its respective destination. On the contrary, if packet A is sent from S1 to S3 and packet B is sent from S1 to S2, the packets can reach their respective destinations without any delay and without causing any congestion.

[0010] Traditional approaches for routing traffic in high-performance computing (HPC) scenarios, such as large-scale or multi-tenant machine learning applications, rely on hash-based routing or other mechanisms that do not attempt to optimize traffic routing. While this traditional approach may work for non-HPC situations involving small amounts of traffic, this traditional approach is not optimized for scenarios involving large amounts of traffic, such as many AI applications. Hash-based routing mechanisms result in artificial grouping of unrelated traffic, leading to inefficient and unstable routing. For example, when a traditional switch receives different flows, the traditional switch assigns all packets in a specific flow to be routed through a specific port. When a flow contains a relatively large number of packets and / or when multiple flows are assigned to the same port, congestion, inefficient routing, and unstable routing can result.

[0011] For switches that route packets based on hashes, each packet of a flow is assigned to the same container, and packets of multiple flows may be assigned to the same container. In addition, while some containers may be overloaded with packets, other containers may be assigned few or no packets. As a result, containers may contain relatively too many packets, while other containers may not be fully utilized. This excessive number of packets may cause line congestion at distant points in the network. For example, a ToR or leaf switch may communicate with a destination through a spine switch. Packets sent from a switch to a spine may be assigned to a container before being sent through an egress port associated with a container. An ARN packet sent to a switch by a spine switch in response to one or more packets may cause an adaptive routing mechanism to switch the egress port associated with the container, causing the data in the container to be sent to a different spine switch, but the same problem will still occur because all the excess traffic assigned to the container will now appear on a different spine. Therefore, the egress port of the container will constantly change and a stable state cannot be achieved. In addition, because the container allocation is constantly changing, the ARN packet may be received too late at the port to properly reallocate the container. When the ARN packet is received, the container associated with the port may have been reallocated.

[0012] Each time a port associated with a container changes, latency may occur. For this reason, a steady state reduces the risk of latency. In addition, handling ARN packets and making decisions in response to ARN packets is costly in terms of time and power efficiency. The systems and methods described herein reduce such costs by implementing a steady state.

[0013] Using the systems or methods described herein, a steady state can be achieved. Packets can be assigned to containers in a uniform manner so that no one container holds an excessive number of packets. When congestion occurs and an ARN packet is received, the container can be reassociated to a different port and the congestion can be resolved. Such a solution can quickly achieve a steady state.

[0014] As described herein, methods of routing packets enable efficient network routing in applications where out-of-order packet arrival is acceptable, such as Ethernet RDMA streams and other HPC scenarios. The systems and methods described herein involve assigning packets to containers through a deterministic or non-deterministic selection process that does not rely on routing packets based solely on the stream to which the packet belongs or the hash of the packet. The systems and methods described herein include randomly assigning packets to containers. The systems and methods described herein avoid artificial grouping of unrelated traffic in traditional systems, which can lead to inefficient and unstable routing.

[0015] The systems and methods described herein also include specifying an egress port and a final destination switch or associating a transmit (TX) container with an egress port and a final destination switch. This associating of containers with egress ports and final destination switches produces fine-grained adaptability, thereby producing a steady-state solution that is not achievable using traditional methods.

[0016] The systems and methods described herein also include providing a response to receiving an ARN by updating local container information (eg, by changing an associated egress port associated with the container for which the ARN has been received).

[0017] The systems and methods described herein allow line rate adaptation to local and global routing inefficiencies, providing technical advantages over conventional solutions that do not handle AI situations well and / or are not applicable to non-local congestion scenarios.

[0018] The present disclosure describes a system and method for enabling a switch or other computing system to route packets in a manner that addresses the above-mentioned shortcomings and other problems by implementing an improved routing method. As an illustrative example aspect of the disclosed system and method, the method may include: receiving a packet associated with a destination; determining a plurality of containers based on the destination of the packet, wherein each container in the plurality of containers is associated with a corresponding port in a plurality of ports; selecting a container in the plurality of containers for forwarding the packet; and forwarding the packet through a port associated with the selected container.

[0019] The above example aspects include receiving an adaptive routing notification (ARN) packet through a first port of a plurality of ports, wherein the first port of the plurality of ports is associated with a first container; and in response to receiving the ARN packet, associating the first container with the second port and disassociating the first container from the first port.

[0020] Any of the above example aspects include where a non-deterministic process is used to select the selected container.

[0021] Any of the above example aspects include where the non-deterministic process is random or pseudo-random.

[0022] Any of the above example aspects includes where the selected container is selected using a round-robin selection process.

[0023] Any of the above example aspects include the method further comprising: receiving a second packet associated with the destination; selecting a second container of the plurality of containers for forwarding the second packet, wherein the second container is different from the first container; and forwarding the second packet through a corresponding port associated with the second container.

[0024] Any of the above example aspects includes where each container in the plurality of containers is further associated with a destination address.

[0025] In another illustrative example, a switch includes one or more circuits to: receive a packet associated with a destination; determine a plurality of containers based on the destination of the packet, wherein each container in the plurality of containers is associated with a corresponding port in a plurality of ports; select a container in the plurality of containers for forwarding the packet; and forward the packet via the port associated with the selected container.

[0026] In another illustrative example, one or more of the circuits are further configured to:

[0027] receiving an adaptive routing notification (ARN) packet via a first port of a plurality of ports, wherein the first port of the plurality of ports is associated with a first container; and

[0028] In response to receiving the ARN packet, the first container is associated with the second port and the first container is disassociated from the first port.

[0029] Any of the above example aspects includes where the selected container is selected using a non-deterministic process.

[0030] Any of the above example aspects includes where the non-deterministic process is random or pseudo-random.

[0031] Any of the above example aspects includes where the selected container is selected using a round-robin selection process.

[0032] Any of the above example aspects includes wherein the one or more circuits are further configured to: receive a second packet associated with a destination; select a second container of the plurality of containers for forwarding the second packet; and forward the second packet via a port associated with the second container.

[0033] Any of the above example aspects includes where each container in the plurality of containers is further associated with a destination address.

[0034] In yet another illustrative example, a network device as described herein includes: a plurality of ports; and one or more circuits to: receive a plurality of packets associated with a destination; determine a plurality of containers based on the destination of the packets, wherein each container in the plurality of containers is associated with a corresponding port in the plurality of ports; and for each packet in the plurality of packets: select a container in the plurality of containers; and forward the packet via the port associated with the selected container.

[0035] The above example aspects include wherein the one or more circuits are further configured to: receive an adaptive routing notification (ARN) packet via a first port of a plurality of ports, wherein the first port of the plurality of ports is associated with a first container; and in response to receiving the ARN packet, associate the first container with the second port and disassociate the first container from the first port.

[0036] Any of the above example aspects includes where a non-deterministic process is used to select the selected container.

[0037] Any of the above example aspects includes where each container in the plurality of containers is further associated with a destination address.

[0038] Any of the above example aspects includes where the selected container is selected using a round-robin selection process.

[0039] Any of the above example aspects includes where each container in the plurality of containers is further associated with a destination address.

[0040] The routing methods described and depicted herein may be applied to switches, routers, or any other suitable type of networking device known or yet to be developed.Additional features and advantages are described herein and will be apparent from the following description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present disclosure is described in conjunction with the accompanying drawings, which are not necessarily drawn to scale:

[0042] Figure 1 is a block diagram showing an illustrative configuration of a computing system in accordance with at least some embodiments of the present disclosure;

[0043] Figure 2 illustrates a computing system and a network of nodes in accordance with at least some embodiments of the present disclosure;

[0044] Figure 3 illustrates local container information and adaptive routing mechanisms according to at least some embodiments of the present disclosure;

[0045] Figure 4A , Figure 4B and Figure 4C illustrates exemplary scenarios according to at least some embodiments of the present disclosure;

[0046] Figure 5A , Figure 5B and Figure 5C illustrates exemplary scenarios according to at least some embodiments of the present disclosure;

[0047] Figure 6 is a flow chart illustrating a method according to at least some embodiments of the present disclosure; and

[0048] Figure 7 is a flow chart illustrating a method in accordance with at least some embodiments of the present disclosure. DETAILED DESCRIPTION

[0049] The following description provides only embodiments and is not intended to limit the scope, applicability or configuration of the claims. Instead, the following description will provide a feasible description for those skilled in the art to implement the embodiments. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the spirit and scope of the appended claims.

[0050] It will be appreciated from the following description that, for reasons of computational efficiency, the components of the system may be arranged in any suitable locations in a distributed network of components without affecting the operation of the system.

[0051] In addition, it should be understood that the various links connecting the elements can be wired, trace or wireless links, or any suitable combination thereof, or any other suitable known or later developed elements capable of providing data to and / or transmitting data from the connected elements. For example, the transmission medium used as a link can be any suitable electrical signal carrier, including coaxial cable, copper wire and optical fiber, electrical traces on a printed circuit board (PCB), etc.

[0052] As used herein, the phrases "at least one", "one or more", "or", and "and / or" are open-ended expressions that are both conjunctions and disjuncts in operation. For example, each of the expressions "at least one of A, B, and C", "at least one of A, B, or C", "one or more of A, B, and C", "one or more of A, B, or C", "A, B and / or C", and "A, B, or C" means: A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0053] As used herein, the term "automatic" and variations thereof refer to any suitable process or operation that can be accomplished without substantial human input when the process or operation is performed. However, if input is received prior to the performance of the process or operation, then the process or operation may be automatic, even if the performance of the process or operation uses substantial or insubstantial human input. Human input is considered substantial if it affects the manner in which the process or operation is performed. Human input that consents to the performance of the process or operation is not considered "substantial."

[0054] As used herein, the terms "determine," "compute," and "calculate" and variations thereof may be used interchangeably and include any suitable type of method, process, operation, or technique.

[0055] Various aspects of the disclosure are described herein with reference to the accompanying drawings, which are schematic illustrations of idealized configurations.

[0056] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present disclosure belongs. It should also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the relevant technology and the present disclosure.

[0057] As used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that when the terms "include", "comprises", and / or "contains" are used in this specification, the presence of the features, integers, steps, operations, elements, and / or components is specified, but the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof is not excluded. The term "and / or" includes any and all combinations of one or more of the associated listed items.

[0058] Reference now Figures 1 to 7, various systems and methods for routing packets between communication nodes will be described. The concepts of packet routing depicted and described herein can be applied to routing information from one computing device to another computing device. The term "packet" as used herein should be interpreted as any appropriate discrete amount of digitized information. The routed data can be in the form of a single packet or multiple packets without departing from the scope of the present disclosure. In addition, certain embodiments will be described in conjunction with systems configured to make centralized routing decisions, while other embodiments will be described in conjunction with systems configured to make distributed and possibly uncoordinated routing decisions. It should be understood that the features and functions of a centralized architecture can be applied to or used in a distributed architecture, or vice versa.

[0059] According to one or more embodiments described herein, Figure 1 The computing system 103 shown can enable various systems (such as switches, servers, personal computers and other computing devices) to communicate across a network. Such a computing system 103 described herein can be a switch or any computing device including multiple ports 106a-106d for connecting to nodes on a network.

[0060] In some implementations, the switches described herein can be used as leaves in a larger hierarchical network topology, such as Figure 2 As shown and described below. This network configuration may be referred to as a leaf-spine architecture and may include multiple leaf switches that interface directly with endpoint devices or nodes while communicating with one or more centralized switches or spines. The network topology may be installed in such a way that packets transmitted between any two leaf switches need only traverse a single spine switch.

[0061] Such a leaf switch can also be characterized as a top-of-rack (ToR) switch. A ToR switch can be located on top of a rack in a data center or server room. Such a ToR switch can facilitate server interconnection within the same rack. By operating as a ToR, a leaf switch can reduce cabling complexity and increase scalability because it can act as a local aggregation point for its directly connected servers.

[0062] The ports 106a-106d of the computing system 103 can be used as communication endpoints, allowing the computing system 103 to manage multiple simultaneous network connections with one or more nodes. Each port 106a-106d can be used to transmit data associated with one or more flows. In some implementations, multiple ports 106a-106d can be used to transmit a single data flow.

[0063] Each container 121a-121d can be associated with or mapped to a port 106a-106d. The ports 106a-106d associated with the containers 121a-121d can be enabled to handle incoming and outgoing data packets associated with the flow. It should be understood that in some implementations, multiple containers 121a-121d can be mapped to the same port or to separate ports. The containers 112a-112d described herein can be data structures that act as intermediate holding circuits for allocated packets. For example, when a packet arrives at or is received at the computing system 103, a classification or allocation mechanism can be triggered, such as the following with respect to Figure 6 The allocation mechanism may determine the appropriate container 112a-112d to which the packet should be allocated.

[0064] Once the packet is assigned to a container 112a-112d, the computing system 103 may initiate a process of forwarding or transmitting the packet. The transmission of the packet may be performed via a port specifically associated or linked to the assigned container 112a-112d. The association between the container and the port may be based on local container information 121 stored in the memory 118. The association between the container and the port may be changed, for example, by the adaptive routing mechanism 115 in response to an ARN received via the port 106a-106d, as described below with respect to Figure 7 The method 700 described herein can be a logical space in a memory, such as a cache or a queue, which can contain packets assigned to the corresponding container 112 until they are routed through a port associated with the container 112.

[0065] The ports 106a-106d of the computing system 103 may be physical connection points that allow a network cable (eg, an Ethernet cable) to connect the computing system 103 to one or more network nodes and / or backbones, such as Figure 2 Each port 106a-106d may be of a different type, including, for example, a 100 Mbps, 1000 Mbps, or 10 Gigabit Ethernet port, each providing a different level of bandwidth.

[0066] Packets received, created, or otherwise handled by computing system 103 for transmission from computing system 103 may be associated with a specific destination. For example, in some implementations, a leaf switch may handle packets carrying specific forwarding information. Each packet traversing through a leaf switch may include detailed information about the final destination, which may be specified by an IP address or another identifier. The final destination information may be used to ensure that data reaches its intended endpoint, even if it must navigate through multiple intermediate devices or networks. In some implementations, a packet may also contain detailed information about the next or next hop switch to which the packet should be forwarded through the network.

[0067] Because each port 106a-106d can be used to send a particular packet, when a packet is received, created, or otherwise handled by the computing system 103 and is to be transmitted from the computing system 103, one or more ports 106a-106d of the computing system 103 can be selected to transmit the packet, such as by method 600 as described below. Transmitting the packet from the port 106a-106d can include assigning the packet to a container 112a-112d associated with the port 106a-106d, as described in more detail below.

[0068] The one or more routing circuits 109 of the computing system may include an internal fabric or path within the computing system 103 through which data propagates between the two ports 106a-106d. In some embodiments, the routing circuit 109 may include one or more network interface cards (NICs). For example, in some embodiments, each port 106a-106d may be associated with a different NIC. The NIC or multiple NICs may include hardware and / or circuitry that can be used to transmit data between the ports 106a-106d.

[0069] The routing circuit 109 may also or alternatively include one or more application specific integrated circuits (ASICs) for performing tasks such as determining to which port a received packet should be sent. The routing circuit 109 may include various components, for example, including a port controller that manages the operation of each port, a network interface card that facilitates data transmission, and an internal data path that directs data flow within the computing system 103. The routing circuit 109 may also include a memory element for temporarily storing data and management software for controlling hardware operations. This configuration enables the routing circuit 109 to accurately track port usage and provide data to the processor and / or adaptive routing mechanism 115 upon request.

[0070] like Figure 1As shown, the computing system 103 may also include an adaptive routing mechanism 115, such as a circuit or device capable of reading instructions from the memory 118 and performing operations. In some implementations, the adaptive routing mechanism 115 may be a process performed by a CPU, a microprocessor, or any circuit or device. The adaptive routing mechanism 115 may execute software instructions to control the operation of the computing system 103, such as by receiving and processing ARNs received via ports 106a-106d and updating local container information 121 in the memory 118 in response, such as by method 700 as described below.

[0071] In some implementations, the processor may serve as the central processing unit of the computing system 103 and execute the operational capabilities of the system. The processor may communicate with the components of the computing system 103 to manage and execute computing operations, thereby ensuring optimal system functionality and performance.

[0072] In more detail, the processor can be designed to perform a wide range of computing tasks. The capabilities of the processor may include executing program instructions, managing data within the system, and controlling the operation of other hardware components (e.g., routing circuit 109). The processor may be a single-core or multi-core processor, and may include one or more processing units, depending on the specific design and requirements of the computing system 103. The architectural design of the processor may allow for efficient instruction execution, data processing, and overall system management, thereby enhancing the performance and practicality of the computing system 103 in a variety of applications. In addition, the processor may be programmed or adjusted to perform specific tasks and operations according to application requirements, thereby potentially enhancing the versatility and adaptability of the computing system 103.

[0073] The computing system 103 may also include one or more memory 118 components. The memory 118 may be configured to communicate with the adaptive routing mechanism 115 and / or the routing circuit 109 of the computing system 103. The communication between the memory 118, the adaptive routing mechanism 115, and the routing circuit 109 may enable various operations including, but not limited to, data exchange, command execution, and memory management.

[0074] The memory 118 may be made up of a variety of physical components, depending on the specific type and design. At the core, the memory 118 may include one or more memory cells capable of storing data in the form of binary information. These memory cells may be made up of transistors, capacitors, or other suitable electronic components, depending on the memory type, such as DRAM, SRAM, or flash memory. To enable data transfer and communication with other parts of the computing system 103, the memory 118 may also include data lines or buses, address lines, and control lines. These physical components may collectively make up the memory 118, contributing to its ability to store and manage data.

[0075] The data stored in the memory 118 may include information about various aspects of port, container, and system usage. Such information may include data about the association of containers 112a-112d with ports 106a-106d and the association of ports 106a-106d and / or containers 112a-112d with destination addresses, as well as other content described in more detail herein. The data in the memory 118 (e.g., local container information 121) may be stored, accessed, and utilized by the adaptive routing mechanism 115 and / or routing circuitry to manage port operations and network communications. For example, the adaptive routing mechanism 115 may utilize the local container information 121 in the memory 118 to respond to the receipt of an ARN or otherwise control the flow of data through the computing system 103, as described in more detail herein.

[0076] In one or more embodiments of the present disclosure, a computing system 103 (eg, a switch) may communicate with a plurality of network nodes 203a-203f, such as Figure 2 As shown. Figure 2 In the network shown, each computing system 103 acts as a leaf, facilitating communication between nodes 203a-203h via backbones 206a-206b and other leaf switch computing systems 103a-103d. Each network node 203a-203h can be a computing system with the ability to send and receive data. Each node 203a-203h can be any of a variety of devices, including but not limited to switches, personal computers, servers, or any other device capable of sending and receiving data in packet form. Through backbones 206a-206b, each leaf switch 103a-103d can be able to communicate with remote systems (e.g., other leaf switch computing systems 103a-103d).

[0077] In some embodiments, the packet sent from node 200A may originate from an application. For example, node 203A may be a computing system executing one or more applications that communicate with a target device acting as another node 203h through one or more leaf switch computing systems 103a-103d and spines 206a-206b.

[0078] The backbone 206a-206b as described herein may be a computing system 103, for example Figure 1 As shown and described above, or may be other types of computing systems capable of receiving and sending packets. In some embodiments, the trunks 206a-206b may be capable of detecting traffic or congestion and generating ARNs in response. The trunks 206a-b may send ARNs to the ports of the leaf switch computing systems 103a-103d if the corresponding ports are sending data through the trunks 206a-206b in a manner that causes excessive traffic or congestion. Such events may be as described with respect to Figures 4A-4C and Figures 5A-5C As described in more detail.

[0079] It should be understood that in some implementations, packets sent from computing systems 103a-103d may originate from within computing systems 103a-103d. For example, computing system 103 may execute one or more applications that communicate with nodes 203a-203h and / or another computing system 103a-103d, for example, via one or more backbones 206a-206b.

[0080] Although Figure 2 The network shown in shows a specific number of nodes 203a-h, leaf switch computing systems 103a-103d, and backbones 206a-206b, but it should be understood that in some implementations, other numbers of each device may be used, and the systems and methods described herein may be performed without involving certain devices (e.g., nodes 203a-203h). In addition, the interconnections of nodes 203a-203h, leaf switch computing systems 103a-103d, and backbones 206a-206b shown should be considered for illustrative purposes only. In some implementations, a single node 203 may be connected to multiple leaf switch computing systems 103 and / or directly to a backbone 206. In addition, in some implementations, some leaf switch computing systems 103a-103d may be connected to only some backbones 206a-206b. It should be appreciated that the leaf switch computing systems 103a-103d, spines 206a-206b, and nodes 203a-203h may be arranged in any number of various interconnection topologies.

[0081] For example, nodes 203a-h can be connected to multiple computing systems 103 and backbone 206 as described herein, thereby forming an interconnected switch network. Multiple computing systems 103 (e.g., switches) can be interconnected in various topologies, such as star, ring, or mesh, depending on the specific requirements and resiliency required for the network. For example, in a star topology, multiple switches can be connected to a central switch, while in a ring topology, each switch can be connected to two other switches in a closed loop. In a mesh topology, each switch can be interconnected with every other switch in the network.

[0082] The computing system 103a-103d can establish a communication channel with the network node 203a-203h via a port. Such a channel can support data transmission in the form of packet streams, following the predetermined protocol of the format, size, transmission method and other aspects of the dominant packet. Each network node 203a-203h can interact with the computing system 103a-103d in various ways. Node 203 can send data packets to the computing system 103a-103d for processing, transmission or other operations, or forward to another node 200. On the contrary, each node 200 can receive data from the computing system 103, which originates from the computing system 103 itself or comes from other network nodes 200a-200f via the computing system 103. In this way, the computing system 103 and the nodes 200a-200f can form a network together, thereby facilitating data exchange, resource sharing and a series of other collaborative operations.

[0083] exist Figure 2 In the example shown, each node 203a is connected to one or more ports of one of the leaf switch computing systems 103a-103d. Each leaf switch computing system 103a-103d is connected to one or more ports of each backbone 206a-206b. Therefore, each node 203a-203h can communicate with any other node 203a-203h via one or more leaf switch computing systems 103a-103d and / or backbones 206a-206b. For example, the computing system 103 can use any port 106b-106f to send a packet from node 200a to another node 200b.

[0084] Figure 3 Local container information 121 and adaptive routing mechanism 115 are shown. Local container information 121 can be used as a structured repository where containers are associated with specific outgoing ports and final destinations (e.g., nodes 203a-203h served by the corresponding container). The outgoing port associated with a container can be the port from which packets assigned to the container are transmitted from the system. In this case, each container acts as a classification or bucketing mechanism for packets. The inclusion of (one or more) final destinations in the local container information 121 can be used to ensure that packets assigned to the corresponding container are forwarded in a manner that directs the packets to their destinations.

[0085] The local container information 121 may also include information indicating an adaptive routing group associated with each container. Each container may be associated with an adaptive routing group. Each adaptive routing group may be associated with multiple containers. As described herein, when a packet is received, an adaptive routing group associated with the packet may be determined, and based on the determined adaptive routing group, a set of containers or multiple containers to which the packet may be assigned may be determined. In some implementations, each adaptive routing container may be associated with a different destination switch, such as a destination ToR switch. Using such a system, packets of a flow associated with one destination ToR switch may not share containers with packets of a flow associated with a different ToR switch. In this way, containers may be pre-divided according to destination.

[0086] In some implementations, the local container information 121 may be stored in the form of a table, such as within the memory 118 of the computing system 103A. In other implementations, the local container information 121 may be implemented by a dedicated circuit system. For example, the local container information 121 may take the form of a hardware-centric approach that may utilize a combination of switches, registers, multiplexers, and / or other electronic devices to encode associations between containers, ports, and destinations. In a hardware-centric approach, the local container information 121 may include a set of electronic devices through which data may flow along a predefined path created by the interconnection of these components, thereby facilitating the association of a container to a port.

[0087] The adaptive routing mechanism 115 may be a software process or hardware device capable of receiving and processing the ARN 300 and updating the local container information 121 in response. For example, the ARN 300 may be received from a device such as the spine 206 via a port of the computing system 103. The ARN 300 may indicate that data sent from the port is causing or being affected by congestion between other devices (e.g., the spine 206 receiving the ARN 300 and another computing system 103 such as a leaf switch and / or node 203).

[0088] The port receiving ARN 300 may be associated with the container described in local container information 121. In response to receiving ARN 300, the adaptive routing mechanism may be as follows: Figure 7 The method described updates the local container information 121 by disassociating the port receiving the ARN 300 from the container associated with the port and reassociating the container with a different outgoing port and / or reassociating the port receiving the ARN 300 with a different container.

[0089] Figures 4A-4C and Figures 5A-5CExemplary scenarios in which an ARN can be received by a computing system 103A and used to update local container information are shown. In each exemplary scenario, a source computing system 103a (e.g., a leaf switch) sends data to one or more destinations via one or more spines 206a-206b. The destination can be other computing systems 103b-103c (e.g., leaf switches), or can be a node.

[0090] exist Figure 4A In the exemplary scenario shown, the source computing system 103a (which may be a leaf switch or a ToR switch, for example) sends data (represented by dashed and solid arrows) to the first trunk 206a. The destination of the data is the target computing system 103b. The first trunk 206a receives the data and forwards the data to the target computing system 103b through one or more ports.

[0091] The data sent from source computing system 103a to target computing system 103b (represented by dashed and solid arrows) can be a single stream or different multiple streams. The data can originate from source computing system 103a or another device (e.g., node 203) that uses source computing system 103a to send data.

[0092] The dashed arrow traffic has been assigned to the first container and is being sent from the first port to the first trunk 206a. The solid arrow traffic has been assigned to the second container and is being sent from the second port to the first trunk 206a. The second trunk 206b, which is connected to the source computing system 103a and the target computing system 103b through one or more ports, is not currently involved in sending the data represented by the dashed and solid arrows to the target computing system 103b.

[0093] After receiving data from the source computing system 103a, the first trunk 206a can assume the responsibility of forwarding the data to the target computing system 103b. For example, once the first trunk 206a identifies the appropriate destination for the data, whether it is a leaf switch or an end node within the network, the first trunk 206a will initiate the process of forwarding the data via one or more ports.

[0094] like Figure 4B As shown, first trunk 206a sends an ARN packet represented by a dotted arrow in response to data represented by a solid arrow sent from source computing system 103a. In response to determining that first trunk 206a is outputting too much traffic (e.g., traffic exceeding a threshold) to target computing system 103b, first trunk 206a may send the ARN packet. Due to congestion issues between the first trunk and target computing system 103b, the ARN packet sent by first trunk 206a to source computing system 103a arrives at the same port that is sending the data represented by the solid arrow.

[0095] In response to the ARN packet, the adaptive routing mechanism may update the local container information to disassociate the container from the port that received the ARN packet and associate the container with a different port. Moving forward, packets assigned to the container may be sent from the source computing system 103a via a different port.

[0096] like Figure 4C As shown, the data represented by the solid line is now routed from a different port of the source computing system 103a to be routed to the target computing system 103b through the second trunk 206b, thereby reducing the amount of traffic between the first trunk 206a and the target computing system 103b. It should be understood that the traffic represented by the dashed arrows is not affected by the ARN packet, even though the dashed arrow traffic is the same flow as the solid arrow traffic. The process of receiving the ARN packet and updating the local container information can be used as described below with respect to Figure 7 The method 700 is described to be performed.

[0097] like Figure 5A As shown, the source computing system 103a (which may be a leaf or ToR switch, for example) transmits first data (represented by a solid arrow) and second data (represented by a dashed arrow) to the first trunk 206a. The destination of the first data is the first target computing system 103b. The destination of the second data is the second target computing system 103c. The first trunk 206a receives the data and forwards the first data to the first target computing system 103b through one or more ports, and forwards the second data to the second target computing system 103c through one or more ports. Another device 500, labeled as noise, sends data (represented by a dashed line) to the first target computing system 103b.

[0098] The first and second data (represented by dashed and solid arrows) sent from source computing system 103a to first and second target computing systems 103b-103c may be a single stream or different multiple streams. For example, each of first and second target computing systems 103b-103c may forward the first data and the second data, respectively, to the same final destination. The first data and the second data may originate from source computing system 103a or from another device (e.g., node 203) that uses source computing system 103A to send data.

[0099] The dashed arrow traffic has been assigned to the first container and is being sent from the first port to the first trunk 206a. The solid arrow traffic has been assigned to the second container and is being sent from the second port to the first trunk 206a. The second trunk 206b, which is connected to the source computing system 103A and the target computing systems 103b-103c through one or more ports, is not currently involved in sending the data represented by the dashed arrows and the solid arrows to the target computing systems 103b-103c.

[0100] After receiving data from the source computing system 103a, the first trunk 206a can assume the responsibility of forwarding the data to the appropriate target computing system 103b-103c. For example, once the first trunk 206a determines the appropriate destination for the data, whether it is a leaf switch or an end node within the network, the first trunk 206a will initiate the process of forwarding the data through one or more ports.

[0101] like Figure 5B As shown, the first trunk 206a sends an ARN packet represented by a dotted arrow in response to data represented by a solid arrow sent from the source computing system 103a.

[0102] In response to determining that the first trunk 206a is outputting too much traffic (e.g., traffic exceeding a threshold) to the first target computing system 103b, the first trunk 206a may send an ARN packet. For example, the excessive traffic may be at least partially due to data sent by the noisy device 500 to the first target computing system 103b via the first trunk 206a. Due to congestion issues between the first trunk and the first target computing system 103b, the ARN packets sent by the first trunk 206a to the source computing system 103a arrive at the same port that sent the data represented by the solid arrow.

[0103] In response to the ARN packet, the adaptive routing mechanism of the source computing system 103a can update the local container information to disassociate the container from the port that received the ARN packet and associate the container with a different port. Subsequently, packets assigned to the container can be sent from the source computing system 103a through a different port.

[0104] like Figure 5C As shown, the data represented by the solid line is now routed from a different port of the source computing system 103a to be routed to the first target computing system 103b through the second trunk 206b, thereby reducing the amount of traffic between the first trunk 206a and the first target computing system 103b. It should be understood that the traffic represented by the dashed arrows is not affected by the ARN packet, even though the dashed arrow traffic is the same flow as the solid arrow traffic. The process of receiving the ARN packet and updating the local container information can be used as described below with respect to Figure 7 The method 700 is described to be performed.

[0105] like Figure 6 As shown, and according to Figure 1 The computing system 103 shown and described above may perform the method 600 to forward a packet to a target device using a specific container. Figure 7 As shown, and according to Figure 1The computing system 103 shown, and as described herein, may perform method 700 to update local container information 121, for example by using an adaptive routing mechanism as described above. The adaptive routing mechanism as described herein may include a software process or a hardware circuit or some combination thereof, and may be capable of changing, for example, Figure 3 The local container information shown. Although the descriptions of methods 600 and 700 provided herein describe that the steps of methods 600 and 700 are performed by specific components of computing system 103, the steps of methods 600 and 700 can be performed by one or more processors, routing circuits 109, controllers, circuits in computing system 103, or some combination thereof. It should be understood that methods 600 and 700 can be implemented by hardware or software. As a result of methods 600 and 700, adaptive routing of packets can be achieved, which can result in efficient traffic routing (e.g., RDMA routing) for applications where packets can arrive at a destination in an order different from the order in which they were sent.

[0106] At 603, computing system 103 (e.g., a leaf or ToR switch) can process the packet. As described herein, processing the packet can include receiving the packet or generating the packet. In some implementations, computing system 103 within the network can receive the packet through one or more ports, while in some implementations, computing system 103 can execute one or more applications that can generate packets that can be sent from computing system 103.

[0107] After receiving or generating a packet, computing system 103 may perform processing of the packet to determine a destination associated with the packet. For example, one or more routing circuits 109 or other components of computing system 103 may perform a header inspection or otherwise determine the destination of the packet. In some implementations, computing system 103 may reference one or more internal tables or databases that map packet attributes to potential destinations. Based on the final destination of the packet, computing system 103 may determine where the packet should be routed next.

[0108] In some implementations, a packet described herein may be a packet of a data stream. A data stream may be a continuous flow or sequence of packets from a single source to a specific destination.

[0109] At 606, computing system 103 may determine a plurality of containers from which packets may be sent to reach a destination. Figure 2As shown, since the computing system 103 can be interconnected to the node 203 through multiple switches and backbones, the computing system 103 can send the packet to the destination through multiple different ports 106. Before selecting a specific container for forwarding the packet, the computing system 103 can identify a set of containers, wherein the packet can be assigned to any container in the set of containers and can reach its destination.

[0110] Each container in the plurality of containers may be associated with a corresponding port in the plurality of ports. For example, local container information 121 may be used to associate each container with a corresponding port. The packet assigned to the container may ultimately be sent via the port associated with the assigned container. As described above, each container in the plurality of containers may also be associated with a destination address. For example, local container information 121 may include information such as the destination node served by each particular container.

[0111] As described above, the local container information 121 may include information indicating an adaptive routing group associated with each container. Each container may be associated with an adaptive routing group. Each adaptive routing group may be associated with multiple containers. As described herein, when a packet is received, an adaptive routing group associated with the packet may be determined, and based on the determined adaptive routing group, a set or multiple containers to which the packet may be assigned may be determined. In some implementations, each adaptive routing container may be associated with a different target switch, such as a target ToR switch. Determining multiple containers from which a packet may be sent to reach a destination may involve determining an adaptive routing group associated with the packet and selecting a set of containers associated with the determined adaptive routing group.

[0112] At 609, a container may be selected from the plurality of containers to forward the packet. A deterministic or non-deterministic process may be used to select a container for forwarding the packet. The method of selecting a container for allocating the packet may vary depending on the requirements of the network or application. For example, in some implementations, the computing system 103 may utilize a randomization mechanism. For example, using a built-in algorithm or an external random number generator, the system may determine a random container in the determined plurality of containers to which the packet should be allocated. This randomness may be used to evenly distribute the load across all containers in the determined plurality of containers.

[0113] In some implementations, various algorithms or methods may be used to achieve randomization. For example, a pseudo-random number generator may be used to determine the container assignment for each packet. Such a generator, while deterministic in nature, produces a sequence of numbers that approximates true randomness, thereby ensuring that each of the determined plurality of containers receives a roughly equal number of packets over time. In other cases, a true random number generator may be used that may rely on an inherently random physical process.

[0114] Furthermore, the granularity of randomization can be adjusted based on the needs of the system. For example, in cases where rapid fluctuations in container allocations are undesirable, the system can introduce a time component that ensures that random reallocations occur only after a certain interval or after a set number of packets have been processed. Conversely, in highly dynamic environments that require rapid adaptability, a higher randomization frequency may be more appropriate.

[0115] Alternatively, in some implementations, computing system 103 may employ a selection process, such as a cyclic mechanism or other deterministic process. For example, in a cyclic approach, packets may be assigned to containers in a determined plurality of containers in a cyclical manner. For example, if there are three containers, the first packet may be assigned to container one, the second packet to container two, the third packet to container three, the fourth packet back to container one, and so on. In some implementations, cyclic container selection may be implemented using a deterministic mapping based on a field that increases by one for each packet in the stream.

[0116] Regardless of the technique used, whether deterministic or non-deterministic, the computing system 103 can assign outgoing packets to containers in a deterministic plurality of containers without regard to the stream or any hash information associated with the packet. By doing so, the system can optimize packet handling operations, ensure balanced utilization of resources, and potentially improve overall network performance, particularly for applications where packet order is not critical, such as RDMA.

[0117] At 612, after selecting a container from the determined plurality of containers, the packet may be assigned to the selected container and forwarded via a port associated with the selected container. In some implementations, after assigning a port to a container, one or more routing circuits 109 may perform a lookup to determine from which port 106a-106d the packet should be sent based on the local container information 121.

[0118] The method 600 may repeat receiving a second packet associated with a destination, selecting a second container of the plurality of containers for forwarding the second packet, wherein the second container is different from the first container, and forwarding the second packet via a corresponding port associated with the second container.

[0119] like Figure 7As shown, a method 700 of adjusting or updating local container information in computing system 103 may be performed in response to receiving an ARN packet. Method 700 may begin when the computing system receives an ARN packet via a port of one or more ports at 703 during the process of sending data from one or more ports of computing system 103. As described above, each port may be associated with a corresponding container. Therefore, when an ARN packet is received through a port, it may be determined by reading local container information 121 (e.g., which may be stored in memory 118) to which container the ARN packet is associated.

[0120] The ARN packet may be handled by the adaptive routing mechanism 115. At 706, in response to receiving the ARN packet, the adaptive routing mechanism 115 may determine which container is associated with the ARN packet and update local container information associated with the container. Specifically, the adaptive routing mechanism may change the port associated with the container. For example, if the container is currently associated with a first port, the adaptive routing mechanism may disassociate the container from the first port and associate the container with a second port.

[0121] In some implementations, updating the local container information associated with the container may involve switching the port associated with the container with a port associated with another container. In some implementations, the port associated with the container after the local container information 121 is updated may be selected by a deterministic process or a non-deterministic process.

[0122] Although the systems and methods described herein relate to the use of ARN grouping, it should be understood that in some implementations, local adaptive routing methods (e.g., port classification mechanisms) can also or alternatively be used to change container allocations to avoid problematic ports. For example, if one flow in a computing system does not support out-of-order grouping or cannot be changed from a specific port, other flows can avoid the port used for that flow.

[0123] Upon receiving the ARN packet, the adaptive routing mechanism may be able to handle the ARN packet received from the port and transmit the ARN packet to the same container that sent the packet that caused the ARN packet.

[0124] At 709 , after the local container information 121 is updated to reflect the new association of the container with the different port, data assigned to the container may be routed from the container via the port newly associated with the container.

[0125] This disclosure covers Figure 6 and Figure 7 (and the corresponding method description), and including all steps identified in Figure 6 and Figure 7The present disclosure also encompasses methods comprising one or more steps from the methods described herein and one or more steps from any other method described herein.

[0126] Embodiments of the present disclosure include a method of routing a packet, the method comprising: receiving a packet associated with a destination; determining a plurality of containers based on the destination of the packet, wherein each container in the plurality of containers is associated with a corresponding port in a plurality of ports; selecting a container in the plurality of containers for forwarding the packet; and forwarding the packet via the port associated with the selected container.

[0127] Embodiments also include a switch comprising one or more circuits to: receive a packet associated with a destination; determine a plurality of containers based on the destination of the packet, wherein each container in the plurality of containers is associated with a corresponding port in a plurality of ports; select a container in the plurality of containers for forwarding the packet; and forward the packet via the port associated with the selected container.

[0128] Embodiments also include a network device comprising: a plurality of ports; and one or more circuits for: receiving a plurality of packets associated with a destination; determining a plurality of containers based on the destination of the packets, wherein each container in the plurality of containers is associated with a corresponding port in the plurality of ports; and for each packet in the plurality of packets: selecting a container in the plurality of containers; and forwarding the packet via the port associated with the selected container.

[0129] Aspects of the above-described methods, switches, and network devices include receiving an adaptive routing notification (ARN) packet via a first port of a plurality of ports, wherein the first port of the plurality of ports is associated with a first container; and in response to receiving the ARN packet, associating the first container with the second port, and disassociating the first container from the first port.

[0130] Aspects of the above methods, switches, and network devices also include where a non-deterministic process is used to select the selected container.

[0131] Aspects of the above methods, switches, and network devices also include where the non-deterministic process is random or pseudo-random.

[0132] Aspects of the above methods, switches, and network devices also include where a round-robin selection process is used to select the selected container.

[0133] Aspects of the above methods, switches, and network devices also include: receiving a second packet associated with a destination; selecting a second container among the plurality of containers for forwarding the second packet, wherein the second container is different from the first container; and forwarding the second packet via a corresponding port associated with the second container. Aspects of the above methods, switches, and network devices also include, wherein each of the plurality of containers is also associated with a destination address.

[0134] It should be understood that any feature described herein may be claimed in combination with any other feature described herein, whether or not such features are from the same described embodiment.

[0135] Specific details are given in the description to provide a comprehensive understanding of the embodiments. However, one of ordinary skill in the art will appreciate that the embodiments may be implemented without these specific details. In other cases, well-known circuits, processes, algorithms, structures, and techniques may not be shown in unnecessary detail to avoid obscuring the embodiments.

[0136] Although illustrative embodiments of the present disclosure are described herein in detail, it should be understood that the concepts of the present invention may be embodied and used in other various ways, and that the appended claims are intended to be interpreted to include such variations unless limited by the prior art.

Claims

1. A method for routing packets, the method comprising: receiving a packet associated with a destination; determining a plurality of containers based on the destination of the packet, wherein each container of the plurality of containers is associated with a corresponding port of a plurality of ports; selecting a first container of the plurality of containers for forwarding the packet; and The packet is forwarded through the port associated with the first container.

2. The method according to claim 1, further comprising: receiving an adaptive routing announcement (ARN) packet via a first port of the plurality of ports, wherein the first port of the plurality of ports is associated with a first container; as well as In response to receiving the ARN packet, the first container is associated with a second port, and the first container is disassociated from the first port. The method of claim 1 , wherein the first container is selected using a non-deterministic process. The method of claim 3 , wherein the non-deterministic process is random or pseudo-random. The method of claim 1 , wherein the first container is selected using a round-robin selection process.

6. The method according to claim 1, further comprising: receiving a second packet associated with the destination; selecting a second container among the plurality of containers for forwarding the second packet, wherein the second container is different from the first container; as well as The second packet is forwarded through a corresponding port associated with the second container.

7. The method of claim 1, wherein each container of the plurality of containers is further associated with a destination address.

8. A switch comprising one or more circuits, wherein the one or more circuits are configured to: receiving a packet associated with a destination; determining a plurality of containers based on the destination of the packet, wherein each container of the plurality of containers is associated with a corresponding port of a plurality of ports; selecting a container from among the plurality of containers for forwarding the packet; and The packet is forwarded through the port associated with the selected container.

9. The switch of claim 8, wherein the one or more circuits are further configured to: receiving an adaptive routing announcement (ARN) packet via a first port of the plurality of ports, wherein the first port of the plurality of ports is associated with a first container; and In response to receiving the ARN packet, the first container is associated with a second port, and the first container is disassociated from the first port.

10. The switch of claim 8, wherein the selected container is selected using a non-deterministic process.

11. The switch of claim 10, wherein the non-deterministic process is random or pseudo-random.

12. The switch of claim 8, wherein the selected container is selected using a round-robin selection process.

13. The switch of claim 8, wherein the one or more circuits are further configured to: receiving a second packet associated with the destination; selecting a second container among the plurality of containers for forwarding the second packet; and The second packet is forwarded through the port associated with the second container.

14. The switch of claim 8, wherein each container of the plurality of containers is further associated with a destination address.

15. A network device comprising: Multiple ports; as well as One or more circuits for: receiving a plurality of packets associated with a destination; determining a plurality of containers based on the destination of the packet, wherein each container of the plurality of containers is associated with a respective port of the plurality of ports; as well as For each group in the plurality of groups: selecting a container from the plurality of containers; as well as The packet is forwarded through the port associated with the selected container.

16. The network device of claim 15, wherein the one or more circuits are further configured to: receiving an adaptive routing announcement (ARN) packet via a first port of the plurality of ports, wherein the first port of the plurality of ports is associated with a first container; and In response to receiving the ARN packet, the first container is associated with a second port, and the first container is disassociated from the first port.

17. The network device of claim 15, wherein the selected container is selected using a non-deterministic process.

18. The network device of claim 17, wherein the non-deterministic process is random or pseudo-random.

19. The network device of claim 15, wherein each container of the plurality of containers is further associated with a destination address.

20. The network device of claim 15, wherein the selected container is selected using a round-robin selection process.

Images