Routing technologies that consider connecting route bandwidth

Abstract

Network switch (102) for routing with link bandwidth consideration, wherein the network switch (102) comprises the following: a large number of entrance ports (106); a large number of output ports (108); and a network switch engine (202) for: Determining an overload level for each of the multitude of output ports (108); Determining a link bandwidth for each of the multiple output ports (108) specifying link bandwidth data; Receiving, from a data link (110) connected to an input port (106) of the multitude of input ports (106) of the network switch (102), a packet containing destination information; Determine, based on the destination information, a primary output port from the plurality of output ports (108) for delivering the packet; Determine, based on the link bandwidth data, at least two alternative output ports from the plurality of output ports different from the primary output port (108); Select, based on the overload level for each of the primary output sport and the at least two alternative output sports, a low overloaded output sport (108) among the primary output sport and the at least two alternative output sports; and Sending the packet via a data link (110) connected to the slightly congested output port (108).

Description

BACKGROUND

[0001] Modern network architectures require adaptive routing to function effectively. Adaptive routing allows a network switch to route a packet along a path that avoids network congestion. It can increase the available bandwidth between two points on the network and improve overall network performance.

[0002] In certain environments, such as high-performance computing (HPC) systems or data centers, high-bandwidth links can be used to connect computing devices and network switches. These high-bandwidth links and the resources required to maintain them can consume a significant amount of power. This is addressed by modulating the bandwidth of each link when high bandwidth is not needed, thus reducing overall power consumption. US 2012 / 0170582 A1, US 2017 / 0048144 A1, and US 2011 / 0038253 A1 address routing in networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The concepts described herein are illustrated in the accompanying figures only as examples and are not exhaustive. For the sake of simplicity and clarity, elements illustrated in the figures are not necessarily drawn to scale. Where deemed appropriate, reference symbols are repeated below the figures to indicate corresponding or analogous elements. Fig. Figure 1 is a simplified block diagram of at least one embodiment of a network for routing with consideration of link bandwidth; Fig. Figure 2 is a simplified block diagram of at least one embodiment of a network switch. Fig. 1; Fig. 3 is a block diagram of at least one embodiment of an environment that is controlled by the network switch of Fig. 1 can be created; Fig. 4 is a simplified flowchart of at least one embodiment of a method for controlling link bandwidth, implemented by the network switch of Fig. 1 can be executed; Fig. Figure 5 is a simplified flowchart of at least one embodiment of a method for distributing congestion data and link bandwidth data through the network switch of Fig. 1 can be executed; and Fig. Figure 6 is a simplified flowchart of at least one embodiment of a routing method that takes link bandwidth into account, implemented by the network switch of Fig. 1 can be executed; THE INVENTION

[0004] The invention is defined by the independent claims. DETAILED DESCRIPTION OF THE DRAWINGS

[0005] Although the concepts of the present disclosure are suitable for a wide variety of modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and will be described in detail here. It is understood, however, that the intention is not to limit the concepts of the present disclosure to the specific disclosed forms; on the contrary, the intention is to cover all modifications, equivalents, and alternatives that are consistent with the present disclosure and the appended claims.

[0006] References in the description to "an embodiment," "a form of embodiment," "an illustrative embodiment," etc., indicate that the described embodiment may include a particular feature, structure, or characteristic. However, not every embodiment may, or necessarily may, include this particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. When a particular feature, structure, or characteristic is described in connection with an embodiment, it is also assumed that it is within the knowledge of a person skilled in the art to achieve such a feature, structure, or characteristic in connection with other embodiments, whether explicitly described or not.Additionally, it is understood that entries in a list of the form "at least one of A, B and C" can mean: (A); (B); (C); (A and B); (B and C); (A and C); or (A, B, and C). Likewise, entries in a list of the form "at least one of A, B or C" can mean: (A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C).

[0007] The disclosed embodiments may in some cases be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more volatile or non-volatile machine-readable (e.g., computer-readable) storage media that can be read and executed by one or more processors. A machine-readable storage medium may be any storage device, storage mechanism, or other physical structure for storing or transmitting information in a machine-readable form (e.g., volatile or non-volatile memory, a media disc, or other media device).

[0008] The drawings may show certain structural or methodological features in specific arrangements and / or sequences. However, it should be noted that such specific arrangements and / or sequences may not be necessary. Rather, in some embodiments, such features may be arranged differently and / or in a different sequence than shown in the illustrative figures. Furthermore, the inclusion of a structural or methodological feature in a specific figure is not intended to imply that such a feature is required in all embodiments and may be omitted or combined with other features in certain embodiments.

[0009] With reference to Fig. Figure 1 comprises a system 100 consisting of several network switches 102, which are interconnected and connected to several computing devices 104. Each of the illustrative network switches 102 is designed to implement a routing algorithm with link bandwidth consideration, which can avoid or reduce congestion and select data link paths 110 with high bandwidth availability. Each output port 108 of each network switch 102 adjusts the available bandwidth based on recent port usage to reduce power consumption. The bandwidth availability and congestion level of each output port 108 are communicated to each input port 106.When selecting which output port 108 a packet from an input port 106 should be routed to, the network switch 102 can first determine a minimum output port 108 associated with a short path to the packet's destination, as well as several alternative output ports 108. The network switch 102 can select the alternative output ports 108 based on link availability from those output ports 108 and then choose the output port 108 with low congestion among the minimum output port 108 and the alternative output ports 108.

[0010] For example, in the Fig. In the illustrative embodiment shown in Figure 1, a first computing device 104-1 communicates with a second computing device 104-2 via one or more data link paths 110 and network switches 102. When a first network switch 102-1 receives a packet from the first computing device 104-1 destined for the second computing device 104-2, it can determine that the minimum output port 108 is the one that connects directly to the second network switch 102-2. If the minimum output port 108 is not congested, the first network switch 102-1 can send the packet to the second network switch 102-2 via the minimum output port 108. Otherwise, the first network switch 102-1 can search for alternative output ports 108, such as the output port 108 connected to a third network switch 102-3.If the alternate output port 108 is less congested than the minimum output port 108, the network switch 102 can send the packet to the third network switch 102-3 using the alternate output port 108.

[0011] The computing device 104 of the system 100 can be implemented as any type of computing device capable of communicating with the network switch 102. For example, the computing device 104 can be implemented as, or be contained in, a server computer, a desktop computer, a smartphone, a cordless phone, a wearable computer, an embedded computing system, a system-on-a-chip (SoC), a tablet computer, a notebook computer, a laptop computer, a handheld device, a messaging device, a camera device, a multiprocessor system, a processor-based system, a consumer electronics device, and / or any other computing device.

[0012] Each data link 110 can be implemented as any type of data link through which the network switches 102 and / or the computing devices 104 can communicate. For example, each data link 110 can be implemented as one or more twisted pairs of wires, one or more coaxial cables, one or more fiber optic cables, or any other wired connection. Additionally or alternatively, in some embodiments, some or all of the data links 110 can be implemented as wireless connections. In the illustrative embodiment, each data link 110 is implemented as a single cable for both sending and receiving data, which is connected to both an input port 106 and an output port 108 of a given network switch 102. Each data link 110 is in Fig. 1 is shown as being split near the end of the data link 110 and represented with a separate input port 106 and output port 108, although it is understood that the data link 110 can be connected to both an input port 106 and an output port 108 in a single connector. For example, each data link 110 can be implemented as a Cat 5, 5e, or 6 category cable with an RJ45 connector. Additionally or alternatively, in some embodiments, a data link 110 can provide only one-way data transfer. For example, a data link 110 can be created or connected from an output port 108 on the first network switch 102-1 to an input port 106 on the second network switch 102-2, without being connected back from an output port 108 on the second network switch 102-2 to an input port 106 on the first network switch 102-1.In the illustrative embodiment, the bandwidth of each data link 110 can be modulated by the output port 108 or the input port 106 to which it is connected. For example, a data link 110 can contain several tracks, which can be implemented as signals on different cables or as signals of different frequencies on the same cable, and the various tracks can be independently enabled and disabled, thereby changing the bandwidth of the data link 110. Of course, the bandwidth of a data link 110 can also be modulated in other ways, such as by changing the frequency of the data link 110, changing the duty cycle of the data link 110, etc.

[0013] With reference to Fig. 2. An illustrative network switch 102 of the system 100 can be implemented as any type of network switch capable of performing the functions described herein. For example, the network switch 102 can be implemented as, or be contained within, a switch, a router, a hub, a server computer, a computing device, an embedded computing system, a system-on-a-chip (SoC), a multiprocessor system, a processor-based system, a consumer electronics device, and / or any other computing device. The illustrative network switch 102 includes a network switch engine 202, one or more input ports 204, one or more output ports 206, and a data storage device 208. In some embodiments, one or more of the illustrative components of the network switch 102 can be incorporated into, or otherwise form part of, another component.For example, in some embodiments the network switch engine 202 or parts thereof may be installed in the input ports 204 or the output ports 206.

[0014] The network switch engine 202 can be implemented as any electrical circuit or component capable of performing the function described here. For example, the network switch engine 202 can be implemented as shown in Fig. Figure 2 shows that the network switch engine 202 can be implemented as a processor 210 and a memory 212, or otherwise include such. In other embodiments, the network switch engine 202 can be implemented as a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an embedded system, and / or other processing or control device, circuit, or assembly of devices and / or circuits. For example, in some embodiments, the network switch engine 202 can have a pipeline architecture composed of one or more discrete processing components. The processor 210 can be implemented as any type of processor capable of performing the functions described herein. For example, the processor 210 can be a single-core or multi-core processor, or...The memory 212 can be implemented as a single- or multi-socket processor, a network processor, a digital signal processor, a graphics processor, a microcontroller, or another type of processor, or as a processing / control circuit. Likewise, the memory 212 can be implemented as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. During operation, the memory 212 can store a variety of data and software used during the operation of the network switch 102, such as operating systems, applications, programs, libraries, and drivers. The memory 212 is communicatively coupled to the processor 210 via an I / O subsystem, which can be implemented as circuits and / or components to facilitate input / output operations with the processor 210, the memory 212, and other components of the network switch 102.The I / O subsystem can be implemented, for example, as memory control hubs, input / output control hubs, firmware devices, communication links (i.e., point-to-point links, bus links, wires, cables, optical fibers, traces on printed circuit boards, etc.), and / or other components and subsystems, or otherwise incorporate them to facilitate input / output operations. In some embodiments, the I / O subsystem can form part of a system-on-a-chip (SoC) and be integrated on a single integrated circuit chip along with the processor 210, the memory 212, and other components of the network switch 102. It is understood that in some embodiments, the network switch engine 202 can be implemented as a parallel or distributed architecture. For example, each input port 204 and each output port 206 can contain an ASIC, an FPGA, a processor 210, a memory 212, etc.include and implement some or all of the functionalities described here.

[0015] Each input port 204 can be implemented as any port capable of receiving data via a data link 110. For example, each input port 204 can be implemented as an electrical connector, an optical antenna, an antenna, etc., and can be connected to any type of data link 110 as described above. Likewise, each input port 206 can be implemented as any port capable of sending data via a data link 110. For example, each output port 206 can be implemented as an electrical connector, an optical antenna, an antenna, etc., and can be connected to any type of data link 110 as described above.

[0016] The data storage device 208 can be designed as any type of device or devices designed for short-term or long-term data storage. For example, the data storage device 208 can include one or more storage devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices.

[0017] Of course, in some embodiments, the network switch 102 may include additional components that are commonly found in a network switch 102 or other computing device, such as one or more peripheral devices 214. The peripheral devices 214 may include one or more buttons, a keyboard, a mouse, a camera, a display, etc.

[0018] With reference to Fig. 3. The network switch 102 in use can include an environment 300, which may include hardware components, software components, firmware components, and / or a combination thereof. The illustrative environment 300 includes a link bandwidth controller 302, a port information distributor 304, a port selector 306, a packet switcher 308, congestion data 310, and link bandwidth data 312. As explained above, in some embodiments, one or more of the components of the environment 300 can be implemented as circuit arrangements or an assembly of electrical devices (e.g., a link bandwidth controller circuit 302, a port information distributor circuit 304, a port selector circuit 306, etc.). It is understood that in such embodiments, the link bandwidth controller circuit 302, the port information distributor circuit 304, the port selector circuit 306, etc.The illustrative components can form part of the network switch engine 202 and / or the processor 210 and / or other components of the network switch 102. Additionally, in some embodiments, one or more of the illustrative components can form part of another component and / or one or more of the illustrative components can be independent of each other. Furthermore, in some embodiments, one or more of the components of the environment 300 can be implemented as virtualized hardware components or emulated architecture, which can be created and maintained by the processor 210 or other components of the network switch 102.

[0019] The link bandwidth controller 302 is designed to control the bandwidth of the data link paths 110 connected to the output ports 206 and / or the input ports 204 of the network switch 102. In the illustrative embodiment, the link bandwidth controller 302 controls the bandwidth of the data link paths 110 from each output port 206 to the corresponding input port 204 at the other end of the data link path 110, and the bandwidth of the data link path 110 to each input port 204 is controlled by the corresponding output port 206 at the other end of the data link path 110. Additionally or alternatively, the link bandwidth controller 302 can control the bandwidth to an input port 204 and from an output port 206 on a given data link path 110. The link bandwidth controller 302 can control the bandwidth in any suitable manner.In the illustrative embodiment, a data link 110 can have multiple tracks which can be independently activated and deactivated. A data link 110 can have a suitable number of tracks, such as 2, 4, 8, or 10 tracks. In the illustrative embodiment, a data link 110 that is activated uses a synchronization signal sent from the output port 206 on one side of the data link 110 to the input port 204 on the other side of the data link 110, whereas a data link 110 that is not activated does not use a synchronization signal.It is understood that when a data link 110 is activated, it can take some time for the synchronization signal to be sent through output port 206 and appropriately processed by input port 204, meaning that activating a data link 110 can take a relatively long time. Deactivating a data link 110 can be done relatively quickly by switching off the synchronization signal.

[0020] As part of controlling the bandwidth of the data link 110, the link bandwidth control 302 is designed to determine whether the current bandwidth availability is too high or too low. The link bandwidth control 302 can determine whether the current bandwidth availability of a data link 110 is too high or too low based on any suitable metric, such as the current occupancy of an output queue associated with the data link 110, the current bandwidth usage of the data link 110, recent occupancy of the output queue associated with the data link 110, and / or recent bandwidth usage of the data link 110. It is understood that in some embodiments, the data link 110 may have a minimum and a maximum permissible bandwidth.In such embodiments, the link bandwidth control 302 will not determine that the current bandwidth availability is too high if the data link 110 is already at the minimum permissible bandwidth, and the link bandwidth control 302 will not determine that the current bandwidth availability is too low if the data link 110 is already at the maximum permissible bandwidth. In the illustrative embodiment, the link bandwidth control 302 determines whether the current bandwidth availability of a data link 110 is too high by determining a recent bandwidth utilization of the data link 110 by calculating an average link bandwidth utilization over a sliding window.If the average link bandwidth is above a predefined threshold relative to the current link bandwidth availability, the link bandwidth controller 302 will activate an additional lane of the data link 110, and if the average link bandwidth is below a predefined threshold relative to the current link bandwidth availability, the link bandwidth controller 302 will deactivate a lane of the data link 110. In the illustrative embodiment, the activation of one lane of the data link 110 corresponds to the minimum bandwidth, and the activation of all lanes on the data link 110 corresponds to the maximum bandwidth.As part of controlling the bandwidth of the data link paths 110, the link path bandwidth control 302 generates the link path bandwidth data 312, which can be distributed and used by the various components of the network switch 102.

[0021] The port information distributor 304 is designed to distribute information about the status of the ports to various components of the network switch 102. In the illustrative embodiment, information relating to each output port 206 is distributed to each input port 204. Additionally or alternatively, information can be collected and processed at a central location, such as a processor 210. The information can be distributed in any suitable way, such as through the primary communication channel used to transport the network data or through a side channel different from the primary communication channel. The side channel can be implemented as any suitable side channel for distributing the information, such as a grid network.

[0022] The port information distributor 304 includes an overload data determiner 314, an overload data distributor 316, and a link bandwidth data distributor 318. The overload data determiner 314 determines overload data 310 for each output port 206, such as by determining a current queue occupancy or a recently existing queue occupancy.

[0023] The overload data distributor 316 is designed to distribute overload data 310 from each output port 206 to each input port 204, and the link bandwidth data distributor 318 is designed to distribute link bandwidth data 312 from each output port 206 to each input port 204. In the illustrative embodiment, the distribution of both the overload data 310 and the link bandwidth data 312 is carried out via a shared side channel, and the overload data 310 and the link bandwidth data 312 cannot be distributed simultaneously via the shared side channel. Therefore, the overload data 310 and the link bandwidth data 312 can be distributed at different times.In the illustrative embodiment, the overload data 310 is distributed frequently, such as at every cycle up to every few cycles of the network switch 102, and the link bandwidth data 312 is distributed less frequently, such as every thousand cycles up to every few thousand cycles of the network switch 102. The illustrative overload data 310 can be distributed as 4 data bits for each output port 206 with an additional bit identifying the data as overload data 310 instead of link bandwidth data 312, and the link bandwidth data 312 can likewise be distributed as 4 data bits for each output port 206 with an additional bit identifying the data as link bandwidth data 312 instead of overload data 310.In the illustrative embodiment, each input port 204 receives the overload data 310 and the link bandwidth data 312 and stores the data locally. Each input port 204 can store the link bandwidth data 312 in lists, with each output port 206 being assigned to a list based on the corresponding link bandwidth data 312.

[0024] Port selector 306 is designed to select an output port 206 for use in transmitting a specific packet received through an input port 204. Port selector 306 includes a minimum output port selector 320 and an alternate output port selector 322. The minimum output port selector 320 is designed to determine a minimum output port 206 (or otherwise a primary output port 206) for use that provides a minimum path (or otherwise preferred path) for the packet to reach its destination. The minimum output port selector 320 can use any suitable approach to determine the minimum output port 206, such as by accessing a routing table.

[0025] The alternate output port selector 322 is designed to select one or more alternate output ports 206. The alternate output port selector 322 selects ports with the highest available link bandwidth from the list of output ports 206. If there are more available output ports 206 in the list than the alternate output port selector 322 needs to select, the alternate output port selector 322 can select the output ports 206 from the list in any way, such as by randomly selecting the output ports 206. If there are fewer available output ports 206 in the list than the alternate output port selector 322 needs to select, then the alternate output port selector 322 selects all of the output ports 206 in the list and then moves to the list of output ports 206 with the next highest available link bandwidth and repeats the above process to select the remaining output ports 206.The alternative output port selector 322 can select any suitable number of alternative output ports 206, such as 1, 2, 4, 5, 10, etc. In the illustrative embodiment, the alternative output port selector 322 selects two alternative output ports 206.

[0026] If the congestion data 310 associated with the minimum output port 206 is below a certain threshold, the port selector 306 can select the minimum output port 206 (and may not even determine the alternative output ports 206). If the congestion data 310 associated with the minimum output port 206 is above the threshold, the port selector 306 can select an output port 206 below the minimum output port 206 selected by the minimum output port selector 320 and the alternative output port 206 selected by the alternative output port selector 322. The port selector 306 compares the congestion data 310 of the minimum port 206 and the alternative output ports 206 and selects the output port 206 with the lowest congestion.In the illustrative embodiment, the port selector 306 applies a base level to the overload data 310 of the minimum port 206 before comparing the overload data 310, so that the minimum port 206 is preferentially selected if the difference between the overload of the minimum port 206 and the output port 206 with the lowest overload is within a threshold value.

[0027] The packet switcher 308 is designed to switch the packet received from an input port 204 to the output port 206 selected by the port selector 306 and to send the packet over the appropriate data link 110. The packet switcher 308 can use any suitable approach for switching and sending the packet, such as using an output queue, and can apply quality-of-service operations, such as dropping or prioritizing the packet when appropriate.

[0028] It goes without saying that the manifold in Fig. The three components described above work together to improve the overall performance of the network switch 102. For example, if the minimum output port 206 is congested, the port selector 306 will consider alternative output ports 206 with higher available bandwidth, thus reducing the congestion at the minimum output port 206. If there are not enough output ports 206 with high link bandwidth availability, the port selector 306 will select an output port 206 with relatively low link bandwidth availability. Because the selected output port 206 uses more bandwidth, the link bandwidth controller 302 will increase the bandwidth of the associated data link 110.Therefore, the available connection bandwidth can be dynamically adjusted, if necessary, based on the use of the corresponding data connection path 110 caused by the port selector 306.

[0029] With reference to Fig. 4. The network switch 102 can execute a method 400 for controlling link bandwidth availability during operation. In some embodiments, the method 400 can represent the logic of the network switch engine 202 (e.g., the logic of an ASIC, an FPGA, or software instructions executed by the processor 210). In the illustrative embodiment, the method 400 can be executed independently at each input port 204. Additionally or alternatively, the method 400 can be executed at some central components of the network switch 102, such as the processor 210. The method 400 begins at block 402, in which the network switch 102 determines a recent link bandwidth utilization of a data link 110.In the illustrative embodiment, the network switch 102 determines a recent bandwidth usage of the data link 110 by determining an average link bandwidth usage over a sliding time window.

[0030] At block 404, the network switch 102 determines whether the available link bandwidth is too high or too low. In the illustrative embodiment, if the average link bandwidth is below a predefined threshold relative to the current link bandwidth availability, the network switch 102 can determine that the available link bandwidth availability is too high, and if the average link bandwidth is above a predefined threshold relative to the current link bandwidth availability, the network switch 102 can determine that the available link bandwidth availability is too low.It is understood that in the illustrative embodiment, if the current link bandwidth is at a minimum level, the network switch 102 will not determine that the available link bandwidth is too high, and if the current link bandwidth is at a maximum level, the network switch 102 will not determine that the available link bandwidth is too low.

[0031] If the link bandwidth at block 406 is not too low, procedure 400 continues to block 414, where network switch 102 makes a decision based on whether the link bandwidth is too high. If the link bandwidth is too low, procedure 400 continues to block 408, where network switch 102 increases the link bandwidth. At block 410, network switch 102 activates one or more additional link lanes of data link 110. At block 410, network switch 102 synchronizes by sending a synchronization signal from output port 206 to a destination input port 204 at the other end of data link 110.It is understood that when a data link 110 is activated, it may take some time until the synchronization signal is sent through the output port 206 and appropriately processed through the input port 204, meaning that activating a data link 110 may take a relatively long time.

[0032] For block 414, if the link bandwidth is not too high, procedure 400 is used in the Fig. The end of procedure 400, as shown in Figure 4, continues where network switch 102 performs other tasks. If the link bandwidth is too high, procedure 400 continues to block 416, where network switch 102 reduces the link bandwidth of data link 110. At block 418, network switch 102 can deactivate one or more link lanes of data link 110 to reduce the link bandwidth.

[0033] With reference to Fig. 5. In operation, the network switch 102 can execute a method 500 for distributing congestion data and link bandwidth data to components in the network switch 102. In some embodiments, the method 500 can represent the logic of the network switch engine 202 (e.g., the logic of an ASIC, an FPGA, or software instructions executed by the processor 210). The method 500 begins at block 502, in which the network switch 102 determines whether congestion data or link bandwidth data should be distributed. The network switch 102 can determine whether congestion data or link bandwidth data should be distributed based on any suitable criteria, such as what data was distributed immediately beforehand, how long ago a type of data was distributed, etc.In the illustrative embodiment, the overload data 310 is distributed frequently, such as at every cycle up to every few cycles of the network switch 102, and the link bandwidth data 312 is distributed less frequently, such as every thousand cycles up to every few thousand cycles of the network switch 102. Therefore, the network switch 102 can determine whether to distribute the link bandwidth data based on how many times the overload data has been distributed since the link bandwidth data was distributed (such as 100, 200, 500, 1000, or 2000 times), or based on how much time has elapsed since the link bandwidth data was distributed (such as 500, 1000, 2000, or 5000 nanoseconds).

[0034] At block 504, if the network switch 102 is to distribute overload data, procedure 500 continues to block 506, where the network switch distributes overload data. The network switch 102 can distribute the information in any suitable way, such as through the primary communication channel used to transport the network data or a side channel different from the primary communication channel. In the illustrative embodiment, at block 508, the network switch 102 distributes the overload information through a side channel. The side channel can be implemented as any suitable side channel for distributing the information, such as a grid network. In the illustrative embodiment, the distributed overload data can be implemented as 4 data bits for each output port 206 with an additional bit identifying the data as overload data 310 instead of link bandwidth data 312.

[0035] Referring back to block 504, if the network switch 102 is not to distribute congestion information, procedure 500 continues to block 510, in which the network switch 102 distributes link bandwidth data. The network switch 102 can distribute the information in any suitable way, such as through the primary communication channel used to transport the network data or a side channel different from the primary communication channel. In the illustrative embodiment, the network switch 102 distributes the congestion information through a side channel received by each input port 204, which executes blocks 512-518 each time the input port 204 receives new link bandwidth data for an output port 206.The distributed link bandwidth data can be implemented as 4 data bits for each output port 206 with an additional bit identifying the data as link bandwidth data instead of overload data.

[0036] At block 512, an input port 204 receives link bandwidth data for an output port 206 of network switch 102 using a side channel. The side channel can be implemented as any suitable side channel for distributing the information, such as a grid network. As discussed above, each input port 204 can maintain multiple lists, each corresponding to a certain available link bandwidth and containing the output ports 206 with the corresponding available link bandwidth. At block 514, input port 204 determines which list output port 206 is currently in. At block 516, if the bandwidth has not changed, input port 204 does not need to change any lists. If the bandwidth has changed, input port 204 moves output port 206 from the previous list to the correct list based on the new link bandwidth data.

[0037] With reference to Fig.6. In operation, the network switch 102 can execute a routing method 600 that takes link bandwidth into account. In some embodiments, the method 600 can represent the logic of the network switch engine 202 (e.g., the logic of an ASIC, an FPGA, or software instructions executed by the processor 210). The method 600 begins at block 602, where, if the network switch 102 has not received a packet at an input port 204, it loops back to block 602 to wait for a packet. If the network switch 102 has received a packet at an input port 204, it proceeds to block 604, where it determines a minimum output port for packet delivery. The network switch 102 can determine a minimum output port 206, which provides a minimum path for the packet to reach its destination.The network switch 102 can use any suitable approach to determine the minimum output port 206, such as by accessing a routing table.

[0038] At block 606, network switch 102 determines an overload of minimum output port 206, such as by accessing corresponding overload data stored locally at input port 204, which received the packet, on minimum output port 206. At block 608, if the overload of minimum output port 206 does not exceed a threshold, procedure 600 continues to block 610, where network switch 102 selects the minimum output port 206 to be used. Procedure 600 then continues from block 610 to block 620, where the packet is sent to the selected port.

[0039] Referring back to block 608, if the overload of the minimum output port 206 exceeds the threshold, procedure 600 continues to block 612, where the network switch 102 determines alternative output ports 206. In the illustrative embodiment, at block 614, the network switch 102 can select ports from the list of output ports 206 with the highest available link bandwidths. If there are more available output ports 206 in the list than the network switch 102 needs to determine, the network switch 102 can select the output ports 206 from the list in any way, such as by randomly selecting them.If there are fewer available output ports 206 in the list than the network switch 102 needs to determine, then the network switch 102 selects all of the output ports 206 in the list, and then considers the list of output ports 206 with the next highest available link bandwidth and repeats the above process to select the remaining output ports 206. In the illustrative embodiment, the network switch 102 determines two alternative output ports 206.

[0040] At block 616, the network switch 102 selects the output port 206 to be used based on the overload data of the minimum and alternative output ports 206. The network switch 102 can compare the overload of the minimum and alternative output ports 206 and select the output port 206 with the lowest overload. In the illustrative embodiment, the network switch 102 applies a baseline level to the overload data of the minimum port 206 before comparing the overload data, so that the minimum port 206 is preferentially selected if the difference between the overload of the minimum port 206 and the output port 206 with the lowest overload is within a threshold value.

[0041] In block 620, network switch 102 sends the packet from the input port 204, which received the packet, to the selected output port 206. In block 622, network switch 102 sends the packet via data link 110, which is connected to the selected output port 206. EXAMPLES

[0042] Illustrative examples of the devices, systems, and methods disclosed herein are provided below. An embodiment of the devices, systems, and methods may include one or more of the examples described below and any combination thereof.

[0043] Example 1 includes a network switch for routing with link bandwidth consideration, wherein the network switch comprises: a plurality of input ports; a plurality of output ports; and a network switching engine for determining a congestion level for each of the plurality of output ports; determining a link bandwidth for each of the plurality of output ports specifying link bandwidth data; receiving, from a data link connected to an input port of the plurality of input ports of the network switch, a packet containing destination information; determining, based on the destination information, a primary output port of the plurality of output ports for delivering the packet; determining, based on the link bandwidth data, at least two alternative output ports from the plurality of output ports other than the primary output port;Select, based on the congestion level for each of the primary output port and the at least two alternative output ports, a less congested output port from among the primary output port and the at least two alternative output ports; and send the packet over a data link connected to the less congested output port.

[0044] Example 2 includes the subject of Example 1 and wherein determining the overload level for each of the multitude of output ports includes receiving the corresponding overload level for each of the multitude of output ports via a side channel of the network switch.

[0045] Example 3 includes the subject of one of Examples 1 and 2, and where for each of the multitude of output ports, the corresponding congestion level comprises a queue occupancy level.

[0046] Example 4 includes the subject of one of Examples 1-3 and where determining the link bandwidth data includes receiving the link bandwidth data via a side channel of the network switch.

[0047] Example 5 includes the subject of one of Examples 1-4 and wherein the link bandwidth data for each of the plurality of output ports includes a number of active link lanes of a data link connected to the corresponding output port.

[0048] Example 6 includes the subject of one of Examples 1-5 and wherein determining the link bandwidth data comprises assigning each input port of the plurality of input ports to each output port of the plurality of output ports a list associated with the corresponding input port, based on the number of active link lanes of the data link associated with the corresponding output port.

[0049] Example 7 includes the subject matter of one of Examples 1-6 and wherein the network switch engine is further to: determine, for each output port of the plurality of output ports, a recent bandwidth usage of a data link connected to the corresponding output port; and determine, for each output port of the plurality of output ports, whether an available link bandwidth of the data link connected to the corresponding output port is too low or too high.

[0050] Example 8 includes the subject matter of one of Examples 1-7 and wherein the network switch engine is furthermore, in response to a determination that the available link bandwidth of the data link connected to at least one output port is too high, to reduce the available link bandwidth of the data link connected to the at least one output port.

[0051] Example 9 includes the subject matter of one of Examples 1-8 and wherein reducing the available link bandwidth of the data link associated with the at least one output port includes disabling one link lane of the data link associated with the at least one output port.

[0052] Example 10 includes the subject matter of one of Examples 1-9 and wherein the network switch engine is furthermore, in response to a determination that the available link bandwidth of the data link connected to at least one output port is too low, is to increase the available link bandwidth of the data link connected to the at least one output port.

[0053] Example 11 includes the subject of one of Examples 1-10 and wherein increasing the available link bandwidth of the data link connected to the at least one output port includes activating a link lane of the data link connected to the at least one output port.

[0054] Example 12 includes the subject of one of Examples 1-11 and wherein activating the link track includes creating a synchronization signal over the link track with a remote input port.

[0055] Example 13 includes the subject of one of Examples 1-12 and where determining the primary output port includes determining the primary output port based on a routing table of the network switch.

[0056] Example 14 includes the subject matter of one of Examples 1-13 and wherein determining the at least two alternative output ports includes selecting the at least two alternative output ports with the highest available link bandwidth based on the link bandwidth data.

[0057] Example 15 includes the subject matter of one of Examples 1-14 and wherein selecting the at least two alternative output ports with the highest available link bandwidth includes randomly selecting the at least two alternative output ports from a set of output ports of the plurality of output ports with the same available link bandwidth.

[0058] Example 16 includes the subject matter of one of Examples 1-15 and wherein selecting the least overloaded output port among the primary output port and the at least two alternative output ports includes selecting the output port with the lowest level of overload among the primary output port and the at least two alternative output ports.

[0059] Example 17 includes the subject matter of one of Examples 1-16 and wherein the network switch engine further applies a baseline level to the congestion level of the primary output port, comprising selecting the slightly congested output port from among the primary output port and the at least two alternative output ports, and comparing the congestion levels of the at least two alternative output ports with the baseline congestion level of the primary output port.

[0060] Example 18 includes a method for link bandwidth-considering routing by a network switch, wherein the method comprises the following steps: Determining, by the network switch, a congestion level for each of a plurality of output ports of the network switch; Determining, by the network switch, a link bandwidth for each of the plurality of output ports specifying link bandwidth data; Receiving, by the network switch and from a data link connected to an input port of a plurality of input ports of the network switch, a packet containing destination information; Determining, by the network switch and based on the destination information, a primary output port of the plurality of output ports for delivering the packet;Determine, by the network switch and based on the link bandwidth data, at least two alternative output ports from the multitude of output ports different from the primary output port; select, by the network switch and based on the congestion level for each of the primary output port and the at least two alternative output ports, a less congested output port below the primary output port and the at least two alternative output ports; and send, by the network switch, the packet over a data link connected to the less congested output port.

[0061] Example 19 includes the subject of Example 18 and includes determining the congestion level for each of the plurality of output ports received by the network switch and for each of the plurality of output ports, the corresponding congestion level via a side channel of the network switch.

[0062] Example 20 includes the subject of one of Examples 18 and 19, wherein for each of the multitude of output ports the corresponding congestion level comprises a queue occupancy level.

[0063] Example 21 includes the subject of one of Examples 18-20 and where determining the link bandwidth data includes receiving the link bandwidth data via a side channel of the network switch.

[0064] Example 22 includes the subject of one of Examples 18-21 and wherein the link bandwidth data for each of the plurality of output ports includes a number of active link lanes of a data link connected to the corresponding output port.

[0065] Example 23 includes the subject matter of one of Examples 18-22 and includes determining the link bandwidth data and assigning it, by each input port of the plurality of input ports, to each output port of the plurality of output ports to a list associated with the corresponding input port, based on the number of active link lanes of the data link associated with the corresponding output port.

[0066] Example 24 includes the subject matter of one of Examples 18-23, and further includes determining, through the network switch and for each output port of the plurality of output ports, a recent bandwidth usage of a data link connected to the corresponding output port; and determining, through the network switch and for each output port of the plurality of output ports, whether an available link bandwidth of the data link connected to the corresponding output port is too low or too high.

[0067] Example 25 includes the subject matter of one of Examples 18-24 and further includes reducing, through the network switch and in response to a determination that the available link bandwidth of the data link connected to at least one output port is too high, the available link bandwidth of the data link connected to the at least one output port.

[0068] Example 26 includes the subject matter of one of Examples 18-25 and wherein reducing the available link bandwidth of the data link connected to the at least one output port includes disabling a link lane of the data link connected to the at least one output port.

[0069] Example 27 includes the subject matter of one of Examples 18-26 and further includes increasing, through the network switch and in response to a determination that the available link bandwidth of the data link connected to at least one output port is too low, the available link bandwidth of the data link connected to the at least one output port.

[0070] Example 28 includes the subject matter of one of Examples 18-27 and includes increasing the available link bandwidth of the data link connected to at least one output port and activating a link lane of the data link connected to the at least one output port.

[0071] Example 29 includes the subject of one of Examples 18-28 and wherein activating the link track includes creating a synchronization signal over the link track with a remote input port.

[0072] Example 30 includes the subject of one of Examples 18-29 and where determining the primary output port includes determining the primary output port based on a routing table of the network switch.

[0073] Example 31 includes the subject matter of one of Examples 18-30 and wherein determining the at least two alternative output ports includes selecting the at least two alternative output ports with the highest available link bandwidth based on the link bandwidth data.

[0074] Example 32 includes the subject matter of one of Examples 18-31 and wherein selecting the at least two alternative output ports with the highest available link bandwidth includes randomly selecting the at least two alternative output ports from a set of output ports of the plurality of output ports with the same available link bandwidth.

[0075] Example 33 includes the subject matter of one of Examples 18-32 and wherein selecting the least congested output port from among the primary output port and the at least two alternative output ports includes selecting the output port with the lowest level of congestion from among the primary output port and the at least two alternative output ports.

[0076] Example 34 includes the subject matter of one of Examples 18-33 and further includes applying a baseline level to the overload level of the primary output sport, wherein selecting the less overloaded output sport from among the primary output sport and the at least two alternative output sports includes comparing the overload levels of the at least two alternative output sports with the baseline overload level of the primary output sport.

[0077] Example 35 includes one or more computer-readable media containing a variety of instructions stored on them which, when executed, cause a network switch to perform the procedure of one of Examples 18-34.

[0078] Example 36 includes a network switch for routing with link bandwidth consideration, wherein the network switch comprises: a plurality of input ports; a plurality of output ports; a network switching engine for receiving, from a data link connected to an input port of the plurality of input ports, a packet containing destination information; a means of determining a congestion level for each of the plurality of output ports of the network switch; a means of determining a link bandwidth for each of the plurality of output ports specifying link bandwidth data; a means of determining, based on the destination information, a primary output port of the plurality of output ports for delivering the packet;A means of determining, based on link bandwidth data, at least two alternative output ports from the plurality of output ports different from the primary output port; and a means of selecting, based on the congestion level for each of the primary output port and the at least two alternative output ports, a less congested output port below the primary output port and the at least two alternative output ports, wherein the network switch engine is further to send the packet via a data link connected to the less congested output port.

[0079] Example 37 includes the subject of Example 36 and wherein the means for determining the overload level for each of the plurality of output ports comprises a means for receiving, for each of the plurality of output ports, the corresponding overload level via a side channel of the network switch.

[0080] Example 38 includes the subject of one of Examples 36 and 37, wherein for each of the multitude of output ports the corresponding congestion level comprises a queue occupancy level.

[0081] Example 39 includes the subject of one of Examples 36-38 and wherein the means for determining the link bandwidth data includes a means for receiving the link bandwidth data via a side channel of the network switch.

[0082] Example 40 includes the subject of one of Examples 36-39 and wherein the link bandwidth data for each of the plurality of output ports includes a number of active link lanes of a data link connected to the corresponding output port.

[0083] Example 41 includes the subject of one of Examples 36-40 and wherein the means of determining the link bandwidth data comprises a means of assigning, by each input port of the plurality of input ports, each output port of the plurality of output ports to a list associated with the corresponding input port, based on the number of active link lanes of the data link associated with the corresponding output port.

[0084] Example 42 includes the subject matter of one of Examples 36-41, and further includes a means of determining, for each output port of the plurality of output ports, a recent bandwidth utilization of a data link connected to the corresponding output port; and a means of determining, for each output port of the plurality of output ports, whether an available link bandwidth of the data link connected to the corresponding output port is too low or too high.

[0085] Example 43 includes the subject matter of one of Examples 36-42 and further includes a means of reducing, in response to a determination that the available link bandwidth of the data link connected to at least one output port is too high, the available link bandwidth of the data link connected to the at least one output port.

[0086] Example 44 includes the subject matter of one of Examples 36-43 and wherein the means for reducing the available link bandwidth of the data link associated with the at least one output port includes a means for disabling a link lane of the data link associated with the at least one output port.

[0087] Example 45 includes the subject matter of one of Examples 36-44 and further includes a means of increasing, in response to a determination that the available link bandwidth of the data link connected to at least one output port is too low, the available link bandwidth of the data link connected to the at least one output port.

[0088] Example 46 includes the subject matter of one of Examples 36-45 and wherein the means for increasing the available link bandwidth of the data link connected to the at least one output port includes a means for activating a link lane of the data link connected to the at least one output port.

[0089] Example 47 includes the subject of one of Examples 36-46 and wherein the means of activating the linking track includes a means of creating a synchronization signal over the linking track with a remote input port.

[0090] Example 48 includes the subject of one of Examples 36-47 and wherein the means of determining the primary output port includes a means of determining the primary output port based on a routing table of the network switch.

[0091] Example 49 includes the subject matter of one of Examples 36-48 and wherein the means for determining the at least two alternative output ports includes a means for selecting the at least two alternative output ports with the highest available link bandwidth based on the link bandwidth data.

[0092] Example 50 includes the subject matter of one of Examples 36-49 and wherein the means of selecting the at least two alternative output ports with the highest available link bandwidth comprises a means of randomly selecting the at least two alternative output ports from a set of output ports of the plurality of output ports with the same available link bandwidth.

[0093] Example 51 includes the subject matter of one of Examples 36-50 and wherein the means for selecting the least overloaded output sport among the primary output sport and the at least two alternative output sports comprises a means for selecting the output sport with the lowest level of overload among the primary output sport and the at least two alternative output sports.

[0094] Example 52 includes the subject matter of one of Examples 36-51 and further includes applying a baseline level to the overload level of the primary output sport, wherein the means for selecting the less overloaded output sport from among the primary output sport and the at least two alternative output sports includes a means for comparing the overload levels of the at least two alternative output sports with the baseline overload level of the primary output sport.

Claims

[1] Network switch (102) for routing with link bandwidth consideration, the network switch (102) comprising: a plurality of input ports (106); a plurality of output ports (108); and a network switch engine (202) for: determining a congestion level for each of the plurality of output ports (108); determining link bandwidth data indicative of a link bandwidth for each of the plurality of output ports (108); Receiving, from a data link (110) connected to an input port (106) of the plurality of input ports (106) of the network switch (102), a packet containing destination information; determining, based on the destination information, a primary output port of the plurality of output ports (108) for delivery of the packet; Determining, based on the link bandwidth data, at least two alternative output ports from the plurality of output ports (108) different from the primary output port; selecting, based on the congestion level for each of the primary output port and the at least two alternative output ports, a lightly congested output port (108) among the primary output port and the at least two alternative output ports; and Sending the packet via a data link (110) connected to the slightly congested output port (108). [2] The network switch (102) of claim 1, wherein determining the link bandwidth data comprises receiving the link bandwidth data over a side channel of the network switch (102). [3] The network switch (102) of claim 2, wherein the link bandwidth data for each of the plurality of output ports (108) comprises a number of active link lanes of a data link (110) connected to the corresponding output port (108). [4] The network switch (102) of claim 3, wherein determining the link bandwidth data comprises, by each input port (106) of the plurality of input ports (106), assigning each output port (108) of the plurality of output ports (108) to a list associated with the corresponding input port (106) based on the number of active link lanes of the data link (110) connected to the corresponding output port (108). [5] The network switch (102) of claim 1, wherein the network switch engine (202) is further to: Determining, for each output port (108) of the plurality of output ports (108), a recent bandwidth usage of a data link (110) connected to the corresponding output port (108); and Determining, for each output port (108) of the plurality of output ports (108), whether an available link bandwidth of the data link (110) connected to the corresponding output port (108) is too low or too high. [6] The network switch (102) of claim 5, wherein the network switch engine (202) is further to, in response to a determination that the available link bandwidth of the data link (110) connected to at least one output port (108) is too high, reduce the available link bandwidth of the data link (110) connected to the at least one output port (108). [7] The network switch (102) of claim 5, wherein the network switch engine (202) is further to increase the available link bandwidth of the data link (110) connected to the at least one output port (108) in response to a determination that the available link bandwidth of the data link (110) connected to the at least one output port (108) is too low. [8] The network switch (102) of claim 1, wherein the network switch engine (202) further applies a baseline level to the congestion level of the primary output port, wherein selecting the lightly congested output port (108) from among the primary output port and the at least two alternate output ports comprises comparing the congestion levels of the at least two alternate output ports to the baseline congestion level of the primary output port. [9] A method for link bandwidth aware routing by a network switch (102), the method comprising the following steps: Determining, by the network switch (102), a congestion level for each of a plurality of output ports (108) of the network switch (102); Determining, by the network switch (102), link bandwidth data indicative of a link bandwidth for each of the plurality of output ports (108); Receiving, by the network switch (102), from a data link (110) connected to an input port (106) of a plurality of input ports (106) of the network switch (102), a packet containing destination information; Determining, by the network switch (102) and based on the destination information, a primary output port of the plurality of output ports (108) for delivery of the packet; Determining, by the network switch (102) and based on the link bandwidth data, at least two alternative output ports from the plurality of output ports (108) different from the primary output port; Selecting, by the network switch (102) and based on the congestion level for each of the primary output port and the at least two alternative output ports, a lightly congested output port (108) among the primary output port and the at least two alternative output ports; and Sending, by the network switch (102), the packet over a data link (110) connected to the lightly congested output port (108). [10] The method of claim 9, wherein determining the link bandwidth data comprises receiving the link bandwidth data via a side channel of the network switch (102). [11] The method of claim 10, wherein the link bandwidth data for each of the plurality of output ports (108) comprises a number of active link lanes of a data link (110) connected to the corresponding output port (108). [12] The method of claim 11, wherein determining the link bandwidth data comprises assigning, by each input port (106) of the plurality of input ports (106), each output port (108) of the plurality of output ports (108) to a list associated with the corresponding input port (106) based on the number of active link lanes of the data link (110) connected to the corresponding output port (108). [13] The method of claim 9, further comprising: Determining, by the network switch (102) and for each output port (108) of the plurality of output ports (108), a recent bandwidth usage of a data link (110) connected to the corresponding output port (108); and Determining, by the network switch (102) and for each output port (108) of the plurality of output ports (108), whether an available link bandwidth of the data link (110) connected to the corresponding output port (108) is too low or too high. [14] The method of claim 13, further comprising decreasing, by the network switch (102) and in response to a determination that the available link bandwidth of the data link (110) connected to the at least one output port (108) is too high, the available link bandwidth of the data link (110) connected to the at least one output port (108). [15] The method of claim 9, further comprising applying a baseline level to the congestion level of the primary output port, wherein selecting the lightly congested output port (108) among the primary output port and the at least two alternative output ports comprises comparing the congestion levels of the at least two alternative output ports with the baseline congestion level of the primary output port. [16] One or more computer-readable media comprising a plurality of instructions stored thereon that, when executed, cause a network switch (102) to: Determining a congestion level for each of a plurality of output ports (108) of the network switch (102); Determining link bandwidth data indicative of a link bandwidth for each of a plurality of output ports (108) of the network switch (102); Receiving, from a data link (110) connected to an input port (106) of the plurality of input ports (106) of the network switch (102), a packet containing destination information; determining, based on the destination information, a primary output port of the plurality of output ports (108) for delivery of the packet; Determining, based on the link bandwidth data, at least two alternative output ports from the plurality of output ports (108) different from the primary output port; selecting, based on the congestion level for each of the primary output port and the at least two alternative output ports, a lightly congested output port (108) among the primary output port and the at least two alternative output ports; and Sending the packet via a data link (110) connected to the slightly congested output port (108). [17] The one or more computer-readable media of claim 16, wherein determining the link bandwidth data comprises receiving the link bandwidth data via a side channel of the network switch (102). [18] The one or more computer-readable media of claim 17, wherein the link bandwidth data for each of the plurality of output ports (108) comprises a number of active link lanes of a data link (110) connected to the corresponding output port (108). [19] The computer-readable medium or multiple computer-readable media of claim 18, wherein determining the link bandwidth data comprises, by each input port (106) of the plurality of input ports (106), assigning each output port (108) of the plurality of output ports (108) to a list associated with the corresponding input port (106) based on the number of active link lanes of the data link (110) connected to the corresponding output port (108). [20] The computer-readable medium or multiple computer-readable media of claim 16, wherein the plurality of instructions further causes the network switch (102) to: Determining, for each output port (108) of the plurality of output ports (108), a recent bandwidth usage of a data link (110) connected to the corresponding output port (108); and Determining, for each output port (108) of the plurality of output ports (108), whether an available link bandwidth of the data link (110) connected to the corresponding output port (108) is too low or too high. [21] The one or more computer-readable media of claim 20, wherein the plurality of instructions further causes the network switch (102) to, in response to a determination that the available link bandwidth of the data link (110) connected to the at least one output port (108) is too high, decrease the available link bandwidth of the data link (110) connected to the at least one output port (108). [22] The one or more computer-readable media of claim 16, wherein the plurality of instructions further cause the network switch (102) to apply a baseline level to the congestion level of the primary egress port, wherein selecting the lightly congested egress port (108) from among the primary egress port and the at least two alternate egress ports comprises comparing the congestion levels of the at least two alternate egress ports to the baseline congestion level of the primary egress port.

Images