Ultra-high radix communication network

Abstract

A communication network includes a first switch interconnected with a second switch. Each first switch includes a first integrated circuit (IC) switch chip, a downlink port, and an uplink port. Each second switch includes a port coupled to at least one uplink port of each of the first switches and a second IC switch chip located in the IC package. In order to allow each second IC switch chip to forward packets between a large number of first switches and to reduce the number of external interconnects of the IC package, each second IC switch chip comprises a set of multiplexer / demultiplexer circuitry, each multiplexer / demultiplexer circuitry is coupled between the external interconnect and a set of the plurality of internal network interfaces of the second IC switch chip. The multiplexer / demultiplexer circuitry demultiplexes data streams from the external interconnect to the plurality of internal network interfaces, and multiplexes a plurality of data streams from the plurality of internal network interfaces to the external interconnect.

Description

Cross-references to related applications

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 532,060, filed August 10, 2023, entitled “Scalable Data Center Network Architecture,” the disclosure of which is expressly incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure generally relates to communication networks, and more specifically, to networks with multi-layer switches. Background Technology

[0003] The methods described in this Background section are possible methods, but not necessarily methods that have been previously conceived or adopted. Therefore, unless otherwise specified, they should not be considered to constitute prior art simply because any of the methods described are included in this section.

[0004] Some network applications require switching between a very large number of ports. For example, a typical data center includes a large number of servers, and switches used to interconnect these servers and communicatively couple them to external network connections such as backbone network links. As another example, some artificial intelligence / machine learning (AI / ML) systems include a large number of processors (e.g., graphics processing units (GPUs)) interconnected through multi-layered networks. In such applications, switching systems capable of switching between numerous ports are utilized to forward traffic between servers, GPUs, backbone network lines, etc. Such switching systems can include a large number of switches, each typically capable of switching between multiple ports. Multi-layered switches are frequently used in data centers, server farms, AI systems, etc., where first-layer switches interconnect second-layer switches, and where the second-layer switches connect to processors, servers, storage devices, etc.

[0005] Expanding a communication network as described above typically involves adding a switching layer. However, adding a switching layer usually increases latency, power consumption, physical footprint, and cost, which can be problematic for some applications. Summary of the Invention

[0006] In one embodiment, a communication network includes: a plurality of first switches, each first switch having: i) a corresponding first integrated circuit (IC) switching chip having a plurality of network interfaces, ii) a corresponding plurality of downlink ports, and iii) a corresponding plurality of uplink ports; and a plurality of second switches, each second switch having: i) a corresponding plurality of ports coupled to at least one uplink port of each of the first switches, ii) a corresponding second IC switching chip in a corresponding IC package having a plurality of external network interfaces coupled to external interconnects of the IC package, and iii) a corresponding plurality of serializers / deserializers (SERDES) communicatively coupling the corresponding plurality of external network interfaces to the corresponding plurality of ports of the second switches. Each second IC switching chip further includes: a plurality of internal network interfaces; a packet processor coupled to the plurality of internal network interfaces, the packet processor being configured to forward packets between the internal network interfaces; and a plurality of multiplexer / demultiplexer circuit devices, each multiplexer / demultiplexer circuit device being coupled to: i) a corresponding external network interface, and ii) the corresponding set of the plurality of internal network interfaces, each multiplexer / demultiplexer circuit device being configured to: i) demultiplex a first data stream received from the corresponding external network interface into a second data stream and a third data stream for transmission to the corresponding set of the plurality of internal network interfaces, and ii) multiplex a fourth data stream and a fifth data stream received via the corresponding set of the plurality of internal network interfaces into a sixth data stream for transmission to the corresponding external network interface.

[0007] In another embodiment, a method for communicating in a communication network includes: i) a plurality of first network switches, and ii) a plurality of second network switches, each first network switch including: i) a corresponding plurality of downlink ports, ii) a corresponding plurality of uplink ports, and iii) a corresponding one or more first integrated circuit (IC) switching chips, the corresponding one or more first IC switching chips being communicatively coupled to the plurality of uplink ports and the plurality of downlink ports. Each second network switch includes: i) a corresponding plurality of ports, the corresponding plurality of ports being coupled to at least one uplink port of each of the first network switches, and ii) a corresponding second IC switching chip, the second IC switching chip being communicatively coupled to the plurality of ports of the second switch. The method includes: receiving packets from a network device via a plurality of downlink ports at each of the plurality of first network switches; forwarding the packets received via the plurality of downlink ports to the plurality of second network switches via corresponding plurality of communication links between each first network switch and each of the plurality of second network switches; transmitting the packets received from the plurality of first switches to an external network interface of a second IC switching chip at each second switch; demultiplexing a corresponding first stream of packets received via the external network interface to a plurality of internal network interfaces of the second IC switching chip in conjunction with each external network interface of the second IC switching chip; and at each second IC switching chip, [further details about the method are needed]. The internal network interfaces of the second IC switching chip forward packets received via the internal network interface of the second IC switching chip; in combination with each external network interface of the second IC switching chip, a second stream and a third stream of packets received via the multiple internal network interfaces of the second IC switching chip are multiplexed to the external network interface; packets received from the external network interface of the corresponding second IC switching chip are forwarded to the multiple first network switches by each second network switch via the corresponding multiple communication links between the second network switch and each of the multiple first network switches; and packets received from the multiple second network switches are transmitted to the multiple network devices by each of the multiple first network switches via the multiple downlink ports. Attached Figure Description

[0008] Figure 1A This is a simplified schematic diagram of an example communication network according to an embodiment, which includes a large number of computing processors interconnected through an example Layer 2 network having multiple large-scale network switches.

[0009] Figure 1B According to the illustrated embodiment Figure 1A A simplified block diagram of two network switches with communication coupling.

[0010] Figure 1C According to the embodiments Figures 1A to 1B A simplified schematic diagram of the switching chips in some network switches.

[0011] Figure 1D According to the embodiments Figures 1A to 1B A simplified schematic diagram of the switching chips in some network switches.

[0012] Figure 2A This is a simplified schematic diagram of another example communication network according to another embodiment, which includes a large number of computing processors interconnected through an example Layer 2 network having multiple large-scale network switches.

[0013] Figure 2B According to the illustrated embodiment Figure 2A A simplified block diagram of two network switches with communication coupling.

[0014] Figure 2C According to another embodiment Figure 2A A simplified block diagram of two network switches with communication coupling.

[0015] Figure 2D According to the embodiments Figure 2A and 2C A simplified schematic diagram of the switching chips in some network switches.

[0016] Figure 3 This is a flowchart of an example method for communicating in a communication network according to an embodiment, the communication network including i) a plurality of first network switches and ii) a plurality of second network switches communicatively coupled to the plurality of first network switches.

[0017] Figure 4 This is a simplified block diagram of an example multiplexer circuit arrangement according to an embodiment, the example multiplexer circuit arrangement being included in, for example, Figure 1A and 2A In the communication network shown in the diagram.

[0018] Figure 5 This is a simplified block diagram of an example demultiplexer circuit arrangement according to an embodiment, the example demultiplexer circuit arrangement being included in, for example, Figure 1A and 2A In the communication network shown in the diagram.

[0019] Figure 6This is a simplified block diagram of an example multiplexer circuit arrangement according to another embodiment, which includes... Figure 2C In the switch.

[0020] Figure 7A This is a simplified block diagram of an example multiplexer circuit arrangement according to another embodiment, the example multiplexer circuit arrangement being included in, for example, Figure 1A and 2A In the communication network shown in the diagram.

[0021] Figure 7B According to the embodiments, by Figure 7A A simplified schematic diagram of an example interleaving function performed by a codeword interleaver circuit.

[0022] Figure 8A This is a simplified block diagram of an example demultiplexer circuit arrangement according to another embodiment, the example demultiplexer circuit arrangement being included in, for example, Figure 1A and 2A In the communication network shown in the diagram.

[0023] Figure 8B According to the embodiments, by Figure 8A A simplified schematic diagram illustrating an example of deinterleaving functionality performed by a codeword deinterleaving circuit. Detailed Implementation

[0024] Figure 1A This is a simplified schematic diagram of an example communication network 100 according to an embodiment, which includes a large number of computing processors interconnected via an example Layer 2 network 104 having multiple large-scale network switches. For example, network 104 is coupled to multiple computing container groups 108, each including multiple computing processors, such as GPUs 112. In one embodiment, the number of computing container groups 108 is 1024. In another embodiment, the number of computing container groups 108 is a suitable number greater than or equal to 900. In another embodiment, the number of computing container groups 108 is a suitable number greater than or equal to 1000. In yet another embodiment, the number of computing container groups 108 is another suitable number.

[0025] In some embodiments, each of at least some of the computing processors in computing processor 112 includes a graphics processing unit (GPU), and for ease of explanation, computing processor 112 is sometimes referred to herein as GPU 112. In other embodiments, each of at least some of the computing processors in computing processor 112 includes a suitable processor other than a GPU, such as a central processing unit (CPU), a digital signal processor (DSP), a graph processor, etc. In some embodiments, at least some of the GPUs in GPU 112 are replaced by other suitable network devices, such as memory devices, network switches, etc.

[0026] In one embodiment, the number of GPUs 112 in each compute container group 108 is 256. In another embodiment, the number of GPUs 112 in each compute container group 108 is a suitable number greater than or equal to 200. In another embodiment, the number of GPUs 112 in each compute container group 108 is a suitable number greater than or equal to 250. In another embodiment, the number of GPUs 112 in each compute container group 108 is another suitable number. In one embodiment, each compute container group 108 includes the same number of GPUs 112. In another embodiment, at least some compute container groups 108 include different numbers of GPUs 112.

[0027] In one embodiment, the communication network 100 includes 262,144 GPUs 112 interconnected via network 104. In another embodiment, the communication network 100 includes at least 200,000 GPUs 112 interconnected via network 104. In yet another embodiment, the communication network 100 includes at least 250,000 GPUs 112 interconnected via network 104. In still other embodiments, the communication network 100 includes another suitable number of GPUs 112 interconnected via network 104.

[0028] Each compute container group 108 is communicatively coupled to a corresponding network switch 116 of network 104. For example, each GPU 112 of compute container group 108 is communicatively coupled to the corresponding network switch 116 via a suitable cable 120 (such as a power cable, optical fiber, etc.). In an embodiment, each GPU 112 includes (or is coupled to) a port (not shown; e.g., an electrical port, an optical port, etc.) configured to be coupled to cable 120 and communicate at a data rate of at least 100 gigabits per second (Gbps). The port of GPU 112 is communicatively coupled to network switch 116 via cable 120, which is configured to communicate at a data rate of at least 100 Gbps. In an embodiment, cable 120 is rated for 100 Gbps Ethernet (GE). In another embodiment, cable 120 is rated for a data rate higher than 100 GE.

[0029] In one embodiment, each network switch 116 includes multiple ports (sometimes referred to herein as "downlink ports"; not shown; e.g., electrical ports, optical ports, etc.) to which communication cables 120 are connected. Each of at least some of the downlink ports is configured to communicate at a data rate of at least 100 Gbps. In one embodiment, network switch 116 includes a number of downlink ports equal to or greater than the number of GPUs 256 in compute container group 108. In one embodiment, the number of downlink ports in each network switch 116 is at least 256. In another embodiment, the number of downlink ports in each network switch 116 is an appropriate number greater than or equal to 200. In another embodiment, the number of downlink ports in each network switch 116 is an appropriate number greater than or equal to 250. In another embodiment, the number of downlink ports in each network switch 116 is another appropriate number. In one embodiment, each network switch 116 includes the same number of downlink ports. In another embodiment, at least some network switches 116 include different numbers of downlink ports.

[0030] Each network switch 116 is communicatively coupled to multiple network switches 124 via multiple communication cables 128 (e.g., electrical cables, optical fibers, etc.). In one embodiment, the cable 128 is rated for a data rate of 100 GE. In another embodiment, the cable 128 is rated for a data rate higher than 100 GE. In one embodiment, each network switch 116 includes multiple ports (sometimes referred to herein as "uplink ports"; not shown; e.g., electrical ports, optical ports, etc.) to which the communication cables 128 are connected. Each of at least some of the uplink ports is configured to communicate at a data rate of at least 100 Gbps.

[0031] Each of at least some of the network switches in network switch 124 is communicatively coupled to a plurality of network switches 116. In an embodiment, each network switch 124 includes a plurality of ports (not shown; e.g., electrical ports, optical ports, etc.), and communication cables 128 are connected to the plurality of ports. Each of at least some of the ports is configured to communicate at a data rate of at least 100 Gbps. For each of the at least some of the network switches in network switch 124, the number of ports is at least the same as the number of network switches 116. Therefore, in Figure 1A In the illustrated example, there are 1024 network switches 116, and each of at least some of these network switches includes at least 1024 ports. In another embodiment, the number of ports for each network switch 124 is an appropriate number greater than or equal to 1000. In another embodiment, the number of ports for each network switch 124 is an appropriate number greater than or equal to 900. In another embodiment, the number of ports for each network switch 124 is another appropriate number. In one embodiment, each network switch 124 includes the same number of ports. In another embodiment, at least some of the network switches 124 include different numbers of ports.

[0032] Each network switch 124 includes a switching integrated circuit (IC) 140, sometimes referred to herein as a "switching chip". Switching chip 140 includes multiple internal network interfaces (not shown) communicatively coupled to ports of network switch 124. Switching chip 140 also includes other components (not shown), such as a memory for storing packet data, a packet processor for analyzing at least the packet header data of packets received via internal network interfaces to determine which internal network interface the packet will be forwarded through.

[0033] The switching chip 140 is included in a suitable chip package having suitable external interconnect structures for inputting / outputting signals to / from the switching chip 140, such as a ball grid array (BGA), a pin grid array (PGA), etc.

[0034] As described above, each of at least some of the network switches in network switch 124 includes at least 1000 ports, which, in some embodiments, are communicatively coupled to switch chip 140. For many IC manufacturing and / or chip packaging technologies, it is difficult to commercially produce a chip package with a switch IC having more than 1000 high-speed external connections. For example, a BGA chip package with a switch IC having more than 1000 network interfaces would require well over 1000 solder balls solely for the network interfaces themselves, and the high-speed nature of the network interfaces would require a large number of additional solder balls specifically for grounding and power connections.

[0035] To significantly reduce the number of external connections required by the switching chip 140, the switching chip 140 includes an external network interface for each of at least some sets of multiple ports, wherein one external network interface is connected to an external interconnect of the IC package, the external interconnect transmitting / receiving packet data corresponding to the sets of multiple ports. For example, in one embodiment, transmitted packet data corresponding to the sets of multiple ports is multiplexed within a combined transmit signal, and received packet data corresponding to the sets of multiple ports is multiplexed within a combined receive signal. Figure 1A In the example, each network switch 124 includes 1024 ports, and the switching chip 140 includes 512 external network interfaces, where each external network interface corresponds to a corresponding port pair.

[0036] On the other hand, in some embodiments, the switching chip 140 includes at least as many internal network interfaces as the number of ports of the network switch 124, and the internal network interfaces are communicatively coupled to external network interfaces via multiplexer / demultiplexer circuitry (not shown). A packet processor (not shown) is configured to forward packets received via the internal network interfaces between the internal network interfaces.

[0037] The network switch 124 includes multiple port modules 144, which are communicatively coupled to the external network interface of the switching chip 140 via corresponding communication links 148. The port modules 144 are optical modules 144. In this embodiment, and... Figure 1A The diagram illustrates optical module 144. In other embodiments, port module 144 includes an electrical port module.

[0038] Each of at least some of the communication links in communication link 148 operates at a data rate at least twice the data rate at which port 144 operates. For example, in an embodiment where port 144 operates at a data rate of 100 Gbps, communication link 148 operates at a data rate of at least 200 Gbps. Each of at least some of the port modules in port module 144 corresponds to a corresponding port pair (e.g., an electrical port, an optical port, etc.), and each of at least some of the communication links in communication link 148 transmits / receives data corresponding to the port pair.

[0039] Each port module 144 includes a multiplexer / demultiplexer circuit arrangement (mux / demux) 152, which is configured to: i) multiplex data received via port pairs (e.g., each 100 Gbps) into a combined receive stream for transmission to switching chip 140 via communication link 148 (e.g., at 200 Gbps), and ii) demultiplex a combined transport stream received from switching chip 140 via communication link 148 (e.g., at 200 Gbps) into a transport stream pair for transmission by port pairs (e.g., at 100 Gbps).

[0040] Figure 1B According to the embodiments Figure 1A A simplified block diagram of network switch 124, in which one network switch is communicatively coupled to two network switches in network switch 116. See below for reference. Figure 1A discuss Figure 1B .

[0041] Switching chip 140 includes a plurality of serializers / deserializers (SERDES) 160, each corresponding to a respective external network interface of switching chip 140. According to an embodiment, each SERDES 160 is communicatively coupled to an optical port pair of network switch 124. Figure 1B For ease of explanation, only one SERDES 160 is illustrated, but as stated above, the switching chip 140 includes multiple SERDES 160s. SERDES 160s are configured to operate at a data rate at least twice the data rate at which port 144 operates. For example, in an embodiment where port 144 operates at a data rate of 100 Gbps, SERDES 160s are configured to operate at a data rate of at least 200 Gbps.

[0042] SERDES 160 is communicatively coupled to the corresponding optical port module 144 via the corresponding communication link 148. Figure 1BFor ease of explanation, only one optical port module 144 is illustrated, but as described above, the network switch 124 includes multiple optical port modules 144. Each optical port module 144 includes SERDES 164 coupled to a communication link 148. SERDES 164 is configured to operate at a data rate at least twice the data rate of the optical port. For example, in an embodiment where the optical port operates at a data rate of 100 Gbps, SERDES 164 is configured to operate at a data rate of at least 200 Gbps. SERDES 164 is coupled to a mux / demux 152.

[0043] The optical port module 144 also includes a corresponding optical transceiver 168 for each optical port. Each optical transceiver 168 is configured to: i) receive an optical signal via a corresponding cable 128 (e.g., a 100 Gbps optical signal), ii) convert the optical signal into an electrical signal (e.g., a 100 Gbps electrical signal), and iii) provide the electrical signal to the mux / demux 152. Each optical transceiver 168 is also configured to: i) receive an electrical signal (e.g., a 100 Gbps electrical signal) from the mux / demux 152, ii) convert the electrical signal into an optical signal (e.g., a 100 Gbps optical signal), and iii) provide the optical signal to the corresponding cable 128.

[0044] Each network switch 116 includes a corresponding switching chip 172, which is coupled to multiple optical port modules 176. Figure 1B For ease of explanation, only one optical port module 176 is illustrated in each network switch 116, but each network switch 116 includes multiple optical port modules 176. The optical port module 176 is communicatively coupled to the switching chip 172 via a communication link 178. The communication link 178 operates at a data rate at least as high as the data rate of the optical port. For example, in an embodiment where the optical port operates at a data rate of 100 Gbps, the communication link 178 operates at a data rate of at least 100 Gbps.

[0045] Optical port module 176 includes optical transceivers 180 corresponding to optical ports of network switch 116. Each optical transceiver 180 is configured to: i) receive optical signals (e.g., 100 Gbps optical signals) via cable 128, ii) convert the optical signals into electrical signals (e.g., 100 Gbps electrical signals), and iii) provide the electrical signals to switching chip 172. Each optical transceiver 180 is also configured to: i) receive electrical signals (e.g., 100 Gbps electrical signals) from switching chip 172, ii) convert the electrical signals into optical signals (e.g., 100 Gbps optical signals), and iii) provide the optical signals to cable 128.

[0046] The network switch 116 also includes a plurality of optical port modules 182, each corresponding to a specific downlink port of the network switch 116. In an embodiment, the optical port modules 182 have the same or similar structure as the optical port module 176. In an embodiment, each optical port module 182 is configured to operate at a rate of at least 100 Gbps.

[0047] Optical port module 182 is communicatively coupled to switching chip 172 via communication link 184. Each communication link 184 operates at a data rate at least as high as the data rate of optical port module 182. For example, in an embodiment where the optical port operates at a data rate of 100 Gbps, each communication link 184 operates at a data rate of at least 100 Gbps.

[0048] In other embodiments, optical port modules 144, 176 are replaced by electrical port modules including electrical transceivers, and cable 128 is a power cable.

[0049] Figure 1C According to the embodiments Figures 1A to 1B A simplified schematic diagram of the switching chip 140 of the network switch 124. The switching chip 140 includes multiple internal network interfaces 186. Figure 1C In one example embodiment, the switching chip 140 includes 1024 internal network interfaces 186. In another embodiment, the switching chip 140 includes at least as many internal network interfaces 186 as the network switch 116. In one embodiment, the switching chip 140 includes at least 1000 internal network interfaces 186. In another embodiment, the switching chip 140 includes at least 950 internal network interfaces 186. In yet another embodiment, the switching chip 140 includes at least 900 internal network interfaces 186.

[0050] The switching chip 140 also includes a packet processor 188 coupled to a plurality of internal network interfaces 186. The packet processor 188 is configured to forward packets between the internal network interfaces 186. For example, the packet processor 188 is configured to analyze at least the packet header data of packets received via the internal network interfaces 186 to determine the internal network interfaces 186 through which the packets will be forwarded.

[0051] According to an embodiment, the packet processor 188 has appropriate processing capabilities to operate in an environment where the internal network interface 186 is communicatively coupled to a large number of switches 116 via a link of at least 100 Gbps.

[0052] The switching chip 140 includes other suitable components (not shown), such as a memory for storing packet data, a memory management circuit device, etc.

[0053] To significantly reduce the number of external connections required by the switching chip 140, the switching chip 140 includes a SERDES for each of at least a set 186 of multiple internal network interfaces, wherein one SERDES transmits / receives packet data corresponding to the set 186 of the multiple internal network interfaces. For example, in an embodiment, transmitted packet data corresponding to the set 186 of the multiple internal network interfaces is multiplexed within a combined transmit signal, and received packet data corresponding to the set 186 of the multiple internal network interfaces is multiplexed within a combined receive signal. Figure 1A In the example, each network switch 124 includes 1024 ports, and the switching chip 140 includes 512 SERDES 160, where each SERDES 160 corresponds to a corresponding internal network interface pair 186.

[0054] As described above, each SERDES 160 is coupled to the corresponding port module 144.

[0055] The switching chip 140 also includes multiple multiplexers / demultiplexers (mux / demux) 190, which are coupled to multiple SERDES 160. Each mux / demux 190 is configured to: i) multiplex data received via internal network interface pair 186 (e.g., each 100 Gbps) into a combined transport stream for transmission to a corresponding port module 144 via communication link 148 (e.g., at 200 Gbps); and ii) demultiplex the combined receive stream received from port module 144 via communication link 148 (e.g., at 200 Gbps) into a receive stream pair for transmission to internal network interface pair 186 (e.g., at 100 Gbps).

[0056] In this embodiment, each SERDES 160 and mux / demux 190 corresponds to an external network interface of the IC switching chip 140. According to this embodiment, each SERDES 160 is coupled to an external interconnect (e.g., solder balls, pins, etc.) of the IC package that encapsulates the switching chip 140.

[0057] Figure 1D According to the embodiments Figures 1A to 1B A simplified schematic diagram of the switching chip 172 of the network switch 116. The switching chip 172 includes multiple network interfaces 194. Figure 1DIn an example embodiment, switch chip 172 includes 512 network interfaces 194. In another embodiment, switch chip 172 includes at least as many network interfaces 194 as the sum of: a) the number of GPUs 112 to which switch chip 172 is communicatively coupled, and b) the number of network switches 124 to which switch chip 172 is communicatively coupled. In another embodiment, switch chip 140 includes at least 500 network interfaces 194. In yet another embodiment, switch chip 172 includes at least 475 network interfaces 194. In yet another embodiment, switch chip 140 includes at least 450 network interfaces 194.

[0058] The switching chip 172 also includes a packet processor 196 coupled to a plurality of network interfaces 194. The packet processor 196 is configured to forward packets between the network interfaces 194. For example, the packet processor 196 is configured to analyze at least the packet header data of packets received via the network interfaces 194 to determine the network interfaces 194 through which the packets will be forwarded.

[0059] According to an embodiment, the packet processor 196 has appropriate processing capabilities to operate in an environment where the network interface 194 is communicatively coupled to a large number of GPUs 112 and a large number of switches 116 via a link of at least 100Gbps.

[0060] The switching chip 196 includes other suitable components (not shown), such as a memory for storing packet data, memory management circuitry, etc.

[0061] Now for reference Figure 1B and 1D Each network interface 194 includes or is coupled to SERDES 198. Each SERDES 198 is configured to operate at a data rate of 100 Gbps. SERDES 198-1 to 198-256 correspond to the uplink ports of switch 116 and are coupled to the corresponding port modules 176 via the corresponding communication links 180. SERDES 198-257 to 198-512 correspond to the downlink ports of switch 116 and are coupled to the corresponding port modules 176 via the corresponding communication links 178.

[0062] According to an embodiment, each SERDES 198 is coupled to an external interconnect (e.g., solder balls, pins, etc.) of the IC package of the packaged switching chip 172.

[0063] Refer again Figure 1ANetwork 104 provides low-latency communication between GPUs 112 in a large number of GPUs (e.g., at least 200,000 GPUs 112). For example, in one embodiment, each GPU 112 in a first compute container group 108 is communicatively connected to another GPU 112 in a second compute container group 108 via a number of alternative paths, where each path traverses at most three network switches 116, 124.

[0064] In an embodiment where the communication network 100 includes 1,024 network switches 116 and 256 network switches 124, the interconnection between the network switches 116 and the network switches 124 includes 262,144 cables 128.

[0065] Figure 2A This is a simplified schematic diagram of another example communication network 200 according to another embodiment, which includes a large number of computing processors interconnected via an example Layer 2 network 204 having multiple large-scale network switches. Communication network 200 includes... Figure 1A Some components of the example communication network 100 are shown below. For the sake of brevity, components with the same number will not be described in detail.

[0066] Network 204 is coupled to multiple compute container groups 108. In one embodiment, the number of compute container groups 108 is 1024. In another embodiment, the number of compute container groups 108 is a suitable number greater than or equal to 900. In another embodiment, the number of compute container groups 108 is a suitable number greater than or equal to 1000. In yet another embodiment, the number of compute container groups 108 is another suitable number.

[0067] Each pair of compute container groups 108 is communicatively coupled to a corresponding network switch 216 of network 204. For example, each GPU 112 of compute container group 108-1 is communicatively coupled to the corresponding network switch 216 via an appropriate cable 120 (such as a power cable, fiber optic cable, etc.).

[0068] In one embodiment, each network switch 216 includes multiple ports (sometimes referred to herein as "downlink ports"; not shown; e.g., electrical ports, optical ports, etc.) to which communication cables 120 are connected. Each of at least some of the downlink ports is configured to communicate at a data rate of at least 100 Gbps. In one embodiment, network switch 216 includes a number of downlink ports equal to or greater than the number of GPUs 112 in the compute container pair 108. In one embodiment, the number of downlink ports in each network switch 216 is at least 512. In another embodiment, the number of downlink ports in each network switch 216 is an appropriate number greater than or equal to 400. In another embodiment, the number of downlink ports in each network switch 216 is an appropriate number greater than or equal to 500. In another embodiment, the number of downlink ports in each network switch 216 is another appropriate number. In one embodiment, each network switch 216 includes the same number of downlink ports. In another embodiment, at least some network switches 216 include different numbers of downlink ports.

[0069] Each network switch 216 is communicatively coupled to multiple network switches 224 via multiple communication cables 228 (e.g., electrical cables, optical fibers, etc.). In one embodiment, the cable 228 is rated for a data rate of 200 GE. In another embodiment, the cable 228 is rated for a data rate higher than 200 GE. In one embodiment, each network switch 216 includes multiple ports (sometimes referred to herein as "uplink ports"; not shown; e.g., electrical ports, optical ports, etc.) to which the communication cables 228 are connected. Each of at least some of the uplink ports is configured to communicate at a data rate of at least 200 Gbps.

[0070] Each of at least some of the network switches in network switch 224 is communicatively coupled to a plurality of network switches 216. In an embodiment, each network switch 224 includes a plurality of ports (not shown; e.g., electrical ports, optical ports, etc.), to which communication cables 228 are connected. Each of at least some of the ports is configured to communicate at a data rate of at least 200 Gbps. For each of the at least some of the network switches in network switch 224, the number of ports is at least the same as the number of network switches 216. Therefore, in Figure 2AIn the illustrated example, there are 512 network switches 216, and each of at least some of the network switches 224 includes at least 512 ports. In another embodiment, the number of ports in each network switch 224 is an appropriate number greater than or equal to 500. In another embodiment, the number of ports in each network switch 224 is an appropriate number greater than or equal to 450. In another embodiment, the number of ports in each network switch 224 is another appropriate number. In one embodiment, each network switch 224 includes the same number of ports. In another embodiment, at least some of the network switches 224 include different numbers of ports.

[0071] Each network switch 224 includes the switching chip 140 discussed above.

[0072] In an embodiment where the communication network 200 includes 512 network switches 216 and 256 network switches 224, the interconnection between network switches 216 and network switches 224 includes 131,072 cables 228, which is significantly fewer than... Figure 1A The number of cables 128 (262,144 cables) in the example communication network 100 (at least in some embodiments).

[0073] In this embodiment, each network switch 216 communicates with network switch 224 via a 200 Gbps stream and with GPU 112 via a 100 Gbps stream. Therefore, according to this embodiment, each network switch 216 includes a port module 244, and each port module 244 has a corresponding mux / demux 256. The port module 244 is an optical module 244, and in this embodiment, and Figure 2A The diagram illustrates optical module 244. In other embodiments, port module 244 includes an electrical port module.

[0074] Each mux / demux 256 is configured to: i) multiplex a first stream pair at a first data rate (e.g., 100 Gbps) into a first combined stream at a higher second data rate (e.g., 200 Gbps), and ii) demultiplex a second combined stream at a higher second data rate (e.g., 200 Gbps) into a second stream pair at the first data rate (e.g., 100 Gbps).

[0075] Figure 2B According to the embodiments Figure 2A A simplified block diagram showing that communication between one network switch in network switch 224 is coupled to one network switch in network switch 216. See below for reference. Figure 2A discuss Figure 2B .

[0076] Similar to Figures 1A to 1B Network switches 124 and 224 each include a switching chip 140. Each network switch 224 includes a plurality of port modules 226, which are communicatively coupled to an external network interface of the switching chip 140 via corresponding communication links 148. Each of at least some of the communication links 148 operates at a data rate at least the same as the data rate at which the optical port of the network switch 224 operates. For example, in an embodiment where the optical port operates at a data rate of 200 Gbps, the communication link 148 operates at a data rate of at least 200 Gbps. Each of the at least some of the port modules 226 corresponds to a corresponding port, and each of the at least some of the communication links 148 transmits / receives data corresponding to the optical port.

[0077] As referenced above Figures 1B to 1C The switching chip 140 includes multiple SERDES 160 (…). Figure 2B (Not shown in the image), each SERDES 160 corresponds to a corresponding internal network interface 186 of the switching chip 140. According to an embodiment, each SERDES 160 is communicatively coupled to a corresponding port module 226 via a corresponding communication link 148. Figure 2B For ease of explanation, only one port module 226 is shown, but as mentioned above, the network switch 224 includes multiple port modules 226.

[0078] Port module 226 includes an optical transceiver 228 for a corresponding optical port. The optical transceiver 228 is configured to: i) receive an optical signal via a corresponding cable 228 (e.g., a 200 Gbps optical signal), ii) convert the optical signal into an electrical signal (e.g., a 200 Gbps electrical signal), and iii) provide the electrical signal to the switching chip 140 via a communication link 148. The optical transceiver 228 is also configured to: i) receive an electrical signal (e.g., a 200 Gbps electrical signal) via the communication link 148, ii) convert the electrical signal into an optical signal (e.g., a 200 Gbps optical signal), and iii) provide the optical signal to the cable 228.

[0079] In some embodiments, port module 226 includes SERDES (not shown), which is configured to interface optical transceiver 228 with communication link 148.

[0080] Each network switch 216 includes multiple switching chips 172. (See above reference.) Figure 1DEach switching chip 172 includes multiple network interfaces 194, which are communicatively coupled to corresponding uplink ports and corresponding downlink ports of the network switch 216. Each switching chip 172 also includes other components ( Figure 2B (Not shown in the diagram) This includes, for example, a memory for storing packet data, and a packet processor for analyzing at least the packet header data of packets received via the network interface to determine the network interface through which the packet will be forwarded. The network interface 194 of the switching chip 172 is configured to operate at a data rate at least as high as the data rate operated by the downlink port. In an embodiment, at least some of the network interfaces 194 of the switching chip 172 are configured to operate at a data rate of at least 100 Gbps.

[0081] In one embodiment, each network interface 194 of the switching chip 172 includes or is coupled to a SERDES 198, which is configured to operate at a data rate at least as high as the data rate of the downlink port of the network switch 216. For example, in an embodiment where the downlink port operates at a data rate of 100 Gbps, the SERDES 198 operates at a data rate of at least 100 Gbps.

[0082] Some of SERDES 198 correspond to downlink ports of switch 216 and are communicatively coupled to port module 182 corresponding to the downlink port. In an embodiment, port module 182 has a similar structure to port module 226 of network switch 224, but operates at a lower data rate corresponding to the downlink port. For example, in an embodiment, port module 182 corresponding to the downlink port operates at a data rate of at least 100 Gbps.

[0083] The network switch 216 includes multiple optical port modules 244, which correspond to the uplink ports of the network switch 216. Figure 2B For ease of explanation, only one optical port module 244 is illustrated in each network switch 216, but each network switch 216 includes multiple optical port modules 244. The optical port modules 244 are communicatively coupled to multiple switching chips 240 via corresponding communication links 178. Communication links 178 operate at a data rate at least as high as the data rate of the downlink port of the network switch 216. For example, in an embodiment where the downlink port operates at a data rate of 100 Gbps, communication link 178 operates at a data rate of at least 100 Gbps. In this embodiment, each communication link 178 is coupled to a corresponding SERDES in the corresponding switching chip 240.

[0084] Optical port module 244 includes optical transceivers 252 corresponding to the uplink ports of network switch 216. Each optical transceiver 252 is configured to: i) receive optical signals via cable 228 (e.g., 200 Gbps optical signals), ii) convert the optical signals into electrical signals (e.g., 200 Gbps electrical signals), and iii) provide the electrical signals to mux / demux 256. Each optical transceiver 244 is also configured to: i) receive electrical signals (e.g., 200 Gbps electrical signals) from mux / demux 256, ii) convert the electrical signals into optical signals (e.g., 200 Gbps optical signals), and iii) provide the optical signals to cable 228.

[0085] The mux / demux 256 is configured to: i) multiplex a first data stream pair (e.g., each 100 Gbps) received from one or both switching chips 240 into a first combined stream (e.g., at 200 Gbps) to be transmitted by the optical port, and ii) demultiplex a second combined stream (e.g., at 200 Gbps) received from the optical port into a second stream pair to be transmitted to one or both switching chips 240 (e.g., at 100 Gbps).

[0086] Each optical port module 244 also includes multiple SERDES 260, 264. SERDES 260, 264 are configured to operate at a data rate at least as high as the data rate at which the downlink ports of network switch 216 operate. For example, in an embodiment where the downlink ports operate at a data rate of 100 Gbps, SERDES 260, 264 are configured to operate at a data rate of at least 100 Gbps.

[0087] Each switching chip 172 includes a plurality of SERDES 198, each SERDES corresponding to a corresponding network interface of the switching chip 172. According to an embodiment, each SERDES 198 is communicatively coupled to an optical module 244. Figure 2B For ease of explanation, only one SERDES 198 is illustrated in each switching chip 240, but as stated above, each switching chip 172 includes multiple SERDES 198 corresponding to multiple uplinks. SERDES 198 are configured to operate at a data rate at least as high as the data rate at which the downlink ports operate. For example, in an embodiment where the downlink ports operate at a data rate of 100 Gbps, SERDES 198 are configured to operate at a data rate of at least 100 Gbps.

[0088] although Figure 2BIn the diagram, SERDES 260 and 264 are communicatively coupled to the network interface of the respective switching chip 172. However, in another embodiment, both SERDES 260 and 264 are communicatively coupled to the network interface of one of the switching chips 172.

[0089] In one embodiment, multiple switching chips 172 of the network switch 216 are mounted on a single printed circuit board (PCB). In another embodiment, at least some of the multiple switching chips 172 and optical modules 244 are mounted on a single PCB.

[0090] Now for reference Figures 2A to 2B Network 204 provides low-latency communication between GPUs 112 in a large number of GPUs (e.g., at least 200,000 GPUs 112). For example, in one embodiment, each GPU 112 in a first compute container group 108 is communicatively connected to another GPU 112 in a second compute container group 108 via a number of alternative paths, wherein each path traverses at most three network switches 216, 224.

[0091] Additional land, with Figure 1A Compared to network 104, network switching system 204 uses a significantly reduced number of cables. For example, there are 131,072 cables 228 between network switch 216 and network switch 224 (in an embodiment), while there are 262,144 cables 128 between network switch 116 and network switch 124 (in an embodiment). Additionally, compared to... Figure 1A Compared to network 104, network 204 uses significantly fewer switching devices. For example, there are 512 switching devices 216 (in an embodiment) compared to 1024 switching devices 116 (in an embodiment).

[0092] In other embodiments, optical port modules 182, 226, 244 are replaced by electrical port modules including electrical transceivers, and cable 128 is a power cable.

[0093] In another embodiment, the optical module 226 of switch 224 includes a mux / demux (not shown) configured to demultiplex a first combined stream received from switching chip 140 via communication link 148 (e.g., at 200 Gbps) into a first stream pair (each at, for example, 100 Gbps) for transmission by optical transceiver 228 via cable 228 at the corresponding optical wavelength. Additionally, the mux / demux (not shown) of optical module 226 is configured to multiplex a second stream pair (each at, for example, 100 Gbps) to generate a second combined stream for transmission to switching chip 140 via communication link 148, the second stream pair having been received by optical transceiver 228 via cable 228 at the corresponding optical wavelength. Additionally, the optical transceiver 252 of optical module 244 of switch 216 is configured to provide the first stream pair (each at, for example, 100 Gbps) received via cable 228 at the corresponding optical wavelength to SERDES 260, 264. Additionally, SERDES 260, 264 provide a second stream pair (each at, for example, 100 Gbps) to optical transceiver 252 for transmission over the corresponding optical wavelength via cable 228. In some such embodiments, mux / demux 256 is omitted.

[0094] Figure 2C According to another embodiment Figure 2A A simplified block diagram showing that communication between one network switch in network switch 224 is coupled to one network switch in network switch 216. See below for reference. Figures 2A to 2B discuss Figure 2C . Figure 2C The schematic diagram is similar to Figure 2B The diagram is shown below. For the sake of simplicity, components with the same number will not be discussed in detail.

[0095] Each network switch 216 includes a corresponding switching chip 272, which is coupled to multiple optical port modules 276, each optical port module corresponding to a corresponding uplink port of the network switch 216. Figure 2C For ease of explanation, only one optical port module 276 is illustrated, but each network switch 216 includes multiple optical port modules 276. The optical port module 276 is communicatively coupled to the switching chip 272 via a communication link 278. The communication link 278 operates at a data rate at least as high as the data rate of the uplink port of the network switch 216. For example, in an embodiment where the uplink port operates at a data rate of 200 Gbps, the communication link 278 operates at a data rate of at least 200 Gbps.

[0096] Each switching chip 272 includes multiple internal network interfaces ( Figure 2C(Not shown in the image), multiple internal network interfaces are communicatively coupled to corresponding uplink and downlink port pairs of network switch 216. Each switch chip 272 (see below) Figure 2D (Discussed in more detail) also includes other components (not shown), such as a memory for storing packet data, a packet processor for analyzing at least the packet header data of packets received via the internal network interface to determine the internal network interface through which the packet will be forwarded, etc.

[0097] The switching chip 272 is also coupled to multiple optical port modules 280, which correspond to the respective downlink ports of the network switch 216. Some of the network interfaces (not shown) of the switching chip 272 are communicatively coupled to the optical port modules 280.

[0098] Figure 2C For ease of explanation, only one optical port module 280 is illustrated, but each network switch 216 includes multiple optical port modules 280. The optical port module 280 is communicatively coupled to the switching chip 272 via a communication link 282. The communication link 282 operates at a data rate at least twice the data rate of the downlink port of the network switch 216. For example, in an embodiment where the downlink port operates at a data rate of 100 Gbps, the communication link 282 operates at a data rate of at least 200 Gbps.

[0099] Each optical port module 280 includes a mux / demux 284 configured to: i) multiplex data received via optical port pairs (e.g., each at 100 Gbps) into a combined receive stream for transmission to switching chip 272 via communication link 282 (e.g., at 200 Gbps); and ii) demultiplex the combined transport stream received from switching chip 272 via communication link 282 (e.g., at 200 Gbps) into a transport stream pair for transmission by optical port pairs (e.g., at 100 Gbps).

[0100] The optical port module 280 also includes SERDES 286 coupled to the communication link 282. SERDES 286 is configured to operate at a data rate at least twice the data rate of the downlink port. For example, in an embodiment where the downlink port operates at a data rate of 100 Gbps, SERDES 286 is configured to operate at a data rate of at least 200 Gbps. SERDES 286 is coupled to a mux / demux 284.

[0101] The optical port module 280 also includes a corresponding optical transceiver 288 for the respective downlink port. Each optical transceiver 288 is configured to: i) receive an optical signal via a corresponding cable 120 (e.g., a 100Gbps optical signal), ii) convert the optical signal into an electrical signal (e.g., a 100Gbps electrical signal), and iii) provide the electrical signal to the mux / demux 284. Each optical transceiver 288 is also configured to: i) receive an electrical signal (e.g., a 100Gbps electrical signal) from the mux / demux 284, ii) convert the electrical signal into an optical signal (e.g., a 100Gbps optical signal), and iii) provide the optical signal to the corresponding cable 120.

[0102] Figure 2D According to the embodiments Figure 2A and 2C A simplified schematic diagram of the switching chip 272 of the network switch 216. The switching chip 272 includes multiple internal network interfaces 290. Figure 2D In one example embodiment, the switch chip 272 includes 1024 internal network interfaces 290. In another embodiment, the switch chip 272 includes at least as many internal network interfaces 290 as the sum of: a) twice the number of network switches 224, and b) the number of GPUs 112 in the two container groups 108. In another embodiment, the switch chip 272 includes at least 1000 internal network interfaces 290. In yet another embodiment, the switch chip 272 includes at least 950 internal network interfaces 290. In yet another embodiment, the switch chip 272 includes at least 900 internal network interfaces 290.

[0103] Each of at least some of the internal network interfaces 290 is configured to operate at a data rate at least as high as the data rate at which the uplink port of network switch 216 operates. For example, in an embodiment where the uplink port operates at a data rate of 200 Gbps, at least some of the network interfaces 290 are configured to operate at a data rate of at least 200 Gbps.

[0104] The switching chip 272 also includes a packet processor 292 coupled to a plurality of internal network interfaces 290. The packet processor 292 is configured to forward packets between the internal network interfaces 290. For example, the packet processor 292 is configured to analyze at least the packet header data of packets received via the internal network interfaces 290 to determine the internal network interfaces 290 through which the packets will be forwarded.

[0105] According to an embodiment, the packet processor 292 has appropriate processing capabilities to operate in an environment where the network interface 290 is communicatively coupled to: i) a large number of switches 224 via a link of at least 200 Gbps, and ii) a large number of GPUs 112 via a link of at least 100 Gbps.

[0106] The switching chip 272 includes other suitable components (not shown), such as a memory for storing packet data, memory management circuitry, etc.

[0107] To significantly reduce the number of external connections required by the switching chip 272, the switching chip 272 includes a SERDES 294 for each of at least a set 290 of multiple internal network interfaces, wherein one SERDES 294 transmits / receives packet data corresponding to the set 290 of the multiple internal network interfaces. For example, in an embodiment, transmitted packet data corresponding to the set 290 of the multiple internal network interfaces is multiplexed within a combined transmit signal, and received packet data corresponding to the set 290 of the multiple internal network interfaces is multiplexed within a combined receive signal. Figure 2A and 2C In the example, each network switch 216 includes 768 ports, and the switching chip 272 includes 512 SERDES 294, where each SERDES 294 corresponds to a corresponding internal network interface pair 290.

[0108] Now for reference Figures 2C to 2D Each SERDES 294-1 to 294-256 corresponds to a corresponding uplink port and is coupled to a corresponding port module 276. Each SERDES 292-257 to 294-512 corresponds to a corresponding downlink port pair and is coupled to a corresponding port module 280. In an embodiment, each SERDES 294 is configured to operate at a data rate at least as high as the data rate of the uplink port of the network switch 216. For example, in an embodiment where the uplink port operates at a data rate of 200 Gbps, SERDES 294 operates at a data rate of at least 200 Gbps.

[0109] The switching chip 272 also includes multiple multiplexers / demultiplexers (mux / demux) 296, which are coupled to multiple SERDES 294. Each mux / demux 296 is configured to: i) multiplex data received via internal network interface pair 290 (e.g., each at 100 Gbps) into a combined transport stream for transmission via communication links 180, 282 (e.g., at 200 Gbps) to the corresponding port modules 276, 280; and ii) demultiplex the combined receive stream received from port modules 276, 280 via communication links 180, 282 (e.g., at 200 Gbps) into a receive stream pair for transmission to internal network interface pair 290 (e.g., at 100 Gbps).

[0110] In this embodiment, each SERDES 294 and mux / demux 296 corresponds to an external network interface of the IC switching chip 272. According to this embodiment, each SERDES 294 is coupled to an external interconnect (e.g., solder balls, pins, etc.) of the IC package that encapsulates the switching chip 272.

[0111] Now for reference Figure 2A and 2C Network 204 provides low-latency communication between GPUs 112 in a large number of GPUs (e.g., at least 200,000 GPUs 112). For example, in one embodiment, each GPU 112 in a first compute container group 108 is communicatively connected to another GPU 112 in a second compute container group 108 via a number of alternative paths, where each path traverses at most three network switches 216, 224.

[0112] Additional land, with Figure 1A Compared to network 104, network 204 uses significantly fewer cables. For example, compared to 262,144 cables 128 between network switches 116 and 124 (in this embodiment), there are 131,072 cables 228 between network switches 216 and 224 (in this embodiment). Additionally, compared to... Figure 1A Compared to network 104, network 204 uses significantly fewer switching devices. For example, it has 512 switching devices 216 (in an embodiment) compared to 1024 switching devices 116 (in an embodiment).

[0113] Additionally, Figure 2C Each network switch 216 includes only one switching chip 272 (with Figure 2B Compared to multiple switching chips 240), therefore, using Figure 2C Network and usage of network switch 216 Figure 2BCompared to the number of switching chips 240 used in the network switch 216, it has significantly fewer switching chips 272.

[0114] In other embodiments, optical port modules 276, 280 are replaced by electrical port modules including electrical transceivers, and cable 232 is a power cable.

[0115] Figure 3 This is a flowchart of an example method 300 for communication in a communication network according to an embodiment, the communication network including i) a plurality of first network switches, and ii) a plurality of second network switches to which communication is coupled. According to the embodiment, method 300 in… Figure 1A The example communication network 100 is implemented therein, and for ease of explanation, refer to [reference needed]. Figures 1A to 1B Method 300 is described. According to another embodiment, method 300 is in Figure 2A The example communication network 200 is implemented, and for ease of explanation, also refer to Figures 2A to 2D Description method 300.

[0116] In other embodiments, method 300 is in conjunction with Figure 1A and 2A The example communication networks 100 and 200 are implemented in another suitable communication network, different from these. Similarly, in some embodiments, Figure 1A Communication network 100 and / or Figure 2A The communication network 200 implements another suitable method that differs from method 300.

[0117] In one embodiment, method 300 is implemented in a communication network, wherein each first network switch (e.g., each network switch 116, each network switch 216, etc.) has at least 200 downlink ports, and each second network switch (e.g., each network switch 124, each network switch 224, etc.) includes a corresponding plurality of ports, the corresponding plurality of ports being coupled to at least one uplink port of each of the first switches. In another embodiment, method 300 is implemented in a communication network, wherein, additionally or alternatively, the plurality of first network switches includes at least a first 1000 first network switches.

[0118] In one embodiment, method 300 is implemented in a communication network, wherein each second network switch includes at least 1000 ports communicatively coupled to a corresponding first network switch, and optionally each port is configured to operate at a data rate of at least 100 Gbps. In another embodiment, method 300 is implemented in a communication network, wherein each second network switch includes at least 500 ports communicatively coupled to a corresponding first network switch, and optionally each port is configured to operate at a data rate of at least 200 Gbps.

[0119] In block 304, each of the plurality of first network switches receives packets from multiple network devices via multiple downlink ports. For example, each network switch 116 receives packets via multiple downlink ports. As another example, each network switch 216 receives packets via multiple downlink ports. In an embodiment, receiving packets in block 304 includes receiving packets via at least 200 downlink ports of the first network switches.

[0120] In one embodiment, each first network switch in block 304 receives packets from multiple GPUs (e.g., GPU 112). In another embodiment, each first network switch in block 304 additionally or alternatively receives packets from other suitable network devices (such as memory devices, servers, other network switches, etc.).

[0121] In one embodiment, receiving packets in block 304 includes a first network switch receiving data at a data rate of at least 100 Gbps on each downlink port.

[0122] In one embodiment, each first network switch is communicatively coupled to a corresponding compute container group within a plurality of compute container groups (e.g., compute container group 108), each compute container group including multiple network devices (e.g., GPUs, memory devices, servers, etc.), and each downlink port of the first network switch is communicatively coupled to a corresponding network device within the compute container group. In another embodiment, each first network switch is communicatively coupled to a corresponding pair of compute container groups within a plurality of compute container groups (e.g., compute container group 108), each compute container group including multiple network devices (e.g., GPUs, memory devices, servers, etc.), and each downlink port of the first network switch is communicatively coupled to a corresponding network device within the pair of compute container groups.

[0123] In block 308, each first network switch forwards packets received in block 304 to multiple second network switches via corresponding multiple communication links between the first network switch and each of the multiple second switches. For example, each network switch 116 forwards packets via multiple uplink ports. As another example, each network switch 216 forwards packets via multiple uplink ports. In this embodiment, packet forwarding in block 308 includes forwarding packets via at least 200 uplink ports of the first network switch.

[0124] In one embodiment, packet forwarding in block 308 includes a first network switch forwarding data at a data rate of at least 100 Gbps on each uplink port. In another embodiment, packet forwarding in block 308 includes a first network switch forwarding data at a data rate of at least 200 Gbps on each uplink port.

[0125] In block 312, each second network switch forwards packets received from the first network switch in connection with block 308 to the first network switch via corresponding multiple communication links between the second network switch and each first network switch. For example, each network switch 124 forwards packets via multiple ports of network switch 124. As another example, each network switch 224 forwards packets via multiple ports of network switch 224.

[0126] In one embodiment, packet forwarding in block 312 includes packet forwarding via at least 1000 ports of a second network switch. In another embodiment, packet forwarding in block 312 includes packet forwarding via at least 500 ports of a second network switch.

[0127] In one embodiment, packet forwarding in block 312 includes data forwarded by the second network switch at a data rate of at least 100 Gbps by each port of the second network switch. In another embodiment, packet forwarding in block 312 includes data forwarded by the second network switch at a data rate of at least 200 Gbps by each port of the second network switch.

[0128] In box 316, each first network switch transmits packets received from multiple second network switches in connection with box 312 to at least 200 network devices via multiple downlink ports.

[0129] For example, each network switch 116 transmits packets via multiple downlink ports. As another example, each network switch 216 transmits packets via multiple downlink ports. In this embodiment, packet transmission in block 316 includes packet transmission via at least 200 downlink ports of the first network switch.

[0130] In one embodiment, each first network switch transmits packets in block 316 to multiple GPUs (e.g., GPU 112). In another embodiment, each first network switch in block 316 additionally or alternatively transmits packets to other suitable network devices (such as memory devices, servers, other network switches, etc.).

[0131] In one embodiment, the packet transmission in block 316 includes a first network switch transmitting data at a data rate of at least 100 Gbps on each downlink port.

[0132] Figure 4 This is a simplified block diagram of an example multiplexer circuit device 400, which is included in a communication network such as described above, and / or other suitable communication networks. For example, in an embodiment, the multiplexer circuit device 400 is included in... Figures 1A to 1B mux / demux 152 and / or Figure 1C In mux / demux 190. In other embodiments, Figures 1A to 1B mux / demux 152 and / or Figure 1C The mux / demux 190 includes other suitable multiplexer circuit devices that are different from the multiplexer circuit device 400.

[0133] As another example, in another embodiment, the multiplexer circuit device 400 is included Figures 2A to 2B In mux / demux 232, 256. As another example, in another embodiment, multiplexer circuitry 400 is included. Figure 2C mux / demux 284 and / or Figure 2D In mux / demux 296. In other embodiments, Figures 2A to 2B mux / demux 232, 256 Figure 2C mux / demux 284, and / or Figure 2D The mux / demux 296 includes other suitable multiplexer circuit devices that are different from the multiplexer circuit device 400.

[0134] The multiplexer circuit arrangement 400 includes a Physical Coding Sublayer (PCS) framing circuit arrangement 404, which is configured to receive first data corresponding to a first data stream at a first data rate (e.g., 100 Gbps) and generate a first PCS frame using the first data. In an embodiment, generating a first PCS frame includes adding a frame marker (FM) to each first PCS frame to indicate the boundary between adjacent first PCS frames.

[0135] The multiplexer circuitry 400 also includes a PCS framing circuitry 408, which is configured to receive second data corresponding to the second data stream at a first data rate (e.g., 100 Gbps) and use the second data to generate second PCS frames. In an embodiment, generating a second PCS frame includes adding an FM to each second PCS frame to represent the boundary between adjacent second PCS frames.

[0136] Frame mark modification circuitry 412 is coupled to PCS framing circuitry 404. Frame mark modification circuitry 412 is configured to modify the FM added by PCS framing circuitry 404 to distinguish PCS frames corresponding to a first data stream from PCS frames corresponding to a second data stream. For example, in one embodiment, frame mark modification circuitry 412 is configured to invert the bits of the FM added by PCS framing circuitry 404. As another example, in another embodiment, frame mark modification circuitry 412 is configured to invert a subset of the bits of the FM added by PCS framing circuitry 404. As another example, in another embodiment, frame mark modification circuitry 412 is configured to modify the bits of the FM added by PCS framing circuitry 404 in another suitable manner.

[0137] In other embodiments, the frame mark modification circuit device 412 is configured to replace the FM added by the PCS framing circuit device 404 with another FM, which is different from the FM added by the PCS framing circuit device 404, in order to distinguish PCS frames corresponding to the first data stream from PCS frames corresponding to the second data stream.

[0138] The frame mark added by the PCS framing circuit device 404 is sometimes referred to herein as the “unmodified frame mark”, and the frame mark output by the frame mark modification circuit device 412 is sometimes referred to herein as the “modified frame mark”.

[0139] In another embodiment, the frame mark modification circuit 412 is omitted, and the PCS framing circuit 404 is configured to add a modified frame mark to each first PCS frame, the modified frame mark being different from the FM added to the second PCS frame by the PCS framing circuit 408.

[0140] The multiplexer circuit arrangement 400 also includes a multiplexer 420. The output of the frame mark modification circuit arrangement 412 is coupled to a first input of the multiplexer 420, and the output of the PCS framing circuit arrangement 408 is coupled to a second input of the multiplexer 420. In embodiments where the frame mark modification circuit arrangement 412 is omitted, the output of the PCS framing circuit arrangement 404 is coupled to the first input of the multiplexer 420.

[0141] In an embodiment, multiplexer 420 is configured to generate an output at a data rate at least twice the input data rate of multiplexer 420 by alternating between: i) providing symbols (e.g., a set of bits) from a first input to the output of the multiplexer, and ii) providing symbols from a second input to the output of the multiplexer.

[0142] In an embodiment where the first and second data streams are each received at 100 Gbps, the multiplexer 420 generates an output data stream at a data rate of 200 Gbps.

[0143] Figure 5 This is a simplified block diagram of an example demultiplexer circuit device 500, which is included in a communication network such as described above, and / or other suitable communication networks. For example, in an embodiment, the demultiplexer circuit device 500 is included in... Figures 1A to 1B mux / demux 152 and / or Figure 1C In mux / demux 190. In other embodiments, Figures 1A to 1B mux / demux 152 and / or Figure 1C The mux / demux 190 includes other suitable demultiplexer circuitry that differs from the demultiplexer circuitry 500.

[0144] As another example, in another embodiment, the demultiplexer circuit device 500 is included. Figures 2A to 2B In mux / demux 232, 256. As another example, in another embodiment, demultiplexer circuitry 500 is included. Figure 2C mux / demux 284 and / or Figure 2D In mux / demux 296. In other embodiments, Figures 2A to 2B mux / demux232, mux / demux256, Figure 2C mux / demux 284, and / or Figure 2D The mux / demux 296 includes other suitable demultiplexer circuitry that differs from the demultiplexer circuitry 500.

[0145] Demultiplexer circuitry 500 includes a demultiplexer 504. The inputs of demultiplexer 504 are configured to receive an input data stream at a first data rate. In an embodiment, demultiplexer 504 is configured to generate two outputs, each at a second data rate half the first data rate, by alternating between: i) providing a symbol (e.g., a set of bits) from the input to a first output of demultiplexer 504, and ii) providing subsequent symbols from the input to a second output of demultiplexer 504.

[0146] In an embodiment where the input data stream is received at a data rate of 200 Gbps, the demultiplexer 504 is configured to generate a first output data stream and a second output data stream, each at 100 Gbps.

[0147] The first output of demultiplexer 504 is coupled to the input of PCS framing circuit device 508, and the second output of demultiplexer 504 is coupled to the input of PCS framing circuit device 512. PCS framing circuit device 508 is configured to: i) identify frame markers in the first output of demultiplexer 504, and ii) output symbols aligned with the frame markers. Additionally, PCS framing circuit device 508 is configured to output an indication: whether the identified frame marker is an unmodified frame marker or a modified frame marker.

[0148] PCS framing circuit device 512 is configured to: i) identify frame markers in the second output of demultiplexer 504, and ii) output symbols aligned with the frame markers. Additionally, PCS framing circuit device 512 is configured to output an indication of whether the identified frame marker is an unmodified frame marker or a modified frame marker.

[0149] Symbols output from PCS framing circuit device 508 are provided to the first input of selector circuit 516 and the first input of selector circuit 520. Symbols output from PCS framing circuit device 512 are provided to the second input of selector circuit 516 and the second input of selector circuit 520.

[0150] Control circuit device 524 receives an instruction from PCS framing circuit device 508: whether the frame marker identified in the first output of demultiplexer 504 is an unmodified frame marker or a modified frame marker. Additionally, control circuit device 524 receives an instruction from PCS framing circuit device 512: whether the frame marker identified in the second output of demultiplexer 504 is an unmodified frame marker or a modified frame marker. Control circuit device 524 uses the instructions received from PCS framing circuit device 508 and PCS framing circuit device 512 to generate control signals for controlling selector circuit 516 and selector circuit 520. Specifically, when the indications received from the PCS framing circuit device 508 and the PCS framing circuit device 512 indicate that: i) the frame mark in the first output of the demultiplexer 504 is a modified frame mark, and ii) the frame mark in the second output of the demultiplexer 504 is an unmodified frame mark, the control circuit device 524 generates a control signal that causes: i) the selector circuit 516 to select the output of the PCS framing circuit device 508, and ii) the selector circuit 520 to select the output of the PCS framing circuit device 512. On the other hand, when the indications received from the PCS framing circuit device 508 and the PCS framing circuit device 512 indicate that: i) the frame mark in the first output of the demultiplexer 504 is an unmodified frame mark, and ii) the frame mark in the second output of the demultiplexer 504 is a modified frame mark, the control circuit device 524 generates a control signal that causes: i) the selector circuit 516 to select the output of the PCS framing circuit device 512, and ii) the selector circuit 520 to select the output of the PCS framing circuit device 508.

[0151] The demultiplexer circuit arrangement 500 also includes a frame mark modification circuit arrangement 532, which is coupled to the output of the selector circuit 516. The frame mark modification circuit arrangement 532 is configured to convert the modified FM in the output of the selector circuit 516 to the unmodified FM.

[0152] Refer again Figure 2A , Figure 2C and Figure 4According to one embodiment, optical transceiver 288-1 is communicatively connected to a first GPU 112, and optical transceiver 288-2 is communicatively connected to a second GPU 112; and multiplexer 420 multiplexes a first stream from the first GPU 112 and a second stream from the second GPU 112. In this embodiment, a first transmission clock of a first network interface (e.g., a first Ethernet interface) of the first GPU 112 is synchronized with a second transmission clock of a second network interface (e.g., a second Ethernet interface) of the second GPU 112. In this embodiment, the first network interface (e.g., the first Ethernet interface) of the first GPU 112 and / or the second network interface (e.g., the second Ethernet interface) of the second GPU 112 includes circuitry configured to synchronize the first transmission clock of the first network interface with the second transmission clock of the second network interface.

[0153] In corresponding Figure 2A and Figure 2C In another embodiment, the first transmission clock of the first network interface (e.g., the first Ethernet interface) of the first GPU 112 is not synchronized with the second transmission clock of the second network interface (e.g., the second Ethernet interface) of the second GPU 112. In some such embodiments, Figure 2C The multiplexer circuitry of the mux / demux 284 includes a circuitry configured to synchronize the first stream from the first GPU 112 with the second stream from the second GPU 112 before multiplexing the first and second streams.

[0154] Figure 6 This is a simplified block diagram of an example multiplexer circuit device 600 according to another embodiment, which is included in a communication network such as described above, and / or other suitable communication networks. For example, in an embodiment, the multiplexer circuit device 600 is included in... Figure 2C In mux / demux 284. In other embodiments, Figure 2C The mux / demux284 includes other suitable multiplexer circuit devices that are different from the multiplexer circuit device 600.

[0155] Figure 6 The multiplexer circuit device 600 is similar to Figure 4 For the sake of brevity, components with the same number are not described in detail in the multiplexer circuit device 400.

[0156] exist Figure 6In one embodiment, the outputs of PCS framing circuitry 404 and PCS framing circuitry 408 are asynchronous. For example, a first transmission clock corresponding to a first 100 Gbps stream is asynchronous with respect to a second transmission clock corresponding to a second 100 Gbps stream. Synchronization circuitry 604 is coupled to PCS framing circuitry 404 and PCS framing circuitry 408 and processes their outputs to generate a first processed stream corresponding to the first 100 Gbps stream, the first processed stream being synchronized with a second processed stream corresponding to the second 100 Gbps stream. In another embodiment, synchronization circuitry 604 includes circuitry configured to generate the first processed stream and the second processed stream, at least as part of generating the synchronized first processed stream and the second processed stream, at least by modifying the length of the inter-frame gap (IFG) in the outputs of PCS framing circuitry 404 and / or PCS framing circuitry 408. In an embodiment, the IFG in the output of PCS framing circuit device 404 and the output of PCS framing circuit device 408 includes free symbols, and modifying the length of the IFG includes removing one or more free symbols or adding one or more free symbols.

[0157] In another embodiment, the synchronization circuit device 604 additionally or alternatively includes one or more memory devices (e.g., one or more registers, one or more latches, etc.) for delaying the output of the PCS framing circuit device 404 and / or the output of the PCS framing circuit device 408, at least as part of the streams for generating the first and second processes of synchronization.

[0158] The multiplexer circuit arrangement 600 further includes a PCS framing circuit arrangement 612 coupled to the synchronization circuit arrangement 604, and a PCS framing circuit arrangement 616 coupled to the synchronization circuit arrangement 604. The PCS framing circuit arrangement 612 is configured to receive first data corresponding to a data stream of a first process, and to generate a first PCS frame using the first data. The PCS framing circuit arrangement 616 is configured to receive second data corresponding to a data stream of a second process, and to generate a second PCS frame using the second data.

[0159] The output of PCS framing circuit device 612 is coupled to the input of FM modification circuit device 412, and the output of PCS framing circuit device 612 is coupled to one of the inputs of multiplexer 420.

[0160] In another embodiment, PCS framing circuit device 612 and PCS framing circuit device 616 are omitted. For example, one output of synchronization circuit device 604 is coupled to an input of FM modification circuit device 412, and the output of PCS framing circuit device 612 is coupled to one input of multiplexer 420.

[0161] In another embodiment, PCS framing circuit device 404 and PCS framing circuit device 408 are omitted.

[0162] In some embodiments, similar to Figure 4 Multiplexer circuit device 400 Figure 6 The multiplexer circuit device such as the multiplexer circuit device 600 includes a circuit device that interleaves the symbols of the forward error correction (FEC) codeword in the output of the multiplexer 420, such that adjacent symbols in the output come from different FEC codewords.

[0163] Figure 7A This is a simplified block diagram of an example multiplexer circuit device 700 according to another embodiment, which is included in a communication network such as described above, and / or other suitable communication networks. For example, in an embodiment, the multiplexer circuit device 700 is included in... Figures 1A to 1B mux / demux 152 and / or Figure 1C In mux / demux 190. In other embodiments, Figures 1A to 1B mux / demux 152 and / or Figure 1C The mux / demux 190 includes other suitable multiplexer circuit devices that are different from the multiplexer circuit device 700.

[0164] As another example, in another embodiment, the multiplexer circuit device 700 is included Figures 2A to 2B In mux / demux 232, 256. As another example, in another embodiment, multiplexer circuitry 700 is included. Figure 2C mux / demux 284 and / or Figure 2D In mux / demux 296. In other embodiments, Figures 2A to 2B mux / demux232, mux / demux256, Figure 2C mux / demux 284, and / or Figure 2D The mux / demux 296 includes other suitable multiplexer circuit devices that are different from the multiplexer circuit device 700.

[0165] Figure 7A The multiplexer circuit device 700 is similar to Figure 4For the sake of brevity, components with the same number are not described in detail in the multiplexer circuit device 400.

[0166] The codeword interleaver circuit 704 is coupled to the PCS framing circuit 404 and processes the output of the PCS framing circuit 404 to generate a first interleaved stream corresponding to the first 100 Gbps stream. For example, the codeword interleaver circuit 704 is configured to interleave symbols of a plurality of FEC codewords received from the PCS framing circuit 404.

[0167] The codeword interleaver circuit 708 is coupled to the PCS framing circuit 408 and processes the output of the PCS framing circuit 408 to generate a second interleaved stream corresponding to the second 100 Gbps stream. For example, the codeword interleaver circuit 708 is configured to interleave symbols of a plurality of FEC codewords received from the PCS framing circuit 408.

[0168] Figure 7B This is a simplified schematic diagram illustrating an example interleaving function performed by a codeword interleaver circuit device 704 and a codeword deinterleaver circuit device 708 according to an embodiment.

[0169] Figure 8A This is a simplified block diagram of an example demultiplexer circuit device 800 according to another embodiment, which is included in a communication network such as described above, and / or other suitable communication networks. For example, in an embodiment, the demultiplexer circuit device 800 is included in... Figures 1A to 1B mux / demux 152 and / or Figure 1C In mux / demux 190. In other embodiments, Figures 1A to 1B mux / demux 152 and / or Figure 1C The mux / demux 190 includes other suitable multiplexer circuit devices that are different from the multiplexer circuit device 400.

[0170] As another example, in another embodiment, the demultiplexer circuit device 800 is included. Figures 2A to 2B In the mux / demux 232. As another example, in another embodiment, the demultiplexer circuitry 800 is included. Figure 2C mux / demux 284 and / or Figure 2D In mux / demux 296. In other embodiments, Figures 2A to 2B mux / demux 232, Figure 2C mux / demux 284, and / or Figure 2D The mux / demux 296 includes other suitable demultiplexer circuitry that differs from the demultiplexer circuitry 800.

[0171] Figure 8A The demultiplexer circuit device 800 is similar to Figure 5 For the sake of brevity, the components with the same number are not described in detail in the demultiplexer circuit device 500.

[0172] Demultiplexer 504 is configured to receive a combined stream alternating from multiple different FEC code characters and demultiplex the combined stream into a first interleaved stream and a second interleaved stream. The first interleaved stream output by demultiplexer 504 includes interleaved FEC codewords corresponding to one of the following: i) a first 100 Gbps stream and ii) a second 100 Gbps stream; and the second interleaved stream output by demultiplexer 504 includes interleaved FEC codewords corresponding to another of the following: i) a first 100 Gbps stream and ii) a second 100 Gbps stream.

[0173] Codeword deinterleaver circuitry 804 is coupled to a first output of demultiplexer 504 and processes a first interleaved stream to generate a first deinterleaved stream, the first deinterleaved stream corresponding to one of: i) a first 100 Gbps stream and ii) a second 100 Gbps stream. For example, codeword deinterleaver circuitry 804 is configured to deinterleave symbols of a plurality of FEC codewords received from demultiplexer 504.

[0174] The codeword deinterleaver circuit 808 is coupled to the second output of the demultiplexer 504 and processes the second interleaving stream to generate a first deinterleaving stream, the first deinterleaving stream corresponding to one of the following: i) a first 100 Gbps stream, and ii) a second 100 Gbps stream. For example, the codeword deinterleaver circuit 808 is configured to deinterleave symbols of a plurality of FEC codewords received from the demultiplexer 504.

[0175] The output of the deinterleaver circuit device 804 is coupled to the input of the PCS framing circuit device 508; and the output of the deinterleaver circuit device 808 is coupled to the input of the PCS framing circuit device 512.

[0176] Figure 8B This is a simplified schematic diagram illustrating an example deinterleaving function performed by codeword deinterleaving circuit device 804 and codeword deinterleaving circuit device 808 according to an embodiment.

[0177] Figure 8B The illustration shows a stream being transmitted within a demultiplexer circuit arrangement 800 via multiple channels (e.g., four). In other embodiments, the stream is transmitted via another suitable number of channels, different from the four channels.

[0178] Demultiplexer 504 receives a combined stream of alternating FEC code characters from multiple different sources and alternately outputs a set of symbols at alternating outputs corresponding to the respective channels. For example, according to one embodiment, demultiplexer 504 outputs two symbols to a first output; then two symbols to a fifth output; then two symbols to a second output; then two symbols to a sixth output; then two symbols to a third output; then two symbols to a seventh output; then two symbols to a fourth output; then two symbols to an eighth output; and so on.

[0179] The first to fourth outputs of demultiplexer 504 correspond to the first interleaved stream, and the fifth to eighth outputs correspond to the second interleaved stream. The operation of demultiplexer 504 in the manner described above results in the first interleaved stream comprising interleaved FEC code characters from the first and second FEC codewords, and the second interleaved stream comprising interleaved FEC code characters from the third and fourth FEC codewords. Additionally, the operation of demultiplexer 504 in the manner described above results in the first interleaved stream corresponding to one of the following: i) a first 100 Gbps stream, and ii) a second 100 Gbps stream, and the second interleaved stream corresponding to the other one of the following: i) a first 100 Gbps stream, and ii) a second 100 Gbps stream.

[0180] exist Figure 8B In the example, symbols from the first FEC codeword and the second FEC codeword alternate on each channel between the demultiplexer 504 and the codeword deinterleaver circuit device 804, and symbols from the third FEC codeword and the fourth FEC codeword alternate on each channel between the demultiplexer 504 and the codeword deinterleaver circuit device 808.

[0181] exist Figure 8B In the example, codeword deinterleaver circuit 804 deinterleaves the first to fourth channels output by demultiplexer 504 to output the entire first FEC codeword before outputting the second FEC codeword. Similarly, codeword deinterleaver circuit 808 deinterleaves the fifth to eighth channels output by demultiplexer 504 to output the entire third FEC codeword before outputting the fourth FEC codeword.

[0182] The first channel output by the codeword deinterleaver circuit device 804 / 808 carries the first, fifth, and ninth symbols of the FEC codeword; the second channel carries the second, sixth, and tenth symbols of the FEC codeword 708; the third channel carries the third, seventh, and eleventh symbols of the FEC codeword; and the fourth channel carries the fourth, ninth, and twelfth symbols of the FEC codeword.

[0183] Refer again Figures 2A to 2CIn other embodiments, more than two streams (e.g., 100 Gbps streams) are transmitted between switches 216 and 224 via a single cable to further reduce the number of cables. As an illustrative example, in one embodiment, four 100 Gbps streams are transmitted between switches 216 and 224 via a single cable suitable for 400 Gbps.

[0184] For example, refer to Figure 2B According to an embodiment, port module 226 is modified to be coupled to two 200 Gbps streams to / from switch chip 140 and to transmit / receive 400 Gbps via 400 G cables; port module 244 is modified to transmit / receive 400 Gbps via 400 G cables and to be coupled to four 100 Gbps streams to / from four switch chips 172. (See reference...) Figure 2C In another embodiment, port module 276 is modified to transmit / receive 400 Gbps via a 400 G cable and coupled to two 200 Gbps streams going to / from two switching chips 272.

[0185] refer to Figure 1C and 2B According to an embodiment, switching chip 140 is modified such that each mux / demux 190 multiplexes four 100 Gbps streams into a 400 Gbps stream and demultiplexes the 400 Gbps streams into four 100 Gbps streams; SERDES 160 operates at a data rate of at least 400 Gbps; port module 226 is modified to couple to / from the 400 Gbps streams of switching chip 140 and to transmit / receive 400 Gbps via 400 G cables; port module 244 is modified to transmit / receive 400 Gbps via 400 G cables and to couple to / from the four switching chips 172. Reference Figure 2CIn another embodiment, switch chip 272 is modified such that each mux / demux 296 multiplexes four 100 Gbps streams into a 400 Gbps stream and demultiplexes the 400 Gbps streams into four 100 Gbps streams; SERDES 294 operates at a data rate of at least 400 Gbps; port module 276 is modified to transmit / receive 400 Gbps via 400 G cables and coupled to 400 Gbps streams going to / from both switch chips 272; port module 280 is modified such that each mux / demux 284 multiplexes four 100 Gbps streams into a 400 Gbps stream and demultiplexes the 400 Gbps streams into four 100 Gbps streams; SERDES 286 operates at a data rate of at least 400 Gbps.

[0186] Example 1: A communication network comprising: a plurality of first switches, each first switch having: i) a corresponding first integrated circuit (IC) switching chip having a plurality of network interfaces, ii) a corresponding plurality of downlink ports, and iii) a corresponding plurality of uplink ports; and a plurality of second switches, each second switch having: i) a corresponding plurality of ports coupled to at least one uplink port of each of the first switches, ii) a corresponding second IC switching chip in a corresponding IC package having a plurality of external network interfaces coupled to external interconnects of the IC package, and iii) a corresponding plurality of serializers / deserializers (SERDES) for communicatively coupling the corresponding plurality of external network interfaces to the corresponding plurality of ports of the second switches. Each second IC switching chip further includes: multiple internal network interfaces, a packet processor coupled to the multiple internal network interfaces and configured to forward packets between the internal network interfaces, and multiple multiplexer / demultiplexer circuit devices, each multiplexer / demultiplexer circuit device being coupled to: i) a corresponding external network interface, and ii) a corresponding set of the multiple internal network interfaces, each multiplexer / demultiplexer circuit device being configured to: i) demultiplex a first data stream received from the corresponding external network interface into a second data stream and a third data stream for transmission to the corresponding set of the multiple internal network interfaces, and ii) multiplex a fourth data stream and a fifth data stream received via the corresponding set of the multiple internal network interfaces into a sixth data stream for transmission to the corresponding external network interface.

[0187] Example 2: According to the communication network of Example 1, wherein the plurality of multiplexer / demultiplexer circuit devices are a plurality of first multiplexer / demultiplexer circuit devices, and wherein each second switch further includes: a corresponding plurality of second multiplexer / demultiplexer circuit devices coupled to a corresponding plurality of SERDES, each second multiplexer / demultiplexer circuit device being configured to: i) demultiplex a sixth data stream received via SERDES from a corresponding external network interface of the second IC switching chip into an eighth and a ninth data stream for transmission via a corresponding port pair of the second switch, and ii) multiplex a tenth and an eleventh data stream received via a corresponding port pair of the second switch into a first data stream for transmission via SERDES to a corresponding external network interface of the second IC switching chip.

[0188] Example 3: The communication network according to Example 2, wherein: each second IC switching chip includes at least 1000 internal network interfaces; each second IC switching chip includes at least 500 external network interfaces; and each second switch includes at least 1000 ports, the at least 1000 ports being communicatively coupled to at least 500 external network interfaces of the corresponding second IC switching chip.

[0189] Example 4: A communication network according to any one of Examples 1 to 3, wherein the plurality of SERDES are a plurality of first SERDES, and wherein each external network interface of the second IC switching chip includes or is coupled to: a corresponding second SERDES, the corresponding second SERDES being coupled to a corresponding multiplexer / demultiplexer circuit device, the corresponding second SERDES being configured to: i) transmit a first data stream from the external network interface to the multiplexer / demultiplexer circuit device, and ii) transmit a sixth data stream from the multiplexer / demultiplexer circuit device to the external network interface.

[0190] Example 5: A communication network according to any one of Examples 1 to 4, wherein each multiplexer / demultiplexer circuit device includes: a forward error correction (FEC) codeword interleaver circuit device configured to: i) interleave symbols of a plurality of FEC codewords in a fourth data stream, and ii) interleave symbols of a plurality of FEC codewords in a fifth data stream; and an FEC codeword deinterleaver circuit device configured to: i) deinterleave symbols of a plurality of FEC codewords in a second data stream, and ii) deinterleave symbols of a plurality of FEC codewords in a third data stream.

[0191] Example 6: A communication network according to any one of Examples 1 to 5, wherein the plurality of SERDES are a plurality of first SERDES, wherein the plurality of multiplexer / demultiplexer circuit devices are a plurality of first multiplexer / demultiplexer circuit devices, and wherein each first switch further includes: at least two corresponding first IC switching chips; and a corresponding plurality of second multiplexer / demultiplexer circuit devices, coupled to the corresponding plurality of second SERDES, each second multiplexer / demultiplexer circuit device being configured to: i) demultiplex a seventh data stream received via a corresponding uplink port into an eighth and a ninth data stream for transmission to a corresponding network interface pair of the first switch, and ii) multiplex a tenth and an eleventh data stream received from a corresponding network interface pair of the first switch into a twelfth data stream for transmission via the corresponding uplink port.

[0192] Example 7: The communication network according to Example 6, wherein: each second IC switching chip includes at least 1000 internal network interfaces; each second IC switching chip includes at least 500 external network interfaces; each external network interface operates at a data rate of at least 200 gigabits per second; and each second switch includes at least 500 ports, the at least 500 ports being communicatively coupled to the at least 500 external network interfaces of the corresponding second IC switching chip.

[0193] Example 8: A communication network according to any one of Examples 1 to 7, wherein the corresponding IC package is a corresponding first IC package, wherein the plurality of internal network interfaces are a plurality of first internal network interfaces, wherein the plurality of external network interfaces are a plurality of first external network interfaces, the plurality of first external network interfaces are coupled to a first external interconnect of the first IC package, wherein the packet processor is a first packet processor, wherein the plurality of multiplexer / demultiplexer circuit devices are a plurality of first multiplexer / demultiplexer circuit devices, and wherein each first IC switching chip of each first switch is included in a corresponding second IC package. Each first IC switching chip includes: a plurality of second external network interfaces coupled to a second external interconnect of a second IC package; a plurality of second internal network interfaces; a second packet processor coupled to the plurality of second internal network interfaces and configured to forward packets between the plurality of second internal network interfaces; and a plurality of second multiplexer / demultiplexer circuit devices, each second multiplexer / demultiplexer circuit device being coupled to: i) a corresponding second external network interface, and ii) a corresponding set of the plurality of second internal network interfaces. Each second multiplexer / demultiplexer circuit device is configured to: i) demultiplex a seventh data stream received from the corresponding second external network interface into an eighth and a ninth data stream for transmission to the corresponding set of the plurality of second internal network interfaces, and ii) multiplex a tenth and an eleventh data stream received via the corresponding set of the plurality of second internal network interfaces into a twelfth data stream for transmission to the corresponding second external network interface.

[0194] Example 9: According to the communication network of Example 8, wherein the plurality of SERDES are a plurality of first SERDES, and wherein each first switch further includes: a plurality of second SERDES, the plurality of second SERDES being coupled to a corresponding second external network interface corresponding to a downlink port of the first switch; and a plurality of third multiplexer / demultiplexer circuit devices being coupled to the plurality of second SERDES, each third multiplexer / demultiplexer circuit device being configured to: i) demultiplex a twelfth data stream received from the corresponding second external network interface of the second IC switching chip via the corresponding second SERDES into a thirteenth and a fourteenth data stream for transmission via the corresponding downlink port pair of the first switch, and ii) multiplex a fifteenth and a sixteenth data stream received via the corresponding downlink port pair of the first switch into a seventh data stream for transmission via the corresponding third SERDES to the corresponding second external network interface of the second IC switching chip.

[0195] Example 10: A communication network according to one of Examples 8 and 9, wherein: each first IC switching chip includes at least 1000 second internal network interfaces; each first IC switching chip includes at least 500 second external network interfaces; and each first switch includes at least 500 downlink ports, the at least 500 downlink ports being communicatively coupled to at least 250 second external network interfaces of the corresponding first IC switching chip.

[0196] Example 11: A method for communicating in a communication network, the communication network comprising: i) a plurality of first network switches, and ii) a plurality of second network switches, each first network switch comprising: i) a corresponding plurality of downlink ports, ii) a corresponding plurality of uplink ports, and iii) a corresponding one or more first integrated circuit (IC) switching chips, the corresponding one or more first IC switching chips being communicatively coupled to the plurality of uplink ports and the plurality of downlink ports, and each second network switch comprising: i) a corresponding plurality of ports, the corresponding plurality of ports being coupled to at least one uplink port of each of the first network switches, and ii) a corresponding second IC switching chip, the second IC switching chip being communicatively coupled to the plurality of ports of the second switch, the method comprising: receiving packets from a network device via the plurality of downlink ports at each of the plurality of first network switches; forwarding the packets received via the plurality of downlink ports to a plurality of second network switches via corresponding plurality of communication links between each of the first network switches and each of the plurality of second network switches. A second network switch; at each second switch, packets received from a plurality of first switches are transmitted to an external network interface of a second IC switching chip; in conjunction with each external network interface of each second IC switching chip, a corresponding first stream of packets received via the external network interface is demultiplexed to a plurality of internal network interfaces of the second IC switching chip; at each second IC switching chip, packets received via the internal network interfaces of the second IC switching chip are forwarded between the internal network interfaces of the second IC switching chip; in conjunction with each external network interface of each second IC switching chip, at least a second stream and a third stream of packets received via the plurality of internal network interfaces of the second IC switching chip are multiplexed to the external network interface; packets received from the external network interface of the corresponding second IC switching chip are forwarded to a plurality of first network switches by each second network switch via a plurality of corresponding communication links between the second network switch and each of the plurality of first network switches; and packets received from the plurality of second network switches by each of the plurality of first network switches are transmitted to a plurality of network devices via a plurality of downlink ports.

[0197] Example 12: The communication method according to Example 11 further includes: combining a set of multiple ports of each second switch, demultiplexing a corresponding fourth stream of packets received via a corresponding external network interface of a corresponding second IC switching chip to a set of multiple ports for transmission to one or more corresponding first network switches via a corresponding communication link; and combining a set of multiple ports of each second switch, multiplexing at least a fifth stream of packets and a sixth stream of packets received via a set of multiple ports to a corresponding external network interface.

[0198] Example 13: The communication method according to Example 12 further includes, for each second switch: multiplexing at least 500 sets of first incoming flows from at least 1000 first incoming flows of packets received via at least 1000 ports into at least 500 corresponding second incoming flows; providing at least 500 corresponding second incoming flows to at least 500 external network interfaces of the second IC switching chip; and at the second IC switching chip, demultiplexing at least 500 second incoming flows of packets received via at least 500 external network interfaces of the second IC switching chip into the second IC switching chip. The second IC switching chip has at least 1000 internal network interfaces; at the second IC switching chip, at least 500 sets of the first outgoing flows from at least 1000 packets received via the at least 1000 internal network interfaces of the second IC switching chip are multiplexed into at least 500 second outgoing flows; at least 500 corresponding second outgoing flows are provided to at least 500 external network interfaces of the second IC switching chip; and at least 500 second outgoing flows of packets received via the at least 500 external network interfaces of the second IC switching chip are demultiplexed to at least 1000 ports of the second switch.

[0199] Example 14: According to the communication method of Example 13, wherein for each second switch: providing at least 500 corresponding second incoming streams to at least 500 external network interfaces of the second IC switching chip includes providing each second incoming stream to the corresponding external network interface at a data rate of at least 200 gigabits per second (Gbps); and providing at least 500 corresponding second outgoing streams to at least 500 external network interfaces of the second IC switching chip includes providing each second outgoing stream to the corresponding external network interface at a data rate of at least 200 Gbps.

[0200] Example 15: A method for communication according to any one of Examples 11 to 14, wherein the method further comprises: combining each first stream of demultiplexed packets, i) demultiplexing the first stream or packets into at least a first substream and a second substream, ii) deinterleaving symbols of a plurality of forward error correction (FEC) codewords in the first substream, and iii) deinterleaving symbols of a plurality of FEC codewords in the second substream; and combining a second stream of multiplexed packets and a third stream of packets, i) interleaving symbols of a plurality of FEC codewords in the second stream, and ii) interleaving symbols of a plurality of FEC codewords in the third stream.

[0201] Example 16: The method for communication according to any one of Examples 11 to 15 further includes, at each first switch: in conjunction with each uplink port of the first switch, demultiplexing a corresponding fourth stream of packets received via the uplink port to at least: i) a corresponding external network interface of one first IC switching chip, and ii) a corresponding external network interface of another first IC switching chip; and in conjunction with each uplink port of the first switch, multiplexing at least the following to a seventh stream of packets for transmission via the uplink port: i) a fifth stream of packets from a corresponding external network interface of one first IC switching chip, and ii) a sixth stream of packets from a corresponding external network interface of another first IC switching chip.

[0202] Example 17: The communication method according to Example 16 further includes: transmitting packets from a corresponding port of a second IC switch to an external network interface at a data rate of at least 200 gigabits per second (Gbps) using each of at least 500 external network interfaces of each second IC switching chip; demultiplexing a corresponding first stream of packets received via the external network interface to multiple internal network interfaces of the second IC switching chip from at least 1000 internal network interfaces of the second IC switching chip using each of the at least 500 external network interfaces of each second IC switching chip; multiplexing a second stream and a third stream of packets received via multiple internal network interfaces of at least 1000 internal network interfaces of the second IC switching chip to the external network interface using each of the external network interfaces of each second IC switching chip; and transmitting packets from the external network interface of the second IC switching chip to a corresponding port of the second IC switching chip at a data rate of at least 200 Gbps using each of the at least 500 external network interfaces of each second IC switching chip.

[0203] Example 18: A method for communication according to any one of Examples 11 to 17, wherein the corresponding IC package is a corresponding first IC package, wherein the plurality of internal network interfaces are a plurality of first internal network interfaces, wherein the plurality of external network interfaces are a plurality of first external network interfaces, the plurality of first external network interfaces are coupled to a first external interconnect of the first IC package, wherein the packet processor is a first packet processor, wherein the plurality of multiplexer / demultiplexer circuit devices are a plurality of first multiplexer / demultiplexer circuit devices, and wherein each first IC switching chip of each first switch is included in a corresponding second IC package, the method further comprising: at each first switch, transmitting packets received from the plurality of second switches to a second interconnect of the first IC switching chip. A first set of external network interfaces; combining each of the second external network interfaces in the first set of second external network interfaces of each first IC switching chip, demultiplexing the corresponding fourth stream of packets received via the first external network interface to a plurality of second internal network interfaces of the first IC switching chip; at each first IC switching chip, forwarding packets received via the second internal network interfaces of the first IC switching chip between the second internal network interfaces of the first IC switching chip; and combining each of the second external network interfaces in the first set of second external network interfaces of each first IC switching chip, multiplexing at least a fifth stream and a sixth stream of packets received via the plurality of second internal network interfaces of the first IC switching chip to the second external network interface.

[0204] Example 19: The communication method according to Example 18 further includes, for each first switch: combining each downlink port of the set of multiple downlink ports of the first switch in a plurality of sets of multiple downlink ports, multiplexing an eighth stream and a ninth stream of at least a packet received via the set of multiple downlink ports to a corresponding tenth stream of the packet; combining each downlink port of the set of multiple downlink ports of the first switch, transmitting the tenth stream of the packet to a corresponding second external network interface of the first IC switching chip; combining each second external network interface of the first IC switching chip, demultiplexing a corresponding eleventh stream of the packet received via the second external network interface to at least a twelfth stream and a thirteenth stream of the packet; and combining each second external network interface of the first IC switching chip, transmitting at least the twelfth stream and the thirteenth stream of the packet to a corresponding set of multiple downlink ports of the first switch.

[0205] Example 20: A method for communication according to one of Examples 18 and 19 further includes, for each first switch: transmitting packets received from a plurality of second switches to a first set of second external network interfaces among at least 500 external network interfaces of a first IC switching chip; demultiplexing a corresponding fourth stream of packets received via the first external network interface to a plurality of second internal network interfaces among at least 1000 second internal network interfaces of the first IC switching chip, in conjunction with each second external network interface in the first set of second external network interfaces of the first IC switching chip; and forwarding packets received via the second internal network interfaces of the first IC switching chip among the at least 1000 second internal network interfaces of the first IC switching chip at each first IC switching chip.

[0206] Some of the blocks, operations, and techniques described above can be implemented using hardware, a processor executing firmware instructions, a processor executing software instructions, or any suitable combination thereof. When implemented using a processor executing software or firmware instructions, the software or firmware instructions can be stored in any suitable computer-readable storage medium. The software or firmware instructions may include machine-readable instructions that, when executed by one or more processors, cause one or more processors to perform the various actions described above.

[0207] When implemented in hardware, the hardware may include one or more of the following: discrete components, integrated circuits, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), etc.

[0208] Although the invention has been described with reference to specific examples, these examples are intended to be illustrative only and not limiting of the invention, but changes, additions and / or deletions may be made to the disclosed embodiments without departing from the scope of the invention.

Claims

1. A communication network, comprising: Multiple first switches, each first switch having: i) a corresponding first integrated circuit (IC) switching chip having multiple network interfaces, ii) a corresponding multiple downlink ports, and iii) a corresponding multiple uplink ports; as well as A plurality of second switches, each second switch having: i) a corresponding plurality of ports coupled to at least one uplink port of each of the first switches in the first switches; ii) a corresponding second IC switching chip in a corresponding IC package, the second IC switching chip having a plurality of external network interfaces coupled to external interconnects of the IC package; and iii) a corresponding plurality of serializers / deserializers (SERDES) for communicatively coupling the corresponding plurality of external network interfaces to the corresponding plurality of ports of the second switch, each second IC switching chip further comprising: Multiple internal network interfaces, A packet processor, coupled to the plurality of internal network interfaces, configured to forward packets between the internal network interfaces of the plurality of internal network interfaces, and Multiple multiplexer / demultiplexer circuit devices, each multiplexer / demultiplexer circuit device being coupled to: i) a corresponding external network interface, and ii) a corresponding set of multiple internal network interfaces, each multiplexer / demultiplexer circuit device being configured to: i) demultiplex a first data stream received from the corresponding external network interface into a second and a third data stream for transmission to the corresponding set of multiple internal network interfaces, and ii) multiplex a fourth and a fifth data stream received via the corresponding set of multiple internal network interfaces into a sixth data stream for transmission to the corresponding external network interface.

2. The communication network according to claim 1, wherein the plurality of multiplexer / demultiplexer circuit devices are a plurality of first multiplexer / demultiplexer circuit devices, and wherein each second switch further comprises: A plurality of corresponding second multiplexer / demultiplexer circuit devices are coupled to the respective plurality of SERDES, each second multiplexer / demultiplexer circuit device being configured to: i) demultiplex the sixth data stream received via the SERDES from the respective external network interface of the second IC switching chip into the eighth and ninth data streams for transmission via the respective port pair of the second switch; and ii) multiplex the tenth and eleventh data streams received via the respective port pair of the second switch into the first data stream for transmission via the SERDES to the respective external network interface of the second IC switching chip.

3. The communication network according to claim 2, wherein: Each second IC switching chip includes at least 1000 internal network interfaces; Each second IC switching chip includes at least 500 external network interfaces; and Each second switch includes at least 1,000 ports, which are communicatively coupled to at least 500 external network interfaces of the corresponding second IC switching chip.

4. The communication network of claim 1, wherein the plurality of SERDES are a plurality of first SERDES, and wherein each external network interface of the second IC switching chip includes or is coupled to: A corresponding second SERDES, which is coupled to the corresponding multiplexer / demultiplexer circuit device, is configured to: i) transmit the first data stream from the external network interface to the multiplexer / demultiplexer circuit device, and ii) transmit the sixth data stream from the multiplexer / demultiplexer circuit device to the external network interface.

5. The communication network of claim 1, wherein each multiplexer / demultiplexer circuit assembly comprises: A forward error correction (FEC) codeword interleaver circuit is configured to: i) interleave symbols of a plurality of FEC codewords in the fourth data stream, and ii) interleave symbols of a plurality of FEC codewords in the fifth data stream; as well as The FEC codeword deinterleaver circuit is configured to: i) deinterleave symbols of a plurality of FEC codewords in the second data stream, and ii) deinterleave symbols of a plurality of FEC codewords in the third data stream.

6. The communication network of claim 1, wherein the plurality of SERDES are a plurality of first SERDES, wherein the plurality of multiplexer / demultiplexer circuit devices are a plurality of first multiplexer / demultiplexer circuit devices, and wherein each first switch further comprises: At least two corresponding first IC switching chips; as well as A plurality of corresponding second multiplexer / demultiplexer circuit devices are coupled to a plurality of corresponding second SERDES, each second multiplexer / demultiplexer circuit device being configured to: i) demultiplex a seventh data stream received via a corresponding uplink port into an eighth and a ninth data stream for transmission to a corresponding network interface pair of the first switch, and ii) multiplex a tenth and an eleventh data stream received from the corresponding network interface pair of the first switch into a twelfth data stream for transmission via the corresponding uplink port.

7. The communication network according to claim 6, wherein: Each second IC switching chip includes at least 1000 internal network interfaces; Each second IC switching chip includes at least 500 external network interfaces; Each external network interface operates at a data rate of at least 200 gigabits per second; and Each second switch includes at least 500 ports, which are communicatively coupled to the at least 500 external network interfaces of the corresponding second IC switching chip.

8. The communication network of claim 1, wherein the corresponding IC package is a corresponding first IC package, wherein the plurality of internal network interfaces are a plurality of first internal network interfaces, wherein the plurality of external network interfaces are a plurality of first external network interfaces, the plurality of first external network interfaces being coupled to a first external interconnect of the first IC package, wherein the packet processor is a first packet processor, wherein the plurality of multiplexer / demultiplexer circuit devices are a plurality of first multiplexer / demultiplexer circuit devices, and wherein each first IC switching chip of each first switch is included in a corresponding second IC package, each first IC switching chip comprising: Multiple second external network interfaces, said multiple second external network interfaces being coupled to a second external interconnect of the second IC package, Multiple secondary internal network interfaces, A second packet processor, coupled to the plurality of second internal network interfaces, is configured to forward packets between the second internal network interfaces among the plurality of second internal network interfaces. A plurality of second multiplexer / demultiplexer circuit devices, each second multiplexer / demultiplexer circuit device being coupled to: i) a corresponding second external network interface, and ii) a corresponding set of a plurality of second internal network interfaces, each second multiplexer / demultiplexer circuit device being configured to: i) demultiplex a seventh data stream received from the corresponding second external network interface into an eighth and a ninth data stream for transmission to the corresponding set of the plurality of second internal network interfaces, and ii) multiplex a tenth and an eleventh data stream received via the corresponding set of the plurality of second internal network interfaces into a twelfth data stream for transmission to the corresponding second external network interface.

9. The communication network of claim 8, wherein the plurality of SERDES are a plurality of first SERDES, and wherein each first switch further comprises: Multiple second SERDES, the multiple second SERDES being coupled to a corresponding second external network interface corresponding to the downlink port of the first switch; as well as Multiple third multiplexer / demultiplexer circuit devices are coupled to the multiple second SERDES, each third multiplexer / demultiplexer circuit device being configured to: i) demultiplex the twelfth data stream received from the corresponding second external network interface of the second IC switching chip via the corresponding second SERDES into thirteenth and fourteenth data streams for transmission via the corresponding downlink port pair of the first switch; and ii) multiplex the fifteenth and sixteenth data streams received via the corresponding downlink port pair of the first switch into the seventh data stream for transmission via the corresponding third SERDES to the corresponding second external network interface of the second IC switching chip.

10. The communication network according to claim 8, wherein: Each first IC switching chip includes at least 1000 second internal network interfaces; Each first IC switching chip includes at least 500 second external network interfaces; and Each first switch includes at least 500 downlink ports, which are communicatively coupled to at least 250 second external network interfaces of the corresponding first IC switching chip.

11. A method for communication in a communication network, the communication network comprising: i) a plurality of first network switches, and ii) a plurality of second network switches, each first network switch comprising: i) a corresponding plurality of downlink ports, ii) a corresponding plurality of uplink ports, and iii) a corresponding one or more first integrated circuit (IC) switching chips, the corresponding one or more first IC switching chips being communicatively coupled to the plurality of uplink ports and the plurality of downlink ports, and each second network switch comprising: i) a corresponding plurality of ports, the corresponding plurality of ports being coupled to at least one uplink port of each of the first network switches, and ii) a corresponding second IC switching chip, the second IC switching chip being communicatively coupled to the plurality of ports of the second switch, the method comprising: At each of the plurality of first network switches, packets are received from the network device via the plurality of downlink ports; Each first network switch forwards packets received via the multiple downlink ports to the multiple second network switches via corresponding multiple communication links between the first network switch and each of the multiple second network switches. At each second switch, packets received from the plurality of first switches are transmitted to the external network interface of the second IC switching chip; By combining each external network interface of each second IC switching chip, the corresponding first stream demultiplexes the packets received via the external network interface to multiple internal network interfaces of the second IC switching chip; At each second IC switching chip, packets received via the internal network interface of the second IC switching chip are forwarded between the internal network interfaces of the second IC switching chip; By combining each external network interface of each second IC switching chip, at least a packetized second stream and a packetized third stream received via multiple internal network interfaces of the second IC switching chip are multiplexed to the external network interface; Each second network switch, via corresponding multiple communication links between the second network switch and each of the plurality of first network switches, forwards packets received from the external network interface of the corresponding second IC switching chip to the plurality of first network switches; and Each of the plurality of first network switches transmits packets received from the plurality of second network switches to the plurality of network devices via the plurality of downlink ports.

12. The method for communication according to claim 11, further comprising: By combining the set of multiple ports of each second switch, the corresponding fourth stream of packets received via the corresponding external network interface of the corresponding second IC switching chip is demultiplexed into the set of multiple ports for transmission to one or more corresponding first network switches via the corresponding communication link. as well as By combining the set of multiple ports of each second switch, at least a fifth flow of packets and a sixth flow of packets received via the set of multiple ports are multiplexed to the corresponding external network interface.

13. The method for communication according to claim 12, further comprising, for each second switch: Multiplex at least 500 sets of the first incoming streams from at least 1000 first incoming streams of packets received via the at least 1000 ports into at least 500 corresponding second incoming streams; The at least 500 corresponding second incoming streams are provided to at least 500 external network interfaces of the second IC switching chip; At the second IC switching chip, the at least 500 second incoming streams of packets received via the at least 500 external network interfaces of the second IC switching chip are demultiplexed to at least 1000 internal network interfaces of the second IC switching chip. At the second IC switching chip, at least 500 sets of the first outgoing streams from at least 1000 first outgoing streams of packets received via the at least 1000 internal network interfaces of the second IC switching chip are multiplexed into the at least 500 second outgoing streams; The at least 500 corresponding second outgoing streams are provided to the at least 500 external network interfaces of the second IC switching chip; as well as The at least 500 second outgoing flows of packets received via the at least 500 external network interfaces of the second IC switching chip are demultiplexed onto the at least 1000 ports of the second switch.

14. The method for communication according to claim 13, wherein for each second switch: Providing the at least 500 corresponding second incoming streams to the at least 500 external network interfaces of the second IC switching chip includes providing each second incoming stream to the corresponding external network interface at a data rate of at least 200 gigabits per second (Gbps); and Providing the at least 500 corresponding second outgoing streams to the at least 500 external network interfaces of the second IC switching chip includes providing each second outgoing stream to the corresponding external network interface at a data rate of at least 200 Gbps.

15. The method for communication according to claim 11, wherein the method further comprises: In combination with each first stream of demultiplexed packets, i) the first stream or packets are demultiplexed into at least a first substream and a second substream, ii) symbols of multiple forward error correction (FEC) codewords in the first substream are deinterleaved, and iii) symbols of multiple FEC codewords in the second substream are deinterleaved; as well as By combining the multiplexing of each at least grouped second stream and the grouped third stream, i) interleaving symbols of multiple FEC codewords in the second stream, and ii) interleaving symbols of multiple FEC codewords in the third stream.

16. The method for communication according to claim 11, further comprising, at each first switch: In conjunction with each uplink port of the first switch, the corresponding fourth stream demultiplexes the packets received via said uplink port to at least: i) a corresponding external network interface of one first IC switching chip, and ii) a corresponding external network interface of another first IC switching chip; and In conjunction with each uplink port of the first switch, at least the following are multiplexed into a seventh stream of packets for transmission via the uplink port: i) a fifth stream of packets from the corresponding external network interface of one of the first IC switching chips, and ii) a sixth stream of packets from the corresponding external network interface of the other first IC switching chip.

17. The method for communication according to claim 16, further comprising: By combining each of the at least 500 external network interfaces of each second IC switching chip, packets are transmitted from the corresponding port of the second IC switch to the external network interface at a data rate of at least 200 gigabits per second (Gbps). In conjunction with each of the at least 500 external network interfaces of each second IC switching chip, the corresponding first stream of packets received via the external network interface is demultiplexed to multiple internal network interfaces of the second IC switching chip from at least 1000 internal network interfaces of the second IC switching chip. By combining each external network interface of each second IC switching chip, at least a packet of second stream and a packet of third stream received via multiple internal network interfaces among the at least 1000 internal network interfaces of the second IC switching chip are multiplexed to the external network interface. as well as By combining each of the at least 500 external network interfaces of each second IC switching chip, packets are transmitted from the external network interface of the second IC switching chip to the corresponding port of the second IC switch at a data rate of at least 200 Gbps.

18. The method for communication according to claim 11, wherein the corresponding IC package is a corresponding first IC package, wherein the plurality of internal network interfaces are a plurality of first internal network interfaces, wherein the plurality of external network interfaces are a plurality of first external network interfaces, the plurality of first external network interfaces being coupled to a first external interconnect of the first IC package, wherein the packet processor is a first packet processor, wherein the plurality of multiplexer / demultiplexer circuit devices are a plurality of first multiplexer / demultiplexer circuit devices, and wherein each first IC switching chip of each first switch is included in a corresponding second IC package, the method further comprising: At each first switch, packets received from the plurality of second switches are transmitted to a first set of second external network interfaces of the first IC switching chip; By combining each of the second external network interfaces in the first set of the first IC switching chip, the corresponding fourth stream of the packets received via the first external network interface is demultiplexed to a plurality of second internal network interfaces of the first IC switching chip. At each first IC switching chip, packets received via the second internal network interface of the first IC switching chip are forwarded between the second internal network interfaces of the first IC switching chip. as well as By combining each of the second external network interfaces in the first set of the first IC switching chip, at least a fifth stream and a sixth stream of packets received via a plurality of second internal network interfaces of the first IC switching chip are multiplexed to the second external network interface.

19. The method for communication according to claim 18, further comprising, for each first switch: In combination with each downlink port in the set of multiple downlink ports of the first switch, at least the eighth stream and the ninth stream of the packet received via the set of multiple downlink ports are multiplexed into the corresponding tenth stream of the packet. By combining each downlink port in the set of multiple downlink ports of the first switch, the tenth stream of the packet is transmitted to the corresponding second external network interface of the first IC switching chip. By combining each of the second external network interfaces of the first IC switching chip, the corresponding eleventh stream of the packet received via the second external network interface is demultiplexed into at least the twelfth stream and the thirteenth stream of the packet; as well as In conjunction with each second external network interface of the first IC switching chip, at least the twelfth and thirteenth packets are transmitted to a corresponding set of multiple downlink ports of the first switch.

20. The method for communication according to claim 18, further comprising, for each first switch: The packets received from the plurality of second switches are transmitted to the first set of second external network interfaces among at least 500 external network interfaces of the first IC switching chip; Combining each of the second external network interfaces in the first set of second external network interfaces of each first IC switching chip, the corresponding fourth stream of packets received via the first external network interface is demultiplexed to a plurality of the at least 1000 second internal network interfaces of the first IC switching chip. as well as At each first IC switching chip, packets received via the second internal network interface of the first IC switching chip are forwarded among the at least 1000 second internal network interfaces of the first IC switching chip.

Images