Modular network element architecture

Abstract

A modular network element includes: one or more circuit boxes, the one or more circuit boxes being a mix between a rack mount module and a circuit module inserted into a chassis; one or more circuit box brackets, the one or more circuit box brackets being rack-mountable, and configured to selectively accommodate the one or more circuit boxes and provide the one or more circuit boxes with a power connector and a data connector; and a core chassis comprising one or more switch modules, one or more controller modules, and a set of connectors located at the rear for wiring to the power connectors and the data connectors on the one or more circuit box brackets.

Description

[0001] This application is a divisional application, and its original application is a PCT application with application number PCT / US2019 / 026489 and application date April 9, 2019, and a Chinese application that entered the Chinese national phase on September 15, 2020. The Chinese application number is 201980019486.4, and the name is "Modular Network Element Architecture". Technical Field

[0002] The present disclosure generally relates to network systems and methods. More particularly, the present disclosure relates to a modular network element architecture that combines rack-mounted units (i.e., "pizzaboxes") and chassis-based systems to provide flexibility, scalability, efficiency, etc., i.e., a hardware platform capable of supporting multiple applications including optical / photonic networking, time division multiplexing (TDM), packet switching, etc. Background Art

[0003] Networks, data centers, cloud infrastructures, etc. are ultimately implemented through physical hardware. Physical hardware is used to implement networking, computing and / or storage functions, including optical / photonic networking, time division multiplexing (TDM), packet switching, etc. Conventionally, there are usually two methods for how to deploy physical hardware—through a chassis and associated modules or through a rack-mounted unit. A chassis-based system includes multiple slots and backplanes (for simplicity, these are referred to as modules) that support different modules, line cards, circuit board cards, blades, etc. A chassis-based system enables deployment flexibility, where the modules needed or required are selectively inserted into the chassis. The chassis includes an electrical (printed circuit board (PCB)) backplane, fixed-size modules, shared power and cooling for all modules, such as a central switch structure through one or more modules, etc. Other advantages of the chassis include a one-time power supply installation for the chassis, front-end access only, redundancy, tight coupling of software and hardware, etc.

[0004] There are several fundamental challenges, namely 1) PCB backplanes are increasingly limited in their ability to support higher Serial-to-Deserializer (SERDES) rates at the same structure-to-module distance, 2) the industry business models that drive these types of designs are changing, 3) the consumption model of lifecycle and depreciation practiced by network operators, which is instead enforced by the merchant Application Specific Integrated Circuit (ASIC) supply chain, means greater modularity is required, etc. That is, the conventional chassis-based approach focuses on fixed form factors, which limits flexibility in size, power, and the ability to handle different applications. In addition, the chassis-based approach requires early lock-in, suboptimal panel usage, fixed slot size and power, fixed cooling, fixed slot bandwidth, etc. The chassis-based approach requires a large initial investment.

[0005] Rack-mount units are also called "boxes" and typically include 1-2 (or more) rack units (RU or simply U, which is a standard unit measuring 44.5mm or 1.75") in height in a fully enclosed form factor. Rack-mount units differ from chassis in that all hardware is already included when the rack-mount unit is deployed. For example, data centers use rack-mount units to implement network elements such as leaf / spine architectures in a fully decentralized and modular form. The advantages of the rack-mount unit approach include standard interfaces, stackability, low cost, adaptability, low initial investment cost, and pay-as-you-go. While this provides flexibility, this approach has interconnect complexity, such as a large amount of wiring in the front end, and this wiring is based on pluggable optics, which lack the density achievable in a chassis-based approach. Likewise, this approach does not include redundancy, expansion to larger systems is complex and inefficient, and power connections are required for each installation. Summary of the invention

[0006] Since there are only two link speed generations left in the current size PCB backplane, there will be a transition to electrical cable-connected backplanes and then to optical cable-connected backplanes. Many people see cable-connected systems as a replacement for PCB backplanes that provide better signal integrity. However, cable-connected backplanes provide a whole new way to think about platform design aspects that are not simply a direct replacement for PCB backplanes.

[0007] Therefore, it would be advantageous to provide a hybrid hardware architecture between a fixed chassis-based approach and fully decentralized rack-mounted units.

[0008] In one embodiment, a modular network element includes: one or more line boxes, the one or more line boxes being a hybrid between a rack-mounted module and a line module inserted into a chassis; one or more line box brackets, the one or more line box brackets being rack-mountable and configured to selectively accommodate one or more line boxes and provide them with power connectors and data connectors; and a core chassis, which includes one or more switch modules, one or more controller modules, and a set of connectors at the rear for wiring to the power connectors and data connectors on the one or more line box brackets. The one or more line boxes can include a plurality of panel ports connected to a fabric circuit that is communicatively coupled to one or more switch modules via wiring; and the one or more switch modules are connected to each other and to the one or more controller modules via a midplane or cables.

[0009] The wiring can include one or more of electrical wiring and optical wiring with multiple pairs of coaxial cables. Each cable between the line box and the core chassis can include multiple bundles, each bundle has multiple links, and the number of multiple bundles is based on the number of one or more switch modules. A chassis management protocol is implemented between the one or more line boxes and the core chassis, and the one or more line boxes and the core chassis can be managed as a single entity. The chassis management protocol can implement timing synchronization, presence, status, ownership, interruption, reset, power and location identification between the one or more line boxes and the core chassis. One or more controller modules provide power and connections to the circuits on each line box to establish a chassis management protocol to turn the line box power on and off. The core chassis can include an access panel that supports each of the core chassis and one or more line boxes. Each of the one or more line boxes can include a display that provides a virtual slot number based on a relative position in the rack.

[0010] One or more line boxes can include multiple line boxes, at least two of which have different spacings. One or more line boxes can include multiple line boxes, at least two of which have different numbers of panel ports and one or more of different types of pluggable optical devices. One or more line boxes can each include an Ethernet switch that is connected to another Ethernet switch in one or more switch modules. One or more line box brackets can include a retractable pin having a spring-loaded mechanism that snaps into a hole in a track on the rack and has a push-to-release mechanism so that the rack can only be approached from the front. One or more line box brackets can have slack in the wiring to the core chassis, allowing movement up and down the rack.

[0011] In another embodiment, a method of deploying a modular network element includes: connecting a line box slot bracket to a rack; providing a line box in the line box slot bracket, wherein the line box is a hybrid between a rack-mounted module and a line module inserted into a chassis, and wherein the line box slot bracket connects the line box to a power connector and a data connector; and providing a core chassis, the core chassis including one or more switch modules, one or more controller modules, and a set of connectors at the rear that are wired to the power connector and the data connector on the line box slot bracket. The method can also include providing a second line box to the second line box slot bracket in response to increasing the capacity of the modular network element. The second line box can be added to the core chassis in service.

[0012] In another embodiment, the modular network element includes: one or more line boxes, which are a hybrid between a rack-mounted module and a line module inserted into a chassis, each of the one or more line boxes including a plurality of panel ports connected to a structural circuit and a processor; and one or more line box brackets, which are rack-mountable and configured to selectively accommodate one or more line boxes and provide power connectors and data connectors thereto. The one or more line boxes can include a plurality of line boxes, which together form a grid via data connectors, and one of the plurality of line boxes can be designated as a master line box for implementing a virtual controller of the modular network element via a processor. A core chassis can be added in service to increase the capacity of the modular network element, and the core chassis can include one or more switch modules, one or more controller modules, and a set of connectors at the rear for wiring to power connectors and data connectors on one or more line box brackets.

[0013] In one embodiment, a method of Ethernet path selection is performed in a modular network element including one or more ingress line modules, a plurality of switch modules in a chassis, and one or more egress line modules, wherein the one or more ingress line modules and the one or more egress line modules are located separately from the chassis and connected to the chassis via wiring. The method includes: distributing one or more Ethernet flows from one or more ingress line modules to one or more egress line modules via a plurality of switch modules; receiving fabric status information from the out-of-band chassis at the one or more ingress line modules via wiring; and, in response to congestion on the path through the plurality of switch modules, the ingress line module selecting a new path for the Ethernet flow. The method can also include creating a gap in the Ethernet flow by buffering before switching the Ethernet flow to the new path, such that the gap is no greater than the path delay difference between the current path and the new path. The size of the gap can be set to the delay of the current path minus the delay of the new path plus a margin.

[0014] The method can also include allocating time division multiplexed (TDM) streams by segmenting the TDM streams and sending them over multiple paths through multiple switch modules. The method can also include allocating timing from an out-of-band chassis via wiring, where the timing is determined at the chassis using time division multiplexed (TDM) streams spread over multiple switch modules. The chassis can use TDM packet headers to estimate relative path delay differences. The method can also include broadcasting the path selection to all other line modules and chassis through each ingress line module to maintain a global path table respectively. The Ethernet stream can include an elephant stream, and if the burst gap of the elephant stream is greater than the delay difference between the current path and the new path, each stream burst of the elephant stream can take a different path.

[0015] In another embodiment, a modular network element configured for Ethernet path selection includes: one or more ingress line modules; one or more switch modules in a chassis; and one or more egress line modules, wherein the one or more ingress line modules and the one or more egress line modules are located separately from the chassis and connected to the chassis via wiring, wherein one or more Ethernet flows are distributed from the one or more ingress line modules to the one or more egress line modules via a plurality of switch modules, wherein the one or more ingress line modules receive fabric status information from the chassis out-of-band via the wiring, and wherein, in response to congestion on the path through the plurality of switch modules, the ingress line modules select a new path for the Ethernet flow.

[0016] In another embodiment, a chassis in a modular network element configured for Ethernet path selection includes: one or more switch modules; one or more controllers; a midplane for communication between the one or more switch modules and the one or more controllers; and wiring for communication to one or more ingress line modules and one or more egress line modules, wherein the one or more ingress line modules and the one or more egress line modules are located separately from the chassis and connected to the chassis via the wiring, wherein one or more Ethernet flows are distributed from the one or more ingress line modules to the one or more egress line modules via a plurality of switch modules, wherein the one or more ingress line modules receive fabric status information from the one or more out-of-band controllers via the wiring, and wherein, in response to congestion on the path through the plurality of switch modules, the ingress line modules select a new path for the Ethernet flow.

[0017] In one embodiment, a modular network element includes a plurality of line modules and zero or more switch modules in a chassis, wherein the plurality of line modules are located separately from the chassis and connected to the chassis and / or to each other via wiring. A method of managing the modular network element as a single entity includes: operating a management plane between the plurality of line modules and the zero or more switch modules via one or more dedicated links in the wiring; managing the plurality of line modules and the zero or more switch modules as a single network element over the management plane using a chassis management protocol; and designating one of the following as a primary virtual processor for the chassis management protocol: a controller in the chassis and a processor in one of the plurality of line modules.

[0018] The method can also include providing a virtual slot number for each of the plurality of line modules, wherein the virtual slot number is used to distinguish the plurality of line modules. The virtual slot number can be automatically assigned based on a position sensor on each of the plurality of line modules or by a chassis that detects the position of each of the plurality of line modules. The management plane can operate through a universal serial bus (USB) pin and a 10 Gigabit Ethernet alternate mode using a subset of the USB pins. The USB can be used to bridge multiple protocols in the management plane and determine the presence of the module, and wherein the 10 Gigabit Ethernet is used for a precision time protocol. The USB can be used for power, presence, primary arbitration, bidirectional status, health monitoring, configuration, data path initialization, software download, interrupt, reset, LED control, I2C bridging, universal asynchronous receiver / transmitter (UART) bridging, SPI bridging, pulse width modulation (PWM) bridging, general purpose input / output (GPIO) bridging, and PCIe bridging.

[0019] The method can also include communicating between a controller in the chassis and circuitry on the line module to establish a chassis management protocol on the line module. The method can also include selecting a candidate clock for synchronization and timing distribution in the chassis management protocol. The method can also include continuously measuring delays on the wire to adjust the phase of the timing distribution. The management plane can operate on the wire out-of-band from the data plane.

[0020] In another embodiment, a modular network element managed as a single entity includes a plurality of line modules; and zero or more switch modules in a chassis, wherein the plurality of line modules are positioned separately from the chassis and from each other and are connected to the chassis and / or to each other via wiring, wherein a management plane is operated between the plurality of line modules and the zero or more switch modules via one or more dedicated links in the wiring, wherein the plurality of line modules and the zero or more switch modules are managed as a single network element on the management plane using a chassis management protocol, and one of the following is designated to function as a primary virtual controller for the chassis management protocol: a controller in the chassis and a processor in one of the plurality of line modules.

[0021] In another embodiment, a device configured to manage modular network elements includes a processor and a memory storing instructions that, when executed, cause the processor to operate a management plane between multiple line modules and zero or more switch modules via one or more dedicated links in wiring between the multiple line modules and zero or more switch modules; and manage the multiple line modules and zero or more switch modules as a single network element on the management plane using a chassis management protocol, wherein the device is designated as a primary device in the chassis management protocol and the device is physically separated from at least one of the multiple line modules and the zero or more switches. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present disclosure is illustrated and described herein with reference to the various drawings, wherein like reference numerals are used to represent like system components / method steps where appropriate, and wherein:

[0023] Figure 1 It is a logical diagram of a modular network element architecture that uses line boxes and core boxes to achieve system capacities from 3.6Tb / s to 80Tb / s and higher;

[0024] Figure 2 yes Figure 1 A logical diagram of the connection between the line box and the core box in different modular network element architecture configurations;

[0025] Figure 3 is a block diagram of the connections between a core box, associated line boxes, and optional slave core boxes;

[0026] Figure 4 It is a block diagram of the data plane and control plane structure connection in the modular network element architecture;

[0027] Figure 5 It is a block diagram of the chassis management of the core box;

[0028] Figure 6 It is a block diagram of the physical connections in the modular network element architecture;

[0029] Figure 7 is a block diagram of a rack equipped with a circuit box bracket and without a circuit box therein;

[0030] Figure 8 is a block diagram of a rack having a circuit box in a circuit box bracket;

[0031] Fig. 9 is a three-dimensional diagram of the circuit box slot bracket;

[0032] Fig.10 is a bottom view of the circuit box slot bracket;

[0033] Fig.11 is a perspective view of a rack having a core box, four line boxes in associated line box bays, and eight empty line box bays, in front and rear views;

[0034] Fig.12 yes Fig.11 Front and rear views of the rack;

[0035] Fig.13 It is a three-dimensional image of the core box;

[0036] Fig.14 It is a block diagram of the communication between the core box and the line box;

[0037] Fig.15 is a logical diagram of Ethernet paths with and without congestion through a modular network element architecture;

[0038] Fig.16 is a logical diagram illustrating an Ethernet path through a modular network element architecture for multicast;

[0039] Fig.17 It is a logical diagram of Ethernet multipath data flow through a modular network element architecture;

[0040] Fig.18 It is a logical diagram of TDM multipath data flow through a modular network element architecture;

[0041] Fig.19 It is a logical diagram of Ethernet flows;

[0042] Fig. 20 is a block diagram showing a modular network element architecture for TDM over Ethernet fabric;

[0043] Fig.21 is a diagram of a single line box interconnected with other line boxes and core boxes in a self-expanding manner;

[0044] Fig. 22 is a front view of an example modular network element architecture with different line boxes and ports;

[0045] Fig.23 is a network diagram of line boxes in a self-expanding grid without core boxes;

[0046] Fig.24 is a front view of a modular network element architecture showing virtual slot numbering and location sensors;

[0047] Fig.25 is a block diagram of an example implementation of a circuit box;

[0048] Fig.26 is a frame diagram of a front view and a rear view of another implementation of the core box;

[0049] Fig. 27 It is a diagram of a twinax cable and connector;

[0050] Fig.28 is a flow chart of the process of deploying modular network elements;

[0051] Fig.29 is a flow chart of a process of Ethernet path selection in a modular network element; and

[0052] Fig.30 is a flow chart of the process of managing modular network elements. DETAILED DESCRIPTION

[0053] The present disclosure relates to combining a modular network element architecture that is a hybrid of a conventional chassis-based approach with a fully decentralized rack-mounted unit approach. The modular network element architecture provides the integrated value from the chassis-based approach while providing the flexibility of a decentralized rack-mounted unit approach. Network elements can be implemented in various ways using the modular network element architecture for packet, hybrid TDM / packet, optical, hybrid TDM / optical, hybrid TDM / packet, hybrid optical / TDM / packet, etc. Aspects of the modular network element architecture include modularity, stackability and scalability; control plane; flexible panel size; decentralized power / cooling; open backplane interface, etc. The modular network element architecture is constructed to outlast several generations of backplane or chipset designs. The modular network element architecture provides the advantages of both the chassis-based approach and the rack-mounted unit approach.

[0054] The modular network element architecture provides operators with the ability to stack and expand. The purpose of the modular network element architecture is to provide a single hardware platform that can meet various markets, applications, etc. In other words, the modular network element architecture can unify different platforms, thereby minimizing development costs and efforts. Slots with variable spacing, variable bandwidth, variable depth, and variable power / cooling mean that the modular network element architecture can provide a platform for many markets. Each market has different requirements for module size based on chipsets, panel components, etc. For example, the hybrid TDM / packet market requires a front-end OTN chipset that the packet-only market does not need. As another example, the packet-only market may have a higher oversubscription rate and therefore require a larger panel area for more pluggable optical devices.

[0055] Modular network element architecture

[0056] Figure 1It is a block diagram of a modular network element architecture 10 that utilizes line boxes 12 and core boxes 14 to achieve system capacities of 3.6Tb / s to 80Tb / s and above. The modular network element architecture 10 is a hybrid between a chassis-based approach and a rack-mounted unit approach. Specifically, the modular network element architecture 10 includes one or more line boxes 12, which are a combination of modules in part rack-mounted units and part chassis. The line box 12 is called a "line box" because of its hybrid nature between a line module and a box-type wavelength division box. The line box 12 can be of variable size and is a fully enclosed rack-mounted module similar to a box-type wavelength division box, but is inserted into a slot bracket in a rack with data connectors (e.g., twinax) and power connectors. The line box 12 enables flexibility, low initial investment cost, pay as you grow, a range of system sizes, and is not limited by slot spacing or slot power (as in a chassis). Additional details of the line box 12 are described herein, but generally the line box 12 has front-end access and provides switching at layer 1 (TDM), layer 2 (packet), etc. For example, the line box 12 can be 1-2U high, while the core box 14 can be 6-12U high. The line box 12 is a hybrid between a box-type wavelength division box and a line module, for example, a box-type wavelength division box that plugs into a slot bracket with connectors for data and power. The core box 14 is similar to a chassis, but is only used for switch modules (SM) 16 and controller (CTM) modules 18.

[0057] At the lower capacity end, e.g., 3.6Tb / s to 7.2Tb / s, the modular network element architecture 10 can employ only one or two line boxes 12. Advantageously, this avoids the need for switch modules (SMs), backplane hardware, cabling, power supplies, etc., which are not needed at this capacity. Core boxes 14 can be added when it is necessary to provide expansion to the modular network element architecture 10, e.g., 19.2Tb / s and above, and to provide intermediate / center level switches. After more than one line box 12 is employed in the modular network element architecture 10, the connection between the line box 12 and possible core boxes 14 is through connectors on the associated slot tray housing the line box 12. The core box 14 can support one to four switch modules 16, two controller modules 18, and an access panel 20. The modular network element architecture 10 can support more than 40Tb / s in a single rack and scale to multiple racks by interconnecting core boxes 14, e.g., in a multi-stage Clos switch.

[0058] Figure 21 is a logical diagram of the connection between the line box 12 and the core box 14 in different modular network element architecture 10 configurations. For illustrative purposes, the modular network element architecture 10 is described with reference to packet switching. However, those skilled in the art will recognize that the same hardware, interconnectivity, etc. can be used for TDM (Optical Transport Network (OTN)) switching, Multi-Protocol Label Switching (MPLS), Internet Protocol (IP) routing, etc. For example, a single line box 12 can include, for example, 16 front-end access ports of 100Gb / s each, i.e., 16x100GbE (Gigabit Ethernet). Two such line boxes 12 can jointly support 32 front-end access ports (16x100GbE×2) and 16x100G ports on the back end through slot brackets. For example, for each switch module 16, the core box 14 can support 128×100GbE. In the case of 8 line boxes 12 and 4 switch modules 16 in one core box 14 at 12.8 Tb / s, each line box 12 can connect 4×100GbE to each switch module 16. In the case of 64 line boxes 12 and 8 switch modules 16 in two core boxes 14 at 102.4 Tb / s, each line box 12 can connect 2×100GbE to each switch module 16.

[0059] Comparison metrics

[0060] Hardware platforms can be compared based on metrics such as watts / Gb / s, volume / Gb / s, and cost / Gb / s. For power, when air cooling is driven to its limits, fan power consumption accounts for a higher percentage of total power, and more power is required to bend the airflow twice relative to front-to-back airflow. The modular network element architecture 10 is an open structure that allows only a sufficient number of fabric links to be used for each application, requiring less power. Similarly, the insertion loss of dual-core coaxial is lower than that of a PCB backplane, so less power is required per distance. The modular network element architecture 10 can include optimized power supplies tuned to the specific load of the line box 12. A system that fully utilizes its chipset is more power efficient than a system that does not fully utilize its chipset. For fixed-pitch systems, the panel area does not always match the chipset.

[0061] If the power is optimized to deliver only the power required in the existing hardware, there is less space and volume. The modular network element architecture 10 can be relocated to work in a shallower or deeper chassis and is therefore more volume efficient. In terms of cost, incorporating high-speed optical modems into fixed-spacing designs requires heat pipe cooling, increased cost of backplanes, high upfront costs to develop proprietary dedicated hardware, etc., which leads to high costs for conventional approaches. These costs are all avoided in the modular network element architecture 10. The modular network element architecture 10 also avoids the need for a large number of pluggable optical transceivers that are space inefficient. In addition, the modular network element architecture 10 enables variable spacing to support multiple markets / applications through one platform. That is, the modular network element architecture 10 is meant to address the problems of various markets / applications, thereby improving the efficiency of suppliers.

[0062] connect

[0063] Figure 3 is a block diagram of the connections between a core box 14 , an associated line box 12 , and optional slave core boxes 14 . Figure 4 It is a block diagram of the data plane and control plane structural connection in the modular network element architecture 10. Figure 5 It is a block diagram of the chassis management of the core box 14. Figure 6 is a block diagram of the physical connections in the modular network element architecture 10.

[0064] exist Figure 3 In the embodiment, the main core box 14 includes a switch module 16, a controller module 18, and an access panel 20. The access panel 20 can include various ports, indicators (e.g., light emitting diodes (LEDs), etc.), etc. The core box 14 can also include a fan 22 and a power supply 24. The switch module 16 includes a 1×100GbE fabric connection to the controller module 18 and a universal serial bus (USB) connection (e.g., a peripheral component express interconnect (PCIe) bridge) to the controller module 18. The controller module 18 can include various connections to the access panel 20, a USB connection to the fan 22 (e.g., a serial peripheral interface (SPI) bridge), and a USB connection to the power supply 24 (e.g., an integrated circuit bus (I2C) bridge). The external connections to the core box 14 can include an N×100GbE fabric link from the line box 12, a 2×USB connection from the line box 12 or the slave core box 14, and a 2×10GbE USB connection from the line box 12 or the slave core box 14.

[0065] The USB management functions in the main core box 14 include power, presence, primary arbitration, bidirectional status, health monitoring, configuration, data path initialization, software download, interrupt, reset, LED control, I2C bridging, universal asynchronous receiver transmitter (UART) bridging, SPI bridging, pulse width modulation (PWM) bridging, general purpose input / output (GPIO) bridging, and PCIe bridging. The 10GbE control structure (USB alternate mode) can perform functions related to fabric congestion status transmission, timing distribution (Synchronous Ethernet (SyncE) and Precision Time Protocol (PTP)), direct communication between line box 12 peers, statistics, external management network, etc. Note that in addition to the line box 12, the modular network element architecture 10 can include optical boxes (for dense wavelength division multiplexing (DWDM)), server boxes (for computing or storage resources), etc. For reference, the term line box 12 is used herein, and the line box 12 includes panel ports and circuits for switching (TDM, packet, etc.). However, the line box 12 can be an optical box (for transceivers, modems, repeaters, etc.), a server box, etc.

[0066] exist Figure 4 , two example line boxes 12 connected to a core box 14 are shown. The line box 12 includes an Ethernet switch 30, a physical (PHY) interface 32 connected to the Ethernet switch 32, and a processor (CPU) 34. Again, for example, a line box 12 and a core box 14 with Ethernet switching are shown, but those skilled in the art will recognize that other protocols are also considered, such as OTN, optical, etc. The physical interface 32 can include a 100G or similar optical transceiver that provides 100GbE. The core box 14 similarly includes multiple Ethernet switches 40 on the switch module 16. The controller module 18 also includes an Ethernet switch 42 and a processor 44. The Ethernet switch 30 on the line box 12 is connected to the Ethernet switch 40 on the switch module 16, for example, via a 100GbE link. The processor 34 is connected to the Ethernet switch 42, for example, via USB / 10GbE.

[0067] The modular network element architecture 10 uses USB for the management plane based on the need to use open interfaces for external connections rather than proprietary connections. That is, there are many software infrastructures around USB, and USB is widely used in chipsets, etc. USB operates in alternate mode or bridging. In USB alternate mode, a subset of USB pins is assigned to a completely different serial protocol (10GE in this example). Therefore, this protocol runs in parallel with the USB protocol instead of running on top of the USB protocol. 10GbE running in alternate mode can be used for SyncE+PTP timing allocation, etc. For USB bridging, there are devices that bridge the following protocols based on the USB protocol: I2C, UART, SPI, PWM, PCIe, and GPIO. USB is a subject: peripheral model. All services are initiated by the subject (usually the controller module 18). Interrupts caused by peripherals are implemented via polling of the subject.

[0068] Figure 5 The details of the core box middle board 50 and the controller module 18 are shown. The controller module 18 has a processor 44 that can be based on x86 and an Ethernet switch 42 that can be 10GbE. Similarly, the controller module 18 includes a timing module (TM), a chassis management protocol, a chassis GPIO database, and a power supply. The controller module 18 is connected to the line box 12 and other core boxes 14 via the core box PCB middle board 50. Any x86 in the system (including the x86's on the line box 12) can become the main control plane or secondary control plane of the operation high-level control plane (e.g., MPLS, operation, management and maintenance (OAM)), but cannot become a low-level chassis management plane (e.g., chassis management protocol, 10GbE switch, system GPIO database, fabric management PCIe, etc.). The main control of the chassis management plane is limited to the controller module 18.

[0069] The master controller module 18 provides 12V power to a small field programmable gate array (FPGA) or other circuit on each line box 12 to establish a chassis control plane that can power on and off the line boxes 12. Any line box 12 can provide candidate timing; the master controller module 18 selects a candidate clock to synchronize its own TM block, and the TM block then distributes the timing to all line boxes. The chassis management protocol includes the establishment of a bidirectional protocol to indicate the presence of the line box 12. The master controller module 18 receives GPIO from all line boxes 12 and periodically broadcasts the chassis GPIO database to all line boxes 12. This implements reset, interrupt, primary-ship request, primary-ship grant, etc. The protocol can have a fixed superframe size, allowing accurate ToD / 1pps / 8kHz transmission / reception to all line boxes. Similarly, each line box 12 can display its virtual slot number on its panel.

[0070] exist Figure 6 In the figure, the side view, front view and rear view of the modular network element architecture 10 are used to illustrate the example physical connection. The modular network element architecture 10 supports installation in a 19" (or 23") rack. Figure 6 The rack is not shown. The core box 14 can have a depth of approximately 450 mm. The switch modules 16, controller modules 18, and access panels 20 do not require a large depth even after taking into account the bending radius of the twinax beam (approximately 0.6"). Depending on the application, the line box 12 can have different depths, such as 450-1000 mm. The line boxes 12 can be targeted at different sizes because they do not have to work within the traditional constraints of size, power, cooling, and connectivity. The modular network element architecture 10 supports front-to-back airflow.

[0071] Figure 6 The example includes a core box 14 with six possible switch modules 16, only three of which are shown equipped. Several different types of line boxes 12 are shown, including an X86 server box providing computing resources, a 36×100G line box (e.g., double height) for additional panel ports, a 16×100G line box (e.g., single height, but extra depth), a fiber deep line box (e.g., double height, extra depth), etc.

[0072] The core box 14 includes a midplane 50 that interconnects the switch modules 16, the controller modules 18, and the access panels 20, among others. The core box 14 can be connected to the line box 12 via a twinax connector 52 that is wired from the back of the core box 14 to a line box bracket, which is described herein in additional detail. Figure 6In the rear view of the circuit box 12, the circuit box 12 can have one, two, or more twinax connectors 52. For example, a double-height circuit box 12 can have two twinax connectors 52, a single-height circuit box 12 can have one twinax connector 52, and so on. The core box 14 can have a twinax connector area 54 wired to each of the circuit box brackets and a control plane connector area 56. For example, each twinax connector 52 can have 36 data plane links, which are divided into 3×12 links to the twinax connector area 54. Of course, other implementations are also considered. The twinax connector 52 can be directly plugged into an Ortho-Direct connector (from Molex), a PCB backplane, or a cable-connected backplane.

[0073] The present disclosure uses the term "twinax" for the twinax connector 52 and the twinax connector region 54. Those skilled in the art recognize that this can also include twinax or any other type of electrical wiring such as coax wiring. That is, the electrical wiring can have any type of implementation.

[0074] Compatibility with Operations, Administration, Maintenance, and Provisioning (OAM&P)

[0075] The line box 12 and the core box 14 enable an extensible chassis control plane with multiplexing protocols, the ability to add x86 servers to the control plane, and the ability to select primary ownership that is implemented only in the line box 12 outside the core box 14. Note that the modular network element architecture 10 is a hybrid between a chassis-based implementation and a box-based WDM box implementation. However, the operation of OAM&P is for chassis-based management. Therefore, the modular network element architecture 10 can be configured to emulate an existing chassis from a software perspective. For example, the modular network element architecture 10 includes the concept of virtual slot numbering to help operators locate the line box 12 in a physical rack. In one embodiment, the modular network element architecture 10 can use a position sensor such as an altimeter on each line box 12 to establish a monotonic virtual slot number sequence in the rack. This provides a unique determination of the physical slot number, allows the use of slots of different physical sizes, a method of sorting the slot numbers based on the position sensor, and the use of a display to display the slot numbers.

[0076] As described in this article, the scalable chassis control plane is able to use USB with 10GbE alternate mode to enable flexible chassis management (primary ownership, remote power, presence, status, health monitoring, configuration, software download, interrupts, reset, LED control, protocol bridging, etc.).

[0077] The chassis control plane can include a split of primary ownership: multiple primary ownerships based on box type and function (i.e., not limited to core box 14). The chassis control plane includes: the ability to distribute multiple clock domains over a single link, remote powering to allow management of line boxes 12 without a main power source, the ability to measure round trip cable time to adjust the clock phase for each box, the ability to upgrade the backplane to which the cables are connected (one cable at a time) without severely impacting the chassis control plane, the ability for the core box 14 to restart or communicate with a remote line box only when the control circuitry of the remote line box 12 is powered, and the ability for the line box 12 to be powered on without the core box 14 in a mesh configuration either standalone or as part of a negotiation process.

[0078] It is not clear how the chassis control plane should be implemented in a rack with a line box 12, especially when it needs to emulate an existing chassis from a software perspective. It also needs to handle N:1 redundancy, line boxes 12 that can manage the entire chassis (standalone and mesh configurations), reduce the number of links to simplify cable management, and the ability to extend the control plane to the server boxes.

[0079] In the past, common practice used to be "1 pin per signal" for the various signals. Historically, this was due to the relative overhead of SERDES capacity relative to backplane pins. This has changed recently and allows for greater multiplexing of many signals on a single SERDES link. So, chassis management protocol link carrying time (ToD), 1pps, 8kHz, GPIO, presence, status, interrupt, reset, primary ownership, etc.

[0080] The chassis management protocol is bidirectional between line boxes 12, core boxes 14, etc., and the establishment of line boxes, core boxes between the two devices is used to indicate the presence of the box. The main controller 18 receives GPIO from all line boxes 12 and broadcasts the chassis GPIO database to all line boxes 12 periodically. This implements reset, interrupt, master ownership request, master ownership permission, virtual slot number, etc. The protocol has a fixed superframe size, allowing accurate ToD / 1pps / 8kHz transmission / reception to all line boxes 12; 25Mhz allocation uses a dedicated link.

[0081] After successful negotiation, any x86 in the system (including the x86's on the line box 12, i.e., the processor 34) can become primary or secondary for the chassis control plane (e.g., MPLS graceful restart, OAM), but not the low-level chassis management plane (e.g., chassis management protocol, 10GbE switch, system GPIO database, fabric management). Negotiation is based on memory and processing power depending on the currently configured functions and reserved capacity. When present in the interconnected modular network element architecture 10, primary ownership of the chassis management plane is limited to the controller 18 on the core box 14.

[0082] The master controller 18 provides 12V power to a small FPGA on each line box 12 and server box to establish a chassis control plane that can power the boxes on and off, hold resets, etc. Any box can provide candidate timing; the master controller 18 selects a candidate clock to synchronize its own timing module 104 block, which then distributes the timing to all boxes. Multiple clock domains can be distributed by sending a frequency and 1pps offset relative to the main superframe phase and a single 25Mhz clock link. This enables multiple timing domains to be distributed in a single chassis if desired.

[0083] As part of the chassis management protocol, round trip delay measurements are used to measure the backplane cable length. This is used to adjust the phase of the timing distributed to each line box 12. This is a continuous process and will apply if the backplane cable is replaced.

[0084] USB is commonly used to connect peripherals, but it is proposed to use USB and its 10GE alternate mode together in a modular network element architecture 10 to manage the chassis; in particular, the use of USB in multiple master modes with master ownership negotiation. Functional partitioning between USB and 10GE alternate modes and how they work together are proposed, and improved implementation methods are provided. For example, USB power is selected instead of PoE (Power over Ethernet). Similarly, module presence is set over USB instead of Ethernet. USB is also used to bridge various protocols such as I2C, UART, SPI, PWM, and PCIe. However, SyncE and PTP timing are distributed over Ethernet instead of USB.

[0085] Circuit box bracket

[0086] Figure 7 1 is a block diagram of a rack 60 in which a circuit box bracket 70 is provided but without a circuit box 12.

[0087] Figure 8is a block diagram of a rack 60 having a circuit box 12 in a circuit box bracket 70. The rack 60 can be a standard telecom / datacom 19" rack with rails 62 on each side. The rails 62 can be C-channel rails (C being the shape of the rails). The box bracket 70 can be an all-metal structure for physically supporting the associated box 12, twinax connector 52, and power connector 64. There are no PCBs or electronics on the box slot bracket 70 (other than the connectors 52, 64), i.e., the box slot bracket 70 is passive. In one embodiment, the box slot bracket 70 includes a retractable pin 72 that locks the box slot bracket 70 in position in the rails 62. The twinax connector 52 is held in a fixed position at the rear of the box slot bracket 70. The box bracket 70 can move up and down in the rack 60 (also referred to as a frame), and the wiring 74 connected to the box bracket 70 includes a slack portion that allows the box slot bracket 70 to move vertically through multiple rack units (U). The wiring 74 is connected to the core box 14.

[0088] exist Figure 8 , an example wiring box 12 is shown placed in a wiring box bracket 70. The wiring box 12 slides horizontally (front to back) into the wiring box bracket 70. The main function of the wiring box bracket 70 is to provide a fixed location for the twinax connector 52 to allow insertion of the wiring box 12 without rear access. The cable length can be configured at the factory to accommodate the customer's specific configuration of the wiring box 12 size.

[0089] The universal bracket design of the circuit box slot bracket 70 can be applied in variations depending on the requirements of different circuit boxes 12 . Figure 8 Four example variations of line box slot brackets 70A, 70B, 70C, 70D are shown. Line box slot bracket 70A includes a duplex coaxial connector 52 and a power connector 64, and can be used to support single or double height line boxes 12. Line box slot bracket 70B includes two duplex coaxial connectors 52 and a power connector 64 located in the middle. Line box slot bracket 70B supports half-sized line boxes 12. Line box slot bracket 70C includes two duplex coaxial connectors 52 and two power connectors 64 to support line boxes 12 of greater height. Finally, line box slot bracket 70D does not include a duplex coaxial connector 52 but includes a power connector 64, for example, for optical boxes that do not require structural plane connections to core boxes 14, etc.

[0090] Fig. 9 is a perspective view of the circuit box slot bracket 70, and Fig.1062. is a bottom view of the circuit cassette slot bracket 70. The circuit cassette slot bracket 70 includes a retractable pin 72 that can include a spring loaded mechanism that snaps into a hole in the track 62 and has a "push to release" feature that avoids the need for side or rear access. For example, the latch 76 can be pulled to release the pin 72, allowing the circuit cassette slot bracket 70 to move up and down.

[0091] Variable pitch backplane virtual slots

[0092] By using the line box 12 with cable-connected backplane connections in the modular network element architecture 10, its physical flexibility can be used to create a slot bracket 70 populated with line boxes 12 whose positions can be moved within a range. This allows line box 12 modules of different panel heights to be stacked together in the chassis without wasting structural capacity.

[0093] The modular network element architecture 10 allows the customer to retain the ability to replace the line box 12 without having to go to the back of the chassis to disconnect / reconnect the structure cables. The cable sockets (connectors 52, 64) are attached to the slot bracket 70, and the slot bracket 70 can slide vertically on the chassis within the range provided by the additional slack in the backplane wiring (e.g., maximum 4U). A vertical physical guide mechanism (e.g., a C-shaped channel in the track 62) can provide a slide rail for the slot bracket 70 on the chassis and a hole for locking in various slot bracket positions based on 1U interval size. The slot bracket 70 has a pin 72 that can be pressed to move it up and down and then lock into the hole of the C channel, all while the backplane cables are still attached. The slot bracket 70 also provides a standard plug alignment mechanism to allow the line box 12 module to be inserted into the slot bracket 70. The slot bracket 70 itself does not need to be moved by the customer to provide value, and the factory configurability allows unique slot size configurations to suit specific customers. Customers can modify the slot position themselves based on the built-in cable slack.

[0094] The slot bracket 70 can have multiple locations for separate cable bundles. Not all cable bundles need to be assembled, which allows the circuit box 12 to obtain the amount of structural bandwidth it needs without wasting structural bandwidth like conventional PCB backplane systems.

[0095] Rack View

[0096] Fig.11 is a perspective view of the front and rear views of a rack 60 having a core box 14 , four line boxes 12 in associated line box bays 70 , and eight empty line box bays 70 . Fig.12 yes Fig.11 Front and rear views of the rack 60. Fig.13 It is a three-dimensional view of the core box 14.

[0097] The modular network element architecture 10 can be deployed as a centralized chassis in various configurations using a line box 12 and an optional core box 14, where cables are docked to the line box 12 in the same rack 60 or in a different rack. The core box 14 can include modules for switch modules 16, controllers 18, timing, etc., which can be linked via a midplane 50 in the core box 14.

[0098] The switch modules 16 can be connected to the circuit boxes 12 via electrical wiring or optical wiring at the rear of the rack 60 as described herein, and the front of the rack 60 is used for the customer-facing interface. As described herein, the rear cables are divided into bundles that match the spacing size of the structural modules. The control plane signals can be in the same cable bundle, or in a cable bundle that is physically separated from the structural cables. The control plane cable bundle includes a multiplexing protocol that transmits signals for ToD / 1pps / frequency, presence, status, primary ownership, interrupt, reset, location identification, Ethernet, and power.

[0099] The core box 14 can provide power to a subset of the control circuits on the line box 12. Different redundancy rates are used to support different functions, such as local Ethernet structure (N:1), timing distribution (1:1), chassis management (1:1), and MPLS control plane (N:1), and switching of one function does not force switching of another function, and each function can reside on a different module. The access panel 20 of the modular network element architecture 10 is centralized in the core box 14 and can serve all line boxes 12 in such a switch system. Advantageously, the cooling of the core box 14 and the line box 12 is independent of each other, and the airflow is significantly improved.

[0100] The modular network element architecture 10 includes physically separate devices (line box 12 and core box 14), but the modular network element architecture is managed as a single entity, for example, alarms, statistics, performance monitoring (PM) data, etc. are aggregated, and the modular network element architecture 10 is a single network element. When the core box 14 is omitted, the single line box 12 can virtualize the core box 14 management functions, such as control plane and structure management.

[0101] Ethernet structure

[0102] Various existing solutions hash Ethernet flows to specific paths that exist during the life cycle of the flow, such as link aggregation. In the Ethernet structure, when elephant flows cause hot spots and unbalance the structure, the slow control plane routes new flows away from the congested hot spots, but this is a slow process that causes congestion aliasing. Elephant flows are very large (total number of bytes) continuous flows on network links. Although there are not many elephant flows, they can occupy too large a proportion of the total bandwidth over a period of time. The conventional method is to disconnect the elephant flow at the natural gap in the elephant flow so that the link can be moved without causing confusion. Of course, unless the elephant flow already has a gap, the conventional method cannot disconnect the elephant flow.

[0103] In addition, conventional methods estimate path delays based on buffer fill levels, but this is not as accurate as the timestamp method using out-of-band timing distribution. Likewise, conventional methods do not use TDM to improve data flow in Ethernet structures. For this reason, a new approach is needed for the modular network element architecture 10 with respect to Ethernet structures.

[0104] In various embodiments, the Ethernet fabric in the modular network element architecture 10 can create gaps in large continuous elephant flows by artificially forcing transient buffering so that the gaps are no larger than the path delay difference between the current path and the new path. This allows a single flow to be spread over many links, thereby improving the utilization of the Ethernet fabric. The modular network element architecture 10 can also include adding post-buffering and shaping to the OTN segment to create evenly spaced mouse flows to improve the utilization of the Ethernet fabric (mouse flows are the opposite of elephant flows).

[0105] The modular network element architecture 10 uses out-of-band timing distribution coupled with TDM segmentation and reassembly (SAR) to measure fabric delays on all paths. The modular network element architecture 10 is able to spread TDM evenly on the paths to measure all paths through the fabric without introducing probe packets. Specifically, the TDM packet SAR header (sequence number) is used to estimate the relative path delay delta and feed it into the path selection algorithm. When the path delay needs to be most accurate, out-of-band congestion notification distribution is used across the cable-connected backplane to estimate path delay tolerance to high congestion. Out-of-band path selection broadcasts allow each line box 12 to establish a global path selection state to prevent fluctuations when flows move away from congested links.

[0106] TDM SAR is used to actively balance the data flows of the Ethernet fabric resulting in an unbalanced fabric. The path selection algorithm has unique rules for unicast, multicast and TDM flows. Together with the above, it is proposed to limit the top service category (CoS) to only TDM, shaped traffic, speeding up the fabric to keep latency within target ranges. Path fluctuations are prevented by communicating path selection status between line boxes 12 to prevent: multiple line boxes 12 from moving large flows to the same link, causing congestion, and moving all flows back, and repeating the cycle. Hashing is used to make a joint decision on which flows to move.

[0107] Fig.14 It is a block diagram of the communication between the core box 14 and the line box 12. Communication is carried out through the control plane using 10GbE or 100GbE links. The core box 14 maintains a system database of path status (such as congestion status, path reservation status, etc.). It is an event-driven publish / subscribe model. This allows the line box 12 to make path selection decisions based on the global system status in the modular network element architecture 10 and the scheduled path selection from other line boxes 12 rather than just the local state of the line box. Negotiated path selection based on global status prevents path fluctuations, whereby all line boxes 12 move large flows to an empty link, congest it, and move their flows away from the link again, etc. The structural congestion status and line box path selection status are transmitted out-of-band.

[0108] Fig.15 is a logical diagram of Ethernet paths with and without congestion through the modular network element architecture 10. In this example, there are three ingress line boxes 12A, 12B, 12C, a core box 14, and one egress line box 12D. The ingress line boxes 12A, 12B, 12C, and the egress line box 12D are all connected to a switch module 16 in the core box 14 at 100G. Fig.15 At the top, the inlet line boxes 12A and 12C provide flows to the first switch module 16, respectively, resulting in congestion on the link between the switch module 16 and the outlet line box 12D. Fig.15 At the bottom, the inlet line box 12C selects a new path, thereby alleviating congestion.

[0109] Fig.161 is a logical diagram of an Ethernet path through a modular network element architecture 10 showing multicast. In this example, there is one inlet line box 12 and two outlet line boxes 12, with a core box 14 located between the two. There are three link aggregation groups (LAGs) LAG1, LAG2, LAG3 between the inlet line box 12 and the switch module 16 in the core box 14. There are two LAGs, namely LAG4, LAG5, between the first switch module 16 and the outlet line box 12. TDM protection requires 1:N multicast (MC), where N≤4. For MC replication flows, the local fabric element (FE) performs a load balancing function for the line box 12, the core box 14, etc. One replication may encounter congestion, while another may not. The congested replication cannot be moved to a different fabric element because this would require MC to occur at the inlet line box 12 and would cause the fabric input to be blocked. If all members of LAG5 are congested, the pre-replication flow must be moved to a different fabric element where both replication paths are not congested. This decision is made by the inlet line box 12 based on the core box 14 congestion status.

[0110] Fig.17 is a logical diagram of Ethernet multipath data flow through the modular network element architecture 10. Fig.17 A packet on a 100GbE ingress port is shown, with a flow A of 40Gb / s and a flow B of 60Gb / s. Both flows A and B are elephant flows, which can lead to poor load balancing, thereby leading to poor fabric utilization (some links are congested, while others are underutilized). For each of flows A and B, there are interburst gaps and flowbursts, and in the modular network element architecture 10 there can be two paths 1 and 2, with delays of 500ns and 600ns, respectively. The modular network element architecture 10 can include sending different flow bursts on different paths to avoid congestion and underutilization. As described herein, a flow burst is a portion of a flow, such as an elephant flow, between burst gaps. Similarly, as described herein, each flow burst can take a different path in the structure.

[0111] Fig.18 is a logical diagram of TDM multipath data flow through the modular network element architecture 10. Note that Optical Data Unit Level 4 (ODU4) is not suitable for 100GbE, and even if it were, it would be an elephant flow that would unbalance the fabric. Fig.18As shown, the TDM rat streams are spread over all fabric links on six example paths. The fragments on different paths can arrive out of order, so they are given sequence numbers and buffered / reassembled in order. By breaking the ODU4 into many rat stream fragments, the bandwidth is evenly distributed over all fabric paths, which leads to good fabric utilization. The TDM fragments are time stamped, which allows continuous measurement of path delays on all paths.

[0112] Enforced gap on elephant flow

[0113] Fig.19 Here is a logical diagram of Ethernet flows. First, a single elephant flow is shown on a panel 100GbE port, i.e., there are no gaps between packets. The problem is that this flow typically wants to stay on a single fabric link to prevent reordering. The fabric may have 150Gb / s of available bandwidth, but that bandwidth may be spread across 15 fabric links so that a given link only has 10Gb / s of available bandwidth. The elephant flow doesn't fit anywhere and will cause congestion no matter which link it is on.

[0114] Next, the elephant flow is split into two bursts. The ingress line box 12 circuit (e.g., FPGA) temporarily buffers the elephant flow and introduces a forced gap. This gap allows the ingress Ethernet switch to send the first burst on link #1, the second burst on link #2, the third burst on link #3, and so on. This allows the elephant flow to be spread across the available bandwidth on many fabric links, thereby achieving higher fabric utilization.

[0115] The enforced gap is not a static value. The size of the gap is determined by the difference in delay between the previous path and the new candidate path. If the candidate path has a higher delay, then there is no need to introduce a gap. However, the path selection algorithm will usually try to find a lower-latency path when possible. In the case where the candidate path has a lower delay, the enforced gap size = old path delay - candidate path delay + margin. Note that this process does not increase the latency of the original flow relative to the latency of the original path.

[0116] Fig.19 The bottom two flows in the figure show fabric links #1, #2 handling different bursts created by forced gaps. Note that the forced gaps in the elephant flow do not waste bandwidth because the ingress Ethernet switch fills the gaps with other flows. In combined TDM / data applications, TDM generates many 512B mouse flows that can easily fill the gaps. Likewise, the elephant flow simply needs to be disconnected if there is no large enough and continuous bandwidth block on the fabric link or the elephant flow has no natural gaps. Therefore, the forced gaps in the elephant flow can depend on congestion, fabric continuous bandwidth, etc.

[0117] Path selection

[0118] By default, path selection can be static at startup or when a new line box 12 is added through a static mapping of ingress ports to egress ports. For line boxes 12, this approach spreads the ingress ports evenly across all available fabrics and fabric links, while for core boxes 14, this approach spreads all ingress ports evenly within each line box 12 trunk. Following this static configuration, depending on the traffic flow, congestion may occur, so the paths must be dynamically modified by the path selection algorithm.

[0119] The path selection algorithm is implemented in the modular network element architecture 10 and includes monitored metrics. Specifically, for each ingress flow, counters are maintained that measure the most recent burst gap and flow bandwidth. Similarly, measurements of path delay are maintained for each path (between line box 12, core box 14). Since TDM fragments are spread over all paths, their timestamps can be used to continuously estimate the delay on all paths. If TDM is not present, an internal delay measurement message (DMM) can therefore be generated. An exponential moving average can be used to smooth the delay estimate.

[0120] Again, the core box 14 provides out-of-band communication of the fabric congestion status to all ingress line boxes 12. Specifically, this exchange is not in-band because the congestion status packet is affected by the congestion that the congestion status packet is attempting to report. Likewise, the out-of-band communication of the routing status is broadcast by all line boxes 12.

[0121] Flow bursts define the path selection interval size. A flow burst is eligible to follow a different path from the previous flow burst if the burst interval of the flow burst is larger than the delay difference between the existing path (high latency) and the new path (low latency). This allows for dynamic movement of flows (rather than static hash-based load balancing).

[0122] For dynamic path selection and manual override, flows can be manually moved without introducing packet reordering. This is done by intentionally pausing the flow and allowing it to buffer until an artificial burst gap is created, which allows the flow to be moved automatically by the path selection algorithm or manually.

[0123] The line box 12 makes path selection based on the congestion estimate for the entire path. It can select a new path inside or outside the LAG trunk. The core box 14 makes path selection only within the LAG trunk. Unreachable paths are not used. This can be due to a failed link, missing structure, etc.

[0124] For unicast streams, whenever a new stream starts or there is a sufficiently large gap in stream bursts, a new path with minimum delay is selected. The delta in delay between the old and new paths must exceed a threshold for the stream to be moved to prevent unnecessary moves for small improvements.

[0125] The local fabric element on the line box 12 performs a load balancing function for each LAG for unicast and multicast. However, for multicast, all members of a given fabric LAG may be congested, so the pre-replicated stream must be moved to a different fabric element with an uncongested LAG. This decision is made by the ingress line box 12 based on the congestion status of the core box 14. Note that the replication stream is prevented from being on a different fabric element, because having the replication stream on a different fabric element would require MC at the ingress line box 12, which could result in fabric input blocking.

[0126] For TDM, the OTN Implementation Protocol based on the Packet Fabric Protocol (OFP) from the Optical Internetworking Forum (November 2011) can be used, the contents of which are incorporated by reference. Here, TDM is broken down into many small streams that require the SAR function to reassemble in the correct order. TDM streams can be moved around without regard for packet ordering, as the SAR will take care of correcting the order. The path selection algorithm identifies TDM and non-TDM streams; TDM streams are not constrained by the burst gap path selection requirement. In cases where the fabric does not identify the stream type, the SAR function can add post-buffering to create evenly spaced burst gaps for the TDM streams.

[0127] For path fluctuations, a given line box 12 sees that there is a structural congestion state, but it does not see what the other line boxes 12 are doing. Two line boxes 12 may simultaneously decide to move flows to an uncongested link, causing it to become congested. Then, two line boxes 12 may move their flows away, causing fluctuations. Each line box 12 broadcasts its path selection state to the other line boxes 12 (and core box 14) so ​​that all can see the global path table. For all flows on the congested link, the flow with the highest hash value is the only flow that has moved during the current congestion interval.

[0128] TDM over Ethernet

[0129] Fig. 201 is a block diagram of a modular network element architecture 10 showing TDM over Ethernet fabric. Specifically, there is an ingress line box 12A, a core box 14, and an egress line box 12B. The line boxes 12A, 12B include an OTN circuit 100 for OTN framing, serialization / serial-to-parallel conversion (SERDES), switching, mapping, etc. The OTN circuit 100 can support up to 50 μs of latency, but the Ethernet switches 30, 40 have much smaller latency variations. The OTN circuit 100 is connected to an Interlaken (ILKN) FPGA 102, which is connected to the Ethernet switch 30. The core box 14 can include a timing module 104, which is communicatively coupled to the OTN circuit 100 and the ILKN FPGA 102.

[0130] For TDM over Ethernet, the Ethernet switch 40 of the core box 14 can turn off switch functions such as MLAG, storm control, ingress access control list (ACL), port mirroring, etc., to bypass the pipeline block and provide cut-through forwarding. Path selection can be configured to reduce the utilization of links with TDM.

[0131] The timing module 104 provides a mechanism for transmitting timing information of optical data unit class k (ODUk) / optical data unit flex (ODUflex) client signals over a packet structure so that the ITU-T Recommendation G.8251 timing specifications are still met without reducing the maximum number of network elements allowed by the G.8251 assumed reference model. The timing module 104 transmits timing information of ODUk / ODUflex client signals over a packet structure for which the structural delay and delay variation are unknown. The timing module 104 can be used to compensate the packet structural delay to a configurable value with a resolution better than 5ns (maximum structural delay ≤ configured value ≤ 100μs). ITU-T Recommendation G.8251 "Control of Jitter and Wander in Optical Transport Networks (OTN)" (09 / 10) is incorporated herein by reference.

[0132] Line Box Self-Expansion System

[0133] Fig.21 Figure 1 is a diagram of a single line box 12 interconnected with other line boxes 12 and core boxes 14 in a self-expanding manner. Each line box 12 can include one or more fabric access processors (FAPs) and fabric elements (FEs) (available from Broadcom). The FAPs and FEs are circuits used for Ethernet switching. Each time a line box 12 is added, the number of FAP to FE links changes. Only a subset of cables are moved (in use). If the cable / connector granularity is 3 links, then 1, 2, 4, 6, 8, and 12 box systems can be supported by one cable.

[0134] Line Box Port

[0135] Fig. 22 1 is a front view of an example modular network element architecture 10 with different line boxes 12A, 12B, 12C and ports. For example, the line box 12A is a half-size module with an optical modem, for example, an optical box that supports high-speed optical transceivers for DWDM transmission. The line box 12B is a large double-height module that supports 96×SPF+ (Small Form Factor Plus) pluggable modules (10Gb / s). The line box 12C is a half-size module that supports 32×QSFP28 (Quad Small Form Factor) pluggable optical modules (4×28Gb / s). The line box 12 can support different oversubscription rates according to customer needs, have different ratios of SFP+ to QSFP28, can be configured with different ratios of ports with and without Media Access Control Security (MACSec) or OTN, can include half-width servers connected by x86 fabrics according to each customer need, etc. The line box 12 can be used with the core box 14 to provide redundancy, larger switches, timing distribution, etc.

[0136] Line Box Self-Expanding Grid

[0137] Fig.23 1 is a network diagram of line boxes 12 in a self-expanding grid without core boxes 14. Each line box 12 contains an Ethernet switch 30 that can perform two functions, namely fabric connectivity and aggregation of low-speed interfaces (e.g., 10Gb / s or 100Gb / s ports) into a 400GbE fabric. Fig.23 In FIG. 1 , any-to-any switch configuration is shown in dashed lines, and edge aggregation configuration is shown in solid lines. Likewise, each line box 12 has a processor 34 that can act as a virtual controller 18 for the entire system.

[0138] The modular network element architecture 10 has the flexibility and advantage of being expandable using only the line box 12 and / or the core box 14. Fig.21 As shown, the line box 12 can include a FAP and a FE. The core box 14 can also include multiple FEs. Note that this is similar to the structure used in existing chassis-based systems, such as FAPs on line modules and FEs on switch modules.

[0139] exist Fig.21 In the embodiment, the system can include only the line boxes 12, where the structure is distributed in each line box 12 instead of the centralized core box 14. This can serve various applications. Furthermore, the line box 12 includes a processor 34, which can instantiate the functions of the virtual controller module 18 and thus can manage the entire chassis. The line box 12 can communicate with other line boxes 12 to arbitrate primary ownership.

[0140] The master line box 12 is configured to distribute timing synchronization to all other line boxes 12, reflect presence information to all line boxes 12, etc. The master line box 12 can accept interrupts and status from all other line boxes 12. Similar to as described herein with respect to the core box 14, the master line box 12 can provide power to a subset of the control circuits in the other line boxes 12. The master line box 12 acts as a central control plane Ethernet switch, a central GPIO reflector, etc.

[0141] For example, each circuit box 12 can contain 2 FAP devices and 1 FE device. The FAP is connected to the local FE via a PCB or cable-connected link in the circuit box 12, and is connected to all other circuit boxes 12 FEs via a cable-connected backplane network. The unused SERDES on each FE can be used for other circuit boxes 12 that contain FAPs that implement self-expansion.

[0142] In a system with a single line box 12, the loopback cable can fully connect the FAP to the local FE within the same line box 12. The backplane cable granularity can support systems of 1, 2, 4, 6, 8 and / or 12 line box 12 sizes. Likewise, the 3 system supports the non-disruptive addition of new line boxes 12 to a self-expanding line box 12 system in service. The process of moving one cable at a time can be enforced by a locking cable assembly that is controlled by detecting an operable structural link so that a new cable cannot be removed until both ends of the previous cable are properly in place.

[0143] The core box 14 can also be incorporated into a self-expanding line box 12 system to increase the range of expansion, increase redundancy, or timing distribution. Thus, the modular network element architecture 10 includes the ability to mix self-expanding systems with a central structure.

[0144] Factory-configurable fabric connections optimize switches for any-to-any switching and aggregation topologies. Link detection through control of connector locking mechanisms enables in-service non-disruptive expansion. Cable granularity allows multiple expansion ranges with a minimum number of cables.

[0145] Slot Number

[0146] In a conventional chassis, the slot number is known, i.e., which numbered slot the module is plugged into. In the case of the modular network element architecture 10, the concept of "physical slot number" is replaced with a "virtual slot number" that can be associated with a numbered cable connector on the core box 14 through the middle plane 50. The physical and virtual slot numbers are not necessarily mapped 1:1, because the line box 12 may be 1U, 2U or 4U, and even though it can occupy a certain range of physical fixed slot positions in the rack, it is still considered a single virtual slot. The technician who needs to replace the "line box virtual slot 3" needs some way to identify the physical location of the line box 12 within the rack. This is achieved through a small display on each line box 12 that shows the virtual slot number.

[0147] It is usually recommended that the line boxes 12 are plugged into the cables sequentially, which results in a monotonous sequence of virtual slot numbers. However, in cases where this may not be the case, the line boxes 12 will have unordered virtual slot numbers. In these cases, a solution is proposed so that the technician does not need to search up and down the rack for a specific virtual slot number: the line box 12 is able to give the line box's virtual slot number based on its local altimeter sensor and continuous calibration of the altimeter sensor involving a known position within, for example, the core box 14.

[0148] Fig.24 1 is a front view of the modular network element architecture 10 showing the virtual slot number 150 and the position sensor 152. The virtual slot number 150 can be a display that provides a number or other unique identifier. The display is shown in the upper left of the panel, but one of ordinary skill in the art recognizes that the actual implementation can be anywhere. The position sensor 152 is used to automatically identify the location to assign the virtual slot number 150. In some embodiments, the position sensor 152 can be a physical device, while in other embodiments, the position sensor 152 can be a technology.

[0149] In one embodiment, a set of coded active transmitters can be attached to the rack 60 at predetermined locations. The line box 12 and the core box 14 can lock to the code and measure the received power level and time stamp the signal from each active transmitter. The line box 12 can pass its measured power and time stamp data to the core box 14 or line box 12, which uses a relative comparison of all the data to determine the relative position of each line box 12. The relative position is then communicated to each line box 12 through a data structure. The determination of the relative position between line boxes 12 is more accurate than the determination of the absolute position.

[0150] As described herein, the line box 12 is phase and frequency synchronized with the core box 14 via a dedicated point-to-point cable connection (sub-nanosecond absolute accuracy on the line box 12). This allows the received "Ping" to have an excellent timestamp, which is not available with ordinary Bluetooth beacons and GPS. Similarly, the active transmitter can be a directional antenna. The active transmitter can be based on radio frequency (RF), but some customers may be worried about RF, and the alternative is to use an ultrasonic sensor. Due to the longer arrival time of the reflected path, the system can filter out reflections. In the case of multiple transmitters, majority voting can help eliminate problematic paths. If the transmitter is located at the rear of the frame, it is easier to power the transmitter. In a preferred embodiment, the transmitter is located at the 8 corners of the core box 14 itself, so there is no need to mount the transmitter on the frame and power it.

[0151] In another embodiment, the rails 62 can include small connectors to determine shelf ID (SID) information from the wiring boxes 12 to locations on the rack 60. Electrical connections can be made via small USB-C patch cords from the side of each wiring box 12 into a vertical rail that connects to the core box 14 with the controller 18. Alternatively, this can be via a free space RF / optical connection.

[0152] Each circuit cassette slot carrier 70 can have a SID encoded via a dual in-line package (DIP) switch (or EPROM) configured with the slot number at the time of manufacture. When a circuit cassette 12 is inserted into a slot carrier 70, it can read the SID. The SID values ​​can be sparse so that a new slot carrier 70 can be optionally inserted between two existing carriers and assigned a new SID number without having to change an existing slot carrier 70. For example, the free numbers can be accumulated by 5, 10, etc.

[0153] In yet another embodiment, there can be some printed pattern on the track 62 that changes continuously over the entire length of the track 62, such as a bar code, quick response (QR) code, etc. The wiring box 12 can scan the pattern using an optical or magnetic sensor. The pattern can be a printed label or pre-etched into the metal.

[0154] In yet another embodiment, the position sensor 152 can be a micro-electromechanical system (MEMS) barometer pressure sensor. These can be accurate to within + / -5cm. In the same way that anemometers can allow weather stations to measure atmospheric pressure in strong winds and gusty conditions, anemometers can be used to calibrate pressure sensors in telecommunication rooms, data centers, and the like. Compact anemometers can be based on measuring temperature changes in fixed heat output devices, for example, lower temperatures mean higher airspeeds. Pitot tubes can be used to eliminate some sources of airflow that may affect pressure measurements. The pressure at an altitude of 1m is 101312.99Pa, and the pressure at an altitude of 1.07m is 101312.12Pa, so the difference is 0.87Pa, which is detectable by the current level of technology.

[0155] Upgradable backplane and cross-technology evolution

[0156] The field replaceable unit (FRU) on the core box 14 has a connector that does not change. The FRU connector can directly mate with another card in an ortho-direct "backplane-less" system. The same FRU connector can be connected to a standard backplane. The same FRU connector can be connected to a midplane that provides a socket to a cable plug, thereby allowing a backplane for cable connection. The cable can be electrical or optical.

[0157] If the cable is optical, an active circuit "line card" can handle the electrical-optical (EO)-optical-optical (OE) conversion as part of the cable. To support this, power and ground outputs are provided on the connector to power the line card. This is not typically done on an existing backplane. As an alternative to locating the EO-OE conversion as part of the line card cable, the EO-OE can be performed on a powered "patch" board.

[0158] In summary, this enables the FRU or circuit box 12 to work in different types of chassis: ortho-direct, backplane, midplane, cabled electrical backplane, and optical cabled backplane. Thus, the circuit box 12 system is able to work in all of these systems and even evolve from one to another in the field. Using a cabled backplane (electrical twinax or optical), non-disruptive backplane field upgradeability is supported.

[0159] To maintain compatibility and upgradeability, signals are routed from connector A columns to connector B columns; there is no crossover between columns. Row-to-row routing uses "long to short" to optimize pair-to-pair skew. The core box 14 provides output power on the backplane connector to power the backplane line plug-ins. The granularity of the cable bundle enables only a subset of the links to be unplugged at a time. Power and ground outputs are provided on the connectors with appropriate management of return current for low electromagnetic interference (EMI).

[0160] Variable bandwidth slots

[0161] As described herein, the modular network element architecture 10 provides flexibility in expansion. There are two types of system sizes, namely, configurable sizes at the factory and configurable sizes at the customer's location. For example, in many cases, customers will simply order "small," "medium," or "large" systems that have enough spare cables to add new line boxes 12 as needed. In addition to the number of cables, cable granularity is also important because it allows some line boxes 12 to obtain more bandwidth than other line boxes. For example, a Jericho-based line box 12 (Jericho is a switching application-specific integrated circuit (ASIC) provided by Broadcom) can use 36 links, while a Jericho2-based line box 12 can use 72 links, even though the physical size of both line boxes is 1U. If the cable bundle granularity is 12 links per bundle, then both types of line boxes 12 can be accommodated in a single system. By effectively allocating structural bandwidth, expansion can be achieved using the same cable granularity.

[0162] The system can support 3 Jericho2 line boxes 12, 6 Jericho line boxes 12, or 12 Arad line boxes 12 (Arad is a switch-specific integrated circuit provided by Broadcom). This is not possible in existing systems. For example, a 10-slot chassis system will support up to 10 of a given card, regardless of whether the card has Arad, Jericho, or Jericho2. The cable granularity is also determined by the number of structures that exist. If there are 3 structures, each structure may have 12 cables for a total of 36 links per virtual slot. The Jericho2 line box 12 may use two virtual slots for a total of 72 cables. All of this can be determined at manufacturing time to create flexibility that is not possible in existing backplane systems.

[0163] Multi-shelf expansion

[0164] The system can be expanded by adding core box 14 structures and establishing direct connections between them. In the chassis-based approach, this is called "multi-shelf" and involves creating a multi-level structure. This is feasible in the case of utilizing a modular network element architecture 10. However, in the case of utilizing a modular network element architecture 10, there is only one possibility, that is, adding a core box 14 to a system that already has a core box 14 and maintaining a single-level structure. In essence, the second core box 14 doubles the number of structures. Therefore, if the core box 14 has 3 structural elements, the two core boxes 14 have a total of 6 FEs, and the system size can be doubled. The challenge is how to redistribute the links of the cable connection from the existing line box 12 that is only connected to the first core box 14. Therefore, if 12 links go to each FE in the core box 14, 6 of the 12 links of each FE will need to be moved to the second core box 14. Part of the design is to select these cable granularities when the cables are manufactured to allow the required expansion level to be achieved at the customer site. However, this is also a way to build systems of different sizes at the time of manufacturing. Existing approaches can require designing new backplanes, power supplies, fabric cards, in order to make systems of different sizes. The modular network element architecture 10 enables this expansion through wiring. In "multi-shelf" expansion, the key point is that the links are dual-purpose - they can connect core box 14 to line box 12 or core box 14 to core box 14. This cannot be done in existing systems.

[0165] Non-disruptive upgrades

[0166] Consider a system with 3 fabrics, with 12 cable links connected from line box 12 to each fabric on core box 14. If the fabrics are 2+1 protected, then 2 fabrics are sufficient to pass the full rate. Therefore, each 12-link cable bundle can be replaced one bundle at a time, while the other two cable-connected bundles carry all traffic with their 24 links. Some deployments may not have a 2+1 redundancy rate; they may require a 30+6 link redundancy rate. In this case, a cable bundle granularity of 6 links may be more conducive to non-disruptive upgrades. This granularity of backplane upgrades is not feasible in existing systems.

[0167] In the case of a cable connection, the cable latch can include a switch that sends a signal to the software that is used to move the cable traffic so that when the cable is unplugged, the in-flight structure data is not lost. This signal can be sent to both ends of the cable bundle so that both TX and RX data can be moved in advance.

[0168] Circuit Box

[0169] Fig.25is a block diagram of an example implementation of the line box 12. As described herein, the line box 12 is an integrated design that provides flexibility, optimal panel area and board volume, optimal framing as needed, optimal architecture, power supply, and interconnection. In this example, the line box 12 is 1U in height and has 24 panel ports 200. The panel ports 200 can be connected to a framing circuit 202 that can provide OTN, no framing, MACSec, coherent digital signal processing (DSP), forward error correction (FEC), etc. The line box 12 can include a power supply 204 (100W, 400W, 1200W, etc.) required for the application and a fan 206 for cooling. The line box 12 can also include a processor 34, a network processing unit (NPU) 208, and a structural circuit 210. The structural circuit 210 can include various FAPs, FEs, etc. required for the application.

[0170] The line box 12 approach can include some of the following options. For optical devices, the line box 12 can include, for example, 1Gb / s SFP ports, 10Gb / s SFP ports, 100Gb / s CFP (100G Form Factor Pluggable), 100Gb / s CFP2, 100Gb / s CFP4, 100Gb / s QSFP28 ports, coherent pluggable, CFP2 analog coherent optics (ACO), etc. The fabric circuit 210 can include a variety of different implementations, such as Petra-B 100G, Arad 200G, Jericho 700G, Qumran 800G, Jericho+800G, Jericho22400G, Xpliant, Intel FM (Fulcrum), etc. The framing circuit 202 can include nothing, include OTN, include MACSec, etc. The modular network element architecture 10 can have various sizes, power supplies / slots, structure densities, number of structures, and different backplane connections (eg, twinax, PCB, and / or active optical cable).

[0171] Core Box / Dual Coaxial Wiring

[0172] Fig.26 It is a framework diagram of the front view and the rear view of another implementation of the core box 14. Fig. 27 FIG. 5 is a diagram of a twinax cable 300 and a connector 52 . Fig.26The core box 14 in the example includes three switch modules 16 and two controller modules 18. The rear view of the core box 14 shows a twinax connector area 54, which supports 16 cables for each module 16. Here, the core box 14 supports 1 or 2 cables from the twinax connector area 54 to each line box 12. Each cable can have 12 links from each switch module 16 and a control link from the controller module 18. The twinax cable 300 can include 72 pairs (36 links), and the core box 14 of this example can support up to 16 cables. The twinax connector area 54 is configured to divide the 72 pairs (36 links) into 24 pairs (12 links) to each switch module 18.

[0173] The process of deploying modular network elements

[0174] Fig.28 is a flow chart of a process 300 for deploying a modular network element. The process 300 includes: connecting a line box slot bracket to a rack (step 302); providing a line box in the line box slot bracket, wherein the line box is a hybrid between a rack-mounted module and a line module inserted into a chassis, and wherein the line box slot bracket connects the line box to power connectors and data connectors (step 304); and providing a core chassis (core box 14), the core chassis including one or more switch modules, one or more controller modules, and a set of connectors at the rear that are wired to power connectors and data connectors on the line box slot bracket (step 306). The process 300 can also include providing a second line box to the second line box slot bracket in response to increasing the capacity of the modular network element (step 308). The second line box can be added to the core chassis in service.

[0175] In another embodiment, a modular network element includes: one or more line boxes 12, which are a hybrid between a rack-mounted module and a line module inserted into a chassis; one or more line box brackets 70, which are rack-mountable and configured to selectively accommodate one or more line boxes 12 and provide them with power connectors 52 and data connectors 64; and a core chassis (core box 14), which includes one or more switch modules 16, one or more controller modules 18, and a group of connectors 54 located at the rear, which are used for cable connection to the power connectors 52 and data connectors 64 on the one or more line box brackets 70. One or more line boxes 12 can include a plurality of panel ports 200, which are connected to a fabric circuit 210, which is communicatively coupled to one or more switch modules 16 via wiring; and one or more switch modules 16 are connected to each other and one or more controller modules 18 via a midplane 50 or a cable.

[0176] The wiring can include one of: electrical wiring with multiple pairs of twinax cables, and optical wiring. Each cable between the line box 12 and the core chassis can include multiple bundles, each bundle has multiple links, and the number of multiple bundles is based on the number of one or more switch modules 16. A chassis management protocol is implemented between one or more line boxes 12 and the core chassis, and the one or more line boxes 12 and the core chassis are managed as a single entity. The chassis management protocol implements timing synchronization, presence, status, ownership, interruption, reset, power, and location identification between one or more line boxes 12 and the core chassis. One or more controller modules 18 provide power and connections to field programmable gate arrays (FPGAs) or other circuits on each line box to establish a chassis management protocol to turn the line box power on and off.

[0177] The core chassis can include an access panel 20 that supports the core chassis and each of the one or more line boxes 12. Each of the one or more line boxes 12 includes a display 150 that provides a virtual slot number based on a relative position in the rack. The one or more line boxes 12 can include a plurality of line boxes 12, wherein at least two of the line boxes 12 have different spacings. The one or more line boxes 12 can include a plurality of line boxes 12, wherein at least two of the line boxes 12 have different numbers of panel ports 200 and one or more of different types of pluggable optical devices. The one or more line boxes 12 each include an Ethernet switch 30 that is connected to another Ethernet switch 40 in the one or more switch modules 16. The one or more line box brackets 70 can include a retractable pin 72 having a spring-loaded mechanism that snaps into a hole in a rail 62 on the rack 60 and has a push-to-disengage mechanism so that the rack can only be approached from the front. One or more of the cassette trays 70 have slack in the wiring to the core rack, allowing movement up and down the rack 60.

[0178] In another embodiment, the modular network element includes one or more line boxes 12, which are a hybrid between a rack-mounted module and a line module inserted into a chassis, each of the one or more line boxes 12 includes: a plurality of panel ports 200 connected to a fabric circuit 210 and a processor 34; and one or more line box brackets 70, which are rack-mountable and configured to selectively accommodate one or more line boxes 12 and provide them with power connectors 52 and data connectors 64. The one or more line boxes 12 can include a plurality of line boxes 12, which together form a mesh via data connectors 64, and wherein one of the plurality of line boxes is designated as a master line box for implementing a virtual controller of the modular network element via processor 34. A core chassis (core box 14) can be added in service to increase the capacity of the modular network element, wherein the core chassis includes one or more switch modules, one or more controller modules, and a set of connectors at the rear for wiring to power connectors and data connectors on one or more line box brackets.

[0179] Ethernet path selection process

[0180] Fig.293 is a flow chart of a process 320 of Ethernet path selection in a modular network element. The process 320 is implemented in a modular network element including one or more ingress line modules, a plurality of switch modules in a chassis, and one or more egress line modules, wherein the one or more ingress line modules and the one or more egress line modules are located separately from the chassis and connected to the chassis via wiring. Note that the line modules can be line boxes 12 and the chassis can be core boxes 14, although other embodiments are also contemplated. The process 320 includes: distributing one or more Ethernet flows from one or more ingress line modules to one or more egress line modules via a plurality of switch modules (step 322); receiving fabric status information from the chassis out-of-band at the one or more ingress line modules via wiring (step 324); and selecting a new path for the Ethernet flow by the ingress line module in response to congestion on the path through the plurality of switch modules (step 326).

[0181] The process 320 can also include, before switching the Ethernet flow to the new path, creating a gap in the Ethernet flow by buffering so that the gap is no larger than the path delay difference between the current path and the new path (step 328). The size of the gap can be set to the delay of the current path minus the delay of the new path plus a margin. The process 320 can also include distributing the time division multiplexed (TDM) flow by segmenting the TDM flow and sending it on multiple paths through multiple switch modules (step 330).

[0182] The process 320 can also include allocating timing from an out-of-band core chassis via wiring, where timing is determined at the core chassis using time division multiplexing (TDM) streams spread across multiple switch modules (step 332). The core chassis can estimate relative path delay differences using TDM packet headers. The process 320 can also include broadcasting the path selection to all other line modules and the core chassis through each ingress line module to maintain a global path table, respectively (step 334). The Ethernet flow can include an elephant flow, and if the burst gap of the elephant flow is greater than the delay difference between the current path and the new path, each flow burst of the elephant flow can take a different path.

[0183] In another embodiment, a modular network element configured for Ethernet path selection includes: one or more ingress line modules (line boxes 12); one or more switch modules 16 in a chassis (core box 14); and one or more egress line modules (line boxes 12), wherein the one or more ingress line modules and the one or more egress line modules are located separately from the chassis and connected to the chassis via wiring, wherein one or more Ethernet flows are distributed from the one or more ingress line modules to the one or more egress line modules via multiple switch modules, wherein the one or more ingress line modules receive fabric status information from an out-of-band core chassis via wiring, and wherein, in response to congestion on the path through the multiple switch modules, the ingress line modules select a new path for the Ethernet flow.

[0184] Before switching the Ethernet flow to a new path, a gap can be created in the Ethernet flow by buffering so that the gap is no larger than the path delay difference between the current path and the new path. The size of the gap can be set to the delay of the current path minus the delay of the new path plus the margin. Time division multiplexing (TDM) flows can be segmented and sent on multiple paths through multiple switch modules. Timing can be allocated from an out-of-band core chassis through wiring, where the timing is determined at the core chassis using time division multiplexing (TDM) flows allocated on multiple switch modules. The core chassis can use the TDM packet header to estimate the relative path delay difference. Each ingress line module can broadcast the path selection to all other line modules and core chassis to maintain the global path table respectively. The Ethernet flow can include an elephant flow, and if the burst gap of the elephant flow is larger than the delay difference between the current path and the new path, each flow burst of the elephant flow can take a different path.

[0185] In another embodiment, a core chassis (core box 14) in a modular network element configured for Ethernet path selection includes: one or more switch modules 16; one or more controllers 18; a middle plane 50, which is used for communication between the one or more switch modules and the one or more controllers; and wiring, which is used for communication to one or more inlet line modules and one or more egress line modules, wherein the one or more inlet line modules and the one or more egress line modules are located separately from the chassis and connected to the chassis via wiring, wherein one or more Ethernet flows are distributed from the one or more inlet line modules to the one or more egress line modules via multiple switch modules, wherein the one or more inlet line modules receive fabric status information from one or more out-of-band controllers via wiring, and wherein, in response to congestion on the path through the multiple switch modules, the inlet line module selects a new path for the Ethernet flow.

[0186] Time division multiplexed (TDM) streams can be segmented and sent over multiple paths through multiple switch modules. Timing can be assigned from an out-of-band core chassis via wiring, where timing is determined at the core chassis using time division multiplexed (TDM) streams spread over multiple switch modules. The core chassis can estimate relative path delay differences using TDM packet headers.

[0187] The process of managing modular network elements

[0188] Fig.30 is a flow chart of a process 340 for managing a modular network element. The process 340 is executed in a modular network element, the modular network element including a plurality of line modules and zero or more switch modules in a chassis, wherein the plurality of line modules are located separately from the chassis and connected to the chassis and / or to each other via wiring. The process 340 includes: operating a management plane between the plurality of line modules and the zero or more switch modules via one or more dedicated links in the wiring (step 342); managing the plurality of line modules and the zero or more switch modules as a single network element over the management plane using a chassis management protocol (step 344); and designating one of the following to be used as a primary virtual processor for the chassis management protocol: a controller in the chassis and a processor in one of the plurality of line modules (step 346).

[0189] The process 340 can also include providing a virtual slot number for each of the plurality of line modules, wherein the virtual slot number is used to distinguish the plurality of line modules (step 348). The virtual slot number can be automatically assigned based on a position sensor on each of the plurality of line modules or by a chassis that detects the position of each of the plurality of line modules. The management plane can operate through a universal serial bus (USB) pin and a 10 Gigabit Ethernet alternate mode that uses a subset of the USB pins. The USB can be used to bridge multiple protocols in the management plane and determine the presence of the module, and wherein the 10 Gigabit Ethernet is used for a precision time protocol. The USB can be used for power, presence, primary arbitration, bidirectional status, health monitoring, configuration, data path initialization, software download, interrupt, reset, LED control, I2C bridging, universal asynchronous receiver transmitter (UART) bridging, SPI bridging, pulse width modulation (PWM) bridging, general purpose input / output (GPIO) bridging, and PCIe bridging.

[0190] The process 340 can also include communicating between a controller in the chassis and a field programmable gate array (FPGA) or circuit on the line module to establish a chassis management protocol on the line module (step 350). The process 340 can also include selecting a candidate clock for synchronization and timing distribution in the chassis management protocol (step 352). The process 340 can also include continuously measuring delays on the wire to adjust the phase of the timing distribution (step 354). The management plane can operate on the wire out-of-band from the data plane.

[0191] In another embodiment, a modular network element managed as a single entity includes a plurality of line modules (line boxes 12); and zero or more switch modules 16 in a chassis (optionally a core box 14), wherein the plurality of line modules are positioned separately from the chassis and from each other, and are connected to the chassis and / or to each other via wiring, wherein a management plane is operated between the plurality of line modules and the zero or more switch modules via one or more dedicated links in the wiring, wherein the plurality of line modules and the zero or more switch modules are managed as a single network element on the management plane using a chassis management protocol, and one of the following is designated to serve as a primary virtual controller for the chassis management protocol: a controller in the chassis and a processor in one of the plurality of line modules.

[0192] A virtual slot number can be provided for each of the plurality of line modules, wherein the virtual slot number is used to distinguish the plurality of line modules. The virtual slot number can be automatically assigned based on a position sensor on each of the plurality of line modules or by a chassis detecting a position of each of the plurality of line modules. The management plane can operate over a universal serial bus (USB) pin and a 10 Gigabit Ethernet alternate mode using a subset of the USB pins. The USB can be used to bridge multiple protocols in the management plane and determine the presence of the module, and wherein the 10 Gigabit Ethernet is used for a precision time protocol.

[0193] The controller in the chassis can communicate with the field programmable gate array (FPGA) or circuit on the line module to establish a chassis management protocol on the line module. Candidate clocks can be selected for synchronization and timing distribution in the chassis management protocol. Delays on the wire can be continuously measured to adjust the phase of the timing distribution. The management plane can operate on the wire out-of-band from the data plane.

[0194] In another embodiment, a device configured to manage modular network elements includes a processor and a memory storing instructions that, when executed, cause the processor to operate a management plane between multiple line modules and zero or more switch modules via one or more dedicated links in wiring between the multiple line modules and zero or more switch modules; and manage the multiple line modules and zero or more switch modules as a single network element on the management plane using a chassis management protocol, wherein the device is designated as a primary device in the chassis management protocol and the device is physically separated from at least one of the multiple line modules and the zero or more switch modules.

[0195] It will be understood that some embodiments described herein may include: one or more general or special processors ("one or more processors"), such as microprocessors; central processing units (CPUs); digital signal processors (DSPs); custom processors, such as network processors (NPs) or network processing units (NPUs), graphics processing units (GPUs), etc.; field programmable gate arrays (FPGAs), etc.; and similar devices with unique stored program instructions (including software and firmware) that are used to control them in combination with certain non-processor circuits to implement some, most, or all of the functions of the methods and / or systems described herein. Alternatively, some or all of the functions may be implemented by a state machine without stored program instructions, or in one or more application-specific integrated circuits (ASICs), where each function or a combination of some specific functions is implemented as a custom logic or circuit. Of course, a combination of the above methods may be used. For some embodiments described herein, the corresponding devices in the hardware and, optionally with software, firmware, and combinations thereof, can be referred to as "configured or adapted circuits", "configured or adapted logic", etc., to perform a set of operations, steps, methods, processes, algorithms, functions, techniques, etc. as described herein for various embodiments on digital and / or analog signals.

[0196] In addition, some embodiments may include a non-transitory computer-readable storage medium having computer-readable code stored thereon for programming a computer, server, device, apparatus, processor, circuit, etc., each of which may include a processor that performs the functions as described and claimed herein. Examples of such computer-readable storage media include, but are not limited to, hard disks, optical storage devices, magnetic storage devices, ROM (read-only memory), PROM (programmable read-only memory), EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), flash memory, etc. When stored in a non-transitory computer-readable medium, software can include instructions executable by a processor or apparatus (e.g., any type of programmable circuit or logic) that, in response to such execution, cause the processor or apparatus to perform a set of operations, steps, methods, processes, algorithms, functions, techniques, etc. as described herein for various embodiments.

[0197] Although the present disclosure has been illustrated and described herein with reference to preferred embodiments and specific embodiments thereof, it is apparent to one of ordinary skill in the art that other embodiments and examples may perform similar functions and / or achieve similar results. All such equivalent embodiments and examples are within the spirit and scope of the present disclosure, are contemplated thereto, and are intended to be covered by the appended claims.

Claims

1. A modular network element architecture (10), comprising: a plurality of modules (12, 14), each of the plurality of modules comprising a plurality of panel ports (200) connected to a fabric circuit (210), the fabric circuit being configured to perform Ethernet switching; and A cable-connected backplane provides connections between the plurality of modules (12, 14), the cable-connected backplane comprising each module (12, 14) of the plurality of modules (12, 14), each module (12, 14) comprising a set of the plurality of panel ports (200) interconnected with other modules (12, 14) of the plurality of modules (12, 14).

2. The modular network element architecture (10) according to claim 1, wherein: The plurality of modules (12, 14) include a plurality of line modules (12) and at least one core module (14), wherein the at least one core module (14) is configured to be interconnected with the plurality of line modules (12).

3. The modular network element architecture (10) according to claim 2, wherein: The at least one core module (14) is taller than the plurality of line modules (12).

4. The modular network element architecture (10) according to claim 2, wherein: The at least one core module includes a chassis supporting a plurality of switch modules (16).

5. The modular network element architecture (10) according to claim 1, wherein: The plurality of modules (12, 14) include a plurality of line modules interconnected to one another in a grid.

6. The modular network element architecture (10) according to claim 1, wherein: The plurality of modules (12, 14) are rack mounted and at least two of the plurality of modules comprise different sizes.

7. The modular network element architecture (10) according to claim 1, wherein: The plurality of panel ports (200) support pluggable modules that support optical or electrical connections.

8. The modular network element architecture (10) according to claim 1, wherein: The plurality of panel ports (200) include twinax connections.

9. The modular network element architecture (10) according to claim 1, wherein: The plurality of modules (12, 14) are physically separated from each other and managed as a single network element.

10. The modular network element architecture (10) according to claim 9, wherein: A chassis management protocol is implemented between the plurality of modules (12, 14).

11. The modular network element architecture (10) according to claim 10, wherein: The chassis management protocol implements timing synchronization, presence, status, ownership, interrupt, reset, power and location identification.

12. The modular network element architecture (10) according to claim 1, wherein: The modular network element architecture (10) is expanded by adding a new module (12, 14) to the plurality of modules (12, 14) and modifying the backplane of the cable connections containing the new module (12, 14).

13. The modular network element architecture (10) according to claim 1, wherein: The plurality of modules (12, 14) include a plurality of line modules (12) interconnected to each other in a grid, and wherein the modular network element (10) is expanded by adding a new core module (14) between the plurality of line modules (12, 14) and modifying a backplane including the cable connections of the new core module (14) between the plurality of line modules (12).

14. The modular network element architecture (10) according to claim 1, wherein: The cable-connected backplane includes a link aggregation group (LAG).

15. The modular network element architecture (10) according to claim 1, wherein: The plurality of modules (12, 14) include a display (150) that provides slot numbers based on relative position in a rack.

16. The modular network element architecture (10) according to claim 1, wherein: The plurality of modules (12, 14) includes at least two modules having different pitches.

17. The modular network element architecture (10) according to claim 1, wherein: The plurality of modules (12, 14) includes at least two modules having different numbers of panel ports (200) and one or more of different types of pluggable optical devices.

18. The modular network element architecture (10) according to claim 1, wherein: The plurality of modules (12, 14) include a plurality of line modules (12) and at least one server module for computing or storage resources.

19. The modular network element architecture (10) according to claim 1, wherein: The plurality of panel ports (200) include at least one dense wavelength division multiplexing (DWDM) interface.

Images