Network architecture with harmonic connections

Abstract

A computer network organized in a logical grid having rows and columns can include network nodes joined according to harmonics. Each network node can be joined to network nodes in the same row using a set of horizontal strands according to a set of horizontal harmonics. Each horizontal harmonic specifies the node distance along the row between adjacent connection points on the corresponding horizontal strand. Each network node can be joined to network nodes in the same column using a set of horizontal strands according to a set of horizontal harmonics. Each vertical harmonic specifies the node distance along the column between adjacent connection points on the corresponding vertical strand.

Description

[Technical Field]

[0001] Embedding by reference The claims are filed concurrently with this specification as part of this application. Each application identified in the claims of the concurrent application, for which this application claims benefit or priority, is incorporated herein by reference in its entirety for any purpose. [Background technology]

[0002] Server computers, such as those supporting cloud computing services, are typically maintained in facilities called data centers. Small data centers may occupy a room or floor of a building, while large data centers may occupy several floors or even an entire building. A typical data center may house thousands of servers communicating with each other over a network. The amount of computing work required from data centers has increased dramatically to support computationally intensive applications such as large-scale machine learning models. As a result, data centers are expanding in size and number to meet this growing workload demand.

[0003] Various embodiments relating to this disclosure can be described with reference to the drawings. [Brief explanation of the drawing]

[0004] [Figure 1] An example of a strand connecting network nodes (e.g., routing devices) according to a particular aspect of this disclosure is shown. [Figure 2A] An example of a strand having reconfigurable channels connecting network nodes, according to a particular aspect of this disclosure, is shown. [Figure 2B] An example of reconstructing the channel of the strand in Figure 2A according to a particular aspect of this disclosure is shown. [Figure 3] An example of a harmonic connection between network nodes according to a particular aspect of this disclosure is shown. [Figure 4A]An example of a network node in a data center, according to a particular aspect of this disclosure, is shown. [Figure 4B] An example of harmonic connectivity between network nodes within a data center, according to a particular aspect of this disclosure, is shown. [Figure 5] This document provides an example of routing traffic between a source and a destination, as described in a particular aspect of this disclosure. [Figure 6] A first exemplary embodiment of the strand according to a particular aspect of this disclosure is shown. [Figure 7] A second exemplary embodiment of the strand, according to a particular aspect of this disclosure, is shown. [Figure 8] A third exemplary embodiment of the strand, according to a particular aspect of this disclosure, is shown. [Figure 9] A fourth exemplary embodiment of the strand, according to a particular aspect of this disclosure, is shown. [Figure 10] A fifth exemplary embodiment of the strand according to a particular aspect of this disclosure is shown. [Figure 11] A flowchart illustrating an example of a process for routing traffic within a network fabric according to a particular aspect of this disclosure is shown. [Figure 12] This diagram shows an example of a process for performing communication within a network fabric according to a particular aspect of this disclosure. [Figure 13] An example of a network device according to a particular aspect of this disclosure is shown. [Modes for carrying out the invention]

[0005] To support data center expansion, the network infrastructure interconnecting the data centers needs to scale to the number of server computers. Multi-layer network fabrics, such as fat tree topologies or other variations of cross-topologies, are common due to their flexibility and non-blocking nature. However, such topologies may not scale linearly with the number of server racks because spine switches and an intermediate switching layer are added between top-of-rack switches and spine switches. Excess equipment also increases power consumption. This makes scaling cross-topologies beyond mega-data centers expensive in terms of the need for additional network devices and overall power consumption.

[0006] The technology disclosed herein provides a network architecture for interconnecting network nodes in a computer network using a single connectivity layer. For example, a computer network implemented in a data center may include multiple network nodes organized as a logical grid. Each network node may be associated with a server rack, and each logical column of the network node may correspond to a passage or other grouping of the server rack. Each network node may be implemented, for example, using routing devices coupled to servers in the server rack. The network nodes may be interconnected using strands (e.g., small optical networks) implemented using multipoint optical techniques to provide optical paths between the network nodes.

[0007] For example, each network node may be connected to network nodes in the same column (e.g., on the same path) using a set of vertical strands, and to network nodes in the same row using a set of horizontal strands. Each strand (e.g., horizontal strand, vertical strand) can connect up to the maximum allowed number of network nodes per strand (e.g., at least four or five or more network nodes per strand). The maximum allowed number of network nodes per strand may depend on the number of optical channels that the strand can support (e.g., the number of physical paths or wavelength-dependent paths). The number of strands connected to each network node may depend on the number of fabric ports (network-facing ports) available on the network node. For example, a routing device with 16 fabric ports can be connected to eight horizontal strands and eight vertical strands.

[0008] Each horizontal strand connects network nodes in the same row according to a horizontal harmonic that specifies the distance between adjacent connection points on the horizontal strand. The horizontal harmonic distance is given in terms of the number of network nodes along the row between adjacent connection points. Each horizontal harmonic can be different from other horizontal harmonics in the set of horizontal strands. Similarly, each vertical strand connects network nodes along the same column according to a vertical harmonic that specifies the distance between adjacent connection points on the vertical strand. The vertical harmonic distance is given in terms of the number of network nodes along the column between adjacent connection points. Each vertical harmonic can be different from other vertical harmonics in the set of vertical strands connected to a network node. The distance specified by a harmonic can also be called the node distance.

[0009] By configuring harmonics based on the number of network nodes in the rows and columns of a logical grid and leveraging the multipoint nature of strands, any two server racks can typically be reached within three switching hops. Congestion can also be avoided by distributing the network load across multiple paths. Single-connection layer topologies also reduce the number of active network devices and total power consumption compared to cross-topologies. Thus, as data centers continue to expand and scale up, the network architectures with harmonic connectivity disclosed herein can scale linearly in terms of the number of server racks.

[0010] Various embodiments are described in the following description. For explanatory purposes, specific configurations and details have been revealed to provide a complete understanding of the embodiments. However, it will be apparent to those skilled in the art that embodiments may be carried out without specific details. Furthermore, well-known functions may be omitted or simplified so as not to obscure the embodiments described.

[0011] FIG. 1 shows a conceptual diagram of an example of a strand connecting network nodes. A strand can refer to an optical path connecting fabric ports of network nodes within a network. In FIG. 1, four network nodes (network node N1 102-1, network node N2 102-2, network node N3 102-3, and network node N4 102-4) can be connected onto strand 150 using multi-point optical technology. Each network node can be implemented using a routing device such as, for example, a router or a switch, and can be coupled to a server rack. Each network node in FIG. 1 is shown with one fabric port, but may include any number of server-facing ports (which may be referred to as server ports) and any number of fabric ports. A server port is a port that connects to one or more servers and / or one or more other networks (e.g., another data center, an external network, etc.). A fabric port is a port that connects to a network fabric. Each fabric port implements an optical ingress / egress path from a network node to other network nodes within the network fabric.

[0012] The connection between two fabric ports along a strand can be referred to as a link, and each link may be implemented as a symmetric pair for ingress / egress. A set of fabric ports belonging to different network nodes connected to the same strand can be referred to as a pool. Fabric ports in the same pool can communicate with each other via the strand. Thus, fabric port 104-1, fabric port 104-2, fabric port 104-3, and fabric port 104-4 can communicate with each other via strand 150. A strand can support connections to a maximum allowable number of network nodes per strand. In some embodiments, the allowable number of network nodes per strand can be a configurable parameter and / or may depend on the embodiment of the strand (e.g., the number of channels and / or wavelengths supported by the strand).

[0013] Figures 2A - 2B show a conceptual diagram of an example of a strand connecting network nodes using reconfigurable multi-point optical technology. Fabric ports from different routers attached to the strand can communicate with each other simultaneously (multi-point connection) within the boundaries of the maximum port bandwidth. The bandwidth assigned to each port pair can be reconfigured in fixed units, and the bandwidth assigned to a particular port pair is independent of the bandwidth assigned to another port pair on the same strand, as long as the maximum bandwidth limit of any port is not violated.

[0014] Referring to Figure 2A, four network nodes (network node N1 202-1, network node N2 202-2, network node N3 202-3, and network node N4 202-4) can be connected on the strand 250 using a reconfigurable multi-point optical technique. Capacitance can be dynamically allocated between any pair of network nodes on the strand 250. For example, the strand 250 may support up to four channels ch1, ch2, ch3, and ch4, as shown in the figure. A channel may refer to an indivisible unit of capacitance allocated between a pair of fabric ports along the strand. In some embodiments, multiple channels supported by the strand can be implemented using their respective optical wavelengths or subcarriers. For example, channel ch1 can be implemented using wavelength λ1, channel ch2 can be implemented using wavelength λ2, channel ch3 can be implemented using wavelength λ3, and channel ch4 can be implemented using wavelength λ4.

[0015] In Figure 2A, channel ch2 connects network node N1 202-1 to network node N3 202-3, bypassing network node N2 202-2 without adding any switching delay. More generally, by utilizing multipoint connections, network nodes on a strand can reach any other network node on the same strand without incurring extra hops. When traffic demand changes, capacity allocation can be adapted, for example, by moving one or more channels between network nodes connected to a strand. For example, if the traffic demand between network node N2 202-2 and network node N3 202-3 increases, channel ch1 can be reassigned from the connection of network node N1 202-1 to network node N2 202-2, as shown in Figure 2A, to the connection of network node N2 202-2 to network node N3 202-3, as shown in Figure 2B.

[0016] Figure 3 shows an example of network nodes connected by strands. Each of the network nodes R1 302-1 to R8 302-8 can be implemented using network devices (e.g., routing devices) within the network fabric. Each of the network nodes R1 302-1 to R8 302-8 can be coupled to a server rack to provide connectivity between servers in the rack and the network fabric. Each network node has multiple connection ports that provide bandwidth capacity to the network node. The connection ports may include several fabric ports and several server ports (e.g., point-to-point server facing ports). For example, network node R1 302-1 may have a set of server ports 304-1 (individual ports are not shown) and a set of fabric ports 306-1. At least half of the network node's bandwidth capacity can be allocated to the fabric ports. In a specific example, network node R1 302-1 may have 16 x 1.6 Tbps fabric ports for a fabric capacity of 25.6 Tbps and 32 x 800 Gbps server ports for a server capacity of 25.6 Tbps, providing a 50 / 50 split of the network node's bandwidth capacity between fabric and server capacities. In some embodiments, bandwidth allocation can provide more than half of the network node's bandwidth capacity to the fabric ports for connecting to the network fabric. For example, a network node could implement an 80 / 20 split between fabric and server capacities. Shifting more capacity to the network fabric can reduce the hop count between source / destination network node pairs, and this can also increase the fabric capacity.

[0017] Network nodes R1 302-1 through R8 302-8 can be interconnected on fabric ports in a specific pattern called a harmonic. A harmonic is defined by the distance between adjacent connection points on a strand, relative to the number of network nodes between them. For example, strand 352 has one harmonic because its connection points are separated by only one network node. Thus, strand 352 connects network node R1 302-1 to network node R2 302-2, network node R3 302-3 to network node R4 302-4. In the illustrated example, each strand has a maximum pool of four network nodes, and therefore, a maximum of four network nodes can be connected to a strand. Strand 358 also has one harmonic, connecting network node R5 302-5 to network node R6 302-6, and network node R7 302-7 to network node R8 302-8.

[0018] As another example, the connection point of strand 354 is separated by only two network nodes, so strand 354 has two harmonics. Therefore, strand 354 connects network node R1 302-1 to network node R3 302-3, and network node R5 302-5 to network node R7 302-7. As yet another example, the connection point of strand 356 is separated by only three network nodes, so strand 356 has three harmonics. Therefore, strand 356 connects network node R1 302-1 to network node R4 302-4, and network node R7 302-7 to a tenth network node (not shown).

[0019] By enabling Weighted Cost Multipathing (WCMP) and intermediate hop routing, the number of paths between pairs of network nodes can be controlled. For example, network node R1 302-1 can reach network node R3 302-3 via a path along strand 354 or via a path along strand 352. As another example, network node R1 302-1 can reach network node R5 302-5 via a path along strand 354, but can also reach network node R7 302-7 via strand 356, and then from network node R7 302-7 to network node R5 302-5 via strand 354. Furthermore, by using multipoint connections, the hop count can be kept low. For example, by utilizing multipoint connections, network node R1 302-1 can reach network node R5 302-5 via strand 354 without incurring a hop at network node R3 302-3.

[0020] In Figure 3, only three fabric ports are connected on each network node, but it should be noted that the network fabric can be wired more densely so that additional or all fabric ports on the network node are connected to a strand. For example, when applied to a data center floor plan with an array of server racks arranged in rows and columns, half of the occupied fabric ports on the network node can be allocated to horizontal strands (connections along the rows) and the other half to vertical strands (connections along the columns). In some embodiments, the allocation of fabric ports to vertical and horizontal strands can be adjusted based on the aspect ratio of the array in the floor plan (e.g., to match or approximate the column-to-row ratio).

[0021] Each of the network nodes R1 302-1 to R8 302-8 may also include processing logic for performing the functions of the network node. The processing logic can be implemented using, for example, one or more of the following: application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), network processing units (NPUs), processors, or systems on a chip (SoC). For example, the processing logic may be able to operate to distribute traffic to a destination node along the corresponding strands of fabric ports to the network nodes, and the traffic can be routed to the destination node through the network fabric. In some embodiments, the processing logic may also be able to operate to transmit traffic demand information for inbound and outbound traffic of the network nodes to the control plane, receive channel allocation information from the control plane based on the traffic demand information, and configure reconfigurable multipoint optical connections with the channel allocation information.

[0022] Figure 4A shows an example of a network node set. In Figure 4, each rectangle can represent a server rack, which may contain one or more server computers and network devices acting as network nodes. Each network node or network device can be implemented using, for example, a routing device, a switch, or other type of network device capable of sending and receiving network traffic. The set of network nodes 400 can form a logical grid. For example, each network node may be assigned an identifier that can be translated into an array index representing the logical row and logical column to which the network node belongs.

[0023] In some embodiments, the network node set 400 can be physically arranged as an array of rows and columns of network nodes in a data center floor plan layout as shown. Referring to Figure 4A, the array of network nodes may include rows 402-1, 402-2, etc., and columns 404-1, 404-2, etc. Each column may correspond to a server rack aisle in the data center. Although only 48 network nodes are shown, it should be understood that the number of network nodes can be extended beyond those illustrated, as indicated by the ellipse.

[0024] In other embodiments, the logical grid of network nodes can be physically arranged in different configurations, and the physical arrangement does not necessarily have to be a rectangular or square layout. It should be understood that the use of the terms grid, array, row(s), and column(s) refers to the logical configuration of network nodes, and not necessarily the physical arrangement of network nodes, and that the logical configuration can be implemented using a physical grid or array.

[0025] In Figure 4A, each network node may connect server racks to other server racks using six fabric ports for connection to the network fabric. Three of the fabric ports on a network node may be assigned to strands that connect network nodes along the same row (e.g., horizontally), and three of the fabric ports may be assigned to strands that connect network nodes along the same column (e.g., vertically). Strands that connect network nodes along the same row may be called horizontal strands, and strands that connect network nodes along the same column may be called vertical strands. Thus, in the illustrated example, the set of horizontal strands connected to network nodes has the same number of strands as the set of vertical strands.

[0026] For example, a horizontal strand may have 1, 2, and 3 horizontal harmonics. The harmonics of each strand connecting network nodes in the same row can be collectively called the set of horizontal harmonics. Referring to the strands connected to network node 412 and its fabric port, strand 452 is a horizontal strand with 1 harmonic, connecting network nodes at a distance of 1 network node in the row direction. In the illustrated example, the maximum number of network nodes per strand is 3. Strand 454 is a horizontal strand with 2 harmonics, connecting network nodes at a distance of 2 network nodes in the row direction. Strand 456 is a horizontal strand with 3 harmonics, connecting network nodes at a distance of 3 network nodes in the row direction.

[0027] A vertical strand can have 1, 2, and 3 vertical harmonics, respectively. The harmonics of each strand connecting network nodes along the same column can be collectively called the set of vertical harmonics. Referring to the strands connected to network node 412 and its fabric port, strand 462 is a vertical strand with 1 harmonic, connecting network nodes at a distance of 1 network node in the column or aisle direction. Strand 464 is a vertical strand with 2 harmonics, connecting network nodes at a distance of 2 network nodes in the column or aisle direction. Strand 466 is a vertical strand with 3 harmonics, connecting network nodes at a distance of 3 network nodes in the column or aisle direction.

[0028] In the illustrated example, the set of vertical harmonics is the same as the set of horizontal harmonics. However, the two sets of harmonics do not need to have the same set of values. For example, one set of harmonics may contain one or more harmonics that are not present in the other set of harmonics, and / or vice versa. Furthermore, in the illustrated example, the number of harmonics is the same in both sets, but the number of harmonics in each set can be different, and the number of horizontal strands connected to a network node can be different from the number of vertical strands. In general, half of the fabric ports of a network node can be allocated to horizontal strands, and the other half can be allocated to vertical strands. However, the number of fabric ports allocated to horizontal strands may differ from that of vertical strands. For example, fabric ports can be allocated to horizontal / vertical strands to match or approximate the aspect ratio of the logical grid.

[0029] Figure 4B shows an example of a network fabric 450 created by extending the harmonic connections described above, with reference to network node 412, to all network nodes in the set of network nodes 400. Each network node may include three fabric ports connected to their respective horizontal strands according to the set of horizontal harmonics [1, 2, 3] (e.g., along the row direction) and three fabric ports connected to their respective vertical strands according to the set of vertical harmonics [1, 2, 3] (e.g., along the column or aisle direction). Each strand connects three network nodes.

[0030] By utilizing all available fabric ports, a higher density wiring network fabric can be created. For example, if a network node has 16 available fabric ports, a set of 8 harmonics can be implemented in both the row (e.g., horizontal) and column (e.g., vertical) directions. The specific values ​​chosen for a harmonic can be based on the number of network nodes in a row (e.g., row size) and / or the number of network nodes in a column (e.g., column size). To minimize the hop count, the largest harmonic can be selected so that the corresponding strand can extend at least the full length of the grid dimension. In other words, a set of horizontal strands can have horizontal strands that extend at least the full length of a column, and / or a set of vertical strands can have vertical strands that extend at least the full length of a column. Strands extending beyond the row / column length can wrap around at the end of the row / column and connect to the next network node from the beginning of the row / column based on the harmonic distance.

[0031] By avoiding harmonics that are multiples of other smaller harmonics (except 1), more diversity can be added to the network nodes that a given network node can connect to. Therefore, in some embodiments, prime numbers are chosen or preferred over non-prime numbers with respect to harmonics. In some embodiments, each horizontal harmonic and each vertical harmonic can be a prime number (for example, the values ​​1 and 2 can be considered prime). In other embodiments, non-prime numbers can also be used, but the set of horizontal harmonics preferably contains at least one prime number greater than 2, and the set of vertical harmonics preferably contains at least one prime number greater than 2. Distributing the harmonics between 1 and the largest harmonic can also reduce the worst-case hop count. Therefore, if there are enough prime numbers between 1 and the largest harmonic, the prime numbers chosen for harmonics may be chosen to maximize the difference between adjacent harmonics.

[0032] In some embodiments, the maximum harmonic in the set of harmonics can be chosen as the largest prime number less than half the number of network nodes along the strand direction. For example, for a row of 40 network nodes, the maximum harmonic could be 19, which is the largest prime number less than 20, half the number of network nodes along the row. With a maximum harmonic of 19, it is possible to implement the set of harmonics for all prime numbers [1, 3, 5, 7, 11, 13, 17, and 19]. Using such techniques, we can see that the typical hop count from any point in the network fabric to another point in the network fabric is 3 hops, and in the worst case it is usually 5 hops.

[0033] Given a maximum harmonic, if the number of primes within the range is insufficient, it should be noted that multiples of another harmonic may be used for the strand so that all available fabric ports are utilized. In some embodiments, if a strand reaches, for example, the end of a row or column and the strand has not exhausted its maximum allowable connections, the strand may wrap around to the beginning of the row or column and connect to the next network at a harmonic distance. In some embodiments, the available fabric capacity on the network devices may be evenly distributed among the strands. In some embodiments, strands with larger harmonics may be allocated more capacity than strands with lower harmonics. It should also be noted that not all harmonics in a set of horizontal or vertical harmonics need to be different, and one or more harmonics in a set may be repeating values ​​such that multiple harmonics along one direction are connected by the same harmonic.

[0034] Figure 5 shows a conceptual diagram of routing traffic between a source and a destination within a network fabric. Network fabric 500 could be, for example, network fabric 450, or another network with harmonic connectivity. Source 510 can be any one of the network nodes in network fabric 500, and destination 520 can be any other network node in network fabric 500. In order to send traffic from source 510 to destination 520, traffic can be distributed to one-hop neighbors of source 510, such as network node S1 512, network node S2 514, network node S3 516, etc., according to flow-level hashing (e.g., 5-tuple hashing). For example, in the case of a network node with 16 fabric ports and strands, each connecting up to 4 network nodes, each fabric port can connect to 3 other network nodes (one-hop neighbors), so source 510 can have 48 one-hop neighbors by utilizing multipoint connectivity. Traffic can be distributed to all one-hop neighbors of source 510, or to a configurable number of one-hop neighbors.

[0035] Each one-hop neighborhood receiving traffic from source 510 becomes a first transit point for traffic distributed to that one-hop neighborhood. Given a one-hop neighborhood for source 510, the corresponding one-hop neighborhood for destination 520 is selected as a second transit point for traffic distributed to that one-hop neighborhood of source 510. For example, destination 520 may have one-hop neighborhoods including network node D1 522, network node D2 524, network node D3 526, etc. Network node D2 524 can be selected as a transit point for traffic distributed from source 510 to network node S1 512. Once network node S1 512 and network node D1 524 are configured as transit points for a route, traffic between the two transit points can be routed through the network fabric 500 based on the shortest path between the transit points. In some embodiments, if there are multiple shortest paths between the two transit points, traffic can be routed using multipath routing between the two transit points. In embodiments where capacity between links can be dynamically adjusted, traffic can be routed using shortest path or equal-cost multipath (ECMP) routing for traffic demand below a threshold. For traffic demand above a threshold, routing can be optimized using weighted-cost multipath (WCMP) routing.

[0036] In some embodiments, a logically centralized control plane can be used to manage traffic within the network fabric. The control plane may receive telemetry flow data from network nodes and maintain an up-to-date traffic demand matrix (the amount of traffic between pairs of network nodes). The traffic demand matrix contains traffic information (e.g., in Gbps units) for each possible pair of network nodes in the network fabric. For example, sources may be located on the y-axis and destinations on the x-axis. Diagonal cells in the traffic demand matrix corresponding to the source and destination being the same network node may be null. The sum of each row corresponds to the total outbound traffic from the network node associated with that row, and the sum of each column corresponds to the total inbound traffic to the network node associated with that column.

[0037] In some embodiments, traffic demand can be characterized using average traffic, central traffic, or peak traffic across network nodes over a single measurement interval. Traffic demand can be a running average over a set of previous measurement intervals, or a weighted average that is more weighted towards the most recent measurement interval. When updating the traffic demand matrix, historical traffic patterns can also be considered (e.g., a cumulative average over the same measurement interval per day over the past few days). Using the traffic demand matrix as input, the control plane can periodically run an optimizer to determine how traffic is transmitted within the network fabric. In some embodiments, optimization can be performed using WCMP across a larger, configurable number of paths (e.g., a set of edge element paths) for traffic demand above a threshold (e.g., 1 Gbps), while for traffic demand below the threshold, ECMP can be used across the shortest path without an optimizer. The optimizer can be used to maximize the available capacity on the most saturated links (e.g., to provide headspace or excess capacity to accommodate unexpected demand).

[0038] To control how traffic demand is routed across the network fabric, the control plane may make decisions regarding (1) the number of channels to allocate between network nodes on a strand, (2) the paths on which traffic is distributed for source-destination network node pairs, and (3) the relative weights of those paths. These decisions can be made in two stages. In the offline stage, which can be performed at periodic intervals (e.g., hourly), the control plane can calculate the channel allocation for strands and sets of raw paths between each pair of network nodes. Past demand can provide an indicator of expected acceptance for the next interval. By default, source network nodes may distribute traffic evenly along these paths (flow-level hashing).

[0039] In the online phase, this can be run at more frequent intervals (e.g., every few minutes), but it can identify network node pairs transmitting at a rate higher than a threshold, and the relative weights of each path between the pairs can be optimized with fixed channel allocation. This calculation considers traffic that continues to be evenly distributed across the paths. This allows for maximizing the minimum headroom available on any link to accommodate bursts and unexpected traffic increases between routers.

[0040] In some embodiments, the optimizer may be implemented within the control plane using a linear programming solver (e.g., a mixed-integer linear programming (MILP) or integer linear programming (ILP) solver) depending on the objective function. Given a traffic demand matrix and wiring topology (which network nodes connect to which other network nodes according to harmonics), channel allocation and forwarding paths can be calculated by the optimizer. All demands in the traffic demand matrix are routed based on the objective function (e.g., minimizing maximum link utilization).

[0041] The inputs to the optimizer may include the following. ● Demand matrix: ○ {dij}: A set of traffic demands from router i to router j (Gbps) ● Encoding of the physical topology (wiring): ○ Lx = [rk]: A list of routers connected to fiber x ● Encoding of valid paths for demands (to control stretch): ○ Pij = {pa,...}, where pa = [lxkl] (the path is a list of links) ● Other constants based on strand implementation / topology: ○ N: The maximum number of channels that can exist on a fiber ○ C: The capacity of each channel (Gbps)

[0042] The outputs of the optimizer may include the following. ● Channel assignment (which may result in a logical link consisting of multiple channels) ○ nxkl: The number of channels assigned to link lxkl. ● How the demand is transmitted: ○ fij,a: A portion of demand dij transmitted on path pa (Gbps)

[0043] The set of constraints of the optimizer may include the following. ● Channel assignment (two models can be used): ○ Model 1, fiber level: The total number of channels on each fiber must be equal to N ■ ∀x · Σ{k,l|rk∈Lx,rl∈Lx,k<l}nxkl = N (k < l ensures that each direction is counted only once) ○ Model 2, port level: The sum of incoming and outgoing for each attached port is fixed ■ ∀x · ∀k when rk ∈ Lx: Σ{l|rl∈Lx}nxkl = N (transmission limit) · When rl ∈ Lx, for all l: Σ{k|rk ∈ Lx} nxkl = N (incoming limit) ● All demands are allocated: ○ r k ∈ L x 、r l ∈ L x 、for all k, l where k < l: n xkl = n xlk ■ For all i, j · Σ{a|p a ∈ P ij} fij,a = dij ● Capacity constraint: ○ r k ∈ L x 、r l ∈ L x 、for all x, k, l where k = Ql: ■ uxkl = Σ{i,j,a|pa ∈ Pij, lxkl ∈ pafij,al (total traffic transmitted by links for all paths that are part of it) ■ uxkl <= nxkl × C (must be less than the capacity)

[0044] The objective function for the optimizer can be based on one or more of the link utilization rate between connection points, headroom (providing excess capacity), latency, and / or redundancy (providing duplicate paths). Examples of objective functions that can be used can include the following. ● Minimize the maximum utilization rate (non - linear): ○ max xkl ( uxkl / ( nxkl × C)) to be minimized ● Maximize the minimum headroom on the link (linear): ○ min xkl (( nxkl × C) - u xkl ) to be maximized

[0045] Figure 6 shows a first exemplary embodiment of a strand in which multipoint optical connectivity is implemented using breakout optics (e.g., DR4 optics) to connect the fabric ports of network nodes along the strand to the fabric ports of each network node. In the illustrated example, strand 600 connects five network nodes 602, 604, 606, 608, and 610. Each network node can be implemented using, for example, a routing device, a switch, or other type of network device that can direct network traffic. Network node 602 may include multiple fabric ports, including fabric port 602-1. For example, only one fabric port is shown, but network node 602 may include 8, 16, 32, or any other number of fabric ports to connect to other strands. Network node 602 may also include multiple server ports to connect to servers or other networks. Other network nodes may have a similar configuration to network node 602.

[0046] Fabric port 602-1 can be a parallel single-mode port and can be implemented, for example, using DR4 optical transceivers to provide four breakout channels. Thus, each fabric port can connect to four other network nodes. Note that each channel is a dual channel with a receive channel and a transmit channel. Strand 600 provides multipoint connectivity, but with fixed bandwidth between port pairs (e.g., bandwidth may not be reconfigurable or incremental / decrementable). Network fabrics built using Strand 600 can be routed, for example, using shortest path or ECMP. Because the bandwidth between port pairs is fixed, optimization by the control plane to reallocate channel capacity can be omitted.

[0047] Figure 7 shows a second exemplary embodiment in which a reconfigurable multipoint optical connection is implemented using breakout optics (e.g., DR4 optics) to connect the fabric ports of network nodes to multiple ports of an optical circuit switch that connects other network nodes along the strand. In the illustrated example, the strand 700 connects four network nodes 702, 704, 706, and 708. Only one fabric port 702-1 is shown for network node 702, but each network node may include multiple fabric ports to connect to other strands. Fabric port 702-1 can be a parallel single-mode port and can be implemented to provide four breakout dual channels, for example, using a DR4 optical transceiver. The breakout fibers from the fiber ports of each network node can be connected to an optical circuit switch (OCS) 750 to form a capacity pool. Each fiber pair from each fabric port along the strand 700 can be connected to any fiber pair from any other fabric port along the strand 700. Because this is a reconfigurable physical path, wavelength division multiplexing (WDM), such as coarse wavelength division multiplexing (CWDM), can be used at the exit of each fiber as needed. Bidirectional use of OCS can be implemented, for example, using a circulator. The Strand 700 allows for independent configuration of bandwidth within the pool without constricting the capacity of any port.

[0048] Figure 8 shows a third exemplary embodiment of a strand in which reconfigurable multipoint optical connections are implemented using optical couplers and splitters to merge and split optical wavelengths along the strand. In the illustrated example, strand 800 connects four network nodes 802, 804, 806, and 808. Only one fabric port 802-1 is shown for network node 802, but each network node may include multiple fabric ports to connect to other strands. Strand 800 includes connecting fabric ports in a pool using a cascade of T-couplers. Individual fabric ports can transmit on an exclusive set of wavelengths to avoid downstream collisions. Coherent techniques using tunable lasers may allow receivers at each port to acquire specific wavelengths directed by a control plane. Alternatively, tunable filters can be used with intensity-modulated direct detection (IMDD) techniques. Transmitters within strand 800 transmit in both directions to reach all fabric ports of the strand. The coupler ratio can be optimized to minimize the transceiver's loss budget. In some embodiments, the coupler / splitter can be part of the fabric port itself (for example, to simplify the cabling infrastructure).

[0049] Figure 9 shows a fourth exemplary embodiment of a strand in which a reconfigurable multipoint optical connection is implemented using a reconfigurable optical add / drop multiplexer (ROADM) to add and interrupt optical wavelengths along the strand. In the illustrated example, the strand 900 connects four network nodes 902, 904, 906, and 908. Only one fabric port 902-1 is shown for network node 902, but each network node may include multiple fabric ports to connect to other strands. The strand 900 involves the use of a reconfigurable optical add / drop multiplexer (ROADM) 950. Each ROADM 950 has access to all wavelengths on the fiber and allows a particular wavelength to be interrupted or added at a certain point, while simultaneously allowing other wavelengths to pass optically without requiring termination. Each ROADM 950 allows a wavelength to be interrupted without continuation in one direction, but the same wavelength can also be added and moved in the opposite direction. Wavelength tunability is utilized in optical networks connecting fabric ports using transceivers and wavelength-selective switches within ROADMs. Using tunable lasers and wavelength-selective switches, receivers at each fabric port can acquire specific wavelengths directed by the control plane. A similar scheme can be implemented with fixed OADMs.

[0050] Figure 10 shows a fifth exemplary embodiment of a strand in which a reconfigurable multipoint optical connection is implemented in silicon photonics using ring resonators to modulate and detect optical wavelengths along the strand. In the illustrated example, the strand 1000 connects four network nodes 1002, 1004, 1006, and 1008. In Figure 10, the fabric ports of the network nodes are represented as silicon photonics (SiPho) transceivers TRX1050, which include ring resonators (e.g., microring resonators) and lasers 1040 (e.g., off-chip lasers). Each ring contributes to the multiplexing, demultiplexing, and modulation operations performed by the laser 1040. Each TRX1050 may include a set of drivers 1052 for driving modulators 1054 corresponding to a particular wavelength, and a set of detectors 1058 for extracting or detecting a particular wavelength for a set of receivers 1056. The rings are tuned to a particular wavelength, and optical channels of other wavelengths are transmitted directly with minimal loss. When the ring is resonating, the corresponding light wavelength is guided into the ring. In this way, receivers in each fabric port can acquire a specific wavelength that is directed by the control plane.

[0051] Figure 11 shows an exemplary flowchart of an example of a process 1100 for routing network traffic between source and destination nodes in a network fabric including network nodes interconnected by strands according to harmonics, each specifying the node distance between adjacent connection points on a corresponding strand. For example, the network fabric can be a logical grid of network nodes connected according to horizontal and vertical harmonics, as described herein. Process 1100 can be performed, for example, by network nodes operating under the direction of a control plane (e.g., a centralized control plane for managing network traffic within the fabric). Certain aspects of process 1100 can be implemented as instructions or commands (e.g., software code) stored in a non-temporary computer-readable medium, which can be executed by network devices to perform various operations.

[0052] Process 1100 may be initiated in block 1102 by a source node distributing network traffic to a set of one-hop neighbors of the source node. The network traffic may correspond to a specific flow (e.g., identified by 5-tuple hashing). One-hop neighbors can be other network nodes connected to a strand of which the source node is part. Each one-hop neighbor of the source node that receives the distributed traffic can forward the distributed traffic to a destination node via a different path.

[0053] In block 1104, the one-hop neighborhood of the source node is configured as a first waypoint for traffic distributed to this one-hop neighborhood. In block 1106, the one-hop neighborhood of the destination node associated with the one-hop neighborhood of the source node is identified. For example, the one-hop neighborhood of the destination node can be identified by searching the source and destination nodes in a mapping table that contains one-to-one mappings of node one-hop neighborhoods. One-to-one mappings or associations between nodes can be configured, for example, by the control plane. Mappings can be assigned randomly, or they can be selected based on a specific metric such as the shortest path or lowest utilization. In block 1108, the identified one-hop neighborhood of the destination node can be configured as a second waypoint for distributed traffic.

[0054] In block 1110, distributed traffic is routed through the network fabric from a first via point to a second via point. In embodiments where each link between connection points on each strand in the network fabric has a fixed bandwidth capacity, routing between the two via points may be performed using equal-cost multipath routing (ECMP). In embodiments where each link between connection points on each strand in the network fabric has a dynamically adjustable bandwidth capacity (for example, each strand can support multiple channels using reconfigurable multi-point optics), routing between the two via points can be performed using weighted-cost multipath routing (WCMP).

[0055] In some embodiments, the routing scheme can be selected based on the traffic demand from the source node to the destination node. For traffic demand below a threshold traffic limit (e.g., 10 Gbps), ECMP can be selected as the routing scheme. For traffic demand above the threshold, WCMP can be selected as the routing scheme. WCMP routing can be more computationally intensive, and therefore, WCMP may be reserved for higher traffic loads when route optimization has a greater impact.

[0056] WCMP can use an optimizer to determine the amount of traffic to carry on each path (e.g., a portion of the traffic demand) and / or the channel allocation on the links between connection points on each strand. The optimizer can be implemented in the control plane using a linear programming solver to find the optimal solution to an objective function (e.g., minimize or maximize the objective function). Depending on the objective function, a mixed-integer linear programming (MILP) solver or an integer linear programming (ILP) solver may be used. The input to the optimizer may include a traffic demand matrix containing traffic information between each pair of network nodes in the network fabric, a connectivity topology including the harmonics of the network fabric, and valid paths for each pair of network nodes in the network fabric. The input to the optimizer may also include the maximum number of channels supported per strand and the bandwidth capacity per channel.

[0057] Traffic information included in the traffic demand matrix can be obtained from network nodes within the network fabric. For example, each network node may periodically provide the control plane with the amount of inbound traffic it receives from other network nodes and the amount of outbound traffic it sends to other network nodes over time intervals. For example, inbound and outbound traffic information from the previous hour can be used as an indicator of traffic demand for the next hour. The set of valid paths for a pair of network nodes considered by the optimizer may include edge paths, point paths, and / or shortest paths.

[0058] The set of optimizer constraints may include traffic constraints indicating that all traffic demand in the traffic demand matrix is ​​routed. The set of constraints may also include channel allocation constraints at the fiber / strand level or fabric port level. Fiber / strand level constraints may indicate that the total number of channels on each fiber / strand is equal to the maximum allowed number of channels per strand. Port level constraints may indicate that the sum of incoming and outgoing channels on a port is set to the maximum allowed number of channels. The set of constraints may further include capacity constraints indicating that the total traffic carried by each link for all paths in which the link is part is less than the capacity allocated to the link.

[0059] The optimizer's objective function can be based on one or more of the following: link utilization between connection points, excess capacity, latency, and / or redundancy. For example, the objective function could be to minimize the maximum utilization of each link between connection points, to maximize the minimum excess capacity or headspace on each link between connection points, to minimize maximum latency (e.g., number of hops), or a combination thereof. The objective function can be adjusted based on network performance priorities (e.g., capacity over latency, or vice versa) and / or the available computing resources of the control plane. The optimizer can be run periodically (e.g., hourly) based on an updated traffic demand matrix that provides the latest traffic patterns between network nodes in the network fabric.

[0060] Figure 12 shows a flowchart of an example of a process 1200 for performing communication within a network fabric including network nodes. Network nodes can be interconnected by strands according to harmonics that specify the node distance between adjacent connection points on the corresponding strands. For example, a network fabric can be a logical grid of network nodes connected according to horizontal and vertical harmonics, as described herein. Process 1200 can be performed, for example, by a network device acting as a network node in the network fabric. Certain aspects of process 1200 can be implemented as instructions or commands (e.g., software code) stored in a non-temporary computer-readable medium, which can be executed by a network device to perform various operations.

[0061] A network device may include connection ports that provide bandwidth capacity to the network device. The connection ports may include a collection of fabric ports capable of connecting to each strand. Each strand may be implemented using, for example, multi-point optical connections. In some embodiments, the multi-point optical connections may be reconfigurable multi-point optical connections that support multiple channels to allow dynamic channel reallocation between connection points on the strand.

[0062] Process 1200 may begin in block 1202 by transmitting traffic demand information for ingress and egress traffic of network devices to a control plane (e.g., a centralized control plane) that manages the network fabric. The ingress / egress traffic information may include bandwidth usage of network nodes collected over a period of time, and the ingress / egress traffic information may be transmitted to the control plane periodically. The control plane may use the ingress / egress traffic information to predict or approximate traffic demand, and may generate a traffic demand matrix from traffic information collected from network nodes of the network fabric.

[0063] In block 1204, the network device may receive channel allocation information from the control plane based on traffic demand information. For example, the control plane may run an optimizer to determine traffic distribution (e.g., the amount of traffic or a portion of the traffic demand to be carried on each path) and / or channel allocation on the links between connection points on each strand. The channel allocation information received by the network device may include channel allocation on the links connected to the fabric ports of the network device.

[0064] In block 1206, a reconfigurable multi-point optical connection can be configured with channel assignment information received from the control plane. For example, a network device may adjust, enable, or disable the wavelengths used by each fabric port according to the channel assignment information. Depending on a particular embodiment, the configuration of the multi-point optical connection may include configuring optical transceivers coupled to the fabric ports and / or configuring external components such as optical circuit switches.

[0065] In block 1208, a network device can distribute traffic for a destination node to network nodes along strands connected to the network device. The traffic can be distributed in a manner determined by the control plane. The distributed traffic can then be routed through the network fabric to the destination node using the techniques disclosed herein. For example, a one-hop neighborhood of a network device and a one-hop neighborhood of a destination node mapped to the one-hop neighborhood of the network device can be set as two waypoints, and routing between waypoints is performed using ECMP or WCMP (for example, based on traffic demand).

[0066] Figure 13 shows an example of a network device 1300. The functions and / or some components of the network device 1300 can be used without limitation, and without being limited to other embodiments disclosed elsewhere in this disclosure. The network device 1300 can be used, for example, as a network node in a network fabric. The network device 1300 can facilitate the processing of packets and / or the forwarding of packets from the network device 1300 to another device. Where used herein, “packet” or “network packet” may refer to a variable or fixed unit of data. In some cases, a packet may include a packet header and a packet payload. The packet header may include information related to the packet, such as source, destination, quality of service parameters, length, protocol, routing label, and error correction information. In certain embodiments, one packet header may show information related to a series of packets, such as a burst transaction. In some embodiments, the network device 1300 may be a receiver and / or generator of packets. In some embodiments, the network device 1300 may modify the contents of a packet before forwarding it to another device. The network device 1300 may be a peripheral device coupled to another computer device, a switch, a router, or any other suitable device enabled to receive and forward packets.

[0067] In one example, the network device 1300 may include a processing logic module 1302, a configuration module 1304, a management module 1306, a bus interface module 1308, a memory module 1310, and a network interface module 1312. These modules may be hardware modules, software modules, or a combination of hardware and software. In certain examples, modules may be used interchangeably with components or engines without departing from the scope of this disclosure. The network device 1300 may include additional modules not shown herein, such as other components of the network node described herein (e.g., fabric ports, server ports, etc.). In some embodiments, the network adapter device 1300 may include fewer modules. In some embodiments, one or more modules may be combined into a single module. One or more modules may communicate with each other on a communication channel 1314. The communication channel 1314 may include one or more buses, meshes, matrices, fabrics, combinations of these communication channels, or several other suitable communication channels.

[0068] The processing logic 1302 may include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), systems-on-chip (SoCs), network processing units (NPUs), processors configured to execute instructions, or any other circuits configured to perform logical arithmetic and floating-point operations. Examples of processors that may be included in the processing logic 1302 may include processors developed by ARM®, MIPS®, AMD®, Intel®, Qualcomm®, and others. In certain embodiments, the processor may include multiple processing cores, each processing core may be configured to execute instructions independently of the other processing cores. Furthermore, in certain embodiments, each processor or processing core may implement multiple processing threads that execute instructions on the same processor or processing core while maintaining logical isolation between the multiple processing threads. Such processing threads running on a processor or processing core may be exposed to software as separate logical processors or processing cores. In some embodiments, multiple processors, processing cores, or processing threads running on the same core may share certain resources, such as a bus, a Level 1 (L1) cache, and / or a Level 2 (L2) cache. Instructions executed by the processing logic 1302 may be stored, for example, in the form of a computer program in a computer-readable storage medium. The computer-readable storage medium may be non-temporary. In some cases, the computer-readable medium may be part of memory 1310.

[0069] Memory 1310 may include either volatile or non-volatile memory, or both volatile and non-volatile types of memory. Memory 1310 may include, for example, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, and / or some other suitable storage media. In some cases, some or all of memory 1310 may be inside the network device 1300, and in other cases, some or all of memory may be outside the network device 1300. Memory 1310 may store an operating system that, when executed by processing logic 1302, provides an execution environment for executing instructions that provide network functionality to the network device 1300. Memory may also store and maintain some data structures and routing tables to facilitate the functionality of the network device 1300.

[0070] In some embodiments, the configuration module 1304 may include one or more configuration registers. The configuration registers may control the operation of the network device 1300. In some embodiments, one or more bits of the configuration registers may represent specific capabilities of the network device 1300. The configuration registers may be programmed by instructions executed in the processing logic 1302 and / or by external entities such as a host device, an operating system running on the host device, and / or a remote server. The configuration module 1304 may further include hardware and / or software that controls the operation of the network device 1300.

[0071] In some embodiments, the management module 1306 may be configured to manage different components of the network device 1300. In some cases, the management module 1306 may configure one or more bits of one or more configuration registers at power-up to enable or disable specific functions of the network device 1300. In certain embodiments, the management module 1306 may use processing resources from the processing logic 1302. In other embodiments, the management module 1306 may have processing logic similar to that of the processing logic 1302, but may be segmented or implemented on a different power plane than the processing logic 1302.

[0072] The bus interface module 1308 may enable communication with external entities, such as a host device and / or other components within a computing system, over an external communication medium. The bus interface module 1308 may include a physical interface or other connection to an external communication medium, such as a cable, socket, or port. The bus interface module 1308 may further include hardware and / or software for managing receive and transmit transactions. The bus interface module 1308 may implement local bus protocols such as Peripheral Component Interconnection (PCI) based protocols, Non-Volatile Memory Express (NVMe), Advanced Host Controller Interface (AHCI), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), Serial AT Attachment (SATA), Parallel ATA (PATA), several other standard bus protocols, or a proprietary bus protocol. The bus interface module 1308 may include a physical layer for any of these bus protocols, including, among other things, connectors, power management, and error handling. In some embodiments, the network device 1300 may include multiple bus interface modules for communicating with multiple external entities. These multiple bus interface modules can implement the same local bus protocol, different local bus protocols, or a combination of the same and different bus protocols.

[0073] The network interface module 1312 may include hardware and / or software for communicating with a network. This network interface module 1312 may include, for example, a physical connector or physical port for a wired connection to the network, and / or an antenna for wireless communication to the network. The network interface module 1312 may further include hardware and / or software configured to implement a network protocol stack. The network interface module 1312 may communicate with the network using, among other things, network protocols such as TCP / IP, Infiniband, RoCE, IEEE 802.13 radio protocol, User Datagram Protocol (UDP), Asynchronous Transfer Mode (ATM), Token Ring, Frame Relay, High-Level Data Link Control (HDLC), Fiber Distributed Data Interface (FDDI), and / or Point-to-Point Protocol (PPP). In some embodiments, the network adapter device 1300 may include multiple network interface modules, each configured to communicate with a different network. For example, in these embodiments, the network device 1300 may include a network interface module for communicating with a wired Ethernet network, a wireless 802.11 network, a cellular network, an Infiniband network, and the like.

[0074] The various components and modules of the network device 1300 described above may be implemented as individual components, as a system-on-a-chip (SoC), as an ASIC, as an NPU, as an FPGA, or as any combination thereof. In some embodiments, the SoC or other components may be communicatively coupled to another computing system to provide various services such as traffic monitoring, traffic shaping, and computing. In some embodiments of this technology, the SoC or other components may include multiple subsystems.

[0075] Embodiments of this disclosure can be described in view of the following clauses.

[0076] Article 1. It is a computer network, A plurality of network nodes arranged in rows and columns, each network node being coupled to a server rack and having a set of fabric ports allocated for connection to other network nodes, comprising a plurality of network nodes, Each network node is joined to network nodes in the same row using a set of horizontal strands, and to network nodes in the same column using a set of vertical strands. Each horizontal strand connects network nodes in the same row according to a horizontal harmonic that specifies the node distance along the row between adjacent connection points on the horizontal strand, and each horizontal harmonic is different from other horizontal harmonics in the set of horizontal strands. Each vertical strand connects network nodes in the same column according to a vertical harmonic, which specifies the node distance along the column between adjacent connection points on the vertical strand, and each vertical harmonic is different from other vertical harmonics in the set of vertical strands, in a computer network.

[0077] Article 2. The computer network described in Clause 1, wherein each strand in a set of horizontal strands and a set of vertical strands connects at least four network nodes.

[0078] Article 3. A computer network as described in either of Clauses 1 or 2, wherein each strand in a set of horizontal strands and a set of vertical strands is implemented using reconfigurable multi-point optical connections.

[0079] Article 4. A computer network as described in any of clauses 1 to 3, wherein each horizontal harmonic and each vertical harmonic is a prime number.

[0080] Article 5. It is a computer network, A logical grid comprising rows and columns of network nodes, Each network node is connected to network nodes in the same row using a set of horizontal strands according to a set of horizontal harmonics, where each horizontal harmonic specifies the node distance along the row between adjacent connection points on the corresponding horizontal strand. A computer network in which each network node is connected to network nodes in the same column using a set of vertical strands according to a set of vertical strands, and vertical harmonics specify the node distance along the column between adjacent connection points on the corresponding vertical strands.

[0081] Article 6. The computer network described in Clause 5, wherein each link between connection points on each strand in a set of horizontal strands and a set of vertical strands has a dynamically adjustable bandwidth capacity.

[0082] Article 7. Each strand has a maximum bandwidth capacity expressed as the number of channels available on the strand, and the bandwidth capacity of each link along the strand is adjusted by reallocating the number of channels available on the strand, as described in Clause 6 of the computer network.

[0083] Article 8. The computer network described in Clause 5, wherein each link between connection points on each strand in a set of horizontal strands and a set of vertical strands has a fixed bandwidth capacity.

[0084] Article 9. A computer network as described in any of clauses 5 to 8, wherein each strand in a set of horizontal strands and a set of vertical strands connects up to the maximum allowable number of network nodes per strand.

[0085] Article 10. A computer network as described in Clause 9, where the maximum number of network nodes allowed per strand is based on the number of channels supported by the strand.

[0086] Article 11. A computer network as described in Clause 10, wherein the number of channels supported by the strand is based on the number of optical wavelengths supported by the strand.

[0087] Article 12. A computer network as described in any of clauses 5 to 11, wherein the set of horizontal strands connected to a network node has the same number of strands as the set of vertical strands connected to a network node.

[0088] Article 13. A computer network as described in any of clauses 5 to 12, wherein each horizontal harmonic in the set of horizontal harmonics is distinct from other horizontal harmonics in the set of horizontal harmonics.

[0089] Article 14. A computer network as described in any of clauses 5 to 13, wherein each vertical harmonic in the set of vertical harmonics is distinct from other vertical harmonics in the set of vertical harmonics.

[0090] Article 15. A computer network as described in any of clauses 5 to 14, wherein the set of horizontal harmonics contains at least one prime number greater than 2.

[0091] Article 16. A computer network as described in any of clauses 5 to 14, wherein the set of vertical harmonics contains at least one prime number greater than 2.

[0092] Article 17. A computer network as described in any of Clauses 5 to 16, wherein a set of horizontal strands includes horizontal strands that extend at least the full length of a row, and a set of vertical strands includes vertical strands that extend at least the full length of a column.

[0093] Article 18. A computer system, It comprises a logical grid having rows and columns of server racks, each containing one or more server computers, Each server rack is joined to another server rack in the same row using a set of horizontal strands according to a set of horizontal harmonics, where each horizontal harmonic specifies the number of server racks along the row between adjacent connection points on the corresponding horizontal strand. A computer system in which each server rack is joined to other server racks in the same column using a set of vertical strands according to a set of vertical harmonics, where each vertical harmonic specifies the number of server racks along the column between adjacent connection points on the corresponding vertical strand.

[0094] Article 19. The computer system according to Clause 18, wherein each link between connection points on each strand in a set of horizontal strands and a set of vertical strands has a dynamically adjustable bandwidth capacity.

[0095] Article 20. The computer system according to Clause 18, wherein each link between connection points on each strand in the set of horizontal strands and the set of vertical strands has a fixed bandwidth capacity.

[0096] Article 21. A computer implementation method for routing network traffic between source nodes and destination nodes in a network fabric having a logical grid of network nodes interconnected by strands, according to harmonics that specify the node distance between adjacent connection points on the corresponding strands, The routing scheme is selected based on the traffic demand from the source node to the destination node, and the routing scheme is selected between equal-cost multipath routing (ECMP) for traffic demand below a threshold and weighted-cost multipath routing (WCMP) for traffic demand above a threshold. Distributing network traffic from the source node to a set of nodes in the immediate vicinity of the source node, For each one-hop neighborhood of a source node within the set of one-hop neighborhoods of a source node, The first waypoint is set to be a location one hop away from the source node, Identifying the one-hop vicinity of the destination node that is mapped to the one-hop vicinity of the source node, Setting a second waypoint is a location near the identified one-hop point of the destination node, A method comprising routing traffic distributed from a first transit point to a second transit point via a network fabric, according to a selected routing scheme.

[0097] Article 22. The computer implementation method described in Clause 21, wherein each strand in the network fabric supports up to the maximum number of channels, and WCMP routing is performed using an optimizer that determines channel allocation on the links between connection points on each strand and a portion of the traffic demand transmitted on each path.

[0098] Article 23. The computer implementation method described in Clause 22, wherein the input to the optimizer includes a traffic demand matrix containing traffic information between each pair of network nodes in the network fabric, a connectivity topology including the harmonics of the network fabric, and valid paths for each pair of network nodes in the network fabric, the maximum number of channels supported per strand, and the bandwidth capacity per channel.

[0099] Article 24. A computer implementation method according to any one of clauses 21 to 22, wherein the objective function of the optimizer is to minimize the maximum utilization or maximize the minimum excess capacity at each link between connection points.

[0100] Article 25. A computer implementation method for routing network traffic between source nodes and destination nodes in a network fabric, which includes network nodes interconnected by strands, according to harmonics that specify the node distance between adjacent connection points on the corresponding strands, Distributing network traffic from the source node to a set of nodes in the immediate vicinity of the source node, For each one-hop neighborhood of a source node within the set of one-hop neighborhoods of a source node, The first waypoint is set to be a location one hop away from the source node, Identifying the one-hop vicinity of the destination node that is mapped to the one-hop vicinity of the source node, Setting a second waypoint is a location near the identified one-hop point of the destination node, A computer implementation method comprising routing traffic distributed from a first transit point to a second transit point via a network fabric.

[0101] Article 26. Routing distributed traffic is performed using equal-cost multipath routing (ECMP) as described in the computer implementation method in Clause 25.

[0102] Article 27. The computer implementation method described in any of clauses 25 to 26, wherein each link between connection points on each strand in the network fabric has a fixed bandwidth capacity.

[0103] Article 28. A computer implementation method as described in any of clauses 25-27, wherein the traffic demand between the source node and the destination node is below the threshold traffic limit used to select a different routing scheme.

[0104] Article 29. Routing distributed traffic is performed using weighted cost multipath routing (WCMP), as described in the computer implementation method in Clause 25.

[0105] Article 30. The computer implementation method according to Clause 25 or Clause 29, wherein each link between connection points on each strand in the network fabric has a dynamically adjustable bandwidth capacity.

[0106] Article 31. A computer implementation method as described in either Clause 25 or Clauses 29-30, wherein the traffic demand between the source node and the destination node is greater than or equal to a threshold traffic limit used to select a different routing scheme.

[0107] Article 32. Routing distributed traffic is performed by an optimizer that determines the amount of traffic to carry on each path, and the optimizer, A traffic demand matrix containing traffic information between each pair of network nodes in the network fabric, Network fabric connectivity topology including harmonics, A computer implementation method according to any of clauses 25 to 31, which receives a set of inputs including valid routes to each pair of network nodes in a network fabric.

[0108] Article 33. The set of inputs to the optimizer further includes the maximum number of channels supported per strand and the bandwidth capacity per channel. The computer implementation method according to Clause 32 further comprises an optimizer that determines the channel assignment on the links between connection points on each strand.

[0109] Article 34. The computer implementation method described in any of clauses 32-33, wherein the optimizer's objective function is based on one or more of the following: link utilization between connection points, excess capacity, latency, or redundancy.

[0110] Article 35. The optimizer is a computer implementation method described in any of clauses 32-34, which is performed periodically based on the updated traffic demand matrix.

[0111] Article 36. A computer implementation method according to any one of clauses 25 to 35, wherein the one-hop neighbor of a destination node mapped to the one-hop neighbor of a source node is identified by searching for the source node and destination node in a mapping table.

[0112] Article 37. A non-temporary, computer-readable medium that, when executed by one or more processors, stores code that causes one or more processors to perform an operation for routing network traffic between source and destination nodes in a network fabric, including network nodes interconnected by strands, according to harmonics that specify the node distance between adjacent connection points on the respective strands, wherein the operation is, Distributing network traffic from the source node to a set of nodes in the immediate vicinity of the source node, For each one-hop neighborhood of a source node within the set of one-hop neighborhoods of a source node, The first waypoint is set to be a location one hop away from the source node, Identifying the one-hop vicinity of the destination node that is mapped to the one-hop vicinity of the source node, Setting a second waypoint is a location near the identified one-hop point of the destination node, A non-transient, computer-readable medium, which includes routing traffic distributed from a first transit point to a second transit point via a network fabric.

[0113] Article 38. Routing distributed traffic is performed using Equal-Cost Multipath Routing (ECMP) in non-transient computer-readable media as described in Clause 37.

[0114] Article 39. Routing distributed traffic is performed using Weighted Cost Multipath Routing (WCMP) in non-transient computer-readable media as described in Clause 37.

[0115] Article 40. Distributed traffic is routed using equal-cost multipath routing (ECMP) when traffic demand between source and destination nodes is below a threshold, and weighted-cost multipath routing (WCMP) when traffic demand is above a threshold, as described in the non-temporary computer-readable media of Clause 37.

[0116] Article 41. A network device that operates as a network node in a network fabric, Multiple connection ports that provide bandwidth capacity to a network device, A set of fabric ports that can operate to connect to each strand, each strand connecting network nodes of a network fabric according to harmonics that specify the node distance between adjacent connection points on the strand, and each strand is implemented using reconfigurable multi-point optical connections that support multiple channels to enable dynamic channel reassignment between connection points on the strand, A set of server ports capable of connecting to one or more servers, or one or more external networks, The system transmits traffic demand information for network device inbound and outbound traffic to the control plane. Based on traffic demand information, channel allocation information is received from the control plane. Configure a multi-point optical connection that can be reconfigured using channel assignment information. A network device comprising multiple connection ports, including processing logic capable of distributing traffic destined for a destination node, routed to the destination node via a network fabric, along the corresponding strands of fabric ports to the network node.

[0117] Article 42. A network device as described in Clause 41, wherein multiple channels supported by strands are implemented on their respective optical wavelengths.

[0118] Article 43. At least half of the bandwidth capacity of the network device is allocated to a set of fabric ports, as described in any of clauses 41-42.

[0119] Article 44. Each network node connected along a strand is a network device as described in any of clauses 41 to 43, including at least four network nodes.

[0120] Article 45. A network device that operates as a network node in a network fabric, Multiple connection ports that provide bandwidth capacity to a network device, A set of fabric ports capable of operating to connect to each strand using multi-point optical connections, wherein each strand connects to a set of fabric ports that connect to network nodes of a network fabric according to harmonics specifying the node distance between adjacent connection points on the strand, A set of server ports capable of connecting to one or more servers, or one or more external networks, A network device comprising multiple connection ports, including processing logic capable of distributing traffic destined for a destination node, routed to the destination node via a network fabric, along the corresponding strands of fabric ports to the network node.

[0121] Article 46. A network device is part of a logical grid of network nodes, and the strands connecting the network nodes of the network fabric include vertical strands connecting network nodes along the same column and horizontal strands connecting network nodes along the same row, as defined in Clause 45.

[0122] Article 47. A network device as described in Clause 46, in which half of the fabric ports are assigned to vertical strands and the other half of the fabric ports are assigned to horizontal strands.

[0123] Article 48. At least half of the bandwidth capacity of the network device is allocated to a set of fabric ports, as described in any of clauses 45-47.

[0124] Article 49. Each strand is a network device as described in any of the clauses 45 to 48, having a fixed bandwidth capacity between connection points on the strand.

[0125] Article 50. A network device as described in any of clauses 45-49, in which multipoint optical connections are implemented using breakout optics to connect the fabric ports of the network device to the fabric ports of each network node along the strand.

[0126] Article 51. Each strand supports multiple channels, and the multipoint optical connection is a reconfigurable multipoint optical connection that allows for dynamic channel reassignment between connection points on each strand, as described in any of the network devices in Clauses 45-48.

[0127] Article 52. The network device described in Clause 51, wherein a reconfigurable multipoint optical connection is implemented using breakout optics to connect the fabric ports of the network device to multiple ports of an optical circuit switch that connects along the strand to other network nodes.

[0128] Article 53. A network device as described in any of clauses 51 to 52, wherein multiple channels are implemented on their respective optical wavelengths by strands.

[0129] Article 54. A network device according to Clause 53, wherein a reconfigurable multipoint optical connection is implemented using optical couplers and splitters to merge and split optical wavelengths along a strand.

[0130] Article 55. The network device described in Clause 53, wherein a reconfigurable multipoint optical connection is implemented using an optical add-drop multiplexer (ROADM) to add and interrupt optical wavelengths along the strand.

[0131] Article 56. A network device according to Clause 53, wherein a reconfigurable multipoint optical connection is implemented in silicon photonics using a ring resonator to modulate and detect optical wavelengths along a strand.

[0132] Article 57. A network device as described in any of clauses 45 to 56, wherein the processing logic is capable of transmitting traffic demand information for the network device's inbound and outbound traffic to the control plane.

[0133] Article 58. A network device as described in Clause 57, wherein the processing logic is capable of receiving channel allocation information from the control plane based on traffic demand information.

[0134] Article 59. Transmitting traffic demand information for inbound and outbound network device traffic to a control plane that manages a network fabric having network nodes interconnected by strands according to harmonics, wherein each harmonic specifies the node distance between adjacent connection points on the corresponding strand, and each strand is implemented using reconfigurable multi-point optical connections that support multiple channels to enable dynamic channel reallocation between connection points on the strand. Based on traffic demand information, channel allocation information is received from the control plane, Configuring reconfigurable multi-point optical connections for network devices using channel assignment information, A method comprising distributing traffic destined for a destination node, which is routed to the destination node via a network fabric, to network nodes along strands connected to network devices.

[0135] Article 60. The method according to Clause 59, wherein configuring a reconfigurable multipoint optical connection includes changing one or more optical wavelengths of the reconfigurable multipoint optical connection.

[0136] The modules described herein may be software modules, hardware modules, or a preferred combination thereof. If a module is a software module, it may be embodied on a non-temporary computer-readable medium and processed by any processor of the computer systems described herein. It should be noted that the described processing and structures may be executed either in real time or in an asynchronous mode prior to any user interaction. The modules may be configured in the manner shown in Figure 13, and / or the functions described herein may be provided by one or more modules existing as separate modules, and / or the module functions described herein may be distributed across multiple modules.

[0137] Therefore, the specification and drawings should be considered illustrative rather than restrictive. However, it is clear that various modifications and changes can be made herein without departing from the broader intent and scope of the disclosure as set forth in the claims.

[0138] Other variations are within the scope of this disclosure. Thus, the disclosed technology may take on a variety of modifications and alternative structures, but its specific embodiments are shown in the drawings and described in detail above. However, it should be understood that this disclosure is not intended to be limited to one or more specific forms disclosed, but rather to encompass all modifications, alternative structures and equivalents that fall within the scope and spirit of this disclosure as defined by the appended claims.

[0139] The use of the terms “a,” “an,” and “the” and similar demonstrative pronouns in the context describing the disclosed embodiments (particularly in the context of the following claims) should be construed to encompass both singular and plural unless otherwise indicated herein or unless clearly contradicted by the context. The terms “equip,” “have,” “include,” and “contain” should be construed to be open terms (i.e., “include but not limited to”) unless otherwise stated. The term “connected” should be construed to mean contained within, attached, or joined together, either partially or whole, even if there is something intervening. The enumeration of ranges of values ​​herein is intended, unless otherwise indicated herein, simply as a shortened way of referring individually to each separate value that falls within the range, and each separate value is incorporated herein as if it were individually described herein. All methods described herein may be performed in any preferred order unless otherwise indicated herein or unless otherwise indicated by the context. The use of some or all of the examples or exemplary language (e.g., "etc.") provided herein is intended solely to illustrate embodiments of the disclosure in a favorable manner and, unless otherwise claimed, does not limit the scope of the disclosure. No language herein should be construed as indicating any unclaimed element essential for the implementation of the disclosure.

[0140] Unless otherwise specified, disjunctive language such as the phrase "at least one of X, Y, or Z" is intended to be understood in context as generally used to indicate that an item, term, etc., may be X, Y, Z, or any combination thereof (e.g., X, Y, and / or Z). Therefore, such disjunctive language is not generally intended, and should not be intended, to imply that a particular embodiment requires the presence of at least one X, at least one Y, or at least one Z, respectively.

[0141] Various embodiments of the Disclosure, including the best mode known to the inventors for carrying out the Disclosure, are described herein. Variations of those embodiments may become apparent to those skilled in the art by reading the foregoing description. The inventors expect that those skilled in the art will utilize such variations as needed, and the inventors intend that the Disclosure may be carried out in ways other than those specifically described herein. Accordingly, the Disclosure includes all modifications and equivalents of the subject matter described in the claims appended herein, as permitted by applicable law. Furthermore, unless otherwise indicated herein, or unless it is clearly inconsistent with the context, any combination of the above elements in all possible variations of the above elements is incorporated herein.

Claims

1. It is a computer network, A logical grid comprising rows and columns of network nodes, Each network node is connected to network nodes in the same row using a set of horizontal strands according to a set of horizontal harmonics, and each of the horizontal harmonics specifies the node distance along the row between adjacent connection points on the corresponding horizontal strand. The computer network wherein each network node is connected to network nodes in the same column using a set of vertical strands according to a set of vertical strands, and each vertical harmonic specifies the node distance along the column between adjacent connection points on the corresponding vertical strand.

2. The computer network according to claim 1, wherein each link between connection points on each strand in the set of horizontal strands and the set of vertical strands has a dynamically adjustable bandwidth capacity.

3. The computer network according to claim 2, wherein each strand has a maximum bandwidth capacity expressed as the number of channels available on the strand, and the bandwidth capacity of each link along the strand is adjusted by reallocating the number of channels available on the strand.

4. The computer network according to claim 1, wherein each link between connection points on each strand in the set of horizontal strands and the set of vertical strands has a fixed bandwidth capacity.

5. The computer network according to any one of claims 1 to 4, wherein each strand in the set of horizontal strands and the set of vertical strands connects up to the maximum allowable number of network nodes per strand.

6. The computer network according to claim 5, wherein the maximum number of network nodes allowed per strand is based on the number of channels supported by the strand.

7. The computer network according to claim 6, wherein the number of channels supported by the strand is based on the number of optical wavelengths supported by the strand.

8. The computer network according to any one of claims 1 to 7, wherein the set of horizontal strands connected to the network node has the same number of strands as the set of vertical strands connected to the network node.

9. The computer network according to any one of claims 1 to 8, wherein each horizontal harmonic in the set of horizontal harmonics is different from other horizontal harmonics in the set of horizontal harmonics, or each vertical harmonic in the set of vertical harmonics is different from other vertical harmonics in the set of vertical harmonics.

10. The computer network according to any one of claims 1 to 9, wherein the set of horizontal harmonics includes at least one prime number greater than 2, or the set of vertical harmonics includes at least one prime number greater than 2.

11. The computer network according to any one of claims 1 to 9, wherein the set of horizontal strands includes horizontal strands extending at least the entire length of the row, and the set of vertical strands includes vertical strands extending at least the entire length of the column.

12. A computer implementation method for routing network traffic between source nodes and destination nodes in a network fabric, which includes network nodes interconnected by strands, according to harmonics that specify the node distance between adjacent connection points on the corresponding strands, Distributing the aforementioned network traffic from the source node to a set of nodes in the vicinity of the source node (one hop away), For each one-hop neighborhood of the source node within the set of one-hop neighborhoods of the source node, Setting the vicinity of the source node in the one-hop vicinity as the first waypoint, Identifying the one-hop vicinity of the destination node that is mapped to the one-hop vicinity of the source node, Setting the identified one-hop vicinity of the destination node as a second waypoint, Routing the network traffic distributed from the first transit point to the second transit point via the network fabric, The computer implementation method, including the above.

13. The computer implementation method according to claim 12, wherein the distributed network traffic is routed using equal-cost multipath routing (ECMP) when the traffic demand between the source node and the destination node is below a threshold, and weighted-cost multipath routing (WCMP) when the traffic demand is above the threshold.

14. The routing of the distributed network traffic is performed by an optimizer that determines the amount of traffic to be carried on each path, and the optimizer, A traffic demand matrix including traffic information between each pair of network nodes in the network fabric, The connection topology of the network fabric including the harmonics, A computer implementation method according to any one of claims 12 to 13, which receives a set of inputs including valid paths to each pair of network nodes in the network fabric.

15. The set of inputs to the optimizer further includes the maximum number of channels supported per strand and the bandwidth capacity per channel, The computer implementation method according to claim 14, wherein the optimizer further determines the channel assignment on the links between connection points on each strand.

Images