Distributed processing architecture

Abstract

Embodiments of the present disclosure include techniques for processing neural networks. Various forms of parallelism can be achieved using a topology of combined processor sequences. In one embodiment, the present disclosure includes a computer system comprising one or more processor groups, each processor group comprising a plurality of processors. A plurality of network switches are coupled to a subset of the plurality of processor groups. In one embodiment, the switches can be optical network switches. A subset of the processors in a processor group can be configured to form a sequence, and the network switches can be configured to form at least one sequence spanning one or more processor groups of the plurality to perform neural network computations. Various configurations for creating Hamiltonian cycles are disclosed to support data parallelism, pipelined parallelism, layer parallelism, or a combination thereof.

Description

Background Technology

[0001] This disclosure relates to distributed computing, and more particularly to distributed processing architectures that can be used to process artificial neural networks.

[0002] In summary, artificial neural networks (hereinafter referred to as neural networks) are becoming increasingly important in artificial intelligence applications and modern computing. Figure 1 An example neural network is shown. Neural network 100 receives input values ​​corresponding to the features to be identified. The input values ​​are multiplied by weights (represented by edges 101) and summed (e.g., summed) at node 102. An activation function is applied to the results at node 102 to generate output values. Values ​​are combined across multiple nodes and node layers to produce network output values ​​corresponding to the results.

[0003] Such systems “learn” to perform tasks by considering examples, typically without being programmed using task-specific rules. Initially, the weights may be untrained. During the training phase, the network processes input values ​​corresponding to known outcomes and compares the differences (or errors) between network output values ​​to known values. The weights can be adjusted based on the error using a process called backpropagation, where computation flows in the opposite direction (e.g., from output to input). Training can involve continuously adjusting the weights across many input samples and their corresponding known network output values. This is typically referred to as the training phase. Once trained, the system can receive inputs and produce meaningful results (e.g., classification or recognition). This is typically referred to as the inference phase.

[0004] Training and / or inference for very large neural networks can involve massive amounts of computation. Therefore, computation for large networks can be partitioned across many processors. However, due to the large number of connections in the network, partitioned network computation can involve significant amounts of potential data communication (partial results) between processors. This connectivity may require multiple processors coupled together, for example, via potentially multiple network switches. Providing full bandwidth for arbitrary communication patterns can become prohibitively expensive as scale and high bandwidth increase. For example, the cost of components and the energy consumption of electrical switches and optical transceivers required to cover long distances in a data center can be disadvantageous. Summary of the Invention

[0005] Embodiments of this disclosure include techniques for processing neural networks. Various forms of parallelism can be achieved using topologies that combine processor sequences. In one embodiment, this disclosure includes a computer system comprising multiple processor groups, each processor group comprising multiple processors. Multiple network switches are coupled to subsets of the multiple processor groups. In one embodiment, the switches may be optical switches. The subsets of processors in the processor groups may be configured to form sequences, and the network switches may be configured to form at least one sequence spanning one or more of the multiple processor groups to perform neural network computations. Various alternative configurations for creating Hamiltonian loops are disclosed to support data parallelism, pipelined parallelism, layer parallelism, or combinations thereof. In some embodiments, the network switches may be optical switches. Attached Figure Description

[0006] Figure 1 An example neural network is shown.

[0007] Figure 2 A neural network partitioned according to an embodiment is shown.

[0008] Figure 3 The neural network communication across three axes is illustrated.

[0009] Figure 4 The following is illustrated: partitioning along three dimensions according to an embodiment. Figure 2 The neural network.

[0010] Figure 5 An example of a plurality of processors that can be configured to form a sequence according to an embodiment is shown.

[0011] Figure 6 An example network switch coupled to multiple processor groups according to an embodiment is shown.

[0012] Figure 7 Another example topology of the processor group and network switch according to the embodiment is shown.

[0013] Figure 8 Another example topology of the processor group and network switch according to the embodiment is shown.

[0014] Figure 9 Another example neural network partitioned according to an embodiment is shown.

[0015] Figure 10 A three-dimensional representation of the partition is shown to illustrate nearest neighbor communication of the processor according to an embodiment.

[0016] Figure 11 Another configuration of the processor in the processor group according to an embodiment is shown.

[0017] Figure 12 The configuration of a cluster of multidimensional processor groups according to an embodiment is shown.

[0018] Figure 13 An example processor group according to another embodiment is shown.

[0019] Figure 14 An example network switch and processing group according to another embodiment are shown.

[0020] Figure 15 A two-layer arrangement of a network switch according to an embodiment is shown.

[0021] Figure 16 An example topology of a cluster along a first cluster dimension is shown according to an embodiment.

[0022] Figure 17 An example topology of a cluster along a second cluster dimension is shown according to an embodiment.

[0023] Figure 18 An example two-dimensional cluster topology according to an embodiment is shown.

[0024] Figure 19 Another example two-dimensional cluster topology according to an embodiment is shown.

[0025] Figure 20 An example optical network switch coupled to multiple processor groups is shown according to an embodiment.

[0026] Figure 21 An example of multiple optical network switches coupled to multiple processor groups according to an embodiment is shown.

[0027] Figure 22 Another example of multiple optical network switches coupled to multiple processor groups according to an embodiment is shown.

[0028] Figure 23 Another example of multiple optical network switches coupled to multiple processor groups according to an embodiment is shown.

[0029] Figure 24 An example of multiple optical network switches coupled to multiple processor groups via an intermediate optical network switch is shown according to an embodiment.

[0030] Figure 25 Another example two-dimensional cluster topology including an optical network switch is shown according to an embodiment.

[0031] Figure 26 Another configuration of the processor in the processor group according to an embodiment is shown.

[0032] Figure 27A Another configuration of the processor in the processor group according to an embodiment is shown.

[0033] Figure 27B This illustrates yet another configuration of the processors in a processor group according to an embodiment. Detailed Implementation

[0034] In the following description, numerous examples and details are set forth for illustrative purposes in order to provide an understanding of various embodiments. However, it will be apparent to those skilled in the art that certain embodiments may be practiced without some of these details, or that certain embodiments may be practiced with modifications or equivalents thereof.

[0035] Figure 2 An artificial neural network 200 partitioned according to an embodiment is illustrated. The features and advantages of this disclosure utilize communication patterns in the neural network to generate a more efficient topology for distributed computing of the neural network. For example, the neural network 200 includes seven layers 201-207, including an input layer 201 and an output layer 202. During inference, input values ​​can be received at the input layer 201, and results are produced at the output layer 207. At each layer, data values ​​are multiplied by weights, summed, and processed by each node using an activation function. During training, data flows in the opposite direction, where the error between the network result and the expected result is determined at the output, and weights are updated layer by layer from layer 207 to layer 201.

[0036] In various embodiments, it may be desirable to partition the neural network so that computations of different partitions of the network can be performed on different processors. Example neural network 200 is a simplified network illustrating communication between partitions. In this example, the network can be partitioned along a pipeline (denoted as Pi) or along pipeline layers (denoted as Li). In this example, network 200 is partitioned into four pipeline stages P1-P4 and two layers L1-L2. Nodes and edges in partitions L1 and P1 can be processed on one processor, while nodes and edges in other partitions can be processed on other processors respectively. However, in order to perform their computations, each processor must receive some data input and send data output to other processors. For example, node 1 in partition L1 and P1 can receive input, multiply the input by weights, apply an activation function, and send the output to node 2 in partition L2 and P1. Similarly, node 3 in L2 and P1 sends output data to node 4 in partition L1 and P1. Therefore, when layer 201 is partitioned across multiple processors, there is intra-layer communication, for example, that occurs between processors. Similarly, pipeline communication can be as follows. The output of node 4 in partitions L1 and P1 is sent to the processor assigned to partitions L1 and P2 as input to node 5, and the output of node 2 in partitions L2 and P1 is sent to the processor assigned to partitions L2 and P2 as input to node 6. Similarly, the output of node 4 in partitions L1 and P1 is sent to the processor assigned to partitions L2 and P2 as input to node 6, and the output of node 2 in partitions L2 and P1 is sent to the processor assigned to partitions L1 and P2 as input to node 5. Therefore, it can be seen that, for example, pipeline stage P1 communicates with pipeline stage P2, pipeline stage P2 communicates with pipeline stage P3, but pipeline stage P3 may not need to communicate with pipeline stage P1.

[0037] Figure 3 This illustrates neural network communication across three axes. In this example, the neural network model is partitioned for layer parallelism, pipeline parallelism, and data parallelism. (The above text is incomplete and requires further context.) Figure 2 Layer parallelism and pipeline parallelism are described in the document. Data parallelism allows data to be exchanged between multiple instances of a model executed on different processors. For example, the same model can be trained in parallel by executing different instances of the model on different processors and periodically synchronizing the weights in each layer.

[0038] In this example, M (integer) instances of the model receive input on layers (0, 0)–(M-1, 0) and produce output on layers (0, N-1)–(M-1, N-1) during inference. For training, data flows in the opposite direction. Thus, pipelined communication flows vertically as shown by arrow 350 (e.g., activation / error). For example, layers 311 and 321 can be partitioned to run on different processors as described above. Here, intra-layer communication between J (integer) processors is shown by arrow 351, where layers 311 and 321 are partitioned into, for example, J partitions. Similarly, during training, the weights in each model can be periodically synchronized as described above. If instances of the neural network model run on different processors (or distributed across multiple sets of different processors), such processors can perform communication to update weights (e.g., using all-reduce algorithms, such as one-dimensional rings, multi-rings, tree-based algorithms, or layer algorithms). A pipeline can operate independently during training, but may need to periodically synchronize its weights due to execution across different instances of the model. For example, during full reduction, weights can be averaged layer by layer across instances. Therefore, in some applications, each layer of the pipeline may periodically perform full reduction along with other instances of the model, such as... Figure 3 As shown by arrow 353, instances of models running in parallel can be configured in a ring, where weight updates between models can flow in both directions around the ring.

[0039] from Figure 2 and Figure 3 In this context, three forms of parallelism exist: data parallelism (e.g., summation, averaging, full reduction), pipeline parallelism, and intra-layer (or simply “layer”) parallelism. These forms of potential parallelism allow for the development of architectures that leverage one or more available parallelisms when performing neural network computations. Various embodiments of this disclosure utilize the understanding that, with certain exceptions, communication dependencies in all three dimensions (intra-layer, pipeline, data) can be mapped (e.g., in the absence of congestion) to communication patterns following a coupled one-dimensional loop structure. Therefore, if the communication patterns are mapped to a network topology, they can be modeled as a set of Hamiltonian loops connecting each dimension. In some example embodiments, the processor sequences can be logically arranged in a three-dimensional Cartesian space. Thus, as described in more detail below, a Hamiltonian loop can refer to, for example, multiple processors configured in series across one or more processor groups via at least one network switch for intra-layer parallelism, pipeline parallelism, data parallelism, or various combinations thereof.

[0040] For example, Figure 4 The partitions along three dimensions are shown. Figure 2The neural network consists of layers (L), pipeline stages (P), and data. In this example, model 200 is processed three times in parallel using instances A, B, and C (data parallelism). The partitions of model 200 can be mapped to... Figure 4 The processing sequence is illustrated. For example, computations of network partitions L1, P1 and L1, P2 can be performed on different processors configured in the sequence to form Hamiltonian loop 401. Similarly, computations for pipeline stages P1-P2-P3-P4 can be performed on multiple processors configured as a sequence to form Hamiltonian loop 402. Finally, for example, computations along the data axis ABC (e.g., full reduction) can be performed on a sequence of multiple processors forming Hamiltonian loop 403. It should be understood that in various example embodiments, the neural network can be partitioned along a data axis, pipeline axis, layer axis, or a combination thereof. For example, processors can be configured to form a sequence, and processors can be configured to process multiple local layers of a partitioned neural network, one or more pipelines of a partitioned neural network, data parallelism (e.g., full reduction) between models of the neural network, and / or combinations thereof.

[0041] Figure 5 An example of multiple processors configurable to form a sequence according to an embodiment is shown. In this example, four (4) processors PR1 501, PR2 502, PR3 503, and PR4 504 are configured serially to form a one-dimensional (1D) array. For example, processors 501-504 may form a processor group 550, and processors 501-504 may be configured as a sequence to process a partitioned neural network. Various embodiments of this disclosure may use various processor types, including but not limited to, x86 architecture processors, ARM architecture processors, graphics processing units (e.g., GPUs), or dedicated artificial intelligence (AI) processors optimized for performing neural network computations.

[0042] Here, processor group 550 includes edge processors 501 and 504 with edge ports 510 and 511. In this example, edge (or boundary) processor 501 is located in the first position in the array and is coupled to port 510 and processor 502. Processor 502 is coupled to adjacent processors 501 and 503, processor 503 is coupled to adjacent processors 502 and 504, and edge (or boundary) processor 504, located in the last position, is coupled to adjacent processor 503 and edge port 511. Edge ports 510 and 511 can be coupled to a network switch, for example, so that multiple processor groups 550 can be coupled together to form a longer sequence to process more partitions of the neural network.

[0043] Figure 6An example network switch 610 coupled to multiple processor groups according to an embodiment is shown. The network switch can be configured to form a sequence across one or more processor groups in the processor group to perform neural network computations. In this simplified illustrative example, four (4) processor groups 620, 630, 640, and 650 each include four (4) processors configured as a one-dimensional array. Each group includes edge ports of ports coupled to the network switch 610, such that different groups can be coupled together to form a longer sequence. For example, if the network is partitioned along any of the three axes (e.g., layers, pipelines, data), the partitions can be loaded into sequential processors, such as processors 621-624 in processor group 620. In some embodiments, the sequence can form a closed loop, for example, such that processors 624 communicate with 621. If partitioning necessitates a longer sequence of processors, the switch 610 can be configured to handle the longer sequence of processors. For example, if 5 to 8 sequential processors are required, port 626 from processor 624 (which is coupled to port 6 of network switch 610) can be coupled to port 635 from processor group 630 (which is coupled to port 7 of network switch 610 for processor 631). Therefore, it is possible to use... Figure 6 The topology shown cascades up to 16 processors together. As further explained below, in one embodiment, switch 610 may be, for example, an optical network switch.

[0044] Figure 7 Another example topology of processor groups and network switches according to an embodiment is shown. In this example, three (3) processor groups 711-713 are coupled to network switch 701, three (3) processor groups 721-723 are coupled to network switch 702, three (3) processor groups 731-733 are coupled to network switch 703, and three (3) processor groups 741-743 are coupled to network switch 704. Each of the processor groups 711-713, 721-723, 731-733, and 741-743 may include, for example... Figure 5The diagram shows multiple processors that can be configured as a sequence. In this example, a greater number of processor sequences can be obtained through network switches 701-704 that are directly coupled together. For example, if each processor group comprises five (5) processors, a sequence of twenty (20) processors can be obtained by coupling groups 711-713 with group 721. In this case, switch 701 is configured to connect ports 7 and 1, ports 2 and 3, ports 4 and 5, and ports 6 and 9, and switch 702 is configured to connect ports 1 and 4, and ports 5 and 2. Ports 5 / 2 of switch 702 can be routed back to ports 1 / 7 on switch 701 via switch 703, which can, for example, connect ports 7 and 8 to close a loop for a dedicated, congestion-free nearest neighbor processor communication loop. Figure 7 The topology shown, for example, can be configured with up to sixty (60) processors in the sequence to perform neural network computations along one of the three axes mentioned above. In one embodiment, switches 701-704 may be, for example, optical network switches.

[0045] Figure 8 Another example topology of processor groups and network switches according to an embodiment is shown. In this example, network switches 801-804 are coupled together via an intermediate (also known as a core or Layer 1) network switch 805. Therefore, one or more processor groups 811-813 can be configured, for example, to form a sequence with any one or more other processor groups 821-823, 831-833, and / or 841-843 via directly connected (also known as Layer 0) switches 801-804 and via the intermediate network switch 805. In various embodiments, switches 801-805 may be, for example, optical network switches.

[0046] Figure 9 Another example neural network 900 partitioned according to an embodiment is shown. The neural network 900 comprises a pipeline of nine (9) layers, each containing nine (9) nodes. For clarity, most edges are omitted, but it should be understood that edges couple the nodes together. In the example shown here, the network 900 is partitioned along layer axes (e.g., L1, L2, L3) and pipeline axes (e.g., P1, P2, P3), where computations for each partition are performed on different processors (e.g., PR11-PR33).

[0047] Figure 10 A three-dimensional representation of the partitions is shown to illustrate sequential communication between the processors. In this example, computation for each partition can be performed on processors PR11-PR33 forming a 3×3 array (see...). Figure 9 and 11Execution is performed on the network. Additionally, each model can be executed four (4) times (e.g., data parallelism A, B, C, D) to form a 3×3×4 array. Embodiments of the invention can map neural network partitions to a processor array for execution, wherein the processors are configured to (e.g.) execute Hamiltonian loops. In this example, along the layer axis, the Hamiltonian loop can capture communication between L1, L2, and L3 during the computation of the first layer P1 of the execution pipeline, as shown at 1001. Along the pipeline axis, the L1 layer computed during P1 can communicate with the L1 layer computed during P2, and subsequently with the L1 layer during P3 to form another Hamiltonian loop, as shown at 1002. Along the data axis, the Hamiltonian loop can capture communication between layers executing across multiple models AD (e.g., performing full reduction operations to adjust the weights of each model), as shown at 1003.

[0048] Figure 11 Another configuration of the processors in processor group 1100 according to an embodiment is shown. In this example, processors 1140-1148 are configured as rows and columns to form a two-dimensional processor array. Here, the two-dimensional processor array (e.g., PR11-PR33) is configured as a 3×3 matrix to form processor group 1100, wherein processors 1140-1148 are coupled to nearest-neighbor processors in an N-dimensional processor array to support the formation of a sequence of processors to perform, for example, Hamiltonian loops. Processor group 1100 includes edge processors 1140-1147, each edge processor 1140-1147 having one or more edge ports. Ports 1101-1106 are coupled to processor rows in processor group 1100, while ports 1110-1115 are coupled to processor columns in processor group 1100. Therefore, Figure 9 Each partition of the neural network 900 can be loaded into a different processor for performing neural network computations on the layers and pipelines of the network 900, where the layers of each pipeline stage can be computed first, followed by computations on each successive pipeline stage. Intra-layer data used to compute each layer can flow vertically across processor columns (e.g., PR11-PR21-PR31). As each layer in each partitioned pipeline stage P1, P2, P3 completes, the results flow horizontally across processor rows (e.g., PR11-PR12-PR13).

[0049] Figure 12A configuration of a multidimensional cluster array 1200 of processor groups according to an embodiment is shown. In this example, the two-dimensional cluster 1200 of processor groups 1201, 1202, 1210, and 1211 can be used to support larger neural networks, more partitions, or the aforementioned partitions, and multiple other executions of model 900 by processor groups that can be combined, for example, via full reduction. For example, larger networks may require more sequences of processors to implement Hamiltonian loops along each axis. For example, Figure 12 The illustrated architecture allows rows or columns in different processor groups to be configured as larger sequences. Specifically, during the execution of neural network computations, a first subset of processors (e.g., rows in processor group 1201) and a second subset of processors (e.g., rows in processor group 1210) can be coupled together to form a sequence comprising a processor string (e.g., A11, A12, A13, B11, B12, B13), wherein each processor communicates with its adjacent processor in the processor string. As described above, in various embodiments the sequence can form a closed loop, for example, such that A11 communicates with B13.

[0050] The network switch topology is configured to couple rows and / or columns of processors between processor groups to produce a processor string for performing neural network computations. In the various embodiments disclosed herein, processor groups along a specific dimension of the cluster array have edge processor ports coupled to corresponding network switches of the same dimension. In this example, processors at the first and last positions along each dimension are coupled to row or column network switches. For example, processor group 1201 includes a row of processors A11, A12, A13, where the edge processor ports of A11 and A13 are coupled to row network switch 1221. Additionally, processor group 1210 includes a row of processors B11, B12, B13, where the edge processor ports of B11 and B13 are coupled to row network switch 1221. Similarly, processor group 1201 includes a row of processors A21, A22, and A23, wherein the edge processor ports of A21 and A23 are coupled to row network switch 1222, and processor group 1210 includes a row of processors B21, B22, and B23, wherein the edge processor ports of B21 and B23 are coupled to row network switch 1222. For example, the third row of processors in groups 1201 and 1210 are also coupled to row network switch 1223. Thus, for example, row switches 1221-1223 form a switch layer corresponding to the processor row in each processor group, which can be used to connect a specific processor row to multiple switch groups.

[0051] The processor columns in each processor group 1201 and 1202 can be similarly coupled together across the first column of cluster 1200. As described for rows in the example above, the edge processor ports on opposite sides of the columns of processors A11, A21, A31 and C11, C21, C31 on processor groups 1201 and 1202 can be coupled to column network switch 1231. Processors Ax2 / Cx2 and Ax3 / Cx3 (x = 1..3) in other columns can, for example, be coupled from edge processor ports to column network switches 1232 and 1233, respectively. Therefore, column switches 1231-1233 form a switch layer corresponding to the processor columns in each processor group, which can be used to connect specific processor columns across multiple switch groups.

[0052] Similarly, as described above for the first row of the cluster, the second row of cluster 1200, including processor groups 1202 and 1211, can be coupled to a multi-row network switch 1225. Finally, as described above for the first column of the cluster, the second column of cluster 1200, including processor groups 1210 and 1211, can be coupled to a multiple-column network switch 1235. As described in more detail below, for example, multiple intermediate switches can be used to link the row and column switches to allow more processors to be configured in a string to compute longer Hamiltonian cycles.

[0053] Although Figure 9-12 The above example illustrated shows an example neural network partitioned to suit a processor group, but different networks can be partitioned in various ways to operate on a network topology according to this disclosure. While various partitions can more efficiently utilize the available resources in a given topology, there is no strict relationship between network partitions that can be used, for example, by a given number of processors in a processor group or a given number of groups in a cluster. For example, each processor can perform computations on one, multiple, or partial layers of a neural network model. Similarly, each processor can perform computations on one pipeline stage, multiple pipeline stages, or a portion of a pipeline stage. Finally, for example, each processor can perform computations on the entire neural network model (all layers and all pipelines) and synchronize the weights on the processor, or multiple groups of processors performing operations on partitioned neural networks can synchronize the weights. In various embodiments, switches 1221-1223, 1225, 1231-1233, and 1235 can be, for example, optical network switches.

[0054] Figure 13An example processor group 1300 according to another embodiment is shown. In this example, a 4×4 processor array PR11…PR44 is arranged to form processor group 1300. Here, each processor has 12 input / output (I / O) communication ports, which may be, for example, Ethernet ports configured in parallel to extend connection bandwidth (e.g., 50Gbps Ethernet). Three (3) I / O ports are coupled to adjacent (e.g., nearest neighbor) processors or network switches, as described in more detail below. Processor group 1300 includes edge processors 1320-1331 with edge processor ports 1301-1308 and 1310-1317, which couple each edge processor to, for example, a specific network switch.

[0055] Figure 14 An example network switch and processing groups according to another embodiment are shown. In this example, N processing groups 1401-1402 are coupled to multiple switches 1410-1412 configured in 12 switch planes, where N is an integer. For example, N could be 24 groups. Figure 13 A 4×4 processor array. As described above, each processor group can have rows and columns. Each row can have 3 edge processor ports, for example, for a total of 24 row processor ports (3 ports on each side (×2) of each of the 4 rows (×4); 3×4×2=24). In this example, the edge processor ports on the opposite side of a row of processors across the first plurality of processor groups are coupled to a row network switch. For example, the edge processor ports on the opposite side of the first row of processors in processor groups 1401-1402 are coupled to network switch 1410. Similarly, the edge processor ports on the opposite side of the second row of processors in processor groups 1401-1402 are coupled to network switch 1411, while the edge processor ports on the opposite side of the last row of processors in processor groups 1401-1402 are coupled to network switch 1412. In this figure, only 3 ports are shown for illustrative purposes, but it should be understood that, Figure 13 As shown in the example, there are 3 ports per row and 3 ports per column. Therefore, in this example, there are 12 switch planes connecting 4 rows of switches across 24 processor groups. Similarly, for each of the 24 processor groups, the edge processor ports on opposite sides of processor columns spanning a second or more processor groups are coupled to column network switches. Thus, for example, there are 12 network switches used to connect to… Figure 14 The rows shown are coupled in essentially the same manner to the processor columns in each of the 24 processor groups. As described above and as will be further described below, switches 1410-1412 may be, for example, optical network switches.

[0056] Figure 15 A two-layer arrangement of network switches according to an embodiment is illustrated. Features and advantages of this disclosure include a topology that allows a subset of processor groups to be coupled together across different network switches (e.g., to form a Hamiltonian cycle). In this example, Layer 0 network switch 1501 is coupled to processor groups 1510A-C, and Layer 0 network switch 1502 is coupled to processor groups 1510D-F. Switches 1501 and 1502 are further coupled to a Layer 1 intermediate network switch 1503. Each connection between the Layer 0 and Layer 1 switches may include two links (e.g., two ports), for example, one for a non-congested data uplink and another for a non-congested data downlink. Figure 15 The topology shown can be applied to Figure 14 A switch plane. For example, similar topologies can be used for other switch planes. In some configurations, a subset of processor groups are coupled together via Layer 0 switches. In other configurations, an intermediate Layer 1 switch 1503 can be used to couple multiple Layer 0 switches together to form a processor sequence longer than the sequence of processors that any particular Layer 0 switch can support. For example, if a sequence of 150 processors is required, processor groups from two switches can be combined (e.g., 24 groups × 4 processors / group × 2 > 150). By cascading the Layer 0 switches and associated processor groups together, longer sequences can be produced; for example, this can form non-congestion loops across multiple Layer 0 and Layer 1 switches. In some example embodiments, switches 1501-1503 can be, for example, optical network switches.

[0057] Switches 1501 and 1502 could be, for example, 64-port switches. Therefore, in this example, 48 ports are used for processor group connections, leaving 16 ports to support inter-switch traffic for 24 groups. Because more switch ports are used for local connections and only 16 are available for inter-switch traffic, the Layer 0 switches in this example topology are referred to as over-subscribed (e.g., all processor groups cannot access the Layer 1 switches simultaneously). Another advantage of some example implementations is that network switches can be over-subscribed without impacting performance because processor groups can be locally configured to sequence before multiple switches are needed. For example, one port of processor group 1510A can be coupled to one port of processor group 1510B, another port of processor group 1510B can be coupled to one port of processor group 1510C via an additional port and processor, and a second port of processor group 1510C can be coupled back to another port of processor group 1510A (e.g., to form a closed loop), or coupled to other Layer 0 switches and processor groups via a link to switch 1503. Because processor groups can typically be configured to form local sequences, there are usually enough ports remaining in an oversubscribed switch to establish communication with other processor groups in other switches to perform the desired computations.

[0058] Figure 16 An example topology of a cluster of processing groups along a first cluster dimension is shown according to an embodiment. In this example, the cluster comprises an N×M two-dimensional array (e.g., 24×24) of processor groups coupled together by a 2-layer switching topology. Figure 16 A switch plane (e.g., a switch topology for one edge port in a row of each processor group) is shown. Other planes for other edge ports in other rows can have substantially similar implementations. In this topology, rows of processor groups in the cluster have edge processor ports coupled to the corresponding network switches in the same row. For example, processor groups 1610-1612 (e.g., processor groups in the first row of the cluster) have edge processor ports coupled to the same row network switch (e.g., network switch 1601 in row 1 of the cluster) (e.g., two per group). Similarly, for example, processor groups 1613-1615 in the second row of the cluster are coupled to row network switch 1602, while processor groups 1616-1618 in the last row of the cluster are coupled to row network switch 1603.

[0059] The row network switches 1601-1603 of the cluster can be coupled together via one or more intermediate Layer 1 row network switches 1620-1621. In this example, each row network switch 1601-1603 includes multiple connections to multiple intermediate row network switches. For example, row network switches 1601-1603 may all include connections to two intermediate row switches 1620-1621. Thus, for example, each row of the processor group can establish a channel to one or more other rows of the processor group via intermediate row switch 1620 or intermediate row switch 1621 to create Hamiltonian loops across one or more rows. In this example, the link between Layer 0 and Layer 1 can be a single full-duplex Ethernet link used for uplink and downlink traffic without congestion. For example, even longer Hamiltonian loops can be obtained by connecting additional clusters to Layer 1 switches 1620 and / or 1621 using the remaining 40 ports. In some embodiments, intermediate row network switches can be coupled to intermediate column network switches, as described below.

[0060] Figure 17 An example topology of a cluster of processing groups along a second cluster dimension is shown according to an embodiment. In this topology, columns of processor groups in the cluster have edge processor ports coupled to corresponding network switches in the same column. For example, processor groups 1610, 1613, and 1616 (e.g., processor groups in the first column of the cluster) have edge processor ports coupled to network switches in the same column (e.g., network switch 1701 in column 1 of the cluster). Similarly, for example, processor groups 1611, 1614, and 1617 in the second column of the cluster are coupled to column network switch 1702, while processor groups 1612, 1615, and 1618 in the last column of the cluster are coupled to column network switch 1703. For example, column switches 1701-1703 can be coupled to substantially the same network switches as... Figure 16 The intermediate layer 1 column of switches 1920 is shown.

[0061] Figure 18 Another example of a multidimensional cluster array according to an embodiment is shown. Figure 16 and 17 Topologies can be combined to form, for example... Figure 18The diagram illustrates a two-dimensional cluster switching topology. For all switch planes, 12 row network switches are available per row (e.g., 12 switches / row × 24 rows = 288 row network switches), and 12 column network switches are available per column (e.g., 12 switches / column × 24 columns = 288 column network switches). The intermediate row network switches for Layer 1 are shown at 1890, and the intermediate column network switches for Layer 1 are shown at 1891. In some embodiments, processor groups in various columns and rows can be indirectly cross-coupled together via, for example, the intermediate row network switches and column network switches shown at 1892. Figure 18 In the illustrated embodiment, each switch plane includes two Layer 1 switches for columns and rows, and there are 12 switch planes (12 edge row ports and 12 edge column ports), so there are a total of 12 planes × 2 Layer 1 switches / planes × 2 (columns and rows) = 48 Layer 1 switches. Figure 18 The example configuration shown has 24×24 groups (9216 processors) and 24 planes (e.g., 12 rows and 12 columns). For each plane, there are 24 Layer 0 switches, each with 16 uplinks. Each uplink connects to a Layer 1 switch. In one example embodiment, a 64-port switch can be used. Therefore, 8 Layer 1 switches can be used (e.g., then the 16 ports per Layer 1 switch can be omitted), resulting in, for example, a total of 24 planes * 8 Layer 1 switches = 192 switches.

[0062] Although Figure 13-18 Example topologies are shown, but it should be understood that these diagrams are merely illustrative. Various numbers of processors can be used in different processor groups, various subsets of these groups can be coupled to network switches, and various cluster sizes and configurations can be used in various embodiments of this disclosure. Furthermore, for any particular hardware topology (e.g., the number of processors in a processor group, the number of processor groups leading to a switch, and / or the number of processor groups in a cluster), any particular topology (e.g., the number of processors in a processor group, the number of processor groups leading to a switch, and / or the number of processor groups in a cluster) can be used. Figure 13-18 The topology shown can be used to compute various neural network sizes and partitions. For example, for a given hardware configuration, smaller or larger neural networks can be computed, which can be adapted to the topology in different ways. For example, a particular hardware topology can handle neural networks with different numbers of layers, different layer partitions, different numbers of pipeline stages, different pipeline stage partitions, and different data volume parallelism. In various embodiments, for example, the techniques described herein can be used to map network models of different sizes to various different processors and sequences of processors in one or more dimensions. The above is combined with... Figure 16-18 The described switch could be, for example, an optical network switch.

[0063] Figure 19Another example two-dimensional cluster topology according to an embodiment is shown. In some embodiments, the middle (e.g., layer 1) row network switches and the middle column network switches in the cluster may be coupled together. In this example, processor groups 1901-1916 are configured in a manner similar to... Figure 12 and 18 In a multidimensional (e.g., two-dimensional) cluster array, for example, processor groups 1901-1916 along a specific dimension of the cluster array have edge processor ports coupled to corresponding network switches of the same dimension. In this example, rows of processor groups 1901-1916 in the cluster array have edge processor ports coupled to corresponding network switches 1920-1923 of the same row, while columns of processor groups in the cluster array have edge processor ports coupled to corresponding network switches 1930-1933 of the same column. Figure 12 and 18 An example of such a connection is shown, but omitted here. In various embodiments, one or more row network switches 1920-1923 may be coupled to one or more column network switches 1930-1933. In this example, row network switch 1920 is coupled to column network switches 1930 and 1931, row network switch 1921 is coupled to column network switches 1931 and 1932, row network switch 1922 is coupled to column network switches 1932 and 1933, and row network switch 1923 is coupled to column network switches 1930 and 1933. This advantageously improves fault tolerance or can support mapping of Hamiltonian cycles across dimensions (e.g., when mapping processor sequences larger than a single dimension together). As described above, indirect cross-dimensional connections can be made between the intermediate (layer 1) row network switches and column network switches as described above.

[0064] Scheduling and Faults

[0065] Some embodiments of this disclosure can differentiate jobs based on job size (e.g., between small and large jobs). For example, a small job can be defined as a job suited to a processor group. For example, such a job may only be scheduled to a processor group that accommodates other small jobs. Large jobs can be distributed across multiple processor groups. In this case, the minimum increment for allocation may be, for example, a single processor group. For example, if the number or geometry of nodes required by a job does not perfectly fit the multiple nodes in a processor group, the nodes may be unused and merely forward traffic.

[0066] Various implementations can route numerous configurations of Hamiltonian cycles and transparently replace processor groups when a failure occurs. For example, in a one-dimensional configuration, the system can swap any and any two processor groups within the same switch, for instance, if enough uplinks can be maintained to sustain the sequence across other switches. In higher-dimensional configurations, the system can swap rows and columns (or contiguous blocks) together if enough links are available. This can simplify finding new processor groups, for example, in the event of a failure or when processing segment allocations after a period of operation.

[0067] routing

[0068] Once the logical portion of the network has been allocated to processor groups and processors, the Hamiltonian cycle can be routed. The cycle can have a constant bandwidth guarantee, reserving at least a portion of the links for communication between processors. For example, the various embodiments described above for one-dimensional, two-dimensional, or more dimensions may include processor closed loops with dedicated, non-congested nearest-neighbor processor communication loops across one or more processor groups and switches. The network can be advantageously optimized to allocate 100% of the link bandwidth to the sequence of processors used in the Hamiltonian cycle. The implementation of bandwidth reservation and routing will depend on the network technology. Simple policy-based routing schemes or more complex schemes based on ECMP routing can be used to establish routes, while QoS buffers and bandwidth reservations can be used to guarantee a certain rate.

[0069] To utilize a global view of the network for routing Hamiltonian cycles, the network manager can allocate available links to cycles and then mark them as used. Routing can then be programmed to use only those cycles. If global traffic is required and links are unavailable, a QoS scheme can allocate a percentage of each Hamiltonian cycle to implement that traffic. For example, if the local topology is longer than two processors in any dimension, some processors can forward global traffic to internal processors. This can be accomplished using simplification mechanisms, as the number of hops in the local topology can be controlled by its size.

[0070] Example of using an optical network switch

[0071] As described above, certain embodiments of this disclosure can be advantageously implemented using optical network switches. Optical network switches according to various embodiments can be used to maintain channels between one or more processor groups to form a sequence. Optical network switches according to various embodiments can be channel-switched, while electrical network switches according to other embodiments can be packet-switched. The difference between packet switching and channel switching is that packet switching can change the switching configuration based on information in each packet. Channel switching can be established between a source and a destination (e.g., two processors in the same or different processor groups) and maintained during a communication cycle. For example, channel switching can dedicate some or all of the channel bandwidth to communication between the source and the destination. Thus, for example, a channel can have an associated minimum dedicated bandwidth to maintain a channel between processors configured to span a sequence of one or more processor groups.

[0072] Figure 20 A processor group coupled to an optical network switch according to an embodiment is illustrated. For example, each processor group may include one or more processors. Processor group 2020 is coupled to optical network switch 2010 via optical links 2030 and 2031. Similarly, processor group 2021 is coupled to optical network switch 2010 via optical links 2032 and 2033. Likewise, processor group 2022 is coupled to optical network switch 2010 via optical links 2034 and 2035. Finally, processor group 2023 is coupled to optical network switch 2010 via optical links 2036 and 2037. Therefore, processor sequences can be formed in various ways. For example, an example sequence can be formed by cascading multiple processors in each processor group (e.g., by connecting ports 1-2, 3-4, 5-6, and 7-8 to form a loop). Another example sequence can be formed by cascading multiple groups. Multiple cascaded configurations of processor groups are possible. For example, two processor groups can be connected in series via ports 2-3 and 1-4, and via ports 6-7 and 5-8. Another series connection can be formed via ports 1-5 and 6-2, and via ports 3-7 and 8-4. In another configuration, all four processor groups are connected in a long chain. In this example, three processor groups can also be configured in series, with one processor group in a loop (e.g., multiple processors in one processor group within a loop). The specific configuration of the processors can be based on the partitioning of the neural network being processed and other factors.

[0073] even though Figure 20The simple example shown can also support full-bandwidth Hamiltonian cycles between any subset of N / 2 nodes. For example, since optical network switches support point-to-point connectivity, it can be seen that arbitrary Hamiltonian cycles can be supported; for example, cycles (2, 4, 3) can be supported by paired ports ([4, 7], [8, 5], [6, 3]) without any congestion. Faulty nodes (e.g., node 1 in the previous example) can be simply omitted. Any node in the same switch can be used as a backup node. In this example, each endpoint can be a member of a single full-bandwidth ring. If the endpoints have more network ports, each endpoint can be connected to multiple switches, for example, with six ports, and each client can connect to three switches.

[0074] In some embodiments, multiple optical network switches can be coupled together such that processor groups coupled to different optical network switches can be configured as a sequence across switches. Figure 21 This diagram illustrates a configuration where optical network switches are directly coupled together. In this example, optical network switch 2101 is coupled to processor groups 2111-2114 via each of two optical links, and optical network switch 2102 is coupled to processor groups 2115-2118 via each of two optical links. Optical network switch 2101 is directly coupled to optical network switch 2102 via optical links 2119 and 2120. Optical link 2119 can be dedicated to establishing a connection between a specific processor group coupled to optical network switch 2101 and another processor group coupled to optical network switch 2102 (e.g., port 8 coupled to processor group 2114 and port 3 coupled to processor group 2115). Similarly, optical link 2120 can be dedicated to establishing another connection between a specific processor group coupled to optical network switch 2101 and another processor group coupled to optical network switch 2102 (e.g., port 10 coupled to processor group 2118 and port 1 coupled to processor group 2111). Therefore, various processor sequences can be formed within or across optical network switches 2101 and 2102.

[0075] Advantageously, if the topology supports a second optical hop, different variants can be designed to achieve greater scheduling flexibility. In the simplest (one-dimensional) configuration, the set of switched ports is divided into two disjoint subsets E (endpoints) and I (inter-switch) ports, such that for |E| = e, there exist N = e + i and |I| = i. The switches themselves can be connected to the I ports as cliques. If k such switches are combined, a total of k·e endpoints can be connected, but i ≥ k - 1 is required. As before, arbitrary rings can be routed within each switch, and if there are c connections between two switches, c rings can be arbitrarily routed between them. The exact network topology is now a trade-off between the parameters e, i, c, and k. Figure 21 A network with two optical switches is shown, N=10, c=2, e=8, i=2. It can support a configuration spanning one Hamiltonian cycle across two switches (e.g., 1, 3, 5, 8), instead of two configurations such as (1, 5), (2, 8), which would require connections between four switches. Different dimensions can be supported in different planes through different network ports.

[0076] Figure 22 Another example of multiple optical network switches coupled to multiple processor groups according to an embodiment is shown. Figure 22 Another example is shown where multiple optical network switches are coupled together via two or more optical links. Here, two optical network switches 2201 and 2202 are coupled together via two processor groups 2214 and 2215. For example, processor group 2214 is coupled to optical network switch 2201 via a first optical link 2220. Processor group 2214 is also coupled to optical network switch 2202 via a second optical link 2221. Another processor group 2215 is coupled to optical network switch 2201 via a third optical link 2222, and processor group 2215 is coupled to optical network switch 2202 via a fourth optical link 2223.

[0077] Figure 23 Further example of multiple optical network switches coupled to multiple processor groups according to an embodiment is shown. This example shows three (3) optical network switches 2301-2303, each optical network switch being coupled to a subset of processor groups. Optical network switches 2301-2303 are directly coupled together (e.g., via an optical link between ports 8-1, 2-8, and 7-7). Thus, for example, up to nine (9) processor groups can be configured to form a sequence. Figure 23 The diagram illustrates a network with three 8-port switches and c=1. For example, for this network, all rings could be bundled into the switches, and the last ring distributed to all three switches. Generally, a larger c value makes configuration easier to allocate.

[0078] Figure 24 An example of multiple optical network switches coupled to multiple processor groups via intermediate optical network switches according to an embodiment is shown. In various embodiments, optical network switches 2401-2402 may be coupled to one or more intermediate optical network switches 2403. Because optical network switches operate using beams generated by lasers, additional optical switches in the long distance and / or signal path between optical transmitters and receivers may require, for example, more powerful lasers. Here, in order to couple processor groups to processors serially coupled to different switches, an additional optical link to intermediate network switch 2403 is used. Figure 24 In the topology shown, the example intermediate switch 2403 with 10 ports can connect up to 5 endpoint switches, supporting a total of 20 endpoints (e.g., processor groups), with, for example, a ring between any two switches, or two rings between any three or more switches.

[0079] Figure 25Another example two-dimensional cluster topology including optical network switches according to an embodiment is shown. In this example, processor groups 2501-2516 are configured in a multi-dimensional (e.g., two-dimensional) cluster array, where processor groups 2501-2516 along a specific dimension of the cluster array have edge processor ports coupled to corresponding optical network switches of the same dimension. In this example, rows of processor groups 2501-2516 in the cluster array have edge processor ports coupled to corresponding row optical network switches 2520-2523, while columns of processor groups in the cluster array have edge processor ports coupled to corresponding column optical network switches 2530-2533. In various embodiments, one or more row optical network switches 2520-2523 may be coupled to one or more column optical network switches 2530-2533. In this example, each optical network switch along one dimension is coupled to at least two optical network switches along another dimension. For example, row optical network switch 2520 is coupled to column optical network switches 2530 and 2531, row optical network switch 2521 is coupled to column optical network switches 2531 and 2532, row optical network switch 2522 is coupled to column optical network switches 2532 and 2533, and row optical network switch 2523 is coupled to column optical network switches 2530 and 2533. Similarly, column optical network switch 2530 is coupled to row optical network switches 2520 and 2523, column optical network switch 2531 is coupled to row optical network switches 2520 and 2521, column optical network switch 2532 is coupled to row optical network switches 2521 and 2522, and column optical network switch 2533 is coupled to row optical network switches 2522 and 2523. This advantageously improves fault tolerance or can support mapping of Hamiltonian cycles across dimensions (e.g., when mapping together processor sequences larger than a single dimension). As described above, indirect cross-dimensional connections can be added between the intermediate (layer 1) row and column optical network switches as described above.

[0080] Based on the example above, the topology can be generalized to extend it to two or more dimensions. This preserves the two-switch hops from any point to any point, but some flexibility is lost in terms of mapping because switches are now arranged along two dimensions (and can "hop" freely within one dimension). Inter-switch links can now be used to connect switches of the same or different sizes. For example, if c = 2, each switch can have two connections to switches in other dimensions. For an n×n layout, and c = 1, all switches will be connected along the main diagonal. For larger c, the invariance of exactly c cross-connections in each row / column can be preserved.

[0081] Figure 26Another configuration of the processors in a processor group according to an embodiment is shown. In this example, edge processors 2601 and 2604 in processor group 2650 are coupled to optical modules 2620 and 2621, respectively. For example, optical modules typically have an electrical interface on the side connected to the inside of the system and an optical interface on the side connected to the outside via optical fiber. For example, optical modules 2620 and 2621 can transmit (Tx) and receive (Rx) optical and electrical signals.

[0082] Figure 27A Another configuration of the processors in a processor group according to an embodiment is shown. In this example, processor group 2700A includes a two-dimensional array of processors 2740-2748. Edge processors 2740-2747 are coupled to one or more optical modules 2720-2731 for transmitting and receiving optical signals via optical channels 2701-2706 and 2710-2715 (e.g., optical fiber).

[0083] Figure 27B Another configuration 2700B of the processor in the processor group according to an embodiment is shown. In some embodiments, optical modules 2720-2731 may be coupled to, for example, splitters / combiners 2760-2763 to transmit and receive multiple different channels. In this example, multiple different communication channels may typically be combined into a common optical fiber, for example using WDM, and then the entire channel set may be extensively switched using a common pair of optical switch ports and a single optical path through the optical switches. For example, optical modules 2720-2731 may receive data from multiple processor groups and combine the data at multiple wavelengths, which are transmitted together on optical fibers 2790-2793. For example, for an input optical signal, a splitter may send different wavelengths to different optical modules 2720-2731 and corresponding processor groups. Advantageously, this example can reduce the total number of optical switch ports, which reduces costs. It also allows processors (with many different “channels” (e.g., Ethernet ports) that have commonalities between their endpoints) to be viewed as a single set, so they can take advantage of their inherent organization and can be switched with a very low number of optical switching ports, where, for example, the possibility of finding a set of channels / ports among ML-specific processors is very high due to the specific nature of neural networks using many parallel channels / ports to support very high total data rates.

[0084] The foregoing description illustrates various embodiments of this disclosure and examples of aspects of how these embodiments can be implemented. The various embodiments described above can advantageously include network topologies capable of achieving an oversubscription factor (e.g., 3:1, with 48 client links and 16 uplinks), meaning significant cost / power savings while still maintaining full Hamiltonian cycle bandwidth. In some embodiments, the network topologies described herein may include software allocation / reassignment methods utilizing locality (e.g., assigning adjacent processor groups to the same Layer 0 switch). The examples and embodiments described above should not be considered as only possible embodiments, but are intended to illustrate the flexibility and advantages of this disclosure as defined by the appended claims. For example, while certain embodiments have been described with respect to specific processes and steps, it will be apparent to those skilled in the art that the scope of this disclosure is not necessarily strictly limited to the described processes and steps. Steps described sequentially may be performed in parallel, the order of steps may be changed, and steps may be modified, combined, added, or omitted. As another example, although certain embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also possible, and a particular operation described as being implemented in software may be implemented in hardware, and vice versa.

[0085] Therefore, the specification and drawings are to be considered illustrative rather than restrictive. Other arrangements, embodiments, implementations, and equivalents will be apparent to those skilled in the art and may be used without departing from the spirit and scope of this disclosure as set forth in the appended claims.

Claims

1. A computer system, comprising: Multiple processor groups, the processor groups including multiple serially configured processors for processing partitioned neural networks, wherein different processors in the multiple processor groups process different layers, levels, or model instances of the partitioned neural network; and One or more optical network switches are coupled to the plurality of processor groups via edge processors located at edge positions, including first and last positions, among the serially configured processors in each processor group. At least a subset of the processors in the processor group are configured to form a sequence such that at least a portion of the processors transmit data for the layer, level, or model instance to at least two adjacent processors in the sequence, and The optical network switch is configured to form a sequence of at least one processor across one or more of the plurality of processor groups to perform neural network computations.

2. The computer system of claim 1, wherein the one or more optical network switches maintain a channel between the one or more processor groups to form the at least one sequence, the channel having an associated minimum dedicated bandwidth to maintain the channel between processors in the at least one sequence across the one or more processor groups.

3. The computer system according to claim 1, wherein the processors at the edge locations of the processor group are coupled to the plurality of optical network switches via optical links.

4. The computer system of claim 3, wherein the edge processor in the processor group is coupled to at least one optical module.

5. The computer system of claim 3, wherein the edge processor port on the opposite side of the processor row is coupled to a first optical network switch.

6. The computer system of claim 5, wherein the edge processor port on the opposite side of the processor column is coupled to a second optical network switch.

7. The computer system of claim 1, wherein the processor group is coupled to a plurality of row optical network switches and a plurality of column optical network switches.

8. The computer system of claim 1, wherein the processor group is configured as a multidimensional cluster array.

9. The computer system of claim 8, wherein the processor group along a specific dimension of the cluster array has an edge processor port coupled to a corresponding optical network switch of the same dimension.

10. The computer system of claim 9, wherein each optical network switch along one dimension is coupled to at least two optical network switches along another dimension.

11. The computer system of claim 1, wherein the one or more optical network switches are multiple optical network switches coupled together by two or more optical links.

12. The computer system of claim 11, wherein two or more optical network switches in the optical network switch are directly coupled together via the two or more optical links.

13. The computer system of claim 11, wherein at least two optical network switches in the optical network switches are coupled together through at least two processor groups, wherein a first processor group is coupled to a first optical network switch through a first optical link, the first processor group is coupled to a second optical network switch through a second optical link, the second processor group is coupled to the first optical network switch through a third optical link, and the second processor group is coupled to the second optical network switch through a fourth optical link.

14. The computer system of claim 1, wherein the plurality of optical network switches are coupled to at least one intermediate optical network switch to cascade processors in at least a subset of the plurality of processor groups.

15. The computer system of claim 1, wherein the processor configured to form the sequence is configured to process a plurality of local layers of the partitioned neural network.

16. The computer system of claim 1, wherein the processor configured to form the sequence is configured to process one or more pipeline stages of the partitioned neural network.

17. The computer system of claim 1, wherein the processor configured to form the sequence is configured to adjust weights across a plurality of model instances of the neural network.

18. A method for processing a neural network, the method comprising: Multiple processors are configured to perform neural network computations in multiple processor groups, wherein the multiple processors are configured serially to process a partitioned neural network, wherein different processors in the multiple processor groups process different layers, levels or model instances of the partitioned neural network, and wherein at least one subset of the processors in the processor groups is configured to form a sequence of processors. Multiple optical network switches are configured to couple a subset of the multiple processor groups together with edge processors at edge positions, including a first position and a last position, of the processors in each processor group configured serially to form a sequence of at least one processor across one or more processor groups, such that at least a portion of the processors transmits data for the layer, level, or model instance with at least two adjacent processors in the sequence. as well as The neural network computation is performed by a sequence of the at least one processor.

Images