Distributed processing architecture

Abstract

Embodiments of the present disclosure include techniques for processing neural networks. Various forms of parallelism can be achieved using a topology of combined processor sequences. In one embodiment, the present disclosure includes a computer system comprising a plurality of processor groups, each processor group comprising a plurality of processors. A plurality of network switches are coupled to a subset of the plurality of processor groups. A subset of the processors in a processor group can be configured to form a sequence, and the network switches can be configured to form at least one sequence across one or more of the plurality of processor groups to perform neural network computations. Various configurations for creating Hamiltonian cycles are disclosed to support data parallelism, pipeline parallelism, layer parallelism, or a combination thereof.

Description

BACKGROUND

[0001] The present disclosure relates to distributed computing, and in particular to a distributed processing architecture that can be used to process artificial neural networks.

[0002] Generally, artificial neural networks (hereinafter neural networks) are becoming increasingly important in artificial intelligence applications and modern computing. Figure 1 An example neural network is shown in FIG. 1. The neural network 100 receives input values corresponding to features to be recognized. The input values are multiplied by weights (represented by edges 101) and summed (e.g., added) in nodes 102. An activation function is applied to the results in the nodes 102 to generate output values. Values are combined across multiple nodes and layers of nodes to produce a network output value corresponding to a result.

[0003] Such systems “learn” to perform a task by considering examples, often without being programmed with task-specific rules. Initially, the weights can be untrained. During a training phase, the network processes input values corresponding to known results, and compares differences (or errors) between network output values and known values. The weights can be adjusted based on the errors using a process called backpropagation, in which calculations flow in the opposite direction (e.g., from output to input). Training can involve continuously adjusting the weights across many input samples and corresponding known network output values. This is often referred to as a training phase. Once trained, the system can receive inputs and produce meaningful results (e.g., classifications or recognitions). This is often referred to as an inference phase.

[0004] Training and / or inference for very large neural networks can involve a large amount of computation. Thus, computation for large networks can be partitioned across many processors. However, due to the large number of connections in the network, partitioned network computation can involve a large amount of potential data communication (partial results) between processors. Such connectivity can require multiple processors coupled together, for example, through potentially multiple network switches. Providing full bandwidth for arbitrary communication patterns can become prohibitively expensive as the scale increases and high bandwidth can become prohibitive. For example, the cost of components and the energy consumption of electrical switches and optical transceivers required to cover longer distances in a data center can be prohibitive. SUMMARY

[0005] Embodiments of the present disclosure include techniques for processing neural networks. Various forms of parallelism can be achieved using a topology of a combined processor sequence. In one embodiment, the present disclosure includes a computer system including a plurality of processor groups, each of the processor groups including a plurality of processors. A plurality of network switches is coupled to a subset of the plurality of processor groups. A subset of the processors in the processor groups are configurable to form sequences, and the network switches are configurable to form at least one sequence across one or more of the plurality of processor groups to perform neural network computations. Various alternative configurations for creating Hamiltonian cycles are disclosed to support data parallelism, pipelined parallelism, layer parallelism, or a combination thereof. BRIEF DESCRIPTION OF DRAWINGS

[0006] Figure 1 An example neural network is shown.

[0007] Figure 2 A neural network partitioned according to an embodiment is shown.

[0008] Figure 3 Neural network communication across three axes is shown.

[0009] Figure 4 A neural network partitioned along three dimensions according to an embodiment is shown. Figure 2

[0010] Figure 5 An example of a plurality of processors configurable to form sequences according to an embodiment is shown.

[0011] Figure 6 An example network switch coupled to a plurality of processor groups according to an embodiment is shown.

[0012] Figure 7 Another example topology of processor groups and network switches according to an embodiment is shown.

[0013] Figure 8 Yet another example topology of processor groups and network switches according to an embodiment is shown.

[0014] Figure 9 Another example neural network partitioned according to an embodiment is shown.

[0015] Figure 10 A three-dimensional representation of a partition is shown to illustrate nearest neighbor communication of processors according to an embodiment.

[0016] Figure 11 Another configuration of processors in a processor group according to an embodiment is shown.

[0017] Figure 12 ​A configuration of a cluster of multi-dimensional processor groups according to an embodiment is shown.

[0018] Figure 13 An example processor group according to another embodiment is shown.

[0019] Figure 14 An example network switch and processing group according to another embodiment is shown.

[0020] Figure 15 A 2-layer arrangement of a network switch according to an embodiment is shown.

[0021] Figure 16 An example topology of a cluster of processing groups along a first cluster dimension according to an embodiment is shown.

[0022] Figure 17 An example topology of a cluster of processing groups along a second cluster dimension according to an embodiment is shown.

[0023] Figure 18 An example two-dimensional cluster topology according to an embodiment is shown.

[0024] Figure 19 Another example two-dimensional cluster topology according to an embodiment is shown. DETAILED DESCRIPTION

[0025] In the following description, for the purposes of explanation, numerous examples and details are set forth in order to provide a understanding of various embodiments. However, it will be apparent to one skilled in the art that certain embodiments can be practiced without some or all of these details, or with modifications or equivalents of these details.

[0026] Figure 2 An artificial neural network 200 partitioned according to an embodiment is shown. The features and advantages of the present disclosure exploit the communication patterns in a neural network to produce a more efficient topology for distributed computation of the neural network. For example, the neural network 200 includes 7 layers 201-207, including an input layer 201 and an output layer 202. During inference, input values can be received on the input layer 201, and results produced on the output layer 207. At each layer, data values are multiplied by weights, summed, and processed by each node using an activation function. During training, data flows in the opposite direction, with errors determined between the network results and desired results at the output, and weights updated layer by layer from layer 207 to layer 201.

[0027] In various embodiments, it can be desirable to partition a neural network so that the computations of different partitions of the network are performed on different processors. The example neural network 200 is a simplified network that illustrates the communication between partitions. In this example, the network can be partitioned along pipelines (denoted as Pi) or along layers of a pipeline (denoted as Li). In this example, the network 200 is partitioned into 4 pipeline stages P1-P4 and 2 layers L1-L2. The nodes and edges in partition L1, P1 can be processed on one processor, while the nodes and edges in the other partitions can each be processed on a different processor. However, in order to perform their computations, each processor must receive certain data inputs and send data outputs to other processors. For example, node 1 in partition L1, P1 can receive an input, multiply the input by a weight, apply an activation function, and send the output to node 2 in partition L2, P1. Similarly, node 3 in L2, P1 sends output data to node 4 in partition L1, P1. Thus, when a layer 201 is partitioned across multiple processors, there is intra-layer communication that occurs, for example, between processors. Similarly, pipeline communication can be as follows. The output of node 4 in partition L1, P1 is sent to the processor assigned to partition L1, P2 as input to node 5, and the output of node 2 in partition L2, P1 is sent to the processor assigned to partition L2, P2 as input to node 6. Likewise, the output of node 4 in partition L1, P1 is sent to the processor assigned to partition L2, P2 as input to node 6, and the output of node 2 in partition L2, P1 is sent to the processor assigned to partition L1, P2 as input to node 5. Thus, it can be seen that, for example, pipeline stage P1 communicates with pipeline stage P2, pipeline stage P2 communicates with pipeline stage P3, but pipeline stage P3 can not need to communicate with pipeline stage P1.

[0028] Figure 3 Neural network communication across three axes is illustrated. In this example, a neural network model is partitioned for layer parallelism, pipeline parallelism, and data parallelism. Layer parallelism and pipeline parallelism are described above in Figure 2 Data parallelism can exchange data between multiple instances of a model executed on different processors. For example, the same model can be trained in parallel by executing different instances of the model on different processors and periodically synchronizing the weights in each layer.

[0029] In this example, M (integer) instances of the model receive input on layers (0, 0) - (M-1, 0) and produce output on layers (0, N-1) - (M-1, N-1) during inference. For training, data flows in the opposite direction. Thus, pipeline communication flows vertically (e.g., activations / errors) as shown by arrows 350. For example, layers 311 and 321 can be partitioned to run on different processors as described above. Here, intra-layer communication between J (integer) processors is shown by arrows 351, where layers 311 and 321 are partitioned, for example, into J partitions. Similarly, during training, weights in each model can be periodically synchronized as described above. If instances of a neural network model run on different processors (or distributed across multiple sets of different processors), such processors can perform communication to update weights (e.g., using an All-Reduce algorithm such as one-dimensional ring, multi-ring, tree-based, or hierarchical algorithm). The pipeline can operate independently during training, but can need to periodically synchronize its weights across execution of different instances of the model. For example, during All-Reduce, weights can be averaged layer-by-layer between instances. Thus, in some applications, each layer of the pipeline can periodically perform All-Reduce with other instances of the model, as shown by arrows 353 in FIG. 3B. Instances of the model running in parallel can be configured in a ring, for example, where weight updates between models can flow in two directions around the ring. Figure 3

[0030] From Figure 2 and Figure 3 there are three forms of parallelism: data parallelism (e.g., sum, average, All-Reduce), pipeline parallelism, and intra-layer (or just “layer”) parallelism. These forms of potential parallelism allow for the development of an architecture that leverages one or more of the available parallelisms in performing neural network computations. Various embodiments of the present disclosure leverage the understanding that, with certain exceptions, communication dependencies in all three dimensions (intra-layer, pipeline, data) can be mapped (e.g., there is no congestion) to a communication pattern that follows a one-dimensional ring structure of couplings. Thus, if the communication pattern is mapped to a network topology, it can be modeled as a set of Hamiltonian cycles connecting each dimension. In some example embodiments, a sequence of processors can be logically arranged in a three-dimensional Cartesian space. Thus, as described in more detail below, a Hamiltonian cycle can refer to, for example, a plurality of processors configured in series across one or more groups of processors via at least one network switch for intra-layer parallelism, pipeline parallelism, data parallelism, or various combinations thereof.

[0031] For example, Figure 4 shows a model partitioned along three dimensions Figure 2 ​The neural network consists of layers (L), pipeline stages (P), and data. In this example, model 200 is processed three times in parallel using instances A, B, and C (data parallelism). The partitions of model 200 can be mapped to... Figure 4 The processing sequence is illustrated. For example, computations of network partitions L1, P1 and L1, P2 can be performed on different processors configured in the sequence to form Hamiltonian loop 401. Similarly, computations for pipeline stages P1-P2-P3-P4 can be performed on multiple processors configured as a sequence to form Hamiltonian loop 402. Finally, for example, computations along the data axis ABC (e.g., full reduction) can be performed on a sequence of multiple processors forming Hamiltonian loop 403. It should be understood that in various example embodiments, the neural network can be partitioned along a data axis, pipeline axis, layer axis, or a combination thereof. For example, processors can be configured to form a sequence, and processors can be configured to process multiple local layers of a partitioned neural network, one or more pipelines of a partitioned neural network, data parallelism (e.g., full reduction) between models of the neural network, and / or combinations thereof.

[0032] Figure 5 An example of multiple processors configurable to form a sequence according to an embodiment is shown. In this example, four (4) processors PR1 501, PR2 502, PR3 503, and PR4 504 are configured serially to form a one-dimensional (1D) array. For example, processors 501-504 may form a processor group 550, and processors 501-504 may be configured as a sequence to process a partitioned neural network. Various embodiments of this disclosure may use various processor types, including but not limited to, x86 architecture processors, ARM architecture processors, graphics processing units (e.g., GPUs), or dedicated artificial intelligence (AI) processors optimized for performing neural network computations.

[0033] Here, processor group 550 includes edge processors 501 and 504 with edge ports 510 and 511. In this example, edge (or boundary) processor 501 is located in the first position in the array and is coupled to port 510 and processor 502. Processor 502 is coupled to adjacent processors 501 and 503, processor 503 is coupled to adjacent processors 502 and 504, and edge (or boundary) processor 504, located in the last position, is coupled to adjacent processor 503 and edge port 511. Edge ports 510 and 511 can be coupled to a network switch, for example, so that multiple processor groups 550 can be coupled together to form a longer sequence to process more partitions of the neural network.

[0034] Figure 6An example network switch 610 coupled to multiple processor groups is shown, according to an embodiment. The network switch can be configured to form sequences across one or more of the processor groups to perform neural network computations. In this simple example, four (4) processor groups 620, 630, 640, and 650 each include four (4) processors configured as a one-dimensional array. Each group includes edge ports coupled to ports of the network switch 610, so that different groups can be coupled together to form longer sequences. For example, if the network is partitioned along any of three axes (e.g., layers, pipelines, data), the partitions can be loaded into sequential processors, such as processors 621-624 in processor group 620. In some embodiments, the sequences can form closed loops, e.g., so that processor 624 communicates with 621. If the partitions require longer sequences of processors, longer sequences of processors can be configured through the switch 610. For example, if 5 to 8 sequential processors are required, port 626 from processor 624 (which is coupled to port 6 of the network switch 610) can be coupled to port 635 from processor group 630 (which is coupled to port 7 of the network switch 610 for processor 631). Thus, a sequence of up to 20 processors can be obtained using Figure 6 The topology shown strings together sequences of up to 16 processors.

[0035] Figure 7 Another example topology of processor groups and network switches is shown, according to an embodiment. In this example, three (3) processor groups 711-713 are coupled to network switch 701, three (3) processor groups 721-723 are coupled to network switch 702, three (3) processor groups 731-733 are coupled to network switch 703, and three (3) processor groups 741-743 are coupled to network switch 704. Each of the processor groups 711-713, 721-723, 731-733, and 741-743 can include, for example Figure 5 The multiple processors shown can be configured as sequences. In this example, a larger number of sequences of processors can be obtained through the network switches 701-704 coupled together directly. For example, if each processor group includes five (5) processors, a sequence of twenty (20) processors can be obtained by coupling groups 711-713 with group 721. In this case, switch 701 is configured to connect ports 7 and 1, ports 2 and 3, ports 4 and 5, and ports 6 and 9, and switch 702 is configured to connect ports 1 and 4 and ports 5 and 2. Port 5 / 2 of switch 702 can be routed back to port 1 / 7 on switch 701 via switch 703, which can connect ports 7 and 8, for example, to close the loop for a dedicated congestion-free nearest-neighbor processor communication loop. Using Figure 7The topology shown, for example, can configure up to sixty (60) processors in a sequence to perform neural network computations along one of the three axes described above.

[0036] Figure 8 Another example topology of processor groups and network switches according to embodiments is shown. In this example, network switches 801-804 are coupled together through an intermediate (a.k.a. core or layer 1) network switch 805. Thus, one or more processor groups 811-813 can be configured to form a sequence with any one or more other processor groups 821-823, 831-833, and / or 841-843, for example, through directly connected (a.k.a. layer 0) switches 801-804 and through intermediate network switch 805.

[0037] Figure 9 Another example neural network 900 partitioned according to embodiments is shown. Neural network 900 includes a pipeline of nine (9) layers, each layer including nine (9) nodes. Most edges are omitted for clarity, but it should be understood that edges couple the nodes together. In this example shown, network 900 is partitioned on a layer axis (e.g., LI, L2, L3) and a pipeline axis (e.g., PI, P2, P3), with computations for each partition performed on different processors (e.g., PR11-PR33).

[0038] Figure 10 A three-dimensional representation of the partitioning is shown to illustrate sequential communication of the processors. In this example, computations for each partition can be performed on processors PR11-PR33 (see Figure 9 and 11 ) forming a 3x3 array. Additionally, each model can be executed four (4) times (e.g., data parallelism A, B, C, D) to form a 3-dimensional 3x3x4 array. Embodiments of the present invention can map the neural network partitioning to an array of processors for execution, with the processors configured to perform, for example, Hamiltonian cycles. In this example, along the layer axis, a Hamiltonian cycle can capture communication between LI, L2, and L3 in performing computations of a first layer PI of the pipeline, as shown at 1001. Along the pipeline axis, the LI layer computed during PI can communicate with the LI layer computed during P2, and subsequently with the LI layer during P3 to form another Hamiltonian cycle, as shown at 1002. Along the data axis, a Hamiltonian cycle can capture communication between layers across the execution of multiple models A-D (e.g., performing a full reduction operation to adjust the weights of each model), as shown at 1003.

[0039] Figure 11Another configuration of processors in a processor group 1100 is shown, in accordance with an embodiment. In this example, the processors 1140-1148 are configured as rows and columns to form a two-dimensional processor array. Here, the 2-dimensional processor array (e.g., PR11-PR33) is configured as a 3x3 matrix to form the processor group 1100, with the processors 1140-1148 coupled to nearest neighbor processors in the N-dimensional processor array to support forming a sequence of processors to perform, for example, a Hamiltonian cycle. The processor group 1100 includes edge processors 1140-1147, each having one or more edge ports. The ports 1101-1106 are coupled to a row of processors in the processor group 1100, while the ports 1110-1115 are coupled to a column of processors in the processor group 1100. Thus, Figure 9 Each partition of the neural network 900 can be loaded into different processors for performing neural network computations for layers and pipelines of the network 900, where layers of each pipeline stage can be computed first, followed by each successive pipeline stage. Intra-layer data for computing each layer can flow vertically across processor columns (e.g., PR11-PR21-PR31). When the layers in each partitioned pipeline stage P1, P2, P3 are complete, the results flow horizontally across processor rows (e.g., PR11-PR12-PR13).

[0040] Figure 12 A configuration of a multi-dimensional cluster array 1200 of processor groups, in accordance with an embodiment, is shown. In this example, the two-dimensional cluster 1200 of processor groups 1201, 1202, 1210, and 1211 can be used to support larger neural networks, more partitions, or other multiple executions of the model 900 that can be combined via a full reduction, for example. For example, larger networks can require more sequences of processors to implement a Hamiltonian cycle along each axis. For example, Figure 12 The architecture shown allows rows or columns in different processor groups to be configured as larger sequences. Specifically, during execution of neural network computations, a first subset of processors (e.g., rows in the processor group 1201) and a second subset of processors (e.g., rows in the processor group 1210) can be coupled together to form a sequence including processor strings (e.g., A11, A12, A13, B11, B12, B13), where each processor communicates with adjacent processors in the processor string. As described above, the sequence can form a closed loop in various embodiments, for example, such that A11 communicates with B13.

[0041] The topology of the network switches is configured to couple rows and / or columns of processors between processor groups to produce strings of processors for performing neural network computations. In various embodiments disclosed herein, processor groups along a particular dimension of the cluster array have edge processor ports coupled to a corresponding same dimension network switch. In this example, processors in first and last positions along each dimension are coupled to row or column network switches. For example, processor group 1201 includes a row of processors Al l, A12, A13, with edge processor ports of Al l and A13 coupled to row network switch 1221. Additionally, processor group 1210 includes a row of processors Bl l, B12, B13, with edge processor ports of Bl l and B13 coupled to row network switch 1221. Similarly, processor group 1201 includes a row of processors A21, A22, A23, with edge processor ports of A21 and A23 coupled to row network switch 1222, and processor group 1210 includes a row of processors B21, B22, B23, with edge processor ports of B21 and B23 coupled to row network switch 1222. For example, the third row of processors in groups 1201 and 1210 are likewise coupled to row network switch 1223. Thus, for example, row switches 1221-1223 form a layer of switches corresponding to the rows of processors in each processor group, which can be used to connect a particular row of processors to multiple groups of switches.

[0042] The columns of processors in each processor group 1201 and 1202 can likewise be coupled together across the first column of the cluster 1200. As described above for the rows in the example, the edge processor ports on opposite sides of the columns of processors Al l, A21, A31 and Cl l, C21, C31 on processor groups 1201 and 1202 can be coupled to column network switch 1231. Other column processors Ax2 / Cx2 and Ax3 / Cx3 (x=l..3) can be coupled from edge processor ports to column network switches 1232 and 1233, respectively, for example. Thus, column switches 1231-1233 form a layer of switches corresponding to the columns of processors in each processor group, which can be used to connect a particular column of processors across multiple groups of switches.

[0043] Likewise, as described above for the first row of the cluster, the second row of the cluster 1200 including processor groups 1202 and 1211 can be coupled to a multi-row network switch 1225. Finally, as described above for the first column of the cluster, the second column of the cluster 1200 including processor groups 1210 and 1211 can be coupled to a multi-column network switch 1235. As described in more detail below, for example, multiple intermediate switches can be used to link the row and column switches to allow more processors to be configured in strings to compute longer Hamiltonian cycles.

[0044] While Figures 9-12 The above examples shown in FIGS. 1-3 illustrate example neural networks that are partitioned to fit a processor group, but different networks can be partitioned in various ways to run on network topologies according to the present disclosure. While various partitions can more efficiently use available resources in a given topology, there is no strict relationship between the network partitioning that can be used, for example, in a given number of processors in one processor group or a given number of groups in a cluster. For example, each processor can perform computations for one layer, multiple layers, or a fraction of a layer of a neural network model. Similarly, each processor can perform computations for one pipeline stage, multiple pipeline stages, or a fraction of a pipeline stage. Finally, for example, each processor can perform computations for the full neural network model (all layers and all pipelines) and synchronize weights across the processors, or multiple groups of processors performing operations on a partitioned neural network can synchronize weights.

[0045] Figure 13 An example processor group 1300 according to another embodiment is shown. In this example, a 4x4 processor array PR11...PR44 is arranged to form the processor group 1300. Here, each processor has 12 input / output (I / O) communication ports, which can be, for example, Ethernet ports (e.g., 50 Gbps Ethernet) configured in parallel to expand connection bandwidth. Three (3) I / O ports are coupled to a neighboring (e.g., nearest neighbor) processor or network switch, as described in more detail below. The processor group 1300 includes edge processors 1320-1331 having edge processor ports 1301-1308 and 1310-1317 that couple each edge processor to, for example, a particular network switch.

[0046] Figure 14 An example network switch and processing group according to another embodiment is shown. In this example, N processing groups 1401-1402 are coupled to a plurality of switches 1410-1412 configured in 12 switch planes, where N is an integer. For example, N can be 24 groups Figure 134x4 array of processors. As described above, each processor group can have rows and columns. Each row can have 3 edge processor ports, e.g., for a total of 24 row processor ports (3 ports per side (x2) for each row of 4 rows (x4); 3 x 4 x 2 = 24). In this example, the edge processor ports on opposite sides of a row of processors across the first plurality of processor groups are coupled to a row network switch. For example, the edge processor ports on opposite sides of the first row of processors in processor groups 1401-1402 are coupled to network switch 1410. Similarly, the edge processor ports on opposite sides of the second row of processors in processor groups 1401-1402 are coupled to network switch 1411, while the edge processor ports on opposite sides of the last row of processors in processor groups 1401-1402 are coupled to network switch 1412. In this figure, only 3 ports are shown for illustration purposes, but it will be appreciated that there are 3 ports per row and per column as shown in the example of Figure 13 there are 4 rows of switches on the 24 processor groups. Likewise, for 4 columns in each of the 24 processor groups, e.g., the edge processor ports on opposite sides of a column of processors across the second plurality of processor groups are coupled to a column network switch. Thus, for example, there are 12 network switches for coupling the columns of processors in each of the 24 processor groups in substantially the same way as the rows Figure 14 are shown.

[0047] Figure 15 A 2-layer arrangement of network switches according to an embodiment is shown. Features and advantages of the present disclosure include a topology that allows a subset of processor groups to be coupled together across different network switches (e.g., to form a Hamiltonian cycle). In this example, layer 0 network switch 1501 is coupled to processor groups 1510A-C, and layer 0 network switch 1502 is coupled to processor groups 1510D-F. Switches 1501 and 1502 are further coupled to layer 1 intermediate network switch 1503. Each connection between the layer 0 and layer 1 switches can include two links (e.g., two ports), e.g., one for a non-congested data uplink and another for a non-congested data downlink. For example, Figure 15 the topology shown can be applied to Figure 14A similar topology can be used for other switch planes. In some configurations, a subset of the processor groups are coupled together through the layer 0 switches. In other configurations, an intermediate layer 1 switch 1503 can be used to couple multiple layer 0 switches together to form a longer sequence of processors than can be supported by any particular layer 0 switch. For example, if a sequence of 150 processors is needed, then the processor groups from two switches can be combined (e.g., 24 groups x 4 processors / group x 2 > 150). By coupling the layer 0 switches and associated processor groups in series together, longer sequences can be produced, for example, which can form a non-congested loop across multiple layer 0 switches and layer 1 switches.

[0048] Switches 1501 and 1502 can be, for example, 64 port switches. Thus, in this example, 48 ports are used for processor group connections, leaving 16 ports to support inter-switch transfers for 24 groups. Because more of the switch ports are used for local connections and only 16 are available for inter-switch transfers, the layer 0 switches in this example topology are referred to as over-subscribed (e.g., all of the processor groups cannot access the layer 1 switch at the same time). Another advantage of some example implementations is that network switches can be over-subscribed without impacting performance because the processor groups can be locally configured into sequences before multiple switches are needed. For example, one port of processor group 1510A can be coupled to one port of processor group 1510B, another port of processor group 1510B can be coupled to one port of processor group 1510C through additional ports and processors, and a second port of processor group 1510C can be coupled back to another port of processor group 1510A (e.g., to form a closed loop), or to other layer 0 switches and processor groups through links to switch 1503. Because the processor groups can generally be configured to form local sequences, there are generally still enough ports in the over-subscribed switches to establish communications with other processor groups in other switches to perform desired computations.

[0049] Figure 16 An example topology of clusters of processor groups along a first cluster dimension is shown in accordance with an embodiment. In this example, the cluster includes a two-dimensional array of N x M processor groups (e.g., 24 x 24) coupled together through a 2-layer switching topology. Figure 16One switch plane (e.g., a switch topology for one edge port of a row in each processor group) is shown. Other planes for other edge ports in other rows can have substantially similar implementations. In this topology, rows of processor groups in the cluster have edge processor ports coupled to a corresponding same-row network switch. For example, processor groups 1610-1612 (e.g., processor groups in a first row of the cluster) have edge processor ports (e.g., 2 per group) coupled to a same-row network switch (e.g., network switch 1601 of cluster row 1). Similarly, for example, processor groups 1613-1615 in a second row of the cluster are coupled to row network switch 1602, while processor groups 1616-1618 in a last row of the cluster are coupled to row network switch 1603.

[0050] Row network switches 1601-1603 of the cluster can be coupled together through one or more intermediate tier-1 row network switches 1620-1621. In this example, each row network switch 1601-1603 includes multiple connections to multiple intermediate row network switches. For example, row network switches 1601-1603 can all include connections to two intermediate row switches 1620-1621. Thus, for example, each row of processor groups can establish a path through intermediate row switch 1620 or intermediate row switch 1621 to one or more other rows of processor groups to create a Hamiltonian cycle across one or more rows of the multiple rows. In this example, the links between tier 0 and tier 1 can be single full-duplex Ethernet links for uplink and downlink traffic without congestion. Even longer Hamiltonian cycles can be obtained, for example, by using the remaining 40 ports to connect additional clusters to tier-1 switches 1620 and / or 1621. In some embodiments, intermediate row network switches can be coupled to intermediate column network switches, as described below.

[0051] Figure 17 An example topology of a cluster of processor groups along a second cluster dimension is shown, according to an embodiment. In this topology, columns of processor groups in the cluster have edge processor ports coupled to a corresponding same-column network switch. For example, processor groups 1610, 1613, and 1616 (e.g., processor groups in a first column of the cluster) have edge processor ports coupled to a same-column network switch (e.g., network switch 1701 of cluster column 1). Similarly, for example, processor groups 1611, 1614, and 1617 in a second column of the cluster are coupled to column network switch 1702, while processor groups 1612, 1615, and 1618 in a last column of the cluster are coupled to column network switch 1703. For example, column switches 1701-1703 can be coupled to an intermediate tier-1 column switch 1920 substantially as shown. Figure 16 Intermediate tier-1 column switch 1920.

[0052] Figure 18 Another example of a multidimensional cluster array according to an embodiment is shown. Figure 16 and 17 Topologies can be combined to form, for example... Figure 18 The diagram illustrates a two-dimensional cluster switching topology. For all switch planes, 12 row network switches are available per row (e.g., 12 switches / row × 24 rows = 288 row network switches), and 12 column network switches are available per column (e.g., 12 switches / column × 24 columns = 288 column network switches). The intermediate row network switches for Layer 1 are shown at 1890, and the intermediate column network switches for Layer 1 are shown at 1891. In some embodiments, processor groups in various columns and rows can be indirectly cross-coupled together via, for example, the intermediate row network switches and column network switches shown at 1892. Figure 18 In the illustrated embodiment, each switch plane includes two Layer 1 switches for columns and rows, and there are 12 switch planes (12 edge row ports and 12 edge column ports), so there are a total of 12 planes × 2 Layer 1 switches / planes × 2 (columns and rows) = 48 Layer 1 switches. Figure 18 The example configuration shown has 24×24 groups (9216 processors) and 24 planes (e.g., 12 rows and 12 columns). For each plane, there are 24 Layer 0 switches, each with 16 uplinks. Each uplink connects to a Layer 1 switch. In one example embodiment, a 64-port switch can be used. Therefore, 8 Layer 1 switches can be used (e.g., then the 16 ports per Layer 1 switch can be omitted), resulting in, for example, a total of 24 planes * 8 Layer 1 switches = 192 switches.

[0053] Although Figures 13-18 Example topologies are shown, but it should be understood that these diagrams are merely illustrative. Various numbers of processors can be used in different processor groups, various subsets of these groups can be coupled to network switches, and various cluster sizes and configurations can be used in various embodiments of this disclosure. Furthermore, for any particular hardware topology (e.g., the number of processors in a processor group, the number of processor groups leading to a switch, and / or the number of processor groups in a cluster), any particular topology (e.g., the number of processors in a processor group, the number of processor groups leading to a switch, and / or the number of processor groups in a cluster) can be used. Figures 13-18The topology shown can be used to compute various neural network sizes and partitions. For example, for a given hardware configuration, smaller or larger neural networks can be computed, which can be adapted to the topology in different ways. For example, a particular hardware topology can handle neural networks with different numbers of layers, different layer partitions, different numbers of pipeline stages, different pipeline stage partitions, and different data volume parallelism. In various embodiments, for example, the techniques described herein can be used to map network models of different sizes to various different processors and sequences of processors in one or more dimensions.

[0054] Figure 19 Another example two-dimensional cluster topology according to an embodiment is shown. In some embodiments, the middle (e.g., layer 1) row network switches and the middle column network switches in the cluster may be coupled together. In this example, processor groups 1901-1916 are configured in a manner similar to... Figure 12 and 18 In a multidimensional (e.g., two-dimensional) cluster array, for example, processor groups 1901-1916 along a specific dimension of the cluster array have edge processor ports coupled to corresponding network switches of the same dimension. In this example, rows of processor groups 1901-1916 in the cluster array have edge processor ports coupled to corresponding network switches 1920-1923 of the same row, while columns of processor groups in the cluster array have edge processor ports coupled to corresponding network switches 1930-1933 of the same column. Figure 12 and 18 An example of such a connection is shown, but omitted here. In various embodiments, one or more row network switches 1920-1923 may be coupled to one or more column network switches 1930-1933. In this example, row network switch 1920 is coupled to column network switches 1930 and 1931, row network switch 1921 is coupled to column network switches 1931 and 1932, row network switch 1922 is coupled to column network switches 1932 and 1933, and row network switch 1923 is coupled to column network switches 1930 and 1933. This advantageously improves fault tolerance or can support mapping of Hamiltonian cycles across dimensions (e.g., when mapping processor sequences larger than a single dimension together). As described above, indirect cross-dimensional connections can be made between the intermediate (layer 1) row network switches and column network switches as described above.

[0055] Scheduling and failure

[0056] Certain embodiments of the present disclosure can distinguish based on job size (e.g., between small jobs and large jobs). For example, a small job can be defined as a job that fits within a processor group. For example, such a job can only be scheduled to a processor group that accommodates other small jobs. A large job can be distributed across multiple processor groups. In this case, the smallest increment of allocation can be, for example, a single processor group. For example, if the number of nodes or geometry required for a job does not fit exactly into the number of nodes in a processor group, then nodes can go unused and simply forward traffic.

[0057] Various embodiments can route many configurations of Hamiltonian cycles and transparently replace processor groups when a failure occurs. For example, in a one-dimensional configuration, the system can swap any processor group in the same switch and any two processor groups, e.g., if enough uplinks can be reserved for maintaining a sequence across other switches. In higher dimensional configurations, the system can swap rows and columns (or contiguous blocks) together if enough links are available. For example, this can simplify finding new processor groups in the event of a failure or when allocating processing segments after operating for a period of time.

[0058] Routing

[0059] Once the logical partitioning of the network to processor groups and processors has been performed, Hamiltonian cycles can be routed. The cycles can have a constant bandwidth guarantee that reserves at least some links for communication between processors. For example, various embodiments described above for one-, two-, or more dimensions can include a closed loop of processors with a dedicated non-congested nearest neighbor communication loop across one or more processor groups and switches. The network can advantageously be optimized so that 100% of the link bandwidth is passed to the sequence of processors used in the Hamiltonian cycle. The implementation of bandwidth reservation and routing will depend on the network technology. A simple routing scheme based on policy or a more complex scheme based on ECMP routing can be used to establish the routing, while QoS buffers and bandwidth reservations can be used to guarantee a certain rate.

[0060] To route Hamiltonian cycles with a global view of the network, the network manager can allocate available links to the cycles and then mark them as used. Routing can then be programmed to use only those loops. If global traffic is required and a link is not available, a QoS scheme can occupy a certain percentage of each Hamiltonian cycle to implement this traffic. For example, if the local topology is longer than two processors in any dimension, some processors can forward global traffic to an internal processor. This can be done using a simplified mechanism, as the number of hops in the local topology can be controlled by its size.

[0061] The above description illustrates various embodiments of the present disclosure and examples of how aspects of these embodiments can be implemented. The various embodiments described above can advantageously include network topologies that can implement an over-subscription factor (e.g., 3: 1, with 48 client links and 16 uplinks), which means significant cost / power savings while still being able to maintain full Hamiltonian cycle bandwidth. In some embodiments, the network topologies described herein can include a software allocation / re-allocation approach that leverages locality (e.g., allocating neighboring processor groups to the same Layer 0 switch). The above examples and embodiments should not be considered as the only embodiments, but rather as examples of the flexibility and advantages of the present disclosure as defined by the appended claims. For example, while certain embodiments have been described in relation to particular process flows and steps, it will be apparent to those skilled in the art that the scope of the present disclosure is not necessarily limited to the described process and steps. Steps described in series can be performed in parallel, the order of steps can be changed, and steps can be modified, combined, added, or omitted. As another example, while certain embodiments have been described using a particular combination of hardware and software, it will be recognized that other combinations of hardware and software are possible, and that particular operations described as implemented in software can be implemented in hardware, and vice versa.

[0062] Embodiments of the present disclosure can further include one or more of the following example embodiments.

[0063] In one example embodiment, the present disclosure includes a computer system comprising: a plurality of processor groups, a processor group comprising a plurality of processors; and a plurality of network switches coupled to a subset of the plurality of processor groups, wherein at least a first subset of processors in a first processor group is configurable to form a sequence, and wherein the network switches are configurable to form at least one sequence across one or more of the plurality of processor groups to perform a neural network computation.

[0064] In one example embodiment, during performance of the neural network computation, at least a first subset of processors in a first processor group and at least a second subset of processors in at least a second processor group are coupled together to form a sequence comprising a string of processors, wherein each processor is in communication with an adjacent processor in the string of processors.

[0065] In one example embodiment, the one or more processor groups comprise processors configured in a string to form a one-dimensional array of processors.

[0066] In one example embodiment, the one or more processor groups comprise processors configured in rows and columns to form a two-dimensional array of processors.

[0067] In one example embodiment, the plurality of processors in a processor group are configured in an N-dimensional array of processors.

[0068] In one example embodiment, processors in edge locations of the processor groups are coupled to multiple network switches.

[0069] In one example embodiment, edge processor ports on opposite sides of a row of processors across the first plurality of processor groups are coupled to a first network switch; and edge processor ports on opposite sides of a column of processors across the second plurality of processor groups are coupled to a second network switch.

[0070] In one example embodiment, the processor groups are coupled to a plurality of row network switches and a plurality of column network switches corresponding to rows and columns of processors in each of the processor groups across the plurality of switch planes.

[0071] In one example embodiment, one or more row network switches are coupled to one or more column network switches.

[0072] In one example embodiment, the processor groups are configured in a multi-dimensional cluster array.

[0073] In one example embodiment, processor groups of a particular dimension of the cluster array have edge processor ports coupled to corresponding same-dimension network switches.

[0074] In one example embodiment, multiple network switches along a particular dimension are coupled together through multiple intermediate network switches.

[0075] In one example embodiment, rows of processor groups in the cluster array have edge processor ports coupled to corresponding same-row network switches and columns of processor groups in the cluster array have edge processor ports coupled to corresponding same-column network switches.

[0076] In one example embodiment, row network switches are coupled together through one or more intermediate row network switches and column network switches are coupled together through one or more intermediate column network switches.

[0077] In one example embodiment, one or more intermediate row network switches are coupled to one or more intermediate column network switches.

[0078] In one example embodiment, multiple network switches are directly coupled together.

[0079] In one example embodiment, multiple network switches are coupled to one or more intermediate network switches.

[0080] In one example embodiment, the multiple network switches and the one or more intermediate network switches form a two-tiered switching network.

[0081] In one example embodiment, each network switch of the plurality of network switches is coupled to a plurality of intermediate network switches so as to be coupled in series the processors of a subset of the plurality of processor groups.

[0082] In one example embodiment, the processors along a first dimension of the plurality of processor groups are coupled to the plurality of network switches.

[0083] In one example embodiment, the first dimension is a row of processors.

[0084] In one example embodiment, the first dimension is a column of processors.

[0085] In one example embodiment, the neural network computation is a computation for training a neural network.

[0086] In one example embodiment, the plurality of processors configured to form a sequence are configured to process a partitioned neural network.

[0087] In one example embodiment, the processors configured to form a sequence are configured to process a plurality of local layers of a partitioned neural network.

[0088] In one example embodiment, the processors configured to form a sequence are configured to process one or more pipeline stages of a partitioned neural network.

[0089] In one example embodiment, the processors configured to form a sequence are configured to adjust weights across a plurality of instances of a neural network.

[0090] In another example embodiment, the disclosure includes a computer system comprising: a plurality of processor groups, a processor group comprising a plurality of processors; a plurality of network switches, wherein the plurality of network switches are coupled to a subset of the plurality of processor groups; and a plurality of intermediate network switches coupled to a subset of the plurality of network switches, wherein at least one subset of processors of the processor groups, one or more of the plurality of network switches, and one or more of the intermediate network switches are configured to perform a Hamiltonian cycle for one or more of: data parallel neural network computation, pipeline parallel neural network computation, and layer parallel neural network computation.

[0091] In another example embodiment, the disclosure includes a method for processing a neural network, the method comprising: configuring a plurality of processors arranged in a plurality of processor groups to perform a neural network computation, wherein at least one subset of processors of the processor groups are configured to form a sequence of processors; a plurality of network switches are configured to couple together a subset of the plurality of processor groups to form at least one sequence of processors across one or more subsets of the plurality of processor groups; and performing the neural network computation by at least one sequence of processors of the plurality of processor groups.

[0092] In another example embodiment, the present disclosure includes a non-transitory computer-readable storage medium having stored thereon program code executable by a computer system for processing a neural network, the program code causing the computer system to: configure a plurality of processors arranged in a plurality of processor groups to perform neural network computations, wherein at least a subset of the processors in a processor group are configured to form a processor sequence; a plurality of network switches are configured to couple together a subset of the plurality of processor groups to form at least one processor sequence across one or more subsets of the plurality of processor groups; and perform the neural network computations by at least one processor sequence in the plurality of processor groups.

[0093] Thus, the description and drawings are to be regarded as illustrative in nature and not restrictive. Other arrangements, embodiments, implementations and equivalents will be apparent to those skilled in the art and can be employed without departing from the spirit and scope of the disclosure as described in the claims.

Claims

1. A computer system comprising: a plurality of processor groups including a plurality of serially configured processors for processing a partitioned neural network, wherein different processors in the plurality of processor groups process one or more different layers, stages, or model instances of the partitioned neural network; and a plurality of network switches coupled to a subset of the plurality of processor groups through edge processors in the serially configured processors in each processor group, wherein at least a subset of the processors in the processor groups are configured to form a sequence such that each processor communicates data for the layers, stages, or model instances with at least two adjacent processors in the sequence, and wherein the network switches are configurable to form at least one sequence across one or more of the plurality of processor groups to perform a neural network computation.

2. The computer system of claim 1, wherein during performance of the neural network computation, at least a first subset of processors in a first processor group and at least a second subset of processors in at least a second processor group are coupled together to form a sequence including a string of processors, wherein each processor communicates with adjacent processors in the string of processors.

3. The computer system of claim 1, wherein one or more processor groups include processors serially configured to form a one-dimensional array of processors.

4. The computer system of claim 1, wherein one or more processor groups include processors configured in rows and columns to form a two-dimensional array of processors.

5. The computer system of claim 1, wherein the plurality of processors in the processor groups are configured as an N-dimensional array of processors.

6. The computer system of claim 1, wherein: edge processor ports on opposite sides of a row of processors across a first plurality of processor groups are coupled to a first network switch; and edge processor ports on opposite sides of a column of processors across a second plurality of processor groups are coupled to a second network switch.

7. The computer system of claim 1, wherein the processor groups are coupled to a plurality of row network switches and a plurality of column network switches corresponding to rows and columns of processors in each processor group across a plurality of switch planes.

8. The computer system of claim 7, wherein one or more of the row network switches are coupled to one or more of the column network switches.

9. The computer system of claim 1, wherein the processor groups are configured as a multi-dimensional cluster array.

10. The computer system of claim 9, wherein processor groups along a particular dimension of the cluster array that indicates either a row of processors or a column of processors have edge processor ports coupled to a network switch corresponding to the same dimension.

11. The computer system of claim 10, wherein a plurality of network switches along the particular dimension are coupled together through a plurality of intermediate network switches.

12. The computer system of claim 9, wherein: ​ a row of processor groups in the cluster array has edge processor ports coupled to a corresponding same row network switch, and a column of processor groups in the cluster array has edge processor ports coupled to a corresponding same column network switch.

13. The computer system of claim 12, wherein: the row network switches are coupled together through one or more intermediate row network switches, and the column network switches are coupled together through one or more intermediate column network switches.

14. The computer system of claim 12, wherein one or more intermediate row network switches are coupled to one or more intermediate column network switches.

15. The computer system of claim 1, wherein the plurality of network switches are coupled together directly.

16. The computer system of claim 1, wherein the plurality of network switches are coupled to one or more intermediate network switches.

17. The computer system of claim 16, wherein the plurality of network switches and the one or more intermediate network switches form a two-tiered switching network.

18. The computer system of claim 16, wherein each network switch of the plurality of network switches is coupled to a plurality of intermediate network switches to couple processors in the subset of the plurality of processor groups in series.

19. The computer system of claim 16, wherein processors in the plurality of processor groups along a first dimension are coupled to the plurality of network switches, the first dimension indicating a row of processors or a column of processors.

20. The computer system of claim 1, wherein the neural network computation is a computation for training a neural network.

21. The computer system of claim 1, wherein the processors configured to form the sequence are configured to process a plurality of local layers of the partitioned neural network.

22. The computer system of claim 1, wherein the processors configured to form the sequence are configured to process one or more pipeline stages of the partitioned neural network.

23. The computer system of claim 1, wherein the processors configured to form the sequence are configured to adjust weights across a plurality of instances of the neural network.

24. A computer system, comprising: a plurality of processor groups, the processor groups including a plurality of serially configured processors to process a partitioned neural network, wherein different processors in the plurality of processor groups process one or more different layers, stages, or model instances of the partitioned neural network; a plurality of network switches, wherein the plurality of network switches are coupled to a subset of the plurality of processor groups through edge processors in the serially configured processors in each processor group; and a plurality of intermediate network switches coupled to a subset of the plurality of network switches. wherein at least a subset of the processors in the processor groups, one or more of the plurality of network switches, and one or more of the intermediate network switches are configured in a sequence such that each processor communicates data for the layer, stage, or model instance with at least two adjacent processors in the sequence to perform a Hamiltonian cycle for one or more of: data-parallel neural network computation, pipelined-parallel neural network computation, and layer-parallel neural network computation.

25. A method for processing a neural network, the method comprising: configuring a plurality of processors arranged in a plurality of processor groups to perform neural network computation on a partitioned neural network, wherein at least a subset of the processors in the processor groups are serially configured to form a sequence of processors, and wherein different processors in the plurality of processor groups process one or more different layers, stages, or model instances of the partitioned neural network; configuring a plurality of network switches to couple together edge processors in the serially configured processors in each processor group with a subset of the plurality of processor groups to form a sequence of at least one processor across one or more subsets of the plurality of processor groups such that each processor communicates data for the layer, stage, or model instance with at least two adjacent processors in the sequence; and performing the neural network computation by the sequence of at least one processor across the plurality of processor groups.

26. A non-transitory computer-readable storage medium having stored thereon program code executable by a computer system for processing a neural network, the program code causing the computer system to: configure a plurality of processors arranged in a plurality of processor groups to perform neural network computation on a partitioned neural network, wherein at least a subset of the processors in the processor groups are serially configured to form a sequence of processors, and wherein different processors in the plurality of processor groups process one or more different layers, stages, or model instances of the partitioned neural network; configure a plurality of network switches to couple together edge processors in the serially configured processors in each processor group with a subset of the plurality of processor groups to form a sequence of at least one processor across one or more subsets of the plurality of processor groups such that each processor communicates data for the layer, stage, or model instance with at least two adjacent processors in the sequence; and perform the neural network computation by the sequence of at least one processor across the plurality of processor groups.

Images