HYBRID DATA MODEL PARALLELITY FOR EFFICIENT DEEP LEARNING

Abstract

Method (500) which exhibits: Selecting a hybrid parallelism technique (140) for distributing the operational load of a layer of a neural network across an array (200, 600) of processors (110, 155), wherein each processor in the array of processors can transfer data to neighboring processors in a first direction (610) and in a second direction (605); and Assigning tasks corresponding to the layer of the neural network to the array of processors using the selected hybrid parallelism technique (145), wherein the hybrid parallelism technique includes the use of a first parallelism technique when data is transferred between processors in the array of processors in the first direction (610), and the use of a second, different parallelism technique when data is transferred between processors in the array of processors in the second direction (605).

Description

BACKGROUND

[0001] The present invention relates to the use of hybrid parallelism techniques to assign layers in a neural network to an array of processors.

[0002] Training a neural network (NN) is a time-consuming process. Consequently, many training platforms use an array of processors (e.g., a 2D torus) to distribute the workload of each layer of the NN. Several well-known techniques exist for distributing the workload across multiple processors. One such technique is data parallelism, where the workload is divided by the characteristics (or inputs) of the layers. With this technique, each processor can perform all tasks for a given batch (or minibatch) of training data. For example, using an image processor NN, each processor can be assigned to process a specific image. To do this, the weights (or cores) for that layer must be passed to each of the processors.In NN layers where the weights are larger than the features (referred to herein as weight-heavy layers), using data parallelism is inefficient due to weight decrementing between chips, where weights are synchronized at the end of each batch or mini-batch training. This is a chip-to-chip computationally expensive operation and often the main weakness of data parallelism.

[0003] Another technique is model parallelism, where the workload for a layer in the output dimension is distributed. This means that the first set of several features is computed in one processor, the next set of several features is computed in another processor, and so on. This avoids having to send all the layer's weights to the processor, thus preventing weight degradation. However, in a neural network, each output feature typically requires the entire input channel. That is, each processor still needs the entire input of the layer (e.g., the output from the previous layer) to compute a new feature. When using model parallelism, the input is replicated across all processors (a process known as "activating").Before the next layer in the neural network can begin, each processor must receive the output computed by every other chip—that is, the activations of all processors are exchanged. Thus, model parallelism is more efficient than data parallelism for weight-heavy layers, but less efficient for feature-heavy layers (where there is more feature data (or input data) than weight data).

[0004] SONG, Linghao [et al.]: HyPar: Towards hybrid parallelism for deep learning accelerator array. In: 2019 IEEE international symposium on high performance computer architecture (HPCA). IEEE, 2019. pp. 56-68. ISBN 978-1-7281-1445-3. https: / / ieeexplore.ieee.org / document / 8675232 [accessed on 13.08.2025] reveals a method for optimizing hybrid parallelism when training deep neural networks across multiple accelerators to minimize communication overhead and improve performance and energy efficiency compared to conventional data or model parallelism.

[0005] US 2018 / 0 293 492 A1 discloses a machine-readable instruction medium that provides an interface for defining a neural network using domain-specific terminology, wherein the interface enables the selection of a network topology and abstracts the low-level details of distributed training.

[0006] EP 000003367310 A1 discloses a computer-implemented method for modifying a deep neural network by parallelizing selected layers across multiple processes, wherein additional layers are inserted to distribute the input data and to combine the output data in order to enable parallel execution of the selected layers. SUMMARY

[0007] According to one embodiment of the present invention, a method is provided that includes selecting a hybrid parallelism technique for distributing the workload of a layer of a neural network across an array of processors, wherein each processor in the array of processors can transfer data to neighboring processors in a first direction and in a second direction. The method further includes assigning tasks corresponding to the layer of the neural network to the array of processors using the selected hybrid parallelism technique, wherein the hybrid parallelism technique comprises using a first parallelism technique when data is transferred between processors in the array of processors in the first direction, and using a second, different parallelism technique when data is transferred between processors in the array of processors in the second direction.

[0008] In another embodiment of the present invention, a system comprising an array of processors and an assignor of a neural network is provided.The assigner is configured to select a hybrid parallelism technique for distributing the workload of a layer of a neural network across an array of processors, where each processor in the array of processors can transfer data to neighboring processors in a first direction and in a second direction, and assign tasks corresponding to the layer of the neural network using the selected hybrid parallelism technique of the array of processors, wherein the hybrid parallelism technique includes using a first parallelism technique when data is transferred between processors of the array of processors in the first direction, and using a second, different parallelism technique when data is transferred between processors of the array of processors in the second direction.

[0009] In a further embodiment of the present invention, a computer-readable storage medium is embodied on which computer-readable program code is represented, wherein the computer-readable program code is readable by one or more computer processors in order to perform an operation.The operation comprises selecting a hybrid parallelism technique to distribute the workload of a layer of a neural network across an array of processors, wherein each processor in the array of processors can transfer data to neighboring processors in a first direction and in a second direction, and assigning tasks corresponding to the layer of the neural network to the array of processors using the selected hybrid parallelism technique, wherein the hybrid parallelism technique includes using a first parallelism technique when data is transferred between processors in the array of processors in the first direction, and using a second, different parallelism technique when data is transferred between processors in the array of processors in the second direction. Brief description of the different views in the drawings

[0010] The following are exemplary embodiments of the invention, with reference to the accompanying drawings, which: Fig. 1 illustrates a NN training system according to an embodiment described herein. Fig. 2 NN training hardware according to an embodiment described herein is illustrated. Fig. Three different dimensions in a NN are illustrated according to an embodiment described herein. Fig. 4 is a flowchart for determining an optimal technique for performing the partitioning of NN layers under the array of processors according to an embodiment described herein. Fig. 5 is a flowchart for selecting a hybrid parallelism technique according to an embodiment described herein. Fig. Figure 6 illustrates a processor array forming x and y rings according to an embodiment described herein. Fig. Figure 7 illustrates a matrix for selecting a parallelism technique for a NN layer according to an embodiment described herein. Fig. 8 illustrates a classification of parallelism techniques based on the properties of the NN layer according to an embodiment described herein. DETAILED DESCRIPTION

[0011] Instead of being limited to data parallelism or model parallelism, the embodiments described herein introduce hybrid parallelism techniques, where a mixture of data and model parallelism can be used to distribute the workload of a layer among an array of processors (e.g., an array of integrated circuits (ICs) or chips). In one embodiment, the array of processors comprises x-rings in a first direction (X-direction) and y-rings in a second direction (Y-direction). When configuring the array, the bandwidth of the processors in one direction can be greater than the bandwidth in the other direction. That is, the system designer can intentionally allocate more data transmission links for exchanging data between processors in the X-direction than in the Y-direction.Each layer can then be characterized as being more feature-heavy or more weight-heavy. Depending on this characterization, the operational load of a neural network layer in the array can be assigned using a hybrid parallelism technique instead of relying solely on data parallelism or solely on model parallelism. For example, a neural network layer might be slightly more weight-heavy than feature-heavy.Since model parallelism is more efficient when running weight-heavy layers, the operational load of the NN layer can be split using a first hybrid parallelism technique, where data parallelism in the X direction (where the bandwidth is higher) is used to achieve gradient reduction along a fast X-direction link, and model parallelism in the Y direction (where the bandwidth is lower, but model parallelism is inherently more efficient for weight-heavy layers).Conversely, if a NN layer is feature-heavy rather than weight-heavy, the operational load can be split using a second hybrid parallelism technique, where model parallelism is used in the X-direction to minimize the negative impact of feature activation, while data parallelism is used in the Y-direction, since data parallelism is inherently more efficient for weight-heavy layers.

[0012] While hybrid parallelism techniques can be used with an array of processors where the bandwidth in one direction differs from the bandwidth in the direction perpendicular to it, hybrid parallelism techniques can also be used when the bandwidth in both directions of the array is the same.

[0013] Fig. Figure 1 illustrates a neural network (NN) training system 100 according to an embodiment described herein. The training system 100 comprises a data processing system 105 that uses a selected technique 145 to distribute the operational load of each layer in an NN assigned to NN training hardware 150. Although the embodiments described herein describe the distribution of an operational load of a layer during NN training, they are limited to this. It may be advantageous to distribute an operational load using the techniques discussed below in situations other than NN training.

[0014] The data processing system 105 comprises a processor 110, which represents any number of processing elements with any number of processing cores, and memory 115, which can include volatile memory elements, non-volatile memory elements, and combinations thereof. As shown, the memory 115 comprises two software applications that run in the data processing system 105: a neural network (NN) assigner 120 and a power estimator 125. The NN assigner 120 determines the selected technique 145, which is used to assign the operational load of the layers in the neural network to the NN training hardware 150. In one embodiment, the NN assigner 120 selects the technique 145 in response to information generated by the power estimator 125 about each layer in the NN. That is, the NN assigner 120 can select a different technique 145 for the different layers in the NN.

[0015] The performance estimator 125 models (or simulates) the neural network (NN) layers to estimate layer performance using a variety of parallelism techniques. In this example, the performance estimator 125 evaluates the layers to determine their performance when using the data parallelism technique 130, the model parallelism technique 135, and one or more hybrid parallelism techniques 140. In one embodiment, the hybrid parallelism techniques 140 are any techniques that are a mixture of two other parallelism techniques used in combination to distribute a task's workload among the various hardware elements in the NN training hardware 150. That is, the hybrid parallelism technique 140 can be one or more techniques that are a combination of the data parallelism technique 130 and the model parallelism technique 135.However, the hybrid parallelism technique 140 is not limited to a combination of techniques 130, 135, but could include a combination of other types of parallelism techniques used to distribute the operational load of a layer under the training hardware 150.

[0016] The NN training hardware 150 comprises multiple processors 155, each comprising multiple cores 160. The cores 160 comprise multiple processing element arrays (PE arrays) (e.g., systolic arrays) formed using multiple PEs 170. As discussed below, the selected technique 145 distributes the workload of a layer among the processors 155. If the selected technique 145 is the data parallelism technique 130, the workload is distributed through minibatches, with each minibatch being assigned to a different processor 155 (e.g., an image used in training an image processing NN is sent to each processor 155). If the selected technique 145 is the model parallelism technique 135, each processor 155 is assigned a subset of the features in the output data (where there are one or more subsets).The 155 processors can then work in parallel to perform their assigned tasks. For example, each 155 processor can process one image at a time, or each 155 processor can process its assigned subset of features simultaneously.

[0017] Fig. Figure 2 illustrates NN training hardware according to an embodiment described herein. Fig. Figure 2 illustrates a 2D array 200 comprising the processors 155, which are interconnected both horizontally (in an X direction) and vertically (in a Y direction) with neighboring processors 155. Therefore, the processors can share processed data. This data sharing is described in more detail below when the hybrid parallelism techniques are discussed.

[0018] Each processor 155 comprises multiple cores 160, which are connected for data transmission purposes using an on-chip interface 202 that allows the cores 160 to exchange data with external memory 205 (e.g., RAM) in the NN training hardware. Each core 160, in turn, comprises multiple PE arrays 165, which are connected to working buffer memory 210, representing any number of caches (e.g., Level 1 and Level 2 caches) that constitute on-chip memory.

[0019] Each PE array 165 comprises several PEs 170 interconnected to form an array or grid (e.g., a systolic array). The inputs and outputs of the PE array 165 include first-in-first-out (FIFO) buffers 215. Each PE 170 comprises multiply-accumulate (MAC) units 220 connected to a registry file 225.

[0020] The operating load of a NN layer can be determined by the factors specified in Fig. The workload is distributed among the various hardware components illustrated in Figure 2. However, the techniques described below focus on distributing the workload of a NN layer among the processors 155. Other techniques can then be used to distribute the workload assigned to each processor 155 among the cores 160, the PE arrays 165, and the PEs 170. That is to say, while the following embodiments discuss selecting an optimal parallelism technique for distributing the workload among the processors 155, the workload can further be distributed among the hardware elements within the processors 155 using additional parallelism techniques.

[0021] Fig. Figure 3 illustrates different dimensions in a neural network according to an embodiment described herein. That is to say, Fig. Figure 3 illustrates different dimensions that can be used to partition a convolutional layer in a neural network. The convolutional layer's operational load can input three data structures: input (Eing), output (Ausg), and weights (kernels). NEing is the number of features (or faces) in the data received from the preceding layer(s) in the neural network. That is, the input has a depth of NEing. Each feature or face has dimensions expressed by Ni and Nj (e.g., a feature size of Ni x Nj). When performing a convolution, each pixel in a feature is processed using one of the kernels (e.g., a dot product). The neural network has a number of NAusg kernels, where each kernel has a height Ki, a width Kj, and a depth of NEing.

[0022] Convolution works by performing the dot product of one of the kernels on a pixel in one of the features in the input. This outputs a scalar value (e.g., a pixel) that is stored in the output. Consequently, the depth of the output is Nout (e.g., the number of features in the output), which is identical to the number of kernels. Moving the kernel using a step size when performing the dot product on other pixels in the feature or face of the input results in a feature (or face) in the output. Assuming a step size of 1, the features or faces of the output have the same width and height (i and j) as the features of the input.

[0023] Performing a convolution results in four dimensions: NEin, Nij, Kij, and NAusg, for distributing the operational load of the convolution layer. However, training a neural network also involves minibatches (mb), where a neural network can be trained using different input training data (e.g., different training images), all using the same weights (kernels) to generate multiple output results. Thus, the mb is a fifth dimension that, together with the other four dimensions, can be used to represent the operational load of a convolution layer as: N{Ein, Ausg, Nij, Kij, mb}.

[0024] Other types of layers can be expressed using different dimensions. For example, when performing matrix multiplication on fully connected layers in a NN, the values ​​Nij and Kij are 1. In any case, the embodiments described herein can be used to distribute different types of layers, which may have different types (and numbers) of dimensions.

[0025] Fig. Figure 4 is a flowchart of a method 400 for determining an optimal technique for performing the partitioning of NN layers among the array of processors according to an embodiment described herein. In block 405, the assigner selects a layer in an NN. In one embodiment, the method 400 is used to iterate through each layer in the NN and select an optimal parallelism technique.

[0026] At block 410, the neural network assigner determines an optimal technique for distributing the layer's execution across a 2D array of processors. That is, the neural network assigner can determine which of the... Fig. 1. The techniques illustrated (the data parallelism technique 130, the model parallelism technique 135 or a hybrid parallelism technique 140) are the optimal technique for the layer in question.

[0027] In one embodiment, the 2D array of processors is a 2D torus in which the processors are interconnected. For example, the processors can be interconnected with adjacent processors in two different directions (e.g., x / Y or north / south) for data transmission purposes. In one embodiment, the bandwidth for processor-to-processor data transmission can vary depending on the direction. For example, the 2D array can be designed such that data transmission between processors in the X direction has a higher bandwidth than data transmission between processors in the y direction. The details for selecting the optimal technique to use for distributing the workload among the processors in a 2D array are described below. Fig. 5 described.

[0028] At block 415, the NN assigner determines whether further layers remain in the NN. That is, the NN assigner determines whether it has selected an optimal parallelism technique for each layer in the NN. If not, the process returns to block 405; however, if it has, the process continues to block 420.

[0029] In block 420, during the training of the neural network (NN), the NN assigns tasks to the 2D array of processors using optimal techniques. If the selected technique is data parallelism or model parallelism, the processors exchange data in both directions by transmitting data according to these techniques. That is, if data parallelism is selected, the processors exchange weights / cores in both the X and Y directions of the 2D array (also known as the gradient direction). If model parallelism is selected, the processors exchange input features along both directions.

[0030] However, if a hybrid parallelism technique is chosen, the processors exchange data in one direction according to data parallelism and in the other direction according to model parallelism. Accordingly, the parallelism technique implemented in the direction of the 2D array with the higher bandwidth has improved performance relative to the technique implemented in the other direction, which has a lower bandwidth. The details of this trade-off are discussed below.

[0031] Fig. Figure 5 is a flowchart of a method 500 for selecting a hybrid parallelism technique according to an embodiment described herein. The method 500 describes several techniques for performing block 410 in the method 400 to select an optimal parallelism technique for a layer in a neural network. For clarity, the method 500 is shown together with Fig. 6 described.

[0032] In block 505, the performance estimator estimates the layer's performance using a variety of parallelism techniques. For example, at each layer, the performance estimator can use a model or simulation to estimate how the layer would perform on the NN training hardware using each of the different parallelism techniques, such as data parallelism, model parallelism, or hybrid parallelism.

[0033] In one embodiment, instead of modeling or simulating each parallelism technique for each layer, the performance estimator can use heuristics to limit the number of techniques to be estimated. For example, the performance estimator can generate a weight-to-feature ratio that can be used to quickly determine which parallelism technique should be employed. If the layer contains significantly more weight data than feature data, according to a predefined threshold, the performance estimator can assume that model parallelism is the optimal parallelism technique without performing any further modeling or simulation. Conversely, if the weight-to-feature ratio indicates that the layer contains significantly more feature data than weight data, data parallelism is considered the optimal parallelism technique.Therefore, the performance estimator does not need to model or test every parallelism technique for every layer.

[0034] As mentioned above, the 2D array of processors can include multiple dimensions, where at least one dimension has a higher bandwidth for processor-to-processor data transfer than another dimension. Fig. Figure 6 illustrates an array of processors forming x- and y-rings in a 2D array 600 according to an embodiment described herein. The x-ring 610 specifies data transmission links that processors 155 use to transfer data in the X-direction, while the y-ring 605 specifies data transmission links that processors 155 use to transfer data in the Y-direction.

[0035] In one embodiment, the 2D array 600 can include limited resources for data transmission links forming the x- and y-rings. Instead of allocating the same number of resources to both rings, the 2D array 600 has different bandwidths for the x- and y-rings. That is, the 2D array 600 is an asymmetric array (or symmetric torus) in which the bandwidth for processor-to-processor data transmission differs depending on the direction of data transmission. For example, each processor-to-processor connection can include multiple physical links. To allocate more bandwidth to the x-rings 610 than to the y-rings 605, the 2D array can have more physical links allocated to each processor-to-processor connection in the x-direction than to the processor-to-processor connections in the y-direction. As a result, the 155 processors are capable of handling data in the X direction (e.g.Transferring data between processors to the left and right is faster than transferring it in the Y direction (e.g., to processors above and below). In this way, the hybrid parallelism techniques described herein are effectively employed, where the less efficient parallelism techniques can be used when transferring data between processors 155 in the X direction (to take advantage of the higher bandwidth), while the more efficient parallelism technique is used when transferring data between processors 155 in the Y direction. For some NN layers, this may not be the case, and it is optimal to use the more efficient parallelism technique in the direction with the higher bandwidth.

[0036] Although the Fig. 2 and Fig. 6. A 2-D array of processors, but the embodiments are not limited to this. If a NN is implemented using an N-dimensional array of processors (where N is greater than 3), at least one of the dimensions (or facilities) can be designed to have a higher bandwidth than at least one other dimension. Therefore, the embodiments described herein can be used to select a hybrid parallelism model where different dimensions use different parallelism techniques.

[0037] Referring again to Procedure 500, the NN assigner at Block 510 uses performance estimates generated at Block 505 to determine whether data or model parallelism is the optimal parallelism technique for the layer. If so, Procedure 500 continues to Block 515, where the NN assigns the data or model parallelism technique for the layer. This means that when splitting tasks and configuring the NN training hardware, data is transferred in both directions in the 2D using the same parallelism technique, regardless of whether it is data model parallelism or model parallelism. Therefore, the same parallelism model is used in both directions, even if one direction might have a higher bandwidth than the other.

[0038] Otherwise, procedure 500 continues to block 520, where the NN assigner uses performance estimates to determine whether the currently evaluated layer is feature-heavy rather than weight-heavy. As mentioned above, for NN layers where the features are more numerous than the weights (e.g., a heavy layer), the data parallelism technique is more efficient than the model parallelism technique. That is, for some NNs, each output feature requires the entire input channel. This means that each processor still needs the entire input of the layer (e.g., the output from the previous layer) to calculate a new feature. When using model parallelism, the input is replicated across all processors, or activations are performed across all processors.Thus, if the layer is a feature-heavy layer, procedure 500 proceeds to block 535, where the NN assigner selects the model parallelism technique for use in the direction with the highest bandwidth in the 2D array. In this way, the negative impact of the activations on the efficient operation of the NN is less pronounced, as these activations are transmitted over the higher-bandwidth processor-to-processor links. At block 540, the NN assigner selects the data parallelism technique for use in the direction with the lowest bandwidth in the 2D array.

[0039] In contrast, if the layer is a more weight-heavy layer, the procedure proceeds from block 520 to block 525, where the NN assigner selects the data parallelism technique for use in the direction with the highest bandwidth in the 2D array, and then to block 530, where the NN assigner selects the model parallelism technique for use in the direction with the lowest bandwidth. Since data parallelism is used on the higher-bandwidth connections, the negative impact of gradient reduction is mitigated. In this way, procedure 500 indicates the use of two different hybrid parallelism techniques.Blocks 535 and 540 represent a hybrid technique, using model parallelism in the dimension with the higher bandwidth of the array of processors and data parallelism in the dimension with the lower bandwidth, while blocks 525 and 530 represent the other, using data parallelism in the dimension with the higher bandwidth of the array of processors and model parallelism in the dimension with the lower bandwidth.

[0040] However, as mentioned above, it can be more efficient to use data parallelism in the direction with higher bandwidth for feature-heavy layers, or model parallelism in the direction with higher bandwidth for weight-heavy layers. For example, long short-term memory (LSTM) layers, although weight-heavy, prefer hybrid parallelism, using model parallelism in the higher-bandwidth direction and data parallelism in the lower-bandwidth direction. In this case, gradient reduction is quite rare (occurring only after all time steps have been completed) and can occur on slower-direction connections without impacting performance.

[0041] Although Method 500 is discussed in the context of an asymmetric array where the bandwidth varies, it can also be used in a symmetric array of processors where the bandwidth is the same in both the X and Y directions. For example, the NN training hardware might include more processors than there are minibatches to be tested. In this case, using only data parallelism is inefficient because the NN assigner could assign a batch (e.g., an image) to any single processor. Using a hybrid parallelism model can improve efficiency in this situation. In another example, the NN assigner might not have enough features to distribute the data across all processors (e.g., NOutput is smaller than the number of processors in the array).Therefore, using only model parallelism can be inefficient, but partitioning the dimensions using a hybrid parallelism model can lead to more efficient use of the NN training hardware. In these examples, it doesn't matter which parallelism is used in which direction, since the directions in a symmetric array have the same bandwidth.

[0042] Fig. Figure 7 illustrates a matrix 700 for selecting a parallelism technique for a NN layer according to an embodiment described herein. The matrix 700 illustrates four parallelism techniques and their relationships to each other. The x-axis indicates the parallelism technique used in the x-direction of the 2D array, while the y-axis indicates the parallelism technique used in the y-direction of the 2D array.

[0043] The "All Data" parallelism technique is one that uses only data parallelism when the operational load is distributed. Therefore, parallelism is achieved in both the X and Y directions of the 2D array using data parallelism.

[0044] The parallelism technique “DataXModelY” is a first hybrid parallelism technique that uses data parallelism in the X direction of the 2-D array, but uses model parallelism in the Y direction of the 2-D array.

[0045] The parallelism technique “ModelXDataY” is a second hybrid parallelism technique that uses model parallelism in the X direction of the 2-D array, but uses data parallelism in the Y direction of the 2-D array.

[0046] The "All Model" parallelism technique uses only model parallelism when the operational load is distributed. Therefore, parallelism in the X and Y directions of the 2D array is achieved using model parallelism.

[0047] Fig.Figure 8 illustrates a classification of parallelism techniques based on the properties of the NN layer according to an embodiment described herein. The hybrid parallelism techniques represent a compromise between the advantages and disadvantages of data- and model-based parallelism. For NN layers that are primarily feature-heavy, using the "all data" parallelism technique in both directions of the 2D array is likely to result in the most efficient use of the NN training hardware. Conversely, for NN layers that are primarily weight-heavy, using the "all model" parallelism technique in both directions of the 2D array is likely to result in the most efficient use of the NN training hardware.

[0048] The use of hybrid parallelism techniques (i.e., "DataXModelY" and "ModelXDataY") can improve efficiency relative to "All Data" or "All Model" techniques for layers that are not primarily feature-heavy or weight-heavy. That is, for layers falling between these two extremes, it can be more efficient to use a hybrid parallelism technique. Specifically, for layers that the performance estimator determines to be feature-heavy rather than weight-heavy, the "ModelXDataY" hybrid parallelism can lead to improved efficiency (assuming the X-direction of the 2D array has more bandwidth than the Y-direction). This means that model parallelism is performed in the X-direction, while data parallelism is used in the Y-direction.This allows the NN training hardware to mitigate the negative impact of the slower input activations of model parallelism, as it is performed in the X-direction where the bandwidth is highest, while the inherently more efficient data parallelism is used in the Y-direction where the bandwidth is lower. For example, convolutional layers with smaller feature sizes (e.g., the final set of convolutional layers in ResNet and GoogLeNet), which are only marginally feature-heavy, prefer the "ModelXDataY" parallelism to optimize activation / error rotation.

[0049] For NN layers that are weight-heavy rather than feature-heavy, the hybrid parallelism “DataXModelY” can lead to improved efficiency because data parallelism is performed in the direction of higher bandwidth (i.e., the X-direction), thus minimizing the negative impact of weight / gradient reduction, while the inherently more efficient model parallelism is performed in the direction of lower bandwidth. For example, small, fully connected layers (e.g., last layers of VGG / ResNet) that are moderately weight-heavy favor “DataXModelY” because it implements gradient reduction along connections in the X-direction.

[0050] The descriptions of the various embodiments of the present invention are intended for illustrative purposes only and are not meant to be exhaustive or limited to the disclosed embodiments. Many modifications and variations are conceivable for those skilled in the art without these representing a deviation from the scope of protection and the fundamental principles of the described embodiments. The terminology used herein has been chosen to explain the fundamental principles of the embodiments, their practical application, and the technical improvements compared to technologies found on the market, or to enable others with the relevant expertise to understand the embodiments disclosed herein.

[0051] The foregoing refers to embodiments presented in this disclosure. However, the scope of protection of this disclosure is not limited to specific described embodiments. Instead, any combination of the following features and elements is conceivable, regardless of whether they are associated with different embodiments, to realize and implement the invention. Although embodiments disclosed herein may offer advantages over other possible solutions or over the prior art, a specified embodiment does not further limit the scope of protection of this disclosure, irrespective of whether a particular advantage is achieved by a specified embodiment.Therefore, the aspects, features, embodiments, and advantages are merely illustrative and are not intended to be elements or limitations of the appended claims, except where expressly stated in a claim or claims. Likewise, reference to "the invention" is not to be interpreted as a generalization of any subject matter disclosed herein and is not to be regarded as an element or limitation of the appended claims, except where expressly stated in a claim or claims.

[0052] Aspects of the present invention may take the form of an embodiment fully realized in hardware, an embodiment fully realized in software (including, but not limited to, firmware, resident software, microcode, etc.), or an embodiment that combines software and hardware aspects, which are herein generally referred to as a “circuit”, “module”, or “system”.

[0053] The present invention may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium (or media) on which computer-readable program instructions are stored to cause a processor to execute aspects of the present invention.

[0054] The computer-readable storage medium can be a physical unit on which instructions for use by a unit for executing instructions can be stored and retained. The computer-readable storage medium can be, for example, but is not limited to, an electronic storage unit, a magnetic storage unit, an optical storage unit, an electromagnetic storage unit, a semiconductor storage unit, or any suitable combination thereof.A non-exhaustive list of more precise examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable read-only memory in the form of a compact disc (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically coded unit such as punched cards or raised structures in a groove with instructions recorded on them, or any suitable combination of the foregoing.A computer-readable storage medium, as used herein, is not to be interpreted as consisting of volatile signals per se, such as radio waves or freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through an optical fiber cable), or electrical signals transmitted via a cable.

[0055] The computer-readable program instructions described herein can be downloaded over a network, such as the internet, a local area network, a wide area network, and / or a wireless network, from a computer-readable storage medium to the relevant data processing units or to an external computer or storage device. The network may include copper transmission cables, fiber optic transmission lines, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in the data processing unit receives computer-readable program instructions from the network and forwards them for storage on a computer-readable storage medium within the respective data processing unit.

[0056] Computer-readable program instructions for performing operations of the present invention can be assembly instructions, ISA instructions (ISA = Instruction Set Architecture), machine instructions, machine-dependent instructions, microcode, firmware instructions, data for setting states, or either source code or object code written in any combination of one or more programming languages, including an object-oriented programming language such as Smalltalk, C++ or similar, and conventional procedural programming languages ​​such as the programming language "C" or similar programming languages.The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, via the internet using an internet service provider).In some embodiments, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), which execute computer-readable program instructions, can be used to personalize the electronic circuit by utilizing state information from the computer-readable program instructions, thus implementing aspects of the present invention.

[0057] Aspects of the present invention are described herein with reference to flowchart representations and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the invention. It will be clear that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented using computer-readable program instructions.

[0058] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a specialized computer, or other programmable data processing devices to create a machine such that the instructions executed by the processor of the computer or other programmable data processing devices provide the means to perform the functions / actions specified in a block or blocks of the flowchart(s) and / or block diagram(s).These computer-readable program instructions may also be stored in a computer-readable medium that can instruct a computer, other programmable data processing devices, or other units to function in a certain manner, such that the computer-readable medium with instructions stored on it constitutes a product containing instructions that implement the function / action specified in a block or blocks of the flowcharts and / or block diagrams.

[0059] The computer-readable program instructions can also be loaded into a computer, other programmable data processing devices, or other units to cause a series of operations to be performed on the computer, other programmable devices, or other units to create a computer-realized process, such that the instructions executed on the computer, other programmable devices, or units realize the functions / actions specified in a block or blocks of the flowcharts and / or block diagrams.

[0060] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this context, each block in the flowcharts or block diagrams can represent a module, segment, or section of instructions that includes one or more executable instructions for implementing the specified logic function(s). In some alternative implementations, the functions specified in the block may be executed in a different order than that shown in the figures. For example, two consecutively listed blocks may actually be executed essentially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functionality associated with them.Furthermore, it should be noted that each block of the block diagrams and / or flowchart representations, as well as combinations of blocks in the block diagrams and / or flowchart representations, can be implemented using dedicated hardware-based systems to perform the specified functions or actions, or using combinations of dedicated hardware and dedicated computer instructions.

[0061] Embodiments of the invention can be provided to end users via a cloud data processing infrastructure. The term "cloud data processing" ("cloud computing") generally refers to the provision of scalable data processing resources as a service over a network. More formally, cloud computing can be defined as a data processing function that provides an abstraction between the data processing resource and its underlying technical architecture (e.g., servers, storage, networks), enabling user-friendly, on-demand network access to a shared collection of configurable data processing resources that can be provisioned and released with minimal administrative overhead or interaction with the service provider.Thus, cloud computing enables a user to access virtual data processing resources (e.g., storage, data, applications, and even complete virtualized data processing systems) in the "cloud," regardless of the underlying physical systems (or the locations of the systems) used to provide the data processing resources.

[0062] Cloud computing resources are typically provided to a user on a pay-per-use basis, where users are only charged for the data processing resources they actually use (e.g., the amount of storage space consumed or the number of virtual systems instantiated). A user can access any of the resources located in the cloud anytime, from anywhere, via the internet. In the context of the present invention, a user can access applications (e.g., the neural network allocation tool or the power estimator) and associated data available in the cloud. For example, the neural network allocation tool or the power estimator could run on a data processing system in the cloud and select the optimal concurrency technique for each layer in a neural network.In such a case, the NN assigner or the power estimator could select the optimal techniques and store these selections in cloud storage. This allows a user to access this data from any data processing system connected to a cloud-connected network (e.g., the internet).

[0063] Although the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention can be developed without deviating from its basic scope of protection, the scope of which is defined by the following claims.

Claims

[1] Method (500) which exhibits: Selecting a hybrid parallelism technique (140) for distributing the operational load of a layer of a neural network across an array (200, 600) of processors (110, 155), wherein each processor in the array of processors can transfer data to neighboring processors in a first direction (610) and in a second direction (605); and Assigning tasks corresponding to the layer of the neural network to the array of processors using the selected hybrid parallelism technique (145), wherein the hybrid parallelism technique includes the use of a first parallelism technique when data is transferred between processors in the array of processors in the first direction (610), and the use of a second, different parallelism technique when data is transferred between processors in the array of processors in the second direction (605). [2] Method (500) according to claim 1, wherein the array (200, 600) of processors (110, 155) has a higher bandwidth when transferring data between the processors in the first direction (610) than when transferring data between the processors in the second direction (605). [3] Method (500) according to claim 2, further comprising: Estimating (505) a layer performance of the neural network using a plurality of different hybrid parallelism techniques (140); and Determine (520) based on the estimated performance whether the layer of the neural network is feature-heavy or weight-heavy. [4] Method (500) according to claim 3, further comprising: After finding that the layer of the neural network is rather feature-heavy, use (535, 540) of model parallelism (135) as the first parallelism technique and of data parallelism (130) as the second parallelism technique. [5] Method (500) according to claim 3, further comprising: After finding that the layer of the neural network is rather weight-heavy, use (525, 530) of data parallelism (130) as the first parallelism technique and of model parallelism (135) as the second parallelism technique. [6] Method (500) according to claim 1, wherein the array (200, 600) of processors (110, 155) forms a 2-D torus, wherein the first direction (610) and the second direction (605) are at right angles to each other. [7] Method (500) according to claim 1, wherein the tasks corresponding to the layer of the neural network include training data used to train the neural network containing the layer of the neural network. [8] Method (500) according to claim 7, further comprising: Estimating the performance of a plurality of layers in the neural network using a plurality of different hybrid parallelism techniques (140); and Assigning one of the plurality of different hybrid parallelism techniques (140) to each of the plurality of layers. [9] System (105) which features: an array (200, 600) of processors (110, 155); and an assigner (120) of a neural network that is configured to: Selecting a hybrid parallelism technique (140) for distributing the operational load of a layer of a neural network across an array of processors, wherein each processor in the array of processors can transfer data to neighboring processors in a first direction (610) and in a second direction (605); and Assigning tasks corresponding to the layer of the neural network to the array of processors using the selected hybrid parallelism technique (145), wherein the hybrid parallelism technique includes the use of a first parallelism technique when data is transferred between processors in the array of processors in the first direction (610), and the use of a second, different parallelism technique when data is transferred between processors in the array of processors in the second direction (605). [10] System (105) according to claim 9, wherein the array (200, 600) of processors (110, 155) has a higher bandwidth when transferring data between the processors in the first direction (610) than when transferring data between the processors in the second direction (605). [11] System (105) according to claim 10, wherein the assigner (120) of the neural network is configured to: Estimating (505) a layer performance of the neural network using a plurality of different hybrid parallelism techniques (140); and Determine (520) based on the estimated performance whether the layer of the neural network is feature-heavy or weight-heavy. [12] System (105) according to claim 11, wherein the assigner (120) of the neural network is configured to: After finding that the layer of the neural network is rather feature-heavy, use (535, 540) of model parallelism (135) as the first parallelism technique and of data parallelism (130) as the second parallelism technique. [13] System (105) according to claim 11, wherein the assigner (120) of the neural network is configured to: After finding that the layer of the neural network is rather weight-heavy, use (525, 530) of data parallelism (130) as the first parallelism technique and of model parallelism (135) as the second parallelism technique. [14] System (105) according to claim 9, wherein the array (200, 600) of processors (110, 155) forms a 2-D torus, wherein the first direction (610) and the second direction (605) are at right angles to each other. [15] A computer-readable storage medium embodying computer-readable program code, wherein the computer-readable program code is readable by one or more computer processors to perform an operation, wherein the operation comprises: Selecting a hybrid parallelism technique (140) for distributing the operational load of a layer of a neural network across an array (200, 600) of processors (110, 155), wherein each processor in the array of processors can transfer data to neighboring processors in a first direction (610) and in a second direction (605); and Assigning tasks corresponding to the layer of the neural network to the array of processors using the selected hybrid parallelism technique (145), wherein the hybrid parallelism technique includes the use of a first parallelism technique when data is transferred between processors in the array of processors in the first direction (610), and the use of a second, different parallelism technique when data is transferred between processors in the array of processors in the second direction (605). [16] Computer-readable storage medium according to claim 15, wherein the array (200, 600) of processors (110, 155) has a higher bandwidth when transferring data between the processors in the first direction (610) than when transferring data between the processors in the second direction (605). [17] Computer-readable storage medium according to claim 16, wherein the operation comprises: Estimating (505) a layer performance of the neural network using a plurality of different hybrid parallelism techniques (140); and Determine (520) based on the estimated performance whether the layer of the neural network is feature-heavy or weight-heavy. [18] Computer-readable storage medium according to claim 17, wherein the operation comprises: After finding that the layer of the neural network is rather feature-heavy, use (535, 540) of model parallelism (135) as the first parallelism technique and of data parallelism (130) as the second parallelism technique. [19] Computer-readable storage medium according to claim 17, wherein the operation comprises: After finding that the layer of the neural network is rather weight-heavy, use (525, 530) of data parallelism (130) as the first parallelism technique and of model parallelism (135) as the second parallelism technique. [20] Computer-readable storage medium according to claim 15, wherein the array (200, 600) of processors (110, 155) forms a 2-D torus, wherein the first direction (610) and the second direction (605) are at right angles to each other.

Images