Hybrid data-model parallelism for efficient deep learning

Abstract

Hybrid parallelism techniques are disclosed in which a mix of data and model parallelism techniques is used to split the workload of a layer across an array of processors. When the array is configured, the bandwidth of the processors in one direction can be greater than the bandwidth in the other direction. Each layer is characterized according to whether they are more feature-heavy or weight-heavy. Based on this characterization, the workload of the NN layers can be distributed to the array using hybrid parallelism techniques, rather than using only data parallelism techniques or only model parallelism techniques. For example, if the NN layers are more weight-heavy than feature-heavy, data parallelism is used in the direction with the larger bandwidth (to minimize the negative impact of weight reduction), while model parallelism is used in the direction with the smaller bandwidth.

Description

Background Art

[0001] The present invention relates to using hybrid parallelism techniques to distribute layers in a neural network to an array of processors.

[0002] Training a neural network (NN) is a time-consuming process. As a result, many training platforms use processor arrays (e.g., 2D torus) to divide the workload of each layer of the NN. There are several known techniques for dividing the workload between multiple processors. One such technique is data parallelism, in which the workload is split by the features (or inputs) of the layer. In this technique, each processor can perform all the tasks of a specific batch (or mini-batch) of training data. Taking an image processor NN as an example, each processor can be assigned to process its own image. To do this, the weights (or kernels) of the layer must be transferred to each of the processors. In NN layers where the weights are larger than the features (referred to herein as weighted layers), the use of data parallelism techniques is inefficient due to the reduction in weights between chips (wherein the weights are synchronized at the end of each batch or mini-batch of training). This is an expensive chip-to-chip operation and is generally a major pain point for data parallelism.

[0003] Another technique is model parallelism, in which the workload of a layer is split in the output dimension. That is, the first few features are calculated in one processor, the next few features are calculated in another processor, and so on. This avoids having to send all the weights of the layer to each processor and therefore avoids weight reduction. However, typically in NNs, each output feature requires an entire input channel. That is, in order to calculate several features, each processor still needs the entire input of the layer (for example, the output from the previous layer). With model parallelism, the input is replicated in all processors (called performing "activations"). Before the next layer in the NN can start, each processor needs to receive the output calculated by every other chip - that is, the activations of all processors are passed to each other. Thus, model parallelism is more efficient than data parallelism for weight-heavy layers, but less efficient for feature-heavy layers (where the feature (or input) data is larger than the weight data). Summary of the Invention

[0004] According to one embodiment of the present invention, a method is provided, comprising selecting a hybrid parallelism technique for splitting a workload of a neural network layer onto a processor array, wherein each processor in the processor array can transmit data to an adjacent processor in a first direction and a second direction. The method further comprises assigning tasks corresponding to the neural network layer to the processor array using the selected hybrid parallelism technique, wherein the hybrid parallelism technique comprises using a first parallelism technique when transmitting data between processors in the processor array in the first direction, and using a different second parallelism technique when transmitting data between processors in the processor array in the second direction.

[0005] Another embodiment of the present invention is a system comprising a processor array and a neural network distributor. The distributor is configured to select a hybrid parallelism technique for splitting a workload of a neural network layer onto the processor array, wherein each processor in the processor array can transfer data to an adjacent processor in a first direction and a second direction, and to distribute tasks corresponding to the neural network layer to the processor array using the selected hybrid parallelism technique, wherein the hybrid parallelism technique includes using a first parallelism technique when transferring data between processors in the processor array in the first direction, and using a different second parallelism technique when transferring data between processors in the processor array in the second direction.

[0006] Another embodiment of the present invention is a computer-readable storage medium having computer-readable program code embodied therein, the computer-readable program code being executable by one or more computer processors to perform operations, the operations comprising selecting a hybrid parallelism technique for splitting a workload of a neural network layer to a processor array, wherein each processor in the processor array can transfer data to an adjacent processor in a first direction and a second direction, and assigning tasks corresponding to the neural network layer to the processor array using the selected hybrid parallelism technique, wherein the hybrid parallelism technique comprises using a first parallelism technique when transferring data between processors in the processor array in the first direction and using a different second parallelism technique when transferring data between processors in the processor array in the second direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0008] Figure 1 A NN training system according to one embodiment described herein is shown.

[0009] Figure 2 NN training hardware according to one embodiment described herein is shown.

[0010] Figure 3 Showing various sizes in a NN according to one embodiment described herein.

[0011] Figure 4 is a flow chart for determining an optimal technique for splitting execution of a NN layer across a processor array, according to one embodiment described herein.

[0012] Figure 5 is a flow chart for selecting a hybrid parallelism technique according to one embodiment described herein.

[0013] Figure 6Illustrated is an array of processors forming an x-ring and a y-ring according to one embodiment described herein.

[0014] Figure 7 Shown is a matrix for selecting parallelism techniques for NN layers according to one embodiment described herein.

[0015] Figure 8 Shown is an ordering of parallelism techniques based on characteristics of NN layers according to one embodiment described herein. DETAILED DESCRIPTION

[0016] Rather than being limited to data parallelism or model parallelism, embodiments herein introduce hybrid parallelism techniques where a mix of data and model parallelism can be used to split the workload of a layer across an array of processors (e.g., an array of integrated circuits (ICs) or chips). In one embodiment, the array of processors includes an x-ring in a first (X) direction and a y-ring in a second (Y) direction. When the array is configured, the bandwidth of the processors in one direction can be greater than the bandwidth in the other direction. That is, the system designer can intentionally allocate more communication links for communication between processors in the X direction than in the Y direction. Each layer can then be characterized according to whether they are feature-heavy or weight-heavy. Based on this characterization, the workload of the NN layers can be distributed to the array using hybrid parallelism techniques, rather than using only data parallelism techniques or only model parallelism techniques. For example, the NN layers can be slightly more weight-heavy (rather than feature-heavy). Since model parallelism is more efficient when executing weight-heavy layers, a first hybrid parallelism technique can be used to split the workload of the NN layers, where data parallelism is used in the X-direction (where bandwidth is greater) to achieve gradient reduction along the fast X-direction link, and model parallelism is used in the Y-direction (where bandwidth is lower, but model parallelism is inherently more efficient for weight-heavy layers). Conversely, if the NN layers are feature-heavy (rather than weight-heavy), a second hybrid parallelism technique can be used to split the workload, where model parallelism is used in the X-direction to minimize the negative impact of feature activations, and data parallelism is used in the Y-direction because data parallelism is inherently more efficient for weight-heavy layers.

[0017] While hybrid parallelism techniques may be used in processor arrays where the bandwidth in one direction is different from the bandwidth in the orthogonal direction, hybrid parallelism techniques may also be used where the bandwidth in both directions of the array is the same.

[0018] Figure 1A NN training system 100 is shown according to one embodiment described herein. The training system 100 includes a computing system 105 that uses a selected technique 145 to split the workload of each layer in a NN assigned to NN training hardware 150. Although the embodiments herein describe splitting the workload of layers when training a NN, they are not limited thereto. In situations other than training a NN, it may be advantageous to use the techniques discussed below to divide the workload.

[0019] The computing system 105 includes a processor 110, which represents any number of processing elements having any number of processing cores, and a memory 115, which may include volatile memory elements, non-volatile memory elements, and combinations thereof. As shown, the memory 115 includes two software applications that execute in the computing system 105: a NN allocator 120 and a performance estimator 125. The NN allocator 120 determines the selected technique 145 for allocating the workload of layers in the NN to the NN training hardware 150. In one embodiment, the NN allocator 120 selects the technique 145 in response to information generated by the performance estimator 125 about each layer in the NN. That is, the NN allocator 120 can select different techniques 145 for various layers in the NN.

[0020] The performance estimator 125 models (or simulates) the NN layer to estimate the performance of the layer using various parallelism techniques. In this example, the performance estimator 125 evaluates the layer to determine its performance when using data parallelism technique 130, model parallelism technique 135, and one or more hybrid parallelism techniques 140. In one embodiment, the hybrid parallelism technique 140 is any technique that is a hybrid of two other parallelism techniques that are used in combination to split the workload of tasks between various hardware elements in the NN training hardware 150. That is, the hybrid parallelism technique 140 can be one or more techniques that are a combination of data parallelism technique 130 and model parallelism technique 135. However, the hybrid parallelism technique 140 is not limited to the combination of techniques 130, 135, but can include a combination of other types of parallelism techniques for splitting the workload of a layer between the training hardware 150.

[0021] The NN training hardware 150 includes a plurality of processors 155, each processor 155 including a plurality of cores 160. The cores 160 include a plurality of processing element (PE) arrays (e.g., systolic arrays) formed using a plurality of PEs 170. As discussed below, the selected technique 145 splits the workload of the layer between the processors 155. For example, if the data parallelism technique 130 is the selected technique 145, the workload is split into small batches, where each small batch is assigned to a different processor 155 (e.g., images used when training an image processing NN are sent to each processor 155). If the model parallelism technique 135 is the selected technique 145, then each processor 155 is assigned a subset of the features in the output data (where a subset is one or more). The processors 155 can then operate in parallel to complete their assigned tasks. For example, each processor 155 can process an image simultaneously, or each processor 155 can process its assigned subset of features simultaneously.

[0022] Figure 2 NN training hardware according to one embodiment described herein is shown. Figure 2 A 2D array 200 is shown that includes processors 155 that are interconnected horizontally (X direction) and vertically (Y direction) with adjacent processors 155. In this way, the processors can share processed data. This data sharing is described in more detail below when discussing hybrid parallelism techniques.

[0023] Each processor 155 includes multiple cores 160 communicatively coupled using an on-chip interconnect 202 that allows the cores 160 to communicate with external memory 205 (e.g., RAM) in the NN training hardware. In turn, each core 160 includes multiple PE arrays 165 coupled to scratchpad memory 210, which represents any number of caches (e.g., level 1 and level 2 caches) that form on-chip memory.

[0024] Each PE array 165 includes a plurality of PEs 170 interconnected to form an array or grid (e.g., a systolic array). The inputs and outputs of the PE array 165 include first-in, first-out (FIFO) buffers 215. Each PE 170 includes a multi-accumulate (MAC) unit 220 coupled to a register file 225.

[0025] The workload of the NN layer can be Figure 2155. However, the following techniques focus on splitting the workload of the NN layer among processors 155. Other techniques can then be used to further split the workload assigned to each processor 155 among cores 160, PE array 165, and PE 170. That is, while the following embodiments discuss selecting the best parallelism technique to distribute the workload among processors 155, additional parallelism techniques can be used to further split the workload among the hardware elements in processor 155.

[0026] Figure 3 shows various sizes in a NN according to one embodiment described herein. That is, Figure 3 The different dimensions that can be used to partition a convolutional layer in a NN are shown. The workload of a convolutional layer can be fed into three data structures: input (Inp), output (Out), and weights (kernels). Nin is the number of features (or faces) in the data received from the previous layer or layers in the NN. That is, the input has a depth of Nin. Each feature or face has a size represented by Ni and Nj (e.g., feature size Ni x Nj). When performing convolution, one of the kernels is used to process each pixel in the feature (e.g., dot product). The NN has Nout number of kernels, where each kernel has a height of Ki, a width of Kj, and a depth of Nin.

[0027] Convolution works by performing a dot product of one of the kernels with a pixel in one of the features in the input. This outputs a scalar value (e.g., a pixel) that is stored in the output. As a result, the depth of the output is Nout (e.g., the number of features in the output), which is the same as the number of kernels. Moving the kernel using the stride when performing a dot product with other pixels in the input feature or face produces the output feature (or face). Assuming a stride of 1, the output feature or face has the same width and height (Ni and Nj) as the input feature.

[0028] The convolution results are performed in four dimensions: Nin, Nij, Kij, and Nout, which is used to split the workload of the convolution layer. However, training a NN also includes mini-batches (mb), where the NN can be trained using different input training data (e.g., different training images) that all use the same weights (kernels) to generate multiple output results. Therefore, mb is the fifth dimension, which can be used together with the other four dimensions to express the workload of the convolution layer as: N{Inp, Out, Nij, Kij, mb}.

[0029] Different sizes can be used to represent other types of layers. For example, when performing matrix multiplication on a fully connected layer in a NN, the Nij and Kij values ​​are 1. In any case, the embodiments herein can be used to separate different types of layers that can have different types (and numbers) of sizes.

[0030] Figure 4 is a flow chart of a method 400 for determining an optimal technique for splitting execution of a NN layer across a processor array, according to one embodiment described herein. At block 405, the NN allocator selects a layer in the NN. In one embodiment, the method 400 is used to iterate through each layer in the NN and select the optimal parallelism technique.

[0031] At block 410, the NN allocator determines the best technique for splitting the execution of the layer on the 2D array of processors. That is, the NN allocator may determine Figure 1 Which of the techniques shown in (data parallelism technique 130, model parallelism technique 135, or hybrid parallelism technique 140) is the best technique to use for a particular layer.

[0032] In one embodiment, the 2D array of processors is a 2D torus in which the processors are interconnected. For example, a processor may be communicatively coupled to an adjacent processor in two different directions (e.g., x / y or north / south). In one embodiment, the bandwidth of processor-to-processor communication may be different depending on the direction. For example, the 2D array may be designed so that communication between processors in the x direction has a higher bandwidth than communication between processors in the y direction. Figure 5 Details for selecting the best technique to use when dividing the workload among processors in a 2D array are described in .

[0033] At block 415, the NN allocator determines whether additional layers remain in the NN. That is, the NN allocator determines whether it has selected the optimal parallelism technique for each layer in the NN. If not, the method 400 returns to block 405, but if so, the method proceeds to block 420.

[0034] At block 420, when training a NN, the NN distributor distributes tasks to the 2D array of processors according to the optimal technique. If data parallelism or model parallelism is the selected technique, the processors exchange data in both directions by transferring data according to these techniques. That is, if data parallelism is selected, the processors exchange weights / kernels in the X and Y directions (also known as the gradient directions) of the 2D array. If model parallelism is selected, the processors exchange input features along both directions.

[0035] However, if hybrid parallelism is selected, the processor exchanges data in one direction based on data parallelism and in the other direction based on model parallelism. Thus, the parallelism technique implemented in the direction of the 2D array with greater bandwidth has improved performance relative to the technique implemented in the other direction with less bandwidth. Details of this tradeoff are discussed below.

[0036] Figure 5 is a flow chart of a method 500 for selecting a hybrid parallelism technique according to one embodiment described herein. The method 500 describes several techniques for performing block 410 of the method 400 to select the best parallelism technique for a layer in a NN. For clarity, Figure 6 The method 500 is described in parallel.

[0037] At block 505, a performance estimator estimates the performance of the layer when using multiple parallelism techniques. For example, at each layer, the performance estimator may use a model or simulation to estimate how the layer will perform on the NN training hardware using each of the different parallelism techniques (e.g., data parallelism, model parallelism, or hybrid parallelism).

[0038] In one embodiment, the performance estimator can use heuristics to limit the number of techniques estimated, rather than modeling or simulating each parallelism technique for each layer. For example, the performance estimator can generate a weight-to-feature ratio that can be used to quickly determine which parallelism technique should be used. If a layer has significantly more weight data than feature data according to a predefined threshold, then the performance estimator can assume that model parallelism is the best parallelism technique without performing any further modeling or simulation. Conversely, if the weight-to-feature ratio indicates that the layer has significantly more feature data than weight data, then data parallelism is considered the best parallelism technique. Thus, the performance estimator does not need to model or test each parallelism technique for each layer.

[0039] As described above, a 2D processor array may include multiple dimensions, where at least one dimension has a greater bandwidth for processor-to-processor communication than another dimension. Figure 6 An array of processors forming an x-ring and a y-ring in a 2D array 600 is illustrated according to one embodiment described herein. X-ring 610 indicates the communication links that processors 155 use to communicate in the X direction, while y-ring 605 indicates the communication links that processors 155 use to communicate in the Y direction.

[0040] In one embodiment, the 2D array 600 may include limited resources for forming communication links for the x- and y-rings. The 2D array 600 has different bandwidths for the x- and y-rings, rather than allocating the same amount of resources to both rings. That is, the 2D array 600 is an asymmetric array (or asymmetric torus), in which the bandwidth used for processor-to-processor communication varies depending on the direction of data transmission. For example, each processor-to-processor link may include multiple physical connections. In order to assign more bandwidth to the x-ring 610 than to the y-ring 605, the 2D array may have more physical connections allocated to each processor-to-processor link in the x-direction than to the processor-to-processor links in the y-direction. Therefore, the processor 155 can transmit data faster in the x-direction (e.g., to the adjacent processors on the left and right) than in the y-direction (e.g., to the adjacent processors above and below). This utilizes the hybrid parallelism technique described herein, in which less efficient parallelism techniques can be used when transmitting data between processors 155 in the x-direction (to utilize greater bandwidth), while more efficient parallelism techniques can be used when transmitting data between processors 155 in the y-direction. However, for some NN layers this may not be the case, and it is optimal to use more efficient parallelism techniques in the direction with greater bandwidth.

[0041] although Figure 2 and 6 A 2D array of processors is shown, but embodiments are not limited thereto. If an N-dimensional array of processors (where N is greater than 3) is used to implement a NN, at least one of the dimensions (or directions) can be designed to have a greater bandwidth than at least one other dimension. As such, embodiments described herein can be used to select a hybrid parallelism model, where different dimensions use different parallelism techniques.

[0042] Returning to method 500, at block 510, the NN allocator uses the performance estimate generated at block 505 to determine whether data or model parallelism is the best parallelism technique for the layer. If so, method 500 proceeds to block 515, where the NN allocator assigns the data or model parallelism technique to the layer. That is, when splitting the task and configuring the NN training hardware, the same parallelism technique (whether data-model parallelism or model parallelism) is used to transfer data in both directions in the 2D array. In this way, the same parallelism model is used in both directions, even if one of the directions may have a greater bandwidth than the other.

[0043] Otherwise, method 500 proceeds to box 520, where the NN allocator uses the performance estimate to determine whether the layer currently being evaluated is feature-heavy (rather than weight-heavy). As described above, in NN layers where features are greater than weights (e.g., heavy layers), data parallelism techniques are more efficient than model parallelism techniques. That is, in some NNs, each output feature requires an entire input channel. That is, in order to compute several features, each processor still needs the entire input to the layer (e.g., the output from the previous layer). Using model parallelism, the input is replicated in all processors or the activations are performed on all processors. Thus, if the layer is a feature-heavy layer, method 500 proceeds to block 535, where the NN allocator selects to use model parallelism techniques in the direction with the largest bandwidth in the 2D array. In this way, the negative impact of the activations has less impact on the efficient operation of the NN because these activations are transmitted on processor-to-processor links with larger bandwidth. In box 540, the NN allocator selects to use data parallelism techniques in the direction with the smallest amount of bandwidth in the 2D array.

[0044] In contrast, if the layer is more of a weight-heavy layer, the method proceeds from box 520 to box 525, where the NN allocator selects to use data parallelism techniques in the direction with the largest bandwidth in the 2D array; and box 530, where the NN allocator selects to use model parallelism techniques in the direction with the smallest bandwidth. Since data parallelism is used on links with larger bandwidth, the negative impact of gradient reduction is reduced. In this way, method 500 indicates the use of two different hybrid parallelism techniques. One hybrid technique is blocks 535 and 540, where model parallelism is used in the higher bandwidth dimension of the array of processors and data parallelism is used in the lower bandwidth dimension, and the other is blocks 525 and 530, where data parallelism is used in the higher bandwidth dimension of the array of processors and model parallelism is used in the lower bandwidth dimension.

[0045] However, as mentioned above, it may be more efficient to use data parallelism in the direction with larger bandwidth for feature-heavy layers, or model parallelism in the direction with larger bandwidth for weight-heavy layers. For example, although weight-heavy, long short-term memory (LSTM) layers prefer hybrid parallelism, where model parallelism is used in the direction with larger bandwidth and data parallelism is used in the direction with smaller bandwidth. In this case, gradient reduction is quite infrequent (occurring only after all time steps are completed) and can occur on the slower direction link without affecting performance.

[0046] Although method 500 is discussed in the context of an asymmetric array with different bandwidths, method 500 can also be used in a symmetric array of processors with the same bandwidth in the X and Y directions. For example, the NN training hardware may include more processors than mini-batches to be tested. In this case, using only data parallelism is inefficient because the NN allocator can allocate batches (e.g., images) to each processor. In this case, using a hybrid parallelism model can improve efficiency. In another example, the NN allocator may not have enough features to distribute to all processors (e.g., Nout is less than the number of processors in the array). Thus, using only model parallelism may be inefficient, but using a hybrid parallelism model to divide the dimensions may result in more efficient use of the NN training hardware. In these instances, since the directions in the symmetric array have the same bandwidth, it does not matter which parallelism is used for which direction.

[0047] Figure 7 A matrix 700 for selecting a parallelism technique for a NN layer according to one embodiment described herein is shown. Matrix 700 shows four parallelism techniques and their relationship to each other. The x-axis represents the parallelism technique used in the x-direction of a 2D array, while the y-axis represents the parallelism technique used in the y-direction of the 2D array.

[0048] The "full data" parallelism technique is a technique that uses only data parallelism when splitting the workload. As such, data parallelism is used to perform parallelism in the X and Y directions of a 2D array.

[0049] The "DataXModelY" parallelism technique is a first hybrid parallelism technique in which data parallelism is used in the X direction of a 2D array, but model parallelism is used in the Y direction of the 2D array.

[0050] The "ModelXDataY" parallelism technique is a second hybrid parallelism technique in which model parallelism is used in the X direction of a 2D array, but data parallelism is used in the Y direction of the 2D array.

[0051] The "full model" parallelism technique is a technique that uses only model parallelism when splitting the workload. As such, model parallelism is used to perform parallelism in the X and Y directions of a 2D array.

[0052] Figure 81 shows a ranking of parallelism techniques based on the features of NN layers according to one embodiment described herein. Hybrid parallelism techniques represent a compromise between the advantages and disadvantages of data and model parallelism techniques. For NN layers that are primarily feature-heavy layers, using full data parallelism techniques in both directions of the 2D array may result in the most efficient use of NN training hardware. Conversely, for NN layers that are primarily weight-heavy layers, using full model parallelism techniques in both directions of the 2D array may result in the most efficient use of NN training hardware.

[0053] Using hybrid parallelism techniques (i.e., DataXModelY and ModelXDataY) can improve efficiency relative to full data or full model techniques for layers that are not primarily feature heavy or weight heavy. That is, for layers that fall between these two extremes, it may be more efficient to use hybrid parallelism techniques. That is, for layers that the performance estimator determines to be closer to feature heavy than weight heavy, ModelXDataY hybrid parallelism can result in improved efficiency (assuming that the X direction of the 2D array has more bandwidth than the Y direction). That is, model parallelism is performed in the X direction, while data parallelism is used in the Y direction. Thus, this enables NN training hardware to mitigate the negative impact of slower input activations of model parallelism because it is performed in the X direction where the bandwidth is largest, while using the inherently more efficient data parallelism in the Y direction where the bandwidth is smaller. For example, convolutional layers with small feature sizes (e.g., the last set of convolutional layers in ResNet and GoogLeNet), which are slightly feature heavy, prefer ModelXDataY parallelism to optimize activation / error rotations.

[0054] For NN layers that are closer to weight weight than feature weight, DataXModelY hybrid parallelism can lead to improved efficiency because data parallelism is performed in the direction with greater bandwidth (i.e., the X direction), which minimizes the negative impact of weight / gradient reduction, while model parallelism, which is inherently more efficient, is performed in the direction with lower bandwidth. For example, small fully connected layers with moderate weight weight (e.g., the last layer of VGG / ResNet) prefer DataXModelY because they implement gradient reduction along the fast X-direction links.

[0055] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, practical applications, or technical improvements over technologies found in the marketplace, or to enable those of ordinary skill in the art to understand the embodiments disclosed herein.

[0056] In the above, reference is made to the embodiments presented in this disclosure. However, the scope of the present disclosure is not limited to the specifically described embodiments. On the contrary, any combination of the following features and elements (whether or not relating to different embodiments) is contemplated to implement and practice the contemplated embodiments. In addition, although the embodiments disclosed herein may achieve advantages over other possible solutions or over the prior art, whether a particular advantage is achieved by a given embodiment does not limit the scope of the present disclosure. Therefore, the aspects, features, embodiments and advantages presented above are merely illustrative and are not considered to be elements or limitations of the appended claims unless expressly stated in the claims. Similarly, reference to the "present invention" should not be interpreted as a generalization of any inventive subject matter disclosed herein and should not be considered to be elements or limitations of the appended claims unless expressly stated in the claims.

[0057] Various aspects of the present invention may be implemented as a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or a combination of hardware and software embodiments, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0058] The present invention may be a system, method, and / or computer program product. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon for causing a processor to perform various aspects of the present invention.

[0059] Computer-readable storage media can be a tangible device that can retain and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card, or a protruding structure in a groove with instructions recorded thereon, and any suitable combination thereof. As used herein, computer-readable storage media should not be interpreted as a temporary signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse passing through an optical fiber cable), or an electrical signal emitted by a wire.

[0060] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or downloaded to an external computer or external storage device. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0061] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data or source code or object code written in any combination of one or more programming languages, these programming languages ​​include object-oriented programming languages ​​(such as Smalltalk, C++ etc.) and conventional procedural programming languages ​​(such as " C " programming languages ​​or similar programming languages). The computer-readable program instructions can be performed completely on the user's computer, partly on the user's computer, performed as an independent software package, partly on the user's computer, partly on a remote computer or fully on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer by any type of network (including local area network (LAN) or wide area network (WAN)), or can be connected to an external computer (for example, using an internet service provider through the internet). In certain embodiments, the electronic circuit comprising for example programmable logic circuit, field programmable gate array (FPGA) or programmable logic array (PLA) can be personalized to perform the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to perform the electronic circuit, so as to perform various aspects of the present invention.

[0062] The present invention will be described below with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0063] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in the flowchart and / or block diagram or multiple blocks. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to operate in a specific manner. Thus, the computer-readable storage medium having the instructions stored therein includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in the flowchart and / or block diagram or multiple blocks.

[0064] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in the flowchart and / or block diagram or multiple boxes.

[0065] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of the systems, methods and computer program products according to different embodiments of the present invention. To this end, each box in the flowchart or block diagram may represent a module, segment or portion of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions annotated in the box may not occur in the order annotated in the figure. For example, depending on the functions involved, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the opposite order. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0066] Embodiments of the present invention may be provided to end users via a cloud computing infrastructure. Cloud computing generally refers to the provision of scalable computing resources as a service over a network. More formally, cloud computing can be defined as the provision of computing capabilities that abstract between computing resources and their underlying technology infrastructure (e.g., servers, storage, networks), thereby enabling convenient, on-demand network access to a shared pool of configurable computing resources that can be quickly provisioned and released with minimal management effort or service provider interaction. Thus, cloud computing allows users to access virtual computing resources (e.g., storage, data, applications, and even complete virtualized computing systems) in a "cloud" without regard to the underlying physical systems (or the location of those systems) used to provide the computing resources.

[0067] Typically, cloud computing resources are provided to users on a pay-per-use basis, where users are charged only for the computing resources actually used (e.g., the amount of storage space consumed by the user or the number of virtualized systems instantiated by the user). Users can access any resource residing in the cloud at any time and from anywhere across the Internet. In the context of the present invention, users can access applications (e.g., NN allocators or performance estimators) or related data available in the cloud. For example, the NN allocator or performance estimator can execute on a computing system in the cloud and select the best parallelism technique for each layer in the NN. In this case, the NN allocator or performance estimator can select the best technique and store these selections at a storage location in the cloud. Doing so allows users to access this information from any computing system attached to a network connected to the cloud (e.g., the Internet).

[0068] While the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope of the invention is determined by the claims that follow.

Claims

1. A method for processing a neural network, comprising: selecting a hybrid parallelism technique for splitting the workload of a neural network layer onto a processor array, wherein each processor in the processor array can transmit data to an adjacent processor in a first direction and a second direction; as well as assigning tasks corresponding to the neural network layers to the processor array using a selected hybrid parallelism technique, wherein the hybrid parallelism technique includes using a first parallelism technique when transferring data between processors in the processor array in the first direction and using a second, different parallelism technique when transferring data between processors in the processor array in the second direction, The method further comprises: estimating the performance of the neural network layer using a plurality of different hybrid parallelism techniques; determining whether the neural network layer is more feature-heavy or weight-heavy based on the estimated performance; When determining that the neural network layer features are heavier, using model parallelism as the first parallelism technique and data parallelism as the second parallelism technique; and When determining which neural network layer weights are heavier, use data parallelism as the first parallelism technique and model parallelism as the second parallelism technique. The processor array has a greater bandwidth when transferring data between the processors in the first direction than when transferring data between the processors in the second direction. 2 . The method of claim 1 , wherein the processor array forms a 2D torus, wherein the first direction and the second direction are perpendicular.

3. The method of claim 1 , wherein the task corresponding to the neural network layer comprises training data for training a neural network comprising the neural network layer.

4. The method according to claim 3, further comprising: estimating the performance of a plurality of layers in the neural network using a plurality of different hybrid parallelism techniques; as well as One of a plurality of different hybrid parallelism techniques is assigned to each of the plurality of layers.

5. A system for processing a neural network, comprising: processor array; as well as The neural network dispatcher is configured to: selecting a hybrid parallelism technique for splitting the workload of a neural network layer onto a processor array, wherein each processor in the processor array can transmit data to an adjacent processor in a first direction and a second direction; as well as assigning tasks corresponding to the neural network layers to the processor array using a selected hybrid parallelism technique, wherein the hybrid parallelism technique includes using a first parallelism technique when transferring data between processors in the processor array in the first direction and using a second, different parallelism technique when transferring data between processors in the processor array in the second direction, The neural network allocator is further configured to: estimating the performance of the neural network layer using a plurality of different hybrid parallelism techniques; determining whether the neural network layer is more feature-heavy or weight-heavy based on the estimated performance; When determining that the neural network layer features are heavier, using model parallelism as the first parallelism technique and data parallelism as the second parallelism technique; and When determining which neural network layer weights are heavier, use data parallelism as the first parallelism technique and model parallelism as the second parallelism technique. The processor array has a greater bandwidth when transferring data between the processors in the first direction than when transferring data between the processors in the second direction.

6. The system of claim 5, wherein the processor array forms a 2D torus, wherein the first direction and the second direction are orthogonal.

7. A computer program product having computer readable program code embodied therewith, the computer readable program code being executable by one or more computer processors to perform operations comprising: selecting a hybrid parallelism technique for splitting the workload of a neural network layer onto a processor array, wherein each processor in the processor array can transmit data to an adjacent processor in a first direction and a second direction; as well as assigning tasks corresponding to the neural network layers to the processor array using a selected hybrid parallelism technique, wherein the hybrid parallelism technique includes using a first parallelism technique when transferring data between processors in the processor array in the first direction and using a second, different parallelism technique when transferring data between processors in the processor array in the second direction, The operations further include: estimating the performance of the neural network layer using a plurality of different hybrid parallelism techniques; determining whether the neural network layer is more feature-heavy or weight-heavy based on the estimated performance; When determining that the neural network layer features are heavier, using model parallelism as the first parallelism technique and data parallelism as the second parallelism technique; and When determining which neural network layer weights are heavier, use data parallelism as the first parallelism technique and model parallelism as the second parallelism technique. The processor array has a greater bandwidth when transferring data between the processors in the first direction than when transferring data between the processors in the second direction.

8. The computer program product of claim 7, wherein the processor array forms a 2D torus, wherein the first direction and the second direction are orthogonal.

Images