Deep learning workload fragmentation on heterogeneous devices

Abstract

The invention relates to DL workload fragmentation on heterogeneous devices and provides a method for heterogeneous fragmentation of DL workloads. The method may include converting an SD graph representing a DL workload into an MD graph based on device information on a plurality of heterogeneous devices, the MD graph including a plurality of sub-graphs representing a plurality of DL sub-workloads to be assigned to DL stages on the plurality of heterogeneous devices to complete the DL workload, respectively; and assigning the plurality of sub-graphs to respective DL stages on the plurality of heterogeneous devices.

Description

Technical Field

[0001] The embodiments described herein generally relate to deep learning (DL) networks, and more specifically to methods and apparatus for sharding DL workloads on heterogeneous devices. Background Art

[0002] The ever-increasing computing demands from DL workloads have prompted hardware (HW) vendors to build specialized HW acceleration units into their HW products and assemble heterogeneous HW chips (such as central processing unit (CPU), graphics processing unit (GPU), tensor processing unit (TPU), application specific integrated circuit (ASIC), field programmable gate array (FPGA), etc.) into a single computing node, for example, CPU+GPU, CPU+TPU, CPU+ASIC, CPU+FPGA, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Various advantages of the embodiments will become apparent to those skilled in the art by reading the following description and appended claims, and by referring to the following drawings, in which:

[0004] Figure 1 An architectural diagram for manually converting a single-device (SD) program to a multi-device (MD) program is illustrated, where the DL framework provides a traditional front-end application interface (API);

[0005] Figure 2 An architectural diagram illustrating automatic sharding of DL workloads on homogeneous devices via graph rewrite passes in the Tensorflow DL framework;

[0006] Figure 3 illustrates an architectural diagram for automatic DL workload sharding on heterogeneous devices via a graph rewriter called XPUAutoShard according to some embodiments of the present disclosure;

[0007] Figure 4 An example Tensorflow front-end program using XPU heterogeneous programming according to some embodiments of the present disclosure is illustrated;

[0008] Figure 5illustrates an example high-level graph rewrite flow for XPUAutoShard according to some embodiments of the present disclosure;

[0009] Figure 6 An example of automatically performing DL workload sharding on CPU and GPU in a pipeline parallel processing manner according to some embodiments of the present disclosure is illustrated;

[0010] Figure 7 An example workflow for completing heterogeneous sharding properties (HSP) annotation for a SD graph using an optimization adjustment loop according to some embodiments of the present disclosure is illustrated;

[0011] Figure 8 illustrates an example pseudo code for completing HSP annotation for a SD graph according to some embodiments of the present disclosure;

[0012] Fig. 9 illustrates an example algorithm for transforming an HSP-annotated SD graph into an MD graph according to some embodiments of the present disclosure;

[0013] Fig.10 illustrates an example SD single convolution training graph according to some embodiments of the present disclosure;

[0014] Fig.11 illustrates an example sliced ​​single convolution training graph to be applied to a CPU with one DL stage and a GPU with two DL stages according to some embodiments of the present disclosure;

[0015] Fig.12 illustrates an example flow chart of a process for heterogeneous sharding of DL workloads according to some embodiments of the present disclosure;

[0016] Fig.13 The block diagram illustrates components capable of reading instructions from a machine-readable or computer-readable medium and performing any one or more of the methods discussed herein, according to some example embodiments;

[0017] Fig.14 is a block diagram of an example processor platform according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] The various aspects of the illustrative embodiments will be described using terms that are commonly used by those skilled in the art to convey the essence of the present disclosure to other persons skilled in the art. However, it will be appreciated by those skilled in the art that many alternative embodiments may also be implemented using some parts of the described aspects. For illustration, specific numbers, materials, and configurations are recorded to provide a thorough understanding of the illustrative embodiments. However, it will be clear to those skilled in the art that alternative embodiments may also be implemented without these specific details. In other cases, known features may be omitted or simplified to avoid blurring the illustrative embodiments.

[0019] In addition, various operations will be described as multiple discrete operations in turn in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be interpreted as meaning that these operations are necessarily order-dependent. In particular, these operations do not need to be performed in the order presented.

[0020] The ever-increasing computing demands from DL workloads have prompted HW vendors to build specialized HW acceleration units into their HW products and assemble heterogeneous HW chips into a single computing node, for example, CPU+GPU, CPU+ASIC, CPU+FPGA, and so on.

[0021] While this trend may provide customers with more options for better performance, it may also bring usability challenges about how to more easily take advantage of these HW chips. Artificial Intelligence (AI) users often have to manually rewrite DL programs written for a single device to work on heterogeneous HW devices. This manual work involves slicing (i.e., partitioning) DL workloads, placing DL workloads on appropriate heterogeneous HW devices, scheduling computations and cross-device communications on individual HW devices, and writing dedicated computational primitives for running on heterogeneous HW devices. For example, some DL frameworks provide front-end APIs (e.g., Horovod, PyTorch DDP, PyTorch pipeline execution, GraphCore sharding) to allow users to manually convert DL programs written for a single device (i.e., SD programs) to MD programs so that they can work on multiple devices via data parallel processing or pipeline parallel processing.

[0022] Figure 1 The diagram shows the architecture of manually converting a single-device (SD) program to a multi-device (MD) program, where the DL framework provides a traditional front-end API. Figure 1As shown, for DL ​​training, AI practitioners always start with an SD program, and the SD program is usually converted to an SD graph representation before being optimized by the device backend and executed on the device backend. The DL framework may provide a front-end API to allow users to manually convert SD programs to MD programs. However, this manual processing may not be practical for distributing DL workloads on heterogeneous HW devices. Given that the system architecture and connectivity speeds of heterogeneous HW devices vary, achieving high efficiency from these HW devices via manual adjustments is beyond the capabilities of ordinary AI users.

[0023] Instead of the manual processing mentioned above, Tensorflow provides an automatic sharding solution via graph rewriting, allowing DL models to run on multiple TPUs of the same type. Figure 2 The following diagram shows the architecture of automatic sharding of DL workloads on homogeneous devices via graph rewriting in the Tensorflow DL framework. Figure 2 As shown, automatic sharding in Tensorflow is also designed for distributed training on homogeneous HW devices. Tensors in the SD graph are sharded evenly due to the fact of homogeneity, and the number of shards in the MD graph is no greater than the number of HW devices. Automatic sharding in Tensorflow can support data parallelism and model parallelism on arbitrary models, but it may only support pipeline parallelism on models with repetitive patterns, such as the Bidirectional Encoder Representation from Transformer (BERT) encoder. In other words, pipeline parallel processing with automatic sharding in Tensorflow requires strong assumptions on the architecture of DL models and thus does not work broadly.

[0024] In addition, OpenVINO recently introduced a feature that automatically performs inference on a set of heterogeneous HW devices. OpenVINO's solution is only for inference models and does not support "multi-stage" sharding. Sharding DL workloads for training models is more complicated than sharding DL workloads for inference models. Specifically, more cross-device communications between sharded workloads are involved during each training iteration, while the sharded inference workloads for each device are essentially independent; more types of operations are involved in the training model, increasing the complexity; due to the need for larger mini-batch sizes and backward calculations for training, the training model requires greater memory bandwidth / capacity, which complicates the sharding algorithm.

[0025] Additionally, gradient accumulation is a common practice to allow large batch models to run on devices with limited memory capacity. Users explicitly change the program code to call the forward and backward training process multiple times before the weights are updated. However, gradient accumulation requires users to manually decide how to split the batch size and the number of forward and backward training. On the other hand, to alleviate slow connectivity during data parallel training, people usually increase the batch size. However, increasing the batch size makes it more difficult for the trained model to fit into the device memory and / or cache, and also affects the convergence of models with batch normalization.

[0026] In view of these problems, this disclosure proposes a concept of automatically performing DL workload sharding on heterogeneous devices via a graph rewriter called XPUAutoShard. The concept of automatic DL workload sharding can be applied to both training-related DL workloads and reasoning-related DL workloads, and can achieve data parallel processing or pipeline parallel processing of any DL model without making any assumptions about the model architecture.

[0027] Figure 3 The diagram illustrates an architecture diagram for automatically sharding DL workloads on heterogeneous devices via XPUAutoShard according to some embodiments of the present disclosure. In these embodiments, a user can run an existing single-device DL program on a virtual device called an XPU, which represents all heterogeneous HW devices on a node to complete the DL workload corresponding to the DL program. The DL framework converts the DL program into a DL computational graph. XPUAutoShard can automatically shard the data (i.e., tensors) and operations in the graph into subgraphs, and place each subgraph (with sharded data and operations) on the device. The resulting sharded graph can be run on existing computational primitives written for a single device.

[0028] Figure 4 An example Tensorflow front-end program with XPU heterogeneous programming according to some embodiments of the present disclosure is illustrated. Figure 4 The program shown in can be a normal SD training program running on a virtual XPU device (see the code in the black box for device placement), the same way as writing an SD training program for a CPU or GPU device. Figure 3 As shown in Figure 4The SD program in can be first converted to an SD graph by the DL framework and then converted to an MD graph by XPUAutoShard. The MD graph can include multiple subgraphs, each of which represents a plurality of DL sub-workloads, which are assigned to the DL stages on the heterogeneous devices for completing the DL workloads. Note that each heterogeneous device can include one or more DL stages for running one or more DL sub-workloads represented by each subgraph.

[0029] Figure 5 An example high-level graph rewriting process of XPUAutoShard according to some embodiments of the present disclosure is illustrated. The input of XPUAutoShard may include an SD graph from a DL framework, device information (DeviceInfo) about multiple heterogeneous devices, and an optional automatic sharding configuration.

[0030] In the example, the SD graph may include some input nodes (labeled as "I" here) and some computation nodes (labeled as "C" or "L" or unlabeled here, where "C" represents a normal computation node and "L" represents a sum-reduce node in all dimensions). The shape of the input tensor can be static or dynamic in each dimension. In the example, the first dimension is static with a size of 768 bits, while the remaining dimensions are dynamic.

[0031] DeviceInfo can describe the device type, computing capability (e.g., number of operations per cycle for vector / matrix calculations, frequency and number of cores / execution units (EUs), etc.), cache capacity, cache latency, cache bandwidth, memory capacity, memory latency, memory bandwidth, etc. of each of multiple heterogeneous devices, and can also describe the interconnection latency and bandwidth between multiple heterogeneous devices. In the example, DeviceInfo describes information about two heterogeneous devices (i.e., CPU and GPU).

[0032] The automatic sharding configuration may include some configurations specified for the automatic sharding algorithm used in XPUAutoShard. For example, the automatic sharding configuration may include a parallel processing mode (e.g., data parallel processing mode or pipeline parallel processing mode) for sharding DL workloads between multiple heterogeneous devices, a minimum batch size per stage for multiple heterogeneous devices, and an indicator for indicating whether the batch size per stage for each of the multiple heterogeneous devices needs to be consistent.

[0033] Note that the automatic sharding configuration can be an optional input to XPUAutoShard. That is, when no automatic sharding configuration is input to XPUAutoShard, XPUAutoShard can convert the SD graph into an MD graph based on DeviceInfo and the default automatic sharding configuration predetermined for XPUAutoShard. On the other hand, setting the automatic sharding configuration can allow AI users to specify their preferred automatic sharding configuration options, such as the preferred parallel processing mode (data parallelism or pipeline parallelism), the preferred minimum batch size per stage, and whether the batch size per stage for each device should be consistent. Figure 5 In the example shown, the parallel processing method is a data parallel processing method, that is, the training data is sharded and placed on each device, each device runs the training program with the sharded training data in parallel, and then the training results from all devices are integrated to complete the training process.

[0034] The output of XPUAutoShard is a multi-device sharding graph. In the example, the SD graph is sharded into three parts along the batch dimension (i.e., the first dimension), with one DL stage on the CPU and two DL stages on the GPU, and the batch size (BS) of each stage is 256 bits. When automatic sharding of DL workloads is performed on heterogeneous HW devices present on a single computing node, how to ensure the convergence of the DL model for the sharded sub-workloads may be a key problem to be solved, especially for DL ​​models with batch normalization, where the statistical batch size must be consistent between heterogeneous HW devices to ensure convergence for known hyperparameters. In view of this problem, according to some embodiments of the present disclosure, each heterogeneous HW device may include one or more DL stages for running one or more DL sub-workloads represented by each subgraph. In other words, some subgraphs in the MD graph can be multi-stage graphs, which makes it possible to perform large batch size training under limited device memory and cache capacity, and also helps to keep the statistical batch size consistent to achieve batch normalization.

[0035] For example, Figure 5As shown, the CPU includes one DL stage, and the GPU includes two DL stages, and each DL stage starts from an ordinary computing node "C" and ends at a sum-reduce node "L". The operation node "S" (meaning "split") is inserted as a post-operation ("post-operation") of the input node "I" to split the input tensor into three slices. Note that a control dependency (shown by a dotted arrow) is added from the sum-reduce node "L" of the first DL stage of the GPU to the ordinary computing node "C" of the second DL stage of the GPU to ensure that the second DL stage runs after the first DL stage is completed, thereby ensuring that the intermediate tensors used by one stage can be released before the start of another stage. Since node "L" is a sum-reduce node, the operation nodes "A" representing the sum-reduce post-operation are inserted after the node "L", respectively. These operation nodes "A" can first apply device-local reduction to each device, and then perform "full reduction" collective communication between devices.

[0036] Next, we will refer to Figure 5 The internal configuration of XPUAutoShard is described in detail. Inside XPUAutoShard, the SD graph can be annotated with a heterogeneous sharding property (HSP) on each tensor to generate an HSP-annotated SD graph, and based on the HSP associated with each tensor in the SD graph, the HSP-annotated SD graph can be transformed into an MD graph. Specifically, Figure 5 As shown, the "HSP completion pass" can annotate all tensors in the SD graph with corresponding HSPs, and the "slice reduction pass" can mechanically rewrite the SD graph into an MD graph according to the HSPs. For each tensor, the HSP can describe how each dimension of the tensor is sliced ​​and placed on each device, and the HSP can also describe the post-operations associated with the tensor.

[0037] Figure 5 The example in shows data parallel sharding. An example HSP for splitting the dimensions of a tensor can be described as follows:

[0038] HSP(split:<dimension>,<post-operation>,<device>:<size or ratio>:<number of stages>,...)

[0039] It should be noted here that the HSP for splitting the dimensions of a tensor can be configured to indicate more than one split operation and the corresponding split dimensions. In other words, there can be multiple fields of "Split: <dimension>" in the HSP. Therefore, one or more split operations along each split dimension can be performed on each tensor.

[0040] An example HSP for replicating a tensor across multiple devices can be described as follows:

[0041] HSP(copy,<post-operation>,<device>,...)

[0042] For example, "HSP(Split:0, Split, CPU:256:1, GPU:512:2)" on the output tensor of input node "I" means splitting the output tensor in the first dimension (e.g., dimension 0) and placing the split tensor on the CPU with a parallel data size of 256 bits and a single stage, and on the GPU with a parallel data size of 512 bits and two stages of 256 bits each. If the tensor shape is dynamic, "Split" can also be specified as a ratio, for example, 1 / 3 on the CPU and 2 / 3 on the GPU. In addition, an operation node "S" representing the split post-operation should be inserted after the input node "I" so that the "Split" post-operation is applied on the output tensor of input node "I". "HSP(Copy, Reduce_Sum, CPU, GPU)" on the output tensor of "L" means copying the tensor on the CPU and GPU, with a "Reduce_Sum" post-operation.

[0043] In general, for each tensor in the SD graph, the corresponding HSP can be configured to indicate the sharding-related operations on the tensor, the post-operations associated with the tensor, and one or more DL devices on which to place the tensor.

[0044] The sharding-related operations on the tensor may include one or more split operations. In this case, the HSP may be configured to further indicate the splitting dimension specified for each splitting operation and the parameters associated with each of the one or more DL devices, and the splitting operation may be configured to split the tensor along the splitting dimension to place the tensor on one or more DL devices based on the parameters associated with the DL device. The parameters associated with each of the one or more DL devices may include the parallel data size of the DL device and the number of DL stages for running each DL sub-workload associated with the DL device. Alternatively, when the tensor has a dynamic tensor shape, the parameters associated with each of the one or more DL devices may include the ratio of the DL sub-workload associated with the DL device to the DL workload, and the number of DL stages for running each DL sub-workload associated with the DL device. In addition to the split operation, the sharding-related operation on the tensor may also be a copy operation, which is configured to copy the tensor and place the copied tensor on one or more DL devices.

[0045] As described above, AI users can specify their preferred automatic sharding configuration, for example, the preferred parallel processing mode (data parallelism or pipeline parallelism). Figure 6 An example of automatically performing DL workload sharding on CPU and GPU in a pipeline parallel processing manner according to some embodiments of the present disclosure is illustrated. Figure 5 The same mechanism described can be applied to Figure 6 In pipeline parallel sharding. With respect to pipeline parallel sharding, the pipeline of a DL workload may be sharded into multiple layers, and the sharded DL sub-workload corresponding to each layer may be assigned to a corresponding DL device. In addition, each device may include one or more DL stages for running one or more DL sub-workloads. Figure 6 In the example of , the input tensor is sharded into three partitions to be processed by three stages of the CPU, and the pipeline of the DL workload is sharded into three layers, where the sharded DL sub-workloads corresponding to the first and third layers are assigned to the CPU, and the sharded DL sub-workload corresponding to the second layer is assigned to the GPU. Note that an operation node "X" is inserted in the MD graph, indicating a post-operation of CPU-to-GPU or GPU-to-CPU data movement.

[0046] like Figure 5 As shown, XPUAutoShard includes an HSP completion pass for generating an HSP-annotated SD graph and a shard reduction pass for transforming the HSP-annotated SD graph into an MD graph. For details on the HSP completion pass and the shard reduction pass, please refer to Figures 7 to 9 To further describe.

[0047] Figure 7 FIG. 1 illustrates an example workflow for completing HSP annotation for an SD graph using an optimization and adjustment loop according to some embodiments of the present disclosure. In an embodiment, the HSP completion pass may follow a general optimization and adjustment loop to annotate all tensors in the SD graph with HSP. Figure 7 The flow of the regulation loop shown is driven by an "HSP regulator". During each regulation iteration, a shard state may be generated as a configuration for creating an "HSP annotator", which is responsible for creating an HSP-annotated graph for evaluation by a "cost model" to score the shard graph corresponding to the shard state. Here, the shard state may include HSPs for tensors in the SD graph. The "cost model" can be used to evaluate the computational cost of each operation in the shard graph given a DeviceInfo, and can be implemented by an analytical model, runtime profiling, a learned model, or a combination. Based on the score of the shard graph, the shard state can be adjusted, the HSP-annotated SD graph can be updated, and the best HSP-annotated SD graph can be recorded as the output of the HSP completion pass.

[0048] More generally, the HSP completion pass may be performed by annotating each tensor in the SD graph with the corresponding HSP in the following manner: adjusting the HSP for each tensor based on a device cost model for evaluating the computational cost of each operation assigned to multiple heterogeneous devices, and obtaining the HSP-annotated SD graph with the best computational cost score as the HSP-annotated SD graph.

[0049] Figure 8 FIG. 4 illustrates an example pseudo code of HSP completing a pass according to some embodiments of the present disclosure. Figure 8 As shown, the key model in the HSP completion pass is the HSP regulator. The implementation options of the HSP regulator will be described in detail below.

[0050] A straightforward implementation of the HSP conditioner and HSP annotator can be based on some heuristics. For data parallel processing, the HSP conditioner can use the semantics of the operations to decide which dimension of the input tensor is a “split” candidate and which input is a “copy” candidate. For example, the first dimension of the data input to the convolution is the batch dimension of the “split” candidate, while the weight input to the convolution is the “copy” candidate. All other HSPs in the graph can be decided by propagating the HSPs throughout the graph according to the semantics of the operations, which is similar to the algorithm in Tensorflow automatic sharding. For example, the split ratio of a HW device can be determined based on the normalized compute power of the HW device calculated by the following formula (w1-w5 are predefined parameters):

[0051] TOPS w1 ×Cache bandwidth w2 ×Memory bandwidth w3 × Cache Delay w4 ×Memory Delay w5

[0052] For example, a grid search can be applied on the batch size at each stage by scanning the power-of-two values ​​for the best score, and the maximum allowed batch size can be calculated using the memory footprint of the graph relative to the device memory capacity.

[0053] On the other hand, for pipeline parallel processing, the HSP regulator can scan the number of stages starting from the number of devices, up to a configured upper limit. The number of stages can determine the number of splits applied to the batch dimension. In the HSP annotator, for a given number of stages, operations can be sorted using a breadth-first traversal, and devices can be greedily assigned to operations one by one, i.e., assigning a device to the next operation "A" with the lowest total cost from the first operation to operation "A" evaluated.

[0054] A more advanced implementation of the HSP conditioner can rely on a Graph Neural Network (GNN) as an implementation of the HSP annotator to predict HSPs. When the parameter values ​​in HSP are discretized, the problem can become a node classification problem for the GNN, i.e., predicting the HSP of the output tensor of the operation node. Then, by sampling HSP solutions from the GNN, the HSP conditioner can identify the best score by evaluating the candidate annotations with a cost model.

[0055] In addition to the HSP completion pass, XPUAutoShard also includes a sharding reduction pass for transforming an HSP-annotated SD graph into an MD graph. Specifically, transforming an HSP-annotated SD graph into an MD graph may include splitting the HSP-annotated SD graph into multiple subgraphs based on the sharding-related operations specified in the HSP on each tensor in the SD graph, and inserting operation nodes corresponding to post-operations associated with each tensor in the SD graph. In addition, when the DL device where the tensor is to be placed includes two or more DL stages, transforming the HSP-annotated SD graph into an MD graph may also include adding a control dependency from an end operation node of one DL stage of the DL device to a start operation node of the next DL stage of the DL device.

[0056] Fig. 9 An example algorithm for transforming an HSP-annotated SD graph into an MD graph according to some embodiments of the present disclosure is illustrated. Fig. 9 The example algorithm shown in mechanistically transforms an HSP-annotated SD graph into an MD graph. In the algorithm, the function "addControlEdgeAmongStages" can add a control dependency from the end operation node of a "split" region in one stage of a DL device to the start operation node of a "split" region in the next stage of the same device. For the sake of brevity, the details of the functions with simple implementation are omitted.

[0057] The concept of automatically performing DL workload sharding on heterogeneous devices via XPUAutoShard according to the embodiments of the present disclosure has been described above with reference to Figures 3 to 9Described. Based on the detailed description, it can be easily understood that the automatic DL workload sharding via XPUAutoShard can be applied to both training-related DL workloads and reasoning-related DL workloads, and can achieve data parallel processing or pipeline parallel processing of any DL model without making any assumptions about the model architecture. XPUAutoShard can automatically decide which dimensions of the tensor to shard and on what device to place the sharded tensor based on some heuristics and device cost models. Based on the proposed automatic DL workload sharding, each device can work on more than one data shard and process a subgraph of the shard with multiple DL stages. In addition, in order to balance work between heterogeneous HW devices, the size of the shards and / or the number of stages may be uneven, and the size of each stage of each shard can be set according to the device cost model to adapt to the device memory and / or maximize the cache residency of the heterogeneous HW device, and to ensure the convergence of the model with batch normalization.

[0058] To further illustrate the proposed solution for automatic sharding of DL workloads on heterogeneous devices, Fig.10 and Fig.11 A real-world example of sharding a single convolutional training workload on both the CPU and GPU is shown in the following. Fig.10 illustrates an example SD single convolution training graph according to some embodiments of the present disclosure, and Fig.11 FIG. 4 illustrates an example sliced ​​single convolution training graph to be applied to a CPU with one DL stage and a GPU with two DL stages according to some embodiments of the present disclosure. Fig.10 and Fig.11 As shown in Figure 1, the SD single convolution training graph has been transformed into a sliced ​​single convolution training graph, which is processed by one DL stage on the CPU and two DL stages on the GPU based on data parallel processing. Fig.11 As shown, the training data (including input data and labels) is sharded into three parts and placed on three DL stages on the CPU and GPU, each DL stage runs single convolution training with the sharded training data in parallel, and then the training results from all DL stages are integrated to complete the training process.

[0059] In order to better understand the overall solution of heterogeneous sharding of DL workloads proposed in this disclosure, reference will be made to Fig.12 The flowchart shown in further describes the proposed heterogeneous DL workload sharding scheme.

[0060] Fig.12An example flow chart of a process for heterogeneous sharding of DL workloads according to some embodiments of the present disclosure is illustrated. The process for heterogeneous sharding of DL workloads may be implemented by a processor circuit and may include operations 1210 to 1220.

[0061] In operation 1210, the processor circuit may convert an SD graph representing a DL workload into a multi-device MD graph based on device information about multiple heterogeneous devices, the MD graph including multiple sub-graphs, which respectively represent multiple DL sub-workloads to be assigned to DL stages on multiple heterogeneous devices to complete the DL workload.

[0062] According to some embodiments, each of the plurality of heterogeneous devices may include one or more DL stages for running one or more DL sub-workloads represented by respective sub-graphs.

[0063] At operation 1220 , the processor circuit may assign the plurality of subgraphs to respective DL stages on the plurality of heterogeneous devices.

[0064] According to some embodiments, the device information may include at least one of the following: device type, computing power, cache capacity, cache latency, cache bandwidth, memory capacity, memory latency, memory bandwidth of each of a plurality of heterogeneous devices, and interconnection latency and bandwidth between a plurality of heterogeneous devices.

[0065] According to some embodiments, the SD graph may be converted to an MD graph based on device information and automatic slicing configuration.

[0066] According to some embodiments, the automatic sharding configuration may include a parallel processing approach for sharding the DL workload among multiple heterogeneous devices. The parallel processing approach may include a data parallel processing approach or a pipeline parallel processing approach.

[0067] According to some embodiments, the auto-sharding configuration may include a minimum per-stage batch size for multiple heterogeneous devices.

[0068] According to some embodiments, the automatic sharding configuration may include an indicator for indicating whether the batch size per stage of each of the plurality of heterogeneous devices needs to be consistent.

[0069] According to some embodiments, the DL workload may include a training-related DL workload or an inference-related DL workload.

[0070] According to some embodiments, converting an SD graph into an MD graph may include: annotating each tensor in the SD graph with a corresponding heterogeneous slice property (HSP) to generate an HSP-annotated SD graph; and transforming the HSP-annotated SD graph into an MD graph based on the HSP associated with each tensor in the SD graph.

[0071] According to some embodiments, for each tensor, the corresponding HSP may be configured to indicate sharding-related operations on the tensor, post-operations associated with the tensor, and at least one of one or more DL devices on which the tensor is to be placed, wherein the DL device is selected from a plurality of heterogeneous devices.

[0072] According to some embodiments, sharding-related operations on a tensor may include one or more split operations, and the HSP may be configured to further indicate a split dimension specified for each split operation and parameters associated with each of the one or more DL devices, and the split operation may be configured to split the tensor along the split dimension to place the tensor on the one or more DL devices based on the parameters associated with the DL devices.

[0073] According to some embodiments, the parameters associated with each of the one or more DL devices may include a parallel data size of the DL device and a number of DL stages for running respective DL sub-workloads associated with the DL device.

[0074] According to some embodiments, under the condition that the tensor has a dynamic tensor shape, the parameters associated with each of the one or more DL devices may include a ratio of a DL sub-workload associated with the DL device to the DL workload, and the number of DL stages used to run each DL sub-workload associated with the DL device.

[0075] According to some embodiments, a shard-related operation on a tensor may include a copy operation configured to copy the tensor and place the copied tensor on one or more DL devices.

[0076] According to some embodiments, transforming the HSP-annotated SD graph into an MD graph may include splitting the HSP-annotated SD graph into multiple subgraphs based on slice-related operations on each tensor in the SD graph, and inserting operation nodes corresponding to post-operations associated with each tensor in the SD graph.

[0077] According to some embodiments, under the condition that the DL device on which the tensor is to be placed includes two or more DL stages for running corresponding two or more DL sub-workloads, transforming the HSP-annotated SD graph into an MD graph may also include adding a control dependency from an end operation node of one DL stage of the DL device to a start operation node of a next DL stage of the DL device.

[0078] According to some embodiments, annotating each tensor in the SD graph with a corresponding HSP may include adjusting the HSP for each tensor based on a device cost model for evaluating the computational cost of each operation assigned to multiple heterogeneous devices, and obtaining an HSP-annotated SD graph with an optimal computational cost score as the HSP-annotated SD graph.

[0079] Fig.13 The block diagram of illustrates components capable of reading instructions from a machine-readable or computer-readable medium (e.g., a non-transitory machine-readable storage medium) and executing any one or more of the methods discussed herein according to some example embodiments. Specifically, Fig.13 A diagrammatic representation of hardware resources 1300 is shown, including one or more processors (or processor cores) 1310, one or more memory / storage devices 1320, and one or more communication resources 1330, each of which may be communicatively coupled via a bus 1340. For embodiments utilizing node virtualization (e.g., NFV), a hypervisor 1302 may be executed to provide an execution environment for one or more network slices / sub-slices utilizing the hardware resources 1300.

[0080] Processor 1310 may include, for example, processor 1312 and processor 1314, and processor 512 and processor 514 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a visual processing unit (VPU), a field programmable gate array (FPGA), or any suitable combination of these.

[0081] The memory / storage device 1320 may include main memory, disk storage, or any suitable combination of these. The memory / storage device 1320 may include, but is not limited to, any type of volatile or non-volatile memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state storage, etc.

[0082] The communication resources 1330 may include an interconnect or network interface component or other appropriate device to communicate with one or more peripheral devices 1304 or one or more databases 1306 via the network 1308. For example, the communication resources 1330 may include a wired communication component (e.g., for coupling via a Universal Serial Bus (USB)), a cellular communication component, an NFC component, Components (e.g., low energy ), components, and other communication components.

[0083] The instructions 1350 may include software, programs, applications, applet programs, apps, or other executable code for causing at least any one of the processors 1310 to perform any one or more of the methods discussed herein. The instructions 1350 may reside in whole or in part within at least one of the processors 1310 (e.g., within a cache memory of the processor), within the memory / storage device 1320, or any suitable combination of these. In addition, any portion of the instructions 1350 may be transmitted to the hardware resources 1300 from any combination of the peripheral device 1304 or the database 1306. Therefore, the memory of the processor 1310, the memory / storage device 1320, the peripheral device 1304, and the database 1306 are examples of computer-readable and machine-readable media.

[0084] Fig.14 1400 is a block diagram of an example processor platform according to some embodiments of the present disclosure. The processor platform 1400 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smart phone, an iPad, etc.), or a processor platform 1401. TM), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a game console, a personal video recorder, a set-top box, headphones or other wearable device, or any other type of computing device.

[0085] The processor platform 1400 of the illustrated example includes a processor 1412. The processor 1412 of the illustrated example is hardware. For example, the processor 1412 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor can be a semiconductor-based (e.g., silicon-based) device. In some embodiments, the processor implements one or more of the methods or processes described above.

[0086] The processor 1412 of the illustrated example includes a local memory 1413 (e.g., cache). The processor 1412 of the illustrated example communicates with a main memory including a volatile memory 1414 and a non-volatile memory 1416 via a bus 1418. The volatile memory 1414 may be a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory (DRAM) Dynamic Random Access Memory, ) and / or any other type of random access memory device. The non-volatile memory 1416 can be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1414, 1416 is controlled by a memory controller.

[0087] The processor platform 1400 of the illustrated example further includes an interface circuit 1420. The interface circuit 1420 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), interface, a near field communication (NFC) interface, and / or a PCI Express interface.

[0088] In the illustrated example, one or more input devices 1422 are connected to the interface circuit 1420. The input device(s) 1422 allow a user to enter data and / or commands into the processor 1412. The input device(s) may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a trackpad, a trackball, and / or a voice recognition system.

[0089] One or more output devices 1424 are also connected to the interface circuit 1420 of the illustrated example. The output device 1424 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. The interface circuit 1420 of the illustrated example thus typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0090] The interface circuitry 1420 of the illustrated example also includes communication devices, such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces, to facilitate data exchange with external machines (e.g., any kind of computing device) via the network 1426. Communications may be via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-sight wireless system, a cellular telephone system, and the like.

[0091] For example, interface circuit 1420 may include a training data set input through input device(s) 1422 or retrieved from network 1426 .

[0092] The processor platform 1400 of the illustrated example also includes one or more mass storage devices 1428 for storing software and / or data. Examples of such mass storage devices 1428 include floppy disk drives, hard disk drives, compact disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0093] The machine-executable instructions 1432 may be stored in the mass storage device 1428, in the volatile memory 1414, in the non-volatile memory 1416, and / or on a removable, non-transitory computer-readable storage medium such as a CD or DVD.

[0094] Additional notes and examples:

[0095] Example 1 includes an apparatus for heterogeneous sharding of a deep learning (DL) workload, the apparatus comprising: an interface circuit; and a processor circuit coupled to the interface circuit and configured to: based on device information about a plurality of heterogeneous devices received via the interface circuit, convert a single device (SD) graph representing the DL workload into a multi-device (MD) graph, the MD graph comprising a plurality of sub-graphs respectively representing a plurality of DL sub-workloads, the plurality of DL sub-workloads being assigned to DL stages on the plurality of heterogeneous devices to complete the DL workload; and assign the plurality of sub-graphs to respective DL stages on the plurality of heterogeneous devices.

[0096] Example 2 includes an apparatus as described in Example 1, wherein the device information includes at least one of the following: device type, computing power, cache capacity, cache latency, cache bandwidth, memory capacity, memory latency, memory bandwidth of each of a plurality of heterogeneous devices, and interconnection latency and bandwidth between the plurality of heterogeneous devices.

[0097] Example 3 includes the apparatus of Example 1 or 2, wherein the processor circuit is configured to convert the SD graph into the MD graph based on the device information and the automatic slicing configuration received via the interface circuit.

[0098] Example 4 includes an apparatus as described in Example 3, wherein the automatic sharding configuration includes a parallel processing method for sharding the DL workload among the multiple heterogeneous devices.

[0099] Example 5 includes the apparatus of Example 4, wherein the parallel processing method includes a data parallel processing method or a pipeline parallel processing method.

[0100] Example 6 includes an apparatus as described in any one of Examples 3 to 5, wherein the automatic sharding configuration includes a minimum per-stage batch size for the plurality of heterogeneous devices.

[0101] Example 7 includes an apparatus as described in any one of Examples 3 to 6, wherein the automatic sharding configuration includes an indicator for indicating whether the batch size of each stage of each of the plurality of heterogeneous devices needs to be consistent.

[0102] Example 8 includes an apparatus as described in any one of Examples 1 to 7, wherein the DL workload includes a training-related DL workload or an inference-related DL workload.

[0103] Example 9 includes an apparatus as described in any one of Examples 1 to 8, wherein the processor circuit is configured to convert the SD graph into the MD graph by: annotating each tensor in the SD graph with a corresponding heterogeneous sharding property (HSP) to generate an HSP-annotated SD graph; and based on the HSP associated with each tensor in the SD graph, transforming the HSP-annotated SD graph into the MD graph.

[0104] Example 10 includes an apparatus as described in Example 9, wherein, for each tensor, the corresponding HSP is configured to indicate at least one of the following: sharding-related operations on the tensor, post-operations associated with the tensor, and one or more DL devices on which to place the tensor, wherein the DL devices are selected from the plurality of heterogeneous devices.

[0105] Example 11 includes an apparatus as described in Example 10, wherein the sharding-related operations on the tensor include one or more split operations, the HSP is configured to further indicate a split dimension specified for each split operation and parameters associated with each of the one or more DL devices, and the split operation is configured to split the tensor along the split dimension to place the tensor on the one or more DL devices based on the parameters associated with the DL devices.

[0106] Example 12 includes the apparatus of Example 11, wherein the parameters associated with each of the one or more DL devices include a parallel data size of the DL device and a number of DL stages for running respective DL sub-workloads associated with the DL device.

[0107] Example 13 includes an apparatus as described in Example 11, wherein, under the condition that the tensor has a dynamic tensor shape, the parameters associated with each of the one or more DL devices include a ratio of a DL sub-workload associated with the DL device to the DL workload, and a number of DL stages for running each DL sub-workload associated with the DL device.

[0108] Example 14 includes the apparatus of Example 10, wherein the shard-related operation on the tensor comprises a copy operation configured to copy the tensor and place the copied tensor on the one or more DL devices.

[0109] Example 15 includes an apparatus as described in any one of Examples 10 to 14, wherein transforming the HSP-annotated SD graph into the MD graph includes partitioning the HSP-annotated SD graph into the multiple subgraphs based on slice-related operations on each tensor in the SD graph, and inserting operation nodes corresponding to post-operations associated with each tensor in the SD graph.

[0110] Example 16 includes an apparatus as described in Example 15, wherein, under the condition that the DL device on which the tensor is to be placed includes two or more DL stages for running corresponding two or more DL sub-workloads, transforming the HSP-annotated SD graph into the MD graph also includes adding a control dependency from an end operation node of a DL stage of the DL device to a start operation node of a next DL stage of the DL device.

[0111] Example 17 includes an apparatus as described in any one of Examples 9 to 16, wherein annotating each tensor in the SD graph with a corresponding HSP includes adjusting the HSP for each tensor based on a device cost model for evaluating the computational cost of each operation assigned to the multiple heterogeneous devices, and obtaining an HSP-annotated SD graph with an optimal computational cost score as the HSP-annotated SD graph.

[0112] Example 18 includes a method for heterogeneous sharding of a deep learning (DL) workload, the method comprising: based on device information about multiple heterogeneous devices, converting a single device (SD) graph representing the DL workload into a multi-device (MD) graph, the MD graph comprising multiple sub-graphs respectively representing multiple DL sub-workloads, the multiple DL sub-workloads being assigned to DL stages on the multiple heterogeneous devices to complete the DL workload; and assigning the multiple sub-graphs to the respective DL stages on the multiple heterogeneous devices.

[0113] Example 19 includes a method as described in Example 18, wherein the device information includes at least one of the following: device type, computing power, cache capacity, cache latency, cache bandwidth, memory capacity, memory latency, memory bandwidth of each of a plurality of heterogeneous devices, and interconnection latency and bandwidth between the plurality of heterogeneous devices.

[0114] Example 20 includes the method as described in Example 18 or 19, wherein converting the SD graph into the MD graph is based on the device information and the automatic sharding configuration.

[0115] Example 21 includes a method as described in Example 20, wherein the automatic sharding configuration includes a parallel processing method for sharding the DL workload among the multiple heterogeneous devices.

[0116] Example 22 includes the method of Example 21, wherein the parallel processing method includes a data parallel processing method or a pipeline parallel processing method.

[0117] Example 23 includes the method of any one of Examples 20 to 22, wherein the automatic sharding configuration includes a minimum per-stage batch size for the plurality of heterogeneous devices.

[0118] Example 24 includes a method as described in any one of Examples 20 to 23, wherein the automatic sharding configuration includes an indicator for indicating whether the batch size of each stage of each of the plurality of heterogeneous devices needs to be consistent.

[0119] Example 25 includes a method as described in any one of Examples 18 to 24, wherein the DL workload includes a training-related DL workload or an inference-related DL workload.

[0120] Example 26 includes a method as described in any one of Examples 18 to 25, wherein converting the SD graph into the MD graph includes: annotating each tensor in the SD graph with a corresponding heterogeneous slice property (HSP) to generate an HSP-annotated SD graph; and based on the HSP associated with each tensor in the SD graph, transforming the HSP-annotated SD graph into the MD graph.

[0121] Example 27 includes a method as described in Example 26, wherein, for each tensor, the corresponding HSP is configured to indicate at least one of the following: sharding-related operations on the tensor, post-operations associated with the tensor, and one or more DL devices on which to place the tensor, wherein the DL devices are selected from the plurality of heterogeneous devices.

[0122] Example 28 includes a method as described in Example 27, wherein the sharding-related operations on the tensor include one or more split operations, the HSP is configured to further indicate a split dimension specified for each split operation and a parameter associated with each of the one or more DL devices, and the split operation is configured to split the tensor along the split dimension to place the tensor on the one or more DL devices based on the parameters associated with the DL devices.

[0123] Example 29 includes the method of Example 28, wherein the parameters associated with each of the one or more DL devices include a parallel data size of the DL device and a number of DL stages for running respective DL sub-workloads associated with the DL device.

[0124] Example 30 includes a method as described in Example 28, wherein, under the condition that the tensor has a dynamic tensor shape, the parameters associated with each DL device of the one or more DL devices include a ratio of a DL sub-workload associated with the DL device to the DL workload, and a number of DL stages for running each DL sub-workload associated with the DL device.

[0125] Example 31 includes the method of Example 27, wherein the shard-related operation on the tensor comprises a copy operation configured to copy the tensor and place the copied tensor on the one or more DL devices.

[0126] Example 32 includes a method as described in any one of Examples 27 to 31, wherein transforming the HSP-annotated SD graph into the MD graph includes partitioning the HSP-annotated SD graph into the multiple subgraphs based on slice-related operations on each tensor in the SD graph, and inserting operation nodes corresponding to post-operations associated with each tensor in the SD graph.

[0127] Example 33 includes a method as described in Example 32, wherein, under the condition that the DL device on which the tensor is to be placed includes two or more DL stages for running corresponding two or more DL sub-workloads, transforming the HSP-annotated SD graph into the MD graph also includes adding a control dependency from an end operation node of a DL stage of the DL device to a start operation node of a next DL stage of the DL device.

[0128] Example 34 includes a method as described in any one of Examples 26 to 33, wherein annotating each tensor in the SD graph with a corresponding HSP includes adjusting the HSP for each tensor based on a device cost model for evaluating the computational cost of each operation assigned to the multiple heterogeneous devices, and obtaining an HSP-annotated SD graph with an optimal computational cost score as the HSP-annotated SD graph.

[0129] Example 35 includes a computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a processor circuit, cause the processor circuit to perform the method of any one of Examples 18 to 34.

[0130] Example 36 includes a device for heterogeneous sharding of deep learning (DL) workloads, the device comprising means for performing the method as described in any one of Examples 18 to 34.

[0131] Various technologies or some aspects or parts thereof may take the form of program codes (i.e., instructions) embodied in tangible media, such as floppy disks, CD-ROMs, hard disk drives, non-transient computer-readable storage media, or any other machine-readable storage media, wherein when the program code is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for implementing various technologies. Non-transient computer-readable storage media may be computer-readable storage media that do not include signals. In the case of program code execution on a programmable computer, a computing system may include a processor, a storage medium (including volatile and non-volatile memory and / or storage elements) that can be read by the processor, at least one input device, and at least one output device. Volatile and non-volatile memory and / or storage elements may be RAM, EPROM, flash drives, optical drives, magnetic hard disk drives, solid-state drives, or other media for storing electronic data. One or more programs that may implement or utilize the various technologies described herein may use application programming interfaces (application programming interfaces, APIs), reusable controls, and the like. Such programs may be implemented in high-level procedural or object-oriented programming languages ​​to communicate with a computer system. However, if desired, the program(s) may be implemented in assembly or machine language. In any case, the language may be a compiled or interpreted language, and combined with a hardware implementation. Exemplary systems or devices may include, but are not limited to, laptop computers, tablet computers, desktop computers, smart phones, computer terminals and servers, storage databases, and other electronic devices utilizing circuits and programmable memory, such as household appliances, smart televisions, digital video disc (DVD) players, heating, ventilating, and air conditioning (HVAC) controllers, light switches, and the like.

[0132] The above detailed description includes reference to the accompanying drawings, which form a part of the detailed description. The accompanying drawings illustrate specific embodiments that can be implemented. These embodiments are also referred to as "examples" in this article. Such examples may include additional elements other than those shown or described. However, the inventors also contemplate examples in which only those elements shown or described are provided. In addition, the inventors also contemplate examples (or one or more aspects thereof) using any combination or arrangement of those elements shown or described, whether for a specific example (or one or more aspects thereof), or for other examples (or one or more aspects thereof) shown or described herein.

[0133] All publications, patents, and patent documents cited in this document are hereby incorporated by reference in their entirety, as if individually incorporated by reference. In the event of inconsistent usages between this document and those documents so incorporated by reference, the usage in the incorporated reference(s) should be considered supplementary to that of this document; for irreconcilable inconsistencies, the usage in this document controls.

[0134] In this document, the term "a" is used to include one or more than one, independent of any other instance or use of "at least one" or "one or more," as is common in patent documents. In this document, the term "or" is used to refer to a non-exclusive or, so that "A or B" includes "A but not B," "B but not A," and "A and B," unless otherwise indicated. In the appended claims, the terms "including" and "wherein" are used as the plain English equivalents of the respective terms "comprising" and "wherein." In addition, in the appended claims, the terms "including" and "comprising" are open-ended, that is, systems, devices, articles, or processes that include other elements in addition to those listed after such terms in the claim are still considered to fall within the scope of the claim. In addition, in the appended claims, the terms "first," "second," and "third," etc. are used merely as labels and are not intended to impose numerical requirements on their objects.

[0135] The above description is intended to be illustrative, not restrictive. For example, the above examples (or one or more aspects thereof) may be used in combination with each other. Other embodiments may be used, for example, by a person of ordinary skill in the art after viewing the above description. The abstract is intended to allow the reader to quickly determine the nature of the technical disclosure and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the above detailed description, various features may be grouped together to simplify the disclosure. This should not be interpreted as a desire to make the disclosed features that are not claimed necessary for any claim. More precisely, the subject matter of the invention may be present in less than all the features of a particular disclosed embodiment. Thus, the attached claims are incorporated herein into the "Specific Implementation" section, with each claim acting independently as a separate embodiment. The scope of the embodiment should be determined with reference to the attached claims and the full scope of equivalents to which such claims are entitled.

Claims

1. An apparatus for heterogeneous sharding of deep learning (DL) workloads, comprising: Interface circuit; and a processor circuit coupled to the interface circuit and configured to: Based on device information about a plurality of heterogeneous devices received via the interface circuit, converting a single device (SD) graph representing the DL workload into a multi-device (MD) graph, the MD graph including a plurality of sub-graphs respectively representing a plurality of DL sub-workloads, the plurality of DL sub-workloads to be assigned to DL stages on the plurality of heterogeneous devices to complete the DL workload; and The plurality of subgraphs are assigned to respective DL stages on the plurality of heterogeneous devices.

2. The device according to claim 1, wherein: The device information includes at least one of the following: device type, computing capability, cache capacity, cache latency, cache bandwidth, memory capacity, memory latency, memory bandwidth of each of the multiple heterogeneous devices, and interconnection latency and bandwidth between the multiple heterogeneous devices.

3. The device according to claim 1, wherein: The processor circuit is configured to convert the SD map into the MD map based on the device information and the automatic tiling configuration received via the interface circuit.

4. The device as claimed in claim 3, wherein: The automatic sharding configuration includes a parallel processing method for sharding the DL workload among the plurality of heterogeneous devices.

5. The device according to claim 4, wherein: The parallel processing method includes a data parallel processing method or a pipeline parallel processing method.

6. The device according to claim 3, wherein: The automatic sharding configuration includes a minimum per-stage batch size for the plurality of heterogeneous devices.

7. The device of claim 3, wherein: The automatic sharding configuration includes an indicator for indicating whether a batch size of each stage of each of the plurality of heterogeneous devices needs to be consistent.

8. The device of claim 1, wherein: The DL workload includes a DL workload related to training or a DL workload related to inference.

9. The device according to any one of claims 1 to 8, wherein: The processor circuit is configured to convert the SD map to the MD map by: annotating each tensor in the SD graph with a corresponding heterogeneous slice property (HSP) to generate an HSP-annotated SD graph; and The HSP-annotated SD map is transformed into the MD map based on the HSPs associated with each tensor in the SD map.

10. The device of claim 9, wherein: For each tensor, the corresponding HSP is configured to indicate at least one of: sharding-related operations on the tensor, post-operations associated with the tensor, and one or more DL devices on which the tensor is to be placed, wherein the DL devices are selected from the plurality of heterogeneous devices.

11. The device of claim 10, wherein: The slicing-related operations on the tensor include one or more split operations, the HSP is configured to further indicate a split dimension specified for each split operation and a parameter associated with each of the one or more DL devices, and the split operation is configured to split the tensor along the split dimension to place the tensor on the one or more DL devices based on the parameters associated with the DL devices.

12. The device of claim 11, wherein: The parameters associated with each DL device of the one or more DL devices include: a parallel data size of the DL device, and a number of DL stages for running respective DL sub-workloads associated with the DL device.

13. The device of claim 11, wherein: Under the condition that the tensor has a dynamic tensor shape, the parameters associated with each DL device of the one or more DL devices include: a ratio of a DL sub-workload associated with the DL device to the DL workload, and a number of DL stages used to run each DL sub-workload associated with the DL device.

14. The device of claim 10, wherein: The shard-related operation on the tensor comprises a copy operation configured to copy the tensor and place the copied tensor on the one or more DL devices.

15. The device of claim 10, wherein: Transforming the HSP-annotated SD graph into the MD graph includes: based on the slice-related operations on each tensor in the SD graph, splitting the HSP-annotated SD graph into the multiple subgraphs, and inserting operation nodes corresponding to post-operations associated with each tensor in the SD graph.

16. The device of claim 15, wherein: Under the condition that the DL device on which the tensor is to be placed includes two or more DL stages for running corresponding two or more DL sub-workloads, transforming the HSP-annotated SD graph into the MD graph also includes: adding a control dependency from an end operation node of a DL stage of the DL device to a start operation node of a next DL stage of the DL device.

17. The device of claim 9, wherein: Annotating each tensor in the SD graph with a corresponding HSP includes: adjusting the HSP for each tensor based on a device cost model for evaluating the computational cost of each operation assigned to the plurality of heterogeneous devices, and obtaining an HSP-annotated SD graph with an optimal computational cost score as the HSP-annotated SD graph.

18. A method for heterogeneous sharding of deep learning (DL) workloads, comprising: Based on device information about a plurality of heterogeneous devices, converting a single device (SD) graph representing the DL workload into a multi-device (MD) graph, the MD graph including a plurality of sub-graphs respectively representing a plurality of DL sub-workloads, the plurality of DL sub-workloads to be assigned to DL stages on the plurality of heterogeneous devices to complete the DL workload; and The plurality of subgraphs are assigned to respective DL stages on the plurality of heterogeneous devices.

19. The method of claim 18, wherein: The device information includes at least one of the following: device type, computing capability, cache capacity, cache latency, cache bandwidth, memory capacity, memory latency, memory bandwidth of each of the multiple heterogeneous devices, and interconnection latency and bandwidth between the multiple heterogeneous devices.

20. The method of claim 18 or 19, wherein: Converting the SD graph into the MD graph comprises: annotating each tensor in the SD graph with a corresponding heterogeneous slice property (HSP) to generate an HSP-annotated SD graph; and The HSP-annotated SD map is transformed into the MD map based on the HSPs associated with each tensor in the SD map.

21. The method of claim 20, wherein: For each tensor, the corresponding HSP is configured to indicate at least one of: sharding-related operations on the tensor, post-operations associated with the tensor, and one or more DL devices on which the tensor is to be placed, wherein the DL devices are selected from the plurality of heterogeneous devices.

22. The method of claim 21, wherein: Transforming the HSP-annotated SD graph into the MD graph includes: based on the slice-related operations on each tensor in the SD graph, splitting the HSP-annotated SD graph into the multiple subgraphs, and inserting operation nodes corresponding to post-operations associated with each tensor in the SD graph.

23. The method of claim 22, wherein: Under the condition that the DL device on which the tensor is to be placed includes two or more DL stages for running corresponding two or more DL sub-workloads, transforming the HSP-annotated SD graph into the MD graph also includes: adding a control dependency from an end operation node of a DL stage of the DL device to a start operation node of a next DL stage of the DL device.

24. The method of claim 20, wherein: Annotating each tensor in the SD graph with a corresponding HSP includes: adjusting the HSP for each tensor based on a device cost model for evaluating the computational cost of each operation assigned to the plurality of heterogeneous devices, and obtaining an HSP-annotated SD graph with an optimal computational cost score as the HSP-annotated SD graph.

25. A computer readable medium having instructions stored thereon, wherein: The instructions, when executed by a processor circuit, cause the processor circuit to perform the method of any one of claims 18 to 24.

Images