Executing computations of neural networks

Abstract

According to an implementation of the disclosure, a solution for executing a computation for a neural network is provided. According to the solution, a data flow graph for a neural network to be computed is obtained, and the data flow graph indicates at least one operation in the neural network and data respectively associated with the at least one operation; scheduling information for the neural network is determined based on the data flow graph and a processing resource configuration of a target device for computing the neural network, the scheduling information indicates a data transformation required to execute the at least one operation; and a computation for the neural network is executed at the target device based on the scheduling information. The implementation of the disclosure supports executing a computation for a neural network at devices with different configurations, providing versatility.

Description

EXECUTING COMPUTATIONS OF NEURAL NETWORKSBACKGROUND

[0001] Building on the recent advancements in deep learning models, there is a growing demand for more powerful computing performance in hardware accelerators like GPUs. The inherent robustness of deep learning to errors enables the use of lower precision arithmetic, setting it apart from traditional workload like scientific computing, which necessitates high precision like float64. In line with this trend, cutting-edge accelerators are increasingly integrating more low-precision computational units, such as 32-bit, 16-bit, and 8-bit floating-point operations, into their new generations.SUMMARY

[0002] According to an implementation of the present disclosure, a solution for executing a computation for a neural network is provided. In the solution, a data flow graph for a neural network to be computed is obtained, and the data flow graph indicates at least one operation in the neural network and data respectively associated with the at least one operation. Scheduling information for the neural network is determined based on the data flow graph and a processing resource configuration of a target device for computing the neural network, and the scheduling information indicates a data transformation required to execute the at least one operation. A computation for the neural network is executed at the target device based on the scheduling information. The implementation of the present disclosure supports executing a computation for a neural network at devices with different configurations, providing versatility. In this way, data involved in the neural network computation may be transformed, so that the transformed data conforms to the processing resource configuration of the target device, thereby improving the computation efficiency of the neural network.

[0003] This section is provided to introduce a selection of objects in a simplified form that is further described below in the Detailed Description. This section is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.BRIEF DESCRIPTION OF THE DRAWINGS[0004JFIG. 1 illustrates a block diagram of an example environment in which various implementations of the present disclosure can be implemented;[0005JFIG. 2 illustrates a schematic diagram of various data formats used in a deep learning model; [0006JFIG. 3A illustrates a schematic diagram of a computation pipeline for mixed-precision matrix multiplication in a conventional approach;

[0007] FIG. 3B illustrates a schematic diagram of a computation pipeline for mixed-precisionmatrix multiplication in accordance with some implementations of the present disclosure;[0008JFIG. 4 illustrates a schematic diagram of an example architecture for performing a computation for a neural network in accordance with some implementations of the present disclosure;[0009JFIG. 5A illustrates an example of data types defined by element granularity or block granularity in accordance with some implementations of the present disclosure;[OO1OJFIG. 5B illustrates an example of implementing a tensor of a specific data type according to some implementations of the present disclosure;[OO11JFIG. 5C illustrates an example of an operator of a neural network in accordance with some implementations of the present disclosure;[0012JFIG. 6A illustrates a tensor expression of an example operation used in a neural network in accordance with some implementations of the present disclosure;[0013JFIG. 6B illustrates a schematic of a hierarchy of a target device and cross-level loading of data according to some implementations of the present disclosure;[0014JFIG. 6C illustrates pseudo code corresponding to a tensor expression according to some implementations of the present disclosure;[0015JFIG. 7 illustrates an example definition of a transformation action according to some implementations of the present disclosure;[0016JFIG. 8 illustrates a flow diagram of a process for performing a computation for a neural network in accordance with some implementations of the present disclosure; and[0017JFIG. 9 illustrates a schematic block diagram of an electronic device capable of implementing various implementations of the present disclosure.DETAILED DESCRIPTION

[0018] The disclosure will now be discussed with reference to a number of example implementations. It should be understood that these implementations are discussed only to enable those of ordinary skill in the art to better understand and thus implement the disclosure, and not to imply any limitation on the scope of the disclosure.

[0019] As used herein, the term "including / comprising" and variations thereof are to be read as an open-ended term that "including / comprising but is not limited to". The term "based on" is to be read as "based at least in part on. " The terms "one implementation" and "an implementation" are to be read as "at least one implementation". The term "another implementation" is to be read as "at least one other implementation. " The terms "first", "second", and the like may refer to different or the same object. Other explicit and implicit definitions may also be included below.

[0020] It is noted that the title of any section / subsection provided herein is not limiting. Various implementations are described throughout herein, and any type of implementation can be includedunder any section / subsection. Furthermore, the implementations described in any section / subsection may be combined in any manner with any other implementations described in the same section / subsection and / or different sections / subsections.

[0021] Herein, unless explicitly stated otherwise, performing a step "in response to A" does not mean that the step is performed immediately after "A", but may include one or more intermediate steps.

[0022] As used herein, the term "model" may learn from training data associations between corresponding inputs and outputs, such that after training is complete a corresponding output may be generated for a given input. The generation of the model may be based on machine learning technology. Deep learning (DL) is a machine learning algorithm, which processes an input by using a multi-layer processing unit and provides a respective output. A neural network model is one example of a model based on deep learning. Herein, "model" may also be referred to as "machine learning model", "learning model", "machine learning network", or "learning network", and these terms may be used interchangeably herein.

[0023] As used herein, the term "tensor" may have any suitable number of dimensions, and each dimension may have any suitable length. For example, the tensor may be a one-dimensional tensor (that is, a vector), a two-dimensional tensor (that is, a matrix), a three-dimensional tensor, or a tensor with any other number of dimensions.Example Environment

[0024] FIG. 1 illustrates a schematic diagram of an example environment 100 in which implementations of the present disclosure can be implemented. In the environment 100 of FIG. 1, a computation of a neural network 120 is performed at a target device 110 to obtain an inference result or to train the neural network 120. The target device 110 may be any hardware system or hardware platform with computing capability. The target device 110 may be any suitable type of general-purpose device or dedicated processing device. Examples of general-purpose processing devices include, but are not limited to, a CPU, a GPU, a field programmable gate array (FPGA), and / or the like. Examples of dedicated processing devices include, but are not limited to, an accelerated processing unit (APU), a TPU, a neural network processing unit (NPU), or other existing or future-developed artificial intelligence (Al chip). It is to be understood that, such an acceleration processing resource is mainly used to improve the computation speed of an Al-related application.

[0025] As shown in FIG. 1, the neural network 120 may be abstracted into one or more operations 130-1, 130-2,... , 130-N for a tensor, which is also collectively or individually referred to as an operation 130, where N is a positive integer. The operation 130 may include a variety7of suitable general-purpose calculations or computations such as, but not limited to, matrix multiplication,convolution, accumulation, transpose, and / or the like. Alternatively, or in addition, the operation 130 may include a custom calculation or computation, such as padding, convolution padding, wildcard matching, or the like.

[0026] It should be appreciated that the structure and functionality of the environment 100 are described for example purposes only and are not intended to imply any limitation on the scope of the disclosure. In the embodiments of the present disclosure, the operations and custom computations performed by the target device 110 may come from other types of services such as models, algorithms, or traditional data processing, without limitation to neural networks.

[0027] While the low-precision computational units are developing, various custom low-precision data types are being investigated, such as mixed precision formats, to strike an optimal balance between model accuracy and training efficiency. Moreover, during the model deployment phase, computations can be converted to even more compact data representations to achieve higher efficiency.

[0028] Ho ever, hardware accelerators are challenging in keeping pace with the diverse and rapidly evolving requirements for supporting various data precision formats, for example, custom data types. This difficulty arises because each accelerator can only integrate a few types of computing units for standard data types, given the limited chip area and high hardware cost. Even forthose recently supported low-precision datatypes, such as those under 16 bits in width, existing softw are is generally inefficient due to the complexity of aligning fine-grained low-bit data access with the coarse-grained memory system. For instance, a type of shared memory bank size is 4 bytes in width, and simply loading or storing 8-bit data elements can easily lead to bandwidth waste. The inadequacy and inefficiency in supporting these new custom data types significantly hinder the innovation for the deployment of both deep learning models and hardware accelerators.

[0029] Despite hardware accelerators lacking computing instructions for those custom datatypes, their memory system can be utilized to store arbitrary7dataty pes by casting them into a data chunk with a fixed bit width. Most custom data types can be losslessly converted to a wider-bits standard data type supported by the computing units in existing hardware. For example, NF4 tensors can be computed with an FP 16 or FP32 operation by converting their data types.[0030JA typical tensor computation pipeline involves loading data from multiple layers of memory hierarchy, such as DRAM. L2 cache, shared memory7, register, and so on. Converting tensor data types in different layers could significantly impact the performance factors like memory footprint, data access traffic, hardware cost, and so on, which is complex to optimize. For example, converting a low-bit data chunk to a higher-bit ty pe in a register could lead to register spill, causing a dramatic performance drop. In addition, a computational pipeline involving different data types usually requires different data layout optimization to align with memory7system, for example, align with memory bank, to maximize the data access throughput. Existing optimizations like swizzling memory7access are mostly7designed for few specific data types, which is hard to be generalized.

[0031] The increasing demand to scale deep learning models to larger sizes, such as Large Language Models (LLM), enhances the requirement of computing in lower bits and mixed precision to increase computation efficiency7and save memory. FP32 data type has been the go-to choice for data representation in deep learning models. However, recent practices suggest that the high precision of FP32 is not always necessary7. Lower precision can deliver the same level of effectiveness while simultaneously reducing costs. During inference, models are frequently- quantized to lower precision, typically down to 8 or 4 bits. FIG. 2 illustrates a schematic diagram 200 of various data formats used in a deep learning model, as shown in FIG. 2, high-accuracy data formats are shifting to low-bit data formats.

[0032] To improve the accuracy and robustness of low-precision deep learning models, a common approach is to use a scaling factor to rescale the values for a more accurate representation of the data distribution. Traditional methods ty pically perform scaling in a tensor-wise or channel-wise manner. However, group-wise scaling, by virtue of its finer granularity7, can better capture the distribution of sub-tensors or groups, leading to improved performance.

[0033] Mixed-precision operations emerge in data quantization due to the varying sensitivity of different tensors to lower bit quantization. For example, mixed-precision training employs a combination of higher and lower bit tensors, such as FP32, FP16, and 8 bits float (FP8). This utilization of precision levels strikes a balance between computational efficiency and precision, thereby optimizing performance.

[0034] Hardware accelerators like GPUs are constantly adapting to the evolving data type requirements in deep learning. Early generations of GPUs, such as standard data types like FP32 and FP64. As deep learning workloads gained relevance, lower precision formats like FP16 were introduced in the GPU architecture, and computational efficiency may be improved by adjusting the precision-performance trade-off. However, hardware ty pically lags behind the requirements of algorithms or models. When encountering unsupported formats, they need to be converted or simulated in a higher-precision supported format. This could lead to significant performance issues and inefficiencies.

[0035] Low-precision computing is challenging to optimize due to the fine-grained data access granularity and special hardware units, such as TensorCore. The hardware utilization of low- precision computing is generally low, for example, less than 50% on average. Some hardware- supported precisions are not well supported by the software. For example, while INT8 is supported in some GPUs, most existing deep learning compilers do not support INT8 computing on theseGPUs.

[0036] A DNN operation is often implemented as a computing pipeline, which continuously loads small data tiles from input tensors across multiple layers of memory' hierarchy to compute in the top-level cores. The problem found in conventional solutions by the present disclosure will be described below by taking a mixed-precision matrix multiplication such as FP16xINT8 as an example. FIG. 3 A illustrates a schematic 300 of a computation pipeline for mixed-precision matrix multiplication in a conventional approach. As illustrated in FIG. 3A, small data tiles are continuously loaded from input tensors across an L2 memory layer 302. an LI memory layer 304, and an L0 memory layer 306 for computation. Each memory layer usually has its preferred minimum access granularity, such as an 8-byte transaction length in the LI memory layer 304. Some of the latest GPUs even introduce built-in instructions for highly efficient data loading, which load a two-dimensional data tile at a time, for instance, load a 2x2 data tile. Given that a data tile is typically stored in a strided memory space, data access often becomes unaligned with the transaction length or instruction shape, potentially leading to low bandwidth utilization. For example, each memory access from the LI memory layer 304 only achieves half utilization for both tensors. Furthermore, due to the absence of computing instructions for FP16*INT8, the operation cannot be supported, even if the corresponding data is loaded into the register.

[0037] In order to solve each of the above problems, implementations of the present disclosure provide solutions for performing computations of a neural network. According to the implementation of the present disclosure, in the solution, a data flow graph for a neural network to be computed is obtained, and the data flow graph indicates at least one operation in the neural network and data respectively associated with the at least one operation; scheduling information for the neural network is determined based on the data flow graph and a processing resource configuration of a target device for computing the neural network, the scheduling information indicates a data transformation required to execute the at least one operation; and a computation for the neural network is executed at the target device based on the scheduling information. The implementation of the present disclosure supports executing a computation for a neural network at devices with different configurations, providing versatility. In this way, data involved in the neural network computation may be transformed, so that the transformed data conforms to the processing resource configuration of the target device, thereby improving the computation efficiency of the neural network.[0038JFIG. 3B illustrates a schematic diagram 350 of a computation pipeline for mixed-precision matrix multiplication in accordance with some implementations of the present disclosure. For example, each 2x2 data tile is stored in contiguous memory space in an LI memory' layer 354 so that the load instructions at the upper layer can fully utilize the bandwidth. Moreover, given thatthe computing instruction only supports the FP 16 data format, a tensor 360 may be converted from INT8 to FP16 during the data loading from an L2 memory layer 352 to the LI memory layer 354. Consequently, the data loading from the L2 memory layer 352 to the LI memory layer 354 efficiently leverages the low traffic due to the low-bit data type, the data loading from the LI memory layer 354 to an L0 memory layer 356 fully utilizes the memory bandwidth through transaction alignment, and the computation is ultimately accelerated in the hardware computing unit by type conversion. This example demonstrates that a DNN computation on a custom data type not supported by hardware can still be scheduled and optimized through a transformation on its layout and data types.

[0039] Some example implementations of the present disclosure will be described in more detail below with reference to the accompanying drawings.Example Architecture

[0040] FIG. 4 illustrates a schematic diagram of an example architecture 400 for performing a computation for a neural network in accordance with some implementations of the present disclosure. In general, the architecture 400 includes a data flow graph 402 for a neural network to be computed and the target device 110 for computing the neural network. The data flow graph 402 indicates at least one operation in the neural network and data respectively associated with the at least one operation. In this example, a matrix multiplication (which is represented by MatMul) operation 404 and datatypes of input data and output data for the MatMul operation 404 are shown, for example, data 408 with an FP16 format and data 410 with an NF4 format. This example also shows an add (which is represented by add) operation 406 and input data and output data for the operation 406. Operations herein may also be referred to as operators.

[0041] Data types in DNN computation are usually defined at either element-wise granularity or block-wise granularity'. To express these data types, the concept of / T pe is introduced in the present disclosure. FIG. 5A illustrates the definition of / Type in accordance with some implementations of the present disclosure. As shown in FIG. 5A, / Type 502 represents a data type that consists of a group of homogeneous elements. A parameter shape 504 represents an n- dimensional array shape, that is, the layout of these elements. Each element shares the same data ty pe. A parameter nElemBits 506 represents the number of bits to store an element. This group of elements may share the same metadata 508, that is, metadata. Since data types may be losslessly represented by some higher-bit data types, c tTypes 510 represents / Type 502 may be losslessly converted to another / Type 502 with a c_func function 512.

[0042] Both existing commonly-used data types and new customized data types can be represented with the / Type 502. For example, the FP16 type can be expressed as the / Type 502 of the parameter shape 504=[l] with the parameter nElemBits 506=16. The element-wise granulated NF4 type canbe expressed as the / Type 502 of the parameter shape 504=[l] with the parameter nElemBits 506=4 and the shared value map in the metadata 508. The NF4 type can be losslessly represented as FP16 type, and therefore there could be a <FP16, NF4_to_FP16_func> entry in the c tTypes 510. The block-wise granulated OCP-MXFP8 type can be expressed as the parameter / Type 502 of shape 504=

[0032] with the parameter nElemBits 506=8 and the shared scaling factor in the metadata 508.

[0043] Based on the / Type 502 that represents a data type, the present disclosure proposes / Tile to represent a tensor of a specific data type. FIG. 5B illustrates the definition of the / Tile in accordance with some implementations of the present disclosure. As illustrated in FIG. 5B, the / Tile 512 is defined as a group of homogeneous elements with the same parameter type 514 (that is, the data type) and the parameter shape 504. Elements in the / Tile 512 share the parameter metadata 508. Besides, elements in the / Tile 512 are stored as row-major.

[0044] A DNN operation (also referred to as DNN operator, such as MatMul) are commonly implemented as a group of independent and homogeneous tasks, where each task processes a tile of the input tensor and outputs a tile of the output tensor. The present disclosure leverages / Tileoperator to represent a DNN operator. FIG. 5C illustrates the definition of a / Tile-operator in accordance with some implementations of the present disclosure. As shown in FIG. 5C, a / Tileoperator 522 represents the tensor computation task over elements. A function get input tTil es() 524 and a function get_output_tTiles() 526 return the input / Tile 512 and the output / Tiles 512 of this computation task. A function computeQ 528 executes the computation defined in the tensor expression expr 530 for the input / Tiles 512 and the output / Tiles 512.

[0045] In some implementations, a tensor expression is determined for each of the at least one operation, and the tensor expression indicates input data for the operation and a data type of the input data, and output data for the operation and a data type of the output data. A tensor expression of an example operation is described with reference to FIG. 6A, which illustrates a tensor expression 600 of a MatMul operation. As shown in FIG. 6A, the tensor expression 600 represents an operation that a FP16 type tensor A 602 with a shape of [M, K] multiplies aNF4 type tensor B 604 with a shape of [N, K], with FP32 ty pe as the accumulation and outputs a FP 16 type tensor C 606 with a shape of [M, N], where M=32, N=32, K=63. The tensor expression 600 indicates input data for the operation (that is, the tensor A 602 and the tensor B 604) and a data type of the input data (that is, the FP 16 type and the NF4 type), and output data for the operation (that is. the tensor C 606) and a data type of the output data (that is, the FP16 type). In the example of FIG. 6A, the data type of the input data and the output data may be indicated using a / Type annotation, that is, “@” and a data type (for example, A[i, k]@FP16) are added following the input data and the output data. In this way. by annotating data with data type, given a shape, input data tiles andoutput data tiles can be inferred, thereby improving the efficiency of neural network computations. It should be noted that the data types, data shapes, and data layout listed in the present disclosure are merely examples, and the present disclosure is not limited thereto.

[0046] After each tensor expression is obtained, the dataflow graph 402 may be determined based on a relationship between the tensor expression determined respectively for the at least one operation and the at least one operation.

[0047] With continuing reference to FIG. 4, scheduling information 404 for the neural network may be determined based on the data flow graph 402 and a processing resource configuration of the target device 110. The scheduling information 408 indicates a data transformation required to execute the at least one operation. The processing resource configuration of the target device 110 may include a data type, a transaction size, and / or the like supported by the target device 110.

[0048] In some implementations, the target device 110 includes a plurality of processing layers. The target device 110 (as an example of the hardware accelerator) typically has a hardware hierarchy that includes a plurality of processing layers, such as the L2 memory layer 352, the LI memory layer 354, the L0 memon layer 356, and a computation layer 358. In an example, the L2 memory layer 352 may be a video memory, the LI memory7layer 354 may be a shared memory7, the LO memory layer 356 may be a register, and the computation layer 358 may be a computational unit. Each processing layer has its preferred data access granularity, for example, the shared memory of a GPU requires a transaction of 324-byte, and a compute unit requires a certain shape of data. In an example, an instruction in a GPU processes at the granularity7of two types of FP16. These requirements (that is, processing resource configuration) may be described as the / Tiles 512.

[0049] In some implementations, the target device 110 may be abstracted as a device represented by tTile, that is, / Tile-device, with each layer represented as the / Tile 512. FIG. 6B illustrates a schematic diagram 630 of a hierarchy of the target device 110 and crossdevel loading of data according to some implementations of the present disclosure. As shown in FIG. 6B, computing instructions of the target device 110 require processing at a granularity of [16,16] and [8, 16] for two inputs, respectively, and the computation layer 358 may be expressed as the / Tile 512 of the shape 504=[l 6, 16] w ith the type 514=FP16. The data loading instruction of the target device 110 requires loading shape [16,2] of data at a granularity7of half8 (that is, 8 FP16 value), and the L0 memory layer 356 may be expressed as the / Tile 512 of the shape 504=[16,2] with the type 514=16B. Besides, the LI memory layer 354 may be expressed as the / Tile 512 of the shape 504=

[0032] with the type 514=4B. The L2 memory layer 352 may be expresses as the / Tile 512 of the shape 504=

[0032] with the ty pe 514=1B.

[0050] In some implementations, for a given operation of the at least one operation, hint information is determined based on the given operation and respective configurations of theplurality of processing layers. The hint information indicates a range of shapes of data blocks available for the given operation in each of the plurality of processing layers. A data transformation to be executed for the given operation between processing layers of the plurality of processing layers is determined based on the hint information. Each processing layer may have a plurality of shapes of data blocks available for the given operation, and thus the data transformation to be executed between processing layers of the plurality of processing layers (for example, transforming the processed data block into a data block with an optimal shape) may be further determined.

[0051] In some implementations, the plurality of processing layers comprises a first processing layer and a second processing layer, and the first processing layer is above the second processing layer. In an example, the first processing layer may comprise the computation layer 358, and the second processing layer may comprise at least one of the L2 memory' layer 352, the LI memory layer 354, and the LO memory layer 356. The first processing layer being above the second processing layer means that the processed data block is stored in the second processing layer, and the processed data needs to be loaded from the second processing layer to the first processing layer for computation.

[0052] In some implementations, a first shape of a data block for the given operation in the first processing layer is determined based on the hint information and a processing bandwidth of the first processing layer. The first shape may be selected from a range of shapes of data blocks available for the given operation in the first processing layer indicated by the hint information, and the first shape may maximize utilization of the processing bandwidth of the first processing layer. In an example, the first processing layer is the computation layer 358 that may process data with a shape [16,16] or a shape [8,16], To maximize utilization of the processing bandwidth of the computation layer 358, the first shape may be determined to be [16,16],

[0053] After the first shape is determined, a second shape of a data block for the given operation in the second processing layer is determined based on the hint information, the first shape, and a bandwidth of a storage resource of the second processing layer. On the basis that the first shape is satisfied, the second shape may be selected from a range of shapes of data blocks available for the given operation in the second processing layer indicated by the hint information, and the second shape may maximize utilization of the bandwidth of storage resources of the second processing layer. In an example, the second processing layer is the LO memory layer 356. To conform to the first shape [16,16] and maximize the utilization of bandwidth of storage resources of the LO memory layer 356, the second shape may be determined to be [16,2],

[0054] Based on the first shape and the second shape, a data transformation to be executed for the given operation between the first processing layer and the second processing layer may bedetermined. For example, the processed data may first be converted to a first shape of data block and then converted to a second shape of data block. In this way, bandwidth utilization of the various layers of the target device 110 may be maximized in the process of inference from an upper layer to a lower layer, thereby improving the efficiency of computations of the neural network.

[0055] In some implementations, a target data block to be loaded from the second processing layer to the first processing layer for the given operation is determined, and a transformation action for the target data block is determined based on the first shape, the second shape, and a target data type configured to the first processing layer. In an example, the target data block may be in any shape, any data type. The transformation action may first change the target data block into a data block with a second shape, and then transform the data block with the second shape into a data block with a first shape and the data type is the target data type.

[0056] In some implementations, the transformation action may include at least one of actions (also referred to as primitives) including a slice action, a map action, a pad action, or a type convert action. The / Tile 512-represented DNN computation in the data flow graph 402 should align with / Tile-device for efficient execution of the neural network. Therefore, the present disclosure proposes a / Tile transformation mechanism to transform the layout of the processed / Tile or / Type into an equivalent / Tile. Specifically, the computation pipeline of the / Tile-operator 522 may be expanded to three stages on the hardware hierarchy, namely transform loading represented by Transform-Load, computation represented by Compute, and transform storage represented by Transform-Store.

[0057] FIG. 6C illustrates pseudo code 640 corresponding to a tensor expression 600 in accordance with some implementations of the present disclosure. Taking the tensor B 604 as an example, a pseudo code fragment 642 is used to transform and load the tensor B 604 from the L2 memory layer 352 to the LI memory layer 354 as FP16 type. The pseudo code fragment 644 is used to load the transformed tensor B 604 from the LI memory layer 354 to the L0 memory layer 356 with a shared storage read instruction (represented by Idmatrix). The pseudo code fragment 646 is used to perform a computation between the tensor A 602 and the tensor B 604 at the computation layer 358, with the FP 32 type as an intermediate accumulated result. The pseudo code fragment 648 indicates that a computation result C of the computation layer 358 is converted and stored to the L2 memory' layer 352 as the FP16 type.

[0058] The actions involved in executing Transform-Load or Transform- Store on the tensor B604 will be described below with continued reference to FIG. 6B. Taking the tensor B 604 as an example, a code fragment 622 is used to perform a slice action on the / Tile describing the tensor B 604 to get a tO with a shape [16, 63], and the slice action is used to slice the processed data blockto get a data block with a first predetermined shape. The code fragment 624 is used to perform a pad action on tO to get tl with a shape of [16, 64], and the pad action is used to pad the boundary of the processed data block to get a data block with a second predetermined shape. The code fragment 626 is used to perform a type convert action on tl to get t2 with the FP16 type, and the type convert action is used to convert the data type of the processed data block. The code fragment 628 is used to perform map action on t2 to get ret meeting the requirements of the LI memory layer 354 and LO memory layer 356. Data block shapes and layouts from 630-1 to 630-4 represent the shape and layout of the tensor B 604 at the L2 memory layer 352, the LI memory layer 354, the LO memory layer 356, and the computation layer 358, respectively, and data types from 632- 1 to 632-4 represent the data type of the tensor B 604 at the L2 memory layer 352, the L 1 memory layer 354, the LO memory layer 356, and the computation layer 358, respectively.

[0059] FIG. 7 illustrates a definition of an operation according to some implementations of the present disclosure. As illustrated in FIG. 7, a slice action 702 is used to slice an input data block represented by a parameter tTile input 704 into an output data block represented by a parameter out_shape 706. A map action 712 is used to map the address of each element in the tTile_input 704 to an expected address by a mapping function map_func 714. A pad action 722 is used to pad the tTile_input 704 with the pad_value 724 on a border given in a parameter pad_shape 726. The type convert action 732 is used to convert the parameter tTile input 704 into a new data type of data block represented by a parameter new_type 732.

[0060] After determining the scheduling information, a computation of the neural network 120 may be executed at the target device 110 based on the scheduling information.

[0061] In some implementations, for a given operation of the at least one operation, a target memory layer in the target device may be determined based on the given operation. In the example of FIG. 4, if the given operation is an intermediate operation (for example, MatMul operation 404), and the result of the computation for the given operation requires for the next operation, the target memory layer may be determined as a higher memory layer (for example, the L0 memory layer 356). thereby improving the efficiency of subsequent computations. If the given operation is a final operation (for example, the Add operation 406), and the result of the computation for the given operation does not require to be used for the next operation, the target memory- layer may be determined as a lower memory layer (for example, the L2 memory' layer 352), thereby conserving resources of the higher memory layer. After the target memory layer is determined, the output result of executing the given operation may be stored to the target memory layer.

[0062] In this way, all custom data types are supported by separating data storage and computation. That is, the tensor is stored and transmitted in custom data ty pes, and the tensor is computed in standard data ty pe by type conversion. In this way, the performance benefits of low-bit data typesmay be efficiently exploited by saving memory traffic and footprint.

[0063] In some implementations, for a given operation of the at least one operation, input data for the given operation may be loaded from a first memory layer of the target device to a computation layer of the target device based on a data transformation indicated by the scheduling information. The given operation is executed at the computation layer to obtain output data for the given operation. The output data is stored to a second memory layer of the target device for a subsequent operation or output. The first memory layer herein may be the same as or different from the second memory layer. In an example, with continued reference to FIG. 4, the output data 405 of the MatMul operation 404 is stored at the L0 memory layer 356, and in order to perform the subsequent Add operation 406, the operation result 405 needs to be loaded from the L0 memory layer 356 (that is, the first memory layer) to the computation layer 358. The output data 407 of the Add operation 406 is stored to the L2 memory layer (that is, the second memory layer), at which point the first memory layer is distinct from the second memory’ layer.Example Process[0064JFIG. 8 illustrates a process 800 for executing a computation for a neural network in accordance with some implementations of the present disclosure. The process 800 may be implemented at a computing system that includes the target device 110.

[0065] At block 810. the computing system obtains a data flow graph for a neural network to be computed. The data flow graph indicates at least one operation in the neural network and data respectively associated with the at least one operation.

[0066] At block 820, the computing system determines scheduling information for the neural network based on the data flow graph and a processing resource configuration of a target device for computing the neural network, and the scheduling information indicates a data transformation required to execute the at least one operation.

[0067] At block 830, the computing system executes, at the target device, a computation for the neural network based on the scheduling information.

[0068] In some implementations, the target device comprises a plurality of processing layers, and determining the scheduling information comprises: for a given operation of the at least one operation, determining hint information based on the given operation and respective configurations of the plurality of processing layers, the hint information indicating a range of shapes of data blocks available for the given operation in each of the plurality of processing layers; and determining, based on the hint information, a data transformation to be executed for the given operation between processing layers of the plurality’ of processing layers.

[0069] In some implementations, the plurality of processing layers comprises a first processing layer and a second processing layer, the first processing layer is above the second processing layer,and determining the data transformation to be executed for the given operation between processing layers of the plurality of processing layers comprises: determining a first shape of a data block for the given operation in the first processing layer based on the hint information and a processing bandwidth of the first processing layer; determining a second shape of a data block for the given operation in the second processing layer based on the hint information, the first shape, and a bandwidth of a storage resource of the second processing layer; and determining, based on the first shape and the second shape, a data transformation to be executed for the given operation between the first processing layer and the second processing layer.

[0070] In some implementations, determining the data transformation to be executed for the given operation between the first processing layer and the second processing layer comprises: determine a target data block to be loaded from the second processing layer to the first processing layer for the given operation; and determining a transformation action for the target data block based on the first shape, the second shape, and a target data type configured to the first processing layer.

[0071] In some implementations, the transformation action comprises at least one of: a slicing action for slicing a processed data block to obtain a data block with a first predetermined shape, a padding action for padding a boundary of the processed data block to obtain a data block with a second predetermined shape, a mapping action for transforming an arrangement of elements in the processed data block, or a type conversion action for converting a data t pe of the processed data block.

[0072] In some implementations, determining the scheduling information comprises: for a given operation of the at least one operation, determining a target memory layer in the target device based on the given operation; and executing, at the target device, the computation for the neural network comprises: storing an output result of executing the given operation to the target memory layer.

[0073] In some implementations, obtaining the data flow graph comprises: determining a tensor expression for each of the at least one operation, the tensor expression indicating input data for the operation and a data type of the input data, and output data for the operation and a data type of the output data; and determining the data flow graph based on a relationship between the tensor expression determined respectively for the at least one operation and the at least one operation.

[0074] In some implementations, executing, at the target device, the computation for the neural network comprises: for a given operation of the at least one operation, loading input data for the given operation from a first memory layer of the target device to a computation layer of the target device based on a data transformation indicated by the scheduling information; executing, at the computation layer, the given operation to obtain output data for the given operation; and storing the output data to a second memory’ layer of the target device for a subsequent operation or output.Example Device[0075JFIG. 9 illustrates a schematic block diagram of an electronic device capable of implementing multiple implementations of the present disclosure. It should be understood that the electronic device 900 shown in FIG. 9 is merely an example and should not constitute any limitation on the functionality and scope of the implementations described in the present disclosure.

[0076] As shown in FIG. 9, the electronic device 900 comprises an electronic device 900 in the form of a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to, one or more processors or processing devices 910, a memory 920, a storage device 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960.

[0077] In some implementations, the electronic device 900 may be implemented as a computing device, a computing system, a server, a mainframe, and / or the like with computing capabilities.

[0078] The processing device 910 may be an actual or virtual processor and can perform various processes according to programs stored in the memory 920. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capabilities of the electronic device 900. The processing device 910 may include a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a controller, and / or a microcontroller, and / or the like.

[0079] The electronic device 900 typically includes a number of computer storage medium. Such medium may be any available medium that is accessible to the electronic device 900, including, but not limited to, a volatile and non-volatile medium, a removable and non-removable medium. The memory 920 may include a volatile memory (for example, a register, a cache, a random access memory (RAM)), a non-volatile memory (for example, a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory), or some combination thereof. The storage device 930 may include a removable or non-removable medium, and may include a computer-readable medium such as a memory, a flash drive, a magnetic disk, or any other medium that can be used to store information and / or data and can be accessed within the electronic device 900.

[0080] The electronic device 900 may further comprise an additional removable / non-removable, volatile / nonvolatile storage medium. Although not shown in FIG. 9, a magnetic disk drive for reading from or writing to a removable, nonvolatile magnetic disk and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data medium interfaces.

[0081] The communication unit 940 enables communication with another computing device overa communication medium. Additionally, the functionality of the components of the electronic device 900 may be implemented in a single computing cluster or multiple computing machines that are capable of communicating over a communication connection. Accordingly, the electronic device 900 may operate in a networked environment using logical connections to one or more other servers, personal computers (PCs), or another general network node.

[0082] The input device 950 may be one or more various input devices such as a mouse, a keyboard, a data import device, and / or the like. The output device 960 may be one or more output devices, such as a display, a data export device, and / or the like. The electronic device 900 may also communicate with one or more external devices (not shown) via the communication unit 940. as desired, an external device, such as a storage device, a display device, and / or the like, in communication with one or more devices enabling a user to interact with the electronic device 900, or in communication with any device that causes the electronic device 900 to communicate with one or more other computing devices (for example, a network card and a modem or the like). Such communication may be performed via an input / output (I / O) interface (not shown).

[0083] In some implementations, in addition to being integrated on a single device, some or all of the various components of the electronic device 900 may also be provided in the form of a cloud computing architecture. In the cloud computing architecture, these components may be arranged remotely and may work together to implement the functionality described in the present disclosure. In some implementations, cloud computing provides computing, software, data access, and storage services that do not require end-user knowledge of the physical location or configuration of the systems or hardware providing these services. In various implementations, cloud computing provides services over a wide area network, such as the Internet, using appropriate protocols. For example, cloud computing providers provide applications over a wide area network, and they may be accessed through a web browser or any other computing component. Software or components of the cloud-computing architecture, and corresponding data, may be stored on a server at a remote location. The computing resources in the cloud computing environment may be consolidated at a remote data center location or they may be dispersed. Cloud computing infrastructures may provide services through shared data centers, even though they appear as a single point of access for a user. Thus, the components and functions described herein may be provided from a service provider at a remote location using a cloud computing architecture. Alternatively, they may be provided from a conventional server, or they may be installed directly or in other ways on the client device.

[0084] The electronic device 900 may be used to implement a computation for a neural network in various implementations of the present disclosure. The memory 920 may include one or more modules having one or more program instructions that may be accessed and executed by theprocessing unit 910 to implement the functionality of the various implementations described herein. For example, the memory 920 may include a neural network computation module 925 for performing a computation of executing a neural network in one or more implementations described above. As shown in FIG. 9, the electronic device 900 may obtain input through the input device 950 that is needed to perform a computation for the neural network, and may provide output through the output device 960 that is needed to perform a computation for the neural network, such as a predicted property. In some implementations, the electronic device 900 may also receive input from other devices (not shown) via the communication unit 940.Example Implementations

[0085] Some example implementations of the present disclosure are listed below.

[0086] In an aspect, the present disclosure provides a computer-implemented method. The method comprises: obtaining a data flow graph for a neural network to be computed, the data flow graph indicating at least one operation in the neural network and data respectively associated with the at least one operation; determining scheduling information for the neural network based on the data flow graph and a processing resource configuration of a target device for computing the neural network, the scheduling information indicating a data transformation required to execute the at least one operation; and executing, at the target device, a computation for the neural network based on the scheduling information.

[0087] In some implementations, the target device comprises a plurality of processing layers, and determining the scheduling information comprises: for a given operation of the at least one operation, determining hint information based on the given operation and respective configurations of the plurality of processing layers, the hint information indicating a range of shapes of data blocks available for the given operation in each of the plurality of processing layers; and determining, based on the hint information, a data transformation to be executed for the given operation between processing layers of the plurality of processing layers.

[0088] In some implementations, the plurality of processing layers comprises a first processing layer and a second processing layer, the first processing layer is above the second processing layer, and determining the data transformation to be executed for the given operation between processing layers of the plurality of processing layers comprises: determining a first shape of a data block for the given operation in the first processing layer based on the hint information and a processing bandwidth of the first processing layer; determining a second shape of a data block for the given operation in the second processing layer based on the hint information, the first shape, and a bandwidth of a storage resource of the second processing layer; and determining, based on the first shape and the second shape, a data transformation to be executed for the given operation between the first processing layer and the second processing layer.

[0089] In some implementations, determining the data transformation to be executed for the given operation between the first processing layer and the second processing layer comprises: determine a target data block to be loaded from the second processing layer to the first processing layer for the given operation; and determining a transformation action for the target data block based on the first shape, the second shape, and a target datatype configured to the first processing layer.

[0090] In some implementations, the transformation action comprises at least one of: a slicing action for slicing a processed data block to obtain a data block with a first predetermined shape, a padding action for padding a boundary' of the processed data block to obtain a data block with a second predetermined shape, a mapping action for transforming an arrangement of elements in the processed data block, or a type conversion action for converting a data type of the processed data block.

[0091] In some implementations, determining the scheduling information comprises: for a given operation of the at least one operation, determining a target memory layer in the target device based on the given operation; and executing, at the target device, the computation for the neural network comprises: storing an output result of executing the given operation to the target memory layer.

[0092] In some implementations, obtaining the data flow graph comprises: determining a tensor expression for each of the at least one operation, the tensor expression indicating input data for the operation and a data ty pe of the input data, and output data for the operation and a data ty pe of the output data; and determining the data flow graph based on a relationship between the tensor expression determined respectively for the at least one operation and the at least one operation.

[0093] In some implementations, executing, at the target device, the computation for the neural network comprises: for a given operation of the at least one operation, loading input data for the given operation from a first memory^ layer of the target device to a computation layer of the target device based on a data transformation indicated by the scheduling information; executing, at the computation layer, the given operation to obtain output data for the given operation; and storing the output data to a second memory layer of the target device for a subsequent operation or output.

[0094] In another aspect, the present disclosure provides an electronic device. The electronic device comprises a processor and a memory7, coupled to the processor and containing instructions stored thereon which, when executed by the processor, cause the device to perform actions comprising: obtaining a data flow graph for a neural network to be computed, the data flow graph indicating at least one operation in the neural network and data respectively associated with the at least one operation; determining scheduling information for the neural network based on the data flow graph and a processing resource configuration of a target device for computing the neural network, the scheduling information indicating a data transformation required to execute the atleast one operation; and executing, at the target device, a computation for the neural network based on the scheduling information.

[0095] In some implementations, the target device comprises a plurality of processing layers, and determining the scheduling information comprises: for a given operation of the at least one operation, determining hint information based on the given operation and respective configurations of the plurality7of processing layers, the hint information indicating a range of shapes of data blocks available for the given operation in each of the plurality of processing layers; and determining, based on the hint information, a data transformation to be executed for the given operation between processing layers of the plurality of processing layers.

[0096] In some implementations, the plurality of processing layers comprises a first processing layer and a second processing layer, the first processing layer is above the second processing layer, and determining the data transformation to be executed for the given operation between processing layers of the plurality of processing layers comprises: determining a first shape of a data block for the given operation in the first processing layer based on the hint information and a processing bandwidth of the first processing layer; determining a second shape of a data block for the given operation in the second processing layer based on the hint information, the first shape, and a bandwidth of a storage resource of the second processing layer; and determining, based on the first shape and the second shape, a data transformation to be executed for the given operation between the first processing layer and the second processing layer.

[0097] In some implementations, determining the data transformation to be executed for the given operation between the first processing layer and the second processing layer comprises: determine a target data block to be loaded from the second processing layer to the first processing layer for the given operation; and determining a transformation action for the target data block based on the first shape, the second shape, and a target datatype configured to the first processing layer.

[0098] In some implementations, the transformation action comprises at least one of: a slicing action for slicing a processed data block to obtain a data block with a first predetermined shape, a padding action for padding a boundary of the processed data block to obtain a data block with a second predetermined shape, a mapping action for transforming an arrangement of elements in the processed data block, or a ty pe conversion action for converting a data type of the processed data block.

[0099] In some implementations, determining the scheduling information comprises: for a given operation of the at least one operation, determining a target memory layer in the target device based on the given operation; and executing, at the target device, the computation for the neural network comprises: storing an output result of executing the given operation to the target memory layer.[OlOOJIn some implementations, obtaining the data flow graph comprises: determining a tensor expression for each of the at least one operation, the tensor expression indicating input data for the operation and a data type of the input data, and output data for the operation and a data type of the output data; and determining the data flow graph based on a relationship between the tensor expression determined respectively for the at least one operation and the at least one operation. [OlOlJIn some implementations, executing, at the target device, the computation for the neural network comprises: for a given operation of the at least one operation, loading input data for the given operation from a first memory layer of the target device to a computation layer of the target device based on a data transformation indicated by the scheduling information; executing, at the computation layer, the given operation to obtain output data for the given operation; and storing the output data to a second memory layer of the target device for a subsequent operation or output.

[0102] In yet another aspect, the present disclosure provides a computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions which, when executed by a device, cause the device to perform actions comprising: obtaining a data flow graph for a neural network to be computed, the data flow graph indicating at least one operation in the neural network and data respectively associated with the at least one operation; determining scheduling information for the neural network based on the data flow graph and a processing resource configuration of a target device for computing the neural network, the scheduling information indicating a data transformation required to execute the at least one operation; and executing, at the target device, a computation for the neural network based on the scheduling information.

[0103] In some implementations, the target device comprises a plurality of processing layers, and determining the scheduling information comprises: for a given operation of the at least one operation, determining hint information based on the given operation and respective configurations of the plurality of processing layers, the hint information indicating a range of shapes of data blocks available for the given operation in each of the plurality of processing layers; and determining, based on the hint information, a data transformation to be executed for the given operation between processing layers of the plurality of processing layers.

[0104] In some implementations, the plurality' of processing layers comprises a first processing layer and a second processing layer, the first processing layer is above the second processing layer, and determining the data transformation to be executed for the given operation between processing layers of the plurality’ of processing layers comprises: determining a first shape of a data block for the given operation in the first processing layer based on the hint information and a processing bandwidth of the first processing layer; determining a second shape of a data block for the given operation in the second processing layer based on the hint information, the first shape, and abandwidth of a storage resource of the second processing layer; and determining, based on the first shape and the second shape, a data transformation to be executed for the given operation between the first processing layer and the second processing layer.

[0105] In some implementations, determining the data transformation to be executed for the given operation between the first processing layer and the second processing layer comprises: determine a target data block to be loaded from the second processing layer to the first processing layer for the given operation; and determining a transformation action for the target data block based on the first shape, the second shape, and a target data type configured to the first processing layer.

[0106] In some implementations, the transformation action comprises at least one of: a slicing action for slicing a processed data block to obtain a data block with a first predetermined shape, a padding action for padding a boundary of the processed data block to obtain a data block with a second predetermined shape, a mapping action for transforming an arrangement of elements in the processed data block, or a type conversion action for converting a data type of the processed data block.

[0107] In some implementations, determining the scheduling information comprises: for a given operation of the at least one operation, determining a target memory7layer in the target device based on the given operation; and executing, at the target device, the computation for the neural network comprises: storing an output result of executing the given operation to the target memory layer.

[0108] In some implementations, obtaining the data flow graph comprises: determining a tensor expression for each of the at least one operation, the tensor expression indicating input data for the operation and a data type of the input data, and output data for the operation and a data type of the output data; and determining the data flow graph based on a relationship between the tensor expression determined respectively for the at least one operation and the at least one operation.

[0109] In some implementations, executing, at the target device, the computation for the neural network comprises: for a given operation of the at least one operation, loading input data for the given operation from a first memory layer of the target device to a computation layer of the target device based on a data transformation indicated by the scheduling information; executing, at the computation layer, the given operation to obtain output data for the given operation; and storing the output data to a second memory' layer of the target device for a subsequent operation or output. [01 lOJIn yet another aspect, the present disclosure provides a computer-readable medium having stored thereon computer-executable instructions which, when executed by a device, cause the device to perform one or more example implementations of the methods of the above aspects.

[0111] The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, example types of hardware logiccomponents that can be used include a field-programmable gate array (FPGA), an applicationspecific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on- a-chip system (SOC), a load programmable logic device (CPLD), and / or the like.

[0112] Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, causes the functions / operations specified in the flowchart and / or block diagrams to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on the remote machine or entirely on the remote machine or server.

[0113] In the context of this disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium may include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory ), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0114] In addition, while operations are depicted in a particular order, this should be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed to achieve desirable results. Multitasking and parallel processing may be advantageous in certain circumstances. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features that are described in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation may also be implemented in multiple implementations separately or in any suitable sub-combination.

[0115] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

CLAIMS1. A computer-implemented method, comprising: obtaining a data flow graph for a neural network to be computed, the data flow graph indicating at least one operation in the neural network and data respectively associated with the at least one operation; determining scheduling information for the neural network based on the data flow graph and a processing resource configuration of a target device for computing the neural network, the scheduling information indicating a data transformation required to execute the at least one operation; and executing, at the target device, a computation for the neural netw ork based on the scheduling information.

2. The method of claim 1, wherein the target device comprises a plurality of processing layers, and determining the scheduling information comprises: for a given operation of the at least one operation, determining hint information based on the given operation and respective configurations of the plurality of processing layers, the hint information indicating a range of shapes of data blocks available for the given operation in each of the plurality of processing layers; and determining, based on the hint information, a data transformation to be executed for the given operation between processing layers of the plurality of processing layers.

3. The method of claim 2, wherein the plurality of processing layers comprises a first processing layer and a second processing layer, the first processing layer is above the second processing layer, and determining the data transformation to be executed for the given operation between processing layers of the plurality of processing layers comprises: determining a first shape of a data block for the given operation in the first processing layer based on the hint information and a processing bandwidth of the first processing layer; determining a second shape of a data block for the given operation in the second processing layer based on the hint information, the first shape, and a bandwidth of a storage resource of the second processing layer; and determining, based on the first shape and the second shape, a data transformation to be executed for the given operation between the first processing layer and the second processing layer.

4. The method of claim 3, wherein determining the data transformation to be executed for the given operation between the first processing layer and the second processing layer comprises: determine a target data block to be loaded from the second processing layer to the first processing layer for the given operation; and determining a transformation action for the target data block based on the first shape, thesecond shape, and a target data type configured to the first processing layer.

5. The method of claim 4, wherein the transformation action comprises at least one of: a slicing action for slicing a processed data block to obtain a data block with a first predetermined shape, a padding action for padding a boundary of the processed data block to obtain a data block with a second predetermined shape, a mapping action for transforming an arrangement of elements in the processed data block, or a type conversion action for converting a data type of the processed data block.

6. The method of claim 1, wherein determining the scheduling information comprises: for a given operation of the at least one operation, determining a target memory layer in the target device based on the given operation; and wherein executing, at the target device, the computation for the neural network comprises: storing an output result of executing the given operation to the target memory layer.

7. The method of claim 1, wherein obtaining the data flow graph comprises: determining a tensor expression for each of the at least one operation, the tensor expression indicating input data for the operation and a data t pe of the input data, and output data for the operation and a data type of the output data; and determining the data flow graph based on a relationship between the tensor expression determined respectively for the at least one operation and the at least one operation.

8. The method of claim 1, wherein executing, at the target device, the computation for the neural network comprises: for a given operation of the at least one operation, loading input data for the given operation from a first memory layer of the target device to a computation layer of the target device based on a data transformation indicated by the scheduling information; executing, at the computation layer, the given operation to obtain output data for the given operation; and storing the output data to a second memory layer of the target device for a subsequent operation or output.

9. An electronic device, comprising: a processing unit; a memory coupled to the processing unit and containing instructions stored thereon which, when executed by the processing unit, cause the device to perform actions comprising: obtaining a data flow graph for a neural network to be computed, the data flow graph indicating at least one operation in the neural network and data respectively associated with the atleast one operation; determining scheduling information for the neural network based on the data flow graph and a processing resource configuration of a target device for computing the neural network, the scheduling information indicating a data transformation required to execute the at least one operation; and executing, at the target device, a computation for the neural network based on the scheduling information.

10. The electronic device of claim 9, wherein the target device comprises a plurality of processing layers, and determining the scheduling information comprises: for a given operation of the at least one operation, determining hint information based on the given operation and respective configurations of the plurality of processing layers, the hint information indicating a range of shapes of data blocks available for the given operation in each of the plurality of processing layers; and determining, based on the hint information, a data transformation to be executed for the given operation between processing layers of the plurality of processing layers.

11. The electronic device of claim 10, wherein the plurality of processing layers comprises a first processing layer and a second processing layer, the first processing layer is above the second processing layer, and determining the data transformation to be executed for the given operation between processing layers of the plurality of processing layers comprises: determining a first shape of a data block for the given operation in the first processing layer based on the hint information and a processing bandwidth of the first processing layer; determining a second shape of a data block for the given operation in the second processing layer based on the hint information, the first shape, and a bandwidth of a storage resource of the second processing layer; and determining, based on the first shape and the second shape, a data transformation to be executed for the given operation between the first processing layer and the second processing layer.

12. The electronic device of claim 11, wherein determining the data transformation to be executed for the given operation between the first processing layer and the second processing layer comprises: determine a target data block to be loaded from the second processing layer to the first processing layer for the given operation; and determining a transformation action for the target data block based on the first shape, the second shape, and a target data type configured to the first processing layer.

13. The electronic device of claim 12, wherein the transformation action comprises at least one of:a slicing action for slicing a processed data block to obtain a data block with a first predetermined shape, a padding action for padding a boundary of the processed data block to obtain a data block with a second predetermined shape, a mapping action for transforming an arrangement of elements in the processed data block, or a type conversion action for converting a data type of the processed data block.

14. The electronic device of claim 9, wherein determining the scheduling information comprises: for a given operation of the at least one operation, determining a target memoiy layer in the target device based on the given operation; and wherein executing, at the target device, the computation for the neural network comprises: storing an output result of executing the given operation to the target memory layer.

15. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions which, when executed by a device, cause the device to perform actions comprising: obtaining a data flow graph for a neural network to be computed, the data flow graph indicating at least one operation in the neural network and data respectively associated with the at least one operation; determining scheduling information for the neural network based on the data flow graph and a processing resource configuration of a target device for computing the neural network, the scheduling information indicating a data transformation required to execute the at least one operation; and executing, at the target device, a computation for the neural netw ork based on the scheduling information.

Images