Cyclic transformation in tensor compiler for deep neural network (DNN)

Abstract

A tensor compiler for DNN may use a trained model to optimize loop nesting in IR. The loop nesting may include loops. The cycle may be nested within another cycle. The loop specifies tensor operations to be repeatedly executed by the processor. A tensor compiler generates a scheduling plan tree for the IR. The scheduling plan tree includes scheduling plans based on hierarchical arrangement. A tensor compiler may select a scheduling plan from the scheduling plan tree by using a trained model that may predict performance of the processor to perform tensor operations according to the IR transformed with the scheduling plan. The tensor compiler then transforms the loop nest with the selected scheduling plan and generates an implementation to be executed by the processor. A tensor compiler may instrumented the implementation to facilitate receiving runtime performance information of the processor. The tensor compiler may use the runtime performance information to further train the model.

Description

Technical Field

[0001] The present disclosure relates generally to neural networks, and more specifically to loop transformations in a tensor compiler for DNNs. Background Art

[0002] DNNs are widely used in various artificial intelligence applications, including computer vision, speech recognition, and natural language processing, due to their ability to achieve high accuracy. However, high accuracy comes at a significant energy cost. DNNs have extremely high computational requirements because each inference may require hundreds of millions of tensor operations, such as convolutions, pooling operations, element-by-element operations, and other types of tensor operations. Therefore, techniques to improve the energy efficiency of DNNs are needed. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To aid this description, similar reference numerals designate similar structural elements. The embodiments are illustrated in the accompanying drawings by way of example and not by way of limitation.

[0004] Figure 1 An example DNN according to various embodiments is illustrated.

[0005] Figure 2 A tensor computing environment according to various embodiments is illustrated.

[0006] Figure 3 is a block diagram of a tensor compiler according to various embodiments.

[0007] Figure 4 is a block diagram of a schedule engine according to various embodiments.

[0008] Figure 5 is a block diagram of a schedule generator according to various embodiments.

[0009] Figure 6 A schedule tree of an intermediate representation (IR) according to various embodiments is illustrated.

[0010] Figure 7 The flowchart of FIG. 1 illustrates a method of loop transformation for deep learning according to various embodiments.

[0011] Figure 8 The flowchart shows a method for generating a scheduling plan tree for an IR according to various embodiments.

[0012] Fig. 9 The flowchart of illustrates another method of generating a scheduling plan tree for an IR according to various embodiments.

[0013] Fig.10A deep learning environment according to various embodiments is illustrated.

[0014] Fig.11 is a block diagram of an example DNN system according to various embodiments.

[0015] Fig.12 is a block diagram of an example computing device in accordance with various embodiments. DETAILED DESCRIPTION

[0016] Overview

[0017] DNNs are widely used in the fields of computer vision, speech recognition, image and video processing, mainly because they can achieve accuracy beyond human level. Tensor computation is the key to deep learning. Tensor computation is often accelerated by vendor-specific experts writing libraries (e.g., OneDNN and CuDNN) for integrated kernels. Another way to accelerate tensor computation is to use microkernel libraries. Compared with integrated kernels, microkernels are more primitive in functionality and are designed to be combined into larger kernels together. Just-In-Time (JIT) compilers have been introduced to optimize DNNs built around microkernels. For example, the PlaidML JIT compiler optimizes DNNs and generates calls to microkernel libraries (i.e., TPP (Tensor Processing Primitive, tensor processing primitives)).

[0018] One problem faced by microkernel-based JIT tensor compilers is how to identify efficient loop transformation sequences that minimize data movement across the memory hierarchy and maximize parallelism within a limited compile time budget. Microkernels focus on optimizing register usage, leaving cache management to the JIT compiler. The compiler often needs to tile a given loop nest with the appropriate tiling factor for each memory (e.g., registers, L1 cache, L2 cache, ..., and last-level cache), match register-level loops with efficient microkernels, order tiled loops at other levels, and parallelize some of the outermost loops so that data reuse and operation intensity are maximized and the many cores of a modern CPU (central processing unit) run in parallel. This process involves multiple loop transformations (e.g., loop tiling, permutation, folding, threading, etc.) and is complicated by the proliferation of possible scheduling schemes and the complexity of modern architectures (e.g., multiple issue, prefetching, cache replacement policies, etc.). Identifying efficient loop transformation sequences is a perpetual challenge even for static compilers, let alone JIT compilers that are constrained by much smaller compile time budgets.

[0019] Polyhedral compilation and analysis models have been used to identify efficient loop transformation sequences. Polyhedral compilers formulate optimization models such as ILP (integer linear programming) models. Cache analysis models are often built for important computations such as matrix multiplication and convolution to guide the selection of loop order and loop range to achieve optimal cache performance. However, solving ILP problems can take too much time, which is not practical for JIT compilers. While coarse-grained analysis modeling can play a role, fine-grained accurate modeling of performance is very difficult in practice due to features of modern architectures such as out-of-order execution and hardware prefetching.

[0020] Autotuning is another approach that has been used to identify sequences of loop transformations. Autotuning can search the parameter space in various ways, such as ridge combing, high variance sampling, etc. Autotuning can also measure the results and identify the best values ​​for the parameters. For example, Apache TVM accepts a tiled loop structure and a specification of the search space, such as permutations of a loop subset, iterates through the tiled loop configuration, generates and runs the code on the hardware, and measures the performance. However, autotuning can take at least several hours, and often several days, which is not directly applicable to JIT compilation.

[0021] In addition, AI (artificial intelligence) models with experience search are also used for loop transformation. With the help of AI models with experience search, the scheduling plan space can be searched by, for example, a tree, where each node is a loop transformation, and the AI ​​model can be queried for predicted performance. Different search algorithms can be applied, such as beam search or Monte Carlo tree search. Various program features, including tensor references, operation counts, load / store bytes, etc., can be used to build AI models. However, the trade-off between model size and accuracy is a challenge faced by JIT compilers. In a JIT environment, small models should be used because of their fast execution, but may lose accuracy. Therefore, there is a need for improved techniques for loop transformation in tensor compilers.

[0022] Embodiments of the present disclosure can improve at least some of the above challenges and problems by using machine learning techniques to identify loop transformation sequences in a tensor compiler (e.g., a microkernel-based JIT tensor compiler). For example, a machine learning model can be used in the present disclosure to predict the performance of a processor when performing tensor calculations and the misses of memory used in the execution, and use the prediction to evaluate the loop transformation sequence and select the optimal loop transformation sequence.

[0023] The present disclosure provides a tensor computing environment in which source code for tensor computing can be converted into IR. IR is a data structure that specifies tensor operations to be performed by a DNN (e.g., performed by one or more layers of a DNN). IR includes loop nests. Loop nests include multiple loops. The loop indicates a tensor operation to be repeatedly performed. The number of times a tensor operation is to be repeatedly performed is the extent of the loop, also known as loop extent or extent. The tensor computing environment includes a tensor compiler, which can use a machine learning model to generate a schedule for the IR, and use the schedule to transform loops in the loop nest. The schedule specifies a loop transformation sequence, which includes one or more loop transformations.

[0024] When the tensor compiler generates a schedule, it may first search in a data repository that stores schedule trees for various IR categories. The schedule tree includes multiple schedules arranged based on their hierarchy. IR categories include IRs with (one or more) the same tensor operations and the same tensor references. But the loop extensions in IRs of the same category may be different. The tensor compiler determines whether the IR belongs to any IR category. In an embodiment where the IR belongs to a certain IR category, the tensor compiler can generate a schedule tree for the IR based on one or more schedule trees of the IR category. In an embodiment where the IR does not belong to a certain IR category, the tensor compiler can generate a schedule tree for the IR from scratch. When generating the schedule tree, the tensor compiler can use a first trained model to predict memory misses and determine the loop extension based on the predicted memory misses.

[0025] After generating the schedule tree, the tensor compiler uses a second trained model to select the optimal schedule to be used to transform the loop nest from the schedule tree. The second trained model can predict the performance of the processor performing the tensor calculation based on the IR transformed with the schedule. For each schedule in the schedule tree, the tensor compiler can input one or more parameters associated with each schedule into the trained model, and the trained model can output a performance score indicating an evaluation of the predicted performance of the processor. The tensor compiler can select the schedule that provides the best predicted performance of the processor. In addition, the tensor compiler transforms the loop nest with the selected schedule to generate an implementation. The tensor compiler can guide the implementation so that after the processor performs the tensor calculation, the runtime performance information of the processor can be provided to the tensor compiler. The tensor compiler can use the selected schedule and runtime performance information to further train the second trained model.

[0026] By incorporating machine learning models into the compilation process, the present disclosure provides an AI-assisted solution for transforming loops in a microkernel-based JIT tensor compiler. The present disclosure can combine the functions of analytical modeling, AI modeling, and on-demand recompilation to produce efficient loop transformations within a limited time budget. Compared with traditional tensor compiler technology, the tensor compiler in the present disclosure has more advantages. For example, the tensor compiler in the present disclosure can utilize existing IR-like scheduling trees by searching for scheduling trees in a data repository. By using a machine learning model that can predict processor performance and memory misses, the tensor compiler can efficiently narrow the scheduling space in the process of generating a scheduling tree. The generated scheduling tree can be fed back to the data repository to expand the scheduling pool for future searches. In addition, the tensor compiler promotes the continuous training of machine learning models, so that the accuracy of machine learning models can be continuously improved.

[0027] For purposes of illustration, specific numbers, materials, and configurations are described to provide a thorough understanding of the illustrative implementations. However, it will be apparent to one skilled in the art that the present disclosure may be implemented without the specific details, or / and may be implemented with only some of the described aspects. In other cases, well-known features are omitted or simplified so as not to obscure the illustrative implementations.

[0028] In addition, reference is made to the accompanying drawings, which form a part of this document, in which embodiments that can be implemented are shown by way of illustration. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description should not be understood in a limiting sense.

[0029] Various operations may be described in turn as a plurality of discrete actions or operations in a manner that is most helpful for understanding the claimed subject matter. However, the order of description should not be interpreted as meaning that these operations must be dependent on the order. Specifically, these operations may not be performed in the order presented. The operations described may be performed in an order different from the described embodiments. Various additional operations may be performed or the operations described may be omitted in additional embodiments.

[0030] For purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B). For purposes of this disclosure, the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B and C). The term "between" when used in reference to a measurement range includes both ends of the measurement range.

[0031] The description uses the phrases "in one embodiment" or "in an embodiment", each of which refers to one or more identical or different embodiments. The terms "including", "comprising", "having", etc., used for the embodiments of the present disclosure, are synonymous. The present disclosure may use perspective-based descriptions, such as "above", "below", "top", "bottom", and "side", to illustrate various features of the drawings, but these terms are only for ease of discussion and do not imply a desired or required orientation. The drawings are not necessarily drawn to scale. Unless otherwise indicated, the use of ordinal adjectives "first", "second", and "third", etc. to describe common objects simply indicates that different instances of similar objects are referenced, and is not intended to imply that the objects described in this way must be in a given sequence in time, space, ranking, or in any other way.

[0032] In the detailed description that follows, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0033] The terms "substantially," "close," "roughly," "near," and "approximately" generally refer to input operating objects based on specific values ​​as described herein or as known in the art, within + / -20% of target values. Similarly, terms indicating the orientation of various elements, such as "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between elements, generally refer to input operating objects based on specific values ​​as described herein or as known in the art, within + / -5-20% of target values.

[0034] Furthermore, the terms "includes," "comprising," "having," or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, apparatus, or DNN accelerator that includes a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such method, process, apparatus, or DNN accelerator. Additionally, the term "or" refers to an inclusive "or" rather than an exclusive "or."

[0035] The DNN systems, methods, and devices of the present disclosure each have multiple innovative aspects, no single one of which is entirely responsible for all of the desirable properties disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the following description and accompanying drawings.

[0036] Example DNN

[0037] Figure 1 An example DNN 100 is illustrated in accordance with various embodiments. For illustration, Figure 1 The DNN 100 in FIG. 1 is a convolutional neural network (CNN). In other embodiments, the DNN 100 may be another type of DNN. The DNN 100 is trained to receive an image and output a classification of an object in the image. Figure 1In an embodiment of the invention, DNN 100 receives an input image 105 including objects 115, 125, and 135. DNN 100 includes a sequence of layers including a plurality of convolutional layers 110 (individually referred to as "convolutional layer 110"), a plurality of pooling layers 120 (individually referred to as "pooling layer 120"), and a plurality of fully connected layers 130 (individually referred to as "fully connected layer 130"). In other embodiments, DNN 100 may include fewer, more, or different layers. In inference of DNN 100, the layers of DNN 100 perform tensor computations including a number of tensor operations such as convolutions (e.g., multiply-accumulate (MAC) operations, etc.), pooling operations, element-wise operations (e.g., element-wise addition, element-wise multiplication, etc.), other types of tensor operations, or some combination thereof.

[0038] Convolutional layer 110 summarizes the presence of features in input image 105. Convolutional layer 110 functions as a feature extractor. The first layer of DNN 100 is convolutional layer 110. In one example, convolutional layer 110 performs convolution on input tensor 140 (also referred to as input feature map (IFM) 140) and filter 150. Figure 1 As shown, IFM 140 is represented by a 7×7×3 three-dimensional (3D) matrix. IFM 140 includes three input channels, each of which is represented by a 7×7 two-dimensional (2D) array. The 7×7 2D array includes 7 input elements (also called input points) per row and 7 input elements per column. Filter 150 is represented by a 3×3×3 3D matrix. Filter 150 includes three kernels, each of which may correspond to a different input channel of IFM 140. The kernel is a 2D weight array, where the weights are arranged in columns and rows. The kernel may be smaller than the IFM. Figure 1 In the embodiment of , each kernel is represented by a 3×3 2D array. The 3×3 kernel includes 3 weights per row and 3 weights per column. The weights can be initialized and updated by back propagation using gradient descent. The magnitude of the weights can indicate the importance of the filter 150 in extracting features from the IFM 140.

[0039] The convolution includes a MAC operation using the input elements in the IFM 140 and the weights in the filter 150. The convolution can be a standard convolution 163, or it can be a depthwise convolution 183. In the standard convolution 163, the entire filter 150 slides on the IFM 140. All input channels are combined to produce an output tensor 160 (also called OFM 160). OFM 160 is represented by a 5×5 2D array. The 5×5 2D array includes 5 output elements (also called output points) per row and 5 output elements per column. For illustration, the standard convolution is Figure 1 In embodiments where there are multiple filters, standard convolution may produce multiple output channels in OFM 160.

[0040] The multiplication applied between the kernel-sized patch of IFM 140 and the kernel can be a dot product. The dot product is an element-by-element multiplication between the kernel-sized patch of IFM 140 and the corresponding kernel, followed by addition, always producing a single value. Because its result is a single value, this operation is often referred to as a "scalar product". Using a kernel smaller than IFM 140 is intentional because it allows the same kernel (a set of weights) to be multiplied with IFM 140 multiple times at different points on IFM 140. Specifically, the kernel is systematically applied to each overlapping portion or kernel-sized patch of IFM 140, from left to right and from top to bottom. The result of multiplying the kernel with IFM 140 once is a single value. Since the kernel is applied multiple times to IFM 140, the multiplication result is a 2D array of output elements. In this way, the 2D output array (i.e., OFM 160) from standard convolution 163 is called OFM.

[0041] In the depthwise convolution 183, the input channels are not combined. Instead, a MAC operation is performed on the individual input channels and the individual kernels, and the MAC operation produces an output channel. Figure 1As shown, the depthwise convolution 183 generates a depthwise output tensor 180. The depthwise output tensor 180 is represented by a 5×5×3 3D matrix. The depthwise output tensor 180 includes three output channels, each of which is represented by a 5×5 2D array. The 5×5 2D array includes 5 output elements per row and 5 output elements per column. Each output channel is the result of a MAC operation of the input channel of the IFM 140 and the kernel of the filter 150. For example, the first output channel (dot pattern) is the result of a MAC operation of the first input channel (dot pattern) and the first kernel (dot pattern), the second output channel (horizontal stripe pattern) is the result of a MAC operation of the second input channel (horizontal stripe pattern) and the second kernel (horizontal stripe pattern), and the third output channel (diagonal stripe pattern) is the result of a MAC operation of the third input channel (diagonal stripe pattern) and the third kernel (diagonal stripe pattern). In such a depthwise convolution, the number of input channels is equal to the number of output channels, and each output channel corresponds to a different input channel. The input channels and output channels are collectively referred to as depth-wise channels. After the depth-wise convolution, a point-wise convolution 193 is then performed on the depth-wise output tensor 180 and the 1×1×3 tensor 190 to produce the OFM 160 .

[0042] The OFM 160 is then passed to the next layer in the sequence. In some embodiments, the OFM 160 is passed through an activation function. An example activation function is a rectified linear activation function (ReLU). ReLU is a calculation that directly returns the value provided as an input, or returns a value of zero if the input is zero or less. The convolution layer 110 can receive several images as input and calculate the convolution of each of them with each kernel. This process can be repeated multiple times. For example, the OFM 160 is passed to a subsequent convolution layer 110 (i.e., a convolution layer 110 after the convolution layer 110 that generates the OFM 160 in the sequence). The subsequent convolution layer 110 performs convolution on the OFM 160 with a new kernel and generates a new feature map. The new feature map can also be normalized and resized. The new feature map can be subjected to a kernel operation again by a further subsequent convolution layer 110, and so on and so forth.

[0043] In some embodiments, the convolutional layer 110 has four hyperparameters: the number of kernels, the size F of the kernel (e.g., the size of the kernel is F×F×D pixels), the stride S of dragging the window corresponding to the kernel on the image (e.g., a stride of one means moving the window one pixel at a time), and zero padding P (e.g., adding a black outline of P pixels thick to the input image of the convolutional layer 110). The convolutional layer 110 can perform various types of convolutions, such as 2-dimensional convolutions, dilated or atrous convolutions, spatially separable convolutions, depthwise separable convolutions, transposed convolutions, and the like. The DNN 100 includes 16 convolutional layers 110. In other embodiments, the DNN 100 may include a different number of convolutional layers.

[0044] The pooling layer 120 downsamples the feature map generated by the convolution layer, for example, by summarizing the presence of features in patches of the feature map. The pooling layer 120 is placed between two convolutional layers 110: the preceding convolutional layer 110 (the convolutional layer 110 before the pooling layer 120 in the sequence of layers) and the subsequent convolutional layer 110 (the convolutional layer 110 after the pooling layer 120 in the sequence of layers). In some embodiments, the pooling layer 120 is added after the convolutional layer 110, for example, after the activation function (e.g., ReLU) is applied to the OFM 160.

[0045] The pooling layer 120 receives the feature map generated by the previous convolutional layer 110 and applies a pooling operation to the feature map. The pooling operation reduces the size of the feature map while retaining their important characteristics. Therefore, the pooling operation improves the efficiency of the DNN and avoids over-learning. The pooling layer 120 can perform a pooling operation by average pooling (calculating the average value for each patch on the feature map), maximum pooling (calculating the maximum value for each patch of the feature map), or a combination of the two. The size of the pooling operation is smaller than the size of the feature map. In various embodiments, the pooling operation is 2×2 pixels applied with a span of two pixels, so that the pooling operation reduces the size of the feature map by 2 times, for example, the number of pixels or values ​​in the feature map is reduced to one-fourth the size. In one example, the pooling layer 120 applied to a 6×6 feature map results in a 3×3 output pooled feature map. The output of the pooling layer 120 is input into the subsequent convolutional layer 110 to further extract features. In some embodiments, the pooling layer 120 operates on each feature map separately to create a new set of pooled feature maps of the same number.

[0046] The fully connected layer 130 is the last layer of the DNN. The fully connected layer 130 can be convolutional or non-convolutional. The fully connected layer 130 receives an input operation object. The input operation object defines the output of the convolution layer 110 and the pooling layer 120, and includes the value of the last feature map generated by the last pooling layer 120 in the sequence. The fully connected layer 130 applies linear combinations and activation functions to the input operation objects and generates individual partial sums. The individual partial sums may contain as many elements as categories: element i represents the probability that the image belongs to category i. Therefore, each element is between 0 and 1, and the sum of all values ​​is one. These probabilities are calculated by the last fully connected layer 130 using a logistic function (binary classification) or a softmax function (multi-class classification) as an activation function.

[0047] In some embodiments, the fully connected layer 130 classifies the input image 105 and returns an operand of size N, where N is the number of classes in the image classification problem. Figure 1 In the embodiment of , N is equal to 3 because there are three objects 115, 125 and 135 in the input image. Each element of the operation object indicates the probability that the input image 105 belongs to a certain category. To calculate the probability, the fully connected layer 130 multiplies each input element by the weight, obtains the sum, and then applies an activation function (for example, if N=2, it is a logical function, if N>2, it is a softmax function). This is equivalent to multiplying the input operation object with a matrix containing weights. In one example, the individual partial sums include three probabilities: the first probability indicates that the object 115 is a tree, the second probability indicates that the object 125 is a car, and the third probability indicates that the object 135 is a person. In other embodiments where the input image 105 includes different objects or a different number of objects, the individual partial sums can be different.

[0048] Example tensor computing environment

[0049] Figure 2 A tensor computing environment 200 according to various embodiments is illustrated. The tensor computing environment 200 provides an environment in which tensor computations in some or all layers of a DNN may be performed, such as Figure 1 The DNN 100 in FIG. 100 may include convolution, pooling operations, element-by-element operations (e.g., element-by-element addition, element-by-element multiplication, etc.), loading, reduction, other types of tensor operations of the DNN, or some combination thereof. Figure 2As shown, the tensor computing environment 200 includes a programming module 210, a conversion module 220, a tensor compiler 230, an abstract module 240, a runtime module 250, and a processor 260. The programming module 210, the conversion module 220, the tensor compiler 230, and the abstract module 240 may be implemented at least in part in software. The processor 260 may be implemented at least in part in hardware. In other embodiments, alternative configurations, different components, or additional components may be included in the tensor computing environment 200. In addition, the functionality of a component of the tensor computing environment 200 may be performed by different components included in the tensor computing environment 200 or by different systems.

[0050] The programming module 210 facilitates the generation of source code 215. The source code 215 is a set of computer program instructions in a human-readable form. The computer instructions will be executed by the processor 260 to perform tensor calculations. In some embodiments, the source code 215 is written in a high-level programming language, such as C, C++, Java, Python, etc. The source code 215 cannot be directly executed by the processing device, but needs to be converted into machine code executable by the processing device.

[0051] In one embodiment, the programming module 210 provides a programming environment in which a user can write computer program instructions using one or more programming languages. The programming languages ​​supported by the programming module 210 may be human-readable programming languages, such as C, C++, Java, Python, and the like. In another embodiment, the programming module 210 may receive source code 215 from another system or device. For example, the programming module 210 may retrieve source code 215 from a library of a deep learning framework. For another example, the programming module 210 may retrieve source code 215 from a memory associated with the DNN. For another example, the programming module 210 may retrieve source code 215 from a computer device communicating with the programming module 210.

[0052] The conversion module 220 receives the source code 215 and converts the source code 215 into an IR 225. The IR 225 is a data structure representing the source code 215. The IR 225 includes a loop nest indicating one or more tensor operations that the processor 260 is to repeatedly perform. The loop nest includes a sequence of loops, where one loop is inside one or more other loops, and these other loops are located after the loop in the sequence. The first loop in the sequence may be the innermost loop, and the last loop in the sequence may be the outermost loop. The loop may include a sequence of programming instructions that is specified only once but can be executed multiple times in succession. The loop may indicate the tensor operations that the processor 260 is to repeatedly perform. The number of times the tensor operation is to be performed is the extension of the loop. A loop with an extension equal to 1 is a unit loop. Tensor operations can be used for tensors of the same dimension.

[0053] IR 225 may include information indicating the attributes of loop nests. The attributes of loop nests may include tensor operations to be repeatedly performed, tensor references, loop extension, other attributes of loop nests, or some combination thereof. Tensor operations may be convolutions, pooling operations, element-by-element operations, tensor reductions, tensor loads, other tensor operations, or some combination thereof. Tensor references may include tensor order (i.e., the number of dimensions of a tensor, such as 1, 2, 3, etc.), tensor shape (i.e., the number of elements in each dimension of a tensor), tensor length (the total number of elements in a tensor), other tensor references, or some combination thereof. IR 225 may include code indicating other attributes of loop nests.

[0054] IR 225 facilitates further processing, such as loop transformations. IR 225 may be in the form of a data structure, such as an in-memory data structure, a special tuple-based code, a stack-based code, or other forms. Compared to source code 215, IR 225 may be in a form more suitable for code improvement transformations before being used to generate machine code for a target device (e.g., a processing device). Transformation module 220 may be part of a programming framework (e.g., Keras, TensorFlow, OpenVino, etc.).

[0055] The tensor compiler 230 receives the IR 225 and optimizes the IR 225. For example, the tensor compiler 230 can optimize the IR 225 by transforming the loop nests in the IR 225. The tensor compiler 230 can perform various types of loop transformations, such as loop permutation, index rewriting, loop unrolling, loop splitting, loop tiling, loop filling, other types of loop transformations, or some combination thereof. Loop permutation can change the order of loops in a loop nest. Index rewriting can change the expression of loop indexes. Loop unrolling can create one or more copies of the loop body and modify the loop index appropriately. Loop splitting can divide a loop with multiple operations into multiple separate loops, each operation corresponding to a different separate loop. Loop fusion can merge multiple loops into one loop, and the new loop contains the operations of multiple loops. Loop tiling can split a loop into nests of loops, each inner loop working on a small piece of data of the original loop. Loop filling can add data elements to an array to change the way the array is mapped to the memory system structure. Loop transformation can increase execution speed and reduce the overhead associated with the loop. Optimization through loop transformation can improve cache performance and effectively utilize parallel processing capabilities.

[0056] The tensor compiler 230 may use a schedule to transform a loop nest. A schedule may specify a sequence of loop transformations. In some embodiments, the tensor compiler 230 generates a schedule tree for the loop nest. The schedule tree includes schedules arranged based on their hierarchy. The schedule tree includes a root, which has the highest hierarchy. The root may be IR 225. The root may be the parent of one or more nodes having the second highest hierarchy. A node at the second level may be the parent of one or more nodes at the third level. The schedule tree may have two or more levels. Each node in the schedule tree is a schedule. More details about the schedule tree will be described below in conjunction with Figure 6 Give a description.

[0057] In some embodiments, the tensor compiler 230 maintains a database, for example by caching, that includes scheduling trees generated or used for previous IRs. Deep learning is an important application area for the tensor compiler 230. Many DNNs include similar or even identical layers that perform the same tensor operations. The tensor compiler 230 can run a similarity search in the database to determine whether the IR is the same or similar to any previous IR. In embodiments where the tensor compiler 230 finds a matching (i.e., the same or similar) IR in the database, the tensor compiler 230 can use an already created scheduling tree for the matching IR to generate a scheduling tree for the IR, which can be more time and resource efficient than generating a scheduling tree from scratch.

[0058] After the tensor compiler 230 generates the schedule tree, the tensor compiler 230 selects a schedule from the schedule tree that can trigger the best predicted performance of the processor 260 when performing tensor operations. The tensor compiler 230 can use the trained model to predict the performance of the processor 260. The tensor compiler 230 can use the schedule to transform loop nests. In embodiments where the schedule is at the third level or lower, one or more other schedules that are ancestors (e.g., father, grandfather, great-grandfather, etc.) of the selected schedule in the schedule tree are also used to transform loop nests. The schedule and one or more other schedules constitute a schedule sequence that specifies a loop transformation sequence.

[0059] After the tensor compiler 230 selects a schedule, the tensor compiler 230 implements the schedule (or the schedule plus one or more ancestors of the schedule) to transform the loop nest. The result of the transformation is an implementation 235. In some embodiments, the tensor compiler 230 also instruments the implementation 235 so that when the implementation 235 runs on the processor 260, information indicating the performance of the processor 260 can be generated, for example, from the runtime module 250 and provided to the tensor compiler 230. The tensor compiler 230 can use the performance information to determine a runtime performance score that indicates an assessment of the runtime performance of the processor 260 when performing tensor operations using the implementation 235. The selected schedule and runtime performance score can be used to further train the trained model. Because the tensor compiler 230 continuously selects schedules and determines the runtime performance scores of the schedules, the tensor compiler 230 can continuously train the trained model. More details about the tensor compiler 230 will be described below in conjunction with Figure 3-Figure 5 Give a description.

[0060] The abstract module 240 is located between the tensor compiler 230 and the processor 260. The abstract module 240 allows the tensor compiler 230 to interact with the processor 260 at a general and abstract level, rather than at a detailed hardware level. The abstract module 240 may include a hardware abstraction layer. The abstract module 240 converts the implementation 235 into an abstract implementation 245. In some embodiments, the abstract module 240 may replace one or more loops in the implementation 235 with a microkernel in the abstract implementation 245. For example, the abstract module 240 may replace one or more loops at the innermost level of the loop nest with a microkernel. The innermost level loop may be a loop corresponding to the highest level of memory, such as a register of the processor 260. The microkernel may be a virtual instruction. The abstract module 240 may hide the differences in the hardware of the processor 260, so that it can run on a processing device with different hardware without changing the code.

[0061] The runtime module 250 facilitates the processor 260 to perform tensor operations and provides an environment in which the abstract implementation 245 runs. The runtime refers to the period when the processor 260 performs tensor operations. The runtime module 250 can solve multiple problems related to the processor 260 performing tensor operations, such as memory management, variable access, interface with the operating system, etc. The runtime module 250 can implement the microkernel in the abstract implementation 245 and generate an implemented implementation 255 that can be executed by the processing device 360. The implemented implementation 255 can be a machine code. In some embodiments, the runtime module 250 includes a library.

[0062] Processor 260 performs tensor operations on input tensor 263 according to implemented implementation 255 and generates output tensor 265. An example of input tensor 263 is Figure 1 An example of an output tensor 265 is Figure 1 OFM 160 in. Processor 260 may constitute one or more layers of a DNN, an example of which is Figure 1 DNN 100 in. Processor 260 includes hardware components that can perform tensor computations. In one embodiment, processor 260 includes multiple processing elements that can perform MAC operations, pooling operations, element-by-element operations, other types of deep learning operations, or some combination thereof. The processing elements can be arranged in one or more tiles. Each tile may include an array of processing elements, wherein the processing elements are arranged in rows and columns. Processor 260 may also include one or more memories, such as registers, caches (e.g., L0 cache, L1 cache, L2 cache, etc.). Data used or generated by processor 260, such as input tensors 263 and output tensors 265, may be stored in some or all of the memories. Although Figure 2 One processor is shown, but in other embodiments, the tensor computing environment 200 may include multiple processors.

[0063] Example Tensor Compiler

[0064] Figure 3 2 is a block diagram of a tensor compiler 230 according to various embodiments. The tensor compiler 230 may be a JIT compiler, such as a microkernel-based JIT compiler, and may quickly obtain a scheduling plan for a transformation loop. Figure 3 As shown, the tensor compiler 230 includes a scheduling engine 310, an implementation module 320, an instrumentation module 330, a performance evaluator 340, and a scheduling update module 350. In other embodiments, alternative configurations, different components, or additional components may be included in the tensor compiler 230. In addition, the functionality attributed to a component of the tensor compiler 230 may be performed by different components included in the tensor compiler 230 or by different systems.

[0065] The scheduling engine 310 generates a scheduling tree for the IR. The scheduling engine 310 can start the generator of the scheduling tree for the IR by searching in the scheduling database. The scheduling database includes scheduling trees associated with one or more IR categories. IR categories include IRs with (one or more) the same tensor operations and the same tensor references. But the loop extensions in IRs of the same category may be different. The scheduling engine 310 determines whether the IR belongs to any IR category in the scheduling database. In response to determining that the IR belongs to a certain IR category, the scheduling engine 310 retrieves one or more scheduling trees associated with the IR category, and uses these scheduling trees to generate a scheduling tree for the IR. The scheduling engine 310 can merge the scheduling trees into a merged tree. The scheduling engine 310 can also modify the merged tree to obtain the scheduling tree for the IR.

[0066] In response to determining that the IR does not belong to any IR category in the database, the scheduling engine 310 can generate a scheduling tree for the IR from scratch. The scheduling engine 310 can partition the loop nest in the IR into multiple loop nests, for example, by permutation. Each of the multiple loop nests corresponds to a different memory level. For example, the multiple loop nests may include register loop nests, L1 cache loop nests, L2 cache loop nests..., last level cache loop nests, and main memory loop nests. The scheduling engine 310 can further modify each loop nest, for example, by changing the order of the loops in the loop nests, adjusting the extension of the loops in the loop nests, or a combination of the two. The scheduling engine 310 can generate a scheduling tree based on the permutation and adjusted loop extension.

[0067] The scheduling engine 310 further selects a schedule from the scheduling tree. To select a schedule, the scheduling engine 310 may determine the predicted performance scores of all schedules in the scheduling tree and select the schedule with the highest predicted performance score. For example, for each schedule in the scheduling tree, the scheduling engine 310 determines the properties of the loop nest after the loop nest is transformed with the schedule and inputs the properties into the trained model. The trained model outputs a predicted performance score that indicates an assessment of the predicted performance of the processor 260 when performing tensor operations using the IR 225 compiled with the schedule. More details about the scheduling engine 310 will be described below in conjunction with Figure 4 and Figure 5 Give a description.

[0068] The implementation module 320 transforms the loop nests in the IR according to the schedule obtained by the scheduling engine 310. The loop transformation can improve the performance of the processor 260 when performing tensor operations. For example, the loop transformation can increase the execution speed of the processor 260, thereby shortening the execution time. It can also reduce the overhead associated with the loop and effectively utilize parallel processing capabilities. It also plays an important role in improving the performance of memory (such as registers and cache memory). Through the transformation, the implementation module 320 generates an implementation, such as implementation 235.

[0069] The plug module 330 plugs the implementation generated by the implementation module 320 to facilitate the generation of performance information. In some embodiments, the plug module 330 adds one or more instructions in the implementation 235. These instructions, when executed by the processor 260, cause information indicating the performance of the processor 260 when performing tensor calculations to be sent to the tensor compiler 230, for example, to the performance evaluator 340. The performance information may include information indicating the time taken by the processor 260 to perform tensor calculations, information indicating memory misses (e.g., cache misses), other information indicating the performance of the processor 260, or some combination thereof. The performance information may be runtime performance information, which may be provided by the runtime module 250.

[0070] The performance evaluator 340 uses the performance information to evaluate the performance of the processor 260 when performing tensor operations. The performance evaluator 340 can determine a runtime performance score indicating the runtime performance of the processor 260 when executing. In some embodiments, the performance evaluator 340 can determine the runtime performance score by summarizing one or more scores. One or more scores can indicate one or more of the execution speed, memory misses, memory utilization, etc. of the processor 260. The score may have a weight. The runtime performance score can be a weighted summary of the score (e.g., a weighted sum or average). The performance evaluator 340 can also compare the runtime performance score with the predicted performance score of the scheduling plan. In an embodiment where the difference between the runtime performance score and the predicted performance score exceeds a threshold, the performance evaluator 340 can request further training of the trained model (e.g., performance predictor 450), and the runtime performance score and the scheduling plan can be provided to a training module (e.g., training module 460) to further train the trained model.

[0071] In some embodiments (e.g., embodiments in which the scheduling engine 310 generates a scheduling tree based on memory misses predicted by a trained model), the performance evaluator 340 may also determine a runtime miss score indicating a runtime memory miss in execution. In some embodiments, the performance evaluator 340 may determine a runtime miss score by summing up one or more memory miss scores. The memory miss score may indicate a miss of a memory associated with the processor 260. The scheduling engine 310 may assign different weights to different memories (e.g., the weight of an L1 cache may be higher than the weight of an L2 cache), and determine a weighted summary (e.g., a weighted sum or average) of the memory miss scores. The performance evaluator 340 may also compare the runtime miss score with the predicted miss score of the scheduling plan. In embodiments in which the difference between the runtime miss score and the predicted miss score exceeds a threshold, the performance evaluator 340 may request further training of the trained model, and may provide the runtime miss score and the scheduling plan to a training module (e.g., training module 570) to further train the trained model.

[0072] The scheduling plan update module 350 can change the scheduling plan selected by the scheduling plan engine 310 for the IR to a different scheduling plan. In some embodiments (such as embodiments in which the trained model is further trained), the scheduling plan update module 350 can use the further trained model to select different scheduling plans from the scheduling plan tree. The scheduling plan update module 350 can also request the implementation module 320 to use different scheduling plans to transform loop nests and generate different implementations. The processor 260 can use different implementations to perform tensor operations in future tensor calculations. The reselection of the scheduling plan update module 350 may be beneficial, especially for reasoning workloads that are repeated many times. If the current scheduling plan is no longer predicted to be the best scheduling plan, it is worthwhile to select and implement a better scheduling plan. Over time, the IR can be optimized and become more and more efficient.

[0073] Figure 4 4 is a block diagram of a scheduling engine 310 according to various embodiments. The scheduling engine 310 includes a search module 410, a scheduling data repository 420, a scheduling generator 430, a scheduling selector 440, a performance predictor 450, and a training module 460. In other embodiments, alternative configurations, different components, or additional components may be included in the scheduling engine 310. In addition, the functions attributed to a component of the scheduling engine 310 may be performed by different components included in the scheduling engine 310 or by different systems.

[0074] The search module 410 searches the schedule data repository 420 for a schedule that can be used for the target IR. An example of a target IR is Figure 2 225 in the scheduling plan data repository 420. The scheduling plan data repository 420 stores multiple IR categories. An IR category is a category of IRs that include the same tensor operations and the same tensor references. But the loop extensions in IRs of the same category may be different. Each IR category in the scheduling plan data repository 420 corresponds to one or more scheduling plan trees, which are also stored in the scheduling plan data repository 420. The scheduling plan tree of an IR category can be a scheduling plan tree that has been used to optimize the IR in the IR category (or a scheduling plan tree that has been proven to be effective for optimization).

[0075] The search module 410 determines whether the target IR belongs to any IR category in the schedule data repository 420. For example, the search module 410 determines tensor operations and tensor references in the target IR. Then, the search module 410 determines whether the tensor operations and tensor references in the target IR match the tensor operations and tensor references in any IR category. In response to determining that the tensor operations in the target IR match the tensor operations and tensor references in the IR category, the search module 410 determines that the target IR belongs to a certain IR category.

[0076] In addition, the search module 410 retrieves the scheduling plan tree of the IR category and generates a scheduling plan tree for the target IR based on the retrieved scheduling plan tree. In some embodiments, the search module 410 uses all retrieved scheduling plan trees as candidate scheduling plan trees for the target IR. In other embodiments, the search module 410 uses a subset of the retrieved scheduling plan trees as candidate scheduling plan trees for the target IR. For example, the search module 410 can determine a similarity score that indicates the degree of similarity between the target IR and the IR of the retrieved scheduling plan tree, for example, based on a comparison of one or more loop extensions in the target IR with one or more corresponding loop extensions in the IR. In response to determining that the similarity score is lower than a threshold similarity score or lower than the similarity scores of some or all other retrieved scheduling plan trees, the search module 410 can remove the scheduling plan tree and use other retrieved scheduling plan trees as candidate scheduling plan trees for the target IR.

[0077] In some embodiments (e.g., embodiments in which the search module 410 obtains multiple candidate scheduling plan trees for a target IR), the search module 410 merges the candidate scheduling plan trees into a merged scheduling plan tree for the target IR. The search module 410 may use the target IR as the root of the merged scheduling plan tree and each candidate scheduling plan tree as a branch of the root. The target IR and each candidate scheduling plan tree have a parent-child relationship. Within an individual candidate scheduling plan tree that includes multiple scheduling plans, these scheduling plans may have a parent-child relationship or a sibling relationship. The search module 410 may assign different priorities to the candidate scheduling plan trees based on their similarity scores. Candidate scheduling plan trees with higher similarity scores (i.e., the IR of the candidate scheduling plan tree is more similar to the target IR) may have a higher priority in the merged scheduling plan tree.

[0078] After the candidate schedule trees are merged, the search module 410 can modify or remove incompatible schedules in the merged schedule tree. For example, the search module 410 can identify a schedule for loop tiling and determine whether the tiling factor of the schedule is compatible with the target IR, for example, by determining whether the result of dividing the loop size in the target IR by the tiling factor is an integer. The search module 410 can identify a schedule based on determining that the IR of the schedule is not the same as the target IR, for example, the loop size of the IR is different from the loop size of the target IR. In response to determining that the tiling factor of the schedule is incompatible with the target IR (for example, some results of the division are not integers), the search module 410 can remove the schedule from the merged schedule tree.

[0079] Alternatively, the search module 410 may modify one or more loops that are incompatible with the schedule. For example, the search module 410 adjusts the tiling factor of the incompatible loop so that the loop tiling is compatible with the loop, for example, so that the result of dividing the loop extension by the tiling factor is an integer. In some embodiments, for example, in embodiments where the loop tiling schedule is incompatible with multiple loops, the search module 410 may modify multiple loops in the loop nest. The search module 410 may start with the innermost loop. The innermost tiled loop may be the loop corresponding to the highest memory hierarchy (e.g., registers). The search module 410 may determine an appropriate loop extension for the innermost tiled loop under consideration so that the performance of the register level microkernel (one or more) may be maintained optimally. Next, the search module 410 may determine an appropriate extension for the L1 cache tiled loop under consideration so that L1 cache misses are minimized, and so on. The modification process may use the heat map and cache miss predictor described below. After modifying or removing the incompatible schedule (if any), the search module 410 may output a schedule tree for the target IR.

[0080] In an embodiment where the search module 410 determines that the target IR does not belong to any IR category in the schedule data repository 420, the search module 410 may request the schedule generator 430 to generate a schedule tree for the target IR from scratch. The schedule generator 430 may generate a schedule tree starting from loop tiling. For example, for all memories associated with the processor 260, the schedule generator 430 partitions the loop nests in the target IR into multiple memory loop nests. Each memory loop nest corresponds to a different memory. For the memory loop nests, the schedule generator 430 may determine one or more permutations for changing the order of the loops in the memory loop nests, and determine the extension of the loops. Then, the schedule generator 430 may generate a schedule tree based on the permutations and loop extensions.

[0081] After the schedule generator 430 generates a schedule tree for the target IR, the schedule generator 430 may also create a new IR category and store the new IR category and schedule tree in the schedule data repository 420. The IR category has the same tensor operations and tensor references as the target IR. However, unlike the loop extension of the target IR, which is a specific number, the corresponding loop extension of the IR category is a range that includes the specific number. For example, the loop extension in the target IR is 16, but the IR category covers IRs with loop extensions in the range of 1 to 64. More details about the schedule generator 430 will be described below in conjunction with Figure 5 Give a description.

[0082] The schedule selector 440 selects a schedule from the schedule tree of the target IR. The schedule will be used to transform the loop nests in the target IR. In some embodiments, the schedule selector 440 selects a schedule based on the predicted performance of the processor 260 in performing tensor calculations based on the schedules in the schedule tree. For example, the schedule selector 440 determines a performance score for the schedule. The performance score indicates the predicted performance of the processor 260 in performing tensor calculations based on the schedule. The schedule selector 440 can rank the schedules based on the performance score and select the schedule with the highest ranking, that is, the schedule that can trigger the best performance of the processor 260.

[0083] The schedule selector 440 can determine the performance score by using a performance predictor 450. The performance predictor 450 is a model that has been trained to receive one or more attributes of the IR after transformation with the schedule and output a performance score for the schedule. These attributes include loop extension, tensor references, tensor operations, other attributes, or some combination thereof. The schedule selector 440 can input one or more attributes of the IR 225 after transformation by each corresponding schedule in the schedule tree. The schedule selector 440 then receives the performance score of each corresponding schedule from the performance predictor 450.

[0084] The training module 460 trains the performance predictor 450. The training module 460 applies machine learning techniques to generate the performance predictor 450, and when the performance predictor 450 is applied to the attributes of the IR transformed with the schedule, it outputs a performance score indicating the predicted performance of the processor 260 performing tensor calculations based on the IR. As part of generating the performance predictor 450, the training module 460 can form a training set. The training set includes training samples and ground truth labels for the training samples. The training samples may include one or more attributes of the IR transformed with the schedule. The training samples may have ground truth performance scores, which may be known performance scores, or may be verified performance scores. The training module 460 extracts feature values ​​from the training set, which are variables that are considered to be potentially related to memory misses. The ordered list of features can be a feature vector. In one embodiment, the training module 460 applies dimensionality reduction (eg, via linear discriminant analysis (LDA), principal component analysis (PCA), etc.) to reduce the amount of data in the feature vectors to a smaller, more representative data set.

[0085] The training module 460 can use supervised machine learning to train the performance predictor 450, for example, using the feature vectors of the training set. Different machine learning techniques, such as linear support vector machines (linear SVMs), enhancements of other algorithms (e.g., AdaBoost), neutral networks, logistic regression, naive Bayes, memory-based learning, random forests, bagged trees, decision trees, boosted trees, or boosted stumps, can be used in different embodiments.

[0086] In some embodiments, the validation set is formed by data associated with additional IRs and additional schedules that have known or verified performance scores, not those in the training set. The training module 460 applies the trained performance predictor 450 to the additional IRs and schedules of the validation set to quantify the accuracy of the performance predictor 450. The accuracy can be determined based on the difference between the performance score determined by the performance predictor 450 and the known or verified performance score. In one embodiment, the training module 460 iteratively retrains the performance predictor 450 until a stopping condition occurs, such as an accuracy measure indicating that the model is sufficiently accurate, or multiple training rounds have occurred.

[0087] In some embodiments, the training module 460 continuously trains a portion or all of the performance predictor 450. For example, after the training module 460 trains the performance predictor 450, the performance predictor 450 receives the properties of the memory loop nest and outputs a performance score. After the processor 260 performs tensor calculations based on the memory loop nest, the training module 460 may receive performance information. The training module 460 may determine a runtime performance score based on the performance information. The runtime performance score indicates a true memory miss during the execution of the tensor calculation. The training module 460 uses the memory loop nest and the runtime performance score as new training samples to further train the performance predictor 450. The training module 460 may continuously generate new training sets and retrain the performance predictor 450 when more performance information is received and more runtime performance scores are determined.

[0088] Figure 5 is a block diagram of a schedule generator 430 according to various embodiments. As described above, the schedule generator 430 generates a schedule tree for a target IR. The schedule generator 430 includes a tiling module 510, a permutation module 520, an extension module 530, a heat map data repository 540, a miss model 550, a schedule tree generator 560, and a training module 570. In other embodiments, alternative configurations, different components, or additional components may be included in the schedule generator 430. In addition, the functionality of a component attributed to the schedule generator 430 may be performed by different components included in the schedule generator 430 or by different systems.

[0089] The tiling module 510 identifies loop nests in the target IR and partitions the loop nests into multiple loop nests by loop tiling. In some embodiments, the tiling module 510 can tile each loop in the loop nest into a separate loop based on the memory level associated with the processor 260. The memory level may include registers, L1 cache, L2 cache... and the last level cache. Each separate loop corresponds to a different memory level. For example, the loop is split into a register loop, an L1 cache loop, an L2 cache loop... and a last level cache loop. After loop tiling, there will be multiple loop nests, each loop nest corresponds to a different memory level, and has N memory loops.

[0090] In one example, the loop nest includes n loops: i1, i2, ...i n , where n is an integer greater than 2. The processor 260 is associated with four memory levels: registers, L1 cache, L2 cache, L3 cache, and main storage (e.g., DRAM (dynamic random access memory), where the L3 cache is the last level cache). The tiling module 510 splits each of the n loops into five memory loops, each memory loop corresponding to one of the five memory levels. As a result, the tiling module 510 generates five memory loop nests. The first memory loop nest is for registers and includes n loops: i 1Register ,i 2Register ,...i nRegister The second memory loop nest is for L1 cache and consists of n loops: i 1L1 ,i 2L1 ,...i nL1 The third memory loop nest is for L2 cache and consists of n loops: i 1L2 ,i 2L2 ,...i nL2 The fourth memory loop nest is for L3 cache and includes n loops: i 1L3 ,i 2L3 ,...i nL3 The fifth memory loop nest is for main memory and includes n loops: i 1Mem ,i 2Mem ,...i nMem . Loop i 1Register 、i 1L1 、i 1L2 、i 1L3 and i 1Mem is generated by partitioning loop i1. Similarly, loop i 2Register 、i 2L1 、i 2L2 、i 2L3 and i 2Memis generated by partitioning loop i2, and loop i nRegister 、i nL1 、i nL2 、i nL3 and i nMem is through the loop i n Generated by partition.

[0091] The permutation module 520 adjusts the order of memory loops in one or more memory loop nests. For a particular memory level, the permutation module 520 determines one or more loop permutations and uses the determined one or more loop permutations to change the order of memory loops in the corresponding memory loop nests to minimize data movement with the memory. The permutation module 520 can generate one or more loop permutations by using analytical model technology. In some embodiments, the permutation module 520 does not adjust the order of all memory loop nests. For example, the permutation module 520 can determine not to adjust the order of register loops in register loop nests. Register loops are located at the innermost level, and these loops and loop bodies must match one or more microkernels. The permutation module 520 can determine that the order of register loops does not need to be adjusted because the register loops will be replaced by the microkernel anyway.

[0092] The stretch module 530 determines one or more loop stretch sets for each memory loop nest. The stretch module 530 may process the memory loop nests in an order determined based on the hierarchy of the memory, for example, from the memory with the highest hierarchy to the memory with the lowest hierarchy. For example, the stretch module 530 may first determine the loop stretch of the registers, then determine the loop stretch of the L1 cache, followed by the L2 cache, and all the way to the last level cache. In some embodiments, the stretch module 530 determines the optimal loop stretch set for the register loop nest. For example, for the brgemm (batch reduction general matrix multiplication) microkernel in TPP (tensor processing primitives), the loop stretch set (bc, bk) = (32, 24) may produce the optimal performance, where bc and bk are the loop stretches of the two register loops. By enabling full utilization (or near full utilization) of registers, the optimal loop stretch set may produce the optimal performance of the processor 260. In some embodiments, the stretch module 530 may maintain a heat map stored in the heat map data repository 540. The heat map represents the relationship between various loop stretch sets and corresponding performance. The stretch module 530 may identify an optimal set of register cycles from the heat map.

[0093] In other embodiments, the stretch module 530 may determine one or more loop stretch sets ("loop stretch sets") for a memory loop nest based on the prediction misses of the corresponding memory. Each loop stretch set includes loop stretches of all memory loops in the memory loop nest and corresponds to a different loop permutation determined by the permutation module 520. The loop stretch sets may minimize memory misses if the memory loop nest is reordered based on the loop permutation. Taking a cache loop nest as an example, the stretch module 530 may determine one or more optimal loop stretch sets for each loop permutation determined by the permutation module 520 for the cache loop nest.

[0094] In order to determine the optimal loop extension set for loop permutation of the memory loop nest, the extension module 530 can obtain multiple candidate loop extension sets. The candidate loop extension set can be generated based on the original loop extension of the memory loop. Then, the extension module 530 uses the miss model 550 to determine whether the candidate loop extension set is the optimal loop extension set. For example, the extension module 530 inputs the loop extensions in the candidate loop extension set into the miss model 550. The extension module 530 can also input other attributes associated with the memory loop nest into the miss model 550. These attributes may include, for example, the loop extension of the inner loop, tensor references, data reuse factors, tensor operations, etc. The data reuse factor indicates the extent to which data can be reused in the loop and can be determined by data reuse analysis. The miss model 550 outputs a miss score, which indicates the number of predicted misses of the memory. The stretch module 530 may rank the candidate cycle stretch sets based on their miss scores and select one or more candidate cycle stretch sets as the optimal cycle stretch set(s) based on the rankings. For example, the stretch module 530 may select a candidate cycle stretch set having a miss score lower than a threshold score or lower than the miss scores of other candidate cycle stretch sets.

[0095] After the permutation module 520 determines the loop permutation and the stretch module 530 determines the loop stretch, the scheduling tree generator 560 generates a scheduling tree for the target IR. The scheduling tree starts with a root that includes the target IR. The root is the first level of the scheduling tree. The root has several children as nodes in the second level as the target IR. Each node in the second level can be a permutation of a tiled loop with a loop stretch set. A node can derive one or more children as new nodes. The scheduling tree generator 560 can remove unit loops, that is, loops with a stretch equal to 1. Additionally or alternatively, the scheduling tree generator 560 can fold adjacent parallel loops into a single parallel loop. Parallel loops are loops that can be executed in parallel, rather than being executed sequentially.

[0096] The training module 570 trains the miss model 550. The training module 570 applies machine learning techniques to generate the miss model 550, which, when applied to the properties of the memory loop nest, outputs a miss score indicating a predicted memory miss. As part of generating the miss model 550, the training module 570 may form a training set. The training set includes training samples and ground truth labels for the training samples. The training samples may include a set of properties of the memory loop nest. The training samples may have ground truth miss scores, which may be known miss scores, or may be miss scores that have been verified. The training module 570 extracts feature values ​​from the training set, which are variables that are considered to be potentially related to memory misses. The ordered list of features may be a feature vector. In one embodiment, the training module 570 applies dimensionality reduction (e.g., via LDA, PCA, etc.) to reduce the amount of data in the feature vector to a smaller, more representative data set.

[0097] The training module 570 can use supervised machine learning to train the miss model 550, for example, with the feature vectors of the positive training set and the negative training set as input. Different machine learning techniques, such as linear support vector machines (linear SVMs), boosting of other algorithms (e.g., AdaBoost), neutral networks, logistic regression, naive Bayes, memory-based learning, random forests, bagged trees, decision trees, boosted trees, or boosted stumps, can be used in different embodiments.

[0098] In some embodiments, the validation set is formed of data associated with additional memory loop nests with known or verified miss scores, not those in the training set. The training module 570 applies the trained miss model 550 to the memory loop nests of the validation set to quantify the accuracy of the miss model 550. The accuracy can be determined based on the difference between the miss scores determined by the miss model 550 and the known or verified miss scores. In one embodiment, the training module 570 iteratively retrains the miss model 550 until a stopping condition occurs, such as an accuracy measure indicating that the model is sufficiently accurate, or a number of training rounds have occurred.

[0099] In some embodiments, the training module 570 continuously trains a portion or all of the miss model 550. For example, after the training module 570 trains the miss model 550, the miss model 550 receives the attributes of the memory loop nest and outputs a miss score. After the processor 260 performs tensor calculations based on the memory loop nest, the training module 570 may receive performance information. The training module 570 may determine a runtime miss score based on the performance information. The runtime miss score indicates a real memory miss during the execution of the tensor calculation. The training module 570 uses the memory loop nest and the runtime miss score as new training samples to further train the miss model 550. The training module 570 may continuously generate new training sets and retrain the miss model 550 when more performance information is received and more runtime miss scores are determined.

[0100] Example Schedule Tree

[0101] Figure 6 An example scheduling plan tree 600 is illustrated according to various embodiments. The scheduling plan tree 600 may be composed of Figure 4 and Figure 5 The scheduling plan generator 430 generates. Figure 6 As shown, the scheduling plan tree 600 includes IR 610 and four scheduling plans 620, 625, 630 and 635. IR 610 is the root of the scheduling plan tree 600. The four scheduling plans 620, 625, 630 and 635 are nodes of the scheduling plan tree 600, wherein the scheduling plans 620 and 625 are children of the root, and the scheduling plans 630 and 635 are grandchildren of the root. The scheduling plans 620 and 630 constitute the first branch of the scheduling plan tree 600. The scheduling plans 625 and 635 constitute the second branch of the scheduling plan tree 600. IR 610 is the first level of the scheduling plan tree 600. The scheduling plans 620 and 625 are the second level. The scheduling plans 630 and 635 are the third level.

[0102] Schedule 620 specifies a permutation of loops i and j, i.e., a change in the order of loops i and j. Schedule 630 is a child of schedule 620, which specifies tiling of loop i according to a tiling factor g. For example, schedule 630 may split schedule i into two loops: an outer loop and an inner loop. The extension of the outer loop may be equal to the original extension of loop i divided by g, and the extension of the inner loop may be equal to g. Alternatively, the extension of the inner loop may be equal to the original extension of loop i divided by g, and the extension of the outer loop may be equal to g. As a child of schedule 620, schedule 630 includes schedule 620. In an embodiment for implementing selection of schedule 630, loops i and j will first be permuted according to the information in schedule 620, and then loop i will be split according to the information in schedule 630.

[0103] Schedule 625, which is parallel to schedule 620 and is a sibling of schedule 625, specifies a permutation of loops m and n, that is, a change in the order of loops m and n. Schedule 635, which is a child of schedule 625, specifies that loop m is tiled according to a tiling factor h. For example, schedule 635 can split loop m into 2 loops: an outer loop and an inner loop. The extension of the outer loop can be equal to the original extension of loop m divided by h, and the extension of the inner loop can be equal to h. Alternatively, the extension of the inner loop can be equal to the original extension of loop m divided by h, and the extension of the outer loop can be equal to h. As a child of schedule 625, schedule 635 contains schedule 625. In an embodiment for implementing the selection of schedule 635, loops m and n will first be permuted according to the information in schedule 625, and then loop m will be split according to the information in schedule 635.

[0104] For simplicity and illustration, Figure 6 The scheduling plan tree 600 in the embodiment includes four scheduling plans, two branches and three levels. In other embodiments, the scheduling plan tree 600 may include a different number of scheduling plans, a different number of branches, a different number of levels, a different structure or some combination thereof.

[0105] Example method for loop transformation for deep learning

[0106] Figure 7 The flowchart of FIG. 700 shows a method 700 for loop transformation for deep learning according to various embodiments. The method 700 may be Figure 2 The tensor compiler 230 in . Although reference Figure 7 The flowchart shown in describes method 700, but many other methods of loop transformation for deep learning may be used instead. For example, Figure 7 The execution order of the steps in the method can be changed. As another example, some steps can be changed, eliminated or combined.

[0107] The tensor compiler 230 generates (710) multiple schedules for the data structure. The data structure may be an IR, such as IR225. The data structure includes loop nests. The loop nests include multiple loops. The loop specifies a tensor operation to be repeatedly performed by the DNN. The tensor operation may be a convolution, a pooling operation, an element-by-element operation, a reduction, a load, or other types of tensor operations. The loop may be nested inside one or more other loops. The schedule specifies a transformation of the loop nest. The transformation may be a loop permutation, an index rewrite, a loop unrolling, a loop splitting, a loop tiling, a loop filling, other types of loop transformations, or some combination thereof. In some embodiments, multiple schedules may be arranged in a schedule tree, where the data structure is a root and the schedule is a node. The schedule tree may include two or more levels. The root is at a first level. One or more schedules may be children of the root and are at a second level of the schedule tree. One or more other schedules may be children of a node in the second level and are at a third level of the schedule tree. For a level including multiple schedules, the schedules are brothers and may be parallel.

[0108] For each corresponding schedule in the plurality of schedules, the tensor compiler 230 inputs (720) one or more attributes of the data structure transformed by the corresponding schedule into the trained model. The one or more attributes of the data structure transformed by the corresponding schedule are selected from the group consisting of: a type of tensor operation, a loop nesting extension indicating a number of times the DNN is to repeat the tensor operation, a tensor rank associated with the tensor operation, a tensor shape associated with the tensor operation, and a tensor length associated with the tensor operation.

[0109] The trained model outputs a prediction performance score indicating an assessment of the prediction performance of the DNN. The prediction performance of the DNN may be the prediction performance of the DNN when the DNN performs tensor operations based on the data structure after the loop nests are transformed using the corresponding schedule. An example of a trained model is Figure 4 The DNN may be implemented in one or more processors, such as processor 260. The DNN may include different levels of memory, such as registers, various levels of cache, etc. An example of a DNN is Figure 1 DNN 100 in .

[0110] The tensor compiler 230 selects (730) a schedule from the plurality of schedules based on the predicted performance scores of the plurality of schedules. For example, the tensor compiler 230 selects the schedule with the highest predicted performance score, i.e., the schedule that is predicted by the trained model to trigger the best performance of the DNN.

[0111] The tensor compiler 230 transforms (740) the loop nests in the data structure based on the schedule. After transforming the loop nests based on the schedule, the data structure is used by the DNN to perform tensor operations. The tensor compiler 230 can generate an implementation based on the transformation of the loop nests. The tensor compiler 230 can also instrument the implementation so that the DNN can generate performance information indicating the performance of the DNN when performing tensor operations, and transmit the performance information to the tensor compiler 230.

[0112] The tensor compiler 230 receives (750) information indicating the runtime performance of the DNN when performing tensor operations. The tensor compiler 230 can evaluate the runtime performance of the DNN based on the information. The tensor compiler 230 can also compare the runtime performance with the predicted performance. In some embodiments, the tensor compiler 230 determines a runtime performance score for the schedule based on the information, and compares the runtime performance score with the predicted performance score determined by the trained model. The tensor compiler 230 can determine the difference between the runtime performance score and the predicted performance score. The difference can indicate the accuracy of the training model in predicting the performance of the DNN.

[0113] The tensor compiler 230 updates (760) the training model based on the evaluation of the runtime performance of the DNN. In some embodiments, the tensor compiler 230 determines whether the difference between the runtime performance score and the predicted performance score exceeds a threshold. In response to determining that the difference exceeds the threshold, the tensor compiler 230 can form a training sample including a scheduling plan and a runtime performance score, and further train the trained model using the training sample. In the process of further training the trained model, the runtime performance score can be used as a ground truth label for the scheduling plan. In some embodiments, the tensor compiler 230 can select different scheduling plans from multiple scheduling plans by using the updated trained model. The tensor compiler 230 can also transform loop nests in the data structure based on different scheduling plans. After transforming the loop nests based on different scheduling plans, the data structure is used for a new execution of the tensor operation by the DNN.

[0114] Example method for generating a scheduling plan tree

[0115] Figure 8 The flowchart of FIG. 8 shows a method 800 for generating a scheduling plan tree for an IR according to various embodiments. The method 800 may be performed by Figure 4 The search module 410 in FIG. Figure 8 The flowchart shown in describes method 800, but many other methods for generating a scheduling plan tree may be used instead. For example, Figure 8 The execution order of the steps in the method can be changed. As another example, some steps can be changed, eliminated or combined.

[0116] The search module 410 determines (810) whether the data structure belongs to a data structure category in the database. The data structure includes a loop nest that specifies tensor operations to be repeatedly performed by the DNN. The tensor operation can be a convolution, a pooling operation, an element-by-element operation, a reduction, a load, or other type of tensor operation. The loop nest includes multiple loops. A loop can be nested inside one or more other loops. The schedule specifies a transformation of the loop nest. The DNN can be implemented in one or more processors, such as processor 260. The DNN can include different levels of memory, such as registers, various levels of cache, etc. An example of a DNN is Figure 1 DNN 100 in .

[0117] In response to determining that the data structure belongs to a data structure category, the search module 410 retrieves (820) a group candidate schedule associated with the data structure category from the database. The candidate schedule specifies a loop transformation. The loop transformation can be a loop permutation, index rewrite, loop unrolling, loop splitting, loop tiling, loop filling, other types of loop transformations, or some combination thereof. The data structure category is a category of data structures that include the same tensor operations and tensor references but may have different loop extensions.

[0118] The search module 410 selects (830) a subset of the candidate scheduling plan group by removing one or more candidate scheduling plans from the group. In some embodiments, the group of candidate scheduling plans includes a first candidate scheduling plan and a second candidate scheduling plan. The search module 410 can determine a similarity score indicating the similarity between the data structure and the data structure associated with the first candidate scheduling plan. The search module 410 determines whether the similarity score is lower than a threshold score. After determining that the similarity score is lower than the threshold score, the search module 410 removes the first candidate scheduling plan from the group. The search module 410 can also determine a similarity score indicating the similarity between the data structure and the data structure associated with the second candidate scheduling plan. The search module 410 determines whether the similarity score of the second candidate scheduling plan is lower than the threshold score. After determining that the similarity score is not lower than the threshold score, the search module 410 determines not to remove the second candidate scheduling plan from the group. The search module 410 includes the second candidate scheduling plan in the subset.

[0119] The search module 410 generates (840) a scheduling tree for the data structure by merging the candidate scheduling plans in the subsets. The candidate scheduling plans in the subsets can be arranged in the scheduling tree based on their hierarchy. In some embodiments, the data structure is the root of the scheduling tree, and the candidate scheduling plans are nodes of the scheduling tree. The scheduling tree may include two or more levels. The root is at the first level. One or more scheduling plans can be children of the root and are at the second level of the scheduling tree. One or more other scheduling plans can be children of the nodes in the second level and are at the third level of the scheduling tree. For levels including multiple scheduling plans, these scheduling plans are brothers and can be parallel.

[0120] The search module 410 may also remove one or more schedules in the schedule tree. For example, the search module 410 may identify a schedule that specifies loop tiling for splitting a loop in a loop nest into multiple new loops. The search module 410 may determine that loop tiling is incompatible with the loop, for example, based on determining that the result of the loop extension of the loop divided by the tiling factor (i.e., the number of new loops split from the loop in the loop nest) is not an integer. Then, the search module 410 may remove the schedule for loop tiling from the schedule tree. Alternatively, the search module 410 may modify the loop that is incompatible with the schedule. For example, the search module 410 adjusts the loop extension so that the loop tiling is compatible with the loop, for example, so that the result of the loop extension divided by the tiling factor is an integer. In some embodiments, for example, in embodiments where the loop tiling schedule is incompatible with multiple loops, the search module 410 may modify multiple loops in the loop nest. The search module 410 may start from the innermost loop, for example, the register loop. After the innermost loop is modified, the search module 410 modifies the loop of the next memory level, for example, the L0 cache loop. The search module 410 may continue this modification process until the last incompatible cycle is modified.

[0121] Fig. 9 The flowchart of FIG. 9 shows another method 900 for generating a scheduling plan tree for an IR according to various embodiments. The method 900 may be performed by Figure 4 The schedule generator 430 in the embodiment is executed. Fig. 9 The flowchart shown in is used to describe the method 900, but many other methods for generating a scheduling plan tree may be used instead. For example, Fig. 9 The execution order of the steps in the method can be changed. As another example, some steps can be changed, eliminated or combined.

[0122] The schedule generator 430 partitions (910) the loop nest into a plurality of memory loop nests. Each of the loop nest and the memory loop nest includes a sequence of loops. The loop indicates a tensor operation to be repeatedly performed by the processor. The loop in the loop nest is partitioned into a plurality of loops. Each loop is in a different memory loop nest in the plurality of memory loop nests and corresponds to a different memory associated with the processor. The partitioning may be accomplished by loop tiling, and the tiling factor is equal to the number of memory levels associated with the processor.

[0123] The schedule generator 430 determines (920) a loop stretch of a loop in a first memory loop nest of the plurality of loop nests. The loop stretch of the loop indicates the number of times the tensor operation in the loop is to be repeatedly performed. In some embodiments, the first memory loop nest may be a register memory loop nest. In other embodiments, the first memory loop nest may be a cache memory loop nest. Before the schedule generator 430 determines the loop stretch, the schedule generator 430 may determine one or more loop permutations of the first memory loop nest.

[0124] The schedule generator 430 determines (930) one or more permutations for a second memory loop nest in the plurality of loop nests. The second memory loop nest targets a different memory level than the first memory loop nest. Each permutation indicates a change in the order of loops in the second memory loop nest. The second memory loop nest may be a cached memory loop nest.

[0125] The schedule generator 430 determines (940) a loop extension for a loop in a second memory loop nest based on one or more permutations. In some embodiments, the schedule generator 430 may determine a plurality of candidate sets. Each candidate set includes candidate loop extensions for loops in the second memory loop nest. For each respective candidate set, the candidate loop extensions in the candidate set and one or more attributes of the data structure are input into an additional trained model. The additional trained model outputs a miss score indicating a predicted miss of the memory corresponding to the second memory loop nest when the DNN performs a tensor operation based on the data structure in which the loop nest is transformed based on the respective candidate set. An example of an additional trained model is Figure 5 The schedule generator 430 then selects a candidate set from the plurality of candidate sets based on the miss scores of the plurality of candidate sets. The candidate set includes loop extensions of loops in the second memory loop nest. The schedule generator 430 may select the candidate set with the lowest miss score, which indicates the least memory misses.

[0126] The schedule generator 430 generates (950) a schedule tree based on the loop extension of the loop in the first memory loop nest, one or more permutations for the second memory loop nest, and the loop extension of the loop in the second memory loop nest. The schedule tree includes a schedule that specifies the transformation of the loop in the first memory loop nest and the loop in the second memory loop nest. In some embodiments, the schedule generator 430 can receive runtime performance information after the processor performs the tensor operation according to the data structure based on the schedule tree transformation. The schedule generator 430 can use the runtime performance information to determine a runtime miss score that indicates an evaluation of memory misses that occurred during the execution of the tensor operation. The schedule generator 430 can use the selected candidate set and the runtime miss score as training samples to further train additional trained models, for example, after determining that the difference between the runtime miss score and the miss score determined by the additional trained model exceeds a threshold.

[0127] Sample deep learning environment

[0128] Fig.10 A deep learning environment 1000 is illustrated according to various embodiments. The deep learning environment 1000 includes a deep learning server 1010 and a plurality of client devices 1020 (individually referred to as client devices 1020). The deep learning server 1010 is connected to the client devices 1020 via a network 1030. In other embodiments, the deep learning environment 1000 may include fewer, more, or different components.

[0129] The deep learning server 1010 uses a neural network to train a DL model. The structure of a neural network is like a human brain and is composed of artificial neurons, also known as nodes. These nodes are stacked on top of each other in three types of layers: an input layer, (one or more) hidden layers, and an output layer. Data provides information to each node in the form of input. The node multiplies the input with a random weight, performs calculations, and adds a bias. Finally, a nonlinear function, also known as an activation function, is applied to determine which neuron to start. The deep learning server 1010 can use various types of neural networks, such as DNN, recurrent neural network (RNN), generative adversarial network (GAN), long short-term memory network (LSTMN), and the like. During the process of training a DL model, the neural network uses unknown elements in the input distribution to extract features, group objects, and find useful data patterns. DL models can be used to solve various problems, such as making predictions, classifying images, and the like. The deep learning server 1010 can build a DL model specifically for a specific type of problem that needs to be solved. The DL model is trained to receive inputs and output solutions to specific problems.

[0130] exist Fig.10 In the example, deep learning server 1010 includes DNN system 1040, database 1050, and distributor 1060. DNN system 1040 trains DNN. DNN can be used to process images, such as images captured by autonomous vehicles, medical equipment, satellites, etc. In one embodiment, DNN receives input images and outputs classifications of objects in the input images. An example of DNN is the above combined Figure 1 In some embodiments, the DNN system 1040 trains the DNN by knowledge distillation (e.g., knowledge distillation based on dense connections). The trained DNN can be used on low-memory systems, such as mobile phones, IOT edge devices, etc. An embodiment of the DNN system 1040 is the above combined Figure 2 Described DNN system 200.

[0131] The database 1050 stores data received, used, generated, or otherwise associated with the deep learning server 1010. For example, the database 1050 stores a training data set used by the DNN system 1040 to train the DNN. In one embodiment, the training data set is an image library that can be used to train a DNN for classifying images. The training data set may include data received from the client device 1020. As another example, the database 1050 stores hyperparameters of a neural network constructed by the deep learning server 1010.

[0132] The distributor 1060 distributes the DL model generated by the deep learning server 1010 to the client device 1020. In some embodiments, the distributor 1060 receives a request for a DNN from the client device 1020 via the network 1030. The request may include a description of a problem that the client device 1020 needs to solve. The request may also include information about the client device 1020, such as information describing the available computing resources on the client device. The information describing the available computing resources on the client device 1020 may be information indicating the network bandwidth, information indicating the available memory size, information indicating the processing power of the client device 1020, and the like. In one embodiment, the distributor may instruct the DNN system 1040 to generate a DNN according to the request. The DNN system 1040 may generate a DNN based on the information in the request. For example, the DNN system 1040 may determine the structure of the DNN and / or train the DNN according to the request.

[0133] In another embodiment, the distributor 1060 may select a DNN from a set of pre-existing DNNs based on the request. The distributor 1060 may select a DNN for a particular client device 1030 based on the size of the DNN and the available resources of the client device 1020. In an embodiment where the distributor 1060 determines that the memory or processing power of the client device 1020 is limited, the distributor 1060 may select a compressed DNN for the client device 1020 instead of an uncompressed DNN with a larger size. The distributor 1060 then transmits the DNN generated or selected for the client device 1020 to the client device 1020.

[0134] In some embodiments, the distributor 1060 may receive feedback from the client device 1020. For example, the distributor 1060 receives new training data from the client device 1020 and may send the new training data to the DNN system 1040 to further train the DNN. As another example, the feedback includes an update of available computer resources on the client device 1020. The distributor 1060 may send a different DNN to the client device 1020 based on the update. For example, after receiving feedback indicating that the computing resources of the client device 1020 have been reduced, the distributor 1060 sends a smaller DNN to the client device 1020.

[0135] The client device 1020 receives the DNN from the distributor 1060 and applies the DNN to perform machine learning tasks, for example, to solve a problem or answer a question. In various embodiments, the client device 1020 inputs an image into the DNN and uses the output of the DNN for various applications, for example, visual reconstruction, augmented reality, robot positioning and navigation, medical diagnosis, weather prediction, and the like. The client device 1020 may be one or more computing devices that are capable of receiving user input and sending and / or receiving data via the network 1030. In one embodiment, the client device 1020 is a conventional computer system, such as a desktop or laptop computer. Alternatively, the client device 1020 may be a device with computer functionality, such as a personal digital assistant (PDA), a mobile phone, a smart phone, an autonomous vehicle, or other appropriate device. The client device 1020 is configured to communicate via the network 1030. In one embodiment, the client device 1020 executes an application that allows a user of the client device 1020 to interact with the deep learning server 1010 (e.g., the distributor 1060 of the deep learning server 1010). The client device 1020 may request the DNN or send feedback to the distributor 1060 through the application. For example, the client device 1020 executes a browser application to enable interaction between the client device 1020 and the deep learning server 1010 via the network 1030. In another embodiment, the client device 1020 executes a browser application on the native operating system of the client device 1020 (e.g. or ANDROID TM ) to interact with the deep learning server 1010.

[0136] In one embodiment, client device 1020 is an integrated computing device that operates as an independent device with network capabilities. For example, client device 1020 includes a display, a speaker, a microphone, a camera, and an input device. In another embodiment, client device 1020 is a computing device for coupling to an external media device, such as a television or other external display and / or an audio output system. In this embodiment, client device 1020 can be coupled to an external media device via a wireless interface or a wired interface (e.g., an HDMI (High Definition Multimedia Interface) cable), and various functions of the external media device can be utilized, such as its display, speaker, microphone, camera, and input device. Here, client device 1020 can be configured to be compatible with a general external media device that does not have specialized software, firmware, or hardware specifically for interacting with client device 1020.

[0137] The network 1030 supports communication between the deep learning server 1010 and the client device 1020. The network 1030 may include any combination of local area networks and / or wide area networks, using wired and / or wireless communication systems. In one embodiment, the network 1030 may use standard communication technologies and / or protocols. For example, the network 1030 may include communication links using technologies such as Ethernet, 10010.11, worldwide interoperability for microwave access (WiMAX), 3G, 4G, code division multiple access (CDMA), digital subscriber line (DSL), etc. Examples of networking protocols for communicating via the network 1030 may include multiprotocol label switching (MPLS), transmission control protocol / Internet protocol (TCP / IP), hypertext transport protocol (HTTP), simple mail transfer protocol (SMTP), and file transfer protocol (FTP). Data exchanged through network 1030 may be represented using any suitable format, such as hypertext markup language (HTML) or extensible markup language (XML). In some embodiments, all or some of the communication links of network 1030 may be encrypted using any one or more suitable techniques.

[0138] Example DNN System

[0139] Fig.111100 is a block diagram of an example DNN system 1100 according to various embodiments. The entire DNN system 1100 or a portion of the DNN system 1100 can be implemented in the computing device 1200. The DNN system 1100 trains DNNs for various tasks, such as image classification, learning relationships between biological cells (e.g., DNA, proteins, etc.), controlling the behavior of devices (e.g., robots, machines, etc.), and the like. The DNN system 1100 includes an interface module 1110, a training module 1120, a verification module 1130, an inference module 1140, and a memory 1150. In other embodiments, alternative configurations, different or additional components may be included in the DNN system 1100. In addition, the functionality attributed to a component of the DNN system 1100 may be performed by different components or different systems included in the DNN system 1100. The DNN system 1100 or a component of the DNN system 1100 (e.g., the training module 1120 or the inference module 1140) may include Fig.12 The computing device 1200 in FIG.

[0140] The interface module 1110 facilitates communication between the DNN system 1100 and other systems. For example, the interface module 1110 establishes communication between the DNN system 1100 and an external database to receive data that can be used to train the DNN or input into the DNN to perform a task. As another example, the interface module 1110 supports the DNN system 1100 to distribute the DNN to other systems, such as computing devices configured to apply the DNN to perform tasks.

[0141] The training module 1120 trains the DNN by using the training data set. The training module 1120 forms the training data set. In an embodiment where the training module 1120 trains the DNN to recognize objects in an image, the training data set includes training images and training labels. The training labels describe the ground truth classification of the objects in the training images. In some embodiments, each label in the training data set corresponds to an object in the training image. In some embodiments, a portion of the training data set may be used to initially train the DNN, while the remainder of the training data set may be retained as a validation subset for the validation module 1130 to validate the performance of the trained DNN. A portion of the training data set that does not include the tuning subset and the validation subset may be used to train the DNN.

[0142] The training module 1120 also determines the hyperparameters for training the DNN. Hyperparameters are variables that specify the DNN training process. Hyperparameters are different from parameters inside the DNN (e.g., weights of filters). In some embodiments, hyperparameters include variables that determine the architecture of the DNN, such as the number of hidden layers, and the like. Hyperparameters also include variables that determine how the DNN is trained, such as batch size, number of rounds, and the like. The batch size defines the number of training samples to be worked through before updating the parameters of the DNN. The batch size is the same as or smaller than the number of samples in the training data set. The training data set can be divided into one or more batches. The number of rounds defines how many times the entire training data set is passed forward and backward throughout the network. The number of rounds defines the number of times the deep learning algorithm works through the entire training data set. One round means that each training sample in the training data set has the opportunity to update the parameters inside the DNN. A round may include one or more batches. The number of rounds may be 11, 110, 500, 1100, or even larger.

[0143] The training module 1120 defines the architecture of the DNN, for example, based on some hyperparameters. The architecture of the DNN includes an input layer, an output layer, and multiple hidden layers. The input layer of the DNN may include a tensor (e.g., a multidimensional array) specifying the attributes of the input image, such as the height of the input image, the width of the input image, and the depth of the input image (e.g., the number of bits specifying the color of the pixels in the input image). The output layer includes the labels of the objects in the input layer. The hidden layer is a layer between the input layer and the output layer. The hidden layer includes one or more convolutional layers and one or more other types of layers, such as a pooling layer, a fully connected layer, a normalization layer, a softmax or a logical layer, and the like. The convolutional layer of the DNN abstracts the input image into a feature map, which is represented by a tensor specifying the feature map height, the feature map width, and the feature map channel (e.g., the red, green, and blue images include three channels). The pooling layer is used to reduce the spatial volume of the input image after convolution. It is used between two convolutional layers. The fully connected layer involves weights, biases, and neurons. It connects neurons in one layer to neurons in another layer. It is used to classify images between different categories through training.

[0144] In the process of defining the architecture of the point DNN, the training module 1120 also adds an activation function to the hidden layer or output layer. The activation function of the layer transforms the weighted sum of the input of the layer into the output of the layer. The activation function can be, for example, a rectified linear unit activation function, a tangent activation function, or other types of activation functions.

[0145] After the training module 1120 defines the architecture of the point DNN, the training module 1120 inputs the training data set into the DNN. The training data set includes multiple training samples. Examples of training samples include objects in images and ground truth labels of objects. The training module 1120 modifies the parameters inside the DNN ("internal parameters of the DNN") to minimize the error between the labels of the training objects generated by the DNN and the ground truth labels of the objects. The internal parameters include the weights of the filters in the convolutional layers of the DNN. In some embodiments, the training module 1120 uses a cost function to minimize the error.

[0146] The training module 1120 may train the DNN for a predetermined number of rounds. The number of rounds is a hyperparameter that defines the number of times the DL algorithm will work through the entire training data set. One round means that each sample in the training data set has a chance to update the internal parameters of the DNN. After the training module 1120 completes the predetermined number of rounds, the training module 1120 may stop updating the parameters in the DNN. The DNN with updated parameters is called a trained DNN.

[0147] The verification module 1130 verifies the accuracy of the trained DNN. In some embodiments, the verification module 1130 inputs the samples in the verification data set into the trained DNN, and uses the output of the DNN to determine the model accuracy. In some embodiments, the verification data set may be formed by some or all samples in the training data set. Additionally or alternatively, the verification data set includes additional samples in addition to the samples in the training set. In some embodiments, the verification module 1130 may determine an accuracy score that measures the precision, recall, or a combination of precision and recall of the DNN. The verification module 1130 may use the following metrics to determine the accuracy score: precision = TP / (TP+FP) and recall = TP / (TP+FN), where precision may be the number of correctly predicted (TP or true positive) by the reference classification model in the total number of its predictions (TP+FP or false positive), and recall may be the number of correctly predicted (TP) by the reference classification model in the total number of objects that do have the relevant attributes (TP+FN or false negative). F-score (F-score = 2*PR / (P+R)) unifies precision and recall into a single metric.

[0148] The verification module 1130 may compare the accuracy score to a threshold score. In an example where the verification module 1130 determines that the accuracy score of the enhanced model is below the threshold score, the verification module 1130 instructs the training module 1120 to retrain the DNN. In one embodiment, the training module 1120 may iteratively retrain the DNN until a stopping condition occurs, such as an accuracy measurement indicating that the DNN may be sufficiently accurate, or multiple training rounds have occurred.

[0149] The reasoning module 1140 applies a trained or verified DNN to perform a task. For example, the reasoning module 1140 inputs an image into the DNN. The DNN outputs a classification of objects in the image. As an example, a DNN can be configured in a security context to detect malicious or dangerous objects in images captured by security cameras. As another example, a DNN can be configured to detect objects (e.g., road signs, hazards, humans, pets, etc.) in images captured by a camera of an autonomous vehicle. The input to the DNN can be formatted according to a predefined input structure, reflecting the way in which the training data set is provided to the DNN. The DNN can generate an output structure, which can be, for example, a classification of an image, a list of detected objects, the boundaries of detected objects, and so on. In some embodiments, the reasoning module 1140 distributes the DNN to other systems, such as a computing device that communicates with the DNN system 1100, so that other systems apply the DNN to perform tasks.

[0150] The memory 1150 stores data received, generated, used, or otherwise associated with the DNN system 1100. For example, the memory 1150 stores data sets used by the training module 1120 and the verification module 1130. The memory 1150 may also store data generated by the training module 1120 and the verification module 1130, such as hyperparameters for training the DNN, internal parameters of the trained DNN (e.g., values ​​of adjustable parameters of the FALU), and the like. Fig.11 In some embodiments, the memory 1150 is a component of the DNN system 1100. In other embodiments, the memory 1150 may be external to the DNN system 1100 and communicate with the DNN system 1100 via a network.

[0151] Example computing device

[0152] Fig.12 is a block diagram of an example computing device 1200 according to various embodiments. In some embodiments, the computing device 1200 may be used as Fig.11 DNN system 1100 in. Fig.12 1200, but any one or more of these components may be omitted or repeated as appropriate for the application. In some embodiments, some or all of the components included in computing device 1200 may be attached to one or more motherboards. In some embodiments, some or all of these components are manufactured onto a single system-on-a-chip (SoC) die. In addition, in various embodiments, computing device 1200 may not include a processor or a processor. Fig.12, but computing device 1200 may include interface circuitry for coupling to the one or more components. For example, computing device 1200 may not include display device 1206, but may include display device interface circuitry (e.g., connectors and driver circuitry) to which display device 1206 may be coupled. In another set of examples, computing device 1200 may not include audio input device 1218 or audio output device 1208, but may include audio input or output device interface circuitry (e.g., connectors and supporting circuitry) to which audio input device 1218 or audio output device 1208 may be coupled.

[0153] The computing device 1200 may include a processing device 1202 (e.g., one or more processing devices). The processing device 1202 processes electronic data from registers and / or memory to transform the electronic data into other electronic data that can be stored in registers and / or memory. One embodiment of the processing device 1202 may be Figure 2 260 in the processor 260. The computing device 1200 may include a memory 1204, which may itself include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid-state memory, and / or a hard drive. In some embodiments, the memory 1204 may include a memory that shares a die with the processing device 1202. In some embodiments, the memory 1204 includes one or more non-transitory computer-readable media in which executable instructions are stored to perform operations for deep learning, such as the above in conjunction with Figure 7-Figure 9 The methods 700, 800 and 900 described above or a combination thereof Figure 2-Figure 5 The operations performed by the tensor compiler 230 are described. The instructions stored in one or more non-transitory computer-readable media may be executed by the processing device 2402.

[0154] In some embodiments, computing device 1200 may include a communication chip 1212 (e.g., one or more communication chips). For example, communication chip 1212 may be configured to manage wireless communications for transmitting data to and from computing device 1200. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, etc. that can communicate data through a non-solid medium using modulated electromagnetic radiation. The term does not mean that the associated devices do not contain any wires, although in some embodiments they may not.

[0155] The communication chip 1212 can implement any of a variety of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards, including Wi-Fi (IEEE 802.10 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendment), Long-Term Evolution (LTE) project and any amendments, updates and / or revisions (e.g., Advanced LTE project, ultramobile broadband (UMB) project (also known as "3GPP2"), etc.). Broadband Wireless Access (BWA) networks compatible with IEEE802.16 are generally referred to as WiMAX networks. This abbreviation stands for Worldwide Interoperability for Microwave Access, which is a certification mark for products that have passed the compliance and interoperability tests of the IEEE 802.16 standard. The communication chip 1212 can operate according to the Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications Syste (UMTS), High Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA) or LTE network. The communication chip 1212 can operate according to Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN) or Evolved UTRAN (E-UTRAN).The communication chip 1212 may operate according to CDMA, Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO) and its derivatives, and any other wireless protocols designated as 3G, 4G, 5G and above. In other embodiments, the communication chip 1212 may operate according to other wireless protocols. The computing device 1200 may include an antenna 1222 to facilitate wireless communications and / or receive other wireless communications (e.g., AM or FM radio transmissions).

[0156] In some embodiments, the communication chip 1212 can manage wired communications, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication chip 1212 may include multiple communication chips. For example, the first communication chip 1212 may be dedicated to shorter-range wireless communications, such as Wi-Fi or Bluetooth, while the second communication chip 1212 may be dedicated to longer-range wireless communications, such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first communication chip 1212 may be dedicated to wireless communications, while the second communication chip 1212 may be dedicated to wired communications.

[0157] The computing device 1200 may include a battery / power circuit 1214. The battery / power circuit 1214 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuits for coupling components of the computing device 1200 to an energy source separate from the computing device 1200 (e.g., AC line power).

[0158] Computing device 1200 may include a display device 1206 (or corresponding interface circuitry, as described above). Display device 1206 may include any visual indicator, such as a heads-up display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light emitting diode display, or a flat panel display.

[0159] Computing device 1200 may include audio output device 1208 (or corresponding interface circuitry, as described above). Audio output device 1208 may include any device that generates audible indications, such as a speaker, headphones, or earbuds.

[0160] The computing device 1200 may include an audio input device 1218 (or corresponding interface circuitry, as described above). The audio input device 1218 may include any device that generates a signal representing sound, such as a microphone, a microphone array, or a digital musical instrument (e.g., a musical instrument with a musical instrument digital interface (MIDI) output).

[0161] Computing device 1200 may include GPS device 1216 (or corresponding interface circuitry, as described above). GPS device 1216 may communicate with a satellite-based system and may receive the location of computing device 1200, as is known in the art.

[0162] Computing device 1200 may include other output devices 1210 (or corresponding interface circuitry, as described above). Examples of other output devices 1210 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or additional storage devices.

[0163] The computing device 1200 may include other input devices 1220 (or corresponding interface circuits, as described above). Examples of other input devices 1220 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device (e.g., a mouse), a stylus, a touchpad, a barcode reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (register file ID) reader.

[0164] The computing device 1200 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cellular phone, a smart phone, a mobile Internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a PDA, an ultra-mobile personal computer, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 1200 may be any other electronic device that processes data.

[0165] Selected Examples

[0166] The following paragraphs provide various examples of the embodiments disclosed herein.

[0167] Example 1 provides a method for deep learning, which includes generating multiple scheduling plans for a data structure including a loop nest, wherein the loop nest includes multiple loops, the loop specifies a tensor operation to be repeatedly performed by a DNN, and the scheduling plan specifies a transformation of the loop nest; for each corresponding scheduling plan in the multiple scheduling plans, one or more attributes of the data structure after being transformed by the corresponding scheduling plan are input into a trained model, and the trained model outputs a predictive performance score indicating an evaluation of the predictive performance of the DNN; selecting a scheduling plan from the multiple scheduling plans based on the predictive performance scores of the multiple scheduling plans; transforming the loop nest in the data structure based on the scheduling plan, wherein after the loop nest is transformed based on the scheduling plan, the data structure is used by the DNN to perform the tensor operation; receiving information indicating the runtime performance of the DNN when performing the tensor operation; and updating the training model based on the evaluation of the runtime performance of the DNN.

[0168] Example 2 provides a method as described in Example 1, wherein updating the training model based on the evaluation of the runtime performance of the DNN includes determining a runtime performance score indicating the evaluation of the runtime performance of the DNN; forming a training sample including the runtime performance score and one or more parameters associated with the scheduling plan; and further updating the trained model by using the training sample.

[0169] Example 3 provides a method as described in Example 1 or 2, further including selecting a different scheduling plan from the multiple scheduling plans by using an updated trained model; and transforming the loop nest in the data structure based on the different scheduling plan, wherein after the loop nest is transformed based on the different scheduling plan, the data structure is used for a new execution of the tensor operation by the DNN.

[0170] Example 4 provides a method as described in any of the preceding examples, wherein generating the multiple scheduling plans for the data structure includes determining whether the data structure is in a data structure category in a database; in response to determining that the data structure is in the data structure category, retrieving candidate scheduling plans associated with the data structure category from the database; and generating the multiple scheduling plans from the candidate scheduling plans.

[0171] Example 5 provides a method as described in Example 4, wherein the candidate scheduling plans include a first candidate scheduling plan and a second candidate scheduling plan, and generating the multiple scheduling plans for the data structure includes determining a similarity score indicating the similarity between the data structure and a data structure associated with the first candidate scheduling plan; and after determining that the similarity score is lower than a threshold score, generating the multiple scheduling plans based on the second candidate scheduling plan instead of based on the first candidate scheduling plan.

[0172] Example 6 provides a method as described in Example 4 or 5, wherein the candidate scheduling plans include a first candidate scheduling plan and a second candidate scheduling plan, the first candidate scheduling plan specifies a loop tiling for splitting the loop in the loop nest into multiple loops, and generating the multiple scheduling plans for the data structure includes determining that the loop tiling and tensors associated with the loop are incompatible; and generating the multiple scheduling plans based on the second candidate scheduling plan rather than the first candidate scheduling plan.

[0173] Example 7 provides a method as described in any of the preceding examples, wherein generating the multiple scheduling plans for the data structure includes partitioning the loop nest into multiple memory loop nests, wherein each of the loop nests and the memory loop nests includes a sequence of loops, the loops indicating tensor operations to be repeatedly performed by the processor, the loops in the loop nests being partitioned into multiple loops, wherein each loop is in a different memory loop nest in the multiple memory loop nests and corresponds to a different memory associated with the processor; determining a loop extension of a loop in a memory loop nest in the multiple loop nests, the loop extension of the loop indicating the number of times the tensor operations in the loop are to be repeatedly performed; and generating the multiple scheduling plans based on the loop extension.

[0174] Example 8 provides a method as described in Example 7, wherein the memory loop nest is a first memory loop nest, and generating the multiple scheduling plans for the data structure also includes determining one or more permutations for a second memory loop nest in the multiple loop nests, each permutation indicating a change in the order of the loops in the second memory loop nest; determining a loop extension of the loop in the second memory loop nest based on the one or more permutations; and further generating the multiple scheduling plans based on the one or more permutations of the second memory loop nest and the loop extension of the loop in the second memory loop nest.

[0175] Example 9 provides a method as described in Example 8, wherein determining the loop extension of the loop in the second memory loop nest includes determining a plurality of candidate sets, each candidate set including a candidate loop extension of the loop in the second memory loop nest; for each corresponding candidate set, inputting the candidate loop extensions in the candidate set and one or more attributes of the data structure into an additional trained model, the additional trained model outputting a miss score indicating a prediction miss of a memory corresponding to the second memory loop nest when the DNN performs the tensor operation based on the data structure in which the loop nest is transformed based on the corresponding candidate set; and selecting a candidate set from the plurality of candidate sets based on the miss scores of the plurality of candidate sets, wherein the candidate set includes the loop extension of the loop in the second memory loop nest.

[0176] Example 10 provides a method as in any preceding example, wherein one or more attributes of the data structure after being transformed by a corresponding schedule are selected from a group consisting of: the type of the tensor operation, a loop nesting extension indicating the number of times the DNN is to repeat the tensor operation, a tensor rank associated with the tensor operation, a tensor shape associated with the tensor operation, and a tensor length associated with the tensor operation.

[0177] Example 11 provides one or more non-transitory computer-readable media storing instructions that are executable to perform operations for deep learning, the operations comprising generating multiple scheduling plans for a data structure including a loop nest, wherein the loop nest includes multiple loops, the loops specify tensor operations to be repeatedly performed by a DNN, and the scheduling plan specifies a transformation of the loop nest; for each corresponding scheduling plan in the multiple scheduling plans, one or more attributes of the data structure after being transformed by the corresponding scheduling plan are input into a trained model, and the trained model outputs a predictive performance score indicating an evaluation of the predictive performance of the DNN; selecting a scheduling plan from the multiple scheduling plans based on the predictive performance scores of the multiple scheduling plans; transforming the loop nests in the data structure based on the scheduling plan, wherein after the loop nests are transformed based on the scheduling plan, the data structure is used by the DNN to perform the tensor operation; receiving information indicating the runtime performance of the DNN when performing the tensor operation; and updating the training model based on the evaluation of the runtime performance of the DNN.

[0178] Example 12 provides one or more non-transitory computer-readable media as described in Example 11, wherein updating the training model based on the evaluation of the runtime performance of the DNN includes determining a runtime performance score indicative of the evaluation of the runtime performance of the DNN; forming a training sample including the runtime performance score and one or more parameters associated with the scheduling plan; and further updating the trained model by using the training sample.

[0179] Example 13 provides one or more non-transitory computer-readable media as described in Example 12, wherein the operation also includes selecting a different scheduling plan from the multiple scheduling plans by using the updated trained model; and transforming the loop nest in the data structure based on the different scheduling plan, wherein after the loop nest is transformed based on the different scheduling plan, the data structure is used for a new execution of the tensor operation by the DNN.

[0180] Example 14 provides one or more non-transitory computer-readable media as described in any of Examples 11-13, wherein generating the plurality of scheduling plans for the data structure includes determining whether the data structure is in a data structure category in a database; in response to determining that the data structure is in the data structure category, retrieving a candidate scheduling plan associated with the data structure category from the database; and

[0181] The plurality of schedules are generated from the candidate schedules.

[0182] Example 15 provides one or more non-transitory computer-readable media as described in Example 14, wherein the candidate scheduling plans include a first candidate scheduling plan and a second candidate scheduling plan, and generating the multiple scheduling plans for the data structure includes determining a similarity score indicating the similarity between the data structure and a data structure associated with the first candidate scheduling plan; and after determining that the similarity score is lower than a threshold score, generating the multiple scheduling plans based on the second candidate scheduling plan instead of based on the first candidate scheduling plan.

[0183] Example 16 provides one or more non-transitory computer-readable media as described in Example 14 or 15, wherein the candidate scheduling plans include a first candidate scheduling plan and a second candidate scheduling plan, the first candidate scheduling plan specifies a loop tiling for splitting the loop in the loop nest into multiple loops, and generating the multiple scheduling plans for the data structure includes determining that the loop tiling and tensors associated with the loop are incompatible; and generating the multiple scheduling plans based on the second candidate scheduling plan rather than based on the first candidate scheduling plan.

[0184] Example 17 provides one or more non-transitory computer-readable media as described in any of Examples 11-16, wherein generating the multiple scheduling plans for the data structure includes partitioning the loop nest into multiple memory loop nests, wherein each of the loop nests and the memory loop nests includes a sequence of loops, the loops indicating tensor operations to be repeatedly performed by the processor, the loops in the loop nests being partitioned into multiple loops, wherein each loop is in a different memory loop nest in the multiple memory loop nests and corresponds to a different memory associated with the processor; determining a loop extension of a loop in a memory loop nest in the multiple loop nests, the loop extension of the loop indicating the number of times the tensor operation in the loop is to be repeatedly performed; and generating the multiple scheduling plans based on the loop extension.

[0185] Example 18 provides one or more non-transitory computer-readable media as described in Example 17, wherein the memory loop nest is a first memory loop nest, and generating the multiple scheduling plans for the data structure also includes determining one or more permutations for a second memory loop nest in the multiple loop nests, each permutation indicating a change in the order of the loops in the second memory loop nest; determining a loop extension of the loop in the second memory loop nest based on the one or more permutations; and further generating the multiple scheduling plans based on the one or more permutations of the second memory loop nest and the loop extension of the loop in the second memory loop nest.

[0186] Example 19 provides one or more non-transitory computer-readable media as described in Example 18, wherein determining the loop extension of the loop in the second memory loop nest includes determining a plurality of candidate sets, each candidate set including a candidate loop extension of the loop in the second memory loop nest; for each respective candidate set, inputting the candidate loop extensions in the candidate set and one or more attributes of the data structure into an additional trained model, the additional trained model outputting a miss score indicating a prediction miss of a memory corresponding to the second memory loop nest when the DNN performs the tensor operation based on the data structure in which the loop nest is transformed based on the respective candidate set; and selecting a candidate set from the plurality of candidate sets based on the miss scores of the plurality of candidate sets, wherein the candidate set includes the loop extension of the loop in the second memory loop nest.

[0187] Example 20 provides one or more non-transitory computer-readable media as described in any of Examples 11-19, wherein one or more attributes of the data structure after being transformed by a corresponding scheduling plan are selected from a group consisting of: the type of the tensor operation, the loop nesting extension indicating the number of times the DNN is to repeat the tensor operation, the tensor rank associated with the tensor operation, the tensor shape associated with the tensor operation, and the tensor length associated with the tensor operation.

[0188] Example 21 provides a device for deep learning, which includes a computer processor for repeatedly performing tensor operations according to computer program instructions; and a tensor compiler configured to perform operations, the operations including generating multiple scheduling plans for a data structure including a loop nest, wherein the data structure is generated based on the computer program instructions, the loop nest includes multiple loops, the loop specifies the tensor operation to be repeatedly performed by the processor, and the scheduling plan specifies the transformation of the loop nest, for each corresponding scheduling plan in the multiple scheduling plans, one or more attributes of the data structure after the corresponding scheduling plan is transformed are input into a trained model, and the trained model outputs a predicted performance score indicating an evaluation of the predicted performance of the processor; selecting a scheduling plan from the multiple scheduling plans based on the predicted performance scores of the multiple scheduling plans, transforming the loop nest in the data structure based on the scheduling plan, wherein after the loop nest is transformed based on the scheduling plan, the data structure is used by the processor to perform the tensor operation, receiving information indicating the runtime performance of the processor when performing the tensor operation, and updating the training model based on the evaluation of the runtime performance of the processor.

[0189] Example 22 provides an apparatus as described in Example 21, wherein updating the training model based on an evaluation of the runtime performance of the processor includes determining a runtime performance score indicative of an evaluation of the runtime performance of the processor; forming a training sample including the runtime performance score and one or more parameters associated with the scheduling plan; and further updating the trained model by using the training sample.

[0190] Example 23 provides an apparatus as described in Example 21 or 22, wherein the operation further includes selecting a different scheduling plan from the multiple scheduling plans by using the updated trained model; and transforming the loop nest in the data structure based on the different scheduling plan, wherein after the loop nest is transformed based on the different scheduling plan, the data structure is used for a new execution of the tensor operation by the processor.

[0191] Example 24 provides an apparatus as described in any one of Examples 21-23, wherein generating the multiple scheduling plans for the data structure includes determining whether the data structure is in a data structure category in a database; in response to determining that the data structure is in the data structure category, retrieving candidate scheduling plans associated with the data structure category from the database; and generating the multiple scheduling plans from the candidate scheduling plans.

[0192] Example 25 provides an apparatus as described in any one of Examples 21-24, wherein generating the multiple scheduling plans for the data structure includes partitioning the loop nest into multiple memory loop nests, wherein each of the loop nests and the memory loop nests includes a sequence of loops, the loops indicating tensor operations to be repeatedly performed by the processor, the loops in the loop nests being partitioned into multiple loops, wherein each loop is in a different memory loop nest in the multiple memory loop nests and corresponds to a different memory associated with the processor; determining a loop extension of a loop in a memory loop nest in the multiple loop nests, the loop extension of the loop indicating the number of times the tensor operation in the loop is to be repeatedly performed; and generating the multiple scheduling plans based on the loop extension.

[0193] Select Example

[0194] Example 1 provides a method for deep learning, the method comprising determining whether a data structure belongs to a data structure category, wherein the data structure includes a loop nest, the loop nest includes multiple loops, and the loop specifies a tensor operation to be repeatedly performed by a DNN; in response to determining that the data structure is in the data structure category, retrieving multiple scheduling plan trees associated with the data structure category, wherein each scheduling plan tree includes one or more scheduling plans, and the scheduling plan specifies a loop transformation of the loop nest; and generating a new scheduling plan tree for the data structure by merging at least some of the multiple scheduling plan trees, wherein the new scheduling plan tree is to be used to transform the loop nest.

[0195] Example 2 provides a method as described in Example 1, wherein determining whether the data structure belongs to the data structure category includes determining whether the tensor operation matches a tensor operation in the data structure category.

[0196] Example 3 provides a method as described in Example 2, wherein determining whether the data structure belongs to the data structure category also includes determining whether one or more tensor references in the data structure match corresponding one or more tensor references in the data structure category, wherein the tensor reference is a tensor order, a tensor shape, or a tensor length.

[0197] Example 4 provides a method as described in any of the preceding examples, further comprising selecting a group of scheduling plan trees from the multiple scheduling plan trees, wherein the group of scheduling plan trees is a subset of the multiple scheduling plan trees, and generating the new scheduling plan tree comprises generating the new scheduling plan tree by merging the group of scheduling plan trees.

[0198] Example 5 provides the method of Example 4, wherein selecting the group of scheduling plan trees from the plurality of scheduling plan trees includes removing one or more scheduling plan trees from the plurality of scheduling plan trees.

[0199] Example 6 provides a method as described in Example 5, wherein removing the one or more scheduling plan trees from the multiple scheduling plan trees includes determining a similarity score for each corresponding scheduling plan tree in the multiple scheduling plan trees, the similarity score indicating the similarity between the data structure and the corresponding data structure in the multiple data structures; and removing the one or more scheduling plan trees based on the similarity scores of the one or more scheduling plan trees.

[0200] Example 7 provides a method as in any preceding example, further comprising, in response to determining that the data structure is not in the data structure category, generating a scheduling plan tree for the data structure based on the loop nest.

[0201] Example 8 provides one or more non-transitory computer-readable media storing instructions that are executable to perform operations for deep learning, the operations comprising determining whether a data structure belongs to a data structure category, wherein the data structure includes a loop nest, the loop nest includes multiple loops, and the loops specify tensor operations to be repeatedly performed by the DNN; in response to determining that the data structure is in the data structure category, retrieving multiple scheduling plan trees associated with the data structure category, wherein each scheduling plan tree includes one or more scheduling plans, and the scheduling plan specifies a loop transformation of the loop nest; and generating a new scheduling plan tree for the data structure by merging at least some of the multiple scheduling plan trees, wherein the new scheduling plan tree is to be used to transform the loop nest.

[0202] Example 9 provides one or more non-transitory computer-readable media as described in Example 8, wherein determining whether the data structure belongs to the data structure category includes determining whether the tensor operation matches a tensor operation in the data structure category.

[0203] Example 10 provides one or more non-transitory computer-readable media as described in Example 9, wherein determining whether the data structure belongs to the data structure category also includes determining whether one or more tensor references in the data structure match corresponding one or more tensor references in the data structure category, wherein the tensor reference is a tensor rank, a tensor shape, or a tensor length.

[0204] Example 11 provides one or more non-transitory computer-readable media as described in any one of Examples 8-10, wherein the operation also includes selecting a group of scheduling plan trees from the multiple scheduling plan trees, wherein the group of scheduling plan trees is a subset of the multiple scheduling plan trees, and generating the new scheduling plan tree includes generating the new scheduling plan tree by merging the group of scheduling plan trees.

[0205] Example 12 provides one or more non-transitory computer-readable media as described in Example 11, wherein selecting the group of scheduling plan trees from the plurality of scheduling plan trees includes removing one or more scheduling plan trees from the plurality of scheduling plan trees.

[0206] Example 13 provides one or more non-transitory computer-readable media as described in Example 12, wherein removing the one or more scheduling plan trees from the multiple scheduling plan trees includes determining a similarity score for each corresponding scheduling plan tree in the multiple scheduling plan trees, the similarity score indicating the similarity between the data structure and the corresponding data structure in the multiple data structures; and removing the one or more scheduling plan trees based on the similarity scores of the one or more scheduling plan trees.

[0207] Example 14 provides one or more non-transitory computer-readable media as described in any of Examples 8-13, wherein the operation further includes generating a scheduling plan tree for the data structure based on the loop nest in response to determining that the data structure is not in the data structure category.

[0208] Example 15 provides a device for deep learning, which includes a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions, which can be executed by the computer processor to perform operations, the operations including determining whether a data structure belongs to a data structure category, wherein the data structure includes a loop nest, the loop nest includes multiple loops, and the loop specifies a tensor operation to be repeatedly performed by a DNN, in response to determining that the data structure is in the data structure category, retrieving multiple scheduling plan trees associated with the data structure category, wherein each scheduling plan tree includes one or more scheduling plans, and the scheduling plan specifies a loop transformation of the loop nest, and generating a new scheduling plan tree for the data structure by merging at least some of the multiple scheduling plan trees, wherein the new scheduling plan tree is to be used to transform the loop nest.

[0209] Example 16 provides an apparatus as described in Example 15, wherein determining whether the data structure belongs to the data structure category includes determining whether the tensor operation matches a tensor operation in the data structure category.

[0210] Example 17 provides an apparatus as described in Example 16, wherein determining whether the data structure belongs to the data structure category also includes determining whether one or more tensor references in the data structure match corresponding one or more tensor references in the data structure category, wherein the tensor reference is a tensor rank, a tensor shape, or a tensor length.

[0211] Example 18 provides an apparatus as described in any one of Examples 15-17, further comprising selecting a group of scheduling plan trees from the multiple scheduling plan trees, wherein the group of scheduling plan trees is a subset of the multiple scheduling plan trees, and generating the new scheduling plan tree comprises generating the new scheduling plan tree by merging the group of scheduling plan trees.

[0212] Example 19 provides the apparatus of Example 18, wherein selecting the group of scheduling plan trees from the plurality of scheduling plan trees comprises removing one or more scheduling plan trees from the plurality of scheduling plan trees.

[0213] Example 20 provides an apparatus as described in Example 19, wherein removing the one or more scheduling plan trees from the multiple scheduling plan trees includes determining a similarity score for each corresponding scheduling plan tree in the multiple scheduling plan trees, the similarity score indicating the similarity between the data structure and a corresponding data structure in the multiple data structures; and removing the one or more scheduling plan trees based on the similarity scores of the one or more scheduling plan trees.

[0214] The above description of the illustrated implementations of the present disclosure, including those described in the Abstract, is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Although specific implementations of the present disclosure and examples thereof are described herein for illustrative purposes, as will be appreciated by those skilled in the relevant art, various equivalent modifications are possible within the scope of the present disclosure. These modifications may be made to the present disclosure in accordance with the above detailed description.

Claims

1. A method for deep learning, the method comprising: generating a plurality of schedules for a data structure comprising a loop nest, wherein the loop nest comprises a plurality of loops, the loops specifying tensor operations to be repeatedly performed by a deep neural network (DNN), and the schedules specify transformations of the loop nest; For each corresponding schedule in the plurality of schedules, inputting one or more attributes of the data structure after being transformed by the corresponding schedule into a trained model, the trained model outputting a prediction performance score indicating an evaluation of the prediction performance of the DNN; selecting a schedule from the plurality of schedules based on predicted performance scores of the plurality of schedules; transforming the loop nests in the data structure based on the scheduling plan, wherein the data structure is used by the DNN to perform the tensor operation after the loop nests are transformed based on the scheduling plan; receiving information indicating a runtime performance of the DNN when performing the tensor operation; and The trained model is updated based on an evaluation of the runtime performance of the DNN.

2. The method of claim 1, wherein: Updating the training model based on the evaluation of the runtime performance of the DNN includes: determining a runtime performance score indicating an assessment of the runtime performance of the DNN; forming a training sample comprising the runtime performance score and one or more parameters associated with the scheduling plan; and The trained model is further trained by using the training samples.

3. The method of claim 1, further comprising: selecting a different schedule from the plurality of schedules by using the updated trained model; and The loop nests in the data structure are transformed based on the different scheduling plan, wherein the data structure is used for a new execution of the tensor operation by the DNN after the loop nests are transformed based on the different scheduling plan.

4. The method of claim 1, wherein: Generating the plurality of scheduling plans for the data structure includes: determining whether the data structure is within a data structure category in a database; In response to determining that the data structure is in the data structure category, retrieving from the database a candidate schedule associated with the data structure category; and The plurality of schedules are generated from the candidate schedules.

5. The method of claim 4, wherein: The candidate scheduling plans include a first candidate scheduling plan and a second candidate scheduling plan, and generating the plurality of scheduling plans for the data structure includes: determining a similarity score indicating a similarity between the data structure and a data structure associated with the first candidate schedule; and After determining that the similarity score is below a threshold score, the plurality of schedules are generated based on the second candidate schedule and not based on the first candidate schedule.

6. The method of claim 4, wherein: The candidate scheduling plans include a first candidate scheduling plan and a second candidate scheduling plan, the first candidate scheduling plan specifies a loop tile for splitting a loop in the loop nest into a plurality of loops, and generating the plurality of scheduling plans for the data structure includes: determining that the loop tiling and a tensor associated with the loop are incompatible; and The multiple scheduling plans are generated based on the second candidate scheduling plans and not based on the first candidate scheduling plans.

7. The method of claim 1, wherein: Generating the plurality of scheduling plans for the data structure includes: partitioning the loop nest into a plurality of memory loop nests, wherein each of the loop nest and the memory loop nest comprises a sequence of loops indicating tensor operations to be repeatedly performed by a processor, the loops in the loop nest being partitioned into a plurality of loops, wherein each loop is in a different memory loop nest of the plurality of memory loop nests and corresponds to a different memory associated with the processor; determining a loop extension of a loop in a memory loop nest of the plurality of loop nests, the loop extension of the loop indicating a number of times a tensor operation in the loop is to be repeatedly performed; and The plurality of schedules are generated based on the cycle extension.

8. The method of claim 7, wherein: The memory loop nest is a first memory loop nest, and generating the plurality of schedules for the data structure further comprises: determining one or more permutations for a second memory loop nest of the plurality of loop nests, each permutation indicating a change in an order of loops in the second memory loop nest; determining a loop extension of a loop in the second memory loop nest based on the one or more permutations; and The plurality of schedules are generated further based on the one or more permutations of the second memory loop nest and loop extensions of loops in the second memory loop nest.

9. The method of claim 8, wherein: Determining the loop extension of the loop in the second memory loop nest includes: determining a plurality of candidate sets, each candidate set comprising candidate loop extensions of loops in the second memory loop nest; for each respective candidate set, inputting candidate loop stretches in the candidate set and the one or more properties of the data structure into an additional trained model, the additional trained model outputting a miss score indicating a prediction miss of the memory corresponding to the second memory loop nest when the DNN performs the tensor operation based on the data structure in which the loop nest is transformed based on the respective candidate set; and A candidate set is selected from the plurality of candidate sets based on the miss scores of the plurality of candidate sets, wherein the candidate set includes loop extensions of loops in the second memory loop nest.

10. The method of claim 1, wherein: The one or more attributes of the data structure after being transformed by the corresponding scheduling plan are selected from the group consisting of: the type of the tensor operation, the loop nesting extension indicating the number of times the DNN is to repeat the tensor operation, the tensor rank associated with the tensor operation, the tensor shape associated with the tensor operation, and the tensor length associated with the tensor operation.

11. One or more non-transitory computer-readable media storing instructions executable to perform operations for deep learning, the operations comprising: generating a plurality of schedules for a data structure comprising a loop nest, wherein the loop nest comprises a plurality of loops, the loops specifying tensor operations to be repeatedly performed by a deep neural network (DNN), and the schedules specify transformations of the loop nest; For each corresponding schedule in the plurality of schedules, inputting one or more attributes of the data structure after being transformed by the corresponding schedule into a trained model, the trained model outputting a prediction performance score indicating an evaluation of the prediction performance of the DNN; selecting a schedule from the plurality of schedules based on predicted performance scores of the plurality of schedules; transforming the loop nests in the data structure based on the scheduling plan, wherein the data structure is used by the DNN to perform the tensor operation after the loop nests are transformed based on the scheduling plan; receiving information indicating a runtime performance of the DNN when performing the tensor operation; and The trained model is updated based on an evaluation of the runtime performance of the DNN.

12. The one or more non-transitory computer readable media of claim 11, wherein: Updating the training model based on the evaluation of the runtime performance of the DNN includes: determining a runtime performance score indicating an assessment of the runtime performance of the DNN; forming a training sample comprising the runtime performance score and one or more parameters associated with the scheduling plan; and The trained model is further trained by using the training samples.

13. One or more non-transitory computer readable media as recited in claim 12, wherein: The operation also includes: selecting a different schedule from the plurality of schedules by using the updated trained model; and The loop nests in the data structure are transformed based on the different scheduling plan, wherein the data structure is used for a new execution of the tensor operation by the DNN after the loop nests are transformed based on the different scheduling plan.

14. The one or more non-transitory computer readable media of claim 11, wherein: Generating the plurality of scheduling plans for the data structure includes: determining whether the data structure is within a data structure category in a database; In response to determining that the data structure is in the data structure category, retrieving from the database a candidate schedule associated with the data structure category; and The plurality of schedules are generated from the candidate schedules.

15. The one or more non-transitory computer readable media of claim 14, wherein: The candidate scheduling plans include a first candidate scheduling plan and a second candidate scheduling plan, and generating the plurality of scheduling plans for the data structure includes: determining a similarity score indicating a similarity between the data structure and a data structure associated with the first candidate schedule; and After determining that the similarity score is below a threshold score, the plurality of schedules are generated based on the second candidate schedule and not based on the first candidate schedule.

16. The one or more non-transitory computer readable media of claim 14, wherein: The candidate scheduling plans include a first candidate scheduling plan and a second candidate scheduling plan, the first candidate scheduling plan specifies a loop tile for splitting a loop in the loop nest into a plurality of loops, and generating the plurality of scheduling plans for the data structure includes: determining that the loop tiling and a tensor associated with the loop are incompatible; and The multiple scheduling plans are generated based on the second candidate scheduling plans and not based on the first candidate scheduling plans.

17. The one or more non-transitory computer readable media of claim 11, wherein: Generating the plurality of scheduling plans for the data structure includes: partitioning the loop nest into a plurality of memory loop nests, wherein each of the loop nest and the memory loop nest comprises a sequence of loops indicating tensor operations to be repeatedly performed by a processor, the loops in the loop nest being partitioned into a plurality of loops, wherein each loop is in a different memory loop nest of the plurality of memory loop nests and corresponds to a different memory associated with the processor; determining a loop extension of a loop in a memory loop nest of the plurality of loop nests, the loop extension of the loop indicating a number of times a tensor operation in the loop is to be repeatedly performed; and The plurality of schedules are generated based on the cycle extension.

18. The one or more non-transitory computer readable media of claim 17, wherein: The memory loop nest is a first memory loop nest, and generating the plurality of schedules for the data structure further comprises: determining one or more permutations for a second memory loop nest of the plurality of loop nests, each permutation indicating a change in an order of loops in the second memory loop nest; determining a loop extension of a loop in the second memory loop nest based on the one or more permutations; and The plurality of schedules are generated further based on the one or more permutations of the second memory loop nest and loop extensions of loops in the second memory loop nest.

19. The one or more non-transitory computer readable media of claim 18, wherein: Determining the loop extension of the loop in the second memory loop nest includes: determining a plurality of candidate sets, each candidate set comprising candidate loop extensions of loops in the second memory loop nest; for each respective candidate set, inputting candidate loop stretches in the candidate set and the one or more properties of the data structure into an additional trained model, the additional trained model outputting a miss score indicating a prediction miss of the memory corresponding to the second memory loop nest when the DNN performs the tensor operation based on the data structure in which the loop nest is transformed based on the respective candidate set; and A candidate set is selected from the plurality of candidate sets based on the miss scores of the plurality of candidate sets, wherein the candidate set includes loop extensions of loops in the second memory loop nest.

20. The one or more non-transitory computer readable media of claim 11, wherein: The one or more attributes of the data structure after being transformed by the corresponding scheduling plan are selected from the group consisting of: the type of the tensor operation, the loop nesting extension indicating the number of times the DNN is to repeat the tensor operation, the tensor rank associated with the tensor operation, the tensor shape associated with the tensor operation, and the tensor length associated with the tensor operation.

21. A device for deep learning, the device comprising: a computer processor for repeatedly performing tensor operations in accordance with computer program instructions; as well as A tensor compiler is configured to perform operations, the operations comprising: generating a plurality of schedules for a data structure comprising a loop nest, wherein the data structure is generated based on the computer program instructions, the loop nest comprising a plurality of loops, the loops specifying the tensor operations to be repeatedly performed by the processor, and the schedules specifying transformations of the loop nests, For each corresponding schedule of the plurality of schedules, inputting one or more attributes of the data structure after being transformed by the corresponding schedule into a trained model, the trained model outputting a predicted performance score indicating an assessment of predicted performance of the processor; selecting a schedule from the plurality of schedules based on the predicted performance scores of the plurality of schedules, transforming the loop nests in the data structure based on the schedule, wherein the data structure is used by the processor to perform the tensor operation after the loop nests are transformed based on the schedule, receiving information indicating a runtime performance of the processor when performing the tensor operation, and The training model is updated based on an evaluation of the runtime performance of the processor.

22. The device of claim 21, wherein: Updating the training model based on the evaluation of the runtime performance of the processor includes: determining a runtime performance score indicative of an assessment of a runtime performance of the processor; forming a training sample comprising the runtime performance score and one or more parameters associated with the scheduling plan; and The trained model is further trained by using the training samples.

23. The device of claim 21, wherein: The operations also include: selecting a different schedule from the plurality of schedules by using the updated trained model; and The loop nests in the data structure are transformed based on the different schedule, wherein the data structure is used for a new execution of the tensor operation by the processor after the loop nests are transformed based on the different schedule.

24. The device of claim 21, wherein: Generating the plurality of scheduling plans for the data structure includes: determining whether the data structure is within a data structure category in a database; In response to determining that the data structure is in the data structure category, retrieving from the database a candidate schedule associated with the data structure category; and The plurality of schedules are generated from the candidate schedules.

25. The apparatus of claim 21, wherein: Generating the plurality of scheduling plans for the data structure includes: partitioning the loop nest into a plurality of memory loop nests, wherein each of the loop nest and the memory loop nest comprises a sequence of loops indicating tensor operations to be repeatedly performed by a processor, the loops in the loop nest being partitioned into a plurality of loops, wherein each loop is in a different memory loop nest of the plurality of memory loop nests and corresponds to a different memory associated with the processor; determining a loop extension of a loop in a memory loop nest of the plurality of loop nests, the loop extension of the loop indicating a number of times a tensor operation in the loop is to be repeatedly performed; and The plurality of schedules are generated based on the cycle extension.

Images