Apparatus, method, system and storage medium for deep learning operations on workloads having dynamic shapes

Abstract

Apparatuses, methods, systems, and storage media for deep learning operations on workloads having dynamic shapes are provided. The apparatus comprises an interface circuit module configured to obtain a configuration file of a tensor shape; and a processing circuitry module coupled to the interface circuitry module and configured to: infer a configuration file of parallel index ranges and a probability of each of the parallel index ranges from the configuration file of a tensor shape; determining a set of candidate partitioning factors for each of the parallel index ranges; and generating a kernel for the deep learning operation under each of the candidate partitioning factors, the kernel for the operation including parallel outer and inner loops, in which a portion of the deep learning operation that were working on a corresponding portion of the workload is verified within a single core.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to PCT International Application No. PCT / CN2022 / 107707, filed on July 25, 2022, entitled “HIGH PERFORMANCE DEEP LEARNING KERNEL GENERATIONWITH PARTIAL SHAPE INFORMATION FOR DEEP LEARNING WORKLOAD”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] Embodiments described herein relate generally to artificial intelligence (AI) and machine learning (ML) techniques, and in particular, to devices, methods, systems, and storage media for performing deep learning operations on workloads with dynamic shapes. Background Art

[0004] Deep learning compilers are often used as the backend of deep learning frameworks as runtime just-in-time (JIT) compilers. To improve efficiency, deep learning compilers are specialized for compiling for specific input tensor shapes. Typically, compiled code is cached so that the compiled code can be reused for the same tensor shape. Unknown tensor shapes trigger a new compilation process that caches the compiled code for future use.

[0005] The input tensor shapes to a deep learning model may change at runtime. For example, in a cloud deployment scenario, a service may buffer different numbers of requests and submit the buffered requests as batches to a deep learning model with different batch sizes. The sentence length of a natural language processing model may change. The number of possible objections to an objection detection model may also change.

[0006] This approach leads to two problems: long compilation delays for unknown shapes and bloated code caches. When the cached compiled code reaches its limit, the compiled code is often recycled, which potentially leads to more compilations and reduces overall performance.

[0007] For every dynamic shape scenario in deep learning (such as machine vision, natural language processing, etc.), not every dimension of the input tensor shape changes at runtime. For example, the dimensions associated with model parameters are often constant, which presents potential opportunities for improving efficiency and optimizing the performance of deep learning operations. Summary of the invention

[0008] According to one aspect of the present disclosure, a device for performing deep learning operations on workloads with dynamic shapes is provided. The device includes an interface circuit module, the interface circuit module is configured to obtain a profile of a tensor shape; and a processing circuit module, the processing circuit module is coupled to the interface circuit module and is configured to: infer a profile of a parallel index range and a probability of each parallel index range in the parallel index range based on the profile of the tensor shape; determine a set of candidate blocking factors for each parallel index range in the parallel index range; and generate a kernel for the deep learning operation under each candidate blocking factor in the candidate blocking factors, the kernel for the operation includes a parallel outer loop and an inner loop, wherein a single-core kernel implements a portion of the deep learning operation working on a corresponding portion of the workload.

[0009] According to another aspect of the present disclosure, a method for performing deep learning operations on a workload with a dynamic shape is provided. The method includes: inferring a profile of a parallel index range and a probability of each parallel index range in the parallel index range based on a profile of a tensor shape; determining a set of candidate blocking factors for each parallel index range in the parallel index range; and generating a kernel for the deep learning operation at each candidate blocking factor in the candidate blocking factors, the kernel for the operation including a parallel outer loop and an inner loop, wherein a single-core kernel implements a portion of the deep learning operation working on a corresponding portion of the workload.

[0010] Another aspect of the present disclosure provides a machine-readable storage medium having instructions stored thereon, which, when executed by a machine, causes the machine to perform the above-mentioned method for performing deep learning operations on a workload with a dynamic shape.

[0011] Another aspect of the present disclosure provides a computing device including components for implementing the above method for performing deep learning operations on workloads with dynamic shapes. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In the accompanying drawings, which are not necessarily drawn to scale, like reference numerals may describe similar components in different views. Like reference numerals with different letter suffixes may represent different instances of similar components. Some embodiments are shown by way of example and not limitation in the figures of the accompanying drawings, in which:

[0013] Figure 1 An overview of a system for performing deep learning operations on workloads with dynamic shapes according to some embodiments of the present disclosure is shown;

[0014] Figure 2 shows exemplary pseudo code that may be used to determine a set of candidate blocking factors for each parallel index range according to some embodiments of the present disclosure;

[0015] Figure 3 An exemplary kernel implementation of computationally intensive operations of a traditional deep learning compilation approach is shown;

[0016] Figure 4 shows a schematic diagram illustrating an implementation of parallelizing computationally intensive operations on input tensors having dynamic shapes according to some embodiments of the present disclosure;

[0017] Figure 5 shows exemplary pseudo code describing how a parallel loop of a parallel index may be transformed to parallelize a reduction index according to some embodiments of the present disclosure;

[0018] Figure 6 shows an exemplary pseudo code for calculating the kernel cost of each candidate kernel according to some embodiments of the present disclosure;

[0019] Figure 7 shows exemplary pseudo code for selecting a blocking factor from a set of predefined blocking factors for a determined parallel index range according to some embodiments of the present disclosure;

[0020] Fig. 8A A flowchart of a process for generating a kernel for deep learning operations on a workload with a dynamic shape according to some embodiments of the present disclosure is shown;

[0021] Figure 8B The use of some embodiments according to the present disclosure is shown by Fig. 8A A flowchart of the process by which a kernel generated by the process performs a deep learning operation on an input tensor;

[0022] Fig. 9 shows an exemplary pseudo code for generating a kernel for a Matmul operation by a conventional compiler according to some embodiments of the present disclosure;

[0023] Fig.10 shows an exemplary pseudo code for generating a kernel for a Matmul operation for which a portion of the shape information is unknown according to some embodiments of the present disclosure;

[0024] Fig.11 is a block diagram illustrating components capable of reading instructions from a machine-readable or computer-readable medium and performing any one or more of the methods discussed herein, according to some example embodiments; and

[0025] Fig.12 is a block diagram of an example processor platform according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0026] The various aspects of the illustrative embodiments will be described using terms commonly used by those skilled in the art to convey the essence of the present disclosure to other persons skilled in the art. However, it will be apparent to those skilled in the art that many alternative embodiments may be practiced using portions of the described aspects. For purposes of explanation, specific quantities, materials, and configurations are set forth to provide a thorough understanding of the illustrative embodiments. However, it will be apparent to those skilled in the art that alternative embodiments may be practiced without specific details. In other cases, well-known features may have been omitted or simplified to avoid blurring the illustrative embodiments.

[0027] In addition, various operations will be described as multiple discrete operations in turn in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be interpreted as implying that these operations are necessarily order-dependent. In particular, these operations do not need to be performed in the order presented.

[0028] The phrases "in an embodiment," "in one embodiment," and "in some embodiments" are used repeatedly herein. The phrases generally do not refer to the same embodiment; however, it may. The terms "comprising," "having," and "including" are synonymous unless the context dictates otherwise. The phrases "A or B" and "A / B" mean "(A), (B), or (A and B)."

[0029] Deep learning workloads spend a significant amount of time on compute-intensive operations such as convolution (Conv) and general matrix multiplication (GEMM). For many deep learning application deployments, having efficient cores for each operation can be critical.

[0030] In order to obtain the best code efficiency for compute-intensive kernels on artificial intelligence (AI) hardware with parallel processing units, deep learning compilers need to consider parallelism at the whole system level and compute efficiency at the processing unit level. When a deep learning compiler decomposes a compute-intensive kernel into subtasks, the decomposition needs to ensure that there is a sufficient number of subtasks for load balancing across all processing units, and the subtask size must be large enough to efficiently use a single parallel processing unit. However, for a given computation size, these two optimization goals may be contradictory, with a larger number of subtasks leading to a smaller subtask size, and a larger subtask size leading to a smaller number of subtasks. Existing deep learning compilers or performance libraries require both the input and output tensor shapes to determine the blocking factor, and then generate specialized code for the given shape.

[0031] One of the previous solutions is to recompile the Conv or GEMM kernel for the input tensor with the new shape, and cache the compiled code so that it can be reused for another input tensor with the same shape. With this solution, under the hood, if the application is involved in a new compilation, it may experience long delays. Sometimes, this may cause user experience issues because the end user may feel that the application is not running smoothly. In addition, over-compilation may cause excessive memory usage of the compiled code, which may cause out-of-memory failures or harm application performance.

[0032] In another previous solution, called the "library solution", the Conv or GEMM kernel is statically pre-built, and therefore does not involve the recompilation overhead of the Conv or GEMM kernel. However, the library solution can only implement Conv and GEMM operations, and cannot compile and execute graphs of deep neural network (DNN) operations. As an example, the Mathematical Kernel Library (MKL) provides a GEMM function with dynamic shape support, but does not allow the GEMM operation to be fused with the neighbor operation. The static pre-built kernel library has an additional binary size and will not be applied to use cases where the user wants to reduce the size of the compiled model group including the dependent library. In addition, the static pre-built kernel library cannot compile the entire graph, and therefore loses performance opportunities.

[0033] In related fields, use cases for processing workloads with dynamic shapes are increasing. In order to solve at least some of the problems raised by the two previous solutions mentioned, an embodiment of the present disclosure provides a method for generating a kernel for performing deep learning operations (such as Conv operations, GEMM operations, DNN graph operations, etc.) on workloads with dynamic shapes. Optionally, as a specific implementation, part of the shape information of the dynamic shape can be known at compile time. For example, some dimensions of the input tensor can be specified as constant, and other dimensions can be dynamically changeable. The present disclosure can be used to reduce repeated compilation without significantly sacrificing the performance of the compiled code.

[0034] In some embodiments, a range of single-core kernels is first built, which are dedicated to predefined shape sizes. A subset of single-core kernels can be selected from the range of single-core kernels based on partial shape information, and a high-performance parallel code with multiple optional paths with these selected single-core kernels can be composed. At execution time, the blocking factor can be determined based on the complete shape information of the input tensor, and then an optimal path is selected accordingly. The compiler can fuse the high-performance code for dynamic tensor shapes generated by the method of the present disclosure with neighbor operations within the DNN graph. The techniques of the present disclosure can minimize the compilation time of models with dynamic input tensor shapes and can be used to support ahead-of-time compilation use cases. These techniques can be integrated into any deep learning compiler (such as oneDNN Graph Compiler Beta) to improve the performance of the deep learning compiler.

[0035] This disclosure describes a method for generating high-performance deep learning kernels with unknown tensor shapes. This disclosure focuses on computationally intensive operations, such as matrix multiplication (Matmul) and convolution (Conv), but can be applied to any operation that can be expressed as a nested loop consisting of parallel loops and reduction loops.

[0036] Figure 1 An overview of a system 100 for performing deep learning operations on workloads with dynamic shapes according to some embodiments of the present disclosure is shown.

[0037] It should be noted that the “deep learning operation” used in this article may refer to any operation involved in a deep learning application, such as a Conv operation, a GEMM operation, a DNN graph operation, etc., without any specific limitation.

[0038] The number, capabilities, and / or capacity of the elements of the system 100 may vary, depending on whether the system 100 is used as a fixed computing device (e.g., a server computer in a data center, a workstation, a desktop computer, etc.) or a mobile computing device (e.g., a smart phone, a tablet computing device, a laptop computer, a game console, an Internet of Things (IoT) device, etc.). In various implementations, the system 100 may include one or more components in a data center, a desktop computer, a workstation, a laptop computer, a smart phone, a tablet computer, a digital camera, a smart appliance, a smart home hub, a network appliance, and / or any other device / system that processes data.

[0039] As a simplified example, system 100 includes input / output (I / O) interface(s) 110 , a pre-processing component 120 , a compiler 130 , and an executor 140 .

[0040] The I / O interface(s) 110 may be configured to receive input data for deep learning operations from a memory / storage device or an input device, and output results of deep learning operations to a memory / storage device or an output device.

[0041] In some embodiments, one or more memory / storage devices may be included in the system 100, or may be coupled to the system 100. The memory / storage device may include a main memory, a disk storage device, or any suitable combination thereof. The memory / storage device may include, but is not limited to, any type of volatile or non-volatile memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a solid-state storage device, etc.

[0042] In some embodiments, various I / O devices may be present within the system 100 or connected to the system 100 via (one or more) I / O interfaces 110. Input devices may include any physical or virtual components for accepting input, including, among other things, one or more physical or virtual buttons (e.g., a reset button), a physical keyboard, a keypad, a mouse, a touchpad, a touch screen, a microphone, a scanner, a headset, etc. Output devices may be included to show or otherwise communicate information, such as sensor readings, (one or more) actuator positions, or other similar information. Data and / or graphics may be displayed on an external device. The output devices may include any number and / or combination of audio or visual displays, including, in particular, one or more simple visual outputs / indicators (e.g., binary state indicators (e.g., light emitting diodes (LEDs)) and multi-character visual outputs, or more complex outputs such as a display device or touch screen (e.g., a liquid crystal display (LCD), LED display, quantum dot display, projector, etc.), where output of characters, graphics, multimedia objects, etc. is generated or produced from the operation of platform 450. The output devices may also include speakers and / or other audio emitting devices, (one or more) printers, etc. Additionally or alternatively, (one or more) sensors may be used as input devices (e.g., image capture devices, motion capture devices, etc.), and one or more actuators may be used as output devices (e.g., actuators that provide tactile feedback, etc.).

[0043] The input data of the deep learning operation may include data indicating shape information of the operator of the deep learning operation, such as a tensor describing the operator. In order to perform deep learning operations on workloads with dynamic shapes, it is assumed in the present disclosure that the input data includes a tensor shape profile describing the shape of each potential tensor of the workload.

[0044] As used herein, the term "tensor shape profile" may be interpreted as including all possible tensor shapes that a person skilled in the relevant art can think of. In practice, a person skilled in the relevant art can select an appropriate cluster of tensor shapes to use as a tensor shape profile.

[0045] After receiving the tensor shape profile, the pre-processing component 120 analyzes the tensor shape profile to infer a parallel index range profile and a probability of each parallel index range according to the tensor shape profile.

[0046] For a given parallel index p0,…,p x , the system 100 attempts to minimize the number of blocking factors (denoted as ф). From the set {PB 0 ..PB ф Each blocking factor of} represents the combination of the blocking factors from each parallel index, for example, PB 0 It can be expressed as {PB 0 0..PB 0 x Assuming there are N possible ways to chunk each parallel index, the number of possible combinations will be exponential in N and the number of parallel indices, X, i.e. N x Since each combination is associated with a kernel implementation, the large number of combinations can result in very large binary sizes.

[0047] In this disclosure, it is assumed that the profile of the parallel index range can be expressed as {P 0 ..P θ}, and the probability of each parallel index range can be expressed as {Prob 0 ..Prob θ}.

[0048] The pre-processing component 120 determines a set of candidate blocking factors for each parallel index range. For example, the pre-processing component 120 can select a set of candidate blocking factors that provide overall good performance for all possible shapes without bloating the binary size, for example, based on the estimated kernel cost corresponding to each potential blocking factor for each parallel index range. The pre-processing component 120 can determine a set of candidate blocking factors for each parallel index range using a blocking heuristic that identifies a blocking factor for a given shape, number of cores, and microarchitecture information.

[0049] An exemplary algorithm by which the pre-processing component 120 determines a set of candidate blocking factors for each parallel index range can be described as follows.

[0050] Algorithm: Select multiple blocking factors {PB 0 ..PB ф}, which for a given profile of a parallel index range {P 0 ..P θ}The effect is best.

[0051] Input: Configuration file for parallel index range {P 0 ..P θ} and the associated probability {Prob 0 ..Prob θ}; ф as the total number of parallel index blocking factor combinations; R as the reduced index range, uArch_info for the microarchitecture information, and core_num for the number of cores.

[0052] Output: Block factor {PB 0 ..PB ф}, which is estimated as the number of parallel index ranges for a given profile {P 0 ..P θ} has the overall lowest kernel cost.

[0053] Figure 2 Exemplary pseudo code that may be used to determine a set of candidate blocking factors for each parallel index range according to some embodiments of the present disclosure is shown.

[0054] After determining a set of candidate blocking factors for each parallel index range, compiler 130 is invoked to generate a kernel for each blocking factor in the set of candidate blocking factors for each parallel index range.

[0055] Assume that if there are θ parallel index ranges and ф candidate blocking factors for each parallel index range, there should be θф kernels to generate.

[0056] In an example, the compiler 130 may include a kernel generator 132 for generating each kernel. For the kernel generator 132, when generating a specific kernel, the parallel index range, the reduction index range, and the blocking factor are all known, and it can generate the specific kernel just like performing a deep learning operation on an input tensor with a static shape.

[0057] The following examples illustrate how to perform deep learning operations on input tensors with static shapes.

[0058]

[0059] Equation (1) shows the general form of a mathematical equation for performing a deep learning operation on input tensors A and B, and C is the result of the operation. The deep learning operation can be any of Matmul, Conv, pooling, element-wise operations, etc. The index of the output tensor C is called the parallel index, denoted as p0,…,px , where P0, .., P x are the index range boundaries and the reduction indices r0,...,r used to perform the reduction t is called, where R0, .., R t The input tensors A and B are indexed by tensor indices i0,...,i y or j0,…,j z to index.

[0060] In equation (1), each index p0,..,p x With index range p0∈{0...P0-1},...p x ∈{0…PX-1}, each index r0,..,r t With index range r0∈{0…R0-1},...r t ∈{0…RT-1}, and from i0,...,i y Each index i or from j0,…,j z The j of all are p0,...,p x and r0,...,r t The affine expression of is α*p+β*r+γ.

[0061] To simplify the description, the present disclosure uses p to represent any parallel index, and P is its index range upper boundary, uses r to represent any reduced index, and R is its index range upper boundary, and similarly uses i and j to represent tensor indexes. Tensor indexes can be mapped to parallel indexes, reduced indexes, or a combination thereof. For example, for the input tensor of 2D Matmul, one matrix index is mapped to a parallel index, and the other is mapped to a reduced index. For the activation input tensor of 3D Conv, the tensor index of a certain dimension consists of both a parallel index and a reduced index. In general, each tensor index can be expressed as an affine expression of a parallel index and a reduced index. In particular, in the deep learning domain, the reduced index range is usually constant, and the parallel index range can change as the shape of the input tensor changes.

[0062] Equation (1) means that for every possible point in the index space of p and r, the multiplication result of the tensor elements from A and B associated with that point is calculated, and then the multiplication result is reduced along the reduction index to produce C.

[0063] Figure 3An exemplary kernel implementation of the computationally intensive operation of equation (1) performed by a conventional deep learning compilation method is shown. As shown in the figure, for a static shape input tensor, the tensor index range is given at compile time, from which the parallel index range (P) and the reduced index range (R) can be inferred. The compiler blocks P with a blocking factor PB and splits the parallel index p into p # and p b , where p = p # *PB+p b . With parallel index p # The outer loop of is fully parallelized and has a parallel index p b The inner loop of is executed on each core. The mathematical equations in the inner loop represent the single-core kernel, which implements some of the original compute-intensive operations on the input and output sub-tensors. Because the parallel index p b Both the upper bounds of the reduction index r are known, and a single-core kernel implementation is dedicated to these index ranges and performs blocking and unblocking for the cache hierarchy and register file.

[0064] Figure 4 A schematic diagram illustrating an implementation of parallelizing computationally intensive operations on input tensors with dynamic shapes according to some embodiments of the present disclosure is shown.

[0065] Will combine Figure 1 To describe Figure 4 The compiler 130 may obtain a specific parallel index range P and a set of candidate blocking factors {PB for the specific parallel index range P. 0 ..PB ф}, and then switch the candidate blocking factor {PB 0 ..PB ф}. With a specific parallel index range P and each candidate blocking factor (as shown in the figure, p b ∈PB 0 ,p b ∈PB 1 ,…,p b ∈PB φ ) The corresponding kernels can be generated in parallel by the kernel generator 132. Similar to the static shape case, each generated kernel can include a parallel outer loop and an inner loop, where a single-core kernel implements a portion of the deep learning operation working on the corresponding portion of the input tensor.

[0066] The compiler 130 may also include a composer 134 for composing the kernels generated by the kernel generator 132 into compiled code for execution.

[0067] A simple compiler implementation generates all possible kernels and then dispatches to the kernel with the best estimated performance at runtime. To improve the performance of the compiler, in some embodiments, a portion of the shape information may be known at compile time. Part of the shape information can be used to reduce the selection of blocking factors and corresponding single-core kernels to reduce binary size. For example, in the case of deep learning, where the reduction index is constant and the blocking factors for each parallel index are predefined, they determine the shape of the input and output sub-tensors of the corresponding single-core kernel, so the single-core kernel can be pre-built and dedicated to the constant index. The blocking factor candidates ensure that the tensor shape can be further blocked to align with the register and cache line sizes to adapt to the hardware, and ensure that the overall size is large enough to allow the single-core kernel to be efficient. Based on the pre-built single-core kernel, the compiler generates an entire kernel with a parallel outer loop that divides any given parallel index range into many sub-ranges and adapts them to the single-core kernel. For each possible combination of the predefined blocking factors for each parallel index, there is a corresponding kernel.

[0068] In some embodiments, there may be such a scenario, where the reduction index loop should also be blocked because not enough parallel subtasks are created for the parallel index loop blocking. In this scenario, the reduction index of the deep learning operation (such as, the K dimension of the Matmul operation) can be blocked with a technique called K-slicing. For example, the compiler 130 can calculate the single-core kernel cost of the single-core kernel obtained by blocking only the parallel index loop, and determine whether the single-core kernel cost exceeds a threshold. The compiler 130 can calculate the single-core kernel cost of the single-core kernel based on any combination of the corresponding blocking factor (PB), the corresponding reduction index range (R) and the micro-architecture information (uArch_info). The single_kernel_cost can be adjusted to reflect the additional reduction operation and the additional memory use of the single-core kernel. The threshold can be predetermined and can be related to the size of the subtask and the number of subtasks, which is not limited in this article. If the single-core kernel cost exceeds the threshold, the reduction index should be sliced ​​to create more parallel subtasks. The preprocessing component 120 can infer a profile of reduced index ranges and a probability of each reduced index range based on a profile of the tensor shape, and determine a set of candidate slicing factors for each reduced index range; and the kernel generator 132 can generate a kernel for a deep learning operation under a combination of each candidate blocking factor and each candidate slicing factor.

[0069] This K-slicing technique can be applied to any reduction index that has a direct mapping to the tensor indices of the two input tensors.

[0070] There may also be extreme scenarios where the parallel index range of a deep learning operation is very small but the reduction index range is large, such as a Matmul operation with a large K dimension and relatively small M or N dimensions. In such scenarios, chunking the parallel index may not be effective, and the reduction index should be chunked to create enough parallel subtasks.

[0071] Figure 5 An exemplary pseudocode is shown describing how a parallel loop of a parallel index (p) is converted to parallelize a reduction index (r0) according to some embodiments of the present disclosure. In this case, the reduction index r0 can be considered to be promoted to a parallel index. The original parallel loop can be considered as a special case where the blocking factor of the reduction index is the same as the size of the reduction index range, and therefore no blocking of the reduction index actually occurs. For the reduction index promoted to a parallel index, the single_kernel_cost needs to be adjusted to reflect the additional reduction operation and the additional memory usage of the single-core kernel.

[0072] Back to Figure 1 After compiler 130 generates compiled code, executor 140 may execute the compiled code to perform deep learning operations on a specific workload.

[0073] When executed, the I / O interface 110 may obtain an input tensor on which a deep learning operation is to be performed from a memory or an input device. The preprocessing component 120 may determine shape information of the input tensor and determine a parallel index range based on the shape information of the input tensor.

[0074] After receiving the determined parallel index range, the executor 140 can find a set of candidate blocking factors and candidate kernels corresponding to the candidate blocking factors for the parallel index range in the compiled code, estimate the kernel cost of each corresponding candidate kernel, and assign one of the corresponding candidate kernels with the minimum kernel cost to perform a deep learning operation on the input tensor.

[0075] Figure 6 An exemplary pseudo code for calculating the kernel cost of each corresponding candidate kernel according to some embodiments of the present disclosure is shown.

[0076] The executor 140 may dispatch the kernels based on the estimated kernel costs, eg, selecting the kernel with the lowest estimated kernel cost. Figure 7 FIG. 1 shows a method for determining a parallel index range P from a set of predefined blocking factors {PB} according to some embodiments of the present disclosure. 0 ..PB ф Example pseudo code for selecting the blocking factor PB in}.

[0077] Under the condition that the parallel index range is not divisible by the blocking factor of the dispatched kernel or is not evenly divisible by the blocking factor of the dispatched kernel (i.e., the reminder of the parallel index range for the blocking factor of the dispatched kernel is not zero), the executor is also configured to pad the slices of the input tensor (so as to be in a computer-friendly size) before feeding them into the single-core kernel of the dispatched kernel, and to unpad the obtained results before saving them back to the corresponding slices of the output tensor. Through padding and unpadding, the generated code can handle any dynamic tensor shape at the expense of handling unused data elements. The padding and unpadding costs can be minimized on hardware, which performs maskload for padding and maskstore for masking out unused data elements.

[0078] Using system 100, the present disclosure can operate workloads with dynamic shapes through multiple kernel implementations, and each kernel implementation has a single-core kernel with static input and output tensor shapes. In this way, each single-core kernel with a static tensor shape can be fused with neighbor operations. Therefore, the techniques described herein can be used to support the fusion of compute-intensive operations with neighbor memory-intensive operations.

[0079] The compiler 130 and the executor 140 of the system 100 may be implemented in hardware, software, or a combination thereof, and they may be implemented on one processor / processing circuit module, or on different processors / processing circuit modules, which is not limited herein.

[0080] Fig. 8A A flow chart of a process 800a for generating kernels for deep learning operations on workloads with dynamic shapes is shown, according to some embodiments of the present disclosure. Figure 8B FIG. 8 is a flowchart of a process 800b for performing a deep learning operation on an input tensor using a kernel generated by process 800a according to some embodiments of the present disclosure. Processes 800a and 800b may be performed, for example, by Figure 1 The system 100 is implemented by one or more processors of any computing device. Examples of processors are described in Fig.12 Shown in.

[0081] like Fig. 8A As shown in , process 800a includes, at block 810, inferring a profile of parallel index ranges and a probability of each parallel index range based on the obtained tensor shape profile. In the present disclosure, it is assumed that a tensor shape profile of a workload with a dynamic shape can be obtained from a memory / storage device or an input device.

[0082] Process 800a includes determining a set of candidate blocking factors for each parallel index range at block 820. For example, a set of candidate blocking factors for each parallel index range may be determined based on an estimated kernel cost corresponding to each potential blocking factor for each parallel index range.

[0083] Process 800a includes, at block 830, generating a kernel for a deep learning operation at each candidate blocking factor. The kernel for the operation may include a parallel outer loop and an inner loop, wherein a single-core kernel implements a portion of the deep learning operation working on a corresponding portion of the workload.

[0084] like Figure 8B As shown in , process 800b includes, at block 840, obtaining an input tensor on which a deep learning operation is to be performed.

[0085] Process 800b includes, at block 850, determining a parallel index range based on shape information of an input tensor.

[0086] Process 800b includes, at block 860, determining a set of candidate blocking factors for the parallel index range and corresponding candidate kernels from the kernels generated by process 800a for the deep learning operation.

[0087] Process 800b includes, at block 870, estimating a kernel cost for each corresponding candidate kernel.

[0088] Process 800b includes, at block 880, assigning one of the corresponding candidate kernels having a minimum kernel cost to perform a deep learning operation on the input tensor.

[0089] More specifically, Fig. 8A Process 800a and Figure 8B The process 800b may be implemented in one or more modules as a set of logic instructions stored in a machine-readable or computer-readable storage medium (such as a random access memory (RAM), a read-only memory (ROM), a programmable ROM (PROM), firmware, flash memory, etc.), in configurable logic (such as, for example, a programmable logic array (PLA), a field programmable gate array (FPGA), a complex programmable logic device (CPLD)), in fixed-functionality logic hardware using circuit technology (such as, for example, an application specific integrated circuit (ASIC), complementary metal oxide semiconductor (CMOS), or transistor-transistor logic (TTL) technology), or in any combination thereof.

[0090] For example, to implement Fig. 8A Process 800a and Figure 8BThe computer program code for the operations shown in process 800b of can be written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as JAVA, SMALLTALK, C++, etc.) and conventional procedural programming languages ​​(such as "C" programming language or similar programming languages). In addition, the logic instructions may include assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, state setting data, configuration data of integrated circuits, state information that personalizes electronic circuit modules and / or other structural components native to the hardware (e.g., host processor, central processing unit / CPU, microcontroller, etc.).

[0091] To facilitate understanding of the technology described herein, a specific implementation using the technology is provided below.

[0092] For the two-dimensional (2D) Matmul operation, there are two operators, i.e., matrix A with indexes m and k, and matrix B with indexes k and n, for which the shape of B is known and the shape of A is unknown. For example, the ranges K and N of indexes k and n are 1024 and 2048 respectively, and the range M of index m is a runtime constant.

[0093] Fig. 9 An exemplary pseudocode for generating a kernel for a Matmul operation by a conventional compiler according to some embodiments of the present disclosure is shown. Assume that the input matrix shape is [M, K], [K, N], and the output matrix shape is [M, N]. A conventional compiler may decompose the Matmul operation into a nested loop nest of the matmul microkernel. M, K, N are provided at compile time, so the compiler generates specialized code based on the shape information. First, the compiler uses a heuristic method to block the M, N, K dimensions by blocking factors MB, NB, and KB. In Fig. 9 In the pseudocode of , m_o, n_o, and k_o are used as loop indices for the blocked dimensions. A[m_o,0:MB-1,k_o,0:KB-1] refers to the submatrix of size [MB-1,KB-1] indexed by m_o and k_o in the original matrix A, and the matmul_micro kernel processes the submatrices of A and B and outputs the submatrix of C on a single processing unit.

[0094] As can be seen, Fig. 9 The code generated in includes microkernels that are specialized according to the blocking factors MB, NB and KB. These blocking factors determine the input submatrix shapes of the microkernels. For each input submatrix shape, the microkernel is specialized because the internal implementation of the loop assumes that the input submatrix shape does not change.

[0095] The blocking heuristics identify the blocking factor for a given shape, number of cores, and microarchitecture information. Since M, N, K are provided at compile time, the compiler uses heuristics to decide the blocking factor that fits the given shape size, and then generates a nested loop dedicated to the blocking factor.

[0096] Fig.10 An exemplary pseudocode for generating a kernel for a Matmul operation according to some embodiments of the present disclosure is shown, for which a portion of the shape information is unknown. As described above, for the Matmul operation, N=1024, K=2048, and M is a runtime constant. According to the techniques provided herein, a set of predefined blocking factors for parallel indexes are selected as [1, 2, 4, 8, 16, 32, 48, 64]. Therefore, the complete combination of blocking factors for parallel indexes (m and n) is 64. Using partial shape information, the compiler is able to select some combinations from the complete set, for example, 3 combinations that may correspond to 3 predefined kernels. Using the complete shape information, the generated code first determines the final blocking factor from the 3 blocking factors, and selects the corresponding case containing the kernel implementation with a parallel loop with a single-core kernel.

[0097] The select_PB_with_lowest_kernel_cost() function selects a specific blocking factor from a set of candidate blocking factors associated with an operation. The criteria used will select the blocking factor that gives the lowest kernel_cost. The kernel then switches to an implementation dedicated to the selected blocking factor.

[0098] According to the above embodiment, a range of single-core kernels is first constructed, which is dedicated to a predefined shape size, a subset of single-core kernels can be selected from the range of single-core kernels based on partial shape information, and a high-performance parallel code with multiple optional paths with these selected single-core kernels can be composed. At execution time, the blocking factor can be determined based on the complete shape information of the input tensor, and then an optimal path is selected accordingly. The compiler can fuse the high-performance code for dynamic tensor shapes generated by the method of the present disclosure with neighbor operations within the DNN graph. The technology disclosed in the present disclosure can minimize the compilation time of models with dynamic input tensor shapes, and can be used to support ahead-of-time compilation use cases.

[0099] Fig.11 is a block diagram illustrating components capable of reading instructions from a machine-readable or computer-readable medium (e.g., a non-transitory machine-readable storage medium) and performing any one or more of the methods discussed herein, according to some example embodiments. Specifically, Fig.11A simplified diagrammatic representation of hardware resources 1100 is shown, including one or more processors (or processor cores) 1110, one or more memory / storage devices 1120, and one or more communication resources 1130, each of which may be communicatively coupled via a bus 1140. For embodiments utilizing node virtualization (e.g., NFV), a hypervisor 1102 may be executed to provide an execution environment for one or more network slices / sub-slices to utilize the hardware resources 1100.

[0100] Processor 1110 may include, for example, processor 1112 and processor 1114, which may be, for example, a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP) (such as a baseband processor), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof.

[0101] The memory / storage device 1120 may include a main memory, a disk storage device, or any suitable combination thereof. The memory / storage device 1120 may include, but is not limited to, any type of volatile or non-volatile memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a solid-state storage device, etc.

[0102] The communication resources 1130 may include an interconnect or network interface component or other suitable means for communicating with one or more peripheral devices 1104 or one or more databases 1106 via the network 1108. For example, the communication resources 1130 may include a wired communication component (e.g., for coupling via a universal serial bus (USB)), a cellular communication component, an NFC component, Components (e.g. Low energy), components and other communication components.

[0103] The instructions 1150 may include software, programs, applications, applet, apps, or other executable code for causing at least any processor 1110 to perform any one or more of the methods discussed herein. The instructions 1150 may reside in whole or in part in at least one processor 1110 (e.g., in a cache memory of a processor), memory / storage 1120, or any suitable combination thereof. In addition, any portion of the instructions 1150 may be transferred to the hardware resources 1100 from any combination of the peripheral device 1104 or the database 1106. Therefore, the memory of the processor 1110, the memory / storage 1120, the peripheral device 1104, and the database 1106 are examples of computer-readable and machine-readable media.

[0104] Fig.12 1 is a block diagram of an example processor platform according to some embodiments of the present disclosure. Processor platform 1200 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smart phone, a tablet computer such as an iPad™), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a game console, a personal video recorder, a set-top box, a headset or other wearable device, or any other type of computing device.

[0105] The processor platform 1200 of the illustrated example includes a processor 1212. The processor 1212 of the illustrated example is hardware. For example, the processor 1212 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired series or manufacturer. The hardware processor can be a semiconductor-based (e.g., silicon-based) device. In some embodiments, the processor implements one or more of the above methods or processes.

[0106] The processor 1212 of the illustrated example includes a local memory 1213 (e.g., a cache). The processor 1212 of the illustrated example communicates with a main memory including a volatile memory 1214 and a non-volatile memory 1216 via a bus 1218. The volatile memory 1214 may be comprised of a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory The non-volatile memory 1216 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1214, 1216 is controlled by a memory controller.

[0107] The processor platform 1200 of the illustrated example also includes an interface circuit module 1220. The interface circuit module 1220 can be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), interface, a near field communication (NFC) interface, and / or a PCI Express interface.

[0108] In the example shown, one or more input devices 1222 are connected to the interface circuit module 1220. The input device(s) 1222 allow a user to enter data and / or commands into the processor 1212. The input device(s) may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a track pad, a trackball, and / or a voice recognition system.

[0109] One or more output devices 1224 are also connected to the interface circuit module 1220 of the illustrated example. The output device 1224 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-situ switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Therefore, the interface circuit module 1220 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0110] The interface circuit module 1220 of the illustrated example also includes a communication device (such as a transmitter, receiver, transceiver, modem, residential gateway, wireless access point and / or network interface) for facilitating the exchange of data with an external machine (e.g., any kind of computing device) via a network 1226. Communication can be via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a field wireless system, a cellular telephone system, etc.

[0111] For example, the interface circuit module 1220 may include a training data set input through the input device(s) 1222 or retrieved from the network 1226 .

[0112] The processor platform 1200 of the illustrated example also includes one or more mass storage devices 1228 for storing software and / or data. Examples of such mass storage devices 1228 include floppy disk drives, hard drive disks, compact disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0113] The machine-executable instructions 1232 may be stored in the mass storage device 1228, in the volatile memory 1214, in the non-volatile memory 1216, and / or on a removable non-transitory computer-readable storage medium such as a CD or DVD.

[0114] The following paragraphs describe examples of various embodiments.

[0115] Example 1 includes a device for performing deep learning operations on a workload with a dynamic shape, comprising: an interface circuit module, the interface circuit module being configured to obtain a profile of a tensor shape; and a processing circuit module, the processing circuit module being coupled to the interface circuit module and being configured to: infer a profile of a parallel index range and a probability of each parallel index range in the parallel index range based on the profile of the tensor shape; determine a set of candidate blocking factors for each parallel index range in the parallel index range; and generate a kernel for the deep learning operation under each candidate blocking factor in the candidate blocking factors, the kernel for the operation comprising a parallel outer loop and an inner loop, wherein a single-core kernel implements a portion of the deep learning operation working on a corresponding portion of the workload.

[0116] Example 2 includes a device according to Example 1, wherein the processing circuit module is further configured to determine the set of candidate blocking factors for each parallel index range in the parallel index range based on an estimated kernel cost corresponding to each potential blocking factor for each parallel index range in the parallel index range.

[0117] Example 3 includes a device according to Example 1 or Example 2, wherein the processing circuit module is further configured to: estimate the single-core kernel cost of the single-core kernel based on any combination of the corresponding blocking factor, the corresponding reduced index range and microarchitecture information; and determine whether the single-core kernel cost exceeds a threshold.

[0118] Example 4 includes a device according to Example 3, wherein, under the condition that the single-core kernel cost exceeds the threshold, the processing circuit module is further configured to: infer a profile of a reduced index range and a probability of each reduced index range in the reduced index range based on the profile of the tensor shape; determine a set of candidate slicing factors for each reduced index range in the reduced index range; and generate the kernel for the deep learning operation under a combination of each candidate blocking factor in the candidate blocking factors and each candidate slicing factor in the candidate slicing factors.

[0119] Example 5 includes an apparatus according to any of Examples 1-4, wherein a portion of the shape information of the dynamic shape is known at compile time to reduce the number of candidate blocking factors and thereby reduce the number of candidate kernels.

[0120] Example 6 includes a device according to any one of Examples 1-5, wherein the interface circuit module is further configured to obtain an input tensor on which the deep learning operation is to be performed; and the processing circuit module is further configured to: determine a parallel index range based on shape information of the input tensor; determine a set of candidate blocking factors and corresponding candidate kernels for the parallel index range; estimate the kernel cost of each corresponding candidate kernel in the corresponding candidate kernels; and dispatch one of the corresponding candidate kernels with the smallest kernel cost to perform the deep learning operation on the input tensor.

[0121] Example 7 includes an apparatus according to Example 6, wherein the kernel cost of each of the corresponding candidate kernels is determined by any combination of the parallel index range, the corresponding blocking factor, the reduced index range, microarchitecture information, and the number of cores.

[0122] Example 8 includes a device according to Example 6, wherein, under the condition that the parallel index range is not divisible by the blocking factor of the dispatched kernel, the processing circuit module is also configured to fill the slice of the input tensor before feeding the slice to the single-core kernel of the dispatched kernel, and to defill the obtained result before saving the obtained result back to the corresponding slice of the output tensor.

[0123] Example 9 includes the apparatus of Example 8, wherein the padding and de-padding are implemented by hardware instructions maskload and maskstore, respectively.

[0124] Example 10 includes an apparatus according to any of Examples 1-9, wherein the deep learning operation is allowed to be fused with a neighbor operation.

[0125] Example 11 includes the apparatus of Example 10, wherein the deep learning operation comprises a compute-intensive operation and the neighbor operation comprises a memory-intensive operation.

[0126] Example 12 includes an apparatus according to any one of Examples 1-11, wherein the set of candidate blocking factors for each of the parallel index ranges is selected from 1, 2, 4, 8, 16, 32, 48 and 64.

[0127] Example 13 includes a method for performing deep learning operations on a workload with a dynamic shape, comprising: inferring a profile of parallel index ranges and a probability of each parallel index range in the parallel index ranges based on a profile of a tensor shape; determining a set of candidate blocking factors for each parallel index range in the parallel index ranges; and generating a kernel for the deep learning operation at each candidate blocking factor in the candidate blocking factors, the kernel for the operation comprising a parallel outer loop and an inner loop, wherein a single-core kernel implements a portion of the deep learning operation working on a corresponding portion of the workload.

[0128] Example 14 includes a method according to Example 13, wherein the set of candidate blocking factors for each parallel index range in the parallel index range is determined based on an estimated kernel cost corresponding to each potential blocking factor for each parallel index range in the parallel index range.

[0129] Example 15 includes the method according to Example 13 or Example 14, further comprising: estimating the single-core kernel cost of the single-core kernel based on any combination of the corresponding blocking factor, the corresponding reduced index range and microarchitecture information; and determining whether the single-core kernel cost exceeds a threshold.

[0130] Example 16 includes the method according to Example 15, further comprising: under the condition that the single-core kernel cost exceeds the threshold: inferring a profile of the reduced index range and a probability of each reduced index range in the reduced index range based on the profile of the tensor shape; determining a set of candidate slicing factors for each reduced index range in the reduced index range; and generating the kernel for the deep learning operation under the combination of each candidate blocking factor in the candidate blocking factors and each candidate slicing factor in the candidate slicing factors.

[0131] Example 17 includes the method of any of Examples 13-16, wherein a portion of the shape information of the dynamic shape is known at compile time to reduce the number of candidate blocking factors and, in turn, the number of candidate kernels.

[0132] Example 18 includes a method according to any one of Examples 13-17, further comprising: obtaining an input tensor on which the deep learning operation is to be performed; determining a parallel index range based on shape information of the input tensor; determining a set of candidate blocking factors and corresponding candidate kernels for the parallel index range; estimating the kernel cost of each corresponding candidate kernel in the corresponding candidate kernels; and dispatching one of the corresponding candidate kernels with the smallest kernel cost to perform the deep learning operation on the input tensor.

[0133] Example 19 includes the method according to Example 18, wherein the kernel cost of each of the corresponding candidate kernels is determined by any combination of the parallel index range, the corresponding blocking factor, the reduced index range, microarchitecture information, and the number of cores.

[0134] Example 20 includes a method according to Example 18, wherein, under the condition that the parallel index range is not divisible by the blocking factor of the dispatched kernel, the method further includes padding the slice of the input tensor before feeding the slice to the single-core kernel of the dispatched kernel, and de-padding the obtained result before saving the obtained result back to the corresponding slice of the output tensor.

[0135] Example 21 includes the method according to 20, wherein the filling and defilling are implemented by hardware instructions maskload and maskstore, respectively.

[0136] Example 22 includes a method according to any of Examples 13-21, wherein the deep learning operation is allowed to be fused with a neighbor operation.

[0137] Example 23 includes a method according to any of Examples 13-22, wherein the deep learning operation includes a compute-intensive operation and the neighbor operation includes a memory-intensive operation.

[0138] Example 24 includes a method according to any one of Examples 13-23, wherein the set of candidate blocking factors for each of the parallel index ranges is selected from 1, 2, 4, 8, 16, 32, 48 and 64.

[0139] Example 25 includes a machine-readable storage medium having instructions stored thereon, which, when executed by a machine, cause the machine to perform a method according to any one of Examples 13 to 24.

[0140] Example 26 includes a computing device comprising components for performing the method according to any one of Examples 13 to 24.

[0141] Example 27 includes a device comprising one or more processors configured to implement one or more of the processes as shown and described in the specification.

[0142] Example 28 includes a method comprising one or more of the processes as shown and described in the specification.

[0143] Example 29 includes a system comprising one or more memories for storing computer readable instructions for implementing one or more of the processes as shown and described in the specification.

[0144] The above description is intended to be illustrative and not restrictive. For example, the above examples (or one or more aspects thereof) may be used in conjunction with each other. Other embodiments may be used, such as by a person of ordinary skill in the art when reviewing the above description. The abstract is to allow the reader to quickly determine the nature of the technical disclosure, and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the above specific embodiments, various features may be grouped together to simplify the disclosure. This should not be interpreted as intended to be necessary for any claim to disclose features that are not claimed. On the contrary, the subject matter of the present invention may be less than all the features of a particular disclosed embodiment. Therefore, the accompanying claims are thus incorporated into the specific embodiments, with each claim itself as a separate embodiment. The scope of the embodiment should be determined with reference to the attached claims together with the full scope of equivalents to which these claims confer rights.

[0145] Although certain embodiments have been shown and described herein for purposes of description, various alternative and / or equivalent embodiments or implementations calculated to achieve the same purpose may be substituted for the embodiments shown and described without departing from the scope of the present disclosure. The present disclosure is intended to cover any adaptation or variation of the embodiments discussed herein. Therefore, it is expressly intended that the embodiments described herein are limited only by the appended claims and their equivalents.

Claims

1. A device for performing deep learning operations on a workload having a dynamic shape, the device comprising: An interface circuit module, wherein the interface circuit module is configured to obtain a configuration file of a tensor shape; as well as a processing circuit module, the processing circuit module being coupled to the interface circuit module and being configured to: inferring a profile of parallel index ranges and a probability of each parallel index range in the parallel index ranges based on the profile of the tensor shape; determining a set of candidate blocking factors for each of the parallel index ranges; as well as A kernel for the deep learning operation is generated at each of the candidate blocking factors, the kernel for the operation comprising a parallel outer loop and an inner loop, wherein a single-core kernel implements a portion of the deep learning operation working on a corresponding portion of the workload.

2. The device according to claim 1, wherein The processing circuit module is further configured to determine the set of candidate blocking factors for each of the parallel index ranges based on the estimated kernel cost corresponding to each potential blocking factor for each of the parallel index ranges.

3. The device according to claim 1, wherein: The processing circuit module is also configured to: estimating a single-core kernel cost for the single-core kernel based on any combination of the corresponding blocking factor, the corresponding reduced index range, and microarchitectural information; and A determination is made as to whether the single core kernel cost exceeds a threshold.

4. The device according to claim 3, wherein: Under the condition that the single-core kernel cost exceeds the threshold, the processing circuit module is further configured to: inferring a profile of reduced index ranges and a probability of each reduced index range in the reduced index ranges based on the profile of the tensor shape; determining a set of candidate slicing factors for each of the reduced index ranges; as well as The kernel for the deep learning operation is generated at a combination of each of the candidate blocking factors and each of the candidate slicing factors.

5. The device according to claim 1, wherein: A portion of the shape information of the dynamic shape is known at compile time to reduce the number of candidate blocking factors and, in turn, the number of candidate kernels.

6. The device according to claim 1, wherein: The interface circuit module is further configured to obtain an input tensor on which the deep learning operation is to be performed; and The processing circuit module is also configured to: Determine a parallel index range based on shape information of the input tensor; Determining a set of candidate blocking factors and corresponding candidate kernels for the parallel index range; estimating a kernel cost for each of the corresponding candidate kernels; as well as One of the corresponding candidate kernels having a minimum kernel cost is dispatched to perform the deep learning operation on the input tensor.

7. The device according to claim 6, wherein: The kernel cost of each of the corresponding candidate kernels is determined by any combination of the parallel index range, the corresponding blocking factor, the reduced index range, microarchitecture information, and the number of cores.

8. The device according to claim 6, wherein: Under the condition that the parallel index range cannot be divided by the blocking factor of the dispatched kernel, the processing circuit module is also configured to pad the slice of the input tensor before feeding the slice to the single-core kernel of the dispatched kernel, and de-pad the obtained result before saving the obtained result back to the corresponding slice of the output tensor.

9. The device according to claim 8, wherein: The filling and defilling are implemented by hardware instructions maskload and maskstore respectively.

10. The device according to any one of claims 1 to 9, wherein: The deep learning operations are allowed to be fused with neighbor operations.

11. The device according to claim 10, wherein: The deep learning operation comprises a computationally intensive operation, and the neighbor operation comprises a memory intensive operation.

12. The device according to any one of claims 1 to 9, wherein: The set of candidate blocking factors for each of the parallel index ranges is selected from 1, 2, 4, 8, 16, 32, 48 and 64.

13. A method for performing deep learning operations on a workload having a dynamic shape, the method comprising: Inferring a profile of parallel index ranges and a probability of each parallel index range in the parallel index ranges based on the profile of the tensor shape; determining a set of candidate blocking factors for each of the parallel index ranges; as well as A kernel for the deep learning operation is generated at each of the candidate blocking factors, the kernel for the operation comprising a parallel outer loop and an inner loop, wherein a single-core kernel implements a portion of the deep learning operation working on a corresponding portion of the workload.

14. The method according to claim 13, wherein: The set of candidate blocking factors for each of the parallel index ranges is determined based on the estimated kernel cost corresponding to each potential blocking factor for each of the parallel index ranges.

15. The method according to claim 13, further comprising: estimating a single-core kernel cost for the single-core kernel based on any combination of the corresponding blocking factor, the corresponding reduced index range, and microarchitectural information; as well as A determination is made as to whether the single core kernel cost exceeds a threshold.

16. The method according to claim 15, further comprising: Under the condition that the single core cost exceeds the threshold: inferring a profile of reduced index ranges and a probability of each reduced index range in the reduced index ranges based on the profile of the tensor shape; determining a set of candidate slicing factors for each of the reduced index ranges; as well as The kernel for the deep learning operation is generated at a combination of each of the candidate blocking factors and each of the candidate slicing factors.

17. The method according to claim 13, wherein: A portion of the shape information of the dynamic shape is known at compile time to reduce the number of candidate blocking factors and, in turn, the number of candidate kernels.

18. The method according to claim 13, further comprising: obtaining an input tensor on which the deep learning operation is to be performed; Determine a parallel index range based on shape information of the input tensor; Determining a set of candidate blocking factors and corresponding candidate kernels for the parallel index range; estimating a kernel cost for each of the corresponding candidate kernels; as well as One of the corresponding candidate kernels having a minimum kernel cost is dispatched to perform the deep learning operation on the input tensor.

19. The method according to claim 18, wherein: The kernel cost of each of the corresponding candidate kernels is determined by any combination of the parallel index range, the corresponding blocking factor, the reduced index range, microarchitecture information, and the number of cores.

20. The method according to claim 18, wherein: Under the condition that the parallel index range cannot be divided by the blocking factor of the dispatched kernel, the method also includes padding the slice of the input tensor before feeding the slice to the single-core kernel of the dispatched kernel, and de-padding the obtained result before saving the obtained result back to the corresponding slice of the output tensor.

21. The method according to claim 20, wherein: The filling and defilling are implemented by hardware instructions maskload and maskstore respectively.

22. The method according to any one of claims 13 to 21, wherein: The deep learning operations are allowed to be fused with neighbor operations.

23. The method according to any one of claims 13 to 21, wherein: The set of candidate blocking factors for each of the parallel index ranges is selected from 1, 2, 4, 8, 16, 32, 48 and 64.

24. A machine-readable storage medium having stored thereon instructions which, when executed by a machine, cause the machine to perform the method according to any one of claims 13 to 23.

25. A computing device comprising means for performing the method according to any one of claims 13 to 23.

Images