Compilation optimization method and system for deep learning operators
Through thread scheduling algorithm and operator dynamic compiler, deep learning operators are mapped to hardware threads and compiled and optimized, which solves the problem of poor hardware architecture adaptability and significantly improves the performance of operators and the overall performance of the framework.
Patent Information
- Application Number
- CN202110677399.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-18
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-06-18
AI Technical Summary
The performance optimization methods of existing deep learning operators are not comprehensive enough and are difficult to adapt to various hardware architectures, resulting in difficulty in migration and optimization.
Using thread scheduling algorithm and operator dynamic compiler, the meta operator is mapped to hardware threads and compiled and optimized according to the hardware type, including strategies such as loop splitting, vectorization, cache prompts and thread allocation to generate compiled and optimized operators. .
The performance of deep learning operators is significantly improved, thereby improving the overall performance of the entire deep learning framework, adapting to different hardware and pursuing its maximum performance.
Smart Images

Figure CN113821208B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of deep learning frameworks, and in particular to a compilation optimization method and system for deep learning operators. Background Art
[0002] The deep learning framework is responsible for the training and reasoning of machine learning models, managing the large-scale data and models required for deep learning applications, and scheduling and resource application of the underlying computing devices. From the Theano framework invented in 2008 to Caffe, TensorFlow, and PyTorch, the deep learning framework is constantly developing in the direction of being easier to use, more complete, and more efficient. At the same time, new deep learning acceleration chips are also being updated, from the initial general-purpose chips such as the Central Processing Unit (CPU) and Graphics Processing Unit (GPU) to the Tensor Processing Unit (TPU), Machine Learning Unit (MLU), Huawei Ascend, and Graphcore. A complete deep learning framework often needs to support thousands of operators, and each operator needs to be adapted to a variety of different hardware. Therefore, different hardware architectures bring huge challenges to the migration and optimization of operators. In addition to hardware optimization, software optimization is also a key factor in improving performance. However, the existing performance optimization for operators is not comprehensive enough, and its optimization method needs to be further improved.
[0003] Therefore, there is an urgent need for a compilation optimization method and system for deep learning operators to solve the above problems. Summary of the invention
[0004] In view of the problems existing in the prior art, the present invention provides a compilation optimization method and system for deep learning operators.
[0005] The present invention provides a compilation optimization method for a deep learning operator, comprising:
[0006] According to the thread scheduling algorithm, the operators of each order in the meta-operator are mapped to the corresponding hardware threads through the operator dynamic compiler;
[0007] According to the type of the hardware thread and the compilation optimization strategy, the operator mapped to the hardware thread is compiled and optimized to obtain the compiled and optimized operator;
[0008] According to the compiled optimized operator, executable code is compiled.
[0009] According to a compilation optimization method for deep learning operators provided by the present invention, before mapping operators of each order in the meta-operator to corresponding hardware threads through the operator dynamic compiler according to the thread scheduling algorithm, the method further includes:
[0010] According to the preset compilation optimization requirements, a thread scheduling algorithm is constructed, wherein the thread scheduling algorithm includes a search strategy and a manual strategy;
[0011] The search strategy is determined based on the actual operating speed of the hardware, and the manual strategy is determined based on the hardware overhead.
[0012] According to a compilation optimization method for deep learning operators provided by the present invention, the search strategy includes a brute force enumeration strategy, a search loop reordering strategy and a loop splitting strategy; the manual strategy includes increasing memory access continuity, reducing atomic operations and reducing register pressure.
[0013] According to a compilation optimization method for a deep learning operator provided by the present invention, the compilation optimization strategy includes: loop splitting, vectorization, cache hint and thread allocation.
[0014] According to a compilation optimization method for deep learning operators provided by the present invention, the operator mapped to the hardware thread is compiled and optimized according to the type of the hardware thread and the compilation optimization strategy to obtain the compiled and optimized operator, including:
[0015] If the type of the hardware thread is a central processing unit, with the goal of maximizing the memory access speed of the central processing unit, loop splitting, vectorization and cache hinting are performed on the operator mapped to the hardware thread in sequence to obtain a compiled and optimized operator.
[0016] According to a compilation optimization method for deep learning operators provided by the present invention, the operator mapped to the hardware thread is compiled and optimized according to the type of the hardware thread and the compilation optimization strategy to obtain the compiled and optimized operator, and further includes:
[0017] If the type of the hardware thread is a graphics processor, the operator mapped to the hardware thread is thread-allocated with the goal of maximizing memory access continuity to obtain a compiled and optimized operator.
[0018] According to a compilation optimization method for deep learning operators provided by the present invention, the operator mapped to the hardware thread is compiled and optimized according to the type of the hardware thread and the compilation optimization strategy to obtain the compiled and optimized operator, and further includes:
[0019] If the type of the hardware thread is an artificial intelligence processor, with the goal of maximizing tensor processing performance, the operator mapped to the hardware thread is sequentially subjected to loop splitting, vectorization, and thread allocation to obtain a compiled and optimized operator.
[0020] The present invention also provides a compilation optimization system for deep learning operators, comprising:
[0021] A mapping module is used to map operators of each order in the meta-operator to corresponding hardware threads through the operator dynamic compiler according to the thread scheduling algorithm;
[0022] A compilation optimization module is used to compile and optimize the operator mapped to the hardware thread according to the type of the hardware thread and the compilation optimization strategy to obtain the compiled and optimized operator;
[0023] The compiling module is used to compile the optimized operator to obtain executable code.
[0024] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the compilation optimization method for deep learning operators as described in any one of the above are implemented.
[0025] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described compilation optimization methods for deep learning operators.
[0026] The compilation optimization method and system for deep learning operators provided by the present invention perform compilation optimization for deep learning operators, which can greatly improve the performance of deep learning operators, thereby improving the overall performance of the entire deep learning framework. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0028] Figure 1 A schematic diagram of the process flow of the compilation optimization method for deep learning operators provided by the present invention;
[0029] Figure 2 A schematic diagram of a CPU optimization strategy based on a broadcast operator provided by the present invention;
[0030] Figure 3A schematic diagram of a GPU optimization strategy based on a broadcast operator provided by the present invention;
[0031] Figure 4 A schematic diagram of the MLU optimization strategy based on the broadcast operator provided by the present invention;
[0032] Figure 5 A schematic diagram of the structure of the compilation optimization system for deep learning operators provided by the present invention;
[0033] Figure 6 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION
[0034] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0035] A complete deep learning framework often needs to support thousands of operators, and each operator needs to be adapted to various hardware. Therefore, different hardware architectures bring huge challenges to the migration and optimization of operators. In addition to hardware optimization, software optimization is also a key factor in improving operator performance. The software optimization strategies of deep learning frameworks can be roughly divided into the following points according to the granularity of their impact, from small to large: 1. Compilation optimization, by optimizing the execution logic of operators on hardware, give full play to the performance of underlying hardware; 2. Scheduling optimization, by optimizing the execution order and execution mode of operators, improve the throughput of the system and reduce latency; 3. Distributed optimization, by optimizing the allocation of computing tasks and communication overhead across nodes, improve the scalability of the system.
[0036] In software optimization, the main purpose of compilation optimization is to implement an operator for a specific task, such as convolution or matrix multiplication, and maximize the performance of the underlying hardware. Compilation optimization needs to take into account the differences in thread logic, cache structure and instruction set of various hardware. Maximizing performance often requires experienced developers to complete. Therefore, how to automatically compile and optimize the program to maximize performance has become a research focus. TVM (TensorVirtual Machine) proposes a set of description languages suitable for deep learning operators to automatically implement optimization strategies such as cache reuse and loop blocking; AutoTVM, based on TVM, realizes automatic optimization through machine learning, and can be comparable to the results of manual optimization; Ansor uses an evolutionary tree search algorithm to better optimize performance. MLIR is based on the LLVM architecture and designs an internal representation (IR) dedicated to deep learning on top of LLVM's IR. Its feature is that various dialects are designed to adapt to various special cases of deep learning.
[0037] Scheduling optimization focuses on the execution order and execution optimization of operators and models, and coordinates the application of resources so that key resources will not become bottlenecks and drag down the overall speed of the system. Rammer abstracts virtual execution units, merges several different operators into a kernel function, and dispatches it to different hardware threads for execution at the same time, greatly improving the hardware throughput. AntMan improves the utilization efficiency of GPU training clusters by performing targeted video memory optimization and scheduling on deep learning frameworks. PipeSwitch optimizes multi-task switching through pipeline algorithms, while improving the utilization of underlying hardware, reducing the overhead of task switching, and the overhead is less than the MPS (Multiprocess server) officially implemented by Nvidia. Clockwork discovered that some requests were delayed too high due to resource competition during the reasoning process. By introducing a global scheduler, the delay of reasoning can be greatly reduced, and there are few cases where some requests are delayed too high. The present invention combines hardware optimization with compilation optimization in software optimization to optimize deep learning operators, improve the performance of deep learning operators, and thus improve the overall performance of the entire deep learning framework. The machine learning model is obtained by training the deep learning framework after optimizing the operator based on the present invention, and the recognition and prediction efficiency in the corresponding fields can be further improved.
[0038] Figure 1 A flow chart of the compilation optimization method for deep learning operators provided by the present invention is shown in FIG. Figure 1 As shown, the present invention provides a compilation optimization method for a deep learning operator, comprising:
[0039] Step 101 , according to the thread scheduling algorithm, through the operator dynamic compiler, map the operators of each order in the meta-operator to the corresponding hardware threads.
[0040] In the present invention, the meta-operator written in Python is used for explanation. The operator dynamic compiler will first use the thread scheduling algorithm to map the operators of each order in the meta-operator to the corresponding hardware thread (or the model built based on the hardware thread) according to the operator content. In the present invention, the thread scheduling algorithm mainly performs thread allocation through search strategy and manual strategy, wherein the search strategy is to determine the thread allocation strategy based on the actual running speed of the hardware through strategies such as brute force enumeration, search loop reordering and loop splitting; the manual strategy is to complete performance tuning without increasing the hardware overhead by strategies such as increasing memory access continuity, reducing atomic operations and reducing register pressure.
[0041] Step 102, compile and optimize the operator mapped to the hardware thread according to the type of the hardware thread and the compile optimization strategy to obtain the compiled and optimized operator;
[0042] Step 103: compile and obtain executable code according to the compiled and optimized operator.
[0043] In the present invention, the operator dynamic compiler will perform intelligent optimization compilation for the CPU, GPU and AI tensor processing chip (i.e., artificial intelligence processor, including TPU and MLU, the present invention is explained with MLU, the compilation optimization of TPU is similar to the optimization process of MLU, and the compilation optimization process of MLU can be referred to) according to the hardware thread type. For the CPU, the operator dynamic compiler mainly performs optimization strategies such as loop splitting, vectorization and cache hints on the deep learning operator, so as to maximize the memory access speed of the CPU; for the GPU, the operator dynamic compiler performs reasonable allocation of threads and blocks according to the thread hierarchy of CUDA (Compute Unified Device Architecture) to maximize the continuity of memory access; for the AI tensor processing chip (MLU), the operator dynamic compiler performs optimization strategies such as loop splitting, vectorization and thread allocation on the deep learning operator, so as to maximize the tensor processing performance. Finally, the operator dynamic compiler compiles the optimized operator into executable code for specific hardware.
[0044] The compilation optimization method for deep learning operators provided by the present invention performs compilation optimization on deep learning operators, which can greatly improve the performance of deep learning operators, thereby improving the overall performance of the entire deep learning framework.
[0045] Based on the above embodiment, before mapping operators of each order in the meta-operator to corresponding hardware threads through the operator dynamic compiler according to the thread scheduling algorithm, the method further includes:
[0046] According to the preset compilation optimization requirements, a thread scheduling algorithm is constructed, wherein the thread scheduling algorithm includes a search strategy and a manual strategy;
[0047] The search strategy is determined based on the actual operating speed of the hardware, and the manual strategy is determined based on the hardware overhead.
[0048] Based on the above embodiments, the search strategy includes a brute force enumeration strategy, a search loop reordering strategy and a loop splitting strategy; the manual strategy includes increasing memory access continuity, reducing atomic operations and reducing register pressure.
[0049] In the present invention, the thread scheduling algorithm integrates all orders of operators and maps them to the hardware thread model reasonably. The thread allocation can be realized by two algorithms according to the compilation optimization requirements: search strategy (requires higher performance optimization) and manual strategy (requires reducing additional overhead). Specifically, the search strategy determines the thread allocation strategy through strategies such as brute force enumeration, search loop reordering and loop splitting, and based on the actual running speed of the hardware. Among them, the performance that can be obtained by brute force enumeration search must be the best, but the search space is relatively large. The number of loop reordering alone is 6! = 120. In addition to loop splitting, the total search amount may reach tens of thousands, and different shapes and sizes need to be searched, which brings huge additional search overhead to the deep learning framework. When the search overhead is huge, a manual strategy can be adopted. By increasing the continuity of memory access, reducing atomic operations and reducing register pressure, etc., 95% of the performance optimization of the search strategy can be achieved without additional overhead. In the present invention, the manual strategy is used as a more preferred method to complete performance tuning without adding additional overhead, and realize the process of mapping operators of each order in the meta-operator to the corresponding hardware threads. Specifically, the description of the specific three strategies for the manual strategy is as follows:
[0050] Adding a memory access continuity strategy can improve the memory access efficiency and throughput of operators, and is cache-friendly and easier to optimize. Different hardware devices have different allocation strategies: for CUDA, the bottom-level orders are allocated to the inner threads first, for example, the W order in NCHW (N stands for number, C stands for channel, H stands for height, and W stands for width) is allocated to threadIdx.x; for CPU, the top-level orders are allocated to the outer threads first, for example, the N order in NCHW is allocated to threadIdx. The reason for the difference between CUDA and CPU is that the number of CUDA threads is much larger than that of CPU, and the memory access models are quite different. CPU uses multi-level cache to achieve high-frequency and low-throughput memory access, while CUDA uses many cores and two-level cache to achieve low-frequency and high-throughput memory access, so the parallel depth of the two will be different.
[0051] Reduce atomic operation strategy, because the reindex simplification operator will generate atomic operations at certain levels, and atomic operations are expensive on all devices. CUDA has a high tolerance for atomic operations, so it can be accelerated by adjusting the allocation strategy, but the efficiency of CPU atomic operations is very low. Therefore, when the backend finds that the CPU operator has atomic operations, it will directly use a single thread to run, thereby avoiding atomic operations. CUDA has two main strategies: if the simplified level is the innermost level, the number of threads allocated to this level is 32 (CUDA warp); if the simplified level is other levels, minimize the number of threads allocated to this level, and it is best not to allocate, but the total number of threads must be saturated (>64k).
[0052] The strategy to reduce register pressure is to merge adjacent orders when conditions permit, for example, merge H and W. Many deep learning operations are element-level, such as adding two 4th-order tensors one by one. The Jittor will create a 4-fold loop according to the definition of the meta-operator, but in fact, so many loop variables are not needed. A large number of loop variables will only consume more register resources and increase the difficulty of thread allocation. Therefore, the backend will compress unnecessary loop variables. For example, when adding two 4th-order tensors, if the tensor memory space is continuous, the Jittor backend will directly compress the 4 loop variables into 1. The reindex operator and reindex simplification operator also use this strategy.
[0053] The present invention utilizes a thread scheduling algorithm to achieve optimal parallelization of operators, thereby greatly improving the execution efficiency of the operators.
[0054] Based on the above embodiment, the compilation optimization strategy includes: loop splitting, vectorization, cache hint and thread allocation.
[0055] In the present invention, compilation optimization is achieved through optimization strategies such as loop splitting, vectorization, cache hints and thread allocation. For the CPU, the operator dynamic compiler mainly performs optimization strategies such as loop splitting, vectorization, cache hints and other optimization strategies on deep learning operators to maximize the CPU's memory access speed. Among them, the loop splitting strategy changes the memory access order by splitting a loop into multiple loops; the vectorization strategy uses a single instruction to access multiple data, such as Intel's AVX instruction set can access 8 32-bit floating-point numbers at a time; the cache hint strategy maximizes cache utilization efficiency by using instructions for cache hints.
[0056] For GPU, the operator dynamic compiler allocates threads and blocks reasonably according to the CUDA thread hierarchy to maximize the continuity of memory access. Specifically, the operator dynamic compiler allocates the same row of continuous memory to the same block according to the CUDA thread hierarchy. When there are too many elements, resulting in exceeding the upper limit of the total number of threads, loop splitting optimization is performed.
[0057] For AI tensor processing chips (MLU), the operator dynamic compiler optimizes deep learning operators with loop splitting, vectorization, and thread allocation strategies to maximize tensor processing performance. The loop splitting strategy splits the loop to ensure that it can be vectorized smoothly later; the vectorization strategy vectorizes the entire memory segment; and the thread allocation strategy improves operator performance through multi-cluster and multi-core concurrent MLUs.
[0058] The present invention performs compilation optimization for specific hardware, can adapt to various computing hardware, and can bring into play the maximum performance of the specific hardware.
[0059] Based on the above embodiment, the operator mapped to the hardware thread is compiled and optimized according to the type of the hardware thread and the compilation optimization strategy to obtain the compiled and optimized operator, including:
[0060] If the type of the hardware thread is a central processing unit, with the goal of maximizing the memory access speed of the central processing unit, loop splitting, vectorization and cache hinting are performed on the operator mapped to the hardware thread in sequence to obtain a compiled and optimized operator.
[0061] In the present invention, for the compilation optimization of the CPU, the deep learning operator has very high requirements for the memory access speed. In the CPU, the factors that most affect the memory access speed are the cache hit rate and the CPU memory prefetch strategy. Figure 2 The schematic diagram of the CPU optimization strategy based on the broadcast operator provided by the present invention can be referred to Figure 2As shown, in this embodiment, taking the broadcast operator as an example, three optimization strategies are demonstrated: 1. Loop splitting: splitting a loop into multiple loops to change the memory access order; 2. Vectorization: using a single instruction to access multiple data, such as Intel's AVX instruction set can access 8 32-bit floating-point numbers at a time; 3. Cache hints: maximizing cache utilization efficiency by using instructions for cache hints.
[0062] Specifically, Figure 2 As shown, taking the CPU to implement row-by-row traversal broadcast operator as an example, Figure 2 In (a), the solid arrows from top to bottom represent that each element in the vector will be broadcasted as a corresponding column of the matrix; the squares in the vector represent instructions to read the elements of the vector one by one and write them into the matrix starting from the first square in the vector; and the dotted arrows in the matrix represent the order of traversal, that is, after broadcasting a row of the matrix, the next row will be broadcasted. Figure 2 In (b), loop splitting optimization is performed to change the order of traversal. Instead of broadcasting a whole row of the matrix, the left half of the matrix is broadcasted before the right half. This optimization is helpful when the length of the broadcasted vector exceeds the L1 cache length, because loop splitting can always ensure that the broadcasted vector resides in the L1 cache, so there is no need to access the broadcasted vector from the main memory.
[0063] Further, please refer to Figure 2 (c) Vectorization is the second important optimization for CPU compilation. In the past, ordinary scalar operation instructions could only read and write one element at a time. With vectorized instruction sets, such as AVX, eight numbers can be read and written at a time. Figure 2 In (c), two blocks are grouped together, which means two elements are read and written at one time. However, there is a constraint when using vectorized instructions, that is, the addresses must be aligned, otherwise performance will also be degraded. In the process of framework implementation, memory allocation methods such as align_alloc are used to ensure the alignment of memory addresses.
[0064] Furthermore, cache hints are the third important optimization for CPU compilation. Broadcast operators have an important characteristic that the input is generally read multiple times, but the output is only written once. Figure 2 (d), for this type of operator, the framework allows the input to reside in the cache for a long time, while the output does not need to reside. Therefore, the backend compiler can use non-temporary write vector instructions to prompt the CPU that it does not need to keep the output in the cache, but can directly write it to the main memory. This optimization ensures that the input vector can reside in the cache for a long time and will not be squeezed out of the cache by the output, thereby greatly improving the cache usage efficiency.
[0065] Based on the above embodiment, the method of compiling and optimizing the operator mapped to the hardware thread according to the type of the hardware thread and the compilation optimization strategy to obtain the compiled and optimized operator further includes:
[0066] If the type of the hardware thread is a graphics processor, the operator mapped to the hardware thread is thread-allocated with the goal of maximizing memory access continuity to obtain a compiled and optimized operator.
[0067] In the present invention, the compilation optimization of GPU is also explained by taking the broadcast operator as an example. The optimization strategy of GPU broadcast operator is simpler than that of CPU, and the key lies in the thread allocation strategy. The thread hierarchy of CUDA is divided into two levels, thread and block, where the number of threads cannot exceed 1024, and block is the upper structure of thread. Threads in the same block are executed in the same physical stream processing. Figure 3 A schematic diagram of a GPU optimization strategy based on a broadcast operator provided by the present invention is shown in FIG. Figure 3 As shown in the figure, different columns in the matrix are assigned to threads, and different rows are assigned to blocks. This order is critical. If it is reversed, the performance will drop significantly. This is because threads in the same block need to meet memory access continuity as much as possible, so the same row of memory needs to be assigned to the same block. It should be noted that if the number of elements is too large, resulting in exceeding the upper limit of the total number of threads, loop splitting needs to be performed according to the actual situation.
[0068] Based on the above embodiment, the method of compiling and optimizing the operator mapped to the hardware thread according to the type of the hardware thread and the compilation optimization strategy to obtain the compiled and optimized operator further includes:
[0069] If the type of the hardware thread is an artificial intelligence processor, with the goal of maximizing tensor processing performance, the operator mapped to the hardware thread is sequentially subjected to loop splitting, vectorization, and thread allocation to obtain a compiled and optimized operator.
[0070] In the present invention, the optimization strategy for the AI tensor processing chip (MLU) is mainly optimized by guiding the maximum performance of the chip. The present invention is illustrated by the optimization of MLU. MLU and GPU have similar two-level thread structures. GPU is divided into thread and block, while MLU is divided into core and cluster. However, the two are different in order of magnitude. A block of GPU can contain up to 1024 threads, while a cluster of MLU contains 4 cores. Different from a single thread in GPU, a single Core in MLU has stronger capabilities. A single thread of GPU can only perform scalar operations, while a single Core of MLU can operate one vector or even a matrix at a time. Multiple ALUs (Arithmetic Logic Units) in MLU, each ALU is in a Core, and ALU is also called Function Unit (FU), which has more powerful vector and matrix operation capabilities than ALUs in CPU and GPU. Each ALU can access the storage NRAM inside the Core, while multiple Cores share SRAM, and multiple Clusters share MLU main memory. Therefore, maximizing the vector and matrix operation capabilities in the ALU and fully utilizing the bandwidth of NRAM and SRAM are the key points of optimizing the MLU operator. Figure 4 A schematic diagram of the MLU optimization strategy based on the broadcast operator provided by the present invention is shown in FIG. Figure 4 The optimization strategy of the broadcast operator on MLU is as follows: 1. Loop splitting: splitting the loop to ensure smooth vectorization in the subsequent process; 2. Vectorization: vectorizing the entire memory segment; 3. Thread allocation: multi-cluster and multi-core in concurrent MLU.
[0071] Specifically, the loop splitting strategy of MLU in this embodiment is similar to the loop splitting purpose of CPU in the above embodiment. The purpose of CPU is to maximize the use of L1 Cache, while the loop splitting of MLU is to give full play to the performance of ALU for whole-segment vector operations. The length of whole-segment vector operations in ALU is much larger than that of CPU, which can reach thousands, but there is also an upper limit. Therefore, by loop splitting, the length of subsequent vectorization is ensured to be within the requirements of ALU. After loop splitting, the deep learning framework can directly map the underlying loop of the broadcast operator to the vectorized operation of MLU. In actual implementation, __memcpy will be used to read and write vectors from MLU main memory to NRAM. Finally, through the thread allocation strategy, the multi-core in concurrent MLU is different from the CUDA thread allocation strategy in that MLU allocates the same 4 columns inside the same core in order to allow the broadcasted vectors to reside in NRAM as much as possible, because NRAM is exclusively occupied by each core. If the threads are divided by rows, the broadcasted vectors will be repeatedly loaded, thereby reducing performance.
[0072] Figure 5 A schematic diagram of the structure of the compilation optimization system for deep learning operators provided by the present invention is shown in FIG. Figure 5 As shown, the present invention provides a compilation optimization system for deep learning operators, including a mapping module 501, a compilation optimization module 502 and a compilation module 503, wherein the mapping module 501 is used to map operators of each order in the meta-operator to corresponding hardware threads through an operator dynamic compiler according to a thread scheduling algorithm; the compilation optimization module 502 is used to compile and optimize the operators mapped to the hardware threads according to the types of the hardware threads and the compilation optimization strategies to obtain the compiled and optimized operators; the compilation module 502 is used to compile and obtain executable code according to the compiled and optimized operators.
[0073] The compilation optimization system for deep learning operators provided by the present invention performs compilation optimization on deep learning operators, which can greatly improve the performance of deep learning operators, thereby improving the overall performance of the entire deep learning framework.
[0074] The system provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.
[0075] Figure 6 A schematic diagram of the structure of an electronic device provided by the present invention, such as Figure 6As shown, the electronic device may include: a processor (processor) 601, a communication interface (CommunicationsInterface) 602, a memory (memory) 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604. The processor 601 can call the logic instructions in the memory 603 to execute the compilation optimization method for deep learning operators, the method comprising: according to the thread scheduling algorithm, through the operator dynamic compiler, mapping the operators of each order in the meta-operator to the corresponding hardware thread; according to the type of the hardware thread, according to the compilation optimization strategy, compiling and optimizing the operator mapped to the hardware thread to obtain the compiled optimized operator; according to the compiled optimized operator, compiling to obtain executable code.
[0076] In addition, the logic instructions in the above-mentioned memory 603 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0077] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the compilation optimization method for deep learning operators provided by the above methods, and the method includes: according to the thread scheduling algorithm, through the operator dynamic compiler, mapping operators of each order in the meta-operator to the corresponding hardware thread; according to the type of hardware thread, according to the compilation optimization strategy, compile and optimize the operator mapped to the hardware thread to obtain the compiled and optimized operator; according to the compiled and optimized operator, compile and obtain executable code.
[0078] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the compilation optimization method for deep learning operators provided in the above-mentioned embodiments, the method comprising: according to the thread scheduling algorithm, through the operator dynamic compiler, mapping operators of each order in the meta-operator to the corresponding hardware thread; according to the type of the hardware thread and the compilation optimization strategy, compiling and optimizing the operator mapped to the hardware thread to obtain the compiled and optimized operator; according to the compiled and optimized operator, compiling and obtaining executable code.
[0079] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0080] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A compilation optimization method for deep learning operators, characterized in that: include: According to the thread scheduling algorithm, the operators of each order in the meta-operator are mapped to the corresponding hardware threads through the operator dynamic compiler; According to the type of the hardware thread and the compilation optimization strategy, the operator mapped to the hardware thread is compiled and optimized to obtain the compiled and optimized operator; Compile and obtain executable code according to the compiled and optimized operator; The compilation optimization strategies include: loop splitting, vectorization, cache hinting and thread allocation; The method of compiling and optimizing the operator mapped to the hardware thread according to the type of the hardware thread and the compilation optimization strategy to obtain the compiled and optimized operator includes: If the type of the hardware thread is a central processing unit, with the goal of maximizing the memory access speed of the central processing unit, loop splitting, vectorization and cache hinting are performed on the operator mapped to the hardware thread in sequence to obtain a compiled and optimized operator; The method further comprises: performing compilation optimization on the operator mapped to the hardware thread according to the type of the hardware thread and the compilation optimization strategy to obtain the compiled optimized operator; If the type of the hardware thread is a graphics processor, with the goal of maximizing memory access continuity, perform thread allocation on the operator mapped to the hardware thread to obtain a compiled and optimized operator; The method further comprises: performing compilation optimization on the operator mapped to the hardware thread according to the type of the hardware thread and the compilation optimization strategy to obtain the compiled optimized operator; If the type of the hardware thread is an artificial intelligence processor, with the goal of maximizing tensor processing performance, the operator mapped to the hardware thread is sequentially subjected to loop splitting, vectorization, and thread allocation to obtain a compiled and optimized operator.
2. The compilation optimization method for deep learning operators according to claim 1, characterized in that: Before mapping operators of each order in the meta-operator to corresponding hardware threads through the operator dynamic compiler according to the thread scheduling algorithm, the method further includes: According to the preset compilation optimization requirements, a thread scheduling algorithm is constructed, wherein the thread scheduling algorithm includes a search strategy and a manual strategy; The search strategy is determined based on the actual operating speed of the hardware, and the manual strategy is determined based on the hardware overhead.
3. The compilation optimization method for deep learning operators according to claim 2, characterized in that: The search strategy includes a brute force enumeration strategy, a search loop reordering strategy and a loop splitting strategy; the manual strategy includes increasing memory access continuity, reducing atomic operations and reducing register pressure.
4. A compilation optimization system for deep learning operators, characterized in that: include: A mapping module is used to map operators of each order in the meta-operator to corresponding hardware threads through the operator dynamic compiler according to the thread scheduling algorithm; A compilation optimization module is used to compile and optimize the operator mapped to the hardware thread according to the type of the hardware thread and the compilation optimization strategy to obtain the compiled and optimized operator; A compiling module, used for compiling the optimized operator to obtain executable code; The compilation optimization strategies include: loop splitting, vectorization, cache hinting and thread allocation; The compilation optimization module is specifically used for: If the type of the hardware thread is a central processing unit, with the goal of maximizing the memory access speed of the central processing unit, loop splitting, vectorization and cache hinting are performed on the operator mapped to the hardware thread in sequence to obtain a compiled and optimized operator; The compilation optimization module is also used for: If the type of the hardware thread is a graphics processor, with the goal of maximizing memory access continuity, perform thread allocation on the operator mapped to the hardware thread to obtain a compiled and optimized operator; The compilation optimization module is also used for: If the type of the hardware thread is an artificial intelligence processor, with the goal of maximizing tensor processing performance, the operator mapped to the hardware thread is sequentially subjected to loop splitting, vectorization, and thread allocation to obtain a compiled and optimized operator.
5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the compilation optimization method for a deep learning algorithm as described in any one of claims 1 to 3 are implemented.
6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the compilation optimization method for a deep learning algorithm as described in any one of claims 1 to 3 are implemented.
