A deep learning model compilation method and compiler supporting multiple hardware

Abstract

The application provides a kind of multi-hardware supported deep learning model compiling method and compiler, comprising: obtaining the computational graph of the deep learning model to be compiled and the target device information;The target device information is converted into hardware attribute intermediate form based on the preset hardware model, the linear operation intermediate form is obtained based on the basic operation of deep learning model, the tensor shape intermediate form is obtained based on the tensor of deep learning model;Based on the tensor and the tensor shape intermediate form, obtain the memory intermediate form, and use the heterogeneous memory transmission folding method to optimize memory copy behavior;Based on the obtained linear operation intermediate form, obtain the loop intermediate form, use the method of fusion, stacking and vectorization to optimize the loop in the loop intermediate form, and convert into vector intermediate form, obtain the heterogeneous device data operation rule intermediate form based on the preset calculation model;Based on the plurality of intermediate forms obtained in the foregoing, obtain the executable code that can be inferred on the target device.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to the technical field of hardware resource utilization of deep learning compiler, and more particularly to a deep learning compiling method and compiler supporting multiple hardware. BACKGROUND

[0002] In the field of deep learning, computer vision, natural language processing, speech recognition and many other fields have made breakthrough progress. These breakthroughs have greatly promoted the development of artificial intelligence technology, making the application of deep learning models in our daily life more and more widespread, from intelligent voice assistants to autonomous vehicles, to medical diagnosis and natural language translation, and so on. The application of deep learning technology is almost everywhere. Behind this rapid development, it is not only the innovation of algorithms and model design, but also how these models can be efficiently deployed on various computing devices. With the increasing size of models, the demand for hardware computing power also grows, which not only requires the continuous improvement of hardware performance to adapt to the growth of model and data volume, but also promotes the continuous evolution of deep learning training and inference framework, so as to realize the optimization of hardware performance and model demand.

[0003] In order to cope with these challenges, a series of hardware accelerators and special chips have been developed to provide more efficient computing power for deep learning applications. The diversity of these hardware platforms provides developers with rich choices, from general-purpose graphics processing units (GPGPU) to various specialized accelerators, each hardware has its unique advantages and specific use scenarios. However, hardware diversity also brings many challenges, each hardware platform has its specific architecture and software ecosystem, which means that in order to fully utilize the performance of these hardware, developers need to do a lot of customized development for each platform. This not only increases the complexity of development, but also raises the threshold of efficiently deploying deep learning models to diverse hardware environments.

[0004] The introduction of compiler technology provides a more automated and universal optimization path. By abstracting the commonality of hardware, the compiler aims to generate efficient execution code for different hardware platforms, reducing the direct dependence of developers on hardware characteristics. This approach not only improves the compatibility and performance of models on diverse hardware, but also helps to reduce development complexity, making the deployment and optimization of deep learning models more efficient and flexible. The progress of compiler technology, especially the development of specialized compilers for deep learning models, provides powerful tools for implementing high-performance deep learning applications across hardware platforms, opening up a new way to optimize the inference performance of deep learning models.

[0005] However, the current deep learning compilation method still has many shortcomings. The current deep learning model compiler mainly includes two types: a compiler developed by a hardware manufacturer and a compiler developed based on an open source framework. The model compilation method of the first type of compiler is customized by the manufacturer according to the released hardware. The inference model compiled by this type of compiler is only applicable to a single hardware device and does not have universality. If a second type of compiler is used to compile the model, the compiler needs to be redeveloped before compilation if the hardware device changes. Because the hardware device changes, the compilation method or model optimization strategy of the compiler may not be suitable for the new hardware device. In order to better exert the performance of the hardware, the compilation method or optimization strategy needs to be modified or the compiler needs to be restructured, resulting in high model inference cost.

[0006] Based on the above analysis, the existing deep learning compiler has the following problems: (1) The compiler does not have universality: In the existing compiler, the compiler is customized according to the hardware device, so it is not possible to use only one compiler to compile the deep learning model into an inference model applicable to multiple hardware devices. The model compilation and inference process has a strong coupling relationship with the hardware device, which has limitations in applicability and cannot realize that one compiler adapts to multiple hardware devices; (2) There are limitations in the compilation method and optimization strategy: The compilation method and optimization strategy in the existing compiler are only applicable to a single hardware device. The compilation method and optimization strategy are specially customized for a certain hardware device. Because different hardware devices differ in architecture, computing power, etc., they cannot be applied to other hardware devices, and there are limitations in the compilation method and optimization strategy; (3) High model compilation cost: Because the compilation method and optimization strategy in each compiler are only applicable to a single hardware device, in order to ensure that the compiled inference model can fully utilize the performance of the hardware device, the compiler needs to be redeveloped according to the hardware device before model compilation, resulting in high model compilation cost. SUMMARY

[0007] Therefore, the purpose of the present application is to overcome the defects of the prior art and provide a deep learning compilation method and compiler supporting multiple hardware.

[0008] The purpose of the present application is achieved by the following technical solutions:

[0009] According to a first aspect of the present application, the present application provides a multi-hardware supported deep learning model compilation method for compiling a deep learning model into executable code that can be used for model inference on a target device, characterized in that the method comprises: an information acquisition step of acquiring a computation graph of a deep learning model to be compiled and target device information, wherein the computation graph at least includes basic operations and tensors of the deep learning model; a logic generation step of converting the target device information into a hardware attribute intermediate form based on a preset hardware model, obtaining a linear operation intermediate form based on the basic operations of the deep learning model, and obtaining a tensor shape intermediate form based on the tensors of the deep learning model; a memory step of obtaining a memory intermediate form based on the acquired tensors and tensor shape intermediate form of the deep learning model, and optimizing the memory copy behavior of each memory segment in the memory intermediate form using a first preset method; a loop processing step of obtaining a loop intermediate form based on the acquired linear operation intermediate form, selecting a corresponding strategy to optimize the loop in the loop intermediate form according to the hardware attribute intermediate form using a second preset method, and converting the optimized loop into a vector intermediate form, in addition, obtaining a heterogeneous device data operation rule intermediate form based on a preset computation model; and a hardware code generation step of obtaining executable code that can be used for inference on the target device based on the hardware attribute intermediate form obtained in the logic generation step, the memory intermediate form obtained in the memory step, and the loop intermediate form obtained in the loop processing step, wherein the hardware information represented by the hardware attribute intermediate form determines the hardware device to which the executable code is adapted.

[0010] Preferably, in the logic generation step, the preset hardware model is a three-level memory structure model, and different target devices convert hardware information into a hardware attribute intermediate form based on the hardware model according to the following steps: simplifying the target device into a hardware model containing a three-level memory structure of memory, cache, and register; obtaining attributes that affect the computation efficiency of the deep learning model based on the hardware model, wherein the attributes at least include hardware architecture, CPU core number, GPU / TPU tensor core, cache size, hardware computing power, and bandwidth size, and the hardware architecture is CPU architecture or GPU architecture or TPU architecture; and converting all the attributes obtained from the hardware model into a hardware attribute intermediate form.

[0011] Preferably, in the step of internalizing, the first preset method is a heterogeneous memory transfer folding method, and the memory copy behavior is optimized according to the following steps: traversing all memory copy behaviors, matching a plurality of memory copy behaviors corresponding to the same memory segment into a group to obtain a plurality of memory copy behavior groups corresponding to different memory segments; and using a deletion / replacement method to optimize redundant memory copy behaviors in each memory copy behavior group and memory management operations of the memory segment corresponding to the group.

[0012] Preferably, in the step of loop processing, the second preset method is: processing redundant loops in the loop intermediate form in a fusion manner to obtain a fused loop intermediate form; adjusting the structures of all loops in the fused loop intermediate form according to the hardware attribute intermediate form to obtain a stacked loop intermediate form; and setting a vectorization length coefficient according to the hardware attribute intermediate form, and converting the loops in the stacked loop intermediate form into a vector intermediate form according to the vectorization length coefficient.

[0013] Preferably, the step of processing redundant loops in the loop intermediate form in a fusion manner to obtain a fused loop intermediate form includes: traversing all loops in the loop intermediate form, and fusing a plurality of loops satisfying a fusion condition into one loop, the fusion condition including: condition 1: the plurality of loops operate on the same memory segment and the memory is not affected between the plurality of loops; and condition 2: the plurality of loops operate on different memories and the iteration numbers of the plurality of loops are consistent; wherein the plurality of loops satisfying any one of the conditions 1 and 2 are regarded as satisfying the loop fusion condition, and otherwise are regarded as not satisfying the fusion condition; and generating a fusion loop intermediate form based on the fused loop and the loop not fused.

[0014] Preferably, the different stacking strategies are adopted to adjust the structure of all loops in the fused loop intermediate form according to the hardware attribute intermediate form to obtain a stacked loop intermediate form, which comprises: obtaining hardware architecture information of a target device based on the hardware attribute intermediate form, and then adopting a corresponding stacking strategy to adjust the structure of the loops in the fused loop intermediate form according to the hardware architecture information, wherein the stacking strategy comprises: for the case that the target device is a homogeneous device, i.e., the hardware architecture of the target device is a CPU architecture, the adopted stacking strategy is to stack all single-layer loop structures into two-layer loop structures, and then to stack 2-4 layers of loops according to the number of CPU cores of the target device; for the case that the target device is a heterogeneous device, i.e., the hardware architecture of the target device is a GPU architecture or a TPU architecture, the adopted stacking strategy is to stack the structure of all loops in the fused loop intermediate form into a multi-layer loop structure of at least three layers containing at least one outer loop, one inner loop and one innermost loop from outside to inside, wherein the outer loop is a grid responsible for managing a plurality of concurrently executed thread blocks, the inner loop is a thread block responsible for managing the execution of a group of threads, and the innermost loop is a thread, and the inner loop of the thread is a single-layer loop or a multi-layer nested loop; the number of loops of the inner loop in all multi-layer loops is adjusted to a preset number, and the remaining number of loops of the inner loop is transferred to the outer loop, wherein the preset number corresponding to the GPU architecture is any integer less than or equal to 1024, and the preset number corresponding to the TPU architecture is any integer less than or equal to 512; and a stacked loop intermediate form is generated based on the stacked loops.

[0015] Preferably, the setting of the vectorization length coefficient according to the hardware attribute intermediate form comprises: for a target device with a CPU architecture, obtaining an instruction set architecture of the target device based on the hardware attribute intermediate form, setting a length coefficient according to the obtained instruction set architecture information, and each loop is vectorized according to the same length coefficient; for a target device with a GPU architecture or a TPU architecture, the length coefficient corresponding to each loop is set by the following method: obtaining a Tensor Core operation scale of the target device based on the hardware attribute intermediate form, wherein the Tensor Core is a tensor core of the GPU or the TPU; and adopting a corresponding strategy to set the length coefficient according to the number of innermost loops of the multi-layer loop, wherein if the innermost loop is a multi-layer nested loop, the length coefficient is set to the Tensor Core operation scale in the hardware attribute of the target device, otherwise, if the innermost loop is a single-layer loop, the length coefficient is set to 128 bytes.

[0016] Preferably, in the step of generating the computing model, the computing model is used to generate a heterogeneous device data operation rule, wherein the heterogeneous device is a target device with a hardware architecture of GPU architecture or TPU architecture, and the computing model generates the heterogeneous device data operation rule in the following manner: standardizing each data operation in the deep learning model to have the same operation step, each data operation being executed according to the operation step; based on all the data operations after the standardization operation step, saving the same steps in the operation steps in the same queue to obtain a plurality of queues, wherein each queue saves the same steps in different data operations; adding data dependencies between event processing queues to constitute a multi-stream parallel data operation running mode, and generating an intermediate form of the data operation rule based on the running mode.

[0017] Preferably, in the step of generating the hardware code, the executable code that can be used for inference on the target device comprises the following steps: obtaining the hardware architecture of the target device based on the hardware attribute intermediate form, so as to determine the instruction set type supported by the target device; and generating the executable code based on the obtained loop intermediate form, vector intermediate form and memory intermediate form, wherein in the generation process, all the loop, vector and memory operations in the deep learning model are converted into computer instructions supported by the target device.

[0018] According to a second aspect of the present application, the present application provides a multi-hardware supported deep learning model compiler for compiling a deep learning model into executable code supporting a target device through target device attributes based on any of the methods of the first aspect of the present application, characterized in that the system comprises a logic generation module, a memoryization module, a loop processing module, and a hardware code generation module, wherein: the logic generation module is configured to obtain a linear operation intermediate form, a hardware attribute intermediate form, and a tensor shape intermediate form based on the obtained deep learning model and target device information, and generate a logic model of the deep learning model to be compiled, wherein the logic model comprises the linear operation intermediate form, a tensor, the tensor shape intermediate form, and the hardware attribute intermediate form; the memoryization module is configured to obtain a memory intermediate form based on the tensor in the logic model and the tensor intermediate form, and optimize memory copy behavior using a memory folding method to generate a memory model of the deep learning model to be compiled, wherein the memory model comprises the linear operation intermediate form, the memory intermediate form, the tensor shape intermediate form, and the hardware attribute intermediate form; the loop processing module is configured to obtain a loop intermediate form based on the linear operation intermediate form in the memory model, and select a corresponding optimization strategy to adjust the structure of the loop according to the hardware attribute intermediate form, and convert the optimized loop into a vector intermediate form, in addition, obtain a target device data operation rule intermediate form based on a preset multi-stream parallel computing model, and generate a loop model of the deep learning model to be compiled through the above operations, wherein the loop model comprises the loop intermediate form, the memory intermediate form, the tensor shape intermediate form, the hardware attribute intermediate form, and the data operation rule intermediate form; and the hardware code generation module is configured to obtain executable code for performing an inference task on the target device based on the loop processing module, wherein the loop, the vector, and the memory operation of the deep learning model are converted into computer instructions supported by the target device according to the hardware attribute intermediate description to generate an inference model of the deep learning model to be compiled.

[0019] According to a third aspect of the present application, a computer readable storage medium is provided, characterized in that a computer program is stored thereon, and the computer program is executable by a processor to implement the method steps of any of the first aspect of the present application.

[0020] According to a third aspect of the present application, an electronic device is provided, characterized by one or more processors; and a memory, wherein the memory is configured to store executable instructions; and the one or more processors are configured to implement the method steps of any of the first aspect of the present application via execution of the executable instructions.

[0021] Compared with the prior art, the present application has the following advantages:

[0022] Firstly, the application will be designed by a hardware model, and the hardware properties of the target device are abstracted based on the model, and the model is compiled according to the hardware properties, and the model is compiled by using the way of obtaining the specific hardware model in the existing compiling method, the application realizes the decoupling of the compiling method and the hardware device, and can meet the needs of model compilation on various hardware devices, secondly, the model optimization strategy proposed in the application can dynamically adjust the optimization path, optimization parameters and the like according to the hardware properties of the target device, and realizes the decoupling of the model optimization and the hardware device, and the compiled inference model can fully utilize the hardware resources of the target device. Finally, based on the compiler constructed by the deep learning model compiling method supporting multiple hardware proposed in the application, the deep learning model can be compiled into executable code supporting the target device according to the properties of the input target device, compared with the prior art in which the target device changes and the compiler needs to be reconfigured. The application realizes that one compiler is suitable for multiple compiling tasks, and effectively reduces the model compiling cost. Compared with the existing deep learning compiler, the model inference precision is lossless, the model inference speed can be kept at the original level at most, and the optimal speed can be improved by 2-3 times. BRIEF DESCRIPTION OF DRAWINGS

[0023] The embodiments of the application will be further described below with reference to the accompanying drawings, in which:

[0024] Figure 1 is a model transformation flowchart according to the embodiments of the application;

[0025] Figure 2 is a deep learning compiling method flowchart supporting multiple hardware according to the embodiments of the application;

[0026] Figure 3 is a raw memory transmission paradigm schematic diagram according to the embodiments of the application;

[0027] Figure 4 is a memory transmission paradigm schematic diagram optimized by the memory folding method according to the embodiments of the application;

[0028] Figure 5 is a computing model schematic diagram according to the embodiments of the application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

[0030] As mentioned in the background section, the existing compiler for compiling deep learning models has the following disadvantages: (1) lack of universality: the existing deep learning compiler lacks universality, and the deep learning model compilation method adopted is customized according to the hardware information of a target device. Since the hardware information of different hardware devices differs in type and quantity, the compilation method cannot be applied to other hardware devices; (2) limitation in optimization strategy: the optimization strategy in the existing compiler is only applicable to a single hardware device, and the optimization strategy is specially customized according to a hardware device. Since different hardware differs in architecture and computing power, it cannot be applied to other hardware devices, and there is limitation in the compilation method and optimization strategy; (3) high model compilation cost: since the compilation method and optimization strategy adopted by the existing compiler are only applicable to a single hardware device, if the hardware of the target device changes, in order to ensure that the compiled inference model can fully utilize the performance of the hardware device, the compiler needs to be redeveloped according to the hardware device before model compilation, so the model compilation cost is high.

[0031] To solve the above problems, the inventors have found through research that existing deep learning compilers lack universality, and the deep learning model compilation method used is customized according to the hardware information of a target device. Since the hardware information of different hardware devices differs in type and quantity, this compilation method cannot be applied to other hardware devices. If a standard is set to abstract the hardware information of different target devices according to the standard, the applicability of the compilation method can be improved. Therefore, a hardware model is proposed in the present application. By designing a hardware model and obtaining the hardware properties of a target device based on the model, different hardware devices can be abstracted into hardware models with the same structure, and model compilation is performed according to the hardware properties, realizing the decoupling of the compilation method and the hardware device, and meeting the needs of model compilation on various hardware devices. In addition, the optimization strategy in the existing compiler is only applicable to a single hardware device. The optimization strategy is specially customized according to a hardware device. Since different hardware devices differ in architecture and computing power, the optimization strategy cannot be applied to other hardware devices, and there are limitations in the compilation method and optimization strategy. If the hardware properties of the target device are used as parameters for the optimization strategy, the decoupling of the model optimization and the hardware device can be realized. Therefore, the model optimization strategy proposed in the present application can dynamically adjust the optimization path and optimization parameters according to the hardware properties of the target device, realize the decoupling of the model optimization and the hardware device, and the compiled inference model can fully utilize the hardware resources of the target device. Finally, the model compilation cost is high: since the compilation method and optimization strategy used by the existing compiler are only applicable to a single hardware device, if the hardware of the target device changes, in order to ensure that the compiled inference model can fully utilize the performance of the hardware device, the compiler needs to be redeveloped according to the hardware device before model compilation, so the model compilation cost is high. If the compiler can adjust the model optimization strategy and the compilation method according to the hardware information of the input target device, it can realize the function of one compiler adapting to multiple compilation tasks. Therefore, the present application proposes a deep learning model compiler supporting multiple hardware. Compared with existing deep learning compilers, the model inference accuracy is not lost, the model inference speed can be kept at the original level at best, and the speed can be improved by 2-3 times at worst.

[0032] Before the embodiments of the present application are specifically introduced, some terms used therein are explained as follows:

[0033] Basic operation: refers to various mathematical and data processing steps in a deep learning model.

[0034] IR (Intermediate Representation): It is an intermediate form used to represent deep learning models. It can represent the structure and computation process of neural network models in the form of computational graphs. By representing deep learning models as IR, it can be used for model cross-framework conversion, model performance optimization, model cross-platform deployment, and model explanation analysis.

[0035] ONNX IR (Open Neural Network Exchange Intermediate Representation): ONNX is an open, cross-platform deep learning model exchange format. It allows different deep learning frameworks (such as TensorFlow, PyTorch, etc.) to share and deploy models. ONNX IR is an intermediate format for representing all operations in deep learning models. It provides a unified model representation to make it easier to exchange models between different deep learning frameworks.

[0036] Tensor IR (Tensor Intermediate Representation): Tensor means multi-dimensional array, usually used to represent data and weights in neural networks. Tensor IR is an intermediate format for representing tensors in deep learning models. This representation helps to operate and optimize data.

[0037] Linalg IR (Linear Algebra Intermediate Representation): Linalg is the abbreviation of "Linear Algebra", which involves vectors, matrices and linear equations, etc. Linalg IR is an intermediate form for representing logical operations in neural networks. The logical operations in this invention mainly include traversing the input and output, and performing common matrix multiplication, vector addition, etc. operations for each element traversed.

[0038] Device IR (Device Intermediate Representation): Device refers to hardware devices such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), TPU (Tensor Processing Unit), etc. Device IR is an intermediate representation of hardware device properties, such as hardware architecture, CPU core number, GPU / TPU tensor core, cache size, hardware computing power, bandwidth size, etc.

[0039] Shape IR (Shape Intermediate Representation): Shape refers to the dimension information of a tensor. Shape IR is an intermediate form that represents the shape information (i.e., the dimensions of a tensor) of tensors in a deep learning model, which is very important for compiler optimization and memory allocation.

[0040] MemRef IR (Memory Reference Intermediate Representation): MemRef is the abbreviation of Memory Reference, which represents memory reference. MemRef IR represents the intermediate form of all memory segments corresponding to logical operations in a deep learning model.

[0041] Affine IR (Affine Intermediate Representation): Affine describes the number of layers and iteration times of a loop in this invention. Affine IR represents the intermediate form of a loop in a deep learning model, which maintains loop conditions, loop times, loop iteration sizes, and other elements.

[0042] Vector IR (Vector Intermediate Representation): Vector refers to a series of data or operations. Vector IR is an intermediate form that represents vectors in a deep learning model.

[0043] DMA IR (Direct Memory Access Intermediate Representation): DMA is the abbreviation of Direct Memory Access, which is a computer technology used to achieve high-speed data transmission. DMA IR is an intermediate form of rules for operations and data transmission on target hardware devices. This is very important for efficient data transmission and memory management.

[0044] LLVM IR (LLVM IR Intermediate Representation): LLVM is a compiler infrastructure used for optimizing and converting code. LLVM IR is machine code generated by the LLVM compiler to represent basic functions such as loop jumps in a deep learning model. This machine code can be deployed on inference devices to perform inference tasks.

[0045] Device ISA (Instruction Set Architecture): It is the set of instructions executable on a device. In the field of deep learning compilation, Device ISA usually refers to the instruction set supported by a specific hardware (such as GPU, TPU, CPU, etc.), which contains instructions that can be executed directly by hardware. The compiler will compile the deep learning model into instruction sequences suitable for the device.

[0046] Homogeneous device: A hardware device that contains only one type of processor, CPU.

[0047] Heterogeneous device: A hardware device that contains one of the two types of processors, CPU and TPU, GPU.

[0048] In order to better understand the present application, the following will be described in detail in combination with specific embodiments.

[0049] According to an embodiment of the present application, the present application proposes a deep learning compilation method supporting multiple hardware, which is used to compile a deep learning model to enable inference process on various hardware devices. The overall compilation process is as shown in Figure 1 and Figure 2 , wherein, as shown in Figure 2 , the deep learning compilation method of the present application comprises five steps: information acquisition step, logic generation step, internalization step, loop processing step, and hardware code generation step. Among them, the logic generation step corresponds to the logic generation process of Figure 1 , the internalization step corresponds to the internalization process of Figure 1 , the loop generation and loop optimization correspond to the loop generation process and loop optimization process of Figure 1 , and the hardware code generation step corresponds to the hardware code generation process of Figure 1 . The following will introduce in detail the deep learning compilation method supporting multiple hardware proposed by the present application in combination with the five steps shown in Figure 2 . It should be noted that the present application involves multiple intermediate form conversion operations (i.e. IR conversion). Since the implementation of IR conversion is known to those skilled in the art, each intermediate form conversion operation in the present embodiment is referred to as IR conversion, and the details of IR conversion technology are not described in detail.

[0050] I. Information acquisition step

[0051] In the logic generation step, the computation graph of the deep learning model to be compiled and the target device information are acquired, wherein the computation graph at least includes the operation and tensor of the deep learning model. In the present application, the basic operation and tensor in the computation graph of the deep learning model to be compiled are represented by ONNX IR and Tensor IR intermediate form respectively.

[0052] II. Logic generation step

[0053] In the logic generation step, the target device information is converted into a hardware attribute intermediate form based on a preset hardware model, a linear operation intermediate form is obtained based on the basic operation of the deep learning model, and a tensor shape intermediate form, i.e., shape IR, is obtained based on the tensor of the deep learning model. In subsequent steps, the memory space of the same size is planned according to the Shape IR to store the elements thereof, and at the same time, the storage manner and arrangement order of the tensor in the memory are determined according to the Shape IR.

[0054] According to an embodiment of the present application, the linear operation intermediate form is Linalg IR, which describes how to traverse the input and output of the operation and what operation needs to be performed for each element traversed by the linear operation.

[0055] According to an embodiment of the present application, the preset hardware model is a three-level memory structure model. In the present application, different target devices are all based on the three-level memory structure model and convert the hardware information into the hardware attribute intermediate form according to the following steps and use Device IR to represent. First, the target device is simplified into a hardware model containing a three-level memory structure of memory, cache, and register. The actual hardware structure of the target device is complex. After being simplified into a three-level structure, it is convenient to obtain the hardware attribute abstraction. Then, based on the hardware model, the attributes affecting the calculation efficiency of the deep learning model are obtained, wherein the attributes at least include hardware architecture, CPU core number, GPU / TPU tensor core, cache size, hardware computing power, and bandwidth size. The hardware architecture is CPU architecture or GPU architecture or TPU architecture. The obtained hardware attributes are combined to constitute the hardware attributes of the target device. Finally, all the attributes obtained from the hardware model are converted into the hardware attribute intermediate form. The compiler adaptively adjusts the compilation method and optimization strategy according to the obtained hardware attributes, so as to convert the deep learning model into executable code that can perform model inference on the target device.

[0056] According to one embodiment of the present application, the acquisition of the attribute affecting the calculation efficiency of the deep learning model needs to consider the following constraints: (1) the total size of the cache on the inference device is limited, and its transmission speed is related to the bandwidth. (2) The time consumption of each calculation is the sum of the time of data transmission from the cache to the register, the calculation time obtained according to the computing power and the operation amount, and the time of the register transmission to the cache, but since the bandwidth from the cache to the register is very high, only the calculation time obtained according to the computing power is considered. (3) The transmission and calculation time of each time are independent of each other. Therefore, when the model is compiled, the hardware attributes such as hardware architecture, CPU core number, GPU / TPU tensor core, cache size, hardware computing power, and bandwidth size need to be provided to the compiler, and the hardware architecture is CPU architecture or GPU architecture or TPU architecture.

[0057] According to one embodiment of the present application, in the logical generation step, a tensor reuse operation is further included, that is, an element-by-element operation on a tensor, if the output result of the operation only has one consumer, the operation can be directly operated in place without using a new tensor. This method can avoid repeated calculation operations, thereby reducing the calculation overhead, improving the calculation efficiency and speed in the model inference process, on the other hand, it can reduce the memory usage, effectively reduce the memory consumption, and improve the performance and efficiency of the model.

[0058] III. Memory step

[0059] In the memory step, based on the acquired intermediate form of the tensor and the tensor shape of the deep learning model, an intermediate form of the memory is acquired, and a first preset method is used to optimize the memory copy behavior corresponding to each memory segment in the intermediate form of the memory.

[0060] According to one embodiment of the present application, the tensor of the deep learning model is the Tensor IR acquired in the information acquisition step, the tensor shape intermediate form is the Shape IR acquired in the logical generation step, and the intermediate form of the memory MemRefIR is generated based on the Tensor IR and the Shape IR through IR conversion. The MemRefIR represents the tensor with a specific memory segment, and can perform offset calculation and memory layout adjustment. In the IR conversion process, the transformation of the tensor will involve the life cycle calculation of the memory segment. The use of a piece of memory involves allocation operation, read-write operation, and release operation. The present application records the life cycle m x, x represents the serial number of the memory segment, and its state is changed according to the influence caused by each operation. Specifically, the possible memory effects of the operator include four kinds, namely, allocation, reading, writing, and release. For each memory segment, the memory allocation operation is included in the initialization, which will be the beginning of the life cycle of the memory segment. Then the operator will perform the corresponding operation on the memory segment, causing the corresponding reading or writing effect. After all operations are completed, the release operation will be performed on the memory segment, which will end the life cycle of the corresponding memory segment. The recorded life cycle will also involve the optimization of memory copy behavior.

[0061] According to an embodiment of the present application, the first preset method is a heterogeneous memory transfer folding method, and the memory copy behavior is optimized by using the method according to the following steps, wherein each memory transfer behavior corresponds to a memory segment. All memory copy behaviors are traversed, and a plurality of memory copy behaviors corresponding to the same memory segment are matched into a group to obtain a plurality of memory copy behavior groups corresponding to different memory segments; and the redundant memory copy behaviors in each memory copy behavior group and the memory management operations of the memory segment corresponding to the group are optimized by using the deletion / replacement method.

[0062] In order to better understand the steps of optimizing the memory transfer operation by using the heterogeneous memory transfer folding method, the following will be described in detail according to the memory transfer paradigm shown in Figure 3 and combined with specific examples. As shown in Figure 3 , the figure shows a memory transfer paradigm including three operation operations and a plurality of memory copy behaviors, the three operation operations are (1) C = GEMM (A, B), which is a matrix multiplication operation on two tensors stored in the memory segment A and the memory segment B on the CPU side (2) D = RELU (C), which is a rectified linear unit operation on the tensor stored in the memory segment C on the CPU side, and (3) Y = ADD (C, D), which is an addition operation on two tensors stored in the memory segment C and the memory segment D on the CPU side. The plurality of memory copies include: (1) the device side initiates copying to the CPU side, that is, memory copy 3, memory copy 5 and memory copy 8, three memory copy behaviors correspond to memory segment C, memory segment D and memory segment Y respectively. (2) The CPU side initiates copying to the device side, that is, memory copy 1, memory copy 2, memory copy 4, memory copy 6 and memory copy 7, five memory copy behaviors correspond to memory segment A, memory segment B, memory segment C, memory segment C and memory segment D respectively. As can be seen, each operation needs to copy data from the CPU side, and then copy the operation result to the CPU side, and the time cost of each copy operation will affect the calculation efficiency of the deep learning model. Illustratively, as shown in Figure 3Taking the transmission paradigm shown as an example, the steps of eliminating redundant memory copy behaviors in the heterogeneous memory transmission folding method are described in detail. First, the operation traverses all memory copy behaviors. If it is a memory copy behavior that initiates a copy from the device side to the CPU side, the memory copy behavior and its correspondence with the memory segment are saved in the global record table. That is, memory copy 3, memory copy 5 and memory copy 8 and their correspondence with the memory segment will be saved in the global record table during the traversal process. If the memory copy behavior is a copy initiated from the CPU side to the device side, then the memory copy behavior with the same memory segment as its memory segment is searched in the global record table. The search results may have two situations. Case 1: There is a memory copy operation that meets the query conditions in the global record table, such as Figure 3 Memory copy 4, memory copy 7, and memory copy 8 in the memory copy 4, memory copy 4 as an example, memory copy 4 corresponds to memory segment C, and memory copy 3 in the global record table also corresponds to memory segment C, that is, the two memory copy behaviors match successfully. The following describes in detail how to eliminate redundant operations in the two memory copy behaviors, delete the copy behavior 3 on the computing device side to write the calculation result of C = GEMM (A, B) back to the CPU's memory segment release operation and memory copy 4 to read the memory segment into the computing device's corresponding memory allocation operation, modify the D = RELU (C) to obtain memory segment C operation to obtain C = GEMM (A, B) and write it back to the CPU's corresponding memory operation, then record the copy behavior initiated from the device side to the CPU side, that is, memory copy 3, into the to-be-deleted list, and delete the memory copy behavior from the CPU side to the device side, that is, delete memory copy 4. The whole process can be summarized as follows: Figure 2 Delete the second half of each pair of behaviors in the dotted box, then add the first half to the list to be deleted, and replace all uses of the second half with the memory corresponding to the first half. Case 2: There is no memory copy behavior that meets the query conditions in the global record table, such as Figure 3 Memory copy 1 and memory copy 2 in the , this case is skipped and not processed. Then, after completing the traversal of all memory copy behaviors, the list to be deleted includes memory copy 3 and memory copy 5, and all memory copy behaviors in the list to be deleted are deleted. Finally, the memory transfer paradigm optimized by the heterogeneous memory transfer folding method is obtained. The optimization results are as follows Figure 4 As shown, compared Figure 3 It can be concluded that the complex memory copy operations between the three calculation operations are removed. The data required for each calculation operation is directly obtained from the previous calculation operation, thus eliminating redundant context switching operations, effectively improving data access and reading speed, and greatly improving the model's inference efficiency on the computing device. At the same time, this algorithm does not involve the hardware parameters of the target device, making model memory optimization independent of device parameters. It can optimize deep learning models on different target devices and has universal applicability.

[0063] 4. Loop processing steps

[0064] In the loop processing step, the loop intermediate form is obtained based on the obtained linear operation intermediate form, a second preset method is adopted, the corresponding strategy is selected to optimize the loop in the loop intermediate form according to the hardware attribute intermediate form, and the optimized loop is converted into the vector intermediate form. In addition, the heterogeneous device data operation rule intermediate form is obtained based on the preset calculation model.

[0065] According to one embodiment of the present application, the Linalg IR obtained in the logic generation step is IR-converted to generate the Affine IR describing the specific loop. In the conversion process, the operation of traversing the taken element is not changed greatly as the loop structure inside, and each dimension of the input and output is taken as a layer of the loop structure, and the traversal mode determines the loop times and the loop condition. The Affine IR contains the basic loop body structure, and maintains the loop condition, the loop times, the loop iteration step size and other elements. The loop condition in the Affine may keep the form of the Affine expression. The Affine expression mathematically describes a high-dimensional space, and the dimension is the number of layers of the loop and the iteration times. The read and write of the data inside the loop constitute the access to the space.

[0066] According to one embodiment of the present application, the second preset method is: processing the redundant loop in the Affine IR in a fusion manner to obtain the fused Affine IR; adjusting the structure of all loops in the fused loop intermediate form according to the Device IR by adopting different stacking strategies to obtain the stacked Affine IR; and setting the vectorization length coefficient according to the hardware attribute intermediate form, and converting the loop in the stacked loop intermediate form into the Vector IR according to the vectorization length coefficient. Compared with the model optimization method in the existing compiler, the optimization strategy of the present method is not limited to a single target device. In the model compilation process, the optimization path can be dynamically adjusted according to the obtained hardware device attribute, the compiled model can reach the native framework level performance on the platforms such as CPU, GPU and TPU, and can even reach 2-3 times acceleration.

[0067] According to one embodiment of the present application, the redundant cycles in the Affine IR are processed in a fusion manner to obtain a fused Affine IR, including: traversing all cycles converted by operation operations, and then fusing multiple cycles meeting a fusion condition into one cycle, the fusion condition including: condition 1: multiple cycle structures operate on the same memory and the memory is not affected between the multiple cycles; condition 2: multiple cycle structures operate on different memories and the multiple cycle structures have consistent iteration times; wherein multiple cycle structures meeting any one of conditions 1 and 2 are considered to meet the cycle fusion condition, and otherwise, two cycle structures are considered to not meet the fusion condition. In order to better understand the steps of fusing redundant cycles, the following will be described in detail in combination with specific examples.

[0068] For example, it is assumed that there are the following cycles in the Affine IR: cycle 1: traversing elements in memory space M1 and adding 5 to each element. Cycle 2: traversing elements in memory space M1 and multiplying each element by 5. Cycle 3: traversing elements in memory space M2, the memory includes 64 elements and is only one-dimensional. Cycle 4: traversing elements in memory space M3, the memory includes 64 elements and is only one-dimensional. There is no other operation that modifies the elements in memory space M1 between cycle 1 and cycle 2. According to the fusion condition, it can be concluded that cycle 1 and cycle 2 meet condition 1, that is, the two cycles operate on the elements of the same memory space, and there is no other memory operation between the two operations. This cycle can be fused into one cycle, that is, traversing all elements in memory space M1 and performing the operation of adding 5 to each element and then multiplying the added result by 5. It should be understood that if multiple cycles meet condition 1, but the multiple cycles have differences in the elements operated and the number of elements operated, the multiple cycles can also be fused. By fusing the redundant cycles in the deep learning model, the memory occupation can be reduced, the memory utilization can be improved, and thus the calculation efficiency and running effect of the model can be optimized.

[0069] According to one embodiment of the present application, different stacking strategies are adopted to adjust the structure of all loops in the fused loop intermediate form according to the hardware attribute information in Device IR to obtain a stacked loop intermediate form, including: obtaining hardware architecture information of a target device based on the hardware attribute in the intermediate form, and then adopting a corresponding stacking strategy to adjust the structure of the loops in the fused loop intermediate form according to the hardware architecture information, wherein the stacking strategy includes: (1) for the case of a homogeneous device as the target device, i.e., the hardware architecture of the target device is a CPU architecture, all single-layer loop structures are stacked into two-layer loop structures, and then 2-4 layers of loops are stacked according to the number of CPU cores of the target device; (2) for the case of a heterogeneous device as the target device, i.e., the hardware architecture of the target device is a GPU architecture or a TPU architecture, the structure of all loops in the fused loop intermediate form is stacked into a multi-layer loop structure of at least three layers containing at least one outer loop, one inner loop and one innermost loop from the outside, wherein the outer loop is a grid responsible for managing a plurality of concurrently executed thread blocks, the inner loop is a thread block responsible for managing the execution of a group of threads, and the innermost loop is a thread, and the inner loop of the thread is a single-layer loop or a multi-layer nested loop. Then, the number of loops of the inner loop in all multi-layer loops is adjusted to a preset number, and the remaining number of loops of the inner loop is transferred to the outer loop, wherein the preset number corresponding to the GPU architecture is any integer less than or equal to 1024, and the preset number corresponding to the TPU architecture is any integer less than or equal to 512. The stacking strategy of the present application has high adaptability and flexibility, and can dynamically optimize the loop structure based on different hardware device attributes, so as to maintain the best computing performance and hardware resource utilization efficiency under various target devices. Such a strategy enables deep learning models to run more efficiently on a variety of hardware platforms.

[0070] According to one embodiment of the present application, setting a vectorization length coefficient according to the hardware attribute intermediate form can improve the hardware resource utilization of the compiled model for the target device, and the vectorization length coefficient setting method comprises: for a CPU architecture target device, obtaining the target device instruction set architecture based on the hardware attribute intermediate form, setting the length coefficient according to the obtained instruction set architecture information, and each loop is vectorized according to the same length coefficient; for a target device of GPU architecture and TPU architecture, the length coefficient corresponding to each loop is set by the following method: obtaining the Tensor Core operation scale of the target device based on the hardware attribute intermediate form, wherein the Tensor Core is the tensor core of the GPU or TPU; according to the number of innermost loops of the multi-layer loop, the corresponding strategy is adopted to set the length coefficient, wherein the innermost loop is a multi-layer nested loop, and the length coefficient is set to the Tensor Core operation scale in the hardware attribute of the target device, otherwise, the innermost loop is a single-layer loop, and the length coefficient is set to 128 bytes. Based on the obtained target hardware attribute, the vectorization parameters are adaptively adjusted, and the hardware computing capacity of different target devices can be fully utilized, and the computing efficiency, data throughput and computing stability are significantly improved.

[0071] According to one embodiment of the present application, the computing model is used to generate a heterogeneous device data operation rule, wherein the heterogeneous device is a parallel mode in which a target device with a hardware architecture of GPU architecture or TPU architecture supports simultaneous execution of multiple computing functions, and the computing model generates a multi-stream parallel data operation rule of the heterogeneous device according to the following steps: standardizing each data operation in a deep learning model to have the same operation step, and each data operation is executed according to the operation step; based on all data operations after the standardization operation step, the same steps in the operation step are saved in the same queue to obtain multiple queues, wherein each queue saves the same steps in different data operations; adding data dependencies between event processing queues to form a multi-stream parallel data operation running mode, and generating a data operation rule intermediate form based on the running mode. The parallelism of the heterogeneous device can be improved, the correctness of the model operation can be ensured, and the hardware resources can be utilized to the maximum extent.

[0072] In order to better understand the steps of fusing the data operation rule generated based on the computing model to generate multi-stream parallel, the following will combine Figure 4 The detailed description is as follows: Figure 4As shown, the figure shows an implementation of the computing model, which standardizes all data operations to have the same steps, i.e., steps including data transfer steps and computing steps, and then stores the data transfer steps and computing steps corresponding to C=GEMM(A, B), D=RELU(C), C=GEMM(A, B) in the data transfer queue and the computing queue, respectively, and solves the data dependency between the queues by adding events. In actual inference, each queue is an operation stream, which fully utilizes the characteristics of the heterogeneous device supporting multiple function parallel execution, and effectively improves the model inference speed. It should be understood that the data operations are standardized to the same operation steps, and the steps included in the operation steps are not limited in number, and users can selectively configure according to needs.

[0073] Five, hardware code generation step

[0074] In the hardware code generation step, based on the loop intermediate form, vector intermediate form and memory intermediate form, and hardware attribute intermediate form obtained in the logic generation step, memory step and loop processing step, executable code that can be inferred on the target device is obtained, wherein the hardware information represented by the hardware attribute intermediate form determines the hardware device to which the executable code is adapted.

[0075] According to an embodiment of the present application, obtaining executable code that can be inferred on the target device comprises the following steps: obtaining the hardware architecture of the target device based on the hardware attribute intermediate form, so as to determine the instruction set type supported by the target device; and generating executable code based on the obtained loop intermediate form, vector intermediate form and memory intermediate form, wherein in the code generation process, all loops, vectors and memory operations in the deep learning model are converted into computer instructions supported by the target device. This method converts the basic operations in the deep learning model into computer instructions supported by the target device based on the properties of the target device, so as to achieve the best adaptation of the model to the target device. In order to better understand the steps of generating executable code, the following will be described in detail with specific examples.

[0076] Illustratively, the loop structure described by Affine IR is further converted into basic judgment and jump instructions, and the memory operations contained in Vector IR and MemRefIR are also converted into specific computer instructions after calculating the offset. This process will be converted into different results according to different hardware, such as AVX on the CPU side, PTX on the GPU side, and TPU on the TPU side. At the same time, a general computing function is used to implement model calling, and a consistent function signature is used to design a general computing function. The same deep learning model is executed on different hardware through the same general computing function to perform inference tasks. During model inference, all operations will be implemented by calling the computer instructions supported on the device, thereby decoupling the model from the hardware. It should be understood that in the hardware code compilation step, the supported hardware instruction set is not limited to AVX, PTX, and TPU. Users can add other hardware instruction sets in the hardware code compilation step according to their needs.

[0077] Based on the above deep learning compilation method supporting multiple hardware, the application further provides a deep learning model compiler supporting multiple hardware, which is used for compiling a deep learning model into executable code supporting a target device through target device attributes, and the system comprises a logic generation module, a memory module, a loop processing module and a hardware code generation module, wherein the logic generation module is used for obtaining a linear operation intermediate form, a hardware attribute intermediate form and a tensor shape intermediate form based on the obtained deep learning model and target device information, and generating a logic model of the deep learning model to be compiled, wherein the logic model comprises the linear operation intermediate form, a tensor, the tensor shape intermediate form and the hardware attribute intermediate form; the memory module is used for obtaining a memory intermediate form based on the tensor in the logic model and the tensor intermediate form, and optimizing memory copy behavior by using a memory folding method to generate a memory model of the deep learning model to be compiled, wherein the memory model comprises the linear operation intermediate form, the memory intermediate form, the tensor shape intermediate form and the hardware attribute intermediate form; the loop processing module is used for obtaining a loop intermediate form based on the linear operation intermediate form in the memory model, and selecting a corresponding optimization strategy to adjust the structure of the loop according to the hardware attribute intermediate form, and converting the optimized loop into a vector intermediate form, in addition, obtaining a target device data operation rule intermediate form based on a preset multi-stream parallel computing model, and generating a loop model of the deep learning model to be compiled through the above operations, wherein the loop model comprises the loop intermediate form, the memory intermediate form, the tensor shape intermediate form, the hardware attribute intermediate form and the data operation rule intermediate form; and the hardware code generation module is used for obtaining executable code capable of performing an inference task on the target device based on the loop processing module, wherein the loop, the vector and the memory operation of the deep learning model are converted into computer instructions supported by the target device according to the hardware attribute intermediate description to generate an inference model of the deep learning model to be compiled.

[0078] In order to better illustrate the technical effects of the application, the inventors designed multiple comparative experiments, and the technical effects of the application are further demonstrated below in combination with the comparative experiments and the results of the comparative experiments.

[0079] In order to verify the compatibility of the multi-hardware supporting compilation method proposed in the present application, by comparing the hardware types supported by the ONNXRuntime deep learning model accelerator, TensorRT, TPU-MLIR and the compiler proposed in the present application, the results shown in Table 1 can be obtained. The analysis of the data in the table can draw the following conclusions: the ONNXRuntime deep learning model accelerator, TensorRT and TPU-MLIR only support part of the hardware in CPU, GPU and TPU. For the adaptation of other hardware, only the ONNXRuntime deep learning model accelerator can adapt to other hardware by re-implementing the calculation kernel. The deep learning compiler proposed in the present application not only supports common CPU, GPU and TPU hardware, but also can complete the adaptation of other hardware through simple extension and addition, and has excellent device compatibility and expansibility.

[0080] Table 1: Comparison of compiler support hardware

[0081]

[0082] In order to further verify the inference accuracy of the inference model compiled by the compiler proposed in the present application, the inventors compared the operation accuracy of Add, Linear, Conv, Relu and Maxpool of the deep learning model when running on the ONNXRuntime deep learning model accelerator and the inference model compiled by the TensorRT, TPU-MLIR and the compiler proposed in the present application. The comparison experiment results are shown in Table 2. The left side of the non-zero item in the table represents the mean of the absolute error, and the right side represents the mean of the relative error. Through the analysis of the experimental results, the following conclusions can be drawn: the model compiled by the compiler proposed in the present application basically maintains no error on different operations. Even if there is an error, it is close to the range of single-precision floating point representation, so it can be considered that there is no error. Therefore, it can be concluded that compared with the existing deep learning compiler, the model inference accuracy of the compiler proposed in the present application has no loss.

[0083] Table 2: Comparison of inference model accuracy

[0084]

[0085] To further verify the inference speed of the inference model compiled by the compiler proposed in the application, the inventors obtain the comparison experimental results shown in Tables 3 and 4 by comparing the operation time and inference time of the inference model when running on the ONNXRuntime deep learning model accelerator and the deep learning model compiled by the TensorRT, TPU-MLIR and the compiler proposed in the application. The analysis of the data in the table can draw the following conclusions: the inference model compiled by the deep learning compiler supporting multiple hardware proposed in the application is optimal in inference speed and operation time compared with the prior art, which can achieve 2-3 times speed improvement, and in the worst case, it can also maintain the same level as the existing compiler.

[0086] Table 3: Comparison of inference model operation time

[0087]

[0088] Table 4: Comparison of inference model inference time

[0089]

[0090] It should be noted that although the above describes each step in a specific order, it does not mean that each step must be performed in the above specific order, in fact, some of these steps can be performed concurrently, or even in reverse order, as long as the desired function can be achieved.

[0091] The application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions loaded thereon for causing a processor to implement various aspects of the application.

[0092] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a hole-in-pole structure having instructions stored thereon, and any suitable combination of the above.

[0093] Having described various embodiments of the application, it is to be understood that the above description is meant to be illustrative only and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art, without departing from the scope and spirit of the described embodiments. The choice of words in this document is intended to best explain the principles of the embodiments, the practical application, or technical improvement over the existing technology, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for compiling a deep learning model that supports multiple hardware components, for compiling a deep learning model into executable code that can perform model reasoning on a target device, characterized in that: The method comprises: Information acquisition step: obtaining a computational graph and target device information of a deep learning model to be compiled, wherein the computational graph includes at least basic operations and tensors of the deep learning model; Logic generation step: convert the target device information into an intermediate form of hardware attributes based on the preset hardware model, obtain the intermediate form of linear operations based on the basic operations of the deep learning model, and obtain the intermediate form of tensor shapes based on the tensors of the deep learning model; Memorization step: obtaining a memory intermediate form based on the obtained tensor and tensor shape intermediate form of the deep learning model, and optimizing the memory copy behavior corresponding to each memory segment in the memory intermediate form using a first preset method; Loop processing step: obtaining a loop intermediate form based on the obtained linear operation intermediate form, using a second preset method to select a corresponding strategy to optimize loops in the loop intermediate form according to the hardware attribute intermediate form, and converting the optimized loop into a vector intermediate form. In addition, obtaining a heterogeneous device data operation rule intermediate form based on a preset computing model; Hardware code generation step: Based on the hardware attribute intermediate form obtained in the logic generation step, the memory intermediate form obtained in the memorization step, and the loop intermediate form obtained in the loop processing step, executable code that can be inferred on the target device is obtained. The hardware information represented by the hardware attribute intermediate form determines the hardware device to which the executable code is adapted.

2. The method according to claim 1, characterized in that In the logic generation step, the preset hardware model is a three-level memory structure model, and different target devices convert hardware information into an intermediate form of hardware attributes based on the hardware model according to the following steps: Simplify the target device into a hardware model that includes a three-level memory structure of memory, cache, and registers; Based on the hardware model, obtain attributes that affect the computational efficiency of the deep learning model, where the attributes include at least hardware architecture, number of CPU cores, GPU / TPU tensor cores, cache size, hardware computing power, and bandwidth size, where the hardware architecture is CPU architecture, GPU architecture, or TPU architecture; Convert all attributes obtained from the hardware model into an intermediate form of hardware attributes.

3. The method according to claim 1, characterized in that In the memorization step, the first preset method is a heterogeneous memory transfer folding method, and the method is used to optimize the memory copy behavior according to the following steps, wherein each memory segment corresponds to multiple memory transfer behaviors, and each memory transfer behavior corresponds to one memory segment: Traverse all memory copy behaviors and match multiple memory copy behaviors corresponding to the same memory segment into groups to obtain memory copy behavior groups corresponding to multiple different memory segments; A delete / replace method is used to optimize the redundant memory copy behaviors in each memory copy behavior group and the memory management operations of the corresponding memory segments of the group.

4. The method according to claim 1, wherein In the loop processing step, the second preset method is: The redundant loops in the loop intermediate form are processed by fusion to obtain the fused loop intermediate form; According to the hardware attribute intermediate form, different stacking strategies are adopted to adjust the structure of all loops in the fused loop intermediate form to obtain the stacked loop intermediate form; A vectorization length coefficient is set according to the hardware attribute intermediate form, and loops in the stacked loop intermediate form are converted into vector intermediate forms according to the vectorization length coefficient.

5. The method according to claim 4, characterized in that The method of processing redundant loops in the loop intermediate form in a fusion manner to obtain a fused loop intermediate form includes: Traverse all loops in the loop intermediate form and merge multiple loops that meet the fusion conditions into one loop. The fusion conditions include: Condition 1: Multiple loops operate on the same memory segment and the memory is not affected between the multiple loops; Condition 2: Multiple loops operate on different memories and the number of iterations of the multiple loops is the same; Among them, if multiple loops meet any one of conditions 1 and 2, they are considered to meet the loop fusion condition, otherwise they are considered to not meet the fusion condition; Generate a fused loop intermediate form based on the fused loop and the unfused loop.

6. The method according to claim 4, characterized in that The intermediate form according to the hardware attributes adopts different stacking strategies to adjust the structure of all loops in the fused loop intermediate form to obtain the stacked loop intermediate form, including: The hardware architecture information of the target device is obtained based on the intermediate form of the hardware attributes, and then a corresponding stacking strategy is adopted according to the hardware architecture information to adjust the structure of the loop in the intermediate form of the fused loop, wherein the stacking strategy includes: If the target device is a homogeneous device, that is, the hardware architecture of the target device is a CPU architecture, the stacking strategy adopted is: Stack all single-layer loop structures into two-layer loop structures. Then, stack 2 to 4 layers of loops according to the number of CPU cores of the target device. If the target device is a heterogeneous device, that is, the hardware architecture of the target device is a GPU architecture or a TPU architecture, the stacking strategy adopted is: The structures of all loops in the fused loop intermediate form are stacked into a multi-layer loop structure with at least three layers, including at least one outer loop, one inner loop, and one innermost loop. The outer loop is a grid, which is responsible for managing multiple concurrently executed thread blocks. The inner loop is a thread block, which is responsible for managing the execution of a group of threads. The innermost loop is a thread, and the loop inside the thread is a single loop or a multi-layer nested loop. Adjust the number of iterations of the inner loops in all multi-layer loops to a preset number, and transfer the remaining number of iterations of the inner loops to the outer loops, where the preset number for GPU architecture is any integer less than or equal to 1024, and the preset number for TPU architecture is any integer less than or equal to 512; Generates a stacked loop intermediate form based on the stacked loop.

7. The method according to claim 4, characterized in that The setting of the vectorization length coefficient according to the intermediate form of the hardware attribute includes: For a target device with a CPU architecture, the target device instruction set architecture is obtained based on the intermediate form of hardware attributes, the length coefficient is set according to the obtained instruction set architecture information, and each loop is vectorized according to the same length coefficient; For GPU and TPU target devices, the length coefficient corresponding to each loop is set as follows: Get the Tensor Core operation scale of the target device based on the intermediate form of hardware attributes, where Tensor Core is the tensor core of the GPU or TPU; According to the number of innermost loop layers of a multi-layer loop, the corresponding strategy is adopted to set the length coefficient. Among them, if the innermost loop is a multi-layer nested loop, the length coefficient is set to the Tensor Core operation scale in the hardware properties of the target device. Conversely, if the innermost loop is a single-layer loop, the length coefficient is set to 128 bytes.

8. The method according to claim 1, characterized in that In the loop processing step, the computing model is used to generate heterogeneous device data operation rules, wherein the heterogeneous device is a target device whose hardware architecture is a GPU architecture or a TPU architecture. The computing model generates the heterogeneous device data operation rules in the following manner: Standardize each data operation in the deep learning model so that it has the same operation steps and each data operation is performed according to the operation steps; Based on all data operations after the standardized operation steps, the same steps in the operation steps are saved in the same queue to obtain multiple queues, wherein each queue saves the same steps in different data operations; Add data dependencies between event processing queues to form a multi-stream parallel data operation operation mode, and then generate the intermediate form of data operation rules based on this operation mode.

9. The method according to claim 1, characterized in that In the hardware code generation step, obtaining executable code that can be inferred on the target device includes the following steps: Obtain the hardware architecture of the target device based on the intermediate form of hardware attributes, thereby determining the type of instruction set supported by the target device; Executable code is generated based on the obtained loop intermediate form, vector intermediate form, and memory intermediate form. During the generation process, all loops, vectors, and memory operations in the deep learning model are converted into computer instructions supported by the target device.

10. A deep learning model compiler supporting multiple hardware based on the method according to any one of claims 1 to 9, used to compile a deep learning model into executable code supporting a target device according to target device attributes, characterized in that: The system includes: a logic generation module, a memory module, a loop processing module, and a hardware code generation module, wherein: The logic generation module is used to obtain a linear operation intermediate form, a hardware attribute intermediate form, and a tensor shape intermediate form based on the acquired deep learning model and target device information, and generate a logic model of the deep learning model to be compiled, wherein the logic model includes the linear operation intermediate form, the tensor, the tensor shape intermediate form, and the hardware attribute intermediate form; The memorization module is used to obtain a memory intermediate form based on the tensors and tensor intermediate forms in the logical model, and optimize the memory copy behavior using a memory folding method to generate a memory model of the deep learning model to be compiled, wherein the memory model includes a linear operation intermediate form, a memory intermediate form, a tensor shape intermediate form, and a hardware attribute intermediate form; The loop processing module is used to obtain a loop intermediate form based on the linear operation intermediate form in the memory model, and select a corresponding optimization strategy to adjust the structure of the loop according to the hardware attribute intermediate form, and convert the optimized loop into a vector intermediate form. In addition, based on the preset multi-stream parallel computing model, the target device data operation rule intermediate form is obtained. The loop model of the deep learning model to be compiled is generated through the above operations, wherein the loop model includes a loop intermediate form, a memory intermediate form, a tensor shape intermediate form, a hardware attribute intermediate form, and a data operation rule intermediate form; The hardware code generation module is used to obtain executable code that can perform inference tasks on the target device based on the loop processing module. According to the intermediate description of hardware properties, the loop, vector, and memory operations of the deep learning model are converted into computer instructions supported by the target device to generate an inference model of the deep learning model that needs to be compiled.

11. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of any one of the methods of claims 1 to 9.

12. An electronic device, characterized in that: include: one or more processors; as well as a memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1 to 9 by executing the executable instructions.

Images