Machine learning computation optimization method and compiler

Abstract

The application discloses a machine learning computation optimization method and a compiler. The method comprises the following steps: identifying a memory-intensive operator in a machine learning computation graph to obtain a plurality of subgraphs; identifying a backbone operator included in the subgraph, and grouping the subgraph based on the backbone operator; propagating index information from the backbone operator of each group to its predecessor operator; and fusing the operators in the memory-intensive operator subgraph into a GPU kernel function. The compilation optimization scheme of the application automatically generates efficient GPU code for large-granularity complex memory-intensive operator subgraphs by jointly considering the dependency relationship characteristics of the computation graph, the data locality on the multi-level storage architecture of the GPU, and the thread concurrency under different data sizes, reduces the additional overhead of GPU kernel function calling and framework layer operator scheduling, avoids unnecessary repeated calculation, reduces off-chip memory access, and simultaneously adapts to various data sizes to obtain optimized parallel efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine learning, and in particular to a machine learning computation optimization method and a compiler. BACKGROUND

[0002] In recent years, with the advent of chips with high parallelism and their supporting computing architecture, high-intensity parallel computing such as convolution and matrix multiplication is no longer a bottleneck for machine learning models. In the prior art, due to the limited automatic optimization capability, it is difficult to efficiently generate code for complex memory-intensive operator subgraphs, so that the additional overhead caused by operator scheduling and data access reduces the overall computing efficiency. In other words, memory-intensive computing becomes a performance bottleneck for today's machine learning models due to its frequent access.

[0003] Therefore, an improved machine learning computation optimization scheme is needed. SUMMARY

[0004] To solve at least one of the above problems, the present application proposes a machine learning computation optimization scheme. This scheme uses the dependency characteristics of the computation graph and the data locality on the multi-level storage architecture of the GPU to fuse a memory-intensive subgraph into a GPU kernel function, so as to reduce the additional overhead of GPU kernel function invocation and framework layer operator scheduling in the actual use of deep computing models, thereby improving the overall computing efficiency.

[0005] According to a first aspect of the present application, a machine learning computation optimization method is provided, comprising: identifying memory-intensive operators in a machine learning computation graph to obtain a plurality of memory-intensive operator subgraphs; identifying backbone operators included in the memory-intensive operator subgraphs and grouping the memory-intensive operator subgraphs, wherein the backbone operators and their preceding operators are grouped into one group; propagating index information from the backbone operators of each group to their preceding operators; and fusing the operators in the memory-intensive operator subgraphs into a GPU kernel function.

[0006] Optionally, the memory-intensive operators include other operators in addition to the computation-intensive operators in the machine learning computation graph, and the obtained plurality of memory-intensive operator subgraphs include a plurality of memory-intensive operator subgraphs separated by computation-intensive operators.

[0007] Optionally, identifying the backbone operators included in the memory-intensive operator subgraphs comprises: identifying reduce operators included in the memory-intensive operator subgraphs as backbone operators.

[0008] Optionally, the identifying the backbone operators included in the memory-intensive operator subgraph and grouping the memory-intensive operator subgraph further comprises: determining whether there is another path between two backbone operators that can propagate index information; in response to determining that there is another path between two backbone operators that can propagate index information, identifying one of the two backbone operators as a secondary backbone operator; and grouping the two operators and their predecessors into the same group.

[0009] Optionally, the propagating index information from each group's backbone operator to its predecessor further comprises: the backbone operator adaptively generating a thread mapping strategy according to the data shape to be processed; and propagating the thread mapping strategy to other operators in the same group.

[0010] Optionally, the adaptively generating a thread mapping strategy by the backbone operator according to the data shape to be processed comprises: setting a thread mapping rule for splicing or cutting a thread block according to the dimension size of the data shape and the real-time running status of the GPU.

[0011] Optionally, the method further comprises: checking the data dependency relationship between the backbone operator and the subsequent operator in the memory-intensive operator subgraph, and performing the following operations: in response to the data dependency relationship being at the thread level, propagating intermediate data through a register; in response to the data dependency relationship being at the thread block level, propagating intermediate data through shared memory; and in response to the data dependency relationship being at the cross-thread block level, propagating intermediate data through global memory.

[0012] Optionally, the method further comprises: analyzing the application and use relationship of shared memory of each operator in each memory-intensive operator subgraph; and causing the shared memory application of the subsequent operator in the same memory-intensive operator subgraph to preferentially select the shared memory application of the predecessor operator.

[0013] According to a second aspect of the present disclosure, a machine learning compiler is provided, comprising: a subgraph obtaining module configured to identify memory-intensive operators from a machine learning computation graph to obtain a plurality of memory-intensive operator subgraphs; a subgraph processing module comprising: an operator grouping submodule configured to identify backbone operators included in the memory-intensive operator subgraph and group the memory-intensive operator subgraph, wherein the backbone operators and their predecessors are grouped into one group; and a propagation submodule configured to propagate index information from each group's backbone operator to its predecessor; and an operator fusion module configured to fuse the propagated operators in each group into one GPU kernel function.

[0014] According to a third aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory having stored thereon executable code that, when executed by the processor, causes the processor to perform the method according to the first aspect.

[0015] According to a fourth aspect of the present disclosure, a non-transitory machine readable storage medium having stored thereon executable code that, when executed by a processor of an electronic device, causes the processor to perform the method according to the first aspect.

[0016] Thus, an optimization scheme for large-grained computation fusion is provided, which automatically generates efficient GPU code for large-grained complex memory-intensive operator subgraphs by jointly considering the dependency relationship characteristics of the computation graph, the data locality on the multi-level storage architecture of the GPU, and the thread concurrency under different data sizes, greatly reduces the additional overhead of GPU kernel function calls and framework layer operator scheduling, avoids unnecessary repeated computation, greatly reduces off-chip memory access, and adapts to various data sizes to obtain optimized parallel efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0017] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout the figures, and in which:

[0018] Figure 1 The software and hardware structure of CUDA is shown.

[0019] Figure 2 A schematic flowchart of a machine learning computation optimization method according to an embodiment of the present disclosure is shown.

[0020] Figure 3 An example of reducing the dimensions of a tensor by a reduce operator is shown.

[0021] Figure 4 An example of grouping memory-intensive operator subgraphs is shown.

[0022] Figure 5 An example of input data shape adaptive operation according to the present disclosure is shown.

[0023] Figure 6 An example of operator grouping, thread mapping propagation, and cross-thread data reuse is shown.

[0024] Figure 7 A constituent schematic diagram of a machine learning compiler according to an embodiment of the present disclosure is shown.

[0025] Figure 8An example of a machine learning compiler based on the present application is shown.

[0026] Figure 9 A structural diagram of a computing device that can be used to implement the machine learning computation optimization method described above according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] Preferred embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms without being limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0028] Currently, machine learning is widely used in image processing, natural language processing, search, recommendation, and advertising, etc. The task of machine learning often needs to consume huge computer computing power, and is often deployed on accelerators such as GPUs for computation. In order to adapt to machine learning computation, parallel computing architectures such as CUDA have been proposed. CUDA is a new hardware and software architecture for operating GPU computing, which regards GPU as a data parallel computing device, does not need to map these computations to a graphics API, and can use multiple computing cores in a graphics processor for general computing processing work. Figure 1 The software and hardware structure of CUDA is shown.

[0029] From the hardware point of view, as shown on the left side of the figure, a GPU includes a number of SMs (Streaming MultiProcessors), each of which includes a number of CUDA cores. Figure 1 As shown on the right side, SP (Streaming Processor) is the most basic processing unit of GPU. SP can also be referred to as CUDA core.

[0030] A plurality of CUDA cores can form a SM (Streaming MultiProcessor). Each SM (also referred to as each "core group") has a different number of CUDA cores according to different GPU architectures, for example, a SM can include 64 CUDA cores. A SM can have its own special operation unit (SFU), shared memory, register file, and thread bundle scheduler (Warp Scheduler), etc. Registers and shared memory are scarce resources, and these limited resources have very strict limitations on the number of effective thread bundles in each SM, which limits the parallelism capability.

[0031] A GPU chip device can include multiple SMs (multiple groups of cores), for example, a commercially available GPU chip can include 60 groups of cores, each group including 64 CUDA cores, so the chip includes 3840 CUDA cores.

[0032] From the software point of view, as Figure 1 As shown on the left, a thread is the execution unit of a CUDA program. A CUDA parallel program will be executed by many threads.

[0033] Several threads are grouped into a thread block. Threads in the same thread block can be synchronized and communicate through shared memory. Multiple blocks can form a grid.

[0034] When operating on a machine learning model, the SM adopts the SIMT (Single-Instruction, Multiple-Thread) architecture, and the warp is the most basic execution unit. A warp includes 32 parallel threads, and these threads execute the same instruction with different data resources.

[0035] When a GPU kernel is executed, the thread blocks in the grid are allocated to the SM. The threads of a thread block can only be scheduled on one SM. An SM can generally schedule multiple thread blocks, and a large number of threads can be distributed to different SMs. Each thread has its own program counter and state register, and executes instructions with its own data, which is called SIMT.

[0036] A CUDA core can execute a thread, and the CUDA cores of an SM are grouped into several warps (i.e., the CUDA cores are grouped in the SM). The Warp Scheduler is responsible for scheduling. The threads in a warp must be in the same thread block. Although the threads in a warp follow the same program address, they can have different behaviors, such as branch structures. Because the GPU stipulates that all threads in a warp execute the same instruction in the same cycle, warp divergence will cause performance degradation. The number of concurrent warps in an SM is limited. Because of resource limitations, the SM allocates shared memory for each thread block and independent registers for each thread in the warp, so the configuration of the SM affects the number of thread blocks and warp concurrency it supports.

[0037] In the calculation of a machine learning model, a computation graph is used to realize the expression of the model network structure. The computation graph is a computation function expressed by a directed graph with operators as nodes. In a machine learning framework, this computation function sequentially calls the operator nodes in the directed graph on the input data (for example, the incoming tensor) and obtains the final output data. In this application, the operators in the computation graph can be decomposed into two categories, namely, computation-intensive operators and memory-intensive operators, based on performance overhead. Here, the computation-intensive operator can be understood as an operator that requires a large amount of parallel computation, such as GEMM (general matrix multiplication) and Convolution (convolution). In actual computation, the above operators involve the call of a specified function library, which is not within the consideration range of the operator fusion optimization of the present application. The memory-intensive operator can refer to an operation that takes more time to access and store than to compute, such as Elementwise Add, BN, etc. The framework overhead caused by the excessive flexibility of the computation graph can also be classified into the proportion of memory-intensive operator overhead. In other words, all other operators in the computation graph except the computation-intensive operators such as GEMM (general matrix multiplication) and Convolution (convolution) can be regarded as memory-intensive operators.

[0038] With the iteration and development of machine learning models, memory-intensive computation begins to account for a large proportion in the end-to-end execution of machine learning models, even more than computation-intensive computation. For memory-intensive computation in machine learning tasks, existing machine learning compilation optimization tools (XLA, TVM, etc.) will fuse multiple memory-intensive machine learning operators into one GPU kernel (GPU kernel function) for execution, to some extent, to alleviate the performance problems caused by memory-intensive computation. However, the existing computation fusion still has the following problems:

[0039] First, due to the limited degree of computation fusion, a memory-intensive computation subgraph is often divided into several different GPU kernels, which brings significant GPU kernel call time overhead, and at the same time, a large number of GPU kernels bring serious machine learning framework scheduling overhead.

[0040] Second, each GPU kernel needs to read data from off-chip storage, and finally write back to off-chip storage for the next GPU kernel to read, and a large number of off-chip storage accesses introduce significant time overhead.

[0041] Third, in the existing technical solutions, data transfer between two operators does not support cross-thread communication, and different threads even need to independently and repeatedly calculate the same data, which brings a large amount of recalculation overhead.

[0042] Finally, the existing technical solutions are often optimized for input data with regular shapes, but there are a large number of irregular input data in industrial applications, and the existing solutions show obvious sub-optimization when facing these irregular input data.

[0043] As can be seen from the above, the current computing fusion technology can only cope with simple computing patterns, and it is difficult to fuse complex computing subgraphs into the same GPU kernel, and it is difficult to meet the performance requirements of users on machine learning tasks.

[0044] Therefore, the present application proposes an improved machine learning computing optimization scheme, which improves the performance of memory-intensive computing in machine learning to improve the execution efficiency of machine learning tasks on GPU.

[0045] Figure 2 A schematic flowchart of a machine learning computing optimization method according to an embodiment of the present application is shown. The method can be performed by a compiler implemented based on the present application, for example, a compiler with Figure 8 The components shown in the compiler perform.

[0046] In step S210, the memory-intensive operators in the machine learning computation graph are identified, and the memory-intensive operator subgraph is obtained.

[0047] As mentioned earlier, in the calculation of the machine learning model, a computation graph is used to express the model network structure, and the computation graph includes computation-intensive operators such as GEMM and Conv and other memory-intensive operators. Therefore, for a given machine learning computation graph, all memory-intensive operators in the computation graph can be identified. Without introducing a ring, the connected memory-intensive operator subgraph is circled. That is, a piece of memory-intensive operator subgraph separated by a computation-intensive operator can be regarded as an independent memory-intensive operator subgraph.

[0048] That is, the memory-intensive operator can include other operators in addition to the computation-intensive operator in the machine learning computation graph, and the plurality of memory-intensive operator subgraphs obtained include a plurality of memory-intensive operator subgraphs separated by computation-intensive operators.

[0049] At step S220, a backbone operator included in the memory access intensive operator subgraph is identified, and the memory access intensive operator subgraph is grouped based on the backbone operator, where the backbone operator and its preceding operators are grouped into one group. Here, the operator grouping can be performed on each memory access intensive operator subgraph based on the backbone operator in the memory access intensive operator subgraph. Here, the backbone operator can be an operator that causes the thread to be unable to be propagated downward due to a shape change of a tensor. In an embodiment, the backbone operator can be a reduce operator. The reduce operator can refer to an operator that performs a specific calculation (such as sum, Max / Min) on a multi-dimensional tensor data input to achieve the purpose of reducing the dimension. Common reduce operators can include tf.reduce_sum under the TensorFlow framework, ArgMax / ArgMin in the MNN framework.

[0050] Figure 3 An example of reducing the dimension of a tensor by a reduce operator is shown. As shown on the left side, the tensor input to the reduce operator is a three-dimensional matrix (N, H, W), where N = 5, H = 4, and W = 6. Then, the reduce operation is performed on Axis = 1, i.e., reduce on the H dimension, for example, to find the minimum value on the H dimension. Thus, the output tensor of (N, W) dimension with H dimension reduced to 1 can be obtained as shown on the right side. Figure 3

[0051] The advantage of selecting the reduce node as the backbone node (i.e., the "backbone" node within the kernel function range obtained by the operator fusion at step S240) is that at other nodes, the index that needs to be calculated within the same thread can be derived and calculated by the index of the backbone node in a radiation propagation manner, and the propagation process of the index and the calculation process of the value are cached within the range of the entire kernel function (obtained by the operator fusion of the memory access intensive operator subgraph), thereby maximizing the avoidance of redundant index calculation and value calculation. Thus, after grouping based on the backbone operator, step S230 can be performed to propagate the index information from the backbone operator of each group to its preceding operator. Here, the propagated index information refers to the index value of which part of a tensor data is calculated by a thread.

[0052] ​After the index propagation, the propagated operators in each group can be fused into a GPU kernel function at step S240. In other words, each memory-intensive operator subgraph in the computation graph can be fused into a GPU kernel function based on the backbone operator grouping and the index propagation by an automatic code generation technique. For example, a computation graph including 10 memory-intensive operator subgraphs separated by 9 computation-intensive operators can be fused into 10 corresponding GPU kernel functions by the backbone operator grouping and the index propagation within each memory-intensive operator subgraph during the compilation.

[0053] Thus, the present application greatly reduces the number of GPU kernel functions during the execution of a machine learning program by fusing any connected memory-intensive operators into the same GPU kernel function, avoiding unnecessary GPU kernel function calls and operator scheduling time.

[0054] In some cases, more than one reduce operator can be included in a group. To this end, identifying the backbone operators included in the memory-intensive operator subgraph and grouping the memory-intensive operator subgraph further includes determining whether there is another path between two backbone operators that can propagate index information, identifying one of the two backbone operators as a secondary backbone operator in response to determining that there is another path between the two backbone operators that can propagate index information, and grouping the two operators and their predecessors into the same group.

[0055] Figure 4 An example of grouping memory-intensive operator subgraphs is shown. Figure 4 An example of a memory-intensive operator subgraph is shown on the left. The subgraph can be the one circled in step S210, consisting of all connected memory-intensive operators between two computation-intensive operators. The graph includes four backbone operators (for ease of illustration, they are labeled as 1, 2, 3, and 4). However, since the 1st and 2nd backbone operators can propagate index information through the operators between them, the 2nd backbone operator and its predecessors can also be incorporated into the group of the 1st backbone operator. Further, since the 3rd and 4th backbone operators can propagate index information through the operators between them, the 4th backbone operator and its predecessors can also be incorporated into the group of the 3rd backbone operator. Since the data shape input to the later backbone operator must be smaller than the data shape input to the earlier backbone operator (for example, the shape of the data fed into the 2nd operator after being reduced by the 1st operator), the earlier backbone operator in the same group can still be considered as a “backbone operator” for index propagation (and subsequent adaptive shape thread mapping), and the later operator can be considered as a “secondary backbone operator”. To this end, in step S230, the 2nd and 4th operators are identified as secondary backbone operators, and the 1st and 3rd operators are identified as backbone operators. Figure 4In the example, the merged backbone operators 2 and 4 can be regarded as the secondary backbone operators of their respective groups (in... Figure 4 The right side is represented by a gray circle, and the subgraph is divided into two groups outlined by dashed lines.

[0056] Furthermore, since the shape of the input data is unknown in practical applications, the present invention also includes adaptively determining the number of threads to be mapped (i.e., how many threads to use for the computation of the current operator) based on the shape of the input data (i.e., the "shape" of the "tensor" of the input operator). To this end, the index propagation step 230 may also include: for each backbone operator, adaptively generating a high-concurrency thread mapping strategy according to the shape of the data to be processed, and then propagating the thread mapping strategy to other operators within the same previously divided group.

[0057] In existing XLA / TVM solutions, for critical reduce operators, if the reduced data dimension is small, the resulting thread blocks are too small to fully utilize GPU computing resources, resulting in poor parallelism. Similarly, if the unreduced dimension is small, the number of resulting thread blocks is too small, also failing to fully utilize GPU computing resources. Therefore, to improve the utilization of GPU computing resources, thread mapping rules can be set. These rules are used to splice or cut thread blocks based on the dimensionality of the data shape and the real-time GPU operating status.

[0058] Figure 5 An example of input data shape adaptation operation according to the present invention is shown. As shown in the figure, when the dimension of the input reduce task's data tensor is too small, the computational cost of a single calculation within the thread block is too low. In this case, the data to be processed by the thread block can be adapted, for example... Figure 5 The horizontal and / or vertical concatenation shown in a) is used. However, when the dimension of the input reduce task's data tensor is too large, the number of computations required by the thread block is too high. In this case, the data to be processed by the thread block can be modified as follows: Figure 5 The cutting is shown in b). In other words, by splicing two-dimensional thread blocks, thread blocks with too small a dimension are combined into thread blocks with a larger dimension, which can solve the problem of GPU computing resource control caused by the input data dimension being too small; by cutting a single thread block into several, the number of thread blocks can be increased, which can solve the problem of GPU computing resource control caused by the input data dimension being too large. For a given data input, this scheme will determine the splicing or cutting of thread blocks according to the size of the data dimension to achieve optimized parallelism.

[0059] Since the adaptive thread mapping is performed in the actual running of the machine learning model, the thread mapping rule can include or be implemented as a conditional judgment, so that the judgment can be made according to the current conditions at the actual execution time. The specific judgment conditions are related to both the tensor shape and the current parallelism of the GPU. For example, the operation to be performed by a certain backbone operator is to reduce a [4x2048] matrix (i.e., the input tensor) to a [4x1] matrix, and there are 16 idle SMs on the GPU, so the thread block can be cut as shown in b) of FIG. 10, so that every 4 SMs perform a [1x2048] reduction, and 16 SMs together complete the entire [4x2048] to [4x1] reduction operation. Figure 5

[0060] Further, since it can be considered that the backbone operator has the same data shape input as its preceding operator, the same thread block can be used to execute the backbone operator and its preceding operator. However, for the backbone operator and its subsequent operator, the data dependency between the backbone operator and its subsequent operator needs to be checked and data reuse planning is performed. If the data dependency is at the thread level, the intermediate data is propagated through the register; if the data dependency is at the thread block level, the intermediate data is propagated through the shared memory; and if the data dependency is at the cross-thread block level, the intermediate data is propagated through the global memory. If shared memory and global memory are used, it indicates that cross-thread data transmission is involved. In addition, in an operator group including a secondary backbone operator, since the shape of the data tensor is changed, the same group can also include intermediate data propagation through shared memory or global memory, and the data dependency between the secondary backbone operator and its subsequent operator also needs to be checked and data reuse planning is performed.

[0061] In addition, when using global memory for data reuse, global synchronization on the GPU is required, which requires that the maximum number of threads launched on the GPU cannot exceed the maximum number of threads that can be scheduled in one wave on the GPU. Therefore, the parallel dimension of the GPU kernel function can be controlled to be within the limit of the maximum number of threads that can be scheduled in one wave on the GPU.

[0062] Figure 6 Examples of operator grouping, thread mapping propagation, and cross-thread data reuse are shown. Figure 6 Figure 4 ​​As shown in the same figure, according to the reduce operator and whether the index information can be propagated between the reduce operators, the illustrated memory-intensive operator subgraph is divided into two groups in step 1). Then, in step 2), the thread mapping propagation within the upper group can be started from the backbone operator (i.e., the No. 1 operator) of the upper group, and the thread mapping propagation within the lower group can be started from the backbone operator (i.e., the No. 3 operator) of the lower group, as shown by the gray arrows in the figure. Figure 6 In step 3), the data reuse planning between the backbone operators and the secondary backbone operators and their subsequent operators can be performed. For example, the dotted line indicates that the intermediate data is propagated between the No. 1 operator and its subsequent operators through shared memory, and the arrow indicates that the intermediate data is propagated between the No. 2 operator and its subsequent operators and between the No. 3 operator and its subsequent operators through global memory.

[0063] The data reuse planning as above is used to determine which type of medium to use for reading and writing. Then, further memory allocation saving can be performed through on-chip memory optimization when it is determined that shared memory is used. Thus, the machine learning computing optimization method of the present application can further include: analyzing the application and use relationship of each operator to shared memory in each memory-intensive operator subgraph; and making the shared memory application of a subsequent operator in the same memory-intensive operator subgraph preferentially select the shared memory application of a preceding operator. For example, when a→b→c is calculated, it is known through the preceding series of operations that the data propagation therebetween needs to be performed through shared memory. That is, it is known that a→[shared memory-X]→b→[shared memory-Y]→c, and then the on-chip memory optimization can be used to analyze whether shared memory-X and shared memory-Y can be allocated only once (i.e., the storage allocation of shared memory-Y is used this time), thereby saving resource allocation.

[0064] Thus, the present application proposes a large-granularity operator fusion technology based on operator grouping, thread mapping propagation and cross-thread data reuse planning, and can generate a high-concurrency thread mapping method adaptive to different data shapes for machine learning operators.

[0065] In one embodiment, the present application can also be implemented as a machine learning computing compiler. Figure 7 A composition diagram of a machine learning compiler according to one embodiment of the present application is shown. As shown in the figure, Figure 7 The machine learning compiler 700 can include a subgraph acquisition module 710, a subgraph processing module 720 and an operator fusion module 730.

[0066] Specifically, the subgraph acquisition module 710 can be used to identify memory-intensive operators from the machine learning computation graph to obtain multiple memory-intensive operator subgraphs. The subgraph processing module 720 may include: an operator grouping submodule, used to identify the backbone operators included in the memory-intensive operator subgraphs and group the memory-intensive operator subgraphs, wherein the backbone operators and their preceding operators are grouped into one group; and a propagation submodule, used to propagate index information from the backbone operators of each group to their preceding operators. The operator fusion module 730 is used to fuse the propagated operators in each group into a GPU kernel function.

[0067] In one embodiment, the machine learning compiler of the present invention can be implemented as having Figure 8 The composition shown. Figure 8 An example of the composition of a machine learning compiler based on the present invention is shown. As shown in the figure, after obtaining the machine learning computation graph, memory-intensive operator subgraphs can be identified, that is, corresponding to the subgraph acquisition module 710 and the above step S210, memory-intensive operator subgraphs are identified. After identifying the memory-intensive operator subgraphs, subgraph optimization can be performed in the automatic code generation module. Specifically, this may include operator grouping based on backbone operators, adaptive shape thread mapping and propagation, and cross-thread data reuse planning. Finally, on-chip memory usage can be optimized (i.e., reusing shared memory allocation as much as possible). After the above optimization, code can be automatically generated, thereby obtaining executable binary files for each subgraph. The above executable binary files are part of the executable binary files obtained after compiling the entire computation graph. Subsequently, the executable binary files obtained after compiling the entire computation graph can be handed over to the GPU for execution to achieve computational performance superior to existing technologies.

[0068] Figure 9 A schematic diagram of a computing device that can be used to implement the above-described machine learning computation optimization method according to an embodiment of the present invention is shown.

[0069] See Figure 9 The computing device 900 includes a memory 910 and a processor 920.

[0070] The processor 920 can be a single core processor or a multiple core processor. In some embodiments, the processor 920 can include a general purpose processor and one or more special purpose processors such as graphics processors (GPUs), digital signal processors (DSPs), and the like. In some embodiments, the processor 920 can be implemented using custom circuitry, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0071] The memory 910 can include various types of storage units such as a system memory, a read-only memory (ROM), and a permanent storage device. The ROM can store static data or instructions that are needed by the processor 920 or other modules of the computer. The permanent storage device can be a read and write memory device. The permanent storage device can be a non-volatile storage device that does not lose stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device employs a mass storage device (e.g., a magnetic or optical disk, a flash memory) as the permanent storage device. In other embodiments, the permanent storage device can be a removable storage device (e.g., a floppy disk, an optical disk). The system memory can be a read and write memory device or a volatile read and write memory device such as a dynamic random access memory. The system memory can store some or all of the instructions and data that the processor needs at runtime. In addition, the memory 910 can include a combination of any or all of the above-mentioned storage devices, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic disks and / or optical disks. In some embodiments, the memory 910 can include a read and / or write removable storage device such as a compact disk (CD), a read-only digital versatile disk (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disk, an ultra density optical disk, a flash memory card (e.g., an SD card, a min SD card, a Micro-SD card, and the like), a magnetic floppy disk, and the like. The computer readable storage media does not include carrier waves and transitory electronic signals propagating through a wire or cable, or a wireless propagation.

[0072] The memory 910 stores executable code that, when processed by the processor 920, causes the processor 920 to perform the machine learning computation optimization methods described above.

[0073] The application proposes a large-granularity computing fusion compilation optimization method, which automatically generates efficient GPU code for large-granularity complex memory-intensive operator subgraphs by jointly considering three aspects of the dependence relationship characteristics of the computation graph, the data locality on the multi-level storage architecture of the GPU, and the thread concurrency under different data sizes, greatly reduces the additional overhead of GPU kernel function calls and framework layer operator scheduling, avoids unnecessary repeated calculations, greatly reduces off-chip memory access, and adapts to various data sizes to obtain optimized parallel efficiency.

[0074] The scheme supports the fusion of memory-intensive operators in the same subgraph into the same GPU kernel function, thereby greatly reducing the number of GPU kernel functions during the execution of the machine learning program, avoiding unnecessary kernel function calls and operator scheduling time; the scheme maximizes the retention of intermediate data in the calculation process in on-chip storage such as registers and shared memory through data reuse planning, avoiding unnecessary access to off-chip storage; the scheme avoids repeated calculations independently performed by different threads through cross-thread data reuse; and the scheme can support high concurrency when processing various input shapes through the method of adaptively mapping threads to input shapes.

[0075] In addition, the method according to the application can also be implemented as a computer program or computer program product, which comprises computer program code instructions for executing the above steps defined in the above method of the application.

[0076] Alternatively, the application can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) having executable code (or computer program, or computer instruction code) stored thereon, which, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to execute each step of the above method according to the application.

[0077] Those skilled in the art will also appreciate that the various example logical blocks, modules, circuits, and algorithm steps described in connection with the present disclosure can be implemented as electronic hardware, computer software, or a combination of the two.

[0078] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0079] In addition, the "first", "second" used in the present application are intended to indicate different objects, not to limit the execution order, etc., for example, the "first part data" and "second part data" referred to herein are intended to indicate different parts belonging to the feature map. The "first subsequent operation" and "second subsequent operation" are only used to distinguish that the two subsequent operations are different subsequent operations.

[0080] The embodiments of the present application have been described above, the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, practical application or improvement of the technology in the market, or to enable other ordinary skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for optimizing machine learning computation, comprising: identifying memory-intensive operators in a machine learning computation graph, obtaining a memory-intensive operator subgraph; identifying backbone operators included in the memory-intensive operator subgraph, and grouping the memory-intensive operator subgraph based on the backbone operators, wherein a backbone operator and its preceding operators are grouped into one group; propagating index information from a backbone operator of each group to its preceding operators; and fusing operators in the memory-intensive operator subgraph into one GPU kernel function; wherein identifying backbone operators included in the memory-intensive operator subgraph, and grouping the memory-intensive operator subgraph based on the backbone operators, comprises: determining whether there is another path between two backbone operators that can propagate index information; in response to determining that there is another path between two backbone operators that can propagate index information, identifying one of the two backbone operators as a secondary backbone operator; and grouping the two operators and their preceding operators into the same group; wherein propagating index information from a backbone operator of each group to its preceding operators comprises: a backbone operator adaptively generating a thread mapping strategy according to a data shape to be processed; and propagating the thread mapping strategy to other operators in the same group.

2. The method of claim 1, wherein, the memory-intensive operators comprise operators other than computation-intensive operators in the machine learning computation graph, and the obtained memory-intensive operator subgraph comprises multiple memory-intensive operator subgraphs separated by computation-intensive operators.

3. The method of claim 1, wherein, identifying backbone operators included in the memory-intensive operator subgraph comprises: identifying reduce operators included in the memory-intensive operator subgraph as backbone operators.

4. The method of claim 1, wherein, a backbone operator adaptively generating a thread mapping strategy according to a data shape to be processed comprises: setting a thread mapping rule for splicing or cutting a thread block according to a dimension size of the data shape and a real-time running status of a GPU. 5.The method of claim 1, further comprising: checking a data dependency relationship between a backbone operator and a subsequent operator in the memory-intensive operator subgraph, and performing the following operations: in response to the data dependency relationship being at a thread level, propagating intermediate data through a register; in response to the data dependency relationship being at a thread block level, propagating intermediate data through shared memory; and in response to the data dependency relationship being at a cross-thread block level, propagating intermediate data through global memory. 6.The method of claim 1, further comprising: analyzing an application and use relationship of shared memory by each operator in each memory-intensive operator subgraph; and making shared memory application of a subsequent operator in the same memory-intensive operator subgraph preferentially select reuse of shared memory application of a preceding operator. 7.A machine learning compiler, comprising: a subgraph obtaining module configured to identify memory-intensive operators from a machine learning computation graph, obtaining a plurality of memory-intensive operator subgraphs; a subgraph processing module comprising: ​ an operator grouping submodule, configured to identify backbone operators included in the memory-intensive operator subgraph, and group the memory-intensive operator subgraph, wherein a backbone operator and its preceding operators are grouped into one group; and a propagation submodule, configured to propagate index information from a backbone operator of each group to its preceding operators; and an operator fusion submodule, configured to fuse the propagated operators in each group into one GPU kernel function; wherein the operator grouping submodule is specifically configured to determine whether two backbone operators include other paths capable of propagating index information; in response to determining that the two backbone operators include other paths capable of propagating index information, identify one of the two backbone operators as a secondary backbone operator; and group the two operators and the preceding operators of the two operators into the same group; wherein the propagation submodule is specifically configured to adaptively generate a thread mapping strategy according to a data shape to be processed by the backbone operator; and propagate the thread mapping strategy to other operators in the same group.

8. A computing device, comprising: a processor; and a memory having stored thereon executable code that, when executed by the processor, causes the processor to perform the method of any one of claims 1-6.

9. A non-transitory machine-readable storage medium having stored thereon executable code that, when executed by a processor of an electronic device, causes the processor to perform the method of any one of claims 1-6. ​

Images