A neural network tensor operator combination automatic tuning method and device

Abstract

The application provides a neural network tensor operator combination automatic tuning method and device, relates to the field of deep neural networks, and comprises the following steps: defining a calculation mode of a tensor operator combination, sequentially obtaining the cyclic block size of a reduction iteration space and a bias iteration space; based on the cyclic block size, performing cyclic block and cyclic merge optimization on the tensor operator combination according to a scheduling template, generating a sketch and annotating the sketch; and outputting tuning parameters of the tensor operator combination according to the automatic tuning module processing the annotated sketch. Through production-consumption relationship data domain size calculation and data alignment cyclic block ratio calculation, the application performs cyclic block on the operator combination with a proportional relationship of block sizes, eliminates the mismatching conflicts between producers and consumers, improves the performance of the operator combination acceleration task with a data dependency relationship, and utilizes an Ansor customized operator dependency relationship search space to realize operator fusion and reduce redundant calculation.

Description

Technical Field

[0001] This application relates to the field of deep neural networks, and in particular to a method and apparatus for automatic tuning of neural network tensor operator combinations. Background Technology

[0002] Given the importance of low-latency inference performance in deep neural networks, researchers and industry practitioners have turned to search-based compilation techniques to automatically generate tensor programs, i.e., low-level implementations of tensor operators. For a single operator or a subgraph consisting of multiple operators, the user defines the computation using a high-level declarative language, and then the compiler searches for customized programs for different hardware platforms.

[0003] Current technology does not take into account the consumer-producer dependency relationship between tensor operators in certain scenarios, resulting in the overhead of generating multiple kernel functions or introducing a large amount of redundant computation, which leads to the less than ideal performance of the entire deep neural network inference. Furthermore, since the effectiveness of compilers based on polyhedral models depends on the effectiveness of the developer's modeling of the target architecture, the AKG designed for Huawei Ascend NPU does not perform well on general CPUs / GPUs. Summary of the Invention

[0004] To address the above problems, an automatic tuning method and apparatus for neural network tensor operator combination is proposed.

[0005] The first aspect of this application proposes an automatic tuning method for neural network tensor operator combinations, including:

[0006] Define the computation method for tensor operator combinations, and obtain the cyclic block size of the reduction iteration space and the bias iteration space in turn;

[0007] Based on the cyclic block size, the tensor operator combination is optimized by cyclic block and cyclic merging according to the scheduling template, a sketch is generated and the sketch is annotated.

[0008] Based on the annotated sketch processed by the automatic tuning module, the tuning parameters of the tensor operator combination are output.

[0009] Optionally, the calculation method for defining the combination of tensor operators includes:

[0010] For an input tensor A, a convolution kernel B, and an output image C after convolution, if a bias statement operation is performed on the input tensor A before the convolution operation, the iteration space corresponding to the first nested loop is defined as the bias iteration space, and the iteration space corresponding to the second nested loop is defined as the reduction iteration space, wherein there is a producer-consumer relationship between the bias iteration space and the reduction iteration space.

[0011] Optionally, obtaining the cyclic block size of the reduction iteration space and the bias iteration space sequentially includes:

[0012] By calling the TVM's overlapable cyclic block scheduling primitive, the reduction iteration space is cyclically divided into blocks to determine the size of the cyclic blocks in the reduction iteration space.

[0013] After determining the cyclic block size of the reduction iteration space, the cyclic block size of the bias iteration space is deduced from the data space.

[0014] Optionally, the step of deriving the cyclic block size of the bias iteration space from the data space includes:

[0015] The loop blocks called by TVM are applied to the reduction iteration space to obtain the memory footprint of each loop block in the reduction iteration space for the input tensor A;

[0016] Based on the memory footprint, the cyclic block size of the bias iteration space is determined according to the data domain size relationship of the input tensor A.

[0017] Optionally, the overlapping cyclic block scheduling primitive implements overlapping block scheduling through the following formula, including:

[0018] Range inner =factor + overlapped num -1;

[0019]

[0020] Among them, Range inner Range is the inner range of the overlapping split leaf iterator. outer Range is the outer range of the overlapping split leaf iterator. root The root iterator range, factor is the block size, and verlapped num This represents the size of the overlapping portion.

[0021] Optionally, the scheduling template is constructed in the following manner, including:

[0022] All nodes of the tensor operator combination sample are accessed in topological order. For operator combinations with consumer-producer relationships, the consumer operator is divided into regular cyclic blocks, and the producer operator is divided into overlapping blocks by calling the overlapping block scheduling primitive. For the cyclic block structure after block division, the SSRSRS block structure is used on the CPU, and the SSSRRSRS block structure is used on the GPU.

[0023] Optionally, the step of optimizing the tensor operator combination by cyclic block division and cyclic merging according to the scheduling template based on the cyclic block size, generating a sketch, and annotating the sketch includes:

[0024] The consumer operator in the tensor operator combination is subjected to cyclic segmentation provided by TVM, and the producer operator is subjected to overlapping segmentation.

[0025] After all the loop blocks are completed, the consumer operator and the producer operator are fused together to obtain the sketch.

[0026] The sketch is randomly annotated according to Ansor.

[0027] Optionally, the automatic tuning module is constructed in the following manner, including:

[0028] An automatic tuning module is generated by fine-tuning the performance tuner in Ansor through evolutionary search and a learnable cost model.

[0029] The second aspect of this application proposes an automatic tuning device for neural network tensor operator combinations, comprising:

[0030] The initialization module is used to define the computation method of tensor operator combination and obtain the cyclic block size of the reduction iteration space and the bias iteration space in turn.

[0031] The processing module, based on the loop block size, optimizes the tensor operator combination by loop block and loop merging according to the scheduling template, generates a sketch, and annotates the sketch;

[0032] The output module is used to output the tuning parameters of the tensor operator combination based on the annotated sketch processed by the automatic tuning module.

[0033] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0034] By calculating the size of the production-consumption relationship data domain and the cyclic block ratio for data alignment, the operator combination is cyclically blocked with a proportional relationship in block size. This eliminates the conflict of mismatch between producers and consumers, improves the performance of accelerating tasks for operator combinations with data dependencies, and utilizes Ansor to customize the search space of operator dependencies, thereby achieving operator fusion and reducing redundant calculations.

[0035] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0036] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0037] Figure 1 This is a flowchart illustrating an automatic tuning method for neural network tensor operator combination according to an embodiment of this application;

[0038] Figure 2 This is a schematic diagram illustrating the calculation method defined according to an embodiment of this application;

[0039] Figure 3 This is a flowchart illustrating an automatic tuning method for neural network tensor operator combination according to an embodiment of this application;

[0040] Figure 4 This is a block diagram of an automatic tuning device for neural network tensor operator combination according to an embodiment of this application. Detailed Implementation

[0041] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0042] Figure 1 This is a flowchart illustrating an automatic tuning method for neural network tensor operator combination according to an embodiment of this application, such as... Figure 1 As shown, it includes:

[0043] Step 101: Define the calculation method for the combination of tensor operators, and obtain the cyclic block size of the reduction iteration space and the bias iteration space in sequence.

[0044] In this embodiment of the application, for an input tensor A, a convolution kernel B, and an output image C after convolution, if a bias statement operation is performed on the input tensor A before the convolution operation, the iteration space corresponding to the first nested loop is defined as the bias iteration space, and the iteration space corresponding to the second nested loop is defined as the reduction iteration space. There is a producer-consumer relationship between the bias iteration space and the reduction iteration space.

[0045] In this process, the bias iteration space writes data into the data space of the input tensor A, and the reduction iteration space reads data from the data space of the input tensor A.

[0046] In this embodiment of the application, the reduction iteration space is cyclically divided by calling the TVM's overlapping cyclic block scheduling primitive, and the size of the cyclic block of the reduction iteration space is determined. After determining the size of the cyclic block of the reduction iteration space, the size of the cyclic block of the bias iteration space is deduced from the data space.

[0047] In one possible embodiment, H = W = 6, KH = KW = 3, and the h and w loops of the reduction iteration space are divided into 2x2 blocks. The iteration space is constructed based on the two-dimensional space corresponding to the h and w loops, as follows: Figure 2 As shown.

[0048] by Figure 2 For example, considering the data space transformation, the loop blocks of the TVM call are applied to the reduction iteration space to obtain the memory footprint of each loop block in the reduction iteration space for the input tensor A. Based on the memory footprint, and according to the data domain size relationship of the input tensor A, the size of the loop blocks in the bias iteration space is determined.

[0049] This allows cyclic blocks of the same color from different computational spaces to be merged because there is no mismatch in memory footprint. Constructing the cyclic block shape of the bias iteration space by considering data space transformations also reduces the size of the cyclic block size search space, since the user only needs to specify the cyclic block size for the reduction iteration space. The cyclic block size of the bias iteration space can be determined simply by searching the search space for the optimal reduction space cyclic block parameters on the target hardware.

[0050] In addition, the overlapping cyclic block scheduling primitive proposed in step 101 is explained.

[0051] The overlapping cyclic block scheduling primitive proposed in this application is a pre-implemented scheduling primitive that performs overlapping block scheduling on tensor operators based on the block factor and the size of the overlapping portion. After implementing cyclic block scheduling, there are overlapping portions between blocks; this type of cyclic block scheduling is called overlapping block scheduling. The cyclic iteration of the overlapping portion between two adjacent overlapping blocks is calculated multiple times by the two blocks, which is also the cost of achieving full parallelism between blocks through overlapping block scheduling.

[0052] Overlapping block scheduling primitives consist of several stages, each with only one operation. Each operation has a root iterable variable list that determines the shape of the output tensor. Each operation also contains several iterable variables representing iterators. All these iterators are linked together through the iterable variable relation list of that operation. When compiling tensor operator combinations, TVM uses boundary inference to determine the boundaries of the iterators and sets the range of the for loop using these boundaries. This functionality is implemented in the boundary inference process. Boundary inference traverses the directed acyclic graph (DAG) composed of scheduling stages, where each stage node is a directed acyclic graph of iterators consisting of iterators and iterator relations.

[0053] For a directed acyclic graph with overlapping splitting of scheduling stages, the overlapping cyclic block scheduling primitive implements overlapping partitioning through the following formulas:

[0054] Range inner =factor + overlapped num -1;

[0055]

[0056] Among them, Range inner Range is the inner range of the overlapping split leaf iterator. outer Range is the outer range of the overlapping split leaf iterator. root The root iterator range, factor is the block size, and verlapped num This represents the size of the overlapping portion.

[0057] Step 102: Based on the cyclic block size, optimize the tensor operator combination by cyclic block and cyclic merging according to the scheduling template, generate a sketch and annotate the sketch.

[0058] In this embodiment of the application, the scheduling template proposed in step 102 will be described first.

[0059] The scheduling template proposed in this application is constructed as follows: all nodes of the tensor operator combination sample are accessed in topological order; for operator combinations with consumer-producer relationships, the consumer operators are divided into regular cyclic blocks, and the producer operators are divided into overlapping blocks by calling the overlapping block scheduling primitive. Specifically, for the cyclic block structure after block division, the SSRSRS block structure is used on the CPU, and the SSSRRSRS block structure is used on the GPU.

[0060] Specifically, a block factor of 1 indicates that the loop corresponding to that iteration variable does not actually participate in loop block optimization. The block factor for each iteration variable should not exceed twice the size of the corresponding iteration range. If the block factor of an iteration variable is too large, the actual computational load within the last computation block of each dimension after loop block optimization will be much smaller than the computation block size. Since subsequent computation and storage resource allocation is based on the computation block size, this results in resource idleness and waste. This constraint effectively reduces this situation by limiting the size of the block factor.

[0061] Secondly, for GPUs, the size of the computation block does not exceed the upper limit of the thread block size. That is, the product of the block factors corresponding to all iteration variables is less than or equal to the upper limit of the thread block size. This is because subsequent loop binding will map the computation within the computation block to the thread of the thread block on a one-to-one basis. This constraint ensures that the result of the loop block is valid for subsequent loop binding. The size of the overlapping part of the overlapping block can be determined by the user based on experience when defining the tensor operator combination computation method. After all the blocks are completed, the producer and consumer blocks are cyclically merged.

[0062] In addition, in the SSRSRS block structure and the SSSRRSRS block structure, S represents a block structure of a space cycle, and R represents a block structure of a reduction cycle.

[0063] In one possible implementation, for matrix multiplication, the SSRSRS block structure expands the original triple for loop (i, j, k) into (i0, j0, i1, j1, k0, i2, j2, k1, i3, j3).

[0064] After explaining the scheduling template, as follows Figure 3 As shown, step 102 can be further subdivided into:

[0065] Step 301: Perform cyclic segmentation of the consumer operator in the tensor operator combination using TVM, and perform overlapping segmentation of the producer operator.

[0066] Step 302: After all the loop blocks are completed, the consumer operator and the producer operator are cyclically merged to obtain a sketch.

[0067] Step 303: Add random annotations to the sketch based on the Ansor.

[0068] In this embodiment, the fused sketches only have a cyclic block structure and no specific cyclic block factor. After annotating the incomplete sketches with Ansor, they become a complete procedure for fine-tuning and evaluation.

[0069] In addition, Ansor randomly fills the loop block size, parallelizes some outer loops, vectorizes some inner loops, and expands some inner loops. Ansor also randomly changes the calculation position of some nodes in the program, and fine-tunes the loop block structure.

[0070] Step 103: Based on the annotated sketch processed by the automatic tuning module, output the tuning parameters of the tensor operator combination.

[0071] In this embodiment, an automatic tuning module is generated by fine-tuning the performance tuner in the Ansor through evolutionary search and a learnable cost model.

[0072] Specifically, in terms of evolutionary search, firstly, evolutionary search is used to find a small batch of programs that meet the performance requirements based on the learned cost model. Then, these programs are measured on hardware to obtain the actual execution time cost. Finally, the analysis data obtained is used to retrain the cost model to make it more accurate.

[0073] Evolutionary search uses a random sampling procedure and a high-quality procedure from the previous evaluation as the initial population, and applies mutation and overlap to generate the next generation.

[0074] Regarding the learnable cost model, it predicts the performance of each program based on the program's throughput, providing a relatively accurate estimate of program performance, while being several orders of magnitude faster than actual predictions. It can compare tens of thousands of programs in the search space within seconds and select the programs that meet the criteria for practical evaluation.

[0075] In addition, the obtained tuning parameters will be stored in a JSON log file, and the optimal configuration will be retrieved from the log file during the compilation process.

[0076] This application embodiment calculates the size of the production-consumption relationship data domain and the cyclic block ratio for data alignment, and performs cyclic block division of the operator combination with proportional block size. This eliminates the mismatch conflict between producers and consumers, improves the performance of accelerating tasks for operator combinations with data dependencies, and utilizes Ansor to customize the search space of operator dependencies, thereby achieving operator fusion and reducing redundant calculations.

[0077] Figure 4 This is a block diagram of an automatic tuning device for neural network tensor operator combination according to an embodiment of this application, including an initialization module 410, a processing module 420 and an output module 430.

[0078] Initialization module 410 is used to define the calculation method of tensor operator combination and obtain the cyclic block size of the reduction iteration space and the bias iteration space in turn.

[0079] Processing module 420 optimizes the combination of tensor operators by performing cyclic block and cyclic merging based on the cyclic block size and according to the scheduling template, generates a sketch and annotates the sketch;

[0080] Output module 430 is used to output the tuning parameters of tensor operator combination based on the annotated sketch processed by the automatic tuning module.

[0081] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0082] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0083] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A neural network tensor operator combination automatic tuning method for eliminating the conflict of mismatch between producers and consumers, characterized in that, include: Define the computation method for tensor operator combinations, and obtain the cyclic block size of the reduction iteration space and the bias iteration space in turn; For an input tensor A, a convolution kernel B, and an output image C after convolution, if a bias statement operation is performed on the input tensor A before the convolution operation, the iteration space corresponding to the first nested loop is defined as the bias iteration space, and the iteration space corresponding to the second nested loop is defined as the reduction iteration space. There is a producer-consumer relationship between the bias iteration space and the reduction iteration space. Based on the cyclic block size, the tensor operator combination is optimized by cyclic block and cyclic merging according to the scheduling template, a sketch is generated and the sketch is annotated; the scheduling template is constructed in the following way, including: accessing all nodes of the tensor operator combination sample in topological order; for operator combinations with consumer-producer relationships, performing regular cyclic block for consumer operators; calling the overlapping block scheduling primitive for producer operators to perform overlapping block, the overlapping cyclic block scheduling primitive is a pre-implemented scheduling primitive; and implementing overlapping block for tensor operators according to the block factor and the size of the overlapping part. For the cyclic block structure after block division, the SSRSRS block structure is used on the CPU, and the SSSRRSRS block structure is used on the GPU. Based on the annotated sketch processed by the automatic tuning module, the tuning parameters of the tensor operator combination are output.

2. The method according to claim 1, characterized in that, The step of sequentially obtaining the cyclic block size of the reduction iteration space and the bias iteration space includes: By calling the TVM's overlapable cyclic block scheduling primitive, the reduction iteration space is cyclically divided into blocks to determine the size of the cyclic blocks in the reduction iteration space. After determining the cyclic block size of the reduction iteration space, the cyclic block size of the bias iteration space is deduced from the data space.

3. The method according to claim 2, characterized in that, The step of deriving the cyclic block size of the bias iteration space from the data space includes: The loop blocks called by TVM are applied to the reduction iteration space to obtain the memory footprint of each loop block in the reduction iteration space for the input tensor A; Based on the memory footprint, the cyclic block size of the bias iteration space is determined according to the data domain size relationship of the input tensor A.

4. The method according to claim 3, characterized in that, The overlapping cyclic block scheduling primitive implements overlapping block scheduling through the following formulas, including: ； ； in, For the inner range of the overlapping split-page iterator, For the outer range of the overlapping split-after-leaf iterator, For the range of the root iterator, For block size, Size of the overlapping portion.

5. The method according to claim 4, characterized in that, The process of optimizing the tensor operator combination based on the cyclic block size and the scheduling template through cyclic block and cyclic merging, generating a sketch, and annotating the sketch includes: The consumer operator in the tensor operator combination is subjected to cyclic segmentation provided by TVM, and the producer operator is subjected to overlapping segmentation. After all the loop blocks are completed, the consumer operator and the producer operator are cyclically fused to obtain a sketch. The sketch is randomly annotated according to Ansor.

6. The method according to claim 5, characterized in that, The automatic tuning module is constructed in the following manner, including: An automatic tuning module is generated by fine-tuning the performance tuner in Ansor through evolutionary search and a learnable cost model.

7. An automatic tuning device for neural network tensor operator combination, characterized in that, The method applied to any one of claims 1-6 includes: The initialization module is used to define the computation method of tensor operator combination and obtain the cyclic block size of the reduction iteration space and the bias iteration space in turn. The processing module, based on the loop block size, optimizes the tensor operator combination by loop block and loop merging according to the scheduling template, generates a sketch, and annotates the sketch; The output module is used to output the tuning parameters of the tensor operator combination based on the annotated sketch processed by the automatic tuning module.

Images