Heterogeneous accelerator calculation scheduling modeling and optimizing method for quantitative neural network

Abstract

The invention discloses a heterogeneous accelerator calculation scheduling modeling and optimization method for a quantization neural network, and the method comprises an abstract description process based on a quantization algorithm, and comprises the steps: building quantization abstraction for the quantization algorithm in an iteration manner; the representation of the quantization operator comprises a tensor expression obtained by a linear layer operator structure of the deep neural network model and quantization algorithm abstraction; the method comprises a quantitative calculation assembly line generation and optimization process, a cyclic tree scheduling modeling process of a quantitative assembly line on a heterogeneous accelerator, and a cyclic tree scheduling optimization process based on a genetic algorithm. By adopting the technical scheme provided by the invention, an efficient calculation process can be generated for a single quantization algorithm or a composite quantization algorithm, and is automatically converted into efficient scheduling representation on the accelerator, so that the calculation delay of heterogeneous accelerator hardware can be minimized, and the calculation efficiency of the accelerator can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of electronic information technology, and relates to the computing scheduling technology of applying a quantized neural network to heterogeneous accelerators (including in-memory computing accelerators, spatial accelerators, etc.). Specifically, it relates to a computing scheduling modeling and optimization method for heterogeneous accelerators based on a quantized neural network. Background Art

[0002] A neural network is a network structure composed of multiple layers of neurons. Each layer of neurons in a neural network can be represented as a tensor computing operator. Taking the output of the previous layer of neurons, the weights of the operator, etc. as parameters, the calculated result is used as the input of the next layer of neurons or the output of the entire network. A tensor, also known as a multi-dimensional array (ND-Array), is the basic organizational form of neurons and data in a neural network. A tensor can contain multiple dimensions. For example, a matrix can be represented as [M, N], which is a 2D tensor, where the size of the first dimension is M and the size of the second dimension is N. A tensor computing operator refers to a computing function that takes a tensor as an operand. For example, the common matrix-matrix multiply (Matrix-Matrix-Multiply, matrix multiplication) in a neural network can be expressed as C[M, N] = A[M, K] * B[N, K]. Here, A and B are the input operands of the operator, and C is the output operand. A, B, and C are all 2D tensors, with dimensions [M, K], [K, N], and [M, N] respectively. The computing process of these tensor computing operators can be represented as a perfect loop. A perfect loop refers to a nested loop structure, and there is only one computing expression in its innermost part. For matrix-matrix multiply, it can be represented as a perfect loop structure with three nested loops. Its three loop dimensions i, j, k respectively traverse the dimensions M, N, K of the three tensors, and there is only one computing expression C[i, j] += A[i, k] * B[j, k] inside the loop. A quantized neural network is a neural network after low-precision representation of the data in the neural network, used to reduce the memory space required for its storage and computing. For example, representing the weight from the original 32-bit floating-point number as a 4-bit integer can compress the memory space required for weight storage to about 1 / 8 of the original. Quantization can be applied to weights (such as the weight Weight in a linear layer), and can also be applied to the input and output of each layer of neurons (also called activation value Activation). W can be used x A yIt is used to represent the form of quantizing weights into x bits and inputs into y bits. Although the data in the quantized network is stored using low precision, it needs to be restored to the high-precision number domain for calculation during the calculation process. This conversion process from low precision to high precision is the dequantization operation (Dequantize). The implementation of the dequantization operation is related to the design of the quantization algorithm. For example, in uniform quantization, dequantization can be expressed as A2 = A1 * scale + zero. Here, A2 is the high-precision number after dequantization, A1 is the low-precision number before dequantization; scale is the scaling factor determined after network quantization, and zero is the zero-point value determined after network quantization. Both the scaling factor and the zero-point value are parameters determined by the quantization algorithm. The quantized neural network consists of quantization operators, and each quantization operator is obtained by quantizing the original operator. The calculation process of the quantization operator is very different from that of the original operator. It breaks the perfect loop structure of the original operator's calculation process because the introduction of the dequantization operation brings new calculation statements. This non-perfect loop structure poses a great challenge to the scheduling optimization of the operator on the accelerator. With the continuous development of quantization methods, the current quantization algorithms are gradually becoming more complex. In addition to the most basic single quantization, there are also more and more complex composite quantization algorithms emerging. Among them, single quantization means that for an operand (weight or input), only one quantization method is adopted; composite quantization means that for an operand, multiple quantization methods are adopted simultaneously. The increasingly complex quantization methods further complicate the scheduling of quantization operators on the accelerator.

[0003] Heterogeneous accelerators refer to neural network accelerators with different configurations and settings, such as spatial accelerators and processing-in-memory accelerators. The spatial accelerator (Spatial Accelerator) contains an efficient tensor calculation unit, such as a matrix-matrix multiplication unit, a vector / scalar calculation unit, and a multi-level on-chip memory hierarchy for caching operands, such as L0 buffer, L1 buffer, etc. The processing-in-memory accelerator (Processing-in-Memory Accelerator) contains a memory array that implements matrix-vector multiplication (also known as a computing memory array), a vector / scalar calculation unit, and a multi-level on-chip memory hierarchy. The scheduling process of the quantization operator on these accelerators needs to schedule two types of calculation units simultaneously. The tensor calculation unit and the computing memory array are used for high-dimensional tensor calculations, which complete the calculation process of the original operator, while the vector / scalar calculation unit completes the dequantization operation. How to efficiently schedule the above two arithmetic operations in the limited on-chip memory space is a very big challenge.

[0004] Existing scheduling optimization work for quantization operators and numerous single-operator scheduling works lack a comprehensive representation of the scheduling and optimization space for quantization operators. Among them, Ladder (Ladder: Enabling Efficient Low-Precision Deep Learning Computing through Hardware-aware Tensor Transformation, 18th USENIX Symposium on Operating Systems Design and Implementation) is a compiler design for quantization operators, but it is a template-based scheduling. On the one hand, the quantization algorithms it supports are very limited, only supporting single quantization; on the other hand, it is a template-based scheduling, and the obtained scheduling result is sub-optimal. Soter (reference doi: 10.1109 / ISCA59077.2024.00076) is the current best-performing scheduling optimization work for single operators. However, it can only support perfect loop operators and cannot directly support the scheduling of quantization operators, with very large limitations. Summary of the Invention

[0005] In order to overcome the deficiencies in the above-mentioned prior art, such as insufficient support for accelerator quantization algorithms and the inability to systematically model and optimize the scheduling of quantization algorithms on heterogeneous accelerators, the present invention provides a method for modeling and optimizing the computing scheduling of heterogeneous accelerators for quantization neural networks. By using the general method for modeling and optimizing the computing scheduling of quantization algorithms on heterogeneous accelerators provided by the present invention, whether it is a single or composite quantization algorithm, an efficient computing process can be generated and automatically transformed into an efficient scheduling representation on the accelerator, which can minimize the computing latency of the heterogeneous accelerator hardware and improve the computing efficiency of the accelerator.

[0006] The term definitions involved in the present invention include:

[0007] Tensor Expression: In the present invention, the tensor expression is obtained from the operator structure in deep neural networks applied to fields such as images, texts, and videos. A tensor expression is an expression that records the tensor connection between operators. It is an equation where there is only one tensor on the left side of the equation, representing the output result, and there can be multiple tensors on the right side of the equation. Each tensor includes two parts, its descriptor and dimension. For example, the tensor expression of matrix-matrix multiplication (matrix multiplication) can be expressed as C[M,N] = A[M,K] * B[N,K]. Here, C is the descriptor of the output matrix, and A and B are the descriptors of the input matrices. M, N, and K are the dimension representations of the matrices. Through the tensor expression, the loop space of the operator can be determined. For example, the loop space of matrix-matrix multiplication has three levels i, j, and k, with sizes M, N, and K respectively, and the calculation within the loop is described as C[i,j] += A[i,k] * B[j,k].

[0008] Computation Pipeline: When there are multiple expressions in the loop space corresponding to an operator, it is called a computation pipeline. The description of the computation pipeline is a supplement to the perfect loop operator.

[0009] Quantized Computation Pipeline: The computation pipeline formed by the computations of quantized operators is called a quantized computation pipeline.

[0010] Loop Tree: A tree-like structure composed of loops. There are two types of nodes in the tree, scheduling nodes and loop nodes. The scheduling nodes control the execution of the subtree. It can have two attributes, parallel or serial, corresponding to the subtree using different hardware resources to execute in parallel or reusing the same hardware resources in time. The loop nodes record the information of the loops, including the loop variables participating in the loop, the range of the loop variables, the memory hierarchy corresponding to the loop variables, and the scheduling method (parallel or serial).

[0011] The technical solution of the present invention is as follows:

[0012] A heterogeneous accelerator computing scheduling modeling and optimization method for quantized neural networks, including abstract description based on quantization algorithms, generation and optimization of quantized computation pipelines, loop tree scheduling modeling of quantized pipelines on heterogeneous accelerators, and loop tree scheduling optimization based on genetic algorithms; specifically including the following steps:

[0013] 1) The user takes the tensor expression and the quantization algorithm abstraction proposed by the present invention as inputs, and these two together constitute the representation of the quantized operator.

[0014] 2) Automatically generate the computation pipeline of the quantized operator and optimize it:

[0015] Construct an initial computation pipeline for the quantization operator through the representation of the quantization operator, and obtain an optimized computation pipeline for the quantization operator by transforming the computation process of the quantization operator. A computation pipeline represents multiple expressions in the loop space corresponding to a quantization operator. The heterogeneous accelerator includes multiple types of hardware computing units, and each expression in the computation pipeline will be mapped to a type of hardware computing unit on the heterogeneous accelerator. Mapping means that the actual computation process of the expression on the heterogeneous accelerator is completed by the specified hardware computing unit. The core of the transformation is three transformation primitives (transformation types): loop space slicing, loop space blocking, and decomposition primitives.

[0016] The hardware computing units on the heterogeneous accelerator can be represented as a set, and each element in the set is a binary tuple composed of the tensor computation type and the precision of the operand. For example, there are 9 hardware computing units on the heterogeneous accelerator Huawei Ascend 910B, namely: 1) two matrix-matrix multiplication computing units with operand precisions of 8-bit integers and 16-bit floating-point numbers respectively; 2) three vector computing units with operand precisions of 32-bit integers, 16-bit floating-point numbers, and 32-bit floating-point numbers respectively; 3) four tensor computing units with operand precisions of 32-bit integers, 16-bit floating-point numbers, 32-bit floating-point numbers, and 64-bit floating-point numbers respectively. These different computing units will exhibit different throughputs and power consumptions. Different types of hardware computing unit sets will exist on different heterogeneous accelerators, including different tensor computation types and operand precisions. In the present invention, the types of heterogeneous accelerators include spatial accelerators and in-memory computing accelerators.

[0017] Adjust and optimize the structure of the computing pipeline. Specifically, by changing the number of statements, the form of statements, and the order of statements in the computing pipeline, a new computing pipeline can be obtained, which can make better use of hardware computing units. Loop space slicing and loop space blocking will introduce new expressions, and each expression can be mapped to a type of hardware computing unit, so that more hardware computing units can be used, increasing the parallelism of computing and reducing the computing latency. Loop space blocking and decomposition will change the form of expressions. Among them, loop space blocking disassembles the complete tensor calculation into multiple sub-tensor calculations and introduces summation statements between sub-tensors. More sub-tensor calculation statements enable us to utilize more hardware computing units; decomposition will reduce the precision of the operands in the tensor calculation, such as changing the 16-bit floating-point number calculation to an 8-bit integer calculation, so that we can use hardware computing units with lower operand precision, reducing the latency and power consumption required for the calculation. Decomposition will change the order between expressions. The decomposition operation transforms the order of the dequantization operation and the tensor calculation operation during the tensor operator calculation process, and changes the number of dequantization operations required and the precision of the operands used in the tensor calculation operation, so that the computationally intensive tensor operations can be mapped to low-precision computing units. The above transformations of the computing pipeline will result in different expressions in the final computing pipeline, and thus different hardware computing units will be used, resulting in different computing performances, including throughput and power consumption.

[0018] 3) Describe the scheduling space of the optimized computing pipeline on heterogeneous accelerator hardware. The core is to establish the correspondence between the computing pipeline and the heterogeneous accelerator through the loop tree.

[0019] The memory hierarchy on the heterogeneous accelerator can be represented as a set. For example, Ascend910B includes 6 memory hierarchies, namely L0A, L0B, L0C, L1, UB, and L2 memory hierarchies. The correspondence represents the mapping of all tensor data participating in the calculation in the computing pipeline to the memory hierarchy.

[0020] A loop tree is a tree structure composed of loop nodes and scheduling nodes. A loop node can represent the mapping of tensor data in each expression to the memory hierarchy on a heterogeneous accelerator. Each loop node represents a loop, including the number of loop iterations, the execution order of the loop (parallel or serial), and the mapping of this loop to the memory hierarchy on the heterogeneous accelerator. After mapping a loop to a memory hierarchy, all the data used in the calculation of this loop needs to be stored in the corresponding memory hierarchy. The number of loop iterations of this loop will affect the storage space requirement of the tensor data. The more loop iterations there are, the more data needs to be stored in this memory space. The number of data to be stored is the number of data used by the expressions in the calculation pipeline in the loop space covered by this loop node. Whether a loop is parallel or serial will affect the number of hardware computing units used, thus affecting the distribution of data among memory hierarchies: If a loop is parallel, then the number of hardware computing units equal to the number of loop iterations on the heterogeneous accelerator needs to be called simultaneously to increase the parallelism of the calculation, but each computing unit needs to store the data it requires in its used memory hierarchy; if a loop is serial, only one hardware computing unit is needed to complete the calculation of the expressions in the loop space covered by this loop, and only the memory hierarchy used by this computing unit needs to store data. A scheduling node can describe the execution among expressions, which can be serial or parallel. Serial means that multiple expressions are executed sequentially on the heterogeneous accelerator, and the computing time on the heterogeneous accelerator should be the sum of the execution times of each expression; parallel means that multiple expressions are executed simultaneously on the heterogeneous accelerator, and the computing time on the heterogeneous accelerator should be the maximum value among the execution times of each expression. The loop tree composed of the above two types of nodes can represent the scheduling space of the calculation pipeline to the heterogeneous accelerator.

[0021] The loop tree is used to establish the connection between the calculation of the quantization operator and the memory structure on the accelerator hardware, that is, to model the hardware scheduling space of the operator calculation pipeline. By mapping each loop space involved in the operator calculation pipeline to different memory hierarchies of the hardware, a scheduling scheme is obtained. By mapping each loop in the loop space of the operator calculation pipeline to a memory hierarchy of the hardware, the tensor data required for the calculation is mapped to a specific memory hierarchy, such as the data buffer and register on the heterogeneous accelerator. Different mapping schemes generate different data mapping (storage) schemes, thus producing different scheduling schemes.

[0022] 4) Explore the scheduling space through the proposed genetic algorithm with the goal of optimizing the overall computational latency. By adjusting the loop space in the computational pipeline, transformations such as loop splitting and loop order swapping are performed to generate a new loop space, and the loop space is mapped onto the hardware to generate a complete scheduling scheme. The core of its optimization is to optimize the use of memory on the hardware, reduce the data movement and the number of external memory accesses on the hardware, thereby improving the computational efficiency and energy efficiency.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0024] The present invention provides a method for computational scheduling modeling and optimization of a heterogeneous accelerator for a quantized neural network. By using the general quantization algorithm provided by the present invention for computational scheduling modeling and optimization on a heterogeneous accelerator, whether it is a single or composite quantization algorithm, an efficient computational process can be generated and automatically transformed into an efficient scheduling representation on the accelerator, which can minimize the computational latency of the heterogeneous accelerator hardware and improve the computational efficiency of the accelerator. Through the modeling and optimization method of the quantization operator provided by the present invention on a heterogeneous accelerator, support can be provided for composite quantization algorithms that cannot be supported by the prior art. Compared with the current best Ladder framework, a 1.7 - 3.74 times improvement in computational efficiency can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the main components constituting the present invention.

[0026] Figure 2 It is an example of the composite quantization operator applied in the present invention.

[0027] Figure 3 It is an example of the computational pipeline generation and optimization process.

[0028] Figure 4 It is an example of the loop tree used in the scheduling modeling of the quantization computational pipeline of the present invention.

[0029] Figure 5 It is a flowchart of the computational pipeline scheduling optimization using the genetic algorithm designed by the present invention.

[0030] Figure 6 It is an example of loop tree gene transformation. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] Next, the present invention will be further clearly and completely described with reference to the accompanying drawings and specific embodiments.

[0032] The present invention provides a method for modeling and optimizing the computational scheduling of a heterogeneous accelerator for a quantized neural network. A general quantization algorithm is used for the computational scheduling modeling and optimization of the heterogeneous accelerator, including single and composite quantization algorithms, which can generate an efficient computational process and automatically transform it into an efficient scheduling representation on the accelerator, capable of minimizing the computational latency of the heterogeneous accelerator hardware and improving the computational efficiency of the accelerator.

[0033] The method of the present invention includes an abstract description method of the quantization algorithm, a generation and optimization method of the quantized computational pipeline, a loop tree scheduling modeling method of the quantized pipeline on the heterogeneous accelerator, and a loop tree scheduling optimization method based on the genetic algorithm. As Figure 1 shown, it includes four components, namely 1) an input module composed of a tensor expression and a quantization abstraction, used to describe the quantization operator; 2) a quantized computational pipeline generation and optimization module, responsible for converting the initial computational pipeline into an optimized computational pipeline through three transformation primitives, namely loop space slicing, loop space blocking, and decomposition primitives; 3) a scheduling model of the computational pipeline on the accelerator with a loop tree representation as the core, used to describe the scheduling space of the computational pipeline on the accelerator hardware; 4) a scheduling optimization module of the computational pipeline on the accelerator, using the genetic algorithm as the optimization algorithm and the computational latency as the objective, to explore the scheduling space, so as to obtain a high-performance mapping scheme.

[0034] In specific implementation, the tensor expression is obtained according to the structure of the deep neural network applied to processing objects such as images, texts, and videos, and the quantization abstraction is obtained according to the design work of the currently proposed quantization algorithm.

[0035] The method for modeling and optimizing the computational scheduling of a heterogeneous accelerator for a quantized neural network according to the present invention includes the following steps:

[0036] 1) Input: It includes two parts, a tensor expression and a quantization abstraction. Specifically as follows:

[0037] a) The tensor expression is the original tensor expression of the operator in the network before quantization, which can be obtained according to the configuration of each network layer in the deep neural network.

[0038] b) The quantization abstraction is a representation based on a general quantization algorithm proposed by the present invention. The present invention proposes a general abstraction of quantization established in an iterative manner. For a single quantization algorithm, it is described using a triple (dtype, dfunc, dstruct), namely the data type of quantization (dtype), the dequantization function (dfunc), and the quantization structure (dstruct). The data type of quantization describes the data format after quantization, such as 4-bit integers, etc.; the quantization structure describes the quantization granularity, that is, in what size of data blocks a set of quantization parameters (such as scaling factors, zero points, etc.) are shared. A composite quantization algorithm can be constructed in an iterative manner: the iterative manner includes two types, existing simultaneously in space, that is, Qs = Q1, Q2, where Qs is a composite quantization algorithm, which applies the two quantization algorithms Q1 and Q2 to different two blocks of the tensor; sequentially in time, that is, Qs = Q1(Q2), where Qs is a composite quantization algorithm, and Q1 and Q2 are two quantization algorithms, which are respectively applied to this tensor, that is, first quantize through Q1, and then quantize using Q2 on this basis. Therefore, the general quantization abstraction can be iteratively described as Qs = Qs, Qs|Qs(Qs)|(beg, end, (dtype, dfunc, dstruct)). The "|" therein represents "or", and the three parts on the right side of the equal sign respectively represent iteration in space, iteration in time, and the final single quantization representation. Beg is the starting position of this quantization method in the tensor, and end is the ending position of this quantization method in the tensor.

[0039] 2) The present invention proposes a method for automatically generating and optimizing the calculation pipeline of quantization operators: through the input operator description and quantization algorithm description, automatically generate and optimize the calculation pipeline of quantization operators.

[0040] a) The present invention generates and optimizes the calculation pipeline form of quantization operator calculations in a search manner: three calculation pipeline transformations are defined in the present invention. By continuously applying them to the current set of feasible calculation pipelines, all feasible quantization calculation pipeline representations can be obtained.

[0041] b) The three transformations respectively correspond to loop space slicing, loop space blocking, and decomposition.

[0042] i) Loop space slicing divides the loop space of operator calculations into two independent blocks. The calculations in each block are relatively independent. Finally, the results in the two loop spaces are spliced together or accumulated together to obtain the original calculation result.

[0043] ii) Loop space blocking is to slice a certain dimension of the loop space and convert it into two dimensions, the inner and the outer. It is required that the inner loop maintains the same calculation form in each instance of the outer loop dimension.

[0044] iii) Decomposition is to move the dequantization operation before the tensor calculation, and only the transformation that ensures the same calculation results before and after is a valid transformation.

[0045] The above three transformations will all change the structure of the calculation pipeline, thus bringing different expressions. Different expressions involve different precisions and calculation forms. Each expression is represented as a binary tuple, which are the calculation form of the expression itself and the precision of the operands in the expression respectively. All the expressions in the pipeline constitute an expression set. Each type of hardware calculation unit on the heterogeneous accelerator is also represented as a binary tuple, which are the form of the calculation completed by the calculation unit and the data precision used respectively. All the calculation units on the heterogeneous accelerator constitute a calculation unit set. Each expression is mapped to a type of hardware calculation unit, indicating that the corresponding calculation unit completes all the calculation processes in the expression. During the mapping process, it is necessary to satisfy both the calculation form matching and the precision matching, that is, the calculation form of the calculation unit can combine all the calculation logics in the expression calculation form, and the precision of each operand of the hardware calculation unit can represent each operand of the expression without loss.

[0046] c) In the initial calculation pipeline, all dequantization operations are performed first, and then the tensor calculation operations in the original operator are performed. The initial set of feasible calculation pipelines only includes this initial calculation pipeline. By iteratively applying the above three transformations to the set of feasible calculation pipelines, all possible quantization operator calculation pipelines can be generated. The set composed of these possible quantization operator calculation pipelines is the optimization space. In this step, each calculation pipeline in the space can be represented as a set of expressions. This step further maps the set represented by each pipeline to the set of hardware calculation units of the heterogeneous accelerator. Each expression is mapped to a calculation unit in the set of hardware calculation units of the heterogeneous accelerator, but each calculation unit can be mapped by multiple expressions at the same time. The set of hardware calculation units that each type of expression can be mapped to is a subset of the set of hardware calculation units on the heterogeneous accelerator, and it is configured through a template in this step. The template directly records an entry table to record all feasible mappings. Each entry is similar to a matrix-matrix multiplication expression of 16-bit floating-point numbers, which can be mapped to a vector calculation unit of 16 or 32-bit floating-point numbers. The matrix-matrix multiplication expression of 16-bit floating-point numbers can be mapped to a 16-bit matrix-matrix multiplication hardware calculation unit.

[0047] In actual operation, through breadth-first search and the above three conversions, the initial computing pipeline is converted into a set of feasible computing pipelines. The implementation logic of the breadth-first search strategy is as follows: 1) Configure templates. For each type of computing pipeline structure, set in advance the conversion schemes that can be performed on this pipeline; 2) For each pipeline in the set of possible pipelines, perform corresponding conversions by checking the templates, and add the obtained feasible computing pipelines to the set of possible pipelines; 3) When the new pipeline set generated after the conversion of all computing pipelines in the set of possible pipelines is the same as that before the conversion or the number of pipelines in the pipeline set reaches a predetermined threshold, stop the search process. In this step, each feasible pipeline can be represented by a set of expressions. For each feasible computing pipeline, each expression therein is further mapped to the computing units of the hardware to ensure that each expression can be computed using one computing unit without affecting the accuracy.

[0048] 3) Quantized computing pipeline scheduling model based on loop tree: Use the loop tree structure to represent the scheduling space of the optimized computing pipeline on the hardware.

[0049] a) The present invention adopts the description method of loop tree to represent the computing scheduling space of quantization operators on heterogeneous accelerators. Each scheduling of the quantized computing pipeline on the hardware can be represented as a loop tree. There are two types of nodes on the loop tree, loop nodes and scheduling nodes. Loop nodes represent the computing process of each expression on the heterogeneous accelerator; scheduling nodes represent the computing process between expressions. Loop nodes record the number of loop iterations, loop scheduling method (parallel / serial), and the memory hierarchy on the heterogeneous accelerator to which the loop is mapped. Scheduling nodes represent the scheduling method (parallel / serial) between different expressions. Loop nodes and scheduling nodes together determine the computing process of a computing pipeline on the heterogeneous accelerator.

[0050] b) The loop nodes map the tensor data required for each expression to different memory hierarchies of the heterogeneous accelerator. The mapping of the data used in each expression to the memory hierarchies is determined by the mapping of the loop nodes to the memory hierarchies. If a loop node is mapped to a specific memory hierarchy, within the loop space represented by that loop node, all the data required for the computation of the expression is stored in that specific memory hierarchy. The number of loop iterations of the loop node controls the size of the storage space required for a memory hierarchy. The number of loop iterations contained in the loop node determines the size of the loop space it covers, and the number of each tensor data required for the expression computation is determined by the size of the loop space. In this step, the storage space required for the tensor data of each operation in each memory hierarchy can be expressed as the number of that data used in the loop space covered by the loop nodes mapped to that memory hierarchy. The number of computing units used is determined by the scheduling attribute (parallel / serial) of the loop node: 1) Serial: The computation of the expression in the loop space represented by this loop is completed by a single hardware unit alone; 2) Parallel: The loop space represented by this loop is completed by the number of hardware computing units equal to the number of loop iterations of this node. The scheduling attribute of the scheduling node determines the execution order of the expressions: 1) Parallel: All the expressions under this scheduling node are computed simultaneously. The tensor operands required for these expressions need to be stored in the on-chip memory hierarchy of the heterogeneous accelerator simultaneously, and the execution time of these expressions; 2) Serial: All the expressions under this scheduling node are computed sequentially,

[0051] b) The loops on the loop tree correspond to the memory hierarchy of the hardware. Taking the Huawei Ascend910B chip as an example, the on-chip memory hierarchy includes L1, L0A, L0B, L0C, UB, L2, etc. By mapping each loop in the loop tree to the above-mentioned memory hierarchy, we can obtain a scheduling scheme for the computing pipeline on the hardware. After mapping a loop to an on-chip memory hierarchy, all the tensor data used for the computation of this loop will also be stored in this memory hierarchy. After obtaining all the loops mapped to a memory hierarchy, the storage space used by this memory hierarchy is determined. The data transfer volume between different memory hierarchies can also be determined through the mapping relationship of the memory hierarchy. Assume that there is a connection between two memory hierarchies L0 and L1, the loop space mapped to L0 is Loop0, and the loop space mapped to L1 is Loop1. The loop space corresponding to Loop0 is contained in Loop1. For each transformation of the points in the loop space of Loop1, if it causes an instance transformation of the loop space of Loop0, the difference set between the tensor data required for this new instance and the tensor data used by the original instance is the data transfer volume each time. With the change of the loop order, loop space transformation, and the mapping relationship transformation from the loop to the hardware memory hierarchy on the loop tree, we can obtain different loop tree structures, and thus different scheduling schemes. There are huge differences in the use of the memory hierarchy on heterogeneous accelerators for different scheduling schemes.

[0052] 4) Scheduling optimization of the quantization computing pipeline on the hardware: Adopt a scheduling space exploration method based on the genetic algorithm.

[0053] a) The present invention regards the loop tree as a genome and designs a genetic algorithm, including genome transformation and fitness metrics for genetic space exploration.

[0054] b) Basic process of the designed genetic algorithm: Perform genome transformation on the optional genomes in the current gene set, and compare the fitness of the transformed genome with that of the original genome. If the fitness of the new genome is higher than before, then add the new genome to the genome set; otherwise, randomly add it to the genome set or discard it to avoid generating local rather than global optimal genomes. Through continuous iteration, the final low-latency scheduling result can be obtained.

[0055] c) Genome transformation method: Each genome corresponds to a loop tree, and the transformation of the genome is the transformation of the loop tree.

[0056] The present invention defines 7 types of loop tree transformation methods. Transformation 1: Change the memory hierarchy of the scheduling nodes of the loop tree; Transformation 2:

[0057] Changing the scheduling attributes of nodes; Transformation 3: Changing the order of subtrees; Transformation 4: Changing the order of scheduling nodes; Transformation 5:

[0058] Changing the loop size of loop nodes; Transformation 6: Changing the order of loop nodes; Transformation 7: Changing the scheduling attributes of loop nodes.

[0059] d) Fitness metric: For a genome, the reciprocal of the execution time of the loop tree structure it represents on hardware is defined as its fitness. That is, the higher the latency, the lower the fitness.

[0060] e) The above genome transformation methods mainly transform the loop space and adjust the mapping scheme of loops to the hardware memory hierarchy. Among them, Transformations 2, 3, 4, 5, 6, and 7 mainly adjust the loop space and the computational statements in the pipeline. Transformations 2, 3, and 4 change the scheduling method of computational statements, that is, whether they are parallel or serial, the execution order between statements, and the structure when statements are executed alternately. These transformations will result in different usage amounts of different memory hierarchies of the hardware computing unit. For example, by adjusting the parallel and serial execution methods, the number of tensor data simultaneously present in the memory hierarchy is different. When computing in parallel, all tensor data needs to be stored, but when computing serially, only the tensor data on one subtree needs to be stored at a time.

[0061] Transformations 5, 6, and 7 change the loop space of the computation, corresponding to the loop size of each loop, the order between loops, and the scheduling method of each loop, that is, serial or parallel. Different loop spaces will also bring different data storage space requirements, thus resulting in different requirements for memory storage space and also affecting the requirements for on-chip data transfer and off-chip data access. The transformation corresponding to 1 corresponds to mapping the loop to different memory hierarchies, which can directly change the amount of tensor data required for each memory hierarchy. For example, after mapping a loop from the L0 buffer to the L1 buffer, the tensor data used by this loop in the corresponding L0 will also be moved to the L1 buffer, thus bringing more data transfer from L1 to L0, but reducing the space required for data storage on L0, and more space can be given to the tensors involved in other statements.

[0062] The present invention designs a modeling and optimization method for a quantization algorithm on a heterogeneous accelerator. Given an accelerator architecture with TensorCore as the core tensor computing unit, matrix-matrix multiplication as the operator input, and a current state-of-the-art spatio-mixed quantization algorithm Atom as the quantization algorithm input (such as Figure 2 ), this algorithm is widely used in the field of large models. The specific process of scheduling modeling and optimization is as follows:

[0063] 1) The input of the present invention is a tensor expression and a quantization abstraction. The tensor expression in this example is the linear layer operator in the currently widely used large language models, which is widely applied to various text processing tasks, such as text generation,

[0064] automatic question answering, etc.; the source of the quantization algorithm is Atom (paper title: Atom: Low-Bit Quantization for

[0065] Efficient and Accurate LLM Serving, published conference: Proceedings of Machine Learning and

[0066] Systems), and in this example, the present invention makes a formal abstraction of it. Specifically as follows:

[0067] a) Tensor expression: Matrix-matrix multiplication operator C[M,N] = A[M,K] * B[N,K], where in this example

[0068] M = N = K = 4096. Here, A and B are input tensors;

[0069] b) Quantization abstraction: For each input tensor (A or B) participating in the operation, an algorithm description of quantization needs to be input. In this example (as Figure 2 shown), the quantization algorithm descriptions of A and B are the same, both as shown below:

[0070] i. Qs = Qs1, Qs2

[0071] ii. Qs1 = ([0,0], [4096, 4096 - 128], (4-bit integer, y = scale * x, [1, 128])), where the range of its action is [0,0] to [4096, 4096 - 128]. Each number is a 4-bit integer, the dequantization function is y = scale * x, scale is the scaling factor, and x and y are the numbers before and after dequantization respectively. [1, 128] is the structure description, representing that every [1, 128] block in the tensor shares a scaling factor.

[0072] iii. Qs2 = ([0, 4096 - 128], [4096, 4096], (8-bit integer, y = scale * x, [1, 128])), and its operating range is from [0, 4096 - 128] to [4096, 4096]. Each number is an 8-bit integer, the dequantization function is y = scale * x, where scale is the scaling factor, and x and y are the numbers before and after dequantization respectively. [1, 128] is the structure description, representing that every [1, 128] block in the tensor shares a scaling factor.

[0073] 2) Automatic generation of the quantization operator calculation pipeline, as Figure 3 shown:

[0074] a) The input operator description and quantization algorithm description first generate an initial quantization calculation pipeline. Its form is to perform all dequantization operations first, and then perform tensor calculation operations, that is, matrix-matrix multiplication. The schematic diagram of its calculation pipeline is Figure 3 (I).

[0075] b) Through loop space slicing transformation, the initial matrix-matrix multiplication can be split into two independent matrix-matrix multiplication operations according to different quantization strategies, and then the results are accumulated. The schematic diagram of its calculation pipeline is Figure 3 (II).

[0076] c) Through loop space blocking transformation, the sliced matrix-matrix multiplication operation can be changed into two steps, first perform block matrix-matrix multiplication, and then accumulate all block results. The schematic diagram of its calculation pipeline is Figure 3 (III).

[0077] d) Through decomposition transformation, move the dequantization operation after the tensor calculation. That is, first perform tensor operation, and then perform dequantization operation. The schematic diagram of its calculation pipeline is Figure 3 (IV).

[0078] 3) Quantization calculation pipeline scheduling modeling, as Figure 4 shown:

[0079] By converting the quantization calculation pipeline into the form of a loop tree, the modeling of the scheduling of the calculation pipeline on hardware is obtained, as Figure 4As shown in the figure. The rounded rectangles represent loop nodes, and the loop nodes represent loops. For example, loops i and j in l1. Each loop node is bound to a memory hierarchy. For example, the l1 loop node is bound to L2, the l2 node is bound to the memory hierarchy L1; the loop nodes of l3 to l9 are bound to the memory hierarchy L0. s1 to s3 are three scheduling nodes that control the execution order of the subtree. Among them, the attribute of s2 is parallel. The expressions represented by loop nodes l6 and l7 use different hardware resources on the hardware and can be parallel; the attribute of s3 is also parallel. The attribute of s1 is serial. The expressions represented by l6 and l8 use the same computing resources (tensor computing resources) on the hardware and complete the calculation serially.

[0080] 4) Scheduling optimization of the quantization calculation pipeline on the accelerator:

[0081] a) Figure 5 The figure shows the schematic flow chart of using the genetic algorithm for scheduling optimization. For the loop tree genes in the loop tree gene set, each gene is as Figure 4 shown. Then, perform gene transformation on it to obtain a new gene representation and measure its running latency. If the latency is lower than before the transformation, add it to the gene set; otherwise, add it to the gene set with a probability, and the probability can be any number less than 1.

[0082] b) Figure 6 The figure shows the gene transformation methods adopted in the scheduling optimization process.

[0083] i. Figure 6 (a) and (b) in it represent the transformation of the scheduling node in the memory hierarchy.

[0084] ii. Figure 6 (c) in it represents the transformation of the scheduling node attribute, that is, serial or parallel.

[0085] iii. Figure 6 (d) in it represents the transformation of the order of the subtree (expression) under the scheduling node.

[0086] iv. Figure 6 (e) in it represents the transformation of the scheduling node order. It can be seen that the order of the two scheduling nodes has changed before and after the transformation.

[0087] v. Figure 6 (f) in it represents the transformation of the loop node size. Taking the right arrow as an example, the size of the top loop node changes from s0 to s0 / t, and the sizes of the following two loop nodes change from s1 and s2 to

[0088] s1*t and s2*t.

[0089] vi. (g) in Figure (6) represents the transformation of the loop node order. Before the transformation, loop j is above i, and after the transformation, loop i is above j.

[0090] vii. Figure 6 (h) in it represents the transformation of the loop node scheduling attribute. Taking the transformation indicated by the right arrow as an example: before the transformation, the loop node is parallel, indicating that all instances in this loop are executed together; after the transformation, the loop node is serial, representing that all loop instances are executed sequentially.

[0091] Through the modeling and optimization method of the quantization operator provided by the present invention on heterogeneous accelerators, support can be provided for complex quantization algorithms that could not be supported before. Compared with the current best Ladder framework, a 1.7 - 3.74 times improvement in computing efficiency can be achieved.

[0092] It should be noted that the purpose of publishing the embodiments is to help further understand the present invention. However, those skilled in the art can understand that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection claimed by the present invention shall be defined by the scope defined in the claims.

Claims

1. A heterogeneous accelerator computing scheduling modeling and optimization method for quantized neural networks, characterized in that It includes the following steps: 1) Establish a quantization abstraction for the quantization algorithm through an iterative approach to obtain the representation of the quantization operator; the representation of the quantization operator includes a tensor expression and a quantization algorithm abstraction; 2) Automatically generate and optimize the computation pipeline of the quantization operator: construct the initial computation pipeline of the quantization operator through the representation of the quantization operator, and obtain the optimized computation pipeline of the quantization operator by transforming the computation process of the quantization operator; Each expression in the computation pipeline is mapped to the hardware computation unit on the heterogeneous accelerator, that is, the actual computation process of the expression on the heterogeneous accelerator is completed by the hardware computation unit; 3) Model the loop tree scheduling of the quantization pipeline on the heterogeneous accelerator: establish the correspondence between the computation pipeline and the heterogeneous accelerator through the loop tree, and describe the scheduling space of the optimized computation pipeline on the heterogeneous accelerator hardware; Map the loop space in the operator computation pipeline to different memory levels of the heterogeneous accelerator hardware, thereby obtaining different scheduling schemes; 4) Optimization of loop tree scheduling based on genetic algorithm: explore the scheduling space through the genetic algorithm with the goal of optimizing the overall computation latency; including: a) Use the loop tree as the genome in the genetic algorithm, design the genetic algorithm, including genome transformation and fitness metrics, for genetic space exploration; the genome transformation method includes multiple transformation types; the fitness is the reciprocal of the execution time of the loop tree structure represented by the genome on the heterogeneous accelerator hardware; b) Design the process of the genetic algorithm as follows: perform genome transformation on the optional genomes in the current genome set; and compare the fitness of the transformed genome with that of the original genome; If the fitness of the new genome is higher, add the new genome to the genome set, otherwise randomly add it to the genome set or discard it; through continuous iteration, the final low-latency scheduling result can be obtained.

2. The heterogeneous accelerator computing scheduling modeling and optimization method for quantized neural networks according to claim 1, characterized in that The specific tensor expression is obtained from the linear layer operator structure in the deep neural network model for processing images, texts, and videos.

3. The heterogeneous accelerator computing scheduling modeling and optimization method for a quantized neural network according to claim 1, characterized in that, The process of automatically generating the quantization operator computation pipeline includes: 21) Generate the initial quantization computation pipeline according to the input operator description and quantization algorithm description; its form is to perform all dequantization operations first, and then perform tensor computation operations, that is, matrix multiplication; determine the loop space of the operator through the tensor expression of matrix multiplication; 22) Through loop space cutting transformation, split the initial matrix multiplication into two independent matrix multiplication operations according to different quantization strategies, and then accumulate the results; 23) Through loop space block transformation, turn the cut matrix multiplication operation into two steps, first perform block matrix multiplication, and then accumulate all the block results; Through decomposition transformation, move the dequantization operation after the tensor computation, that is, perform tensor operation first and then dequantization operation.

4. The heterogeneous accelerator computing scheduling modeling and optimization method for a quantized neural network according to claim 1, characterized in that, In step 2), specifically, a new computation pipeline is obtained by changing the number of statements, statement forms, and statement order of the computation pipeline.

5. The heterogeneous accelerator computing scheduling modeling and optimization method for the quantized neural network according to claim 1, wherein The types of transformation include loop space cutting, loop space block, and decomposition primitive.

6. The heterogeneous accelerator computing scheduling modeling and optimization method for a quantized neural network according to claim 1, wherein Genomic transformation methods are used to transform the loop space and adjust the mapping of loops to the heterogeneous accelerator hardware memory hierarchy; including 7 types of loop tree transformation methods; Transformation 1: Change the memory hierarchy of the loop tree scheduling node; Transformation 2: Change the scheduling attributes of the node; Transformation 3: Change the order of subtrees; Transformation 4: Change the order of scheduling nodes; Transformation 5: Change the loop size of the loop node; Transformation 6: Change the order of loop nodes; Transformation 7: Change the scheduling attributes of loop nodes.

7. The heterogeneous accelerator computing scheduling modeling and optimization method for a quantized neural network according to claim 6, wherein Transformations 2 to 7 adjust the loop space and the computational statements in the pipeline; the transformation corresponding to Transformation 1 corresponds to mapping the loop to different memory hierarchies, which can directly change the amount of tensor data required for each memory hierarchy.

8. The heterogeneous accelerator computing scheduling modeling and optimization method for a quantized neural network according to claim 7, characterized in that, Transformations 2 to 4 change the scheduling method of computational statements, that is, whether they are parallel or serial, the execution order between statements, and the structure when statements are executed alternately; thus resulting in different usage amounts of different memory hierarchies by the hardware computing unit; Transformations 5 to 7 change the loop space of the computation, corresponding to the loop size of each loop, the order between loops, and the scheduling method of each loop, that is, serial or parallel; different loop spaces bring different data storage space requirements, resulting in different requirements for memory storage space, and affecting the requirements for on-chip data transfer and off-chip data access.

Images