Blocking strategy generation method and device for tensor calculation

Abstract

A method for generating a blocking strategy for tensor calculation, the method comprising: acquiring information about a plurality of tensor operations corresponding to tensor calculation, the information about each tensor operation comprising a tensor calculation dimension corresponding to the tensor operation, a data type of an element corresponding to the tensor calculation dimension, and a priority of the tensor calculation dimension (S301); determining a correspondence between the plurality of tensor operations and a plurality of hardware units, the plurality of hardware units being used to compute tensor calculations (S302); obtaining a partitioning strategy of the tensor calculation dimension corresponding to the tensor operation based on the characteristic information of the hardware unit corresponding to the tensor operation, the storage space corresponding to the tensor operation, the data type of the element corresponding to the tensor calculation dimension and the priority of the tensor calculation dimension (S303); and obtaining a blocking strategy of tensor calculation based on a blocking strategy of a tensor calculation dimension corresponding to the plurality of tensor operations (S304). The partitioning strategy code with excellent performance is generated through the method, and the overall performance of tensor calculation is improved.

Description

A method and device for generating a block strategy for tensor calculation Technical Field

[0001] The present application relates to the technical field of artificial intelligence technology, and in particular to a method and device for generating a block strategy for tensor calculation. Background Art

[0002] Deep learning is playing an increasingly important role in our daily lives, for example in natural language processing (NLP) applications such as language translation and text understanding and analysis, and in computer vision applications such as facial recognition. Deep learning primarily relies on tensor computations, which are typically stored in computers as multidimensional arrays. Tensor computations can be described by a series of nested loops.

[0003] To improve the performance of tensor computations, various new heterogeneous hardware acceleration solutions have been designed and launched, such as heterogeneous accelerators that support matrix multiplication, improve data reuse, and utilize multi-level caches. Hardware vendors typically provide supporting computational libraries, but developing these libraries requires significant expert time and effort, and they can only adapt hardware to a limited set of known operators. With the rapid advancements in deep learning, neural network models are becoming increasingly diverse and complex, requiring computational graphs to describe these models and structures with countless operator combinations.

[0004] However, when the block strategy for tensor calculation generated according to the prior art is applied to tensor calculation, the performance of tensor calculation is still relatively low.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a method for generating a block strategy for tensor computing, which generates block strategy code with excellent performance, reduces the computing overhead of the hardware accelerator, and thereby improves the overall performance of tensor computing.

[0007] In the first aspect, the present application provides a method for generating a block strategy for tensor calculation, obtaining information of multiple tensor operations corresponding to the tensor calculation, the information of each tensor operation including the tensor calculation dimension corresponding to the tensor operation, the data type of the element corresponding to the tensor calculation dimension, and the priority of the tensor calculation dimension; determining the correspondence between multiple tensor operations and multiple hardware units, and the multiple hardware units are used to perform tensor calculations; based on the characteristic information of the hardware unit corresponding to the tensor operation, the storage space corresponding to the tensor operation, the data type of the element corresponding to the tensor calculation dimension, and the priority of the tensor calculation dimension, obtaining the block strategy of the tensor calculation dimension corresponding to the tensor operation, the characteristic information of the hardware unit indicates the storage characteristics or computing characteristics of the hardware unit; based on the block strategy of the tensor calculation dimension corresponding to multiple tensor operations, a block strategy for tensor calculation is obtained.

[0008] The block strategy generation method for tensor calculation provided in the embodiment of the present application adapts the block strategy of the tensor calculation dimension corresponding to each tensor operation to the required hardware characteristics, ensuring the quality of the block strategy, so that the code generated based on the block strategy generation method for tensor calculation provided in the embodiment of the present application has excellent performance. At the same time, there is no need to measure the pros and cons of the block strategy through execution and measurement, thereby improving the generation speed of the block strategy for tensor calculation.

[0009] In one possible implementation, based on the characteristic information of the hardware unit corresponding to the tensor operation, the storage space corresponding to the tensor operation, the data type of the element corresponding to the tensor calculation dimension, and the priority of the tensor calculation dimension, a blocking strategy for the tensor calculation dimension corresponding to the tensor operation is obtained, including: obtaining the blocking strategy for the first tensor calculation dimension based on the characteristic information of the hardware unit corresponding to the tensor operation and the data type of the element corresponding to the first tensor calculation dimension; obtaining the blocking strategy for the second tensor calculation dimension based on the storage space corresponding to the tensor operation and the blocking strategy for the first tensor calculation dimension; obtaining the blocking strategy for the tensor calculation dimension corresponding to the tensor operation based on the blocking strategy for the first tensor calculation dimension and the blocking strategy for the second tensor calculation dimension; wherein, the first tensor calculation dimension and the second tensor calculation dimension are both tensor calculation dimensions corresponding to the tensor operation, and the priority of the first tensor calculation dimension is higher than the priority of the second tensor calculation dimension.

[0010] In this possible implementation, a blocking strategy is gradually generated for the tensor calculation dimension of each tensor operation. That is to say, a blocking strategy is first generated for the tensor calculation dimension with high priority among the multiple tensor calculation dimensions corresponding to each tensor operation, and it is made to meet the characteristics of the hardware and achieve legalization. Then, a maximized blocking strategy is generated for the tensor calculation dimension with lower priority, achieving legality first and then efficiency, and generating the best performance segmentation strategy for each tensor calculation dimension.

[0011] In one possible implementation, the storage space corresponding to the tensor operation is related to the number of tensor operations corresponding to the hardware unit.

[0012] In another possible implementation, it also includes: performing storage reuse analysis on each tensor operation to determine the tensor operations with reusable storage space among the multiple tensor operations corresponding to the hardware unit; and updating the number of tensor operations corresponding to the hardware unit based on the storage reuse analysis results of each tensor operation.

[0013] In another possible implementation, the information of the tensor operation also includes the operation type corresponding to the tensor operation; determining the correspondence between multiple tensor operations and multiple hardware units includes: obtaining at least one hardware unit corresponding to the tensor operation based on the correspondence between the operation type of the tensor operation and the hardware unit.

[0014] In another possible implementation, the multiple hardware units include a storage unit and a computing unit, the characteristics of the storage unit include storage granularity, and the characteristics of the computing unit include computing batch size.

[0015] It is understood that storage granularity means the minimum granularity of each read or write of a storage unit. For example, if the storage granularity of storage unit A is 32B, it means that the minimum granularity of each read or write of storage unit A is 32B. Computational batch size means the computing resources required for a computing unit to perform a calculation. For example, if the computational batch size of computing unit A is 128B, it means that computing unit A needs to consume 128B of computing resources to perform a calculation.

[0016] In another possible implementation, the multiple hardware units include different types of storage units and / or different types of computing units.

[0017] In the second aspect, the present application provides a method for generating a block strategy for tensor calculation, including: obtaining information of multiple tensor operations corresponding to tensor calculation, the information of each tensor operation includes an operation type and a hardware label, and the hardware label represents the hardware unit on which the operation type depends; determining a topological structure corresponding to multiple hardware units, the topological structure includes multiple nodes, and the nodes represent hardware units related to tensor calculation; based on the operation type and the hardware label, establishing a mapping relationship between multiple tensor operations and multiple nodes in the topological structure corresponding to multiple hardware units, the topological structure is a tree-like structure, including a root node, a branch node and a leaf node; determining a block strategy for the tensor calculation dimension corresponding to the tensor operation corresponding to the node for multiple nodes in the direction from the root node to the leaf node; obtaining a block strategy for tensor calculation based on the block strategy for the tensor calculation dimension corresponding to the tensor operation corresponding to each node in the multiple nodes.

[0018] The block strategy generation method for tensor calculation provided in the embodiment of the present application abstracts the structure of the hardware accelerator into a tree topology structure, maps each tensor operation of the tensor calculation to each node of the tree topology structure, and converts the block strategy analysis of the tensor calculation into an analysis and solution of the graph structure (tree topology structure), that is, the analysis of each node in the tree topology structure. The block strategy of tensor calculation can be obtained by analyzing each node, which does not require expensive and sensitive profiling hardware to measure performance, thereby improving the generation speed of the tensor calculation block strategy.

[0019] In a possible implementation, the topology structure further includes multiple unidirectional edges, where the unidirectional edges are used to connect different nodes and represent the relationship between the nodes.

[0020] In another possible implementation, the multiple unidirectional edges include first-type unidirectional edges and / or second-type unidirectional edges and / or third-type unidirectional edges; wherein, the first-type unidirectional edges represent that the blocking strategy of the tensor calculation dimension corresponding to the tensor operations corresponding to the nodes connected by the first-type unidirectional edges is the same; the second-type unidirectional edges represent that there is no need to determine the blocking strategy for the tensor calculation dimension corresponding to the tensor operations corresponding to the nodes pointed to by the second-type unidirectional edges; the third-type unidirectional edges represent the communication connection between the hardware units represented by the nodes connected by the third-type unidirectional edges.

[0021] In another possible implementation, the blocking strategy of the tensor calculation dimension corresponding to the tensor operation corresponding to the node is related to the allocated storage space of the hardware unit represented by the node; the allocated storage space of the hardware unit represented by the node is related to the number of available sibling nodes of the node, where the sibling node is a node at the same layer as the node and represents the same type of hardware unit.

[0022] In another possible implementation, the number of available sibling nodes of a node is related to the data size corresponding to the tensor operation and the number of sibling nodes.

[0023] In another possible implementation, the hardware units represented by the root node and the branch node are storage units, and the hardware units represented by the leaf node are computing units.

[0024] In another possible implementation, the multiple hardware units include different types of storage units and / or different types of computing units.

[0025] In a third aspect, the present application provides a device for generating a block strategy for tensor calculation, comprising:

[0026] An acquisition module, configured to acquire information of multiple tensor operations corresponding to tensor calculations, wherein the information of each tensor operation includes a tensor calculation dimension corresponding to the tensor operation, a data type of an element corresponding to the tensor calculation dimension, and a priority of the tensor calculation dimension;

[0027] a determination module, configured to determine a correspondence between the plurality of tensor operations and a plurality of hardware units, the plurality of hardware units being configured to perform operations on the tensor calculations;

[0028] a blocking strategy generating module, configured to obtain a blocking strategy for the tensor calculation dimension corresponding to the tensor operation based on characteristic information of the hardware unit corresponding to the tensor operation, a storage space corresponding to the tensor operation, a data type of an element corresponding to the tensor calculation dimension, and a priority of the tensor calculation dimension, wherein the characteristic information of the hardware unit indicates a storage characteristic or a calculation characteristic of the hardware unit;

[0029] Based on the block strategies of the tensor calculation dimensions corresponding to the multiple tensor operations, a block strategy for the tensor calculation is obtained.

[0030] In one possible implementation, the block strategy generation module is specifically configured to:

[0031] Obtaining a blocking strategy for the first tensor calculation dimension based on characteristic information of the hardware unit corresponding to the tensor operation and a data type of an element corresponding to the first tensor calculation dimension;

[0032] Obtaining a blocking strategy for a second tensor calculation dimension based on a storage space corresponding to the tensor operation and the blocking strategy for the first tensor calculation dimension;

[0033] Obtaining a blocking strategy for the tensor calculation dimension corresponding to the tensor operation based on the blocking strategy for the first tensor calculation dimension and the blocking strategy for the second tensor calculation dimension;

[0034] The first tensor calculation dimension and the second tensor calculation dimension are both tensor calculation dimensions corresponding to the tensor operation, and the priority of the first tensor calculation dimension is higher than the priority of the second tensor calculation dimension.

[0035] In another possible implementation, the storage space corresponding to the tensor operation is related to the number of the tensor operations corresponding to the hardware unit.

[0036] For example, the storage space corresponding to the tensor operation is 8KB, that is, the storage space that the hardware unit A corresponding to the tensor operation can allocate for tensor calculation is 8KB. The number of tensor operations corresponding to the hardware unit A is 4, then the storage space that the hardware unit A can allocate for one tensor operation is 8 / 4=2KB, that is, the storage space corresponding to one tensor operation is 2KB.

[0037] In another possible implementation, the method further includes:

[0038] a reuse analysis module, configured to perform storage reuse analysis on the plurality of tensor operations corresponding to the hardware unit to determine a tensor operation with reusable storage space among the plurality of tensor operations corresponding to the hardware unit;

[0039] An update module is used to update the number of tensor operations corresponding to the hardware unit based on the storage reuse analysis results of the respective tensor operations.

[0040] In another possible implementation, the information of the tensor operation further includes an operation type corresponding to the tensor operation;

[0041] The determining module is specifically configured to:

[0042] Based on the correspondence between the operation types corresponding to the multiple tensor operations and the hardware units, at least one hardware unit corresponding to the tensor operation is obtained.

[0043] In another possible implementation, the multiple hardware units include a storage unit and a computing unit, the characteristics of the storage unit include storage granularity, and the characteristics of the computing unit include computing batch size.

[0044] In another possible implementation, the multiple hardware units include different types of storage units and / or different types of computing units.

[0045] In a fourth aspect, the present application further provides a device for generating a block strategy for tensor calculation, comprising:

[0046] an acquisition module, configured to acquire information of multiple tensor operations corresponding to the tensor calculation, wherein the information of each tensor operation includes an operation type and a hardware tag, wherein the hardware tag represents a hardware unit on which the operation type depends, and the multiple hardware units are used to perform the tensor calculation;

[0047] a determination module, configured to determine a topological structure corresponding to the plurality of hardware units, the topological structure comprising a plurality of nodes, the nodes representing the hardware units related to the tensor calculation;

[0048] a mapping module, configured to establish, based on the operation type and the hardware label, a mapping relationship between the plurality of tensor operations and a plurality of nodes in a topological structure corresponding to the plurality of hardware units, wherein the topological structure is a tree structure including a root node, branch nodes, and leaf nodes;

[0049] A block strategy generation module is used to determine, for a plurality of nodes, a block strategy of tensor calculation dimensions corresponding to the tensor operations corresponding to the nodes in a direction from the root node to the leaf node;

[0050] Based on the block strategy of the tensor calculation dimension corresponding to the tensor operation corresponding to each of the multiple nodes, the block strategy of the tensor calculation is obtained.

[0051] In another possible implementation, the topological structure further includes a plurality of unidirectional edges, where the unidirectional edges are used to connect different nodes, and the unidirectional edges represent the relationship between the nodes.

[0052] In another possible implementation, the plurality of unidirectional edges include first-type unidirectional edges and / or second-type unidirectional edges and / or third-type unidirectional edges;

[0053] The first type of unidirectional edge represents that the tensor calculation dimension corresponding to the tensor operation corresponding to the nodes connected by the first type of unidirectional edge is the same as the blocking strategy;

[0054] The second type of unidirectional edge representation does not require determining a blocking strategy for the tensor calculation dimension corresponding to the tensor operation corresponding to the node pointed to by the second type of unidirectional edge;

[0055] The third type of unidirectional edge represents the communication connection between the hardware units represented by the nodes connected by the third type of unidirectional edge.

[0056] In another possible implementation, the blocking strategy of the tensor calculation dimension corresponding to the tensor operation corresponding to the node is related to the allocated storage space of the hardware unit represented by the node;

[0057] The allocated storage space of the hardware unit represented by the node is related to the number of available sibling nodes of the node. The sibling nodes are nodes located at the same layer as the node and represent the same type of hardware units.

[0058] In another possible implementation, the number of available sibling nodes of the node is related to the data size corresponding to the tensor operation and the number of the sibling nodes.

[0059] In another possible implementation, the hardware units represented by the root node and the branch node are storage units, and the hardware units represented by the leaf nodes are computing units.

[0060] In another possible implementation, the plurality of hardware include different types of storage units and / or different types of computing units.

[0061] In a fifth aspect, the present application provides a computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method provided in the first aspect and / or second aspect of the present application.

[0062] In a sixth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the method provided in the first aspect and / or the second aspect of the present application.

[0063] In a seventh aspect, the present application provides a computer program or a computer program product, wherein the computer program or computer program product comprises instructions, and when the instructions are executed, the method provided in the first aspect and / or the second aspect of the present application is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] FIG1a is a schematic diagram of an application scenario of a block strategy generation method for tensor calculation;

[0065] Figure 1b is a schematic diagram of tensor calculation without block division;

[0066] Figure 1c is a schematic diagram of the calculation after tensor calculation is divided into blocks;

[0067] FIG1d is a flow chart of a method for implementing the first solution;

[0068] FIG1e is a flow chart of a method for implementing the second solution;

[0069] FIG2a is a method for generating a block strategy for tensor calculation provided in an embodiment of the present application;

[0070] FIG2 b is a method for generating another block strategy for tensor calculation provided in an embodiment of the present application;

[0071] FIG2c is another method for generating a block strategy for tensor calculation provided in an embodiment of the present application;

[0072] FIG3 is a schematic diagram of a tree topology;

[0073] FIG4 is a schematic diagram of analyzing each node layer by layer based on a tree topology structure;

[0074] FIG5 is a diagram illustrating the relationship between tensor operations and hardware in the Ascend 910;

[0075] FIG6 is a schematic diagram of the tree topology corresponding to the Ascend 910;

[0076] FIG7 is a schematic diagram of tensor calculation axis information;

[0077] FIG8 is a schematic diagram of a tree topology structure after tensor operation mapping;

[0078] FIG9 is a tree topology diagram of a tensor operation mapping after reuse analysis;

[0079] FIG10 is a schematic diagram of tensor calculation axis information with available storage size information;

[0080] FIG11 is a schematic diagram of tensor calculation axis information after optimizing and adjusting available storage size information;

[0081] FIG12 is a schematic diagram of tensor calculation axis information with block strategy information;

[0082] FIG13 is a schematic diagram of a block strategy generated by a block strategy generation method for tensor calculation provided in an embodiment of the present application;

[0083] FIG14 is a schematic diagram of another tensor calculation axis information;

[0084] FIG15 is a schematic diagram of another tree topology structure after tensor operation mapping;

[0085] FIG16 is a tree topology diagram of another tensor operation mapping after reuse analysis;

[0086] FIG17 is a schematic diagram of another tensor calculation axis information with available storage size information;

[0087] FIG18 is a schematic diagram of another type of tensor calculation axis information after optimizing and adjusting the available storage size information;

[0088] FIG19 is a schematic diagram of another tensor calculation axis information with block strategy information;

[0089] FIG20 is a schematic diagram of another tensor calculation axis information with block strategy information;

[0090] FIG21 is a schematic diagram of another tensor calculation axis information with block strategy information;

[0091] FIG22 is a schematic diagram of another tensor calculation axis information with block strategy information;

[0092] FIG23 is a schematic diagram of another tensor calculation axis information with block strategy information;

[0093] FIG24 is a schematic diagram of a block strategy generated by a block strategy generation method for tensor calculation provided in an embodiment of the present application;

[0094] FIG25 is a schematic diagram showing the comparison of execution time of the fusion operator with reduction under different block strategies;

[0095] FIG26 is a schematic diagram showing the comparison of execution time of the matrix multiplication fusion operator under different blocking strategies;

[0096] FIG27a is a schematic structural diagram of a block strategy generating device for tensor calculation provided by an embodiment of the present application;

[0097] FIG27 b is a schematic structural diagram of another apparatus for generating a block strategy for tensor calculations provided in an embodiment of the present application;

[0098] FIG27c is a schematic structural diagram of another apparatus for generating a block strategy for tensor calculations provided in an embodiment of the present application;

[0099] Figure 28 is a schematic diagram of the structure of the computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0100] The technical solution of the present application is further described in detail below through the accompanying drawings and examples.

[0101] In order to better understand the technical solutions provided by the embodiments of the present application, some of the terms are briefly introduced below.

[0102] An operator can be thought of as a mapping from one function space to another. Any operation on any independent variable, such as raising a variable to a power or taking a square root, can be considered an operator.

[0103] A tensor can be understood as an n-dimensional array, where n is an integer greater than or equal to 1. It is an n-dimensional generalization of scalars, vectors, and matrices. For example, a scalar is an order-0 tensor, a vector is an order-1 tensor, and a matrix is ​​an order-2 tensor. In machine learning model training, both training data and intermediate computation results can be considered tensors.

[0104] A computational graph refers to a method of describing tensor computations (such as neural network models) using a graph structure. A neural network model can be abstracted as a directed graph structure consisting of tensors and operators. The graph structure includes nodes and directed edges. Nodes represent operators, and directed edges describe the relationship between operators. They are used to represent the dependency between operators and the data flow of tensors.

[0105] Tensor computing hardware accelerators are specialized accelerated computing hardware used for tensor computations (e.g., multiplication and addition of tensor data). Examples of tensor computing hardware accelerators include graphics processing units (GPUs) from NVIDIA, tensor processing units (TPUs) from Google, and neural-network processing units (NPUs) from Huawei.

[0106] Figure 1a is a schematic diagram of an application scenario of a block strategy generation method for tensor calculation. As shown in Figure 1a, the tensor data in the tensor calculation (such as tensor A and tensor B) are blocked, and the blocked tensors can be called tensor blocks. The blocked tensor blocks are then calculated separately to finally obtain the result of the tensor calculation. For example, the tensor is a second-order tensor, and the tensor calculation is a matrix calculation. Before the tensor is blocked, each row and column of tensor A and tensor B needs to be read from the memory, which has a large memory overhead. After blocking, tensor A and tensor B are blocked into multiple smaller tensor blocks. The tensor blocks can be placed in the cache and reused, saving memory overhead.

[0107] Figure 1b is a schematic diagram of a tensor calculation without block division. As shown in Figure 1b, tensor calculation is a matrix calculation, and tensor calculation is not block-divided. Tensors A and B are input tensors, and after the operator operation, tensor C is output. In other words, tensor A and tensor B are calculated to obtain tensor C. For each point in tensor C (representing each element in tensor C), a row of tensor A and a column of tensor C must be fully read once. At this time, the memory overhead required for tensor calculation is 2×M×N×K, where M represents the length of the vertical dimension of tensor A (that is, the number of elements in a column of tensor A), N represents the length of the horizontal dimension of tensor B, and K represents the length of the horizontal dimension of tensor A or the length of the vertical dimension of tensor B.

[0108] Figure 1c is a schematic diagram of the calculation after tensor calculation is divided into blocks. As shown in Figure 1c, tensor calculation is a matrix calculation. After the tensor calculation is divided into blocks, tensors A and B are divided into T×T tensor blocks. The memory overhead required for tensor calculation is 2×M×N×K / T. Tensors A and B are divided into T×T tensor blocks and can be retained in the cache for reuse.

[0109] After the tensor calculation is divided into blocks, the communication time can be reduced through the multi-level storage unit of the hardware accelerator, the parallel computing unit can be used to reduce the computing time, and the storage and computing pipeline can be used to hide the time.

[0110] In order to improve the performance of tensor computing, increasingly complex hardware accelerators have been designed. For example, the GPU launched by NVIDIA includes hardware units such as SM, Warp, and Tensor Core; the TPU launched by Google includes hardware units such as TPU Chip, TPU Core, MXU, and VPU; and the Ascend NPU launched by Huawei includes hardware units such as AI Core, CUB, VEC, UB, L1, LOA, L0B, and L0C.

[0111] Most of the above-mentioned accelerators are heterogeneous accelerators, including heterogeneous parallel computing units and heterogeneous storage units. Different types of computing units have different execution modes, parallelism and granularity, etc. Different types of storage units have different storage spaces and multi-level storage topologies, etc.

[0112] The blocking strategy for tensor computation needs to fully consider the topological structure of the hardware accelerator and the characteristics of each hardware unit to generate an excellent blocking strategy so that the tensor computation can be more closely aligned with the hardware accelerator and better utilize the performance of the hardware accelerator.

[0113] However, traditional block strategy generation methods are mostly targeted at neatly nested homogeneous architectures, making heterogeneous architectures difficult to analyze clearly and efficiently. In addition, the fusion operator calculation graph requires the mixed use of different computing and storage units, further increasing the difficulty of analysis.

[0114] In order to obtain a more optimal tiling strategy (tiling size) for tensor computation, so that tensor computation is more compatible with the hardware accelerator and the performance of the hardware accelerator is better utilized, the embodiments of the present application propose the following solutions:

[0115] The first approach uses a loop-nested scheduling tree as input, finds the mapping between storage size limits and loop-nested axes, calculates the ratio of data movement to computation, and uses these as constraints to search for the best block strategy using a greedy algorithm. The specific implementation, as shown in Figure 1d, includes the following steps: S1. Based on the computational dependencies of operators in the loop-nested scheduling tree, deduce the surviving operators at each moment; S2. Map the surviving operators at each moment to storage, and establish a mapping relationship between each operator axis and the storage size limit; S3. Find the data axis involving data movement and the computation axis involving computation in the scheduling tree; S4. Using the found axes, generate a mapping between data movement and axes, then a mapping between computation and axes, and calculate the ratio between the data movement mapping and the computation mapping. S5. Using the storage mapping from step S2 and the ratio from step S4 as constraints, use a greedy algorithm to search for the best block strategy.

[0116] However, this solution has the following problems: when the computation graph is composed of multiple operators, multiple different hardware units of the accelerator need to be called simultaneously. The combination of different computing units and storage units produces different constraints. Using a single data-to-computation ratio is difficult to meet all conditions at the same time, and it is impossible to search for an excellent blocking strategy for complex operators; and storage deduction only considers the computational dependencies between operators, but does not consider the possibility of concurrency between data reading, computing, and writing back. Therefore, it is impossible to accurately calculate the possibility of reading, writing, and computing, and therefore it is impossible to accurately calculate the storage size of the accelerator that can read, write, and compute concurrently. As a result, it may be impossible to find an excellent blocking strategy or the blocking strategy found is an invalid blocking strategy.

[0117] The second solution is: using a loop nested template as input, using an evolutionary model to generate a block strategy, and measuring the performance to determine the quality of the strategy, continuously updating the evolutionary model, and approaching the best block strategy. For details, please refer to Figure 1e, which includes the following steps: S11. If it is the first iteration, then obtain the block strategy range within the loop nested template and randomly generate a block strategy; if it is not the first iteration, then update the evolutionary model and add the block strategy generated by the model to the new round of randomly generated block strategies. S12. Deploy the block strategy in the loop block calculation of the fusion operator, compile and generate executable code, execute it on the hardware and measure the performance. S13. Determine whether better performance is no longer obtained or whether the number of iterations threshold is reached; if so, then output the block strategy; if not, then continue to execute step S11; S14. Repeat steps S11 to S13 until the iteration stops and the final block strategy is output.

[0118] However, this solution has the following problems: the search will stop when the number of iterations reaches the threshold, resulting in the excellent block strategy still not being searched; each generated block strategy needs to be actually executed and measured to determine the performance, which takes a lot of time and cannot quickly find an excellent block strategy; and the performance measurement is based on the execution time measurement value, which is very sensitive to software and hardware configuration and load changes, and is easily affected, leading to subsequent search errors and other problems.

[0119] In order to solve the problems existing in the above-mentioned solutions and the blocking strategies of the prior art, an embodiment of the present application provides a blocking strategy generation method for tensor calculation, which generates the blocking strategy code with the best performance, reduces the computing overhead of the hardware accelerator, and thereby improves the overall performance of tensor calculation; at the same time, there is no need to measure the pros and cons of the blocking strategy through execution and measurement, thereby improving the generation speed of the blocking strategy for tensor calculation.

[0120] FIG2a is a method for generating a block strategy for tensor calculation provided by an embodiment of the present application. As shown in FIG2a, the method includes steps S201 to S205.

[0121] In step S201, information of each tensor operation corresponding to the tensor calculation is obtained.

[0122] Among them, the information of each tensor operation may include the tensor calculation dimension corresponding to the tensor operation, the number of elements in the tensor calculation dimension, the data type of the element, the operation type and the hardware label, and the hardware label indicates the hardware unit on which the operation type depends.

[0123] In one example, information about each tensor operation corresponding to the tensor calculation can be obtained based on the intermediate representation (IR) corresponding to the tensor calculation and the relationship table between the tensor calculation and the hardware accelerator. The intermediate representation corresponding to the tensor calculation includes the tensor calculation dimensions corresponding to each tensor operation corresponding to the tensor calculation, as well as the number of elements corresponding to each tensor calculation dimension, the data type of the elements, and the operation type of the tensor operation; the relationship table between the tensor calculation and the hardware accelerator includes the correspondence between the operation type and each hardware unit in the hardware accelerator.

[0124] By analyzing the intermediate expressions corresponding to tensor calculations, we can obtain the corresponding nested loop expressions. The nested loop expressions corresponding to tensor calculations describe each operation in the tensor calculation during the execution of the computational graph, including the type of operation and the dimensions involved. For example, the addition operation of two-dimensional tensors can be expressed as: result(dim0, dim1) = operant0(dim0, dim1) + operant1(dim0, dim1), which includes the addition operation "+", the two operands "operant0" and "operant1", and the two dimensions "dim0" and "dim1" involved in the operation.

[0125] In some other examples, tensor calculation operations can also be expressed through other methods such as tree structures and pointer forms.

[0126] The relationship table between tensor operations and hardware accelerators includes information about the hardware units used for each tensor operation (read, write, or compute). For example, matrix multiplication requires a matrix compute unit, vector multiplication requires a vector compute unit, and reading data from a primary storage unit requires a primary storage unit.

[0127] It should be noted that, unless otherwise specified, the tensor calculation axis or axis mentioned below has the same meaning as the tensor calculation dimension.

[0128] In step S202 , a topology structure corresponding to the hardware accelerator is obtained; the topology structure includes a plurality of nodes, and the nodes represent hardware units related to tensor calculations.

[0129] Hardware accelerators are specialized computing hardware used to perform tensor addition and multiplication operations, reducing the time required for tensor calculations. Examples of hardware accelerators include GPUs, Tensor Processing Units (TPUs), and NPUs.

[0130] Obtain hardware information of the hardware accelerator, where the hardware information may include information about each hardware unit of the hardware accelerator and topological structure information of the hardware units. Exemplarily, the information about each hardware unit includes characteristic information of each hardware unit. For example, when the hardware unit is a storage unit, the characteristics of the storage unit include storage granularity; when the hardware unit is a computing unit, the characteristics of the computing unit include computing batch size. For example, the topological structure information of the hardware unit includes communication connection relationships between the hardware units, such as the communication connection relationships between the storage units and the communication connection relationship between the storage unit and the computing unit.

[0131] Optionally, the characteristics of the storage unit also include a read speed.

[0132] Optionally, the memory cell includes a programmable memory cell and / or a non-programmable memory cell.

[0133] The hardware information is abstractly expressed as a graph structure, which includes multiple nodes and connecting edges representing the relationship between nodes. The nodes represent hardware units related to tensor calculations (such as storage units and computing units), and the connecting edges represent the communication relationship between the various hardware units.

[0134] Optionally, the graph structure is a tree topology structure, that is, the hardware information of the hardware accelerator is abstractly expressed as a tree topology structure, which includes a root node, branch nodes and leaf nodes. The root node and branch nodes represent hardware units for storage, and the leaf nodes represent hardware units for computing.

[0135] In one example, as shown in FIG3 , the tree topology consists of N layers of storage nodes (root nodes and branch nodes) and one layer of calculator nodes. The nodes are connected by edges.

[0136] Each node in the tree topology has information about the corresponding hardware characteristics. For example, a compute node has information about the computational batch size, and a storage node has information about the storage granularity. Each compute node has its own input storage and output storage.

[0137] In another example, the connecting edge is a unidirectional edge, and the unidirectional edge includes a first type of edge and / or a second type of edge and / or a third type of edge; wherein the first type of unidirectional edge represents that the node to which the unidirectional edge points (in the direction of the arrow) has the same block strategy as the node at the other end of the unidirectional edge; the second type of unidirectional edge represents that the node to which the unidirectional edge points (in the direction of the arrow) does not need to determine the block strategy; the third type of edge represents the communication connection between the hardware units represented by the nodes connected by the unidirectional edge. For example, the third type of unidirectional edge is an ordinary unidirectional edge, that is, there is no label on the unidirectional edge, the first type of unidirectional edge is Cost = 0, that is, Cost = 0 is labeled on the unidirectional edge, representing that the unidirectional edge is a second type of unidirectional edge, the second type of unidirectional edge is Bypass, that is, Bypass is labeled on the unidirectional edge, representing that the unidirectional edge is a third type of unidirectional edge.

[0138] Through this step, the hardware information of the hardware accelerator is expressed as a hierarchical topology description, and the concurrency of computing and reading and writing is converted into the relationship between layers to participate in the analysis of block strategy generation.

[0139] It should be noted that, in the embodiment of the present application, the hardware information of the hardware acceleration is simply abstractly expressed as a tree topology structure, which does not mean that the topology structure of each hardware unit in the hardware accelerator is actually a tree topology structure.

[0140] In step S203 , each tensor operation is mapped to a corresponding node based on the operation type and the hardware label.

[0141] The tensor operations corresponding to the operation types are mapped to the corresponding nodes in the tree topology structure according to their corresponding hardware labels.

[0142] In step S204, based on the characteristic information of the hardware unit corresponding to the node, the allocated storage space of the hardware unit and the data type of the element corresponding to the tensor calculation dimension corresponding to the tensor operation, the blocking strategy of the tensor calculation dimension corresponding to the tensor operation corresponding to the node is obtained. The allocated storage space of the hardware unit is obtained based on the storage space of the hardware unit and the number of tensor operations corresponding to the node.

[0143] The tensor calculation dimensions corresponding to the nodes have different priorities. The order of generating the block strategies for the tensor calculation dimensions corresponding to the nodes is determined according to the priorities of the tensor calculation dimensions.

[0144] There are many different ways to determine the priority of tensor calculation dimensions (also called tensor calculation axes or axes). For example, all known compilers prioritize storage from the outermost axis (such as axis ax0 in Figure 10) to the innermost axis (such as axis ax1 in Figure 10) and calculation on the innermost axis. Therefore, the priority of storage is from the outermost axis to the innermost axis, and the priority of calculation is the innermost axis. Of course, the implementation of this application also supports other types of priority determination methods, such as from the inside to the outside or starting from the second axis, etc. You can choose the appropriate tensor calculation axis priority determination method as needed. The embodiment of this application does not specifically limit the specific tensor calculation axis priority determination method.

[0145] For example, the tensor calculation dimension corresponding to the node includes a first tensor calculation dimension and a second tensor calculation dimension, and the priority of the first tensor calculation dimension is higher than the priority of the second tensor calculation;

[0146] First, based on the characteristic information of the hardware unit corresponding to the node and the data type of the elements corresponding to the first computational dimension, a blocking strategy for the first tensor computational dimension is obtained; that is, the first computational dimension is made to satisfy the characteristics of the hardware unit corresponding to the node. Then, based on the allocated storage space of the hardware unit and the blocking strategy for the first tensor computational dimension, a blocking strategy for the second tensor computational dimension is determined. In other words, a maximum blocking strategy that does not exceed the allocated storage space is generated for computational dimensions other than the first computational dimension.

[0147] The allocated storage space corresponding to the node is obtained based on the storage space of the hardware unit and the number of tensor operations corresponding to the node. For example, if the storage space of the hardware unit corresponding to the node is 256KB and the number of tensor operations corresponding to the node is 2, then the allocated storage space corresponding to the node is 256 / 2=128KB.

[0148] It can be understood that the allocated storage space corresponding to a node means the storage space allocated by the hardware unit corresponding to the node to the tensor calculation dimension corresponding to the node.

[0149] It should be noted that, unless otherwise specified, the available storage size mentioned below has the same meaning as the allocated storage space.

[0150] In one example, the allocated storage space of the hardware unit corresponding to the node is also related to the data of the available sibling nodes corresponding to the node. For example, if the number of sibling nodes of the node is N and the available number of sibling nodes is M, then the allocated storage space corresponding to the node is the storage space of the node divided by M. For example, if the node M is 32 and the storage space corresponding to the node is 256KB, then the allocated storage space of the node is 256 / 32=8KB.

[0151] The number of available sibling nodes M is related to the data size S of the tensor operation and the number of similar sibling nodes N. For example, if S>N, M takes the greatest common divisor of S and N, and if S≤N, M=S.

[0152] In one example, an analysis order of multiple nodes in a tree topology structure is determined based on the tree topology structure, and a block strategy is generated for each node according to the order, corresponding to a tensor calculation dimension of a tensor operation corresponding to the node.

[0153] That is, based on the tree topology, a block strategy is progressively prepared at each node in a specific order. For example, based on the tree topology, nodes are traversed layer by layer from top to bottom (from root to leaf nodes). That is, based on the hierarchical relationship determined by the tree topology, block strategies are determined for the tensor computation dimensions corresponding to the nodes in each layer in a top-down (from root to leaf) order. As shown in Figure 4, a block strategy is first determined for the nodes in layer N (i.e., the tensor computation dimensions corresponding to the nodes), then for the nodes in layer N-1…, then for the nodes in layer 2, then for the nodes in layer 1, and finally for the computation nodes in the computation layer. Each layer is traversed in an arbitrary order. When nodes of different types appear in the same layer, the priority of node analysis is determined based on the priority information of the connecting edges in the tree topology. That is, the connecting edges also have priority attributes. A block strategy generation method is called at each non-root node to generate block strategies corresponding to all tensor computation axes. If there are multiple blocking strategies for the same tensor computation axis, the blocking strategy that satisfies all conditions is selected. If you do not need to output the blocking strategy of a certain node, you can skip calling the blocking strategy generation method for that node.

[0154] In this step, the blocking strategy of each node is analyzed in topological order, so that different hardware units called by the computation graph composed of complex fusion operators can be fully analyzed.

[0155] In step S205 , a blocking strategy for tensor calculation is obtained based on the blocking strategies for each tensor calculation dimension.

[0156] The blocking strategy of each tensor calculation dimension is used to determine the blocking strategy of the entire tensor calculation. According to the blocking strategy of the tensor calculation, the tensor data is blocked into multiple tensor blocks, and then the blocked tensor blocks are calculated separately to finally obtain the result of the tensor calculation.

[0157] In another example, the method for generating a blocking strategy for tensor calculations provided in an embodiment of the present application also includes performing memory reuse analysis on the tensor calculation operations corresponding to each node, and updating the number of tensor calculation operations corresponding to each node based on the memory reuse analysis results.

[0158] Optionally, the hardware accelerator has a heterogeneous architecture, that is, the hardware accelerator includes multiple storage units, the multiple storage units include different types of storage units, and / or includes multiple computing units, the multiple computing units include different types of computing units.

[0159] The block strategy generation method for tensor calculation provided in the embodiment of the present application optimizes the iterative optimization of the uncertainty in the original block strategy of tensor calculation into a few steps of approximation to generate an excellent block strategy for tensor calculation through a progressively prepared block strategy; at the same time, it provides a block strategy generation method for tensor calculation based on hardware unit characteristics, such as the parallelism of each hardware unit, the storage granularity of the storage unit, and the calculation batch size of the computing unit, to derive an excellent block strategy without having to measure the pros and cons of the block strategy through execution and strategy.

[0160] On the other hand, it also avoids the geometric expansion of the solution space caused by combining computing constraints and hardware architecture constraints, while avoiding the convergence problems and local optimal problems of heuristic algorithms.

[0161] The method for generating a blocking strategy for tensor calculation provided in the embodiment of the present application can be applied to a variety of scenarios involving tensor calculation. For example, in the training scenario of a neural network, an efficient blocking strategy for tensor calculation can be provided within a given time for situations where new calculation graphs are frequently generated due to operations such as modulating training parameters and changing network structure; it can be applied to scientific computing scenarios to meet the demand for instant blocking strategies for calculations generated by optimizations such as dynamic shape; and it can be applied to usage scenarios of hardware design manufacturers so that their computing libraries can quickly generate blocking strategies to meet users' performance requirements for new complex operators.

[0162] Figure 2b shows a flow chart of another method for generating a block strategy for tensor calculation. As shown in Figure 2b, the method includes steps S301 to S304.

[0163] In step S301, information of multiple tensor operations corresponding to tensor calculations is obtained, where the information of each tensor operation includes the tensor calculation dimension corresponding to the tensor operation, the data type of the element corresponding to the tensor calculation dimension, and the priority of the tensor calculation dimension.

[0164] The specific method for obtaining information about multiple tensor operations corresponding to tensor calculations can be found in the above description. For the sake of brevity, it will not be repeated here.

[0165] In step S302, a correspondence between a plurality of tensor operations and a plurality of hardware units is determined, and the plurality of hardware units are used to perform operations on tensor calculations.

[0166] The information of the tensor operation also includes the operation type corresponding to the tensor operation; based on the operation type corresponding to the tensor operation and the relationship table between the tensor operation and the hardware accelerator, the correspondence between the tensor operation and each hardware unit in the hardware accelerator is obtained; wherein the relationship table between the tensor operation and the hardware accelerator includes the correspondence between the operation type and each hardware unit in the hardware accelerator.

[0167] In step S303, based on the characteristic information of the hardware unit corresponding to the tensor operation, the storage space corresponding to the tensor operation, the data type of the element corresponding to the tensor calculation dimension, and the priority of the tensor calculation dimension, the blocking strategy of the tensor calculation dimension corresponding to the tensor operation is obtained, wherein the characteristic information of the hardware unit indicates the storage characteristics or computing characteristics of the hardware unit. For example, when the hardware unit is a storage unit, the characteristics of the hardware unit are storage characteristics (such as storage granularity); when the hardware unit is a computing unit, the characteristics of the hardware unit are computing characteristics (such as computing batch size).

[0168] Based on the characteristic information of the hardware unit corresponding to the tensor operation and the data type of the element corresponding to the first tensor calculation dimension, the blocking strategy of the first tensor calculation dimension is obtained; based on the storage space corresponding to the tensor operation and the blocking strategy of the first tensor calculation dimension, the blocking strategy of the second tensor calculation dimension is obtained; based on the blocking strategy of the first tensor calculation dimension and the blocking strategy of the second tensor calculation dimension, the blocking strategy of the tensor operation is obtained; wherein, the first tensor calculation dimension and the second tensor calculation dimension are both tensor calculation dimensions corresponding to the tensor operation, and the priority of the first tensor calculation dimension is higher than the priority of the second tensor calculation dimension.

[0169] Exemplarily, the tensor calculation dimensions corresponding to the tensor operation include at least a first tensor calculation dimension and a second tensor calculation dimension, and the data types of the elements of the first tensor calculation dimension and the second tensor calculation dimension corresponding to the tensor operation are both float32; the hardware unit corresponding to the tensor operation is a storage unit, and the characteristic information of the storage unit is the storage granularity. For example, the storage granularity of the storage unit is 32B, and the allocated storage space of the storage unit is 4KB. When storing, the priority of the first tensor calculation dimension is higher than the priority of the second tensor calculation dimension. Therefore, the blocking strategy is first generated for the first tensor calculation dimension (higher priority), and then the blocking strategy is generated for the second tensor calculation dimension (lower priority). Make the element type of the first tensor calculation dimension in the tensor operation (i.e. float32, size is 4B) meet the characteristics of the storage unit (storage granularity is 32B), that is, 32B / 4B=8, and the blocking strategy of the first tensor calculation dimension corresponding to the tensor operation is 8. Then, according to the allocated storage space 4KB based on the storage unit and the blocking strategy 8 of the first tensor calculation dimension, the blocking strategy of the second tensor calculation dimension is obtained, that is, 44*1024B / 4B / 8=128. Finally, the blocking strategy of the first tensor calculation dimension corresponding to the tensor operation is 8 (i.e., every 8 elements are a tensor block), and the blocking strategy of the second tensor calculation dimension is 128 (every 128 elements are a tensor block).

[0170] In step S304, a blocking strategy for the tensor calculation is obtained based on the blocking strategies for the tensor calculation dimensions corresponding to the plurality of tensor operations.

[0171] Based on the partitioning strategy of the tensor calculation dimensions corresponding to each tensor operation, the partitioning strategy for the entire tensor calculation is determined. Based on this partitioning strategy, the tensor data is partitioned into multiple tensor blocks. The partitioned tensor blocks are then calculated separately to obtain the final tensor calculation result, thereby improving the speed of the entire tensor calculation.

[0172] Figure 2c shows a flow chart of another method for generating a block strategy for tensor calculation. As shown in Figure 2c, the method includes steps S401 to S405.

[0173] In step S401, information of multiple tensor operations corresponding to tensor calculations is obtained. The information of each tensor operation includes an operation type and a hardware tag. The hardware tag represents the hardware unit on which the operation type depends. Multiple hardware accelerators are used to perform operations on tensor calculations.

[0174] The specific method for obtaining information about multiple tensor operations corresponding to tensor calculations can be found in the above description. For the sake of brevity, it will not be repeated here.

[0175] In step S402 , a topological structure corresponding to a plurality of hardware units is determined. The topological structure includes a plurality of nodes, and the nodes represent hardware units related to tensor calculation.

[0176] Obtain hardware information of the hardware accelerator, where the hardware information may include information about each hardware unit of the hardware accelerator and topological structure information of the hardware units. Exemplarily, the information about each hardware unit includes characteristic information of each hardware unit. For example, when the hardware unit is a storage unit, the characteristics of the storage unit include storage granularity; when the hardware unit is a computing unit, the characteristics of the computing unit include computing batch size. For example, the topological structure information of the hardware unit includes communication connection relationships between the hardware units, such as the communication connection relationships between the storage units and the communication connection relationship between the storage unit and the computing unit.

[0177] Optionally, the characteristics of the storage unit also include a read speed.

[0178] Optionally, the memory cell includes a programmable memory cell and / or a non-programmable memory cell.

[0179] The hardware information is abstractly expressed as a graph structure, which includes multiple nodes and connecting edges representing the relationship between nodes. The nodes represent hardware units related to tensor calculations (such as storage units and computing units), and the connecting edges represent the communication relationship between the various hardware units.

[0180] Optionally, the graph structure is a tree topology structure, that is, the hardware information of the hardware accelerator is abstractly expressed as a tree topology structure, which includes a root node, branch nodes and leaf nodes. The root node and branch nodes represent hardware units for storage, and the leaf nodes represent hardware units for computing.

[0181] In one example, as shown in FIG3 , the tree topology consists of N layers of storage nodes (root nodes and branch nodes) and one layer of calculator nodes. The nodes are connected by edges.

[0182] Each node in the tree topology has information about the corresponding hardware characteristics. For example, a compute node has information about the computational batch size, and a storage node has information about the storage granularity. Each compute node has its own input storage and output storage.

[0183] In another example, the connecting edge is a unidirectional edge, and the unidirectional edge includes a first type of edge and / or a second type of edge and / or a third type of edge; wherein the first type of unidirectional edge represents that the node to which the unidirectional edge points (in the direction of the arrow) has the same block strategy as the node at the other end of the unidirectional edge; the second type of unidirectional edge represents that the node to which the unidirectional edge points (in the direction of the arrow) does not need to determine the block strategy; the third type of edge represents the communication connection between the hardware units represented by the nodes connected by the unidirectional edge. For example, the third type of unidirectional edge is an ordinary unidirectional edge, that is, there is no label on the unidirectional edge, the first type of unidirectional edge is Cost = 0, that is, Cost = 0 is labeled on the unidirectional edge, representing that the unidirectional edge is a second type of unidirectional edge, the second type of unidirectional edge is Bypass, that is, Bypass is labeled on the unidirectional edge, representing that the unidirectional edge is a third type of unidirectional edge.

[0184] Through this step, the hardware information of the hardware accelerator is expressed as a hierarchical topology description, and the concurrency of computing and reading and writing is converted into the relationship between layers to participate in the analysis of block strategy generation.

[0185] It should be noted that, in the embodiment of the present application, the hardware information of the hardware acceleration is simply abstractly expressed as a tree topology structure, which does not mean that the topology structure of each hardware unit in the hardware accelerator is actually a tree topology structure.

[0186] In step S403, based on the operation type and the hardware label, a mapping relationship between multiple nodes in a topological structure corresponding to multiple tensor operations and multiple hardware units is established. The topological structure is a tree structure including a root node, branch nodes, and leaf nodes.

[0187] The tensor calculation operations corresponding to the operation types are mapped to the corresponding nodes of the tree topology structure according to their corresponding hardware labels.

[0188] In step S404, a blocking strategy for tensor calculation dimensions corresponding to tensor operations corresponding to the nodes is determined for multiple nodes in a direction from the root node to the leaf nodes.

[0189] Optionally, the order of the nodes can be determined based on the hierarchical relationship, and the block strategy of the tensor calculation dimension corresponding to the tensor operation corresponding to the node can be determined for each node in the order from high to low levels (i.e., from the root node to the leaf node). For example, the N-1 layer (the layer closest to the root node) branch node first starts to determine the block strategy of the tensor calculation dimension corresponding to the tensor operation corresponding to the node, followed by the N-2 layer branch node to determine the block strategy of the tensor calculation dimension corresponding to the tensor operation corresponding to the node, and so on, and finally the block strategy of the tensor calculation dimension corresponding to the tensor operation corresponding to the node is determined for the leaf node.

[0190] That is, based on the tree topology, a block strategy is progressively prepared at each node in a specific order. For example, based on the tree topology, nodes are traversed layer by layer from top to bottom (from root to leaf nodes). That is, based on the hierarchical relationship determined by the tree topology, block strategies are determined for the tensor computation dimensions corresponding to the nodes in each layer in a top-down (from root to leaf) order. As shown in Figure 4, a block strategy is first determined for the nodes in layer N (i.e., the tensor computation dimensions corresponding to the nodes), then for the nodes in layer N-1…, then for the nodes in layer 2, then for the nodes in layer 1, and finally for the computation nodes in the computation layer. Each layer is traversed in an arbitrary order. When nodes of different types appear in the same layer, the priority of node analysis is determined based on the priority information of the connecting edges in the tree topology. That is, the connecting edges also have priority attributes. A block strategy generation method is called at each non-root node to generate block strategies corresponding to all tensor computation axes. If there are multiple blocking strategies for the same tensor computation axis, the blocking strategy that satisfies all conditions is selected. If you do not need to output the blocking strategy of a certain node, you can skip calling the blocking strategy generation method for that node.

[0191] In this step, the blocking strategy of each node is analyzed in topological order, so that different hardware units called by the computation graph composed of complex fusion operators can be fully analyzed.

[0192] In step S405 , a blocking strategy for tensor calculation is obtained based on the blocking strategy for the tensor calculation dimension corresponding to the tensor operation corresponding to each node in the plurality of nodes.

[0193] According to the blocking strategy of the tensor calculation dimension corresponding to the tensor operation corresponding to each node, the blocking strategy of the tensor calculation is obtained, and the tensor data is blocked into multiple tensor blocks according to the blocking strategy of the tensor calculation, and then the blocked tensor blocks are calculated respectively, and finally the result of the tensor calculation is obtained, thereby improving the operation speed of the entire tensor calculation.

[0194] The following takes the Ascend 910 as an example of a hardware accelerator to describe in detail the implementation of the method for generating a block strategy for tensor calculations provided in an embodiment of the present application in practical applications.

[0195] First, taking the Ascend 910 as the hardware accelerator and the computation graph formed by the fusion of multiple vectorized computing operators as an example, the specific implementation of the method for generating the block strategy for tensor computing provided in the embodiment of the present application is introduced.

[0196] Get the loop nested expression of tensor calculation, that is, the description of each operation of tensor calculation in the calculation graph of multiple vectorized calculation operators, including the type of operation and corresponding information. The obtained loop nested expression of tensor calculation is as follows:

[0197]

[0198]

[0199]

[0200] Obtain the relationship between tensor operations and hardware of the Ascend 910, as shown in Figure 5.

[0201] Build a tree-like topology structure after abstracting the hardware information of the Ascend 910.

[0202] As shown in Figure 6, the tree-like topology corresponding to the Ascend 910 includes three levels of storage layers. The root node of the third-level storage layer is the primary storage node with a storage size of 32GB.

[0203] The secondary storage layer is a branch node with 32 Tier 2 storage UB nodes. Each UB storage node has 256KB of storage space and a 32B data read granularity. The unidirectional edge from the primary storage node to the Tier 2 storage UB node has the Bypass attribute, meaning that data can be read directly from the Tier 3 storage layer to the Tier 1 storage layer, skipping the Tier 2 storage layer. The unidirectional edge from the Tier 2 storage UB node to the primary storage node is a standard unidirectional edge.

[0204] The primary storage layer is also a branch node. It includes multiple groups of storage nodes, each consisting of a Level 1 storage UB node and a Level 1 storage L1 node. Each group of nodes is connected to a Level 2 storage UB node in the Level 2 storage layer. The Level 1 storage UB node has a storage capacity of 256KB and a node characteristic of 32B data read granularity. The Level 1 storage L1 node has a storage capacity of 1MB and a node characteristic of 32B data read granularity. Unidirectional edges from Level 2 storage UB nodes to Level 1 storage UB nodes and from Level 1 storage UB nodes to Level 2 storage UB nodes have a Cost = 0 attribute, meaning that Level 2 storage UB nodes and Level 1 storage UB nodes are completely equivalent, sharing the same attributes and partitioning strategies. Unidirectional edges from Level 2 storage UB nodes to Level 1 storage L1 nodes and from Level 1 storage L1 nodes to Level 2 storage UB nodes are standard unidirectional edges.

[0205] The computing layer nodes are leaf-layer nodes, including multiple computing nodes, such as vector and cube nodes. A vector node is connected to a level-1 storage node called UB. Its computational batch size is 128 Bytes. Its input storage is UB, with a storage size of 256 KB, equivalent to a level-1 storage node called UB. Its output storage is UB, with a size of 256 KB, equivalent to a level-1 storage node called UB. A cube node is connected to a level-1 storage node called L1. Its computational batch size is 16*16*16. It has two input storages, L0A and L0B, each with a storage size of 64 KB. Its output storage is L0C, with a storage size of 256 KB, and can skip the level-1 storage node called L1.

[0206] Extract information about tensor operations from loop nested expressions, including corresponding axes, axis lengths, data types, and corresponding hardware tags.

[0207] The first operation in the nested loop expression reads data from the main storage into the underlying memory (UB). The corresponding dimensions are ax0 and ax1, with lengths of 6144 and 1024, respectively, and the data type is float32. The Ascend 910's tensor operation and hardware relationship table shows that the corresponding storage hardware is UB. This information is stored in the axis information.

[0208] The second operation in the nested loop expression reads data from the main storage into the UB. The corresponding dimensions are ax0 and ax1, with lengths of 6144 and 1024, respectively, and the data type is float32. The Ascend 910's tensor operation and hardware relationship table shows that the corresponding storage hardware is UB. However, the same information already exists in the axis information and does not need to be stored again.

[0209] The third operation in the nested loop expression is vector subtraction. The corresponding dimensions are ax0 and ax1, with lengths of 6144 and 1024, respectively, and the data type is float32. The table of tensor operations and hardware relationships for the Ascend 910 shows that the corresponding storage hardware and computing hardware are VECTOR. This information is stored in the axis information.

[0210] The remaining operations in the nested loop expression are all reads from the main storage to UB operations or vector operations. The corresponding information is the same and does not need to be stored again.

[0211] The obtained tensor calculation axis information can be seen in Figure 7. The length of the tensor calculation axis means the number of elements on the calculation axis, and the data type means the data type of the element. For example, the length of the tensor calculation axis ax0 is 6144, which means that the axis has 6144 elements of data type float32.

[0212] Tensor operations are mapped into a tree topology through hardware labels.

[0213] The first operation in the loop nested expression is to read data from the main storage to UB. The storage label corresponding to the tensor calculation dimension is UB, so it is mapped to the UB storage node in the tree topology structure.

[0214] The second operation in the loop nested expression is to read data from the main storage to UB. The storage label corresponding to the tensor calculation dimension is UB, so it is mapped to the UB storage node in the tree topology structure.

[0215] The third operation in the nested loop expression is vector subtraction. The storage label of the corresponding tensor calculation dimension is UB, so it is mapped to the UB storage node in the tree topology structure.

[0216] The storage labels for the remaining operations in the nested loop expression are all UB, so they are mapped to the UB storage nodes in the tree topology. See Figure 8 for a diagram of the tree topology after tensor operation mapping.

[0217] Perform reuse analysis on tensor operations in each node.

[0218] For example, dependency analysis is performed on the input and output of all tensor operations on each storage node (or other methods are used to obtain the maximum number of tensor operations that exist simultaneously in each storage node):

[0219] Primary storage node: No tensor operation mapping.

[0220] The UB nodes in level 2 storage have the following tensor operations obtained from the loop nesting expression:

[0221] Operation 1: read data from primary storage to UB;

[0222] Operation 2: read data from primary storage to UB;

[0223] Operation 3: Subtract the data read from operation 2 from the data read from operation 1, and store the result in UB;

[0224] Operation 4, read data from the main storage to UB;

[0225] Operation 5: Multiply the result of operation 3 by the data read in operation 4, and store the result in UB;

[0226] Operation 6, read data from the main storage to UB;

[0227] Operation 7: multiply the result of operation 5 by the data read in operation 6, and store the result in UB;

[0228] Operation 8, read data from the main storage to UB;

[0229] Operation 9: Add the result of operation 7 to the data read in operation 8, and store the result in UB;

[0230] Operation 10 converts the data type of the result of operation 9 and stores the result in UB.

[0231] Operation 1 analysis: Since operation 1 is the first operation, operation 1 is retained.

[0232] Analysis of Operation 2: Because the result of Operation 1 still needs to be retained for use by Operation 3, the space occupied by Operation 2 cannot be reused by Operation 1, so Operation 2 is retained.

[0233] Analysis of Operation 3: Because the result of Operation 1 is no longer used, and the space occupied by the result of Operation 1 is the same as the space occupied by the input of Operation 3, the output of Operation 3 can reuse the space occupied by the result of Operation 1, and Operation 1 is removed.

[0234] Analysis of Operation 4: Because the result of Operation 2 is no longer used and the space occupied by Operation 4 is the same as that of Operation 2, Operation 4 can reuse the space occupied by Operation 2 and Operation 2 can be removed.

[0235] Analysis of Operation 5: Because the result of Operation 3 is no longer used, and the space occupied by the result of Operation 3 is the same as the space occupied by the input of Operation 5, the output of Operation 5 can reuse the space occupied by the result of Operation 3, and Operation 3 is removed.

[0236] Analysis of Operation 6: Because the result of Operation 4 is no longer used and the space occupied by Operation 6 is the same as that of Operation 4, Operation 6 can reuse the space occupied by Operation 4 and Operation 4 can be removed.

[0237] Analysis of Operation 7: Because the result of Operation 5 is no longer used, and the space occupied by the result of Operation 5 is the same as the space occupied by the input of Operation 7, the output of Operation 7 can reuse the space occupied by the result of Operation 5, and Operation 5 is removed.

[0238] Analysis of Operation 8: Because the result of Operation 6 is no longer used and the space occupied by Operation 8 is the same as that of Operation 6, Operation 8 can reuse the space occupied by Operation 6 and Operation 6 can be removed.

[0239] Analysis of Operation 9: Because the result of Operation 7 is no longer used, and the space occupied by the result of Operation 7 is the same as the space occupied by the input of Operation 9, the output of Operation 9 can reuse the space occupied by the result of Operation 7, and Operation 7 is removed.

[0240] Analysis of operation 10: Because the result of operation 8 is no longer used, and the space occupied by the result of operation 8 is the same as the space occupied by the input of operation 10, the output of operation 10 can reuse the space occupied by the result of operation 8, and operation 8 is removed.

[0241] Finally, only operations 9 and 10 are retained in the level 2 storage UB node.

[0242] Tier 1 storage UB node: Equivalent to Tier 2 storage UB node.

[0243] Level 1 storage L1 node: no tensor operation mapping.

[0244] Input storage and output storage of the VECTOR calculator: equivalent to the level 1 storage UB node.

[0245] CUBE calculator input storage and output storage: no tensor operation mapping.

[0246] The tree topology diagram after tensor operation mapping after reuse analysis is shown in Figure 9.

[0247] A blocking strategy for generating tensor calculation axes corresponding to each node in the order of the tree topology.

[0248] For the primary storage layer, the root node represents the entire storage space of the hardware accelerator and does not need to be divided into blocks.

[0249] For the secondary storage layer, the secondary storage layer includes multiple level 2 storage UB nodes, and the available storage size is generated according to the tensor operation mapping of the tree topology storage nodes: there are only two operations (operation 9 and operation 10) in the level 2 storage UB node, and their corresponding axes are ax0 and ax1, so ax0 and ax1 can use half of the UB storage size, that is, 256KB / 2=128KB (see Figure 10).

[0250] Adjust the available storage size according to the sibling node information of the tree topology structure: the current node in the tree topology structure has 32 sibling nodes of the same type, and the tensor operation data size is 6144*1024, so 32 sibling nodes can be called, so the available storage size of 128KB is divided by 32 to obtain the new available storage size of 128KB / 32=4KB (see Figure 11).

[0251] A block strategy is generated based on the characteristics of the nodes in the tree topology: the current node is a Level 2 storage UB node, and the node characteristic is a data read granularity of 32B. Axis ax0 has a higher priority than axis ax1, so the block strategy is generated for axis ax0 first. Axis ax0 is selected to meet the UB data read granularity, so the block strategy for ax0 is set to 32B / 4B = 8 (32B is the node's data read granularity, and 4B is the size of the float32 data type). Axis ax1 uses the maximum block strategy of 4*1024B / 4B / 8 = 128 (see Figure 12).

[0252] For the level 1 storage UB of the primary storage layer: the attribute of the edge between the level 1 storage UB and the level 2 storage UB is cost = 0, so the two are equivalent and use the same blocking strategy.

[0253] For the level 1 storage L1 of the first-level storage layer: there is no mapping of tensor operations, so there is no corresponding axis, so it does not need to be blocked.

[0254] For the computation layer, VECTOR calculator nodes have tensor operation mappings, while CUBE calculator nodes do not. Therefore, you only need to analyze the VECTOR calculator nodes and determine the block strategy for the tensor computation axis corresponding to the VECTOR calculator nodes.

[0255] The available storage size is generated based on the tensor operation mapping of the tree topology storage nodes: the input storage and output storage of VECTOR are equivalent to the level 1 storage UB (cost = 0), so they have the same available storage size.

[0256] Adjust the available storage size based on the sibling node information of the tree topology: the current node has no sibling nodes, so the available storage size is not adjusted

[0257] The optimization strategy is described based on the characteristics of the nodes in the tree topology: the current node is a vector node, and the node characteristic is a computation batch size of 128 bytes. Based on the default priority (axis ax1 has a higher priority than ax0), axis ax1 is selected to meet the vector computation batch size. Therefore, the blocking strategy for ax1 must be a multiple of 128 bytes / 4 bytes = 32. The current blocking strategy for axis ax1 is 256, which is a multiple of 32 and meets the computation batch size, so no adjustment is required. Therefore, the final blocking strategy for tensor calculations is 8 blocks for axis ax0 and 128 blocks for axis ax1 (see Figure 13).

[0258] It can be understood that the blocking strategy of axis ax0 is 8, which means that every 8 elements in axis ax0 are blocked as a block. For example, if the length of axis ax0 is 6144 and the blocking strategy is 8, the number of blocks in axis ax0 is 6144 / 8=768, and the size of each block is 8. The blocking strategy of axis ax1 is 128, which means that every 128 elements in axis ax1 are blocked as a block. For example, if the length of axis ax1 is 1024 and the blocking strategy is 128, the number of blocks in axis ax1 is 1024 / 128=8, and the size of each block is 128.

[0259] Next, taking the Ascend 910 as the hardware accelerator and the computation graph formed by the fusion of vectorization operators and matrix multiplication operators as an example, the specific implementation of the method for generating the block strategy for tensor calculation provided in the embodiment of the present application is introduced.

[0260] Get the loop nested expression of tensor calculation, that is, the description of each operation of tensor calculation in the calculation graph of the fusion of vectorization operator and matrix multiplication operator, including the type of operation and corresponding information. The obtained loop nested expression of tensor calculation is as follows:

[0261]

[0262]

[0263] For a table showing the relationship between tensor operations and hardware on the Ascend 910, see Figure 5.

[0264] The tree-like topology structure after abstracting the hardware information of the Ascend 910 can be found in the description above and will not be repeated here for the sake of brevity.

[0265] Extract information about tensor operations from loop nested expressions, including corresponding axes, axis lengths, data types, and corresponding hardware tags.

[0266] The first operation in the nested loop expression reads data from the main storage to L1 and then to L0A. The corresponding dimensions are ax1 and ax2, with lengths of 512 and 256, respectively, and the data type is float16. The table showing the relationship between tensor operations and hardware in the Ascend 910 shows that the corresponding storage hardware is L1 and L0A. This information is stored in the axis information.

[0267] The second operation in the nested loop expression reads data from the main storage into L1 and L0B. The corresponding dimensions are ax0 and ax2, with lengths of 64 and 256, respectively, and the data type is float16. The table showing the relationship between tensor operations and hardware in the Ascend 910 shows that the corresponding storage hardware is L1 and L0B. This information is then stored in the axis information.

[0268] The third operation in the nested loop expression performs matrix multiplication on the data in L0A and L0B, then reads the result into L0C and then into UB. The corresponding dimensions are ax0, ax1, and ax2 (the dimensions corresponding to reading data into UB are ax0 and ax1), with lengths of 64, 512, and 256, respectively, and the data type is float16. The table of tensor operations and hardware relationships for the Ascend 910 shows that the corresponding storage hardware is L0C and UB, and the computing hardware is CUBE. This information is stored in the axis information.

[0269] The fourth operation in the nested loop expression reads data from the main storage into the underlying memory (UB). The corresponding dimensions are ax0 and ax1, with lengths of 64 and 512, respectively, and the data type is float16. The table showing the relationship between tensor operations and hardware in the Ascend 910 shows that the corresponding storage hardware is UB. This information is then stored in the axis information.

[0270] The fifth operation in the nested loop expression performs vector multiplication on the data in UB. The corresponding dimensions are ax0 and ax1, with lengths of 64 and 512, respectively, and the data type is float16. The table of tensor operations and hardware relationships for the Ascend 910 shows that the corresponding computing hardware is VECTOR. This information is stored in the axis information.

[0271] The obtained tensor calculation axis information can be seen in Figure 14.

[0272] Tensor operations are mapped into a tree topology through hardware labels.

[0273] The storage labels of the dimension (ax1, ax2) corresponding to the first operation in the loop nested expression are UB, L1, L0A, L0B, and L0C, so they are mapped to the corresponding storage nodes in the tree topology structure.

[0274] The storage labels of the dimension (ax0, ax2) corresponding to the second operation in the loop nested expression are UB, L1, L0A, L0B, and L0C, so they are mapped to the corresponding storage nodes in the tree topology structure.

[0275] The storage labels of the dimensions (ax0, ax1, ax2) corresponding to the third operation in the loop nested expression are UB, L1, L0A, L0B, and L0C, so they are mapped to the corresponding storage nodes in the tree topology structure.

[0276] The storage labels of the dimension (ax0, ax1) corresponding to the fourth operation in the loop nested expression are UB, L1, L0A, L0B, and L0C, so they are mapped to the corresponding storage nodes in the tree topology structure.

[0277] The storage labels for the fifth operation in the nested loop expression, corresponding to the dimension (ax0, ax1), are UB, L1, L0A, L0B, and L0C, so they are mapped to the corresponding storage nodes in the tree topology. See Figure 15 for a diagram of the tree topology after tensor operation mapping.

[0278] Perform reuse analysis on tensor operations in each node.

[0279] For example, dependency analysis is performed on the input and output of all tensor operations on each storage node (or other methods are used to obtain the maximum number of tensor operations that exist simultaneously in each storage node):

[0280] Primary storage node: No tensor operation mapping.

[0281] The UB nodes in level 2 storage have the following tensor operations obtained from the loop nesting expression:

[0282] Operation 1: Read data from main storage to L1 and then to L0A;

[0283] Operation 2: Read data from main storage to L1 and then to L0B;

[0284] Operation 3: Perform matrix multiplication on the data read from operation 1 and the data read from operation 2, and read the result to L0C and then to UB;

[0285] Operation 4: Read data from primary storage to UB;

[0286] Operation 5: Multiply the result of Operation 3 by the data read in Operation 4.

[0287] Operation 1 analysis: Operation 1 does not involve the Level 2 storage UB node. Remove Operation 1.

[0288] Operation 2 analysis: Operation 2 does not involve the Level 2 storage UB node. Operation 2 is removed.

[0289] Operation 3 analysis: Because operation 3 is the first operation of the level 2 storage UB node, operation 3 is retained.

[0290] Analysis of Operation 4: Because the result of Operation 3 still needs to be retained for use by Operation 5, the space occupied by Operation 4 cannot be reused by Operation 3, so Operation 4 is retained.

[0291] Analysis of Operation 5: Because the result of Operation 3 is no longer used, and the space occupied by the result of Operation 3 is the same as the space occupied by the input of Operation 5, the output of Operation 5 can reuse the space occupied by Operation 3, and Operation 3 is removed.

[0292] Tier 1 storage UB node: Equivalent to Tier 2 storage UB node.

[0293] Level 1 storage L1 node:

[0294] Operation 1: Read data from main storage to L1 and then to L0A;

[0295] Operation 2: Read data from main storage to L1 and then to L0B;

[0296] Operation 3: Perform matrix multiplication on the data read from operation 1 and the data read from operation 2, and read the result to L0C and then to UB;

[0297] Operation 4: Read data from primary storage to UB;

[0298] Operation 5: Multiply the result of Operation 3 by the data read in Operation 4.

[0299] Operation 1 analysis: Because operation 1 is the first operation of the level 1 storage L1 node, operation 1 is retained.

[0300] Analysis of Operation 2: Because the result of Operation 1 still needs to be retained for use by Operation 3, the space occupied by Operation 2 cannot be reused by Operation 1, so Operation 2 is retained.

[0301] Operation 3 analysis: Operation 3 does not involve the Level 1 storage L1 node. Operation 3 is removed.

[0302] Operation 4 analysis: Operation 4 does not involve the Level 1 storage L1 node. Operation 4 is removed.

[0303] Operation 5 analysis: Operation 5 does not involve the Level 1 storage L1 node. Operation 5 is removed.

[0304] Input storage and output storage of the VECTOR calculator: equivalent to the level 1 storage UB node.

[0305] The input of the CUBE calculator is stored in L0A:

[0306] Operation 1: Read data from main storage to L1 and then to L0A;

[0307] Operation 2: Read data from main storage to L1 and then to L0B;

[0308] Operation 3: Perform matrix multiplication on the data read from operation 1 and the data read from operation 2, and read the result to L0C and then to UB;

[0309] Operation 4: Read data from primary storage to UB;

[0310] Operation 5: Multiply the result of Operation 3 by the data read in Operation 4.

[0311] Operation 1 analysis: Because operation 1 is the first operation of the input storage L0A of the CUBE calculator, operation 1 is retained.

[0312] Operation 2 analysis: Operation 2 does not involve the input storage L0A of the CUBE calculator, so operation 2 is removed.

[0313] Operation 3 analysis: Operation 3 does not involve the input storage L0A of the CUBE calculator, so operation 3 is removed.

[0314] Operation 4 analysis: Operation 4 does not involve the input storage L0A of the CUBE calculator, so operation 4 is removed.

[0315] Operation 5 analysis: Operation 5 does not involve the input storage L0A of the CUBE calculator, so operation 5 is removed.

[0316] The input of the CUBE calculator is stored in L0B:

[0317] Operation 1: Read data from main storage to L1 and then to L0A;

[0318] Operation 2: Read data from main storage to L1 and then to L0B;

[0319] Operation 3: Perform matrix multiplication on the data read from operation 1 and the data read from operation 2, and read the result to L0C and then to UB;

[0320] Operation 4: Read data from primary storage to UB;

[0321] Operation 5: Multiply the result of Operation 3 by the data read in Operation 4.

[0322] Operation 1 analysis: Operation 1 does not involve the input storage L0B of the CUBE calculator, so operation 1 is removed.

[0323] Operation 2 analysis: Because operation 2 is the first operation of the input storage L0B of the CUBE calculator, operation 2 is retained.

[0324] Operation 3 analysis: Operation 3 does not involve the input storage L0B of the CUBE calculator, so operation 3 is removed.

[0325] Operation 4 analysis: Operation 4 does not involve the input storage L0B of the CUBE calculator, so operation 4 is removed.

[0326] Operation 5 analysis: Operation 5 does not involve the input storage L0B of the CUBE calculator, so operation 5 is removed.

[0327] The output of the CUBE calculator is stored in L0C:

[0328] Operation 1: Read data from main storage to L1 and then to L0A;

[0329] Operation 2: Read data from main storage to L1 and then to L0B;

[0330] Operation 3: Perform matrix multiplication on the data read from operation 1 and the data read from operation 2, and read the result to L0C and then to UB;

[0331] Operation 4: Read data from primary storage to UB;

[0332] Operation 5: Multiply the result of Operation 3 by the data read in Operation 4.

[0333] Operation 1 analysis: Operation 1 does not involve the output storage L0C of the CUBE calculator, so remove Operation 1.

[0334] Analysis of Operation 2: Operation 1 does not involve the output storage L0C of the CUBE calculator, so Operation 1 is removed.

[0335] Analysis of Operation 3: Since Operation 3 is the first operation of the CUBE calculator's output storage L0C, Operation 3 is retained.

[0336] Operation 4 analysis: Operation 4 does not involve the output storage L0C of the CUBE calculator, remove operation 4

[0337] Analysis of Operation 5: Operation 5 does not involve the output storage L0C of the CUBE calculator, so Operation 5 is removed.

[0338] The tree topology diagram after tensor operation mapping after reuse analysis is shown in Figure 16.

[0339] A blocking strategy for generating tensor calculation axes corresponding to each node in the order of the tree topology.

[0340] For the primary storage layer, the root node represents the entire storage space of the hardware accelerator and does not need to be divided into blocks.

[0341] The secondary storage layer includes multiple Level 2 storage UB nodes. The available storage size is generated based on the tensor operation mapping of the tree-like topology storage nodes. There are two operations (operation 4 and operation 5) in the Level 2 storage UB node, and the available storage size for each operation is 256 / 2 = 128KB. Operations 4 and 5 require the tensor computation axes to be partitioned in the storage UB node, ax0 and ax1, respectively. Therefore, the available storage size for axes ax0 and ax1 in the storage UB node is 128KB (see Figure 17).

[0342] Adjust the available storage size according to the sibling node information of the tree topology structure: the current node in the tree topology structure has 32 sibling nodes of the same type, and the tensor operation data size is 64*512, so 32 sibling nodes can be called, so the available storage size is divided by 32 to obtain the new available storage size of 128KB / 32=4KB (see Figure 18).

[0343] Based on the characteristics of the nodes in the tree topology, the optimization strategy is described: the current node is a Level 2 storage UB node, and the node characteristic is a data read granularity of 32B. Based on the default priority (axis ax0 has a higher priority than axis ax1), axis ax0 is selected to meet the UB data read granularity, so the block strategy for ax0 is set to 32B / 2B = 16 (32B is the node's data read granularity, and 2B is the size of the float16 data type). Axis ax1 adopts the maximum block strategy of 4*1024B / 2B / 16 = 128 (see Figure 19).

[0344] For the level 1 storage UB of the primary storage layer: the attribute of the edge between the level 1 storage UB and the level 2 storage UB is cost = 0, so the two are equivalent and use the same blocking strategy.

[0345] For the first-level storage layer L1:

[0346] The available storage size is generated based on the tensor operation mapping of the tree topology storage nodes: There are two operations (operation 1 and operation 2) in the level 1 storage L1 node, so it can use half of the L1 storage (1MB / 2=512KB).

[0347] The axes that need to be blocked in L1 for operation 1 are ax0, ax1, ax2, so the available storage size of ax0, ax1, ax2 in L1 is 512KB (see Figure 20).

[0348] Adjust the available storage size based on the sibling node information of the tree topology: The current node has no sibling nodes, so the storage size is not adjusted.

[0349] Based on the characteristics of the nodes in the tree topology, the optimization strategy is described: the current node is a level 1 storage L1 node, and the node characteristic is a data read granularity of 32B. Based on the default priority, the ax0 axis is selected to meet the L1 data read granularity, so the block strategy of ax0 is set to 32B / 2B=16 (the fp16 data size is 2B). The ax1 axis adopts the maximum block strategy of 512 (reaching the maximum axis length), and the ax2 axis adopts the maximum block strategy of 64 (512*1024B / 2B / 16 / 512=32) (see Figure 21).

[0350] For the computing layer, both the VECTOR calculator node and the CUBE calculator node have tensor operation mappings, so both need to be analyzed to determine the blocking strategy for the tensor calculation axes corresponding to the VECTOR calculator node and the CUBE calculator node.

[0351] For VECTOR calculator nodes:

[0352] The available storage size is generated based on the tensor operation mapping of the tree topology storage nodes: the input storage and output storage of the VECTOR calculator node are equivalent to the level 1 storage UB (cost = 0), so the same blocking strategy is used.

[0353] Adjust the available storage size based on the sibling node information of the tree topology: The current node has no sibling nodes, so the storage size is not adjusted.

[0354] Describe the optimization strategy based on the characteristics of the nodes in the tree topology: The current node is a vector calculator node with a computation granularity of 128 bytes. Among the axes corresponding to the UB tag, ax1 is selected based on the default priority to meet the computation batch size of the vector calculator node. Therefore, the block strategy for ax1 must be a multiple of 128 bytes / 4 bytes (32). The block strategy for ax1 in UB is 128, a multiple of 32, which meets the computation granularity, so the block strategy is not adjusted.

[0355] For CUBE calculator nodes:

[0356] Generate the available storage size based on the tensor operation mapping of the tree topology storage nodes:

[0357] Input storage L0A: There is only one operation (operation 1) in input storage L0A, so it can use the entire storage of L0A (64KB).

[0358] Input storage L0B: There is only one operation (operation 2) in the input storage L0B, so it can use the entire storage of L0B (64KB).

[0359] Output Storage L0C: There is only one operation (operation 3) in the Output Storage L0C, so it can use the entire storage of the L0C (256KB) (see Figure 22).

[0360] Adjust the available storage size based on the sibling node information of the tree topology: The current node has no sibling nodes, so the storage size is not adjusted.

[0361] Based on the characteristics of the nodes in the tree topology, the optimization strategy is described: the current node is a CUBE calculator node, and the node characteristic is a computation batch size of 16:16:16. A block strategy is applied to the three axes ax0, ax1, and ax2 involved in input storage L0A and L0B, using multiples of 16 and the same proportions. By allocating the available storage capacity in the same proportions, a block strategy of 2√(64*1024B / 2B)=181 is obtained. The largest multiple of 16 smaller than 181 is then found: 176. Therefore, the block strategy for both L0A and L0B is 176.

[0362] The output storage L0C has no block strategy, and a maximum block strategy is generated for it: the block strategies of ax0, ax1, and ax2 in L0C are 64 (reaching the maximum axis length), 512 (reaching the maximum axis length), and 4 (=256*1024B / 2B / 64 / 512, the available storage size is divided by the data type and the used block strategy to obtain a larger block strategy that does not exceed the storage size) (see Figure 23).

[0363] The final block strategy is shown in Figure 24.

[0364] The block strategy generation method for tensor calculation provided by the embodiment of the present application takes into account the storage size of each level of storage unit and the batch size of the storage unit and the computing unit, ensuring the legitimacy of the block strategy. For example, in the reduction operation (Reduce), the calculations between the elements of the input tensor are interdependent, and it is necessary to consider the batch size of each hardware unit and the available space of the storage unit at the same time. Among the 159 comparison examples of fusion operators containing reduction selected by MindSpore in actual combat, the block strategy generation method for tensor calculation provided by the embodiment of the present application successfully generated block strategies for all examples, and some performance was higher than the examples successfully processed by the prior art (see Figure 25).

[0365] The block strategy generation method for tensor calculation provided in the embodiment of the present application progressively adapts the block strategy of each axis to the required hardware features, ensuring the quality of the block strategy and allowing the code generated based on the block strategy generation method for tensor calculation provided in the embodiment of the present application to have excellent performance. In the 40 comparison examples of fusion operators containing matrix multiplication selected by MindSpore in actual combat, the execution speed of the code generated by the block strategy generated by the block strategy generation method for tensor calculation provided in the embodiment of the present application is significantly better than the execution speed of the code generated by the prior art, and the present invention also successfully generates excellent segmentation strategies for examples that were not successfully processed by the existing calculations (see Figure 26).

[0366] Based on the same concept as the embodiment of the aforementioned tensor calculation block strategy generation method, the embodiment of the present application also provides a tensor calculation block strategy generation device 2700, which includes units or modules for implementing each step in the tensor calculation block strategy generation method shown in Figure 2a.

[0367] FIG27a is a schematic diagram of the structure of a block strategy generation device for tensor calculation provided by an embodiment of the present application. The device is applied to a computing device, as shown in FIG27a, and the block strategy generation device 2700 for tensor calculation includes at least:

[0368] An acquisition module 2701 is configured to acquire information about each tensor operation corresponding to a tensor calculation, where the information includes the corresponding tensor calculation dimension, the number of elements within the tensor calculation dimension, the data type of the elements, the operation type, and a hardware tag, where the hardware tag indicates the hardware unit on which the operation type depends.

[0369] Obtaining a topology structure corresponding to a hardware accelerator, the hardware accelerator being used to accelerate the tensor calculation; the topology structure comprising a plurality of nodes, each node representing a hardware unit related to the tensor calculation;

[0370] A mapping module 2702 is configured to map each of the tensor operations to the corresponding node based on the operation type and the hardware tag;

[0371] A blocking strategy generating module 2703 is configured to obtain a blocking strategy for the tensor computation dimension corresponding to the tensor operation corresponding to the node based on characteristic information of the hardware unit corresponding to the node, allocated storage space of the hardware unit, and a data type of the element corresponding to the tensor computation dimension corresponding to the tensor operation, wherein the allocated storage space of the hardware unit is obtained based on the storage space of the hardware unit and the number of tensor operations corresponding to the node;

[0372] Based on the block strategies of each of the tensor calculation dimensions, a block strategy for the tensor calculation is obtained.

[0373] In a possible implementation, the topological structure further includes a plurality of connection edges, where the connection edges are used to connect different nodes, and the connection edges represent the relationship between the nodes;

[0374] The blocking strategy of the tensor calculation dimension corresponding to the node is also related to the connection edge corresponding to the node.

[0375] In another possible implementation, the connecting edge is a unidirectional edge, the unidirectional edge has an attribute, and the attribute is a first attribute, a second attribute, or a third attribute;

[0376] The first attribute represents that the blocking strategy of the tensor calculation dimension corresponding to the node pointed to by the unidirectional edge is determined based on characteristic information of the hardware unit corresponding to the node, the allocated storage space of the hardware unit, and the data type of the element corresponding to the tensor calculation dimension corresponding to the tensor operation;

[0377] The second attribute represents a block strategy for the tensor calculation dimension corresponding to the node pointed to by the unidirectional edge, which is the same as the block strategy for the tensor calculation dimension corresponding to another node connected by the unidirectional edge;

[0378] The third attribute representation does not require determining a block strategy for the tensor calculation dimension corresponding to the node to which the unidirectional edge points.

[0379] In another possible implementation, it further includes:

[0380] A reuse analysis module 2704 is used to perform memory reuse analysis on the tensor operations corresponding to each of the nodes;

[0381] The updating module 2705 is used to update the number of tensor operations corresponding to each of the nodes based on the memory reuse analysis result.

[0382] In another possible implementation, the topology is a tree topology, the multiple nodes include a root node, branch nodes, and leaf nodes, the root node and branch nodes represent hardware units for storage, and the leaf nodes represent hardware units for computing;

[0383] Also includes:

[0384] An order determination module 2706 is configured to obtain an order of the plurality of nodes based on the tree topology structure;

[0385] Based on the order of the multiple nodes, the block strategy of the tensor calculation dimension corresponding to the node is determined in sequence.

[0386] In another possible implementation, the allocated storage space of the hardware unit corresponding to the node is also related to the number of available sibling nodes corresponding to the node.

[0387] In another possible implementation, the tensor calculation dimension corresponding to the node includes a first tensor calculation dimension and a second tensor calculation dimension, and the priority of the first tensor calculation dimension is higher than the priority of the second tensor calculation dimension;

[0388] The block strategy generation module 2703 is specifically used to:

[0389] Obtaining a blocking strategy for the first tensor calculation dimension based on characteristic information of the hardware unit corresponding to the node and a data type of the element corresponding to the first calculation dimension;

[0390] Based on the allocated storage space of the hardware unit and the blocking strategy of the first tensor calculation dimension, a blocking strategy of the second tensor calculation dimension is determined.

[0391] In another possible implementation, the hardware unit includes a storage unit and a computing unit, the characteristics of the storage unit include storage granularity, and the characteristics of the computing unit include computing batch size.

[0392] In another possible implementation, the obtaining module 2701 is specifically configured to:

[0393] Obtaining information of each calculation dimension corresponding to the tensor calculation based on the intermediate expression corresponding to the tensor calculation and the relationship table between the tensor calculation and the hardware accelerator;

[0394] Among them, the intermediate expression corresponding to the tensor calculation includes the number of elements corresponding to each calculation dimension corresponding to the tensor calculation, the data type of the element, and the operation type; the relationship table between the tensor calculation and the hardware accelerator includes the correspondence between the operation type and each hardware unit in the hardware accelerator.

[0395] In another possible implementation, the hardware accelerator is a heterogeneous structure.

[0396] According to the block strategy generation device 2700 for tensor calculation according to the embodiment of the present application, it can correspond to the execution of the method described in the embodiment of the present application, and the above-mentioned and other operations and / or functions of each module in the block strategy generation device 2700 for tensor calculation are respectively for realizing the corresponding processes of each method in Figure 2a. For the sake of brevity, they will not be repeated here.

[0397] Based on the same concept as the embodiment of the aforementioned tensor calculation block strategy generation method, the embodiment of the present application also provides another tensor calculation block strategy generation device 3700, which includes units or modules for implementing each step in the tensor calculation block strategy generation method shown in Figure 2b.

[0398] FIG27 b is a schematic diagram of the structure of a block strategy generation device for tensor calculation provided by an embodiment of the present application. The device is applied to a computing device, as shown in FIG27 b , and the block strategy generation device 3700 for tensor calculation includes at least:

[0399] An acquisition module 3701 is configured to acquire information of multiple tensor operations corresponding to tensor calculations, where the information of each tensor operation includes a tensor calculation dimension corresponding to the tensor operation, a data type of an element corresponding to the tensor calculation dimension, and a priority of the tensor calculation dimension.

[0400] a determination module 3702 for determining a correspondence between the plurality of tensor operations and a plurality of hardware units for performing operations on the tensor calculations;

[0401] A blocking strategy generating module 3703 is configured to obtain a blocking strategy for the tensor calculation dimension corresponding to the tensor operation based on characteristic information of the hardware unit corresponding to the tensor operation, a storage space corresponding to the tensor operation, a data type of an element corresponding to the tensor calculation dimension, and a priority of the tensor calculation dimension, wherein the characteristic information of the hardware unit indicates a storage characteristic or a calculation characteristic of the hardware unit;

[0402] Based on the block strategies of the tensor calculation dimensions corresponding to the multiple tensor operations, a block strategy for the tensor calculation is obtained.

[0403] In one possible implementation, the block strategy generation module 3703 is specifically configured to:

[0404] Obtaining a blocking strategy for the first tensor calculation dimension based on characteristic information of the hardware unit corresponding to the tensor operation and a data type of an element corresponding to the first tensor calculation dimension;

[0405] Obtaining a blocking strategy for a second tensor calculation dimension based on a storage space corresponding to the tensor operation and the blocking strategy for the first tensor calculation dimension;

[0406] Obtaining a blocking strategy for the tensor calculation dimension corresponding to the tensor operation based on the blocking strategy for the first tensor calculation dimension and the blocking strategy for the second tensor calculation dimension;

[0407] The first tensor calculation dimension and the second tensor calculation dimension are both tensor calculation dimensions corresponding to the tensor operation, and the priority of the first tensor calculation dimension is higher than the priority of the second tensor calculation dimension.

[0408] In another possible implementation, the storage space corresponding to the tensor operation is related to the number of the tensor operations corresponding to the hardware unit.

[0409] In another possible implementation, the method further includes:

[0410] A reuse analysis module 3704 is configured to perform storage reuse analysis on the plurality of tensor operations corresponding to the hardware unit to determine a tensor operation with reusable storage space among the plurality of tensor operations corresponding to the hardware unit;

[0411] The updating module 3705 is used to update the number of the tensor operations corresponding to the hardware unit based on the storage reuse analysis results of the various tensor operations.

[0412] In another possible implementation, the information of the tensor operation further includes an operation type corresponding to the tensor operation;

[0413] The determining module 3702 is specifically configured to:

[0414] Based on the correspondence between the operation types corresponding to the multiple tensor operations and the hardware units, at least one hardware unit corresponding to the tensor operation is obtained.

[0415] In another possible implementation, the multiple hardware units include a storage unit and a computing unit, the characteristics of the storage unit include storage granularity, and the characteristics of the computing unit include computing batch size.

[0416] In another possible implementation, the multiple hardware units include different types of storage units and / or different types of computing units.

[0417] According to the block strategy generation device 3700 for tensor calculation according to the embodiment of the present application, it can correspond to the execution of the method described in the embodiment of the present application, and the above-mentioned and other operations and / or functions of each module in the block strategy generation device 3700 for tensor calculation are respectively for realizing the corresponding processes of each method in Figure 2b. For the sake of brevity, they will not be repeated here.

[0418] Based on the same concept as the embodiment of the aforementioned tensor calculation block strategy generation method, the embodiment of the present application also provides another tensor calculation block strategy generation device 4700, which includes units or modules for implementing each step in the tensor calculation block strategy generation method shown in Figure 2c.

[0419] FIG27c is a schematic diagram of the structure of a block strategy generation device for tensor calculation provided by an embodiment of the present application. The device is applied to a computing device, as shown in FIG27c, and the block strategy generation device 4700 for tensor calculation includes at least:

[0420] An acquisition module 4701 is configured to acquire information about multiple tensor operations corresponding to the tensor calculation, where the information about each tensor operation includes an operation type and a hardware tag, where the hardware tag represents a hardware unit on which the operation type depends, and the multiple hardware units are used to perform the tensor calculation.

[0421] A determination module 4702 is configured to determine a topological structure corresponding to the plurality of hardware units, wherein the topological structure includes a plurality of nodes, each of which represents the hardware units related to the tensor calculation.

[0422] A mapping module 4703 is configured to establish, based on the operation type and the hardware label, a mapping relationship between the plurality of tensor operations and a plurality of nodes in a topological structure corresponding to the plurality of hardware units, wherein the topological structure is a tree structure including a root node, branch nodes, and leaf nodes;

[0423] A block strategy generating module 4704 is configured to determine, for a plurality of nodes, a block strategy for tensor calculation dimensions corresponding to the tensor operations corresponding to the nodes in a direction from the root node to the leaf nodes;

[0424] Based on the block strategy of the tensor calculation dimension corresponding to the tensor operation corresponding to each of the multiple nodes, the block strategy of the tensor calculation is obtained.

[0425] In another possible implementation, the topological structure further includes a plurality of unidirectional edges, where the unidirectional edges are used to connect different nodes, and the unidirectional edges represent the relationship between the nodes.

[0426] In another possible implementation, the plurality of unidirectional edges include first-type unidirectional edges and / or second-type unidirectional edges and / or third-type unidirectional edges;

[0427] The first type of unidirectional edge represents that the tensor calculation dimension corresponding to the tensor operation corresponding to the nodes connected by the first type of unidirectional edge is the same as the blocking strategy;

[0428] The second type of unidirectional edge representation does not require determining a blocking strategy for the tensor calculation dimension corresponding to the tensor operation corresponding to the node pointed to by the second type of unidirectional edge;

[0429] The third type of unidirectional edge represents the communication connection between the hardware units represented by the nodes connected by the third type of unidirectional edge.

[0430] In another possible implementation, the blocking strategy of the tensor calculation dimension corresponding to the tensor operation corresponding to the node is related to the allocated storage space of the hardware unit represented by the node;

[0431] The allocated storage space of the hardware unit represented by the node is related to the number of available sibling nodes of the node. The sibling nodes are nodes located at the same layer as the node and represent the same type of hardware units.

[0432] In another possible implementation, the number of available sibling nodes of the node is related to the data size corresponding to the tensor operation and the number of the sibling nodes.

[0433] In another possible implementation, the hardware units represented by the root node and the branch node are storage units, and the hardware units represented by the leaf nodes are computing units.

[0434] In another possible implementation, the plurality of hardware units include different types of storage units and / or different types of computing units.

[0435] According to the block strategy generation device 4700 for tensor calculation according to the embodiment of the present application, it can correspond to the execution of the method described in the embodiment of the present application, and the above-mentioned and other operations and / or functions of each module in a block strategy generation device 4700 for tensor calculation are respectively for realizing the corresponding processes of each method in Figure 2c. For the sake of brevity, they will not be repeated here.

[0436] An embodiment of the present application also provides a computing device, including at least one processor, a memory, and a communication interface, wherein the processor is used to execute the methods described in Figures 1a-24.

[0437] Figure 28 is a schematic diagram of the structure of the computing device provided in an embodiment of the present application.

[0438] As shown in Figure 28, the computing device 2800 includes at least one processor 2801, a memory 2802, a hardware accelerator 2803 and a communication interface 2804. Among them, the processor 2801, the memory 2802, the hardware accelerator 2803 and the communication interface 2804 are communicatively connected, and the communication connection can be achieved by wired means (such as a bus) or by wireless means. The communication interface 2804 is used to receive data sent by other devices; the memory 2802 stores computer instructions, and the processor 2801 executes the computer instructions to execute the block strategy generation method and / or classification method for tensor calculation in the aforementioned method embodiment. The hardware accelerator 2803 is used to accelerate tensor calculations.

[0439] It should be understood that in the embodiment of the present application, the processor 2801 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0440] The memory 2802 may include a read-only memory and a random access memory, and provides instructions and data to the processor 2801. The memory 2802 may also include a nonvolatile random access memory.

[0441] The memory 2802 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0442] It should be understood that the computing device 2800 according to the embodiment of the present application can execute the method shown in Figures 1a-24 in the embodiment of the present application. The detailed description of the implementation of the method is given above and will not be repeated here for the sake of brevity.

[0443] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer instructions are executed by a processor, the above-mentioned method for generating a blocking strategy for tensor calculation is implemented.

[0444] An embodiment of the present application provides a chip comprising at least one processor and an interface, wherein the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to implement the above-mentioned method for generating a block strategy for tensor calculation.

[0445] An embodiment of the present application provides a computer program or a computer program product, which includes instructions that, when executed, cause a computer to execute the above-mentioned method for generating a blocking strategy for tensor calculation.

[0446] Those skilled in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0447] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0448] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this application. It should be understood that the above description is only the specific implementation methods of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A method for generating a block strategy for tensor calculation, characterized in that: Obtain information of multiple tensor operations corresponding to the tensor calculation, where the information of each tensor operation includes a tensor calculation dimension corresponding to the tensor operation, a data type of an element corresponding to the tensor calculation dimension, and a priority of the tensor calculation dimension; Determining a correspondence between the plurality of tensor operations and a plurality of hardware units, the plurality of hardware units being used to perform operations on the tensor calculations; Obtaining a blocking strategy for the tensor calculation dimension corresponding to the tensor operation based on characteristic information of the hardware unit corresponding to the tensor operation, a storage space corresponding to the tensor operation, a data type of an element corresponding to the tensor calculation dimension, and a priority of the tensor calculation dimension, wherein the characteristic information of the hardware unit indicates a storage characteristic or a calculation characteristic of the hardware unit; Based on the block strategies of the tensor calculation dimensions corresponding to the multiple tensor operations, a block strategy for the tensor calculation is obtained.

2. The method according to claim 1, characterized in that The obtaining of a block strategy for the tensor calculation dimension corresponding to the tensor operation based on characteristic information of the hardware unit corresponding to the tensor operation, a storage space corresponding to the tensor operation, a data type of an element corresponding to the tensor calculation dimension corresponding to the tensor operation, and a priority of the tensor calculation dimension includes: Obtaining a blocking strategy for the first tensor calculation dimension based on characteristic information of the hardware unit corresponding to the tensor operation and a data type of an element corresponding to the first tensor calculation dimension; Obtaining a blocking strategy for a second tensor calculation dimension based on a storage space corresponding to the tensor operation and the blocking strategy for the first tensor calculation dimension; Obtaining a blocking strategy for the tensor calculation dimension corresponding to the tensor operation based on the blocking strategy for the first tensor calculation dimension and the blocking strategy for the second tensor calculation dimension; The first tensor calculation dimension and the second tensor calculation dimension are both tensor calculation dimensions corresponding to the tensor operation, and the priority of the first tensor calculation dimension is higher than the priority of the second tensor calculation dimension.

3. The method according to claim 1 or 2, characterized in that The storage space corresponding to the tensor operation is related to the number of the tensor operations corresponding to the hardware unit.

4. The method according to claim 3, characterized in that Also includes: Performing storage reuse analysis on the plurality of tensor operations corresponding to the hardware unit to determine a tensor operation with reusable storage space among the plurality of tensor operations corresponding to the hardware unit; Based on the storage reuse analysis results of the respective tensor operations, the number of the tensor operations corresponding to the hardware unit is updated.

5. The method according to any one of claims 1 to 4, characterized in that The information of the tensor operation also includes the operation type corresponding to the tensor operation; Determining the correspondence between the plurality of tensor operations and the plurality of hardware units includes: Based on the correspondence between the operation type of the tensor operation and the hardware unit, at least one hardware unit corresponding to the tensor operation is obtained.

6. The method according to any one of claims 1 to 5, characterized in that The multiple hardware units include a storage unit and a computing unit. The characteristics of the storage unit include storage granularity, and the characteristics of the computing unit include computing batch size.

7. The method according to any one of claims 1 to 6, characterized in that The plurality of hardware units include different types of storage units and / or different types of computing units.

8. A method for generating a block strategy for tensor calculation, characterized in that: Obtaining information about a plurality of tensor operations corresponding to the tensor calculation, where the information about each tensor operation includes an operation type and a hardware tag, where the hardware tag represents a hardware unit on which the operation type depends; Determine a topological structure corresponding to the plurality of hardware units, the topological structure comprising a plurality of nodes, the nodes representing the hardware units related to the tensor calculation; Based on the operation type and the hardware label, establishing a mapping relationship between the multiple tensor operations and multiple nodes in a topological structure corresponding to the multiple hardware units, where the topological structure is a tree structure including a root node, branch nodes, and leaf nodes; Determine, for a plurality of nodes, a block strategy for determining tensor calculation dimensions corresponding to the tensor operations corresponding to the nodes in a direction from the root node to the leaf nodes; Based on the block strategy of the tensor calculation dimension corresponding to the tensor operation corresponding to each of the multiple nodes, the block strategy of the tensor calculation is obtained.

9. The method according to claim 8, characterized in that The topological structure further includes a plurality of unidirectional edges, wherein the unidirectional edges are used to connect different nodes, and the unidirectional edges represent the relationship between the nodes.

10. The method according to claim 9, characterized in that The plurality of unidirectional edges include first-type unidirectional edges and / or second-type unidirectional edges and / or third-type unidirectional edges; The first type of unidirectional edge represents that the tensor calculation dimension corresponding to the tensor operation corresponding to the nodes connected by the first type of unidirectional edge is the same as the blocking strategy; The second type of unidirectional edge representation does not require determining a blocking strategy for the tensor calculation dimension corresponding to the tensor operation corresponding to the node pointed to by the second type of unidirectional edge; The third type of unidirectional edge represents the communication connection between the hardware units represented by the nodes connected by the third type of unidirectional edge.

11. The method according to any one of claims 8 to 10, characterized in that: A blocking strategy for a tensor calculation dimension corresponding to the tensor operation corresponding to the node is related to an allocated storage space of a hardware unit represented by the node; The allocated storage space of the hardware unit represented by the node is related to the number of available sibling nodes of the node, wherein the sibling nodes are nodes located at the same layer as the node and represent the same type of hardware units.

12. The method according to claim 11, characterized in that The number of available sibling nodes of the node is related to the data size corresponding to the tensor operation and the number of the sibling nodes.

13. The method according to any one of claims 8 to 12, characterized in that: The hardware units represented by the root node and the branch node are storage units, and the hardware units represented by the leaf node are computing units.

14. The method according to any one of claims 8 to 13, characterized in that: The plurality of hardware units include different types of storage units and / or different types of computing units.

15. A block strategy generation device for tensor calculation, characterized in that: include: An acquisition module, configured to acquire information of multiple tensor operations corresponding to tensor calculations, wherein the information of each tensor operation includes a tensor calculation dimension corresponding to the tensor operation, a data type of an element corresponding to the tensor calculation dimension, and a priority of the tensor calculation dimension; a determination module, configured to determine a correspondence between the plurality of tensor operations and a plurality of hardware units, the plurality of hardware units being configured to perform operations on the tensor calculations; A block strategy generation module is used to obtain a block strategy for the tensor calculation dimension corresponding to the tensor operation based on the characteristic information of the hardware unit corresponding to the tensor operation, the storage space corresponding to the tensor operation, the data type of the element corresponding to the tensor calculation dimension, and the priority of the tensor calculation dimension; Based on the block strategies of the tensor calculation dimensions corresponding to the multiple tensor operations, a block strategy for the tensor calculation is obtained.

16. The device according to claim 15, characterized in that The block strategy generation module is specifically used to: Obtaining a blocking strategy for the first tensor calculation dimension based on characteristic information of the hardware unit corresponding to the tensor operation and a data type of an element corresponding to the first tensor calculation dimension; Obtaining a blocking strategy for a second tensor calculation dimension based on a storage space corresponding to the tensor operation and the blocking strategy for the first tensor calculation dimension; Obtaining a blocking strategy for the tensor calculation dimension corresponding to the tensor operation based on the blocking strategy for the first tensor calculation dimension and the blocking strategy for the second tensor calculation dimension; The first tensor calculation dimension and the second tensor calculation dimension are both tensor calculation dimensions corresponding to the tensor operation, and the priority of the first tensor calculation dimension is higher than the priority of the second tensor calculation dimension.

17. The device according to claim 15 or 16, characterized in that The storage space corresponding to the tensor operation is related to the number of the tensor operations corresponding to the hardware unit.

18. The device according to claim 17, characterized in that Also includes: a reuse analysis module, configured to perform storage reuse analysis on the plurality of tensor operations corresponding to the hardware unit to determine a tensor operation with reusable storage space among the plurality of tensor operations corresponding to the hardware unit; An update module is used to update the number of tensor operations corresponding to the hardware unit based on the storage reuse analysis results of the respective tensor operations.

19. The device according to any one of claims 15 to 18, characterized in that The information of the tensor operation also includes the operation type corresponding to the tensor operation; The determining module is specifically configured to: Based on the correspondence between the operation types of the multiple tensor operations and the hardware units, at least one hardware unit corresponding to the tensor operation is obtained.

20. The device according to any one of claims 15 to 19, characterized in that The multiple hardware units include a storage unit and a computing unit. The characteristics of the storage unit include storage granularity, and the characteristics of the computing unit include computing batch size.

21. The device according to any one of claims 15 to 20, characterized in that The plurality of hardware units include different types of storage units and / or different types of computing units.

22. A block strategy generation device for tensor calculation, characterized in that: include: an acquisition module, configured to acquire information of a plurality of tensor operations corresponding to the tensor calculation, wherein the information of each tensor operation includes an operation type and a hardware tag, wherein the hardware tag represents a hardware unit on which the operation type depends; a determination module, configured to determine a topological structure corresponding to the plurality of hardware units, the topological structure comprising a plurality of nodes, the nodes representing the hardware units related to the tensor calculation; a mapping module, configured to establish, based on the operation type and the hardware label, a mapping relationship between the plurality of tensor operations and a plurality of nodes in a topological structure corresponding to the plurality of hardware units, wherein the topological structure is a tree structure including a root node, branch nodes, and leaf nodes; A block strategy generation module is used to determine, for a plurality of nodes, a block strategy of tensor calculation dimensions corresponding to the tensor operations corresponding to the nodes in a direction from the root node to the leaf node; Based on the block strategy of the tensor calculation dimension corresponding to the tensor operation corresponding to each of the multiple nodes, the block strategy of the tensor calculation is obtained.

23. The device according to claim 22, characterized in that The topological structure further includes a plurality of unidirectional edges, wherein the unidirectional edges are used to connect different nodes, and the unidirectional edges represent the relationship between the nodes.

24. The device according to claim 23, characterized in that The plurality of unidirectional edges include first-type unidirectional edges and / or second-type unidirectional edges and / or third-type unidirectional edges; The first type of unidirectional edge represents that the tensor calculation dimension corresponding to the tensor operation corresponding to the nodes connected by the first type of unidirectional edge is the same as the blocking strategy; The second type of unidirectional edge representation does not require determining a blocking strategy for the tensor calculation dimension corresponding to the tensor operation corresponding to the node pointed to by the second type of unidirectional edge; The third type of unidirectional edge represents the communication connection between the hardware units represented by the nodes connected by the third type of unidirectional edge.

25. The device according to any one of claims 22 to 24, characterized in that A blocking strategy for a tensor calculation dimension corresponding to the tensor operation corresponding to the node is related to an allocated storage space of a hardware unit represented by the node; The allocated storage space of the hardware unit represented by the node is related to the number of available sibling nodes of the node. The sibling nodes are nodes located at the same layer as the node and represent the same type of hardware units.

26. The device according to claim 25, characterized in that The number of available sibling nodes of the node is related to the data size corresponding to the tensor operation and the number of the sibling nodes.

27. The device according to any one of claims 22 to 26, characterized in that The hardware units represented by the root node and the branch node are storage units, and the hardware units represented by the leaf node are computing units.

28. The device according to any one of claims 22 to 27, characterized in that The plurality of hardware units include different types of storage units and / or different types of computing units.

29. A computing device comprising a memory and a processor, characterized in that: The memory stores executable code, and the processor executes the executable code to implement the method described in any one of claims 1 to 7 and / or the method described in any one of claims 8 to 14.

30. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 7 and / or the method according to any one of claims 8 to 14.

Images