NPU-based Strassen matrix multiplication acceleration method

Abstract

The present invention discloses an NPU-based Strassen matrix multiplication acceleration method, which belongs to the field of hybrid programming technology. The problem that there is no method for realizing fast matrix multiplication using open source hybrid programming operators for Ascend chips in the prior art is solved; the present invention designs an Ascend_Strassen operator data path, performs data transfer and segmentation on the input matrix to obtain a block matrix; performs a first matrix addition and subtraction operation based on the block matrix to obtain the input data of the matrix multiplication stage, performs zero addition operation on some block matrices to obtain a converted matrix; performs matrix multiplication operation to obtain a matrix multiplication result; performs a second matrix addition and subtraction operation based on the matrix multiplication result to obtain a submatrix; performs data transfer and merging on the submatrix to obtain an output matrix. The present invention effectively shortens the matrix operation time and improves the calculation accuracy, and can be applied to Ascend chip hybrid programming.

Description

Technical Field

[0001] The invention relates to a Strassen matrix multiplication acceleration method, in particular to a Strassen matrix multiplication acceleration method based on an NPU, and belongs to the technical field of hybrid programming. Background Art

[0002] In the existing technology, Huawei first launched the Huawei Ascend series chips covering all scenarios of artificial intelligence and the Atlas intelligent computing platform with mainstream heterogeneous computing components in the industry. Later, it officially launched the operator programming language Ascend C, which is benchmarked against NVIDIA CUDA C++. Its predecessor is the TIK C++ programming model in the Ascend self-developed heterogeneous computing architecture (CANN). Ascend C is a programming language launched by CANN for operator development scenarios. It natively supports C and C++ standard specifications, which can maximize the matching of user development habits. Through key technologies such as multi-layer interface abstraction, automatic parallel computing, and twin debugging, it greatly improves the efficiency of operator development and helps AI developers complete operator development and model tuning and deployment at low cost. However, as an emerging programming language model, the operator programming language Ascend C still needs to be continuously iterated and improved. The essence of the operator programming language Ascend C is a combination of C++ and a set of specially developed APIs, that is, it natively supports C and C++ standard specifications, and additionally provides a set of highly encapsulated high-performance class library interfaces for developers to assemble operator core logic. Ascend C is used to develop operators that run on Ascend chips. Operators are generally executed in neural networks and refer to functions or operators that perform specific mathematical operations. Operators can be simple mathematical operations, such as addition and multiplication, or complex operations, such as convolution, pooling, activation functions, etc. The core part of an operator is called the kernel function, which contains the program logic that the operator needs to execute and is the main part of an operator. For the kernel function of Ascend C, Ascend officials have also provided certain programming paradigm examples. Developers can adapt their own operators based on this sample template, so that they can more conveniently and quickly write kernel functions that can run correctly on NPU (network processor) devices.

[0003] The Strassen algorithm is essentially a recursive divide-and-conquer algorithm for matrix multiplication. Its core idea is to divide two large matrices A and B into smaller sub-matrices, and then calculate their product through optimized recursive calls. However, it can complete matrix multiplication in a shorter time than traditional methods. The key idea of ​​the Strassen algorithm is to reduce the amount of calculation by reducing the number of multiplications. Although the number of operations in each step is more than the traditional method, by reducing the number of recursive calls, the overall amount of calculation is reduced. The Strassen algorithm also decomposes each matrix into 4 However, unlike the basic divide-and-conquer algorithm, the Strassen algorithm reduces the number of matrix multiplications from 8 to 7 by cleverly optimizing the recursive call process. The time complexity of the Strassen algorithm for decomposing the matrix is ​​Θ(1), which can be ignored. The time complexity of the Strassen algorithm for the first matrix addition and subtraction is Θ(n 2 ), the time complexity of the Strassen algorithm for 7 matrix multiplications is The time complexity of the second matrix addition and subtraction operation of the Strassen algorithm is Θ(n 2 );

[0004] The time complexity T(n) of the Strassen algorithm is expressed as:

[0005]

[0006] The time complexity of the Strassen algorithm can be calculated by the Master Theorem:

[0007] The Strassen matrix multiplication acceleration method can usually be applied to image processing, scientific computing and machine learning. In the field of image processing, especially in convolutional neural networks (CNN), convolution operations are essentially large-scale matrix multiplications, which usually consume a lot of computing resources. In order to accelerate convolution operations, the Strassen matrix multiplication method can be used to reduce the time complexity of matrix multiplication. Assuming there is a large-resolution image and a large filter, traditional convolution calculations require a large number of matrix multiplication operations. If conventional matrix multiplication algorithms are used directly, the amount of calculation will be very large. However, using the Strassen matrix multiplication acceleration method to accelerate can reduce the number of multiplications to It can significantly reduce the calculation time when performing image processing;

[0008] Ascend chip also provides a matrix multiplication high-level API, so that developers can complete matrix multiplication more conveniently. The matrix multiplication high-level API includes the registration and calculation part in the kernel function on the device side (the part running on the NPU board) and the calculation and segmentation information function (referred to as tiling function, this part is mainly calculated on the host side, does not occupy the scalar scalar calculation unit on the board, thereby increasing the overall function running speed) on the host side (the part running on the host, that is, the part running on the non-NPU board). The operation principle of the matrix multiplication high-level API is that multi-core parallel computing can be used, so the complete data can be distributed to multiple cores, each core only calculates part of the data, and finally the data calculated on each core is unified to form the final complete result. The calculation content in each kernel function is independent and will not be related to the data being calculated in other cores. Ascend C high-level matrix multiplication API is a standard single-core API, which is only responsible for calculating the data assigned to the current core. For reference, Figure 3 , a matrix multiplication example with 8 cores participating in the calculation, the original left matrix size is M×K, the original right matrix size is K×N, and the result matrix is ​​M×N. Because of multi-core parallelism, the amount of data required to be calculated in a single core will be smaller than the original matrix. The green part in the figure is an example of segmentation, representing the amount of calculation on a single core and the amount of data of the output result. Therefore, the size of the left matrix that actually performs the calculation in a core is SingleCoreM×K, and the size of the right matrix is ​​K×SingleCoreN. The result matrix core3; In a single core, it is necessary to continue to set the operation control parameters, Ascend The C high-level matrix multiplication API provides two calculation modes. The first is the IterateAll mode, which can be understood as a direct calculation mode. It can complete the calculation of all data allocated to the core in one step. The calculation kernel encapsulated by the API is the traditional Mmad matrix multiplication and addition operation. The advantage of the IterateAll mode is that the operation is simple and intuitive, and it is fully capable of computing tasks for operators without special engineering requirements. The second is the Iterate mode, which can be understood as a fine-grained control calculation mode. It is also necessary to set more detailed parameters to complete the matrix calculation of the single core. It is suitable for matrix multiplication operations with special engineering requirements. The calculation kernel encapsulated by the API is also the traditional Mmad matrix multiplication and addition operation. Figure 4Developers need to further allocate baseM on the scale of SingleCoreM and baseN on the scale of SingleCoreN. An Iterate process calculates the result of multiplying the baseM×SingleCoreK block shown in the blue part with the SingleCoreK×baseN block. Under default conditions, the SingleCoreK×baseN block in the right matrix will be fixed first, and the left matrix will be traversed from top to bottom according to the scale of baseM. After traversing the scale of SingleCoreM, the fixed block in the right matrix will be moved for the next round of traversal calculation. In an Iterate calculation process, baseK will be automatically split according to the size of SingleCoreK, and matrix multiplication and addition will be performed. However, the current version does not support developers to customize the value of baseK.

[0009] In summary, Ascend has not yet officially released the matrix multiplication operator implemented using the divide-and-conquer algorithm. The Strassen algorithm contains both matrix addition and subtraction (vector operations) and matrix multiplication, that is, operations involving both vector and matrix computing resources. According to Ascend's official operator classification method, it can be divided into hybrid programming operators, but there is currently no official open source example of hybrid programming operators implementing matrix multiplication. In order to speed up the computing efficiency between matrices and optimize and expand the hybrid programming operators adapted to the Ascend chip, an NPU-based Strassen matrix multiplication acceleration method is needed. Summary of the invention

[0010] A brief overview of the present invention is provided below in order to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to a more detailed description discussed later.

[0011] In view of this, in order to solve the problem that there is no method for implementing fast matrix multiplication using open source hybrid programming operators for Ascend chips in the prior art, the present invention provides a Strassen matrix multiplication acceleration method based on NPU.

[0012] The technical solution is as follows: The NPU-based Strassen matrix multiplication acceleration method includes the following steps:

[0013] S1. Design the Ascend_Strassen operator data path to move and split the input matrix to obtain a block matrix;

[0014] S2. Perform the first matrix addition and subtraction operation according to the block matrix to obtain the input data of the matrix multiplication stage, add zero to some block matrices to obtain the converted matrix;

[0015] S3. Perform a matrix multiplication operation according to the input data in the matrix multiplication stage and the converted matrix to obtain a matrix multiplication result;

[0016] S4. Perform a second matrix addition and subtraction operation based on the matrix multiplication result to obtain a submatrix;

[0017] S5. Move and merge the sub-matrices to obtain the output matrix and complete the Ascend_Strassen operator data path design.

[0018] Furthermore, in S1, the Ascend CAPI library is queried, and the parameters are set as required using the DataCopy interface to move the input matrix stored on the Host side to the Device side. According to the TPosition type of the data, the data flow direction is obtained as GM to VECIN, and the two input matrices with a size of 32×32 and a data type of float16, namely the first input matrix A and the second input matrix B, are respectively divided into four matrices with a size of 16×16 and a data type of float16, and the eight block matrices are respectively the first matrix A 11 , the second matrix A 12 , the third matrix A 21 、The fourth matrix A 22 、The fifth matrix B 11 、The sixth matrix B 12 、The seventh matrix B 21 、The eighth matrix B 22 , the TPosition types of the 8 block matrices are all set to VECIN.

[0019] Furthermore, in S2, the input data of the matrix multiplication stage is calculated using the Add interface and the Sub interface according to the 8 block matrices {S 1 , S 2 ,......S 10}, query the Ascend CAPI library, according to the high-level matrix multiplication API constraints of Ascend C, the TPosition type of the input data in the matrix multiplication stage is VECOUT, and the data flow direction is VECIN to VECOUT. The first matrix A is added through the scalar binary addition instruction Adds. 11 、The fourth matrix A 22 、The fifth matrix B 11 、The fourth matrix B 22 Perform the zero-adding operation to obtain the converted matrices, which are the converted first matrix NA11 、The fourth matrix after transformation is NA 22 、The converted fifth matrix NA 11 、The eighth matrix NB after transformation 22 And convert its TPosition type to VECOUT.

[0020] Furthermore, in S3, according to the input data {S 1 , S 2 ,......S 10} and the converted matrix, use the Ascend C matrix multiplication high-level API: Matmul instruction to complete 7 groups of matrix multiplications and get the matrix multiplication result {P 1 , P 2 ,......P 7}, according to the high-level matrix multiplication API constraints of Ascend C, the TPosition type of all input data in the matrix multiplication process is VECOUT, the data flow direction is VECOUT to VECIN, and the matrix multiplication result {P 1 , P 2 ,......P 7}'s TPosition type is set to VECIN.

[0021] Further, in S4, according to the matrix multiplication result {P 1 , P 2 ,......P 7}, use the Add interface and the Sub interface to calculate the submatrix {C 11 , C 12 , C 21 , C 22}, submatrix {C 11 , C 12 , C 21 , C 22}The TPosition type is set to VECOUT, and the data flow direction is from VECIN to VECOUT.

[0022] Furthermore, in S5, the submatrix {C 11 , C 12 , C 21 , C 22 The data of} is transferred from the Device side to the Host side, and the sub-matrices are merged to obtain the output matrix C, whose size is 32×32. The data flow direction is VECOUT to GM, completing the Ascend_Strassen operator data path design.

[0023] The beneficial effects of the present invention are as follows: the present invention organically combines the Strassen algorithm with the operator programming language Ascend C to obtain a divide-and-conquer matrix multiplication operator, namely the Ascend_Strassen operator, which belongs to a hybrid programming operator. After actual operation tests on the AtlasA2 training series products, the multiplication operation of two matrices consisting of 1024 float16 (half) type data is processed, and the execution time is about 49.233 microseconds on average. The present invention can shorten the matrix operation time in the Ascend C hybrid programming mode and improve the accuracy of the operation results;

[0024] The operator result correctness test and operator performance test are as follows: Figure 8 First, use the random method of the numpy library in Python to generate two 32×32 random number matrices of half type as input matrices, and input them to the custom operator and numpy.matmul method respectively to obtain the result real calculated by the custom operator and the result golden calculated by numpy.matmul. The result data calculated by numpy.matmul must be correct, that is, golden is the true value result. If the value of the result real is the same as the true value result golden, it can be verified that the execution logic of the custom operator is completely correct. Convert the result real and the true value result golden into binary data in .bin format and use the md5sum method to compare them. If the results are consistent, it proves that the value and precision of the output result of the custom operator are completely correct. At the same time, you can also explicitly use the print method to print out the two output results for comparison to see whether the reason for the difference in binary data is precision. Refer to Fig. 9 The last two rows of data are the md5sum values ​​of the matrix multiplication results completed using the numpy library and the md5sum values ​​of the calculation results output by Ascend_Strassen under the same input data. The same values ​​prove that the calculation results and accuracy are correct.

[0025] The operator performance test is conducted in the Ascend AtlasA2 training series product environment. The test method is to use the msprof software included in the CANN software package to record the operator execution performance data information. The kernel function execution time can be obtained after the operator is executed.

[0026] Referring to Table 1, a total of 10 experiments were conducted, and each time the input data was a different random number, and the running time of the Ascend_Strassen operator kernel function was obtained;

[0027] Table 1

[0028] No.1 No.2 No.3 No.4 No.5 No.6 No.7 No.8 No.9 No.10 49.301 49.501 48.781 49.801 49.361 50.801 48.701 48.761 48.661 48.561

[0029] The calculation shows that the execution time of 10 experimental evaluations is 49.233 microseconds. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0031] Figure 1 It is a flowchart of the NPU-based Strassen matrix multiplication acceleration method;

[0032] Figure 2 This is a schematic diagram of the AI ​​Core abstract architecture;

[0033] Figure 3 This is a schematic diagram of multi-core data segmentation calculation;

[0034] Figure 4 This is a schematic diagram of in-core computing;

[0035] Figure 5 Schematic diagram of the data path design for the Ascend_Strassen operator;

[0036] Figure 6 This is a schematic diagram of the operation of data segmentation and import in STEP1 of the Ascend_Strassen operator;

[0037] Figure 7 This is a schematic diagram of the operation of data merging and moving out in STEP5 of the Ascend_Strassen operator;

[0038] Figure 8 This is a schematic diagram for the correctness test of the Ascend_Strassen operator;

[0039] Fig. 9 This is a schematic diagram of the Ascend_Strassen operator correctness test results;

[0040] Fig.10 A schematic diagram of the code defined for the kernel function operator class;

[0041] Fig.11 Code diagram for the Init() function implementation;

[0042] Fig.12 Code diagram for the Process() function;

[0043] Fig.13 This is a code diagram for the CopyIn() function.

[0044] Fig.14 Code diagram for the AddandSub1() function;

[0045] Fig.15 This is the code diagram for the first part of the MatmulCompute() function implementation;

[0046] Fig.16 The code diagram for the second part of the MatmulCompute() function implementation;

[0047] Fig.17 Code diagram for the AddandSub2() function;

[0048] Fig.18 Code diagram for the CopyOut() function implementation;

[0049] Fig.19 A schematic diagram of the code implemented for the operator kernel function body;

[0050] Fig. 20 A code diagram for setting up a function to implement matrix multiplication information. DETAILED DESCRIPTION

[0051] In order to make the technical solutions and advantages of the embodiments of the present invention more clearly understood, the exemplary embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0052] refer to Figure 1-Figure 20 The present embodiment is described in detail, and the NPU-based Strassen matrix multiplication acceleration method specifically includes the following steps:

[0053] S1. Design the Ascend_Strassen operator data path to move and split the input matrix to obtain a block matrix;

[0054] S2. Perform the first matrix addition and subtraction operation according to the block matrix to obtain the input data of the matrix multiplication stage, add zero to some block matrices to obtain the converted matrix;

[0055] S3. Perform a matrix multiplication operation according to the input data in the matrix multiplication stage and the converted matrix to obtain a matrix multiplication result;

[0056] S4. Perform a second matrix addition and subtraction operation based on the matrix multiplication result to obtain a submatrix;

[0057] S5. Move and merge the sub-matrices to obtain the output matrix, and complete the Ascend_Strassen operator data path design;

[0058] Specifically, one of the main features of the operator programming language Ascend C is that it is strongly related to hardware. The following is a brief introduction to some content related to Ascend hardware. The core computing unit in the Ascend chip is the AI ​​Core. Figure 2 , AI Core completely separates the matrix calculation module (AIC) from the vector calculation module (AIV) and connects them through a bus (MAIN BUS). The above operation greatly increases the parallelism of the calculation, and can realize the simultaneous matrix and vector operations, while also providing more optimization angles for operator writing; in the matrix operation module, ScalarALU is used to control the scalar instruction flow, the L1 Buffer register is used to store the incoming matrix data, the FeatureMap L0A register and the Weights L0B register are used to store the two input matrix data after data format preprocessing, the CUBEALU (the unit that actually performs matrix operations) is used to perform various operations between matrices, the L0C register is used to store the result matrix, and the matrix calculation module is mainly responsible for performing matrix multiplication related operations; in the vector operation module, there is only one general register UB (UnifiedBuffer) for storing all vector data, the ScalarALU is used to control the scalar instruction flow, the VECALU (the unit that actually performs vector calculations) is used to perform all vector operations, and the vector calculation module is mainly responsible for performing all vector operations;

[0059] Due to the complete separation of the matrix calculation module and the vector calculation module, Ascend officially divides the operator types of the operator programming language Ascend C into three categories, namely vector operators (only using vector calculation modules), matrix operators (only using matrix calculation modules) and hybrid programming operators (the vector calculation module and the matrix calculation module are used in the operator calculation process). At the same time, due to the above-mentioned characteristics that are strongly related to hardware, the hardware position of each data needs to be explicitly considered in the Ascend C programming process, that is, Ascend C will give each data a location attribute, which is crucial in planning the operator data path. In Ascend C, the data type stored outside the board is called GlobalTensor, and the data type stored on the board is called LocalTensor. Each LocalTensor has two attributes: size and location information, which are called Buffer respectively. Size and TPosition, among which TPosition is particularly important. Refer to Table 2 for the mapping relationship between TPosition types and physical memory hardware locations. TPosition types include GM, VECIN, VECOUT, VECCALC, A1, A2, B1, B2, CO1, and CO2. Among them, VECIN, VECCALC, and VECOUT are mainly used for vector programming, A1, A2, B1, B2, CO1, and CO2 are used for matrix programming, and GM represents external storage.

[0060] In actual programming, multiple different LocalTensors may be under the same TPosition type. Therefore, in order to facilitate data flow management, the actual definition of TPosition is a higher-level data structure TQue, that is, it is necessary to declare the TPosition type attribute of TQue first, and then TQue allocates LocalTensor. The TPosition attribute of LocalTensor inherits the TPosition type attribute of the TQue to which it is allocated;

[0061] Table 2

[0062]

[0063] Furthermore, in S1, the Ascend CAPI library is queried, and the DataCopy interface is used to set parameters as required to move the input matrix stored on the Host side to the Device side. According to the TPosition (logical mapping of a data in a physical position in AscendC for easy management) type of the data, the data flow direction is obtained as GM (a TPosition, an off-board storage space) to VECIN (a TPosition, data is moved into a logical position), and two input matrices with a size of 32×32 and a data type of float16, namely the first input matrix A and the second input matrix B, are respectively divided into four matrices with a size of 16×16 and a data type of float16, and the data type of the input matrix is ​​float16 (half), and the eight block matrices obtained are the first matrix A and the second input matrix B. 11 , the second matrix A 12 , the third matrix A 21 、The fourth matrix A 22 、The fifth matrix B 11 、The sixth matrix B 12 、The seventh matrix B 21 、The eighth matrix B 22 , the TPosition types of the 8 block matrices are all set to VECIN.

[0064] Furthermore, in S2, according to the 8 block matrices, the input data of the matrix multiplication stage is calculated using the Add (vector addition) interface and the Sub (vector subtraction) interface {S 1 , S 2 ,......S 10}, query the Ascend CAPI library, according to the high-level matrix multiplication API constraints of Ascend C, the TPosition type of the input data in the matrix multiplication stage is VECOUT, and the data flow direction is VECIN to VECOUT. The first matrix A is added through the scalar binary addition instruction Adds. 11 、The fourth matrix A 22 、The fifth matrix B 11 、The fourth matrix B 22 Perform the zero-adding operation to obtain the converted matrices, which are the converted first matrix NA 11 、The fourth matrix after transformation is NA 22 、The converted fifth matrix NA 11 、The eighth matrix NB after transformation 22 And convert its TPosition type to VECOUT;

[0065] Specifically, since the first matrix A will be used in the subsequent matrix multiplication stage 11、The fourth matrix A 22 、The fifth matrix B 11 、The fourth matrix B 22 However, the TPosition type of the above four matrices is still VECIN and cannot be used directly. They need to be converted through the scalar binary addition instruction Adds as an intermediate step to adjust their TPosition type to VECOUT.

[0066] Furthermore, according to the input data {S 1 , S 2 ,......S 10} and the converted matrix, use the matrix multiplication Matmul instruction to complete 7 sets of matrix multiplications and get the matrix multiplication result {P 1 , P 2 ,......P 7}, according to the high-level matrix multiplication API constraints of Ascend C, the TPosition type of all input data in the matrix multiplication process is VECOUT, the data flow direction is VECOUT to VECIN, and the matrix multiplication result {P 1 , P 2 ,......P 7}'s TPosition type is set to VECIN.

[0067] Further, according to the matrix multiplication result {P 1 , P 2 ,......P 7}, use the Add interface and the Sub interface to calculate the submatrix {C 11 , C 12 , C 21 , C 22}, because the next stage of data removal and merging requires the use of the DataCopy interface to transfer data to the off-board storage GM, and the DataCopy interface can only realize data transfer from VECOUT to GM, the sub-matrix {C 11 , C 12 , C 21 , C 22}The TPosition type can only be set to VECOUT, and the data flow direction is from VECIN to VECOUT.

[0068] Furthermore, in S5, the submatrix {C 11 , C 12 , C 21 , C 22} data is transferred from the Device side to the Host side, and the sub-matrices are merged to obtain the output matrix C, whose size is 32×32, and the data flow direction is from VECOUT to GM. The Ascend_Strassen operator data path design is completed, and the data type of the output matrix C is float32;

[0069] Specifically, refer to Figure 5 In this embodiment, the software environment for the Ascend_Strassen operator is CANN Community Edition 8.0.RC2.alpha001, and the hardware environment is AtlasA2 training series products. The specific implementation steps are as follows:

[0070] (1) Class definition and main function implementation

[0071] refer to Fig.10 , the class of Ascend_Strassen operator is named strassenKernel, which mainly includes 2 main functions and 5 stage functions. The main functions are initialization function Init() and logic execution function Process(). The stage functions are CopyIn() (data moving in function, corresponding to step S1 of the specific implementation method), AddandSub1() (first matrix addition and subtraction function, corresponding to step S2 of the specific implementation method), MatmulCompute() (matrix multiplication function, corresponding to step S3 of the specific implementation method), AddandSub2() (second matrix addition and subtraction function, corresponding to step S4 of the specific implementation method) and CopyOut() (data moving out function, corresponding to step S5 of the specific implementation method), which respectively correspond to the 5 steps of Strassen algorithm, namely matrix decomposition, first matrix addition and subtraction, matrix multiplication, second matrix addition and subtraction and matrix merging;

[0072] The main function part mainly includes the definition of the matrix multiplication class, which is named mm1, the setting of the matrix multiplication execution information structure, which is named tiling, the definition of the off-board input and output, that is, the definition of the input as GMmatrixA and GMmatrixB, the definition of the output as GMmatrixC, the declaration of the on-chip TQue variable, including the need to use the address offset method on the chip to store the block matrix obtained by decomposing the first input matrix A and the second input matrix B, which is used to store the S in the result of the first matrix addition and subtraction operation. 1 , S 2 , S 3 , S 4 (i.e. S1234) and S 5 , S 6 , S 7 , S 8 (i.e. S5678) and S9 , S 10 (ie, S910), N for performing TPosition type conversion, P for storing the result of the matrix multiplication operation 1 , P 2 , P 3 , P 4 (i.e. P1234) and P 5 , P 6 , P 7 (i.e., P567), used to store the block matrix C of the output matrix C;

[0073] refer to Fig.11 , the initialization Init() function mainly performs four operations. The first is to move in the matrix segmentation information. The second is to locate the off-chip input and output memory location, using the SetGlobalBuffer() method (locating the off-board input space function). The third is to allocate the workspace (work running space) unique to the Ascend C programming operator. The fourth is to initialize all the applied TQues, which requires explicit allocation of memory space size.

[0074] refer to Fig.12 ,The Process() function is mainly responsible for executing the calculation logic function, that is, executing the 5-stage function;

[0075] (2) Specific implementation of the five-stage functions

[0076] refer to Fig.13 , CopyIn() function is implemented. In CopyIn(), the TPosition type of the first input matrix A and the second input matrix B is GM. The matrix segmentation and data transfer operations need to be implemented using the DataCopy interface. However, the DataCopy interface has restrictions on the TPosition type of input and output data. When the input TPosition type is GM, the output result can only be VECIN. Therefore, the first matrix A 11 , the second matrix A 12 , the third matrix A 21 、The fourth matrix A 22 、The fifth matrix B 11 、The sixth matrix B 12 、The seventh matrix B 21 and the eighth matrix B 22 The TPosition type can only be set to VECIN, which is allocated by the first variable TQueA and the second variable TQue B. Among them, quarterSize is a global constant set, and its value is 256, which is the data amount of 1 small matrix;

[0077] In AscendC, the number of memory blocks allocated to the same TPosition type TQue in a kernel function is strictly limited to no more than 8, and the sum of all allocated memory sizes cannot exceed the buffer size of the Ascend hardware environment used. That is, there is no limit on the number of LocalTensors in a kernel function, but there is a strict limit on the number of TQues. Therefore, in order to control the number of buffers allocated by the TQue type to less than 8, when writing operators, you cannot allocate a separate TQue to each intermediate variable. You need to allocate a piece of memory for some related variables together, and determine the storage start address by offset.

[0078] refer to Figure 6 , the data length of a small matrix is ​​256 and is stored continuously. A LocalTensor with a length of 1024 is allocated through TQue, and a LocalTensor is used to store the four split matrices. In this embodiment, the global fixed parameter quarterSize is set to 256, and the offset storage is performed in the following order: the first matrix A 11 The first address offset is 0, and the second matrix A 12 The first address offset is quarterSize, and the third matrix A 21 The first address offset is 2*quarterSize, the fourth matrix A 22 The first address offset is 3*quarterSize, and so on. The maximum length of the data volume of a LocalTensor is 1024.

[0079] When using DataCopy to split a matrix and store it continuously, the physical storage of the matrix data is row-major by default. In Ascend C, this data format is called ND format. The broken line in the figure represents the arrangement order of data in physical space. The DataCopy interface can realize data transfer and set the total number of transfers blockCount, the amount of data transferred at a time blockLen, the source operand transfer step srcStride, the destination operand receiving data step dstStride and other information. One DataCopy transfers 16 half-type data and is called a block. The above parameters include the amount of data transferred at a time blockLen, the source operand transfer step srcStride and the destination operand receiving data step dstStride. The length dstStride is in blocks. It takes 16 moves to get one small matrix. The total number of moves blockCount is set to 16. Each move is 1 block, which means 16 half-type data. The amount of data moved at one time blockLen is set to 1. The source operand is not moved continuously. One move requires 16 half-type data jumps, which means 1 block. The source operand move step srcStride is set to 1. The destination operand receives and stores data continuously. The destination operand receives data step dstStride is set to 0. The first matrix A is moved. 11 When the second matrix A is moved from the source operand offset = 0 12 When the third matrix A is moved, it starts from the source operand offset = 16. 21 When the fourth matrix A is moved from the source operand offset = 512 22 When , it starts from the source operand offset = 528;

[0080] refer to Fig.14 , AddandSub1() function is implemented. In AddandSub1(), 8 small matrices are added and subtracted with each other. The Add interface and Sub interface in Ascend C are used to obtain 10 intermediate matrices S 1 , S 2 ,......S 10 , the TPosition type of the 10 intermediate matrices is VECOUT. At the same time, considering that when performing matrix multiplication in step S3, four matrices, namely the first matrix A, are involved 11 、The fourth matrix A 22 、The fifth matrix B 11 And the fourth matrix B 22However, because the input data type of the matrix multiplication API can only be VECOUT, the TPosition type of the four matrices is still VECIN, so it is necessary to newly allocate the converted first matrix NA 11 、The fourth matrix after transformation is NA 22 、The converted fifth matrix NA 11 、The eighth matrix NB after transformation 22 There are 4 intermediate matrices of TPosition type VECOUT. 11 、The fourth matrix A 22 、The fifth matrix B 11 And the fourth matrix B 22 The data in is transferred to it. The above process is implemented by using the Adds scalar binary addition instruction, which adds the same scalar size to each data and sets the added scalar size to 0 to realize the data transfer function.

[0081] refer to Fig.15 and Fig.16 , MatmulCompute() function is implemented. In MatmulCompute(), the Ascend C high-level matrix multiplication API is used to complete 7 times of 16×16 matrix multiplication with 16×16 matrix to obtain a matrix multiplication result containing 7 matrices {P 1 , P 2 ,......P 7}, and considering that the matrix multiplication result is still needed for addition and subtraction operations in the future, the TPosition type of the above 7 LocalTensors can only be set to VECIN due to the restrictions on the output TPosition type of the advanced matrix multiplication API and the restrictions on the input of the Add interface and Sub interface;

[0082] refer to Fig.17 , AddandSub2() function is implemented, and the splicing of this stage can be completed by using the Add interface and the Sub interface. Similarly, the data of the previous stage needs to be taken out through the DeQue (the function to take data out of the queue in Ascend C) operation, and the memory space required for this stage is allocated. The submatrix {C 11 , C 12 , C 21 , C 22 The storage space of the first sub-matrix C 11 The storage address of the first sub-matrix is ​​CLOCAL[0], and the second sub-matrix C 12 The storage address of the third submatrix C is CLOCAL[quarterSize]. 21The storage address of the fourth submatrix C is CLOCAL[2*quarterSize]. 22 The storage first address is CLOCAL[3*quarterSize];

[0083] refer to Fig.18 , CopyOut() function is implemented. In CopyOut(), the DataCopy interface needs to be used to copy the first submatrix C 11 , the second sub-matrix C 12 , the third sub-matrix C 21 , the fourth sub-matrix C 22 The data is transferred to the output matrix C stored outside the board. At this time, the physical storage format of the data is ND, that is, row-major order storage;

[0084] refer to Figure 7 , it takes 16 data transfers to transfer 1 small block, so the total number of transfers blockCount is set to 16, and because the data type is float at this time, and 1 data transfer can transfer 1 block, that is, 16 half-type data, so to transfer 1 whole row of 16 float type data at a time, it takes 2 blocks to transfer, so the data transfer amount blockLen is set to 2, the source operand is transferred continuously, the source operand transfer step srcStride is set to 0, the destination operand is received non-continuously, and 16 float data need to be jumped for each reception. The 16 float data type is 2 blocks long, so the destination operand receiving data step dstStride is set to 2, the first sub-matrix C 11 Since the offset is moved to the output matrix C, the second submatrix C 12 Since it is moved to the offset=16 position of the output matrix C, the third submatrix C 21 Since it is moved to the offset = 512 position of the output matrix C, the fourth sub-matrix C 22 Because it is moved to the offset=528 position of the output matrix C;

[0085] (3) Operator kernel function body and matrix multiplication information setting

[0086] refer to Fig.19The second line indicates the declaration of a strassenKernel class named op, the third line indicates the application of the TPipe type of the Ascend C-specific on-chip memory management system, which is a common step for all operator programming languages ​​Ascend C, the fourth line indicates the execution of the Init function, the fifth line indicates the registration of information for matrix multiplication calculation, and the sixth line indicates the execution of the Process function. At this point, the kernel function part of the Ascend_Strassen operator has been written;

[0087] refer to Fig. 20 ,Using the advanced matrix multiplication API, you also need to write the tiling code on the Host side, which includes the matrix multiplication operation registration information and the operation scale setting information. In this embodiment, the TPosition type of the left matrix is ​​VECOUT, the CubeFormat is ND type, the DataType is half type, that is, float16, and the transposition enable parameter is 0. The TPosition type of the right matrix is ​​VECOUT, the CubeFormat is ND type, the DataType is half type, that is, float16, and the transposition enable parameter is 0. The TPosition type of the C matrix is ​​VECIN, and the CubeFormat is ND type , DataType is float32. Since Bias is not needed, the Bias parameter can be set arbitrarily. The Ascend_Strassen operator only needs single-core operation, so the MatmulApiTiling class for single-core operation is created as the data structure passed to the kernel function. Since the matrices involved in the multiplication in this operator are two matrices of data type float16 and size 16×16, and only a single core is used for calculation, the simplest calculation method is selected, that is, directly calculating the entire matrix. The original matrix M (number of rows in the left matrix), N (number of columns in the right matrix), and K (number of columns in the left matrix and number of rows in the right matrix) are all 16. At the same time, SingleCoreM(Ascend Ascend C matrix multiplication advanced API parameter, the number of rows of the left matrix assigned to a single core), SingleCoreK (Ascend C matrix multiplication advanced API parameter, the number of columns of the left matrix or the number of rows of the right matrix assigned to a single core), SingleCoreN (Ascend C matrix multiplication advanced API parameter, the number of columns of the left matrix or the number of columns of the right matrix assigned to a single core) are all set to 16, baseM (Ascend C matrix multiplication advanced API parameter, the number of rows of the result matrix obtained by an actual operation) and baseN (Ascend C matrix multiplication advanced API parameter, the number of columns of the result matrix obtained by an actual operation) are also 16, which can achieve the purpose of direct whole block calculation;

[0088] The present invention can be applied to image processing based on convolutional neural networks to perform convolution operations and speed up processing time.

[0089] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.

Claims

1. NPU-based Strassen matrix multiplication acceleration method, characterized in that: The following steps are involved: S1. Design the Ascend_Strassen operator data path to move and split the input matrix to obtain a block matrix; S2. Perform the first matrix addition and subtraction operation according to the block matrix to obtain the input data of the matrix multiplication stage, add zero to some block matrices to obtain the converted matrix; S3. Perform a matrix multiplication operation according to the input data in the matrix multiplication stage and the converted matrix to obtain a matrix multiplication result; S4. Perform a second matrix addition and subtraction operation based on the matrix multiplication result to obtain a submatrix; S5. Move and merge the sub-matrices to obtain the output matrix, and complete the Ascend_Strassen operator data path design; In S1, the Ascend CAPI library is queried, and the parameters are set as required using the DataCopy interface to move the input matrix stored on the Host side to the Device side. According to the TPosition type of the data, the data flow direction is obtained as GM to VECIN. The two input matrices with a size of 32×32 and a data type of float16, namely the first input matrix A and the second input matrix B, are respectively divided into four matrices with a size of 16×16 and a data type of float16, and the eight block matrices are respectively the first matrix A 11 , the second matrix A 12 , the third matrix A 21 、The fourth matrix A 22 、The fifth matrix B 11 、The sixth matrix B 12 、The seventh matrix B 21 、The eighth matrix B 22 , the TPosition types of the 8 block matrices are all set to VECIN.

2. The NPU-based Strassen matrix multiplication acceleration method according to claim 1, characterized in that: In S2, according to the 8 block matrices, the input data {S1, S2, ... S 10 }, query the Ascend CAPI library, according to the high-level matrix multiplication API constraints of Ascend C, the TPosition type of the input data in the matrix multiplication stage is VECOUT, and the data flow direction is VECIN to VECOUT. The first matrix A is added through the scalar binary addition instruction Adds. 11 、The fourth matrix A 22 、The fifth matrix B 11 、The fourth matrix B 22 Perform the zero-adding operation to obtain the converted matrices, which are the converted first matrix NA 11 、The fourth matrix after transformation is NA 22 、The converted fifth matrix NA 11 、The eighth matrix NB after transformation 22 And convert its TPosition type to VECOUT.

3. The NPU-based Strassen matrix multiplication acceleration method according to claim 2, characterized in that: In S3, according to the input data {S1, S2, ... S 10 } and the converted matrix, use the Ascend C matrix multiplication advanced API: Matmul instruction to complete 7 groups of matrix multiplications to obtain the matrix multiplication results {P1, P2, ... P7}. According to the constraints of the Ascend C advanced matrix multiplication API, the TPosition type of all input data in the matrix multiplication process is VECOUT, the data flow direction is VECOUT to VECIN, and the TPosition type of the matrix multiplication results {P1, P2, ... P7} are all set to VECIN.

4. The NPU-based Strassen matrix multiplication acceleration method according to claim 3, characterized in that: In S4, according to the matrix multiplication result {P1, P2, ... P7}, the submatrix {C 11 , C 12 , C 21 , C 22 }, submatrix {C 11 , C 12 , C 21 , C 22 }The TPosition type is set to VECOUT, and the data flow direction is from VECIN to VECOUT.

5. The NPU-based Strassen matrix multiplication acceleration method according to claim 4, characterized in that: In S5, the DataCopy interface is used to copy the submatrix {C 11 , C 12 , C 21 , C 22 } data is moved from the Device side to the Host side, and the sub-matrices are merged to obtain the output matrix C, whose size is 32×32, and the data flow direction is VECOUT to GM, completing the Ascend_Strassen operator data path design.

Images