Sparse matrix vector multiplication optimization method for domestic intelligent chip

Abstract

The invention discloses a sparse matrix vector multiplication optimization method for a domestic intelligent chip, and the method comprises the steps: firstly obtaining a sparse matrix in a memory, and carrying out the partitioning of the obtained sparse matrix; secondly, a row offset array, a column array and a value array of the sparse matrix are read, the row offset array, the column array and the value array are stored in a sparse matrix structure in an aligned mode, three-dimensional representation of the sparse matrix is obtained, and operators of addition and subtraction are reloaded. And then based on the three-dimensional representation of the reloaded sparse matrix, uniformly distributing the calculation task to each thread to realize multi-thread load balancing. And finally, integrating the calculation tasks of each thread to complete the calculation of the SpMV. According to the method, the optimization of the average reasoning time of the sparse matrix vector multiplication is realized, and the calculation performance of the sparse matrix vector multiplication from a ten-thousand-level scale to a ten billion-level scale under the condition of high sparseness is improved.

Description

Technical Field

[0001] The present invention belongs to the field of edge computing technology, and specifically is an optimization method for sparse matrix-vector multiplication (SpMV) based on a domestic smart chip. Background Art

[0002] Sparse Matrix-Vector Multiplication (SpMV) is a common and critical calculation that plays an important supporting role in the fields of artificial intelligence, graph computing, and data mining. In the field of artificial intelligence, a large number of SpMV calculations are involved in machine learning algorithms such as convolutional neural networks and deep neural networks. Improving the computational efficiency of SpMV can effectively shorten the model training time and reduce time costs. Existing smart chips can provide a good foundation for parallel computing for SpMV algorithms. Therefore, it is of great significance to parallel optimize the SpMV algorithm on existing smart chips.

[0003] Domestic independent smart chips have a complete technology chain. However, the current optimization of mathematical libraries for domestic smart chip platforms is mainly focused on the Basic Linear Algebra Subroutines (BLAS) and the Automatic Tuning Linear Algebra Software Library (ATLA). As an emerging high-performance computing processor manufacturer, domestic smart chips have not yet released customized function libraries, but related research is underway. Although it is not currently possible to optimize domestic smart chips by calling high-performance computing libraries specifically for hardware characteristics, developers can use their open source vector instruction sets to rewrite high-level language programs to convert scalar calculations into more efficient vector instructions, thereby significantly improving computing performance.

[0004] However, simply rewriting it into instructions does not necessarily perform better than the two-level optimization method of first optimizing the algorithm of the high-level language program and then converting the scalar calculation part into vector instructions. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a sparse matrix-vector multiplication optimization method for domestic smart chips, improves the storage strategy and parallel computing strategy of the algorithm, and at the same time takes advantage of the hardware characteristics of domestic smart chips to effectively utilize their vector instruction set to further optimize the algorithm.

[0006] This method includes the following steps:

[0007] The storage strategy optimization described in the present invention includes the following steps:

[0008] First, obtain the sparse matrix in the memory and divide the obtained sparse matrix into blocks.

[0009] The block height of the sparse matrix block is obtained by dividing the bit width supported by the processor vector instruction by the length of the value type in the sparse matrix.

[0010] For the block width, the present invention considers the non-zero elements in the matrix to select a suitable block width. First, the block width x is randomly initialized, and the vector instruction length of the domestic smart chip needs to be obtained. If the block width is greater than the vector instruction length of the domestic smart chip, the block width x is adjusted to the average number of non-zero elements per row. Then the cache line size and the single instruction multiple data (SIMD) register width of the operating system are obtained. If x is greater than the cache line size and x is an even number, in order to adapt to a larger sparse matrix, the block width is shifted left by one bit. In order to comply with data alignment and avoid additional memory access, if x is greater than the cache line size and an odd number, x is first incremented by 1 and then shifted left by one bit. If x is less than the cache line size and greater than the single instruction multiple data (SIMD) register width, x is aligned downward to the single instruction multiple data (SIMD) register width. If x is less than the cache line size and less than the single instruction multiple data SIMD register width, x is rounded up to ten.

[0011] Read the row offset array, column array, and value array of a sparse matrix.

[0012] According to the read row offset array, column array and value array, the row offset array, column array and value array are aligned and stored in the sparse matrix structure to obtain a three-dimensional representation of the sparse matrix.

[0013] And overload the addition and subtraction operators for the resulting three-dimensional representation of the sparse matrix.

[0014] The parallel computing strategy optimization of the present invention includes the following steps:

[0015] After determining the block size, each block is a computing task. The number of computing tasks for the entire program is determined. Based on the three-dimensional representation of the overloaded sparse matrix, the computing tasks are evenly distributed to each thread to achieve multi-threaded load balancing.

[0016] While using multi-threaded computing, use system monitoring tools in domestic smart chips such as vmstat and top instructions to confirm the bottleneck type, and then use performance analysis tools in domestic smart chips such as perf instructions to analyze the SpMV hotspot function of the bottleneck type in the process, and use assembly-level performance analysis instructions in domestic smart chips such as perfannotate instructions to obtain assembly-level SpMV hotspot functions.

[0017] Assembly-level fused multiplication-addition vector instructions are used to replace the multiplication and addition in the SpMV hotspot function, and the assembly load instruction is overloaded to optimize the loading of matrix data.

[0018] Finally, the computing tasks of each thread are integrated to complete the calculation of SpMV.

[0019] Beneficial effects of the present invention:

[0020] The average inference time of sparse matrix-vector multiplication is optimized for domestic smart chips by optimizing storage strategies, parallel computing frameworks, and vector instructions. The computing performance of sparse matrix-vector multiplication in high sparsity conditions is improved from tens of thousands to tens of billions by targeting the hardware characteristics of domestic smart chips. The inference performance when using sparse matrix-vector multiplication instead of the YOLOv5 convolution operator is compared, and the inference performance is greatly improved when inferring power line images of different orders of magnitude. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is the optimization flow chart of sparse matrix-vector multiplication based on Loongson 3A5000;

[0022] Figure 2 This is the Loongson independent instruction set architecture diagram;

[0023] Figure 3 This is the register structure diagram of Loongson 3A5000;

[0024] Figure 4 Store instance graphs in CSR2 format;

[0025] Figure 5 This is a sample diagram of multi-threaded load balancing;

[0026] Figure 6 Flowchart of the vector instruction replacement algorithm;

[0027] Figure 7 This is a performance comparison chart of sparse matrix SpMV algorithms within the 10,000-level scale (10 iterations);

[0028] Figure 8 This is a performance comparison chart of sparse matrix SpMV algorithms in the billion-level scale (10 iterations);

[0029] Fig. 9 This is a performance comparison chart of sparse matrix SpMV algorithms in the tens of billions of scale (10 iterations);

[0030] Fig.10 The performance comparison chart of the sparse matrix SpMV algorithm in ascending order of the number of non-zero elements (10 iterations);

[0031] Fig.11 Comparison of inference time for different sets of image numbers. DETAILED DESCRIPTION

[0032] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings in the implementation examples of the present invention. The implementation described is only a part of the embodiments of the present invention and cannot cover all implementation examples. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0033] The present invention realizes a sparse matrix-vector multiplication optimization method for a domestically produced intelligent chip.

[0034] Compared with traditional SpMV, the present invention optimizes the average reasoning time of SpMV for domestic smart chips by optimizing storage strategies, parallel computing frameworks, and vector instructions. Figure 1 This embodiment takes Loongson 3A5000 as an example. Figure 2 and Figure 3 shown.

[0035] First, a sparse matrix is ​​read and divided into blocks. For a given sparse matrix, the block size has a great influence on the performance of the algorithm program. If the block is too small, the cache will be insufficiently utilized, and more memory access times will be generated, affecting the program performance; and if the block is too large, the number of generated blocks will be insufficient, the number of parallel threads will be insufficient, and the functions of all processing units cannot be brought into play. For this purpose, the present invention designs a block size selection algorithm, and for a given sparse matrix, a more appropriate block size can be selected.

[0036] For the height of the block, the bit width supported by the processor vector instruction is divided by the length of the numerical type in the matrix. The situation on the LoongArch lasx instruction set (256-bit width) is shown in Table 1.

[0037] Table 1 Matrix block height-data type correspondence table

[0038]

[0039] For the width of the blocks, the present invention selects a suitable block width by considering the non-zero elements in the matrix. The specific process is as follows:

[0040] The initial width of the sparse matrix block width is set to the width of the sparse matrix. If the initial width is less than the vector instruction length of the domestic smart chip, the block width will not be adjusted. If it is greater than the vector instruction length of the domestic smart chip, the following steps will be performed:

[0041] Calculate the average number of non-zero elements per row as the initial width.

[0042] Get the operating system's cache line size.

[0043] If the initial width is larger than the cache line size and is an even number, in order to accommodate a larger sparse matrix, the block width is shifted left by one position. In order to comply with data alignment and avoid additional memory access, if the block width is larger than the cache line size and is an odd number, the block width is first incremented by 1 and then shifted left by one position.

[0044] Then get the Single Instruction Multiple Data (SIMD) register width.

[0045] If the initial width is smaller than the cache line size and larger than the SIMD register width, the tile width is aligned down to the SIMD register width.

[0046] First, the row offset array, column array, and value array of the read sparse matrix file are aligned using the structure storage form. The structure is used to simulate the 256-bit state of the register, and the input data is aligned. The sparse matrix is ​​stored by aligning the row offset array, column array, and value array. Operator overloading is used to overload the 256-bit addition and multiplication in the structure. The specific algorithm is shown in Algorithm 2. This algorithm defines aligned storage to implement CSR2 format storage of the matrix. The CSR2 format is as follows Figure 4 As shown, the entire sparse matrix is ​​saved through the row offset array, column array and value array of the sparse matrix, and the addition and subtraction operators are reloaded on the aligned structure.

[0047] The specific process of optimizing the parallel computing framework described in the present invention is to determine the size of the blocks, each block is a computing task, and the number of computing tasks of the entire algorithm program is determined. The present invention adopts the method of evenly distributing the computing tasks to each thread to achieve multi-thread load balancing, and uses the parallel computing framework OpenMP, such as Figure 5 As shown, each block computing task is assigned to a thread for computing and then the computing results of each thread are merged.

[0048] The vector instruction optimization is to use vmstat and top commands to confirm the bottleneck type while using multi-threaded calculations, then use perf to analyze the function hotspots in the process, and finally use perf annotate to analyze the hotspots at the assembly level. After finding the calculation hotspots, the instruction-level optimization can improve the performance of the algorithm. Through the performance bottleneck analysis of the algorithm implemented in the top-level C language, it is found that the calculation hotspots exist in addition and multiplication operations, so vector instructions are used to implement addition and multiplication operations respectively to improve the calculation performance. The specific replacement strategy for vector instructions is as follows: Figure 6 As shown, the core code description is shown in Algorithm 3.

[0049] The assembly fusion multiply-add vector instruction is shown in Algorithm 2:

[0050] The first step is to use the fetch instruction to fetch the double-precision floating-point vector in memory address one into vector register one, and repeat the operation to fetch the double-precision floating-point vector in memory address two into vector register two; the second step is to perform element-level multiplication instructions on the vectors in register one and register two, and store the result vector in vector register four; the third step is to use the fetch instruction to fetch the double-precision floating-point vector in memory address three into vector register four; the fourth step is to perform element-level addition instructions on the vectors in registers three and four, and store the result in register two; finally, write the vector in vector register two back to memory address two. Implement floating-point vector multiplication and addition instructions, and finally, replace the multiplication and addition in the hotspot function.

[0051] Algorithm 1: Assembly fused multiply-add vector instruction algorithm example

[0052]

[0053] Finally, vector instructions were used to replace the two basic algorithms involved in sparse matrix-vector multiplication. At the same time, in order to improve the loading performance, the method of loading the matrix was optimized through instructions. The load assembly code was reloaded to load the matrix through a single instruction multiple data stream. The process is as follows:

[0054] First, four double-precision floating-point numbers are taken out from the memory through the data fetch instruction and stored in four floating-point registers; then, the floating-point numbers in the floating-point registers are inserted into the vector register as offsets in the order in which they are obtained; finally, the data in the vector register is returned to the memory to realize parallel loading of matrix data.

[0055] In order to verify the performance of the SpMV optimization strategy proposed in this invention on domestic smart chips, a series of experiments were carried out using Loongson 3A5000 to demonstrate the superiority of the SpMV optimization strategy over traditional methods.

[0056] For the general algorithm SpMV, the classic sparse matrix calculations involved in non-artificial intelligence fields and model reasoning in the artificial intelligence field are two major aspects of measuring algorithm performance. In the sparse matrix calculations in non-artificial intelligence fields, researchers have designed many representative sparse matrices to measure algorithm efficiency and computing performance. These sparse matrices have different scales and sparsity (the proportion of non-zero elements in the matrix) and different numbers of non-zero rows in the matrix, which can well measure the efficiency and computing performance of sparse matrix algorithms.

[0057] The hardware parameters during the experiment are shown in Table 2.

[0058] Table 2 Experimental hardware configuration

[0059]

[0060] The experimental scheme is shown in Table 3.

[0061] Table 3 Experimental plan

[0062]

[0063] The task compares the performance of the four schemes and the Base scheme, and compares the performance of sparse matrix-vector multiplication when different optimization methods are used. The performance of regular and irregular matrices with different scales and different sparsities is tested. The parameters of the sparse matrices are shown in Tables 4, 5, and 6.

[0064] Table 4 Parameters of classic regular and irregular sparse matrices

[0065]

[0066] Table 5 Parameters of irregular sparse matrix within 100 million level

[0067]

[0068]

[0069] Table 6 Parameters of irregular sparse matrix in the tens of billions level

[0070]

[0071] In order to verify the superiority of the present invention in different scenarios, sparse matrices of different scales and different sparsities are used to simulate sparse matrix-vector multiplication operations in various situations in high-performance computing. Based on the Loongson smart chip, experiments and performance analysis were carried out for irregular sparse matrices of different scales and sparsities and regular matrices of different scales. The specific experimental design is as follows:

[0072] Experiments were conducted on sparse matrices of different sizes, with matrix sparsity ranging from 0.0004996% to 24.52%. Considering that running fewer times may cause cache problems, resulting in faster reading of the same sparse matrix, different types and sizes of matrices were tested with 10, 100, and 1000 iterations and the average was taken to minimize the impact of cache.

[0073] The core indicator for evaluating experimental results is the average calculation time Tmean, which is calculated as follows:

[0074] Using the average computation time can reduce the impact of chip frequency fluctuations and better measure the performance of the optimization strategy.

[0075]

[0076] T mean =T / iteration (2)

[0077] This experiment designed multiple comparative tests on the performance changes of sparse matrices of different sizes and different numbers of iterations for four SpMV algorithm optimization methods. The performance differences of sparse matrices of 10,000, 1 million, 100 million, 10 billion and regular sparse matrices under 10 iterative calculations were compared. In addition, the performance differences of sparse matrices in ascending order of the number of non-zero elements under 10 iterative calculations were compared.

[0078] Figure 7 What is shown is the performance comparison of 10 iterations of sparse matrices in 4 modes at a scale of 10,000. The performance analysis of the SpMV algorithm with different sparsities at a scale of 10,000 using 10 iterations shows that the average calculation time of the unoptimized sparse matrix-vector multiplication is much lower than that of the parallel computing framework OPENMP. Since there is synchronization overhead between the threads of the parallel computing framework, the main thread must wait for other threads to finish before continuing to execute. Therefore, when the amount of calculation is small, the thread overhead has a greater impact on the average calculation time. The CSR2_S algorithm, which uses the computing program of the optimized storage strategy and parallel computing strategy of the present invention, achieves the best performance. Since the computing performance of vector instructions is better than that of scalar computing instructions, CSR2_S is the best in terms of the average calculation time of sparse matrices at a scale of 10,000, while the performance of the parallel computing framework CSR2_C_MP is the worst.

[0079] Figure 8 The performance comparison of 10 iterations of sparse matrices in the billion-scale under four modes is shown. The performance of the four modes is analyzed when the sparsity is less than 2% in the billion-scale. The performance of each matrix calculation using the CSR2_S method is the best, and the performance of the Baseline method is the worst. Due to the further reduction of sparsity, the improvement in computing efficiency brought by the parallel computing framework has offset the thread overhead.

[0080] Fig. 9The performance comparison of sparse matrices in the tens of billions of scale in 4 modes after 10 iterations is shown. The performance of the four calculation methods of sparse matrices in the tens of billions of scale is analyzed and compared. The strategy using the parallel computing framework and the strategy using the CSR2_S are better than the Baseline method. Since the Baseline method performs calculations in a single-threaded manner, the performance gradually decreases as the scale of the sparse matrix increases. The scalar calculation method using the optimized storage strategy also has poor performance, and the performance on some matrices such as flickr is even worse than the Baseline method. Since scalar calculations essentially require the calculation of values ​​one by one, while vector calculations can perform addition and multiplication on multiple numbers at once, the performance is significantly ahead.

[0081] Fig.10 This is a performance line graph of 10 iterations of sparse matrix multiplication by vector operation in four modes in ascending order of the number of non-zero elements. It can be clearly seen from the line graph that the original 47 sparse matrices are numbered from 1 to 47, and ten iterations are performed in the order of increasing non-zero elements. The experimental results show that the more non-zero elements there are, the lower the sparsity, the longer the average calculation time, and the performance of the CSR2_S algorithm is better when the number of non-zero elements is low. As the number of non-zero elements increases, its performance is basically the same as the OpenMP algorithm. The following conclusion can be drawn: As the number of non-zero elements increases, the operation program using the CSR2 optimization method takes significantly longer than the other three methods.

[0082] In summary, from the test results, we can find that the CSR2 optimization method of embedded Loongson vector instructions can achieve different speedup ratios in different scale and sparsity matrices, and the effect is particularly prominent in small matrices.

[0083] In order to verify the SpMV inference effect of the present invention on a real power line detection dataset, the following experiments are designed:

[0084] The experimental configuration table is shown in Table 7.

[0085] Table 7 Experimental configuration table

[0086]

[0087] The distribution of the dataset is shown in Table 8.

[0088] Table 8 Dataset distribution table

[0089]

[0090] The YOLOv5 algorithm is used to train the power line detection dataset on the TensorFlow training framework to obtain the weight file, and the weight file is converted into the intermediate representation format tmfile through the Tengine framework, and finally inference is performed through the Loongson 3A5000 device operator. At the same time, the convolution operation is converted into a matrix-vector multiplication operation. And as a replacement operator for the convolution operation in the inference calculation, the performance of the two algorithms is compared. The experiment uses the power line dataset as input and tests the average inference time of the two inference methods when inferring different orders of magnitude of 10, 100, and 1000 pictures.

[0091] The experimental evaluation index is the average calculation time Tmean, as shown in formulas 1 and 2.

[0092] 10, 100, and 1000 images were randomly selected from the power line test set. The Baseline method was used to test the average inference time of images at three orders of magnitude, and SpMV was used to replace the Conv operator to test the average inference time at three orders of magnitude. Under the Baseline method, the average inference time at three different orders of magnitude was 157.02ms, 152.93ms, and 155.365ms, respectively. The average inference time using the SpMV algorithm to replace the convolution calculation was 139.23ms, 141.22ms, and 131.26ms, respectively. Excluding the influence of individual image features, the inference time at three orders of magnitude was reduced by 11.3%, 7.657%, and 15.51%, respectively. By optimizing the original convolution operator using sparse matrix multiplication, the inference performance of domestic smart chips can be effectively improved. The experimental results are as follows Fig.11 As shown in the figure, the optimized SpMV can significantly reduce the inference time of domestic smart chips.

[0093] Although embodiments of the present invention have been shown, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A sparse matrix-vector multiplication optimization method for a domestic smart chip, characterized in that: The following steps are involved: S1. Obtain a sparse matrix in memory and divide the obtained sparse matrix into blocks; S2, read the row offset array, column array and value array of the sparse matrix, and align and store the row offset array, column array and value array in the sparse matrix structure to obtain a three-dimensional representation of the sparse matrix; S3, overloading the addition and subtraction operators for the obtained three-dimensional representation of the sparse matrix; S4, based on the three-dimensional representation of the overloaded sparse matrix, the computing task is evenly distributed to each thread to achieve multi-thread load balancing; S5. Integrate the computing tasks of each thread to complete the calculation of SpMV.

2. The sparse matrix vector multiplication optimization method for a domestic smart chip according to claim 1, characterized in that: The obtained sparse matrix is ​​divided into blocks, and the height and width of the blocks are as follows: Block height: The block height of the sparse matrix block is obtained by dividing the bit width supported by the processor vector instruction by the length of the value type in the sparse matrix; Block width: First, randomly initialize the block width x and obtain the vector instruction length of the domestic smart chip. If the block width is greater than the vector instruction length of the domestic smart chip, the block width x is adjusted to the average number of non-zero elements in each row; then obtain the cache line size and the single instruction multiple data SIMD register width of the operating system. If x is greater than the cache line size and x is an even number, shift the block width left by one bit; if x is greater than the cache line size and an odd number, first increase x by 1 and then shift it left by one bit; if x is less than the cache line size and greater than the single instruction multiple data SIMD register width, align x downward to the single instruction multiple data SIMD register width; if x is less than the cache line size and less than the single instruction multiple data SIMD register width, round x up to ten.

3. The sparse matrix vector multiplication optimization method for a domestic smart chip according to claim 2 is characterized in that: The specific implementation process of step S4 is as follows: After determining the block size, each block is a computing task. Determine the number of computing tasks for the entire program and evenly distribute the computing tasks to each thread to achieve multi-thread load balancing. While using multi-threaded computing, use the system monitoring tool in the domestic smart chip to confirm the bottleneck type, then use the performance analysis tool in the domestic smart chip to analyze the SpMV hotspot function of the bottleneck type in the process, and use the assembly-level performance analysis instructions in the domestic smart chip to obtain the assembly-level SpMV hotspot function; Assembly-level fused multiplication-addition vector instructions are used to replace the multiplication and addition in the SpMV hotspot function, and the assembly load instruction is overloaded to optimize the loading of matrix data.

4. The sparse matrix vector multiplication optimization method for a domestic smart chip according to claim 3 is characterized in that: The specific implementation process of the assembly fusion multiply-add vector instruction is as follows: Use the fetch instruction to fetch the double-precision floating-point vector in memory address one into vector register one, and repeat the operation to fetch the double-precision floating-point vector in memory address two into vector register two; Perform an element-wise multiplication instruction on the vectors in register one and register two, and store the result vector in vector register four; Use the fetch instruction to fetch the double-precision floating-point vector at memory address three into vector register four; Perform an element-wise addition instruction on the vectors in register three and register four, and store the result in register two; Write the vector in vector register two back to memory address two to implement floating-point vector multiplication and addition instructions, replacing the multiplication and addition in the hotspot function.

5. The sparse matrix vector multiplication optimization method for a domestic smart chip according to claim 3 is characterized in that: The specific implementation process of reloading the assembly load instruction is as follows: Four double-precision floating-point numbers are taken out from the memory through the data fetch instruction and stored in four floating-point registers; then, the floating-point numbers in the floating-point registers are inserted into the vector register as offsets in the order in which they are obtained; finally, the data in the vector register is returned to the memory to realize parallel loading of matrix data.

Images