Winograd convolution optimization method and system suitable for ARMv8 multi-core architecture

Abstract

The invention discloses a Winograd convolution optimization method and system suitable for an ARMv8 multi-core architecture, and belongs to the technical field of deep learning. The method comprises the following steps: determining a tile block value Tblk, an input channel block value Cblk and an output channel block value Kblk, and dividing input data according to the tile block value Tblk to obtain tile blocks; global domain conversion of the filter is completed through double-layer nested loop, and a global domain conversion result is stored in a FilterOut array according to a first matrix multiplication data layout; and traversing the tile blocks according to the tile block value Tblk, and performing in-block input conversion, coupled matrix multiplication and output conversion on each tile block to obtain a Winograd convolution optimization result. According to the method, the calculation efficiency of the Winograd algorithm can be greatly improved.

Description

Technical Field

[0001] The present application belongs to the field of deep learning technology, and specifically relates to a Winograd convolution optimization method and system suitable for ARMv8 multi-core architecture. Background Art

[0002] Winograd convolution uses the Winograd minimum filtering algorithm to reduce the inherent computational complexity of convolution operations by dividing the input data into multiple tiles and mapping them from the spatial domain to the Winograd domain. This algorithm reduces the number of arithmetic operations required, thereby improving the computational efficiency of convolution operations. Taking two-dimensional convolution as an example, F(m×m, r×r) means using an r×r convolution kernel to calculate m×m outputs, where the input tile size is m+r-1. Assume and are the row and column coordinates of the input tile, b is the batch index, and the output element corresponding to the b-th batch and the k-th output channel It can be expressed as

[0003]

[0004] Here ⊙ represents element-by-element multiplication; B, G, A are the transformation matrices of input, filter, and output, respectively, which can be generated using the Chinese Remainder Theorem; g = F k,c , that is, the element value of the filter tensor F at the kth output channel and cth input channel; d is the row and column coordinates of the input tile in the bth batch and the cth input channel The element value of U k,c and V b,c They are the filter tile and input tile (corresponding to the bth batch, kth output channel and cth input channel) mapped to the Winograd domain after multiplication with the transformation matrix. One technical solution for Winograd convolution is to perform element-by-element multiplication of the input and filter data in the Winograd domain after completing the domain conversion.

[0005] Element-by-element multiplication belongs to level-1 BLAS operation. In order to improve the arithmetic intensity, the original formula can be rewritten as follows:

[0006]

[0007] Where (x, y) represents the coordinates of the elements involved in the element-by-element multiplication, and ξ is the original coordinate represents the convolution result of the Winograd domain. This formula can be regarded as a matrix multiplication operation.

[0008] At present, based on Winograd convolution, it can be divided into three stages: conversion of input and convolution kernel, matrix multiplication, and inverse conversion of output results. For example, the existing implementation scheme disclosed by Zhen Jia et al. ① (Zhen Jia, Aleksandar Zlateski, Fredo Durand, and Kai Li. 2018. Optimizing N-dimensional, winograd-based convolution for manycore CPUs. In Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 109–123.), and the existing implementation scheme disclosed by Xueying Wang et al. ② (Xueying Wang, Guangli Li, Zhen Jia, Xiaobing Feng, and Yida Wang. 2024. Fast convolution meets low precision: Exploring efficient quantized Winograd convolution on modern CPUs. ACM Transactions on Architecture and Code Optimization 21, 1 (2024), 1–26.). However, in the two existing implementations, the data layout after conversion only considers blocking and vectorization, and cannot guarantee continuous memory access for matrix multiplication. Matrix multiplication only considers optimizing cache blocking, without optimizing microkernel configuration. The conversion stage uses the same method for different Winograd tile sizes, without considering register reuse for overlapping parts between tiles.

[0009] In summary, the shortcomings of the existing technical solutions mainly include the following aspects:

[0010] (1) Using a non-fused model to perform Winograd convolution, the input conversion, calculation, and output conversion are processed as three independent stages, or only some of the stages are coupled, which cannot fully utilize cache locality.

[0011] (2) Matrix multiplication is used for Winograd domain calculation. Since cross-row memory access introduces additional domain conversion overhead, the intrinsic method used in the prior art to implement domain conversion cannot fully reduce the conversion overhead.

[0012] (3) The data layout of existing Winograd domain computation based on matrix multiplication only considers block partitioning, which limits performance.

[0013] (4) The adopted parallel strategy is not optimized for specific layers in the network, resulting in low parallel efficiency in some layers. Summary of the invention

[0014] In view of the technical problems existing in the implementation of existing Winograd convolution on ARM, such as insufficient utilization of cache locality, additional overhead introduced by strided memory access, and insufficient refinement of parallel strategy, the present invention discloses a Winograd convolution optimization method and system suitable for ARMv8 multi-core architecture based on the data characteristics of common convolutional neural network models, which can greatly improve the computational efficiency of the Winograd algorithm.

[0015] To achieve the above objectives, the technical solution of the present invention includes the following contents.

[0016] A Winograd convolution optimization method suitable for ARMv8 multi-core architecture, the method comprising:

[0017] Determine the tile value T blk , input channel block value C blk And the output channel block value K blk , and according to the tile block value T blk Divide the input data into tiles;

[0018] The overall domain conversion of the filter is completed through a double nested loop, and the overall domain conversion result is stored in the FilterOut array according to the first matrix multiplication data layout;

[0019] By tile value T blk Traversing the tile blocks, performing intra-block input conversion, coupled matrix multiplication and output conversion on each tile block, and obtaining a Winograd convolution optimization result; wherein, performing intra-block input conversion, coupled matrix multiplication and output conversion on each tile block, and obtaining a Winograd convolution optimization result, includes:

[0020] Block value C by input channel blk Loop through the loop, and in each traversal process, call the input conversion core to complete the input domain conversion, and store it into the TransInOut temporary array according to the second matrix multiplication data layout;

[0021] Block value K by output channel blk Loop through, and in each traversal, first perform L matrix multiplications, each matrix multiplication is based on the input channel block value C blkCall GEMM to check the TransInOut temporary array T for the stride traversal blk ×C blk Block and FilterOut arrays in C blk ×K blk The block performs intra-block matrix multiplication and accumulation, and stores the matrix multiplication results in the GEMMOut temporary array according to the data layout; then, the output conversion kernel is called to perform output conversion on the matrix multiplication results in the GEMMOut temporary array to obtain the Winograd convolution optimization result; where L represents the number of elements in a Winograd convolution tile.

[0022] Furthermore, the determination of the tile block value T blk , input channel block value C blk And the output channel block value K blk ,include:

[0023] Minimize data movement overhead based on cache capacity and determine the tile value T blk , input channel block value C blk And the output channel block value K blk , and combined with the characteristics of the input and output channels in the convolutional neural network, the tile block value T blk , input channel block value C blk And the output channel block value K blk To be limited.

[0024] Furthermore, the overall domain conversion of the filter is completed by double-layer nested loops, including:

[0025] The outer loop divides the output channel into blocks K blk Traverse;

[0026] The inner loop divides the input channel into blocks C blk Traverse;

[0027] The filter conversion kernel is called in the loop for iterative conversion; the size of one iterative conversion is L×C blk ×K blk data.

[0028] Furthermore, when the number of row and column elements of the output tile matrix is ​​m=2, the calling of the input conversion kernel to complete the input domain conversion includes:

[0029] Step 4.1: Load L = 16 elements in a Winograd convolution tile into vector registers v0 to v15; where one element represents θ data values ​​across C input channels in a tile loaded into the same vector register, where θ is the number of data contained in a vector register;

[0030] Step 4.2: Left-multiply the Winograd convolution tile corresponding to the L vector registers by the matrix B T , and release vector registers v0, v1, v8, and v9, and combine vector registers v16 to v27 to store the tile left multiplication matrix B T The temporary result of ; where B represents the input transformation matrix;

[0031] Step 4.3: Multiply the tile by matrix B T The temporary result is right-multiplied by matrix B and stored in vector registers v16 to v31;

[0032] Step 4.4: Store the results in vector register v16 to vector register v31 into the TransInOut temporary array;

[0033] Step 4.5: After loading the non-overlapping data in the next Winograd convolution tile into vector registers v0, v1, v4, v5, v8, v9, v12, and v13, execute steps 4.2 to 4.4, and store the result corresponding to the next Winograd convolution tile into the TransInOut temporary array;

[0034] Step 4.6: Continue to load the non-overlapping data of the next Winograd convolution tile into vector registers v2, v3, v6, v7, v10, v11, v14, and v15, then execute steps 4.2 to 4.4, and store the result corresponding to the next Winograd convolution tile to be loaded into the TransInOut temporary array;

[0035] Step 4.7: Return to step 4.5 until all Winograd convolution tiles are processed.

[0036] Furthermore, when the number of row and column elements of the output tile matrix is ​​m=6, the calling of the input conversion core to complete the input domain conversion includes:

[0037] Perform d×B conversion on the i-th row of data in a Winograd convolution tile and get the result tmp i ; Wherein, the result tmp i Transpose Calculate the factor f0 = d (i,3) +d (i,7) -4.25×d (i,4) , calculate the factor f1 = d (i,2) -4.25×d (i,4) +d (i,6) , calculation factor f2 = 1.25 × d (i,5) , calculation factor f3 = 2.5 × d (i,4) , calculation factor f4 = 0.25 × d (i,3) -f2+d (i,7) , calculation factor f5 = 0.5 × d (i,2) -f3+2×d (i,6) , calculate the factor f6 = 4 × (d (i,3) -f2)+d (i,7) , calculate the factor f7 = 2 × d (i,2) -f3+0.5×d (i,6) , d represents the input tile matrix for input domain conversion calculation, d (i,n) represents the element in the i-th row and n-th column of the input tile, where n ranges from 1 to 8, and B represents the input transformation matrix;

[0038] The result tmp i As a row of data in the temporary array tmp of 8×8×2θ, θ represents the number of data contained in a vector register, and 2θ is the number of input channels across which a row of data is processed in one iteration;

[0039] Multiply the temporary array tmp by matrix B column by column T and store the corresponding results in the TransInOut temporary array.

[0040] Furthermore, the first matrix multiplication data layout is [K / K blk ][l][C / C blk ][K blk / η][C blk ][η], the second matrix multiplication data layout is [L][C / C blk ][T blk / α][C blk / θ][α][θ], the data layout of the matrix multiplication result is [L][K blk / η][T blk / α][η / θ][α][θ]; where K represents the number of output channels of the convolutional layer, C represents the number of input channels of the convolutional layer, α and η represent the parameters of the matrix multiplication microkernel, and θ represents the number of floating-point numbers stored in the vector register.

[0041] 7. The method according to claim 6, characterized in that the method further comprises:

[0042] When the number of row and column elements of the output tile matrix is ​​m=2, the input conversion kernel first traverses the image width W of the convolutional network input tensor, and then to the image height H until the tile block value T is completed. blk Transformation of θ elements of all tiles in the input channel direction C; finally traverse the input channel direction;

[0043] When the number of row and column elements of the output tile matrix is ​​m=6, the traversal order of the input conversion kernel is to first complete the overall conversion of a tile in the input channel direction, and then traverse the image width W and height H directions of the convolution network input tensor to convert all tiles;

[0044] The iteration method in the filter conversion core is to load the value loaded into the same vector register across the output channels K and follow In-order traversal.

[0045] Furthermore, the implementation process of the matrix multiplication microkernel includes:

[0046] Based on the constraints of parameter α and parameter η, determine the optimal microkernel parameter α1 and parameter η1 that maximizes the computation-to-memory ratio; wherein the constraints include: 2α+η / 2+α×η / 4≤32 and η%4=0;

[0047] According to the characteristics of the convolutional neural network, the parameters α2 and η2 corresponding to the suboptimal microkernel are selected;

[0048] According to the parameters α1 and η1 of the selected optimal microkernel and the parameters α2 and η2 corresponding to the suboptimal microkernel, a ping-pong strategy is adopted to implement the operation of the matrix multiplication microkernel.

[0049] Furthermore, the method further comprises:

[0050] In the shallow convolutional layer of the convolutional neural network model, OpenMP is used to couple the outermost layer of the Winograd algorithm with tiles T blk Parallelize the loop that iterates through the tiles for the stride, and set the maximum number of threads to Where T represents the total number of tiles divided by the input tensor;

[0051] In the deep convolutional layer of the convolutional neural network model, set the tile value T blk = T, and use OpenMP technology to parallelize the C and K dimensions, setting the maximum number of threads to Among them, C represents the number of input channels of the convolutional layer, and K represents the number of output channels of the convolutional layer;

[0052] In the middle convolution layer of the convolutional neural network model, Pthreads thread pool technology and lock-free task queues are used for multi-dimensional parallelism, and atomic snapshots are used to complete the synchronization between stages; among them, T-dimensional parallelization is the initial subtask of the task queue. Each T-dimensional subtask pushes C and K-dimensional subtasks into the queue to achieve parallelization. The total number of subtasks is The maximum number of threads is empirically set to

[0053] A Winograd convolution optimization system suitable for ARMv8 multi-core architecture, the system comprising:

[0054] The tile value determination module is used to determine the tile value T blk , input channel block value C blk And the output channel block value K blk , and according to the tile block value T blk Divide the input data into tiles;

[0055] A filter conversion module, used to complete the overall domain conversion of the filter through a double nested loop, and store the overall domain conversion result into the FilterOut array according to the first matrix multiplication data layout;

[0056] Winograd convolution optimization module for tile-based block value T blk Traversing the tile blocks, performing intra-block input conversion, coupled matrix multiplication and output conversion on each tile block, and obtaining a Winograd convolution optimization result; wherein, performing intra-block input conversion, coupled matrix multiplication and output conversion on each tile block, and obtaining a Winograd convolution optimization result, includes:

[0057] Block value C by input channel blk Loop through the loop, and in each traversal process, call the input conversion core to complete the input domain conversion, and store it into the TransInOut temporary array according to the second matrix multiplication data layout;

[0058] Block value K by output channel blk Loop through, and in each traversal, first perform L matrix multiplications, each matrix multiplication is based on the input channel block value C blk Call GEMM to check the TransInOut temporary array T for the stride traversal blk ×C blk Block and FilterOut arrays in C blk ×K blkThe block performs intra-block matrix multiplication and accumulation, and stores the matrix multiplication results in the GEMMOut temporary array according to the data layout; then, the output conversion kernel is called to perform output conversion on the matrix multiplication results in the GEMMOut temporary array to obtain the Winograd convolution optimization result; where L represents the number of elements in a Winograd convolution tile.

[0059] Compared with the prior art, the present invention has at least the following beneficial effects.

[0060] 1. The present invention improves the cache locality of the convolution calculation process by coupling the three stages of the Winograd algorithm and making full use of the three-level cache of the ARM architecture.

[0061] 2. The present invention combines the conversion of input data, output data and filter data in the spatial domain and Winograd domain with the data layout designed for the matrix multiplication calculation process, thereby alleviating the stride memory access overhead caused by converting data into the GEMM format.

[0062] 3. The present invention carefully designs a matrix multiplication microkernel, utilizes ping-pong technology and customized data layout, maintains continuous memory access during the calculation process, and overlaps data loading and calculation, which greatly improves the calculation efficiency.

[0063] 4. The present invention achieves the effect of higher parallel efficiency of each layer of convolution by adapting the parallel mode according to the data features of different layers of the convolutional network.

[0064] 5. The present invention improves the overall computational efficiency of Winograd convolution, and obtains test results that are superior to those of the STOA library in both single-core and multi-core tests. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is the algorithm framework diagram of the present invention.

[0066] Figure 2 This is an overview of the entire process of processing a Winograd tile convolution.

[0067] Figure 3 This is a schematic diagram of the assembly-level register multiplexing optimization method for input conversion when m=2.

[0068] Figure 4 It is a schematic diagram of the input data layout for continuous memory access.

[0069] Figure 5 This is a schematic diagram of the filter data layout for continuous memory access.

[0070] Figure 6 This is a diagram of the arrangement of the microkernel (4,16) vector registers.

[0071] Figure 7 It is the microkernel (7,8) vector register arrangement.

[0072] Figure 8 This is a comparison of the single-core convolution running time between the present invention and the prior art.

[0073] Fig. 9 This is a comparison of the running time of 16-core and 32-core convolutions of the present invention and the prior art.

[0074] Fig.10 It is a flow chart of further integrating matrix multiplication and output conversion steps in the present invention. DETAILED DESCRIPTION

[0075] The present invention is further described in detail below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention but not to limit the scope of the present invention.

[0076] The present invention provides a convolutional neural network optimization method based on Winograd algorithm. Figure 1 The figure shows the overall algorithm flow of the present invention. This is a fusion block method to implement Winograd convolution by coupling the input conversion, matrix multiplication and output conversion stages; first, the three stages are coupled by blocking the tiles, and then the matrix multiplication and output conversion are further coupled by blocking the number of output channels K. The input and output of the algorithm and the corresponding data storage format are consistent with general convolution ( Figure 1 The input and output of the _D_PAGE_STRING_Tensor specify the storage format of the input tensor, filter tensor, and output tensor).

[0077] The specific steps of the fusion block strategy are described as follows:

[0078] 1) Based on the cache capacity, a heuristic method is used to determine the tile size, including the tile value T blk , input channel block value C blk And the output channel block value K blk .

[0079] The heuristic method for determining the block size in the present invention is to minimize the data movement overhead based on the cache capacity. Considering the characteristics of the input and output channels in the convolutional neural network, Figure 1 K blk and C blk It is configured as an integer multiple of 16 to limit the block value, avoid potential edge cases, and simplify the calculation process.

[0080] 2) To take advantage of Cache locality, a temporary array TransInOut[L×T blk×C] and allocate memory to store the result of converting the input data in the block to the Winograd domain; create a temporary array GEMMOut[L×T blk ×K blk ] and allocate memory to store the results of matrix multiplication operations within the block.

[0081] 3) Through double nested loops, the outer loop is divided into blocks K according to the output channel blk Traversal, the inner loop is divided into blocks C according to the input channel blk Traversal, calling the filter conversion kernel in the loop, the size of one iteration conversion is L×C blk ×K blk The data is converted into the overall domain of the filter and stored in the FilterOut[L×C×K] array according to the matrix multiplication data layout. In the inference-only mode, since the weights of the neural network are pre-trained and will not change, this step can be completed and stored in advance as preprocessing.

[0082] 4) First, divide the tile value T blk Traverse the tile blocks and perform the input transformation within the block and the coupled matrix multiplication and output transformation in sequence:

[0083] 4.1) Divide into blocks C according to input channels blk The loop traverses, calling the input conversion core to complete the input domain conversion and storing it in the temporary array TransInOut in step 2) according to the data layout designed by the present invention; the data size of one iteration conversion is C blk ×T blk .

[0084] 4.2) Divide into blocks K by output channel blk The loop traverses to complete further coupled matrix multiplication and output transformation.

[0085] The process of step 4.2 is as follows Fig.10 As shown, the completed size is T blk The matrix multiplication and output transformation of the tile block. bk is the iteration index value of the output channel K, and each iteration processes a size of L×T blk ×K blk The processing method is as follows:

[0086] a. Loop and perform L matrix multiplications. Each matrix multiplication is divided into blocks C according to the input channel. blk For step size traversal, call the GEMM kernel to perform intra-block matrix multiplication, and the corresponding input matrix is ​​the T of the TransInOut temporary array blk ×C blk Block and FilterOut arrays in C blk ×K blkblock, the accumulated result (T blk ×K blk blocks) into the GEMMOut temporary array.

[0087] Among them, if Winograd domain calculation does not use matrix multiplication but uses element-by-element multiplication (TEWMM), the calculation data can still be packaged into TEWMM continuous memory access data layout, combining packaging with domain conversion.

[0088] b. Call the output conversion kernel on the matrix multiplication result data in the GEMMOut temporary array to perform output conversion.

[0089] 5) Handle tile edge cases.

[0090] In order to precisely control the vector registers and make full use of software prefetching, the present invention uses assembly language to implement the filter transformation core of step 3, the input transformation core of step 4.1, the GEMM core of step 4.2a, and the output transformation core of step 4.2b.

[0091] Figure 2 As shown in FIG. 1 , the process of processing a tile of input data in the present invention includes three stages of Winograd convolution (filter / input conversion, matrix multiplication and output conversion). The gray rectangular blocks are the input, filter data blocks and output data to be processed, and the darkened small blocks are loaded into the same vector register and processed simultaneously. Figure 2 The meanings of important symbols are as follows:

[0092] θ: The number of floating point numbers that can be stored in a vector register. For ARMv8 architecture processors, vector registers are 128-bit, so single-precision floating point numbers θ = 4 and double-precision floating point numbers θ = 2.

[0093] L: represents the number of elements in a Winograd convolution tile, that is, L = (m+r-1)×(m+r-1).

[0094] T: Total number of tiles.

[0095] (α,η): Parameters of the matrix multiplication microkernel, each time the α rows of matrix V are multiplied by the η columns of matrix U, the result matrix is ​​obtained The (α,η) size sub-block.

[0096] The following will describe in detail the optimization strategy designed for the three stages of Winograd convolution in the present invention, as well as the parallel strategy applicable to the algorithm framework integrating the three stages of the present invention.

[0097] 1. Domain conversion

[0098] Due to the influence of different L values ​​on the use of assembly-level vector registers, the present invention uses different methods to handle the two conversions of m=2 and m=6. The main difference between the two methods is whether 32 vector registers can be used to complete the conversion of a tile spanning channel number θ=4 at one time.

[0099] Method 1: When m=2, L=16. The number of vector registers allows 16 inputs and 16 conversion results to be saved across the number of channels θ=4, so the conversion of one tile can be processed at one time. The Winograd convolution input conversion uses the overlap-add (OLA) method, and the data is stored in row-major order. The present invention uses the characteristics of adjacent tiles sharing elements to realize register reuse, significantly reducing the number of elements that need to be loaded, such as Figure 3 shown. Figure 3 Method 1 shows the register arrangement of the input conversion of a single tile using assembly language, where the numbers represent vector register index values, and each vector register contains θ elements. The specific processing steps are as follows:

[0100] a. Initial iteration: load L = 16 elements into vector registers (v0 ~ v15), and the remaining registers are used to store a tile for left multiplication by B during the conversion calculation. T The intermediate result (corresponding to Figure 3 The register index in the upper left rectangle) and the result after the conversion (corresponding to Figure 3 The register index in the upper right rectangle of the figure);

[0101] b. A tile corresponding to vector registers v0~v15 is multiplied by B on the left T , registers v0, v1, v8, and v9 are released to store tile left multiplied by B T The temporary result (corresponding to Figure 3 The last row of register index in the upper left rectangle);

[0102] c. Figure 3 Multiply the upper left rectangle by B T The temporary result is multiplied right by B and stored in vector registers v16~v31;

[0103] d. Store the results in vector registers v16~v31 into the corresponding positions of the temporary array TransInOut in step 4.1);

[0104] e. Load the non-overlapping data in the next tile ( Figure 3 The values ​​of the registers in the dashed box can be reused, so we only need to load the non-overlapping elements of the next tile into vector registers v0, v1, v4, v5, v8, v9, v12, and v13, corresponding to Figure 3 The 8 register indexes to the right of the dashed box in the figure);

[0105] f. Execute steps b, c, and d for the newly loaded tile, and then continue to load the non-overlapping data of the next tile. The difference from step e is that the register index value reused in this iteration is the same as Figure 3 The opposite is true, that is, the values ​​in registers v0, v1, v4, v5, v8, v9, v12, and v13 can be reused, and registers v2, v3, v6, v7, v10, v11, v14, and v15 need to load non-overlapping elements in the next tile. Execute steps b, c, and d for the newly loaded tile, and the way to load non-overlapping elements of the next tile is alternately executed according to steps e and f.

[0106] Method 2: When m=6, L=64, which exceeds the number of available vector registers. The present invention adopts a design that processes one row (8 elements) in one iteration. At this time, the number of channels spanning 2θ can fully utilize the vector registers. In order to continuously access the data stored in row-major order, first perform the d×B step in the input conversion, store the temporary result in a temporary array tmp of size 8×8×8, and then perform left multiplication B on tmp by column. T , store the result in the temporary array TransInOut of step 4.1).

[0107] By taking advantage of the special structure of the transformation matrix B when m=6, the overall computational complexity can be reduced by extracting common calculation factors. The following formula shows the simplification of the d×B transformation of the i-th row of data in the input tile to obtain a row of data in the temporary array tmp: i Among them, f0~f7 are the calculation factors that can be shared, d (i,n) Represents the element in the i-th row and n-th column of the input tile (n ranges from 1 to 8). The temporary array tmp is multiplied by B on the left. T We can also use the common factors f0 to f7 in the formula to simplify the calculation. In this case, tmp i Should be a column of the matrix resulting from the Winograd transformation of the input tile.

[0108]

[0109] 2. Matrix multiplication data layout

[0110] The present invention designs a matrix multiplication-friendly data layout, which can ensure continuous memory access for the matrix multiplication steps in Winograd convolution. The layout after input conversion is [L][C / C blk ][T blk / α][C blk / θ][α][θ], the layout of the filter after conversion is [K / K blk ][L][C / C blk ][Kblk / η][C blk ][η], respectively Figure 4 , Figure 5 The implementation of this layout is integrated with the transformation of inputs / filters to the Winograd domain.

[0111] To alleviate the stride memory access during the conversion process, when m=2, the input conversion is first traversed in the W direction; when m=6, the traversal order is C→W→H. In the data layout after the filter conversion, η is the fastest changing direction, so the value loaded into the same vector register crosses the output channel K and follows In-order traversal.

[0112] When the size is L×T blk ×K blk After the intra-block matrix multiplication is completed, the output conversion is performed to better utilize cache locality. The data layout of the temporary array GEMMout is also set to [L][K] to accommodate the continuous memory access of the matrix multiplication. blk / η][T blk / α][η / θ][α][θ], this layout is also conducive to efficient data loading for output conversion.

[0113] 3. Matrix multiplication microkernel

[0114] The present invention designs a matrix multiplication microkernel based on the characteristics of each dimension of the input data of the common model of the convolutional neural network. The microkernel size is set according to the need to achieve a high computational memory access ratio while taking into account the need to reduce edge case processing. In order to adapt to single-precision floating-point SIMD loading, η is set to satisfy η%4=0. Due to the number of vector registers required for loading and storing the converted input / filter data and output data, and the "ping-pong" technology used by the present invention to reduce pipeline pauses, α and η need to meet the following conditions:

[0115] 2α+η / 2+α×η / 4≤32

[0116] Through the above constraints, the optimal microkernel size α=7, η=8 is obtained to maximize the computation-to-memory ratio. Considering the general trend that the dimension T decreases and C and K increase as the number of layers of the convolutional neural network model deepens, the edge cases of the T dimension in the deep network are more time-consuming. The present invention selects α=4, η=16 as the suboptimal microkernel, thereby alleviating the edge processing overhead in the deep network (when C and K exceed T in magnitude). Both microkernel configurations can ensure full utilization of 32 vector registers.

[0117] The implementation of the "ping-pong" technology and register usage in two microkernels (α=4, η=16, hereinafter abbreviated as (4,16); α=7, η=8, hereinafter abbreviated as (7,8)) is as follows Figure 6 , Figure 7 As shown in the figure, “#” indicates the pipeline stage number in the “ping-pong” strategy; the numbers represent the vector register index values; the left, top, and bottom right rectangles represent the input, filter, and result data, respectively.

[0118] In the (4,16) configuration, the present invention uses registers v0~v3, v4~v7 as two groups of registers for loading input data, two groups of registers v8~v11, v12~v15 are used to load filter data, and the remaining registers are used to store corresponding calculation results. A group of registers storing input data is released after passing through 4 pipeline stages, and the filter registers are released at each stage. When the microkernel starts to execute, v0~v3 loads input data, and v8~v15 loads filter data. In each subsequent pipeline stage, the present invention pre-fetches 4 elements of one of the other groups of input registers (v4~v7), and pre-fetches 16 elements of another group of filter registers, so that there is enough distance between the loading and calculation instructions for the same data.

[0119] The prefetch elements input in configuration (7,8) are grouped into four pipeline stages, with the first three stages of each group containing eight elements and the last stage containing four elements.

[0120] It should be noted that the optimal and suboptimal microkernels ((7,8), (4,16)) are configured for single-precision floating point, but the optimization and configuration methods for the microkernel can be extended to other precisions by simply replacing the number of data that can be stored in the vector register (e.g., the double-precision floating point number θ is 2).

[0121] 4. Parallel Strategy

[0122] The present invention is Figure 1 The framework shown designs a multi-dimensional parallelization method that combines OpenMP and Pthreads technologies to adapt parallel modes to different problem sizes.

[0123] Specifically, the T value of the shallow network is large, while C and K are relatively small. Therefore, the present invention selects OpenMP to couple the outermost layer of the Winograd algorithm with tile blocks T blk To parallelize the loop that steps through the tiles, set the maximum number of threads to In deep networks, T is small and the parallel benefit is not high. blk = T, using OpenMP technology to parallelize the C and K dimensions, the maximum number of threads is set to The C, K, and T dimensions of the middle-layer network are all suitable for parallelization, but the use of OpenMP nested parallelism introduces additional overhead. The present invention uses Pthreads thread pool technology and lock-free task queues for multi-dimensional parallelism, and uses atomic snapshots to complete the synchronization between stages. T-dimensional parallelization is the initial subtask of the task queue. Each T-dimensional subtask pushes the C and K-dimensional subtasks into the queue to achieve parallelization. The total number of subtasks is The maximum number of threads is empirically set to

[0124] In summary, the present invention couples the three stages of Winograd convolution (input / filter conversion, matrix multiplication, and output conversion) to obtain a fused block traversal strategy that is beneficial to cache locality.

[0125] The present invention designs a microkernel for Winograd domain matrix multiplication calculation, designs two matrix multiplication microkernel sizes according to the characteristics of each dimension of the input data of the convolutional neural network model, achieves a high computing access memory ratio and a small amount of edge case processing, including: obtaining the optimal microkernel with the highest computing access memory ratio according to the limit of the number of vector registers required for computing and loading data; obtaining a suboptimal microkernel according to the need to reduce the edge case processing in the tile partitioning direction of the deep network. The microkernel uses a "ping-pong" strategy to overlap the loading and calculation of input data and filter data by dividing the pipeline stage and the vector register group, thereby reducing pipeline pauses.

[0126] The present invention packages the input data and filter data involved in the matrix multiplication operation into a "z" shape according to the different size configurations and block sizes of the microkernel. The layout enables continuous memory access during the matrix multiplication calculation process to improve the calculation efficiency. The input data and filter data of the layout are directly used as the input of the microkernel, and the implementation of the packaging process of the two is integrated with the Winograd domain conversion.

[0127] The Winograd algorithm of the present invention divides the input into multiple tiles, and selects two methods for implementing domain conversion at the assembly level according to whether the number of available vector registers can simultaneously contain twice the number of elements in the tile. First, one iteration completes the processing of 4 elements across the input channel of a tile, and register reuse is realized by utilizing the characteristic of shared elements of adjacent tiles. Second, one iteration completes the conversion of a part of the elements in a tile (here is a row), so that the number of elements across the input channel is increased to fully utilize the vector registers, and the special structure of the conversion matrix is ​​used to extract the common calculation factors and reduce the calculation complexity.

[0128] The present invention uses a multi-dimensional parallel approach to the three-mode parallel strategy of the fusion block algorithm framework, combines OpenMP and Pthreads technologies, and adapts different modes to different problem scales.

[0129] Figure 8 and Fig. 9 The running time comparison of each convolution layer of the Winograd convolution implemented by the present invention, NCNN and NNPACK in the three models of FusionNet, VGG-16 and ResNet-50 is shown respectively, where red is the present invention, orange is NCNN, and blue is NNPACK. The results show that in the case of single core, the acceleration ratio is 1.21 to 2.35 times compared with NCNN, and NNPACK is 1.30 to 2.39 times. In the case of 16 threads, the present invention achieves a geometric mean speedup ratio of 1.47 and 1.66 times relative to NCNN and NNPACK, respectively, while 32 threads are 2.06 times and 1.59 times, respectively.

[0130] Although the specific embodiments of the present invention are disclosed for the purpose of illustration, the purpose is to help understand the content of the present invention and implement it accordingly, those skilled in the art will understand that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the best embodiment, and the scope of the present invention is subject to the scope defined in the claims.

Claims

1. A Winograd convolution optimization method suitable for ARMv8 multi-core architecture, characterized in that: The method comprises: Determine the tile value T blk , input channel block value C blk And the output channel block value K blk , and according to the tile block value T blk Divide the input data into tiles; The overall domain conversion of the filter is completed through a double nested loop, and the overall domain conversion result is stored in the FilterOut array according to the first matrix multiplication data layout; By tile value T blk Traversing the tile blocks, performing intra-block input conversion, coupled matrix multiplication and output conversion on each tile block, and obtaining a Winograd convolution optimization result; wherein, performing intra-block input conversion, coupled matrix multiplication and output conversion on each tile block, and obtaining a Winograd convolution optimization result, includes: Block value C by input channel blk Loop through the loop, and in each traversal process, call the input conversion core to complete the input domain conversion, and store it into the TransInOut temporary array according to the second matrix multiplication data layout; Block value K by output channel blk Loop through, and in each traversal, first perform L matrix multiplications, each matrix multiplication is based on the input channel block value C blk Call GEMM to check the TransInOut temporary array T for the stride traversal blk ×C blk Block and FilterOut arrays in C blk ×K blk The block performs intra-block matrix multiplication and accumulation, and stores the matrix multiplication results in the GEMMOut temporary array according to the data layout; then, the output conversion kernel is called to perform output conversion on the matrix multiplication results in the GEMMOut temporary array to obtain the Winograd convolution optimization result; where L represents the number of elements in a Winograd convolution tile.

2. The method according to claim 1, characterized in that Determining the tile partition value T blk , input channel block value C blk And the output channel block value K blk ,include: Minimize data movement overhead based on cache capacity and determine the tile value T blk , input channel block value C blk And the output channel block value K blk , and combined with the characteristics of the input and output channels in the convolutional neural network, the tile block value T blk , input channel block value C blk And the output channel block value K blk To be limited.

3. The method according to claim 1, characterized in that The overall domain conversion of the filter is completed by double-layer nested loops, including: The outer loop divides the output channel into blocks K blk Traverse; The inner loop divides the input channel into blocks C blk Traverse; The filter conversion kernel is called in the loop for iterative conversion; the size of one iterative conversion is L×C blk ×K blk data.

4. The method according to claim 1, characterized in that: When the number of row and column elements of the output tile matrix is ​​m=2, the calling of the input conversion kernel to complete the input domain conversion includes: Step 4.1: Load L = 16 elements in a Winograd convolution tile into vector registers v0 to v15; where one element represents θ data values ​​across C input channels in a tile loaded into the same vector register, where θ is the number of data contained in a vector register; Step 4.2: Left-multiply the Winograd convolution tile corresponding to the L vector registers by the matrix B T , and release vector registers v0, v1, v8, and v9, and combine vector registers v16 to v27 to store the tile left multiplication matrix B T The temporary result of ; where B represents the input transformation matrix; Step 4.3: Multiply the tile by matrix B T The temporary result is right-multiplied by matrix B and stored in vector registers v16 to v31; Step 4.4: Store the results in vector register v16 to vector register v31 into the TransInOut temporary array; Step 4.5: After loading the non-overlapping data in the next Winograd convolution tile into vector registers v0, v1, v4, v5, v8, v9, v12, and v13, execute steps 4.2 to 4.4, and store the result corresponding to the next Winograd convolution tile into the TransInOut temporary array; Step 4.6: Continue to load the non-overlapping data of the next Winograd convolution tile into vector registers v2, v3, v6, v7, v10, v11, v14, and v15, then execute steps 4.2 to 4.4, and store the result corresponding to the next Winograd convolution tile to be loaded into the TransInOut temporary array; Step 4.7: Return to step 4.5 until all Winograd convolution tiles are processed.

5. The method according to claim 1, characterized in that When the number of row and column elements of the output tile matrix is ​​m=6, the calling of the input conversion kernel to complete the input domain conversion includes: Perform d×B conversion on the i-th row of data in a Winograd convolution tile and get the result tmp i ; Wherein, the result tmp i Transpose Calculate the factor f0 = d (i,3) +d (i,7) -4.25×d (i,4) , calculate the factor f1 = d (i,2) -4.25×d (i,4) +d (i,6) , calculation factor f2 = 1.25 × d (i,5) , calculation factor f3 = 2.5 × d (i,4) , calculation factor f4 = 0.25 × d (i,3) -f2+d (i,7) , calculation factor f5 = 0.5 × d (i,2) -f3+2×d (i ,6) , calculate the factor f6 = 4 × (d (i,3) -f2)+d (i,7) , calculate the factor f7 = 2 × d (i,2) -f3+0.5×d (i,6) , d represents the input tile matrix for input domain conversion calculation, d (i,n) represents the element in the i-th row and n-th column of the input tile, where n ranges from 1 to 8, and B represents the input transformation matrix; The result tmp i As a row of data in the temporary array tmp of 8×8×2θ, θ represents the number of data contained in a vector register, and 2θ is the number of input channels across which a row of data is processed in one iteration; Multiply the temporary array tmp by matrix B column by column T and store the corresponding results in the TransInOut temporary array.

6. The method according to claim 1, characterized in that The first matrix multiplication data layout is [K / K blk ][L][C / C blk ][K blk / η][C blk ][η], the second matrix multiplication data layout is [L][C / C blk ][T blk / α][C blk / θ][α][θ], the data layout of the matrix multiplication result is [l][K blk / η][T blk / α][η / θ][α][θ]; where K represents the number of output channels of the convolutional layer, C represents the number of input channels of the convolutional layer, α and η represent the parameters of the matrix multiplication microkernel, and θ represents the number of floating-point numbers stored in the vector register.

7. The method according to claim 6, characterized in that The method further comprises: When the number of row and column elements of the output tile matrix is ​​m=2, the input conversion kernel first traverses the image width W of the convolutional network input tensor, and then to the image height H until the tile block value T is completed. blk Transformation of θ elements of all tiles in the input channel direction C; finally traverse the input channel direction; When the number of row and column elements of the output tile matrix is ​​m=6, the traversal order of the input conversion kernel is to first complete the overall conversion of a tile in the input channel direction, and then traverse the image width W and height H directions of the convolution network input tensor to convert all tiles; The iteration method in the filter conversion core is to load the value loaded into the same vector register across the output channels K and follow In-order traversal.

8. The method according to claim 6, characterized in that The implementation process of the matrix multiplication microkernel includes: Based on the constraints of parameter α and parameter η, determine the optimal microkernel parameter α1 and parameter η1 that maximizes the computation-to-memory ratio; wherein the constraints include: 2α+η / 2+α×η / 4≤32 and η%4=0; According to the characteristics of the convolutional neural network, the parameters α2 and η2 corresponding to the suboptimal microkernel are selected; According to the parameters α1 and η1 of the selected optimal microkernel and the parameters α2 and η2 corresponding to the suboptimal microkernel, a ping-pong strategy is adopted to implement the operation of the matrix multiplication microkernel.

9. The method according to claim 1, characterized in that: The method further comprises: In the shallow convolutional layer of the convolutional neural network model, OpenMP is used to couple the outermost layer of the Winograd algorithm with tiles T blk Parallelize the loop that iterates through the tiles for the stride, and set the maximum number of threads to Where T represents the total number of tiles divided by the input tensor; In the deep convolutional layer of the convolutional neural network model, set the tile value T blk = T, and use OpenMP technology to parallelize the C and K dimensions, setting the maximum number of threads to Among them, C represents the number of input channels of the convolutional layer, and K represents the number of output channels of the convolutional layer; In the middle convolution layer of the convolutional neural network model, Pthreads thread pool technology and lock-free task queues are used for multi-dimensional parallelism, and atomic snapshots are used to complete the synchronization between stages; among them, T-dimensional parallelization is the initial subtask of the task queue. Each T-dimensional subtask pushes C and K-dimensional subtasks into the queue to achieve parallelization. The total number of subtasks is The maximum number of threads is empirically set to 10. A Winograd convolution optimization system for ARMv8 multi-core architecture, characterized in that: The system comprises: The tile value determination module is used to determine the tile value T blk , input channel block value C blk And the output channel block value K blk , and according to the tile block value T blk Divide the input data into tiles; A filter conversion module, used to complete the overall domain conversion of the filter through a double nested loop, and store the overall domain conversion result into the FilterOut array according to the first matrix multiplication data layout; Winograd convolution optimization module for tile-based block value T blk Traversing the tile blocks, performing intra-block input conversion, coupled matrix multiplication and output conversion on each tile block, and obtaining a Winograd convolution optimization result; wherein, performing intra-block input conversion, coupled matrix multiplication and output conversion on each tile block, and obtaining a Winograd convolution optimization result, includes: Block value C by input channel blk Loop through the loop, and in each traversal process, call the input conversion core to complete the input domain conversion, and store it into the TransInOut temporary array according to the second matrix multiplication data layout; Block value K by output channel blk Loop through, and in each traversal, first perform L matrix multiplications, each matrix multiplication is based on the input channel block value C blk Call GEMM to check the TransInOut temporary array T for the stride traversal blk ×C blk Block and FilterOut arrays in C blk ×K blk The block performs intra-block matrix multiplication and accumulation, and stores the matrix multiplication results in the GEMMOut temporary array according to the data layout; then, the output conversion kernel is called to perform output conversion on the matrix multiplication results in the GEMMOut temporary array to obtain the Winograd convolution optimization result; where L represents the number of elements in a Winograd convolution tile.

Images