Implementation method of fusion operator for convolutional neural network calculation

Abstract

The implementation method of the fusion operator of the convolutional neural network calculation of the present invention includes a convolution layer, a normalization layer, and an activation layer; wherein the convolution layer includes a first convolution layer for completing the main calculation task and a second convolution layer for completing the data folding and sorting process; the normalization layer includes a first normalization layer for completing the mean and variance calculation and a second normalization layer for completing the scaling and correction process; the calculation process of the fusion operator is: the first convolution layer calculates the convolution result, and writes the output sequence to the memory in a continuous manner; the first normalization layer performs mean variance calculation; the second convolution layer reads the calculation result of the first convolution layer from the memory, and uses a separate folding operator to sort it into the final convolution result, and then writes it to the memory; the second normalization layer performs scaling and correction calculation; the activation layer performs activation calculation. The present invention splits the convolution into two-stage calculations, and avoids writing out the 64B block size by increasing the number of reads and writes, which significantly improves the performance.

Description

Technical Field

[0001] The present invention relates to a convolutional neural network computing mode, and in particular to a method for implementing a fusion operator of convolutional neural network computing. Background Art

[0002] Convolutional neural networks are a large category of deep learning models, mainly composed of layers such as convolution, normalization (BatchNorm), activation (taking ReLU as an example), and pooling. Convolutional neural network calculations have multiple fixed calculation modes, that is, fixed calculation operator connection methods. "Convolution-normalization-activation" is one of the most commonly used calculation modes, and the calculation time of this mode dominates the training process of the overall model.

[0003] like Figure 1 The operator shown in area a of the convolutional neural network is a "convolution-normalization-activation" calculation mode diagram, in which the two dotted lines represent an independent operator, that is, an operator function call; each rectangular box represents the logical internal calculation stage of the operator. Figure 1 Taking the BatchNorm_0 operator in area a as an example, "BatchNorm_0 mean variance calculation" represents the first calculation stage of the normalization operator, and "BatchNorm_0 scaling correction" represents the second calculation stage of the normalization operator, and these two calculation stages are completed through one operator call. Figure 1 The calculation process of a requires 6 storage reads and 6 memory writes to complete two "convolution-normalization-activation" calculation modes.

[0004] like Figure 1 The operator shown in area b is Figure 1 The optimized operator fusion strategy on the calculation mode of the operators in zone a designs a new "scaling correction-activation-convolution calculation-mean calculation" calculation mode to fuse the three operators of normalization, activation, and convolution calculation into one operator, reducing the number of operator calls and function calls. Most importantly, the calculation process after the optimization and fusion only requires 4 storage reads and 4 storage writes, reducing the number of memory accesses and thus speeding up the overall calculation process. In today's processor architecture, the storage wall effect is already very obvious, and reducing the number of storage accesses means reducing the overall calculation time.

[0005] For the Taichu AI processor, the fusion operator shown in 1b above still has two shortcomings:

[0006] 1) The "scaling correction-activation-convolution calculation-mean calculation" fusion operator structure is not flexible enough and has a limited scope of application. The fusion operator starts with "scaling correction-activation", which means that in order to give full play to the performance of the fusion operator, two or more "convolution-normalization-activation" calculation modes must be strictly connected in series. Otherwise, the first-layer fusion operator will have redundant "scaling correction-activation" calculations and corresponding redundant storage accesses, reducing the computing efficiency and model development difficulty.

[0007] 2) The Taichu AI processor uses a 32*32 systolic array to accelerate convolution calculations. The systolic array structure outputs the convolution calculation results in a 64B data block size. At the same time, the Taichu AI processor provides an efficient DMA engine for data transmission. The DMA engine can perform well in the case of continuous and large data block data transmission. However, its efficiency is less than 10% for the 64B data block. Therefore, when data is output, the storage bandwidth utilization is very low.

[0008] Therefore, most of the computing time of the fused operator is spent on writing the calculation results, that is, writing the final results in 64B blocks, which leads to low overall operator efficiency and fails to achieve the acceleration effect of operator fusion. Summary of the invention

[0009] The present invention provides a method for implementing a fusion operator for convolutional neural network calculations, which can fully utilize the performance of the Taichu AI processor by reducing stride data writing and improving storage bandwidth utilization.

[0010] The implementation method of the fusion operator of the convolutional neural network calculation described in the present invention is applicable to the "convolution-normalization-activation" calculation mode, and the fusion operator includes a convolution layer (Convolution), a normalization layer (BatchNorm), and an activation (taking ReLU as an example) layer; wherein,

[0011] The convolution layer includes the first convolution layer, which is used to complete the main calculation task, and the second convolution layer, which is used to complete the data folding and sorting process;

[0012] The normalization layer includes the first normalization layer, which is used to complete the mean and variance calculation, and the second normalization layer, which is used to complete the scaling and correction process;

[0013] The calculation process of the fusion operator is:

[0014] The first convolutional layer calculates the convolution results and writes the output sequence to the memory in a continuous manner;

[0015] The first normalization layer performs mean variance calculation;

[0016] The second convolutional layer reads the calculation results of the first convolutional layer from the memory, uses a separate folding operator to organize them into the final convolution results, and then writes them out to the memory;

[0017] The second normalization layer performs scaling and correction calculations;

[0018] The activation layer performs activation calculations.

[0019] In one embodiment, the function call of the fusion operator includes:

[0020] Combine the convolution result calculated by the first convolution layer and the mean variance calculation by the first normalization layer, and make the first function call;

[0021] Combine the convolution result calculated by the second convolution layer and the scaling correction calculation of the second normalization layer, and make a second function call;

[0022] The activation layer performs activation calculations and makes a third function call.

[0023] In one embodiment, the function call of the fusion operator includes:

[0024] Combine the convolution result calculated by the first convolution layer and the mean variance calculation by the first normalization layer, and make the first function call;

[0025] The convolution result calculated by the second convolution layer, the scaling correction calculation performed by the second normalization layer, and the activation calculation performed by the activation layer are combined to make a second function call.

[0026] The present invention splits Convolution into two stages of calculation. Although the number of memory reads and writes increases, the writing of 64B blocks is avoided, so that the performance of the final two-stage calculation is still significantly ahead of the original single-stage Convolution operator performance. Taking the Taichu AI chip as an example, the performance of reading and continuous writing is much higher than that of writing of 64B blocks. The present invention significantly improves performance.

[0027] The fusion method of function calling of the present invention reduces the overall number of storage accesses and improves the overall execution efficiency.

[0028] The fusion method adopted by the present invention has no restrictions on pre- or post-operator conditions, can be connected with any operator, and does not have the problem of additional computational overhead caused by existing fusion methods, so it can be better applied to different network models. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Diagram of the "convolution-normalization-activation" calculation mode in the existing convolutional neural network

[0030] Figure 2Schematic diagram of embodiment 1 of the method for implementing the fusion operator of the convolutional neural network calculation of the present invention

[0031] Figure 3 A diagram showing the calculation process of a fusion operator applied to a model network according to the first embodiment of the present invention

[0032] Figure 4 Schematic diagram of embodiment 2 of the method for implementing the fusion operator of the convolutional neural network calculation of the present invention

[0033] Figure 5 A diagram showing the calculation process of a fusion operator applied to a model network according to the second embodiment of the present invention DETAILED DESCRIPTION

[0034] In order to clearly describe the implementation method of the fusion operator of the convolutional neural network calculation described in the present invention, it is now further explained in conjunction with the accompanying drawings.

[0035] Embodiment 1

[0036] The method for implementing the fusion operator of the convolutional neural network calculation described in the present invention is applicable to the "convolution-normalization-activation" calculation mode, such as Figure 2 As shown in area a, the fusion operator includes a convolution layer (Convolution), a normalization layer (BatchNorm), and an activation (ReLU) layer;

[0037] The convolution layer includes the first convolution layer, which is used to complete the main calculation task and calculate the output five-dimensional tensor; the second convolution layer is used to complete the folding and sorting process of the convolution result; the normalization layer includes the first normalization layer, which is used to complete the calculation of the mean and variance, and the second normalization layer, which is used to complete the scaling and correction process;

[0038] like Figure 2 As shown in the middle b area, the first function call calculation process of the fusion operator includes: the first convolution layer calculates the convolution result and writes the output sequence to the memory in a continuous manner; the first normalization layer calculates the mean variance;

[0039] The second function call calculation process of the fusion operator includes: the second convolution layer reads the calculation result of the first convolution layer from the memory, uses a separate folding operator to organize it into the final convolution result, and then writes it to the memory; the second normalization layer performs scaling and correction calculations; and the activation layer performs activation calculations.

[0040] In this embodiment, the convolution layer is divided into two stages of calculation: the first stage calculates the result of the convolution, but writes the output sequence to the memory in a continuous manner; the second stage uses a separate folding operator to read the first stage data into the final output, and then writes it to the memory continuously. In this way, the original 64B block size write operation is replaced by two consecutive writes + one read in. The performance of reading and continuous writing is much higher than that of writing 64B block size. Therefore, the convolution is split into two stages of calculation. Although the number of memory reads and writes is increased, the writing of 64B block size is avoided, solving the problem of low writing performance of 64B block size (stride data) of convolution. Taking the Taichu AI chip as an example, the performance of the final two-stage calculation is still significantly ahead of the original single-stage convolution operator performance.

[0041] like Figure 3 As shown, the fusion operator of this embodiment is taken as a whole, which can easily replace the original independent operator and be applied to the model network calculation process of an independent operator performing two operations.

[0042] like Figure 3 As shown, when the fusion operator of this embodiment performs two consecutive operations, the "BatchNorm mean variance calculation" is fused with the first stage of Convolution, and the "BatchNorm scaling correction" and "ReLU" activation are fused with the second stage of Convolution, which reduces the overall number of storage accesses. Now storage access has been repeatedly proven to be a key factor limiting the overall model training, reducing the number of storage accesses and improving the overall execution efficiency.

[0043] The fusion method adopted by the present invention has no pre- or post-operator condition requirements, does not require multiple fusion operators to be connected in series, and can be connected with any operator; compared with the existing Figure 1 The existing fusion operator in region b has no additional computational overhead in this embodiment and can be better applied to different network models with higher efficiency.

[0044] Embodiment 2

[0045] The implementation method of the fusion operator of the convolutional neural network calculation described in the present invention is applicable to the "convolution-normalization-activation" calculation mode, such as Figure 4 As shown in area a, the fusion operator includes a convolution layer (Convolution), a normalization layer (BatchNorm), and an activation (ReLU) layer;

[0046] The convolution layer includes the first convolution layer, which is used to complete the main calculation task and calculate the output five-dimensional tensor; the second convolution layer is used to complete the folding and sorting process of the convolution result; the normalization layer includes the first normalization layer, which is used to complete the calculation of the mean and variance, and the second normalization layer, which is used to complete the scaling and correction process;

[0047] like Figure 4 As shown in the middle b area, the first function call calculation process of the fusion operator includes: the first convolution layer calculates the convolution result and writes the output sequence to the memory in a continuous manner; the first normalization layer calculates the mean variance;

[0048] The second function call calculation process of the fusion operator includes: the second convolution layer reads the calculation result of the first convolution layer from the memory, uses a separate folding operator to organize it into the final convolution result, and then writes it to the memory; the second normalization layer performs scaling and correction calculation;

[0049] Compared with the first embodiment, the fusion operator of this embodiment performs activation calculation on the activation layer, which is an independent third function call calculation process.

[0050] like Figure 5 As shown, the fusion operator of this embodiment is taken as a whole, which can easily replace the original independent operator and be applied to the model network calculation process of an independent operator performing two operations.

[0051] Compared with the first embodiment of the present invention, although this embodiment adds a function calling process, compared with the prior art, it still has the advantages of improved read and write performance due to dividing the convolution layer into two stages of calculation; it also still has the advantages of no pre- or post-operator condition requirements, no need for multiple fusion operators to be connected in series, and can be connected to any operator.

Claims

1. A method for implementing a fusion operator for convolutional neural network calculation, applied to a processor using a 32*32 systolic array, wherein the systolic array structure outputs convolution calculation results in a 64B data block size, and the processor provides a DMA engine for data transmission. The fusion operator is suitable for a "convolution-normalization-activation" calculation mode, including a convolution layer, a normalization layer, and an activation layer; characterized in that: The convolution layer includes the first convolution layer, which is used to complete the main calculation task, and the second convolution layer, which is used to complete the data folding and sorting process; The normalization layer includes the first normalization layer, which is used to complete the mean and variance calculation, and the second normalization layer, which is used to complete the scaling and correction process; The calculation process of the fusion operator is: The first convolutional layer calculates the convolution results and writes the output sequence to the memory in a continuous manner; The first normalization layer performs mean variance calculation; The second convolutional layer reads the calculation results of the first convolutional layer from the memory, uses a separate folding operator to organize them into the final convolution results, and then writes them out to the memory; The second normalization layer performs scaling and correction calculations; The activation layer performs activation calculations.

2. The method for implementing the fusion operator of the convolutional neural network calculation according to claim 1, characterized in that: The function call of the fusion operator includes: Combine the convolution result calculated by the first convolution layer and the mean variance calculation by the first normalization layer, and make the first function call; Combine the convolution result calculated by the second convolution layer and the scaling correction calculation of the second normalization layer, and make a second function call; The activation layer performs activation calculations and makes a third function call.

3. The method for implementing the fusion operator of the convolutional neural network calculation according to claim 1, characterized in that: The function call of the fusion operator includes: Combine the convolution result calculated by the first convolution layer and the mean variance calculation by the first normalization layer, and make the first function call; The convolution result calculated by the second convolution layer, the scaling correction calculation performed by the second normalization layer, and the activation calculation performed by the activation layer are combined to make a second function call.

Images