An implementation method of a hollow convolution based on NNA1.0

Abstract

The application provides an implementation method of a cavity convolution based on NNA1.0, comprising the following steps: S1, for the cavity convolution, due to the range limitation of the NNA register parameters, when the size of the expanded convolution kernel is greater than 3x3, the convolution kernel matrix needs to be split; S2, the cavity convolution process of the NNA: write the convolution kernel data into the WRAM, then write the required input image data into the FRAM, set the convolution kernel size through the NNA hardware register, wherein KH and KW must be less than or equal to 3, the pad and stride parameters, then start the NNA to perform the convolution calculation, obtain the convolution result output, and finally obtain the cavity convolution output result of one row and four pixels. The method is based on the implementation method of the fast cavity convolution calculation of the NNA1.0, and the operation speed of the cavity convolution is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of neural network operator calculation, and particularly relates to a method for implementing a dilated convolution based on NNA1.0. BACKGROUND

[0002] In the prior art, a neural network is a mathematical model for imitating the structure and function of a biological neural network, and is increasingly widely applied. The neural network obtains the ability to analyze or express sample data by learning the internal law of the sample data, and can be applied to various application fields such as target detection, scene classification, and character recognition. The process and method of training the neural network are deep learning.

[0003] A convolutional neural network has a wide range of applications in the fields of images, videos, and speech. As neural networks become more and more complex, a large amount of resources are required for training and evaluation, and the performance growth and generalization of hardware accelerators are becoming slower and slower.

[0004] A convolutional neural network mainly consists of an input layer, a convolutional layer, a pooling layer, and a fully connected layer. The convolutional layer is the core layer of the convolutional neural network, and most of the computational load in the network is generated by the convolutional layer. Therefore, the running speed of the convolutional neural network basically depends on the operation speed of the convolutional layer.

[0005] NNA (neural network accelerator) is a high-performance and low-power neural network accelerator for processing convolutional calculations. It has its own instruction set and flexible programmable characteristics, and can be applied to different types of neural networks. By configuring related register parameters, convolutional kernel matrix dimensions less than or equal to 3x3 can be accelerated, thereby reducing the running time of the neural network, improving real-time performance in practical applications, and providing a better user experience. The neural network accelerator NNA1.0 supports fast operation of convolutional kernel matrix dimensions less than or equal to 3x3. The dilation coefficient is the magnitude of the expansion of the convolutional kernel matrix. Dilation convolution is to fill 0 between the elements of the convolutional kernel, and the size of the expanded convolutional kernel is controlled by the dilation coefficient, so as to extract multi-scale information. The larger the dilation coefficient, the larger the convolutional kernel.

[0006] However, the convolution process involves a large number of multiply-accumulate calculations, and the operation speed is slow, which does not have real-time performance in practical applications. NNA1.0 only supports fast operation of convolutional kernel matrix dimensions less than or equal to 3x3. When the dilation coefficient is greater than 1 and the dimension of the expanded convolutional kernel matrix is greater than 3x3, the dilated convolution process cannot be accelerated by NNA1.0.

[0007] In addition, the commonly used terms in the prior art include:

[0008] 1、Image Convolution: For each pixel of the input image, calculate the product of its neighborhood pixels and the corresponding elements of the convolution kernel matrix, and accumulate it as the value of the pixel position. By continuously sliding the convolution kernel matrix, the final convolution output result is obtained.

[0009] 2、Feature Extraction: Use a computer to extract image information to determine whether each pixel belongs to an image feature. The result is to divide the points on the image into different subsets, which often belong to independent points, continuous curves or continuous regions.

[0010] 3、FRAM: NNA internal on-chip RAM for storing input image data.

[0011] 4、WRAM: NNA internal on-chip RAM for storing convolution kernel data.

[0012] 5、ORAM: On-chip general-purpose RAM.

[0013] 6、pixel: The smallest unit of the input image.

[0014] 7、pad: Edge padding of the input image, divided into pad_top, pad_bottom, pad_left, pad_right, indicating the size of the edge padding of the image.

[0015] 8、stride: The step size of the convolution kernel matrix sliding, divided into stride_x and stride_y, indicating the step size of the convolution kernel matrix sliding horizontally and vertically.

[0016] 9、dilate: The dilation of the convolution kernel, also known as the hole coefficient, divided into dilate_x and dilate_y, indicating the dilation of the convolution kernel horizontally and vertically. SUMMARY

[0017] To solve the above problems, the purpose of the present application is to provide a fast computing method for implementing dilated convolution based on NNA1.0, which realizes NNA convolution acceleration by splitting the convolution kernel. Among them:

[0018] 1、Convolution kernel splitting: For dilated convolution, when the size of the expanded convolution kernel exceeds the NNA register parameter range, NNA convolution acceleration is realized by convolution kernel splitting.

[0019] 2、NNA register configuration: NNA-related registers must be correctly configured for convolution acceleration using NNA. The step size of the convolution, the size of the convolution kernel and other parameters are configured through NNA registers. These parameters must be within the parameter range supported by NNA to obtain the correct convolution result.

[0020] Specifically, the application provides an implementation method of a hollow convolution based on NNA1.0, which comprises the following steps:

[0021] S1, for the hollow convolution, when the size of the expanded convolution kernel is greater than 3*3 due to the limitation of the NNA register parameter range, the convolution kernel matrix needs to be split:

[0022] S1.1, the expanded convolution kernel: the size of the input image is IH*IW, the size of the convolution kernel matrix is KH*KW, the edge padding size of the input image is pad_top, pad_bottom, pad_left and pad_right, the convolution step is stride_x and stride_y, the hollow coefficient of the convolution kernel is dilate_x and dilate_y, the size of the expanded convolution kernel is dilate_kh and dilate_kw, which are calculated by the following formula:

[0023]

[0024] S1.2, the NNA convolution acceleration is realized by splitting the convolution kernel:

[0025] The NNA supports a 3*3 convolution kernel at most, when dilate_kh or dilate_kw is greater than 3, the expanded convolution kernel matrix needs to be split:

[0026] n=(dilate_kh+2) / 3

[0027] m=(dilate_kw+2) / 3

[0028] Wherein n is the count of splitting the expanded convolution kernel in the vertical direction, m is the count of splitting the expanded convolution kernel in the horizontal direction, n*m is the total number of the split convolution kernels, and the size of each split convolution kernel is not greater than 3*3;

[0029]

[0030] Wherein the size of the K11 to K(n-1)(m-1) convolution kernel is 3*3, the size of the K1m to K(n-1)m convolution kernel is 3*(dilate_kw%3), the size of the Kn1 to Kn(m-1) convolution kernel is (dilate_kh%3)*3, and the size of the Knm convolution kernel is (dilate_kh%3)*(dilate_kw%3), % represents the remainder;

[0031] When the input image size is IH*IW, no edge padding is performed on the input image, i.e., pad_top, pad_bottom, pad_left, and pad_right are all 0, the convolution kernel size KH and KW are both 3, the convolution step stride_x and stride_y are both 1, and the convolution kernel dilation coefficients dilate_x and dilate_y are both 2, the size of the output image OH*OW is obtained:

[0032]

[0033]

[0034] S2, the dilation convolution process of the NNA: the convolution kernel data is written into the WRAM, the required input image data is written into the FRAM, the convolution kernel size is set through the NNA hardware register, wherein KH and KW must be less than or equal to 3, the pad and stride parameters are set, and then the NNA is started to perform convolution calculation to obtain the convolution result output, and finally the dilation convolution output result of 1 row and 4 pixels is obtained.

[0035] The specific implementation of the expansion in step S1.1 is as follows:

[0036] The convolution kernel size is 3x3,

[0037] If the dilation coefficient is 2, the expanded convolution kernel size is 5x5;

[0038] If the dilation coefficient is 3, the expanded convolution kernel size is 7x7;

[0039]

[0040] 3x3 convolution kernel data arrangement;

[0041]

[0042] Dilation coefficient is 2, expanded convolution kernel;

[0043]

[0044] Dilation coefficient is 3, expanded convolution kernel.

[0045] The splitting in step S1.2 further includes:

[0046] Assuming the convolution kernel size KH, KW are both 3, the dilation coefficients dilate_y, dilate_x are both 2, according to the formula, the expanded convolution kernel size dilate_kh, dilate_kw are both 5: NNA supports a maximum of 3x3 convolution kernel, and the expanded 5x5 convolution kernel is split: n = (dilate_kh+2) / 3 = 2

[0047] m = (dilate_kw+2) / 3 = 2

[0048] n*m = 4 is the total number of split convolution kernels, and the size of each split convolution kernel is not more than 3*3:

[0049] Split the expanded 5x5 convolution kernel into 3x3, 3x2, 2x3, and 2x2 convolution kernels:

[0050] As shown below:

[0051]

[0052] The data arrangement of the convolution kernel before expansion;

[0053]

[0054] The data arrangement of the convolution kernel after expansion;

[0055] After splitting:

[0056]

[0057] The data arrangement of the 3x3 convolution kernel;

[0058]

[0059] The data arrangement of the 3x2 convolution kernel;

[0060]

[0061] The data arrangement of the 2x3 convolution kernel;

[0062]

[0063] The data arrangement of the 2x2 convolution kernel.

[0064] The step S2, the convolution kernel data is written into the WRAM according to the split convolution kernel size, and the required input data is written into the FRAM at one time, further comprising:

[0065] S2.1, set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of the WRAM and the FRAM, and start the NNA to calculate the 3x3 convolution;

[0066] S2.2, set the convolution kernel size as 3x2 by configuring the NNA register, set the read address of the WRAM and the FRAM, and start the NNA to calculate the 3x2 convolution;

[0067] S2.3, set the convolution kernel size as 2x3 by configuring the NNA register, set the read address of the WRAM and the FRAM, and start the NNA to calculate the 2x3 convolution;

[0068] S2.4, set the convolution kernel size as 2x2 by configuring the NNA register, set the read address of the WRAM and the FRAM, and start the NNA to calculate the 2x2 convolution.

[0069] Therefore, the advantage of the present application is that the implementation method of the fast calculation of the dilated convolution based on the NNA1.0 improves the operation speed of the dilated convolution. BRIEF DESCRIPTION OF DRAWINGS

[0070] The accompanying drawings, which are included to provide a further understanding of the application and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application.

[0071] Figure 1 is a flowchart of the method. DETAILED DESCRIPTION

[0072] In order to enable a clearer understanding of the technical content and advantages of the present application, the present application will be further described in detail in conjunction with the drawings.

[0073] The embodiment of the present application relates to an implementation method of dilated convolution based on NNA1.0. Due to the limitation of the parameter range of the NNA register, only convolution calculation with a convolution kernel matrix dimension less than or equal to 3x3 can be processed, and the general applicability is not high in actual application. In order to ensure the general applicability and improve the convolution operation speed, when the NNA related parameter range is not met, the input image and the convolution kernel matrix are split, reorganized and accumulated to realize NNA convolution acceleration.

[0074] The input image size is IH*IW, the convolution kernel matrix size is KH*KW, the input image edge padding size is pad_top, pad_bottom, pad_left, pad_right, the convolution stride is stride_x, stride_y, the dilated coefficient of the convolution kernel is dilate_x, dilate_y, and the dilated convolution kernel size dilate_KH and dilate_KW are calculated by the following formula:

[0075] dilate__KH = dilate_y * (KH - 1) + 1

[0076] dilate_KW = dilate_x * (KW - 1) + 1.

[0077] Implementation:

[0078] If the hole coefficient is 2, the expanded convolution kernel size is 5x5;

[0079] If the hole coefficient is 3, the expanded convolution kernel size is 7x7;

[0080]

[0081] 3x3 convolution kernel data arrangement;

[0082]

[0083] Dilated convolution kernel with a hole coefficient of 2;

[0084]

[0085] Dilated convolution kernel with a hole coefficient of 3;

[0086] The size OH*OW of the final output image is calculated by the following formula:

[0087]

[0088]

[0089] NNA convolution process: write the convolution kernel data into the WRAM, write the required input image data into the FRAM, set the convolution kernel size (KH and KW must be less than or equal to 3), pad, stride, etc. through NNA hardware registers, then start NNA for convolution calculation, get the convolution result output, which can get 1 row of 4 pixel output data at a time.

[0090] For example: the convolution kernel size is 3x3, the input image size is 64x64, no edge padding is performed on the input image, i.e. pad_top, pad_bottom, pad_left, pad_right are all 0, and the convolution step stride_x, stride_y are all 1, the output image size is 62x62.

[0091]

[0092] 3x3 convolution kernel data arrangement;

[0093] Set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of the WRAM and FRAM, start NNA calculation of 3x3 convolution, and get 1 row of 4 pixel 3x3 convolution output result.

[0094] For the empty convolution, due to the limitation of the NNA register parameter range, when the expanded convolution kernel size exceeds 3x3, the convolution kernel matrix needs to be split.

[0095] Specific implementation:

[0096] The convolution kernel size is 3x3, the empty coefficient is 2, the expanded convolution kernel size is 5x5, and the NNA can support a 3x3 convolution kernel at most. The expanded 5x5 convolution kernel is split into 3x3, 3x2, 2x3, and 2x2 convolution kernels.

[0097]

[0098] The data arrangement of the convolution kernel before expansion;

[0099]

[0100] The data arrangement of the convolution kernel after expansion;

[0101] After splitting:

[0102]

[0103] The data arrangement of the 3x3 convolution kernel;

[0104]

[0105] The data arrangement of the 3x2 convolution kernel;

[0106]

[0107] The data arrangement of the 2x3 convolution kernel;

[0108]

[0109] The data arrangement of the 2x2 convolution kernel.

[0110] The input image size is 64x64, no edge padding is performed on the input image, i.e., pad_top, pad_bottom, pad_left, and pad_right are all 0, and the convolution step stride_x and stride_y are both 1, and the output image size is 60x60.

[0111] The convolution kernel data is written into the WRAM according to the split convolution kernel size, and the required input data is written into the FRAM at one time.

[0112] (1) Set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of the WRAM and the FRAM, and start the NNA to calculate the 3x3 convolution;

[0113] (2) Set the convolution kernel size to 3x2 by configuring the NNA registers, set the read addresses of the WRAM and FRAM, and start the NNA to calculate the 3x2 convolution;

[0114] (3) Set the convolution kernel size to 2x3 by configuring the NNA registers, set the read addresses of the WRAM and FRAM, and start the NNA to calculate the 2x3 convolution;

[0115] (4) Set the convolution kernel size to 2x2 by configuring the NNA registers, set the read addresses of the WRAM and FRAM, and start the NNA to calculate the 2x2 convolution;

[0116] Finally, the 1-row 4-pixel hole convolution output result is obtained.

[0117] As shown in the above, the method comprises the following steps: Figure 1

[0118] S1, for the hole convolution, due to the range limitation of the NNA register parameters, when the size of the expanded convolution kernel exceeds 3x3, the convolution kernel matrix needs to be split:

[0119] S1.1, the expanded convolution kernel: the size of the input image IH*IW, the size of the convolution kernel matrix KH*KW, the edge padding size of the input image pad_top, pad_bottom, pad_left, pad_right, the convolution step size stride_x, stride_y, the convolution kernel dilation coefficient dilate_x, dilate_y, the size of the expanded convolution kernel dilate_kh and dilate_kw, which are calculated by the following formula:

[0120]

[0121] S1.2, the NNA convolution acceleration is realized by splitting the convolution kernel:

[0122] The NNA supports a 3*3 convolution kernel at most, when dilate_kh or dilate_kw is greater than 3, the expanded convolution kernel matrix needs to be split:

[0123] n=(dilate_kh+2) / 3

[0124] m=(dilate_kw+2) / 3

[0125] Wherein n is the count of splitting the expanded convolution kernel vertically, m is the count of splitting the expanded convolution kernel horizontally, n*m is the total number of split convolution kernels, and the size of each split convolution kernel is not more than 3*3;

[0126] ​

[0127] wherein the convolution kernel size of K11 to K(n-1)(m-1) is 3*3, the convolution kernel size of K1m to K(n-1)m is 3*(dilate_kw%3), the convolution kernel size of Kn1 to Kn(m-1) is (dilate_kh%3)*3, and the convolution kernel size of Knm is (dilate_kh%3)*(dilate_kw%3), % represents the remainder;

[0128] When the input image size is IH*IW, no edge padding is performed on the input image, i.e., pad_top, pad_bottom, pad_left, and pad_right are all 0, the convolution kernel size KH and KW are both 3, the convolution step stride_x and stride_y are both 1, and the convolution kernel hole coefficient dilate_x and dilate_y are both 2, the size of the output image OH*OW is obtained:

[0129]

[0130]

[0131] The S2, the dilated convolution process of the NNA: the convolution kernel data is written into the WRAM, and the required input image data is written into the FRAM, the convolution kernel size is set through the NNA hardware register, wherein KH and KW must be less than or equal to 3, pad, stride parameters, and then the NNA is started to perform convolution calculation, and the convolution result output is obtained, and finally the dilated convolution output result of 1 row and 4 pixels is obtained.

[0132] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the embodiments of the present application can be variously changed and modified. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An implementation method of a cavity convolution based on NNA1.0, characterized in that, The method Comprise the following steps: S1, for the hollow convolution, due to the NNA register parameter range limit, when the expanded convolution kernel size exceeds 3x3, the convolution kernel matrix needs to be split: S1.1, the expanded convolution kernel: the input image size IH*IW, the convolution kernel matrix size KH*KW, the input image edge padding size pad_top, pad_bottom, pad_left, pad_right, the convolution step size stride_x, stride_y, the convolution kernel hole coefficient dilate_x, dilate_y, the expanded convolution kernel size dilate_kh and dilate_kw, calculated by the following formula: ； S1.2, the NNA convolution acceleration is realized by splitting the convolution kernel: The NNA supports 3*3 convolution kernel at most, when dilate_kh or dilate_kw is greater than 3, the expanded convolution kernel matrix needs to be split: ； Where n is the count of vertical splitting of the expanded convolution kernel, m is the count of horizontal splitting of the expanded convolution kernel, n*m is the total number of split convolution kernels, and the size of each split convolution kernel is not more than 3*3; ； Where K11 to K(n-1)(m-1) convolution kernel size is 3*3, K1m to K(n-1)m convolution kernel size is 3*(dilate_kw%3), Kn1 to Kn(m-1) convolution kernel size is (dilate_kh%3)*3, Knm convolution kernel size is (dilate_kh%3)*(dilate_kw%3), % represents the remainder; The input image size is IH*IW, no edge padding is performed on the input image, i.e. pad_top, pad_bottom, pad_left and pad_right are all 0, the convolution kernel size KH and KW are both 3, the convolution step size stride_x and stride_y are both 1, and the convolution kernel hole coefficient dilate_x and dilate_y are both 2, so that the output image size OH*OW is obtained: ； S2, the hollow convolution process of the NNA: write the convolution kernel data into the WRAM, then write the required input image data into the FRAM, set the convolution kernel size, pad, stride parameters through the NNA hardware register, wherein KH and KW must be less than or equal to 3, then start the NNA to perform convolution calculation to obtain the convolution result output, and finally obtain 1 row and 4 pixels of the hollow convolution output result.

2. The implementation method of the hollow convolution based on NNA1.0 according to claim 1, characterized in that, The specific implementation of the expansion in the step S1.1: The convolution kernel size is 3x3, If the hole coefficient is 2, the expanded convolution kernel size is 5x5; If the hole coefficient is 3, the expanded convolution kernel size is 7x7; ； 3x3 convolution kernel data arrangement; ； The hole coefficient is 2, and the expanded convolution kernel; ； The hole coefficient is 3, and the expanded convolution kernel.

3. The implementation method of the hollow convolution based on NNA1.0 according to claim 2, characterized in that, The splitting in the step S1.2 further comprises: The convolution kernel size KH, KW is 3, the dilation coefficient dilate_y, dilate_x is 2, according to the formula, the dilated convolution kernel size dilate_kh, dilate_kw is 5: The NNA supports a maximum of 3x3 convolution kernel, and the dilated 5x5 convolution kernel is split: ； n*m=4 is the total number of split convolution kernels, and the size of each split convolution kernel is not more than 3*3: The dilated 5x5 convolution kernel is split into 3x3, 3x2, 2x3 and 2x2 four convolution kernels: As follows: ； The pre-dilation convolution kernel data arrangement; ； The post-dilation convolution kernel data arrangement; After splitting: ； 3x3 convolution kernel data arrangement; ； 3x2 convolution kernel data arrangement; ； 2x3 convolution kernel data arrangement; ； 2x2 convolution kernel data arrangement.

4. The implementation method of the hollow convolution based on NNA1.0 according to claim 3, characterized in that, The step S2, the convolution kernel data is written into the WRAM according to the split convolution kernel size, and the required input data is written into the FRAM at one time, further comprising: S2.1, set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of the WRAM and the FRAM, and start the NNA calculation of 3x3 convolution; S2.2, set the convolution kernel size to 3x2 by configuring the NNA register, set the read address of the WRAM and the FRAM, and start the NNA calculation of 3x2 convolution; S2.3, set the convolution kernel size to 2x3 by configuring the NNA register, set the read address of the WRAM and the FRAM, and start the NNA calculation of 2x3 convolution; S2.4, set the convolution kernel size to 2x2 by configuring the NNA register, set the read address of the WRAM and the FRAM, and start the NNA calculation of 2x2 convolution.

Images