Implementation method of cavity convolution based on NNA1.0

Abstract

The invention provides an NNA1.0-based dilated convolution implementation method, which comprises the following steps: S1, for dilated convolution, due to the parameter range limitation of an NNA register, when the size of an expanded convolution kernel exceeds 3 * 3, a convolution kernel matrix needs to be split; the convolution kernel data is written into a WRAM, then needed input image data is written into an FRAM, the size of the convolution kernel is set through an NNA hardware register, KH and KW must be smaller than or equal to 3, pad and stride parameters, then the NNA is started to conduct convolution calculation, convolution result output is obtained, and finally a one-line four-pixel cavity convolution output result is obtained. According to the implementation method for rapidly calculating the dilated convolution based on NNA1.0, the operation speed of the dilated convolution is improved.

Description

Technical Field

[0001] The present invention belongs to the field of neural network operator computing technology, and in particular relates to a method for realizing a dilated convolution based on NNA1.0. Background Art

[0002] In the prior art, neural networks, as mathematical models that mimic the structure and function of biological neural networks, are increasingly widely used. They acquire the ability to analyze or express sample data by learning the inherent laws of training sample data, and can be applied to a variety of application fields such as target detection, scene classification, and text recognition. The process and method of training neural networks is deep learning.

[0003] Convolutional neural networks have been widely used in the fields of images, videos, and speech. As neural networks become more and more complex, they require a lot of resources to train and evaluate, while the performance growth and versatility of hardware accelerators are becoming increasingly slow.

[0004] Convolutional neural networks are mainly composed of input layers, convolutional layers, pooling layers, and fully connected layers. The convolutional layer is the core layer of the convolutional neural network, and most of the computational complexity in the network is generated by it. Therefore, the running speed of the convolutional neural network basically depends on the computing speed of the convolutional layer.

[0005] NNA (neural network accelerator) is a high-performance, low-power neural network accelerator for processing convolution calculations. It has its own instruction set and flexible programmable features, which can be applied to different types of neural networks. By configuring the relevant register parameters, the convolution calculation acceleration of the convolution kernel matrix dimension less than or equal to 3x3 can be achieved, thereby reducing the running time of the neural network, with higher real-time performance and better user experience in practical applications. Neural network accelerator NNA1.0 supports fast operations with convolution kernel matrix dimensions less than or equal to 3x3. The hole coefficient is the amplitude of the expansion of the convolution kernel matrix. The hole convolution fills 0 between the elements of the convolution kernel, and controls the size of the expanded convolution kernel through the hole coefficient, thereby extracting multi-scale information. The larger the hole coefficient, the larger the convolution kernel.

[0006] However, the convolution process involves a large number of multiplication and accumulation calculations, and the operation speed is slow, so it is not real-time in practical applications. NNA1.0 only supports fast operations when the convolution kernel matrix dimension is less than or equal to 3x3. When the dilation coefficient is greater than 1 and the dimension of the dilated convolution kernel matrix is ​​greater than 3x3, the dilated convolution process cannot be accelerated by NNA1.0.

[0007] In addition, terms commonly used in the prior art include:

[0008] 1. Image convolution: For each pixel of the input image, the product of its neighborhood pixels and the corresponding elements of the convolution kernel matrix is ​​calculated and accumulated as the value of the pixel position. By continuously sliding the convolution kernel matrix, the final convolution output result is obtained.

[0009] 2. Feature extraction: Use computers to extract image information and determine whether each pixel belongs to an image feature. The result is that the points on the image are divided into different subsets, which often belong to independent points, continuous curves or continuous areas.

[0010] 3. FRAM: On-chip RAM inside the NNA that stores input image data.

[0011] 4. WRAM: On-chip RAM inside the NNA that stores convolution kernel data.

[0012] 5. ORAM: On-chip general-purpose RAM.

[0013] 6. Pixel: The smallest unit of input image.

[0014] 7. pad: pad the edges of the input image, divided into pad_top, pad_bottom, pad_left, pad_right, indicating the size of the padding of the top, bottom, left and right edges of the image.

[0015] 8. stride: The step size of the convolution kernel matrix sliding, divided into stride_x and stride_y, which represent the step size of the horizontal and vertical sliding of the convolution kernel matrix.

[0016] 9. dilate: The expansion amplitude of the convolution kernel, also called the dilation coefficient, is divided into dilate_x and dilate_y, which indicate the horizontal and vertical expansion amplitude of the convolution kernel. Summary of the invention

[0017] In order to solve the above problems, the purpose of this application is to propose a method for quickly calculating dilated convolution based on NNA1.0, and to achieve NNA convolution acceleration by splitting the convolution kernel.

[0018] 1. Convolution kernel splitting: For dilated convolution, when the size of the expanded convolution kernel exceeds the NNA register parameter range, NNA convolution acceleration is achieved by splitting the convolution kernel.

[0019] 2. NNA register configuration: When using NNA for convolution acceleration, the NNA related registers must be configured correctly. The convolution step size, convolution kernel size and other parameters are configured through the NNA registers. These parameters must be within the parameter range supported by NNA to obtain the correct convolution results.

[0020] Specifically, the present invention provides a method for implementing a dilated convolution based on NNA1.0, the method comprising:

[0021] S1, for dilated convolution, due to the limitation of NNA register parameter range, when the size of the convolution kernel exceeds 3x3 after expansion, the convolution kernel matrix needs to be split:

[0022] S1.1, dilated convolution kernel: input image size IH*IW, convolution kernel matrix size KH*KW, input image edge padding size is pad_top, pad_bottom, pad_left, pad_right, convolution step size is stride_x, stride_y, convolution kernel void coefficient is dilate_x, dilate_y, the dilated convolution kernel size dilate_kh and dilate_kw are calculated by the following formula:

[0023]

[0024] S1.2, NNA convolution acceleration is achieved by splitting the convolution kernel:

[0025] NNA supports a maximum of 3*3 convolution kernels. When dilate_kh or dilate_kw is greater than 3, the expanded convolution kernel matrix needs to be split:

[0026] n=(dilate_kh+2) / 3

[0027] m=(dilate_kw+2) / 3

[0028] Where n is the count of the vertical split of the dilated convolution kernel, m is the count of the horizontal split of the dilated convolution kernel, n*m is the total number of convolution kernels after the split, and the size of each convolution kernel after the split does not exceed 3*3;

[0029]

[0030] Among them, the convolution kernel size of K11 to K(n-1)(m-1) is 3*3, the convolution kernel size of K1m to K(n-1)m is 3*(dilate_kw%3), the convolution kernel size of Kn1 to Kn(m-1) is (dilate_kh%3)*3, and the convolution kernel size of Knm is (dilate_kh%3)*(dilate_kw%3), % means taking the remainder;

[0031] The input image size is IH*IW, and the edge padding of the input image is not performed, that is, pad_top, pad_bottom, pad_left, and pad_right are all 0, the convolution kernel size KH and KW are both 3, the convolution step size stride_x and stride_y are both 1, and the convolution kernel void coefficient dilate_x and dilate_y are both 2. The size of the output image is OH*OW:

[0032]

[0033]

[0034] S2, NNA's dilated convolution process: write the convolution kernel data into WRAM, then write the required input image data into FRAM, set the convolution kernel size through the NNA hardware register, where KH and KW must be less than or equal to 3, pad, and stride parameters, then start NNA for convolution calculation, get the convolution result output, and finally get a dilated convolution output result of 1 row and 4 pixels.

[0035] The specific implementation of the expansion in step S1.1 is:

[0036] The convolution kernel size is 3x3,

[0037] If the dilation coefficient is 2, the size of the expanded convolution kernel is 5x5;

[0038] If the dilation coefficient is 3, the size of the expanded convolution kernel is 7x7;

[0039]

[0040] 3x3 convolution kernel data arrangement;

[0041]

[0042] The dilated convolution kernel has a dilation coefficient of 2.

[0043]

[0044] The dilated convolution kernel has a dilation coefficient of 3.

[0045] The splitting described in step S1.2 further comprises:

[0046] Assuming that the convolution kernel sizes KH and KW are both 3, and the dilation coefficients dilate_y and dilate_x are both 2, the expanded convolution kernel sizes dilate_kh and dilate_kw are both 5 according to the formula: NNA supports a maximum of 3x3 convolution kernels, and splits the expanded 5x5 convolution kernel into: n = (dilate_kh + 2) / 3 = 2

[0047] m=(dilate_kw+2) / 3=2

[0048] n*m=4 is the total number of convolution kernels after splitting. The size of each convolution kernel after splitting does not exceed 3*3:

[0049] Split the expanded 5x5 convolution kernel into 4 convolution kernels: 3x3, 3x2, 2x3, and 2x2:

[0050] As shown below:

[0051]

[0052] Convolution kernel data arrangement before expansion;

[0053]

[0054] Convolution kernel data arrangement after expansion;

[0055] After splitting:

[0056]

[0057] 3x3 convolution kernel data arrangement;

[0058]

[0059] 3x2 convolution kernel data arrangement;

[0060]

[0061] 2x3 convolution kernel data arrangement;

[0062]

[0063] 2x2 convolution kernel data arrangement.

[0064] The step S2, writing the convolution kernel data into WRAM in sequence according to the size of the split convolution kernel, and writing the required input data into FRAM at one time, further includes:

[0065] S2.1, set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x3 convolution;

[0066] S2.2, set the convolution kernel size to 3x2 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x2 convolution;

[0067] S2.3, set the convolution kernel size to 2x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 2x3 convolution;

[0068] S2.4, set the convolution kernel size to 2x2 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 2x2 convolution.

[0069] Therefore, the advantages of this application are: an implementation method for quickly calculating dilated convolution based on NNA1.0, thereby improving the computing speed of dilated convolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0071] Figure 1 is a schematic diagram of the process of this method. DETAILED DESCRIPTION

[0072] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0073] The embodiment of the present application relates to a method for implementing a dilated convolution based on NNA1.0. Due to the limitation of the NNA register parameter range, it can only process convolution calculations with a convolution kernel matrix dimension less than or equal to 3x3, and is not very versatile in practical applications. In order to ensure versatility and improve the speed of convolution operations, when the NNA related parameter range is not met, NNA convolution acceleration is achieved by splitting, reorganizing, and accumulating the input image and the convolution kernel matrix.

[0074] The input image size is IH*IW, the convolution kernel matrix size is KH*KW, the input image edge padding size is pad_top, pad_bottom, pad_left, pad_right, the convolution step size is stride_x, stride_y, the convolution kernel void coefficient is dilate_x, dilate_y, and the expanded convolution kernel size dilate_KH and dilate_KW are calculated by the following formula:

[0075] dilate__KH=dilate_y*(KH-1)+1

[0076] dilate_KW=dilate_x*(KW-1)+1.

[0077] Specific implementation:

[0078] The convolution kernel size is 3x3. If the dilation coefficient is 2, the size of the expanded convolution kernel is 5x5.

[0079] If the dilation coefficient is 3, the size of the expanded convolution kernel is 7x7;

[0080]

[0081] 3x3 convolution kernel data arrangement;

[0082]

[0083] The dilated convolution kernel has a dilation coefficient of 2.

[0084]

[0085] The dilated convolution kernel has a dilation coefficient of 3.

[0086] The size of the final output image OH*OW is calculated by the following formula:

[0087]

[0088]

[0089] NNA convolution process: write the convolution kernel data into WRAM, then write the required input image data into FRAM, set the convolution kernel size (KH and KW must be less than or equal to 3), pad, stride and other parameters through the NNA hardware register, and then start NNA to perform convolution calculation to obtain the convolution result output. One row of 4 pixels of output data can be obtained at a time.

[0090] For example: when the convolution kernel size is 3x3, the input image size is 64x64, the input image is not padded at the edges, that is, pad_top, pad_bottom, pad_left, and pad_right are all 0, and the convolution step size stride_x and stride_y are both 1, the output image size is 62x62.

[0091]

[0092] 3x3 convolution kernel data arrangement;

[0093] By configuring the NNA register, the convolution kernel size is set to 3x3, the read addresses of WRAM and FRAM are set, and the NNA is started to calculate the 3x3 convolution to obtain the 3x3 convolution output result of 1 row of 4 pixels.

[0094] For dilated convolution, due to the limitation of NNA register parameter range, when the size of the convolution kernel exceeds 3x3 after expansion, the convolution kernel matrix needs to be split.

[0095] Specific implementation:

[0096] The convolution kernel size is 3x3, the dilation coefficient is 2, and the size of the convolution kernel after expansion is 5x5. NNA can support a maximum convolution kernel of 3x3. The 5x5 convolution kernel after expansion is split into 4 convolution kernels: 3x3, 3x2, 2x3, and 2x2.

[0097]

[0098] Convolution kernel data arrangement before expansion;

[0099]

[0100] Convolution kernel data arrangement after expansion;

[0101] After splitting:

[0102]

[0103] 3x3 convolution kernel data arrangement;

[0104]

[0105] 3x2 convolution kernel data arrangement;

[0106]

[0107] 2x3 convolution kernel data arrangement;

[0108]

[0109] 2x2 convolution kernel data arrangement.

[0110] The input image size is 64x64, and the edge padding is not performed on the input image, that is, pad_top, pad_bottom, pad_left, and pad_right are all 0, and the convolution step size stride_x and stride_y are both 1, then the output image size is 60x60.

[0111] The convolution kernel data is written into WRAM in sequence according to the size of the split convolution kernel, and the required input data is written into FRAM at one time.

[0112] (1) Set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x3 convolution;

[0113] (2) Set the convolution kernel size to 3x2 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x2 convolution;

[0114] (3) Set the convolution kernel size to 2x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 2x3 convolution;

[0115] (4) Set the convolution kernel size to 2x2 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 2x2 convolution;

[0116] Finally, we get a row of 4 pixels of the dilated convolution output.

[0117] In summary, if Figure 1 As shown, the method comprises the following steps:

[0118] S1, for dilated convolution, due to the limitation of NNA register parameter range, when the size of the convolution kernel exceeds 3x3 after expansion, the convolution kernel matrix needs to be split:

[0119] S1.1, dilated convolution kernel: input image size IH*IW, convolution kernel matrix size KH*KW, input image edge padding size is pad_top, pad_bottom, pad_left, pad_right, convolution step size is stride_x, stride_y, convolution kernel void coefficient is dilate_x, dilate_y, the dilated convolution kernel size dilate_kh and dilate_kw are calculated by the following formula:

[0120]

[0121] S1.2, NNA convolution acceleration is achieved by splitting the convolution kernel:

[0122] NNA supports a maximum of 3*3 convolution kernels. When dilate_kh or dilate_kw is greater than 3, the expanded convolution kernel matrix needs to be split:

[0123] n=(dilate_kh+2) / 3

[0124] m=(dilate_kw+2) / 3

[0125] Where n is the count of the vertical split of the dilated convolution kernel, m is the count of the horizontal split of the dilated convolution kernel, n*m is the total number of convolution kernels after the split, and the size of each convolution kernel after the split does not exceed 3*3;

[0126]

[0127] Among them, the convolution kernel size of K11 to K(n-1)(m-1) is 3*3, the convolution kernel size of K1m to K(n-1)m is 3*(dilate_kw%3), the convolution kernel size of Kn1 to Kn(m-1) is (dilate_kh%3)*3, and the convolution kernel size of Knm is (dilate_kh%3)*(dilate_kw%3), % means taking the remainder;

[0128] The input image size is IH*IW, and the edge padding of the input image is not performed, that is, pad_top, pad_bottom, pad_left, and pad_right are all 0, the convolution kernel size KH and KW are both 3, the convolution step size stride_x and stride_y are both 1, and the convolution kernel void coefficient dilate_x and dilate_y are both 2. The size of the output image is OH*OW:

[0129]

[0130]

[0131] S2, NNA's dilated convolution process: write the convolution kernel data into WRAM, then write the required input image data into FRAM, set the convolution kernel size through the NNA hardware register, where KH and KW must be less than or equal to 3, pad, and stride parameters, then start NNA for convolution calculation, get the convolution result output, and finally get a dilated convolution output result of 1 row and 4 pixels.

[0132] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for implementing dilated convolution based on NNA1.0, characterized in that: The method comprises: S1, for dilated convolution, due to the limitation of NNA register parameter range, when the size of the convolution kernel exceeds 3x3 after expansion, the convolution kernel matrix needs to be split: S1.1, dilated convolution kernel: input image size IH*IW, convolution kernel matrix size KH*KW, input image edge padding size is pad_top, pad_bottom, pad_left, pad_right, convolution step size is stride_x, stride_y, convolution kernel void coefficient is dilate_x, dilate_y, the dilated convolution kernel size dilate_kh and dilate_kw are calculated by the following formula: S1.2, NNA convolution acceleration is achieved by splitting the convolution kernel: NNA supports a maximum of 3*3 convolution kernels. When dilate_kh or dilate_kw is greater than 3, the expanded convolution kernel matrix needs to be split: n=(dilate_kh+2) / 3 m=(dilate_kw+2) / 3 Where n is the count of the vertical split of the dilated convolution kernel, m is the count of the horizontal split of the dilated convolution kernel, n*m is the total number of convolution kernels after the split, and the size of each convolution kernel after the split does not exceed 3*3; Among them, the convolution kernel size of K11 to K(n-1)(m-1) is 3*3, the convolution kernel size of K1m to K(n-1)m is 3*(dilate_kw%3), the convolution kernel size of Kn1 to Kn(m-1) is (dilate_kh%3)*3, and the convolution kernel size of Knm is (dilate_kh%3)*(dilate_kw%3), % means taking the remainder; The input image size is IH*IW, and the edge padding of the input image is not performed, that is, pad_top, pad_bottom, pad_left, and pad_right are all 0, the convolution kernel size KH and KW are both 3, the convolution step size stride_x and stride_y are both 1, and the convolution kernel void coefficient dilate_x and dilate_y are both 2. The size of the output image is OH*OW: S2, NNA's dilated convolution process: write the convolution kernel data into WRAM, then write the required input image data into FRAM, set the convolution kernel size through the NNA hardware register, where KH and KW must be less than or equal to 3, pad, and stride parameters, then start NNA for convolution calculation, get the convolution result output, and finally get a dilated convolution output result of 1 row and 4 pixels.

2. The method for implementing dilated convolution based on NNA1.0 according to claim 1, characterized in that: The specific implementation of the expansion in step S1.1 is: The convolution kernel size is 3x3, If the dilation coefficient is 2, the size of the expanded convolution kernel is 5x5; If the dilation coefficient is 3, the size of the expanded convolution kernel is 7x7; 3x3 convolution kernel data arrangement; The dilated convolution kernel has a dilation coefficient of 2. The dilated convolution kernel has a dilation coefficient of 3.

3. The method for implementing dilated convolution based on NNA1.0 according to claim 2, characterized in that: The splitting described in step S1.2 further comprises: Assuming that the convolution kernel sizes KH and KW are both 3, and the dilation coefficients dilate_y and dilate_x are both 2, the expanded convolution kernel sizes dilate_kh and dilate_kw are both 5 according to the formula: NNA supports a maximum of 3x3 convolution kernels, and splits the expanded 5x5 convolution kernel into the following: n=(dilate_kh+2) / 3=2 m=(dilate_kw+2) / 3=2 n*m=4 is the total number of convolution kernels after splitting. The size of each convolution kernel after splitting does not exceed 3*3: Split the expanded 5x5 convolution kernel into 4 convolution kernels: 3x3, 3x2, 2x3, and 2x2: As shown below: Convolution kernel data arrangement before expansion; Convolution kernel data arrangement after expansion; After splitting: 3x3 convolution kernel data arrangement; 3x2 convolution kernel data arrangement; 2x3 convolution kernel data arrangement; 2x2 convolution kernel data arrangement.

4. The method for implementing dilated convolution based on NNA1.0 according to claim 3, characterized in that: The step S2, writing the convolution kernel data into WRAM in sequence according to the size of the split convolution kernel, and writing the required input data into FRAM at one time, further includes: S2.1, set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x3 convolution; S2.2, set the convolution kernel size to 3x2 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x2 convolution; S2.3, set the convolution kernel size to 2x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 2x3 convolution; S2.4, set the convolution kernel size to 2x2 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 2x2 convolution.

Images