Acceleration method of cavity convolution when cavity coefficient is greater than 5

Abstract

The invention provides a method for accelerating cavity convolution when a cavity coefficient is greater than 5, which comprises the following steps of: S1, for the cavity convolution, due to the parameter range limitation of an NNA register, when the cavity coefficient is greater than 5 and the size of an expanded convolution kernel exceeds 3 * 3, splitting a convolution kernel matrix; s2, a cavity convolution process of the NNA: when a cavity coefficient is greater than 5, convolution kernel data of a part of split convolution kernels are all 0, only the convolution kernels of which the convolution kernel data are not all 0 are written into a WRAM, then required input image data are written into an FRAM, the sizes of the convolution kernels are set through an NNA hardware register, KH and KW must be smaller than or equal to 3, pad and stride parameters, then the NNA is started to perform convolution calculation, and the convolution kernel data are not all 0; and a convolution result is obtained and output.

Description

Technical Field

[0001] The present invention belongs to the field of neural network computing technology, and in particular relates to an acceleration method for dilated convolution when a dilated coefficient is greater than 5. Background Art

[0002] In the prior art, neural networks, as mathematical models that mimic the structure and function of biological neural networks, are increasingly widely used. They acquire the ability to analyze or express sample data by learning the inherent laws of training sample data, and can be applied to a variety of application fields such as target detection, scene classification, and text recognition. The process and method of training neural networks is deep learning.

[0003] Convolutional neural networks have been widely used in the fields of images, videos, and speech. As neural networks become more and more complex, they require a lot of resources to train and evaluate, while the performance growth and versatility of hardware accelerators are becoming increasingly slow.

[0004] Convolutional neural networks are mainly composed of input layers, convolutional layers, pooling layers, and fully connected layers. The convolutional layer is the core layer of the convolutional neural network, and most of the computational complexity in the network is generated by it. Therefore, the running speed of the convolutional neural network basically depends on the computing speed of the convolutional layer.

[0005] NNA (neural network accelerator) is a high-performance, low-power neural network accelerator for processing convolution calculations. It has its own instruction set and flexible programmable features, which can be applied to different types of neural networks. By configuring the relevant register parameters, the convolution calculation acceleration of the convolution kernel matrix dimension less than or equal to 3x3 can be achieved, thereby reducing the running time of the neural network, with higher real-time performance and better user experience in practical applications. Neural network accelerator NNA1.0 supports fast operations with convolution kernel matrix dimensions less than or equal to 3x3. The hole coefficient is the amplitude of the expansion of the convolution kernel matrix. The hole convolution fills 0 between the elements of the convolution kernel, and controls the size of the expanded convolution kernel through the hole coefficient, thereby extracting multi-scale information. The larger the hole coefficient, the larger the convolution kernel.

[0006] However, the convolution process involves a large number of multiplication and accumulation calculations, and the operation speed is slow. It is not real-time in practical applications. NNA only supports fast operations when the convolution kernel matrix dimension is less than or equal to 3x3. When the void coefficient is large, the dimension of the expanded convolution kernel matrix is ​​much larger than 3x3. The NNA is started by splitting the convolution kernel to accelerate the void convolution process, but the operation speed is still slow and cannot meet the requirements of practical applications.

[0007] In addition, terms commonly used in the prior art include:

[0008] 1. Image convolution: For each pixel of the input image, the product of its neighborhood pixels and the corresponding elements of the convolution kernel matrix is ​​calculated and accumulated as the value of the pixel position. By continuously sliding the convolution kernel matrix, the final convolution output result is obtained.

[0009] 2. FRAM: On-chip RAM inside the NNA that stores input image data.

[0010] 3. WRAM: On-chip RAM inside the NNA that stores convolution kernel data.

[0011] 4. Pixel: The smallest unit of input image.

[0012] 5. pad: pad the edges of the input image, divided into pad_top, pad_bottom, pad_left, pad_right, indicating the size of the padding of the top, bottom, left and right edges of the image.

[0013] 6. stride: The step size of the convolution kernel matrix sliding, divided into stride_x and stride_y, which represent the step size of the horizontal and vertical sliding of the convolution kernel matrix.

[0014] 7. dilate: The expansion amplitude of the convolution kernel, also called the dilation coefficient, is divided into dilate_x and dilate_y, which indicate the horizontal and vertical expansion amplitude of the convolution kernel. Summary of the invention

[0015] In order to solve the above problems, the purpose of this application is: the general method of dilated convolution will calculate the convolution results of each small convolution kernel and the corresponding input image after the dilated convolution kernel is split, and when the dilation coefficient is greater than 5, that is, in some special scenarios (the dilation coefficient is greater than 5, such as QR code, voice and other applications), the general method is inefficient and slow in operation. In order to improve the operation speed of dilated convolution in these special scenarios and make it more real-time in actual applications, this method is proposed.

[0016] NNA convolution acceleration is achieved by splitting the convolution kernel. Among them:

[0017] 1. Convolution kernel splitting: For dilated convolution, when the size of the expanded convolution kernel exceeds the NNA register parameter range, NNA convolution acceleration is achieved by splitting the convolution kernel.

[0018] 2. NNA register configuration: When using NNA for convolution acceleration, the NNA related registers must be configured correctly. The convolution step size, convolution kernel size and other parameters are configured through the NNA registers. These parameters must be within the parameter range supported by NNA to obtain the correct convolution results.

[0019] Specifically, the present invention provides an acceleration method for dilated convolution when the dilation coefficient is greater than 5. When the dilation coefficient is greater than 5, the dilated convolution kernel is split, and the convolution kernel data of a part of the small convolution kernels after the splitting are all 0. At this time, these small convolution kernels are not convolved with the corresponding input image. The method includes:

[0020] Assuming the convolution kernel size is 3x3, when the dilate coefficients dilate_x and dilate_y are greater than 5, as shown in the following table, those greater than 5 are all in the following table. The ellipsis in the table indicates that it is greater than 6: the dilate coefficient is equal to 6, which means that 5 zeros are filled between adjacent convolution kernel elements. The dilate coefficient is equal to 7, and 6 zeros are filled between adjacent convolution kernel elements. At this time, the ellipsis represents 1 zero. The dilate coefficient is equal to 8, and 7 zeros are filled between adjacent convolution kernel elements. At this time, the ellipsis represents 2 zeros. And so on. The convolution kernel before expansion is:

[0021] k11 k12 k13 K21 K22 K23 K31 K32 K33 ;

[0023] Convolution kernel after expansion: The gray part in the table indicates the small convolution kernel whose convolution kernel data is not all 0 after splitting:

[0024]

[0025] The dilation coefficients dilate_x and dilate_y, after the convolution kernel is expanded, dilate_x-1 zeros are filled between adjacent horizontal real convolution kernel data, and dilate_y-1 zeros are filled between adjacent vertical real convolution kernel data; when the dilation coefficients dilate_x and dilate_y are greater than 5, the expanded convolution kernel is split, and the size of each small convolution kernel after the split does not exceed 3x3, and at least one of the small convolution kernel data after the split is all 0. At this time, the convolution calculation of the small convolution kernel and the corresponding input image is not performed, thereby reducing the number of NNA startups and improving the calculation speed of the dilation convolution.

[0026] The method further comprises the following steps:

[0027] S1, for dilated convolution, due to the limitation of NNA register parameter range, when the size of the convolution kernel exceeds 3x3 after expansion, the convolution kernel matrix needs to be split:

[0028] S1.1, dilated convolution kernel: input image size IH*IW, convolution kernel matrix size KH*KW, input image edge padding size is pad_top, pad_bottom, pad_left, pad_right, convolution step size is stride_x, stride_y, convolution kernel void coefficient is dilate_x, dilate_y, the dilated convolution kernel size dilate_kh and dilate_kw are calculated by the following formula:

[0029] dilate_kh=dilate_y*(KH-1)+1

[0030] dilate_kw=dilate_x*(KW-1)+1

[0031] The convolution kernel sizes KH and KW are both 3, the dilation coefficients dilate_y and dilate_x are both 6, and the expanded convolution kernel sizes dilate_kh and dilate_kw are both 13 according to the formula. NNA supports a maximum of 3x3 convolution kernels.

[0032] S1.2, split the convolution kernel matrix:

[0033] Split the convolution kernel of size 13x13 after expansion:

[0034] n=(dilate_kh+2) / 3=5

[0035] m=(dilate_kw+2) / 3=5

[0036] Where n is the count of the vertical split of the dilated convolution kernel, m is the count of the horizontal split of the dilated convolution kernel, n*m=25 is the total number of convolution kernels after splitting, and the size of each convolution kernel after splitting does not exceed 3*3:

[0037]

[0038] Among them, the convolution kernel size of K11 to K(n-1)(m-1) is 3*3, the convolution kernel size of K1m to K(n-1)m is 3*(dilate_kw%3), the convolution kernel size of Kn1 to Kn(m-1) is (dilate_kh%3)*3, and the convolution kernel size of Knm is (dilate_kh%3)*(dilate_kw%3), % means taking the remainder;

[0039] That is, the convolution kernel with a size of 13x13 after expansion is split into 25 small convolution kernels, of which the convolution kernel size from K11 to K44 is 3x3, the convolution kernel size from K15 to K45 is 3x1, the convolution kernel size from K51 to K54 is 1x3, and the convolution kernel size from K55 is 1x1;

[0040] The input image size is IH*IW, and the edge padding of the input image is not performed, that is, pad_top, pad_bottom, pad_left, and pad_right are all 0, the convolution kernel size KH and KW are both 3, the convolution step size stride_x and stride_y are both 1, and the convolution kernel void coefficient dilate_x and dilate_y are both 6, then the output image size OH*OW is obtained:

[0041] dilate_kh=dilate_y*(KH-1)+1=13

[0042] dilate_kw=dilate_x*(KW-1)+1=13

[0043]

[0044]

[0045] S2, NNA's dilated convolution process: When the dilation coefficient is greater than 5, the expanded convolution kernel is split. After the split, the convolution kernel data of a part of the convolution kernel is all 0. Then, only the convolution kernel data of which is not all 0 is written to WRAM, and then the required input image data is written to FRAM. The convolution kernel size is set through the NNA hardware register, where KH and KW must be less than or equal to 3, pad, and stride parameters, and then NNA is started to perform convolution calculation to obtain the convolution result output.

[0046] The splitting in step S1.2 further comprises:

[0047] The convolution kernel size is 3x3, the dilation coefficient is 6, and the size of the convolution kernel after expansion is 13x13. NNA supports a maximum convolution kernel of 3x3. The 13x13 convolution kernel after expansion is split into 25 small convolution kernels. The convolution kernel matrix dimension is less than or equal to 3x3, as shown below:

[0048]

[0049] Convolution kernel data arrangement before expansion;

[0050]

[0051]

[0052] Convolution kernel data arrangement after expansion.

[0053] In the step S2, if NNA is started for calculation for each of the 25 small convolution kernels after splitting, the operation speed will be relatively slow and cannot meet the requirements of practical applications. In order to ensure the operation speed, only the small convolution kernels whose convolution kernel data after splitting are not all 0, i.e., the gray area, are started for calculation by NNA. In this way, only convolution calculation of 9 small convolution kernels is required.

[0054] In the step S2, the convolution kernel data is sequentially written into the WRAM according to the size of the split convolution kernel, and the required input data is written into the FRAM at one time, further comprising:

[0055] S2.1, set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x3 convolution;

[0056] S2.2, set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x3 convolution;

[0057] S2.3, set the convolution kernel size to 3x1 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x1 convolution;

[0058] S2.4, set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x3 convolution;

[0059] S2.5, set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x3 convolution;

[0060] S2.6, set the convolution kernel size to 3x1 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x1 convolution;

[0061] S2.7, set the convolution kernel size to 1x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 1x3 convolution;

[0062] S2.8, set the convolution kernel size to 1x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 1x3 convolution;

[0063] S2.9, set the convolution kernel size to 1x1 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 1x1 convolution.

[0064] The void coefficient greater than 5 includes at least coefficients 8, 10, 12, 24, and 36.

[0065] Therefore, the advantage of this application is that this method will consider that when the hole coefficient is greater than 5, some of the data of the split small convolution kernels are all 0. At this time, these small convolution kernels are not convolved with the corresponding input image, which can greatly reduce the number of operations, thereby improving the operation speed. Increasing the operation speed of the hole convolution when the hole coefficient is large solves the problem of reduced efficiency caused by the large hole coefficient in the general method. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0067] Figure 1 is a schematic diagram of the process of this method. DETAILED DESCRIPTION

[0068] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0069] The embodiment of the present application relates to an acceleration method for dilated convolution when the dilated coefficient is greater than 5. Due to the limitation of the NNA register parameter range, it can only process convolution calculations with a convolution kernel matrix dimension less than or equal to 3x3, and its versatility is not high in practical applications. Although the implementation method of the general dilated convolution adopted to solve the versatility can meet the versatility, the efficiency of the general method is low and the operation speed is slow in some special scenarios (the dilated coefficient is greater than 5, such as QR code, voice and other applications). In order to improve the operation speed of dilated convolution in these special scenarios and make it more real-time in practical applications, this method is proposed.

[0070] The input image size is IH*IW, the convolution kernel matrix size is KH*KW, the input image edge padding size is pad_top, pad_bottom, pad_left, pad_right, the convolution step size is stride_x, stride_y, the convolution kernel void coefficient is dilate_x, dilate_y, and the expanded convolution kernel size dilate_kh and dilate_kw can be calculated by the following formula:

[0071] dilate_kh=dilate_y*(KH-1)+1

[0072] dilate_kw=dilate_x*(KW-1)+1

[0073] Specific implementation:

[0074] The convolution kernel size is 3x3. If the dilation coefficient is 2, the dilated convolution kernel size is 5x5; if the dilation coefficient is 3, the dilated convolution kernel size is 7x7.

[0075]

[0076] 3x3 convolution kernel data arrangement

[0077]

[0078] The dilated convolution kernel has a dilation coefficient of 2.

[0079]

[0080]

[0081] The dilated convolution kernel has a dilation coefficient of 3.

[0082] The size of the final output image OH*OW can be calculated by the following formula:

[0083] dilate_kh=dilate_y*(KH-1)+1

[0084] dilate_kw=dilate_x*(KW-1)+1

[0085]

[0086]

[0087] NNA convolution process: write the convolution kernel data into WRAM, then write the required input image data into FRAM, set the convolution kernel size (KH and KW must be less than or equal to 3), pad, stride and other parameters through the NNA hardware register, and then start NNA for convolution calculation to obtain the convolution result output.

[0088] For dilated convolution, due to the limitation of NNA register parameter range, when the dilation coefficient is greater than 5, the size of the convolution kernel exceeds 3x3 after expansion, and the convolution kernel matrix needs to be split.

[0089] Specific implementation: Assuming the convolution kernel size is 3x3, the dilation coefficient is 6, and the size of the expanded convolution kernel is 13x13, NNA can support a maximum of 3x3 convolution kernels. Split the expanded 13x13 convolution kernel into 25 small convolution kernels (the size of each small convolution kernel does not exceed 3x3).

[0090]

[0091] Convolution kernel data arrangement before expansion

[0092]

[0093]

[0094] Convolution kernel data arrangement after expansion

[0095] If NNA is started for calculation for each of the 25 small convolution kernels after splitting, the operation speed will be slow and cannot meet the requirements of practical applications. In order to ensure the operation speed, NNA is started only for the small convolution kernels (gray area) whose convolution kernel data after splitting is not all 0. In this way, only convolution calculations of 9 small convolution kernels are required, which greatly reduces the number of NNA startups and thus improves the operation speed.

[0096] Write 9 small convolution kernels whose convolution kernel data are not all 0 into WRAM, and write the required input data into FRAM at one time.

[0097] (1) Set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x3 convolution;

[0098] (2) Set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x3 convolution;

[0099] (3) Set the convolution kernel size to 3x1 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x1 convolution;

[0100] (4) Set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x3 convolution;

[0101] (5) Set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x3 convolution;

[0102] (6) Set the convolution kernel size to 3x1 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate the 3x1 convolution;

[0103] (7) Set the convolution kernel size to 1x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 1x3 convolution;

[0104] (8) Set the convolution kernel size to 1x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 1x3 convolution;

[0105] (9) Set the convolution kernel size to 1x1 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 1x1 convolution;

[0106] Finally, the dilated convolution output is obtained.

[0107] In summary, if Figure 1 As shown, the specific embodiment of the method includes the following steps:

[0108] S1, for dilated convolution, due to the limitation of NNA register parameter range, when the size of the convolution kernel exceeds 3x3 after expansion, the convolution kernel matrix needs to be split:

[0109] S1.1, dilated convolution kernel: input image size IH*IW, convolution kernel matrix size KH*KW, input image edge padding size is pad_top, pad_bottom, pad_left, pad_right, convolution step size is stride_x, stride_y, convolution kernel void coefficient is dilate_x, dilate_y, the dilated convolution kernel size dilate_kh and dilate_kw are calculated by the following formula:

[0110] dilate_kh=dilate_y*(KH-1)+1

[0111] dilate_kw=dilate_x*(KW-1)+1

[0112] The convolution kernel sizes KH and KW are both 3, the dilation coefficients dilate_y and dilate_x are both 6, and the expanded convolution kernel sizes dilate_kh and dilate_kw are both 13 according to the formula. NNA supports a maximum of 3x3 convolution kernels.

[0113] S1.2, split the convolution kernel matrix:

[0114] Split the convolution kernel of size 13x13 after expansion:

[0115] n=(dilate_kh+2) / 3=5

[0116] m=(dilate_kw+2) / 3=5

[0117] Where n is the count of the vertical split of the dilated convolution kernel, m is the count of the horizontal split of the dilated convolution kernel, n*m=25 is the total number of convolution kernels after splitting, and the size of each convolution kernel after splitting does not exceed 3*3:

[0118]

[0119] Among them, the convolution kernel size of K11 to K(n-1)(m-1) is 3*3, the convolution kernel size of K1m to K(n-1)m is 3*(dilate_kw%3), the convolution kernel size of Kn1 to Kn(m-1) is (dilate_kh%3)*3, and the convolution kernel size of Knm is (dilate_kh%3)*(dilate_kw%3), % means taking the remainder;

[0120] That is, the convolution kernel with a size of 13x13 after expansion is split into 25 small convolution kernels, of which the convolution kernel size from K11 to K44 is 3x3, the convolution kernel size from K15 to K45 is 3x1, the convolution kernel size from K51 to K54 is 1x3, and the convolution kernel size from K55 is 1x1;

[0121] The input image size is IH*IW, and the edge padding of the input image is not performed, that is, pad_top, pad_bottom, pad_left, and pad_right are all 0, the convolution kernel size KH and KW are both 3, the convolution step size stride_x and stride_y are both 1, and the convolution kernel void coefficient dilate_x and dilate_y are both 6, then the output image size OH*OW is obtained:

[0122] dilate_kh=dilate_y*(KH-1)+1=13

[0123] dilate_kw=dilate_x*(KW-1)+1=13

[0124]

[0125]

[0126] S2, NNA's dilated convolution process: When the dilation coefficient is greater than 5, the expanded convolution kernel is split. After the split, the convolution kernel data of a part of the convolution kernel is all 0. Then, only the convolution kernel data of which is not all 0 is written to WRAM, and then the required input image data is written to FRAM. The convolution kernel size is set through the NNA hardware register, where KH and KW must be less than or equal to 3, pad, and stride parameters, and then NNA is started to perform convolution calculation to obtain the convolution result output.

[0127] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for accelerating dilated convolution when the dilated coefficient is greater than 5, characterized in that: When the dilation coefficient is greater than 5, the expanded convolution kernel is split, and the convolution kernel data of a part of the small convolution kernels after the split are all 0. At this time, these small convolution kernels are not convolved with the corresponding input image. The method includes: Assuming the convolution kernel size is 3x3, when the dilate coefficients dilate_x and dilate_y are greater than 5, as shown in the following table, the ellipsis in the table indicates that it is greater than 6: when the dilate coefficient is equal to 6, 5 zeros are filled between adjacent convolution kernel elements; when the dilate coefficient is equal to 7, 6 zeros are filled between adjacent convolution kernel elements, and the ellipsis indicates 1 zero; when the dilate coefficient is equal to 8, 7 zeros are filled between adjacent convolution kernel elements, and the ellipsis indicates 2 zeros, and so on. The convolution kernel before expansion is: ； Convolution kernel after expansion: The gray part in the table indicates the small convolution kernel whose convolution kernel data is not all 0 after splitting: The dilation coefficients are dilate_x and dilate_y. After the convolution kernel is expanded, dilate_x-1 zeros are filled between adjacent horizontal real convolution kernel data, and dilate_y-1 zeros are filled between adjacent vertical real convolution kernel data. When the dilation coefficients are dilate_x and dilate_y greater than 5, the expanded convolution kernel is split, and the size of each small convolution kernel after the split does not exceed 3x3. At least one of the small convolution kernel data after the split is all 0. At this time, the convolution calculation is not performed on the small convolution kernel and the corresponding input image.

2. The acceleration method of dilated convolution when the dilated coefficient is greater than 5 according to claim 1, characterized in that: The method further comprises the following steps: S1, for dilated convolution, due to the limitation of NNA register parameter range, when the size of the convolution kernel exceeds 3x3 after expansion, the convolution kernel matrix needs to be split: S1.1, dilated convolution kernel: input image size IH*IW, convolution kernel matrix size KH*KW, input image edge padding size is pad_top, pad_bottom, pad_left, pad_right, convolution step size is stride_x, stride_y, convolution kernel void coefficient is dilate_x, dilate_y, the dilated convolution kernel size dilate_kh and dilate_kw are calculated by the following formula: dilate_kh=dilate_y*(KH-1)+1 dilate_kw=dilate_x*(KW-1)+1 The convolution kernel sizes KH and KW are both 3, and the dilation coefficients dilate_y and dilate_x are both 6. The formula is used to get the size of the expanded convolution kernel dilate_kh and dilate_kw, both of which are 13. NNA supports a maximum of 3x3 convolution kernels. S1.2, split the convolution kernel matrix: Split the convolution kernel of size 13x13 after expansion: n=(dilate_kh+2) / 3=5 m=(dilate_kw+2) / 3=5 Where n is the count of the vertical split of the dilated convolution kernel, m is the count of the horizontal split of the dilated convolution kernel, n*m=25 is the total number of convolution kernels after splitting, and the size of each convolution kernel after splitting does not exceed 3*3: Among them, the convolution kernel size of K11 to K(n-1)(m-1) is 3*3, the convolution kernel size of K1m to K(n-1)m is 3*(dilate_kw%3), the convolution kernel size of Kn1 to Kn(m-1) is (dilate_kh%3)*3, and the convolution kernel size of Knm is (dilate_kh%3)*(dilate_kw%3), % means taking the remainder; That is, the convolution kernel with a size of 13x13 after expansion is split into 25 small convolution kernels, of which the convolution kernel size from K11 to K44 is 3x3, the convolution kernel size from K15 to K45 is 3x1, the convolution kernel size from K51 to K54 is 1x3, and the convolution kernel size from K55 is 1x1; The input image size is IH*IW, and the edge padding of the input image is not performed, that is, pad_top, pad_bottom, pad_left, and pad_right are all 0, the convolution kernel size KH and KW are both 3, the convolution step size stride_x and stride_y are both 1, and the convolution kernel void coefficient dilate_x and dilate_y are both 6, then the output image size OH*OW is obtained: dilate_kh=dilate_y*(KH-1)+1=13 dilate_kw=dilate_x*(KW-1)+1=13 S2, NNA's dilated convolution process: When the dilation coefficient is greater than 5, the expanded convolution kernel is split. After the split, the convolution kernel data of a part of the convolution kernel is all 0. Then only the convolution kernel data of which is not all 0 is written to WRAM, and then the required input image data is written to FRAM. The convolution kernel size is set through the NNA hardware register, where KH and KW must be less than or equal to 3, pad, and stride parameters, and then NNA is started to perform convolution calculation to obtain the convolution result output.

3. The acceleration method of dilated convolution when the dilated coefficient is greater than 5 according to claim 1, It is characterized in that The splitting in step S1.2 further comprises: The convolution kernel size is 3x3, the dilation coefficient is 6, and the size of the convolution kernel after expansion is 13x13. NNA supports a maximum convolution kernel of 3x3. The 13x13 convolution kernel after expansion is split into 25 small convolution kernels. The convolution kernel matrix dimension is less than or equal to 3x3, as shown below: Convolution kernel data arrangement before expansion; Convolution kernel data arrangement after expansion.

4. The acceleration method of dilated convolution when the dilated coefficient is greater than 5 according to claim 3, characterized in that: In the step S2, if NNA is started for calculation for each of the 25 small convolution kernels after splitting, the operation speed will be relatively slow and cannot meet the requirements of practical applications. In order to ensure the operation speed, only the small convolution kernels whose convolution kernel data after splitting are not all 0, i.e., the gray area, are started for calculation by NNA. In this way, only convolution calculation of 9 small convolution kernels is required.

5. The acceleration method of dilated convolution when the dilated coefficient is greater than 5 according to claim 4, characterized in that: In the step S2, the convolution kernel data is sequentially written into the WRAM according to the size of the split convolution kernel, and the required input data is written into the FRAM at one time, further comprising: S2.1, set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x3 convolution; S2.2, set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x3 convolution; S2.3, set the convolution kernel size to 3x1 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x1 convolution; S2.4, set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x3 convolution; S2.5, set the convolution kernel size to 3x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x3 convolution; S2.6, set the convolution kernel size to 3x1 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 3x1 convolution; S2.7, set the convolution kernel size to 1x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 1x3 convolution; S2.8, set the convolution kernel size to 1x3 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 1x3 convolution; S2.9, set the convolution kernel size to 1x1 by configuring the NNA register, set the read address of WRAM and FRAM, and start NNA to calculate 1x1 convolution.

6. The acceleration method of dilated convolution when the dilated coefficient is greater than 5 according to claim 1, characterized in that: The void coefficient greater than 5 includes at least coefficients 8, 10, 12, 24, and 36.

Images