A method for accelerating a convolution with a cavity coefficient greater than 5

Abstract

The application provides an acceleration method for a cavity convolution when a cavity coefficient is greater than 5, comprising the following steps: S1, for the cavity convolution, due to the range limitation of the NNA register parameter, when the cavity coefficient is greater than 5, the size of the expanded convolution kernel exceeds 3x3, and it is necessary to split the convolution kernel matrix; S2, the cavity convolution process of the NNA: when the cavity coefficient is greater than 5, the convolution kernel data of a part of the split convolution kernel is all 0, only the convolution kernel data which is not all 0 is written into the WRAM, the required input image data is written into the FRAM, the size of the convolution kernel is set through the NNA hardware register, wherein KH and KW must be less than or equal to 3, the pad and the stride parameters, and then the NNA is started to perform the convolution calculation, and the convolution result output is obtained.

Description

Technical Field

[0001] This invention belongs to the field of neural network computing technology, and specifically relates to an accelerated method for dilated convolution when the dilation coefficient is greater than 5. Background Technology

[0002] In current technologies, neural networks, as mathematical models that mimic the structure and function of biological neural networks, are increasingly widely used. By learning the inherent patterns in training sample data, they gain the ability to analyze or express sample data, and can be applied to various fields such as object detection, scene classification, and character recognition. The process and methods of training neural networks are known as deep learning.

[0003] Convolutional neural networks have wide applications in image, video, and speech processing. As neural networks become more complex, they require a lot of resources for training and evaluation, while the performance growth and versatility improvement of hardware accelerators are slowing down.

[0004] Convolutional neural networks (CNNs) mainly consist of an input layer, convolutional layers, pooling layers, and fully connected layers. Among them, the convolutional layer is the core layer of the CNN, generating the vast majority of the computational load. Therefore, the operating speed of a CNN basically depends on the computational speed of the convolutional layer.

[0005] NNA (Neural Network Accelerator) is a high-performance, low-power neural network accelerator for processing convolutional computations. It has its own instruction set, offering flexible programmability suitable for different types of neural networks. By configuring relevant register parameters, it can accelerate convolutional computations with kernel matrix dimensions of 3x3 or less, thereby reducing neural network runtime and providing higher real-time performance and a better user experience in practical applications. The NNA 1.0 neural network accelerator supports fast computation with convolutional kernel matrix dimensions of 3x3 or less. The dilation factor is the extent to which the convolutional kernel matrix is ​​expanded. Dilated convolution fills the elements of the convolutional kernel with zeros. The dilation factor controls the size of the expanded convolutional kernel, thereby extracting multi-scale information. The larger the dilation factor, the larger the convolutional kernel.

[0006] However, the convolution process involves a large number of multiplication and accumulation calculations, which is slow and not real-time in practical applications. NNA only supports fast operation when the convolution kernel matrix dimension is less than or equal to 3x3. When the dilation coefficient is large, the dimension of the dilated convolution kernel matrix is ​​much larger than 3x3. Even if NNA is started to accelerate the dilated convolution process by splitting the convolution kernel, the operation speed is still slow and cannot meet the requirements of practical applications.

[0007] In addition, commonly used terms in the prior art include:

[0008] 1. Image Convolution: For each pixel in the input image, the product of its neighboring pixels and the corresponding element of the convolution kernel matrix is ​​calculated and accumulated to obtain the value of that pixel position. By continuously sliding the convolution kernel matrix, the final convolution output result is obtained.

[0009] 2. FRAM: On-chip RAM that stores input image data inside the NNA.

[0010] 3. WRAM: On-chip RAM that stores convolution kernel data inside the NNA.

[0011] 4. pixel: The smallest unit of input image.

[0012] 5. pad: Fills the edges of the input image, divided into pad_top, pad_bottom, pad_left, and pad_right, which represent the size of the padding on the top, bottom, left, and right edges of the image.

[0013] 6. stride: The stride of the convolution kernel matrix, divided into stride_x and stride_y, representing the stride of the convolution kernel matrix in the horizontal and vertical directions.

[0014] 7. dilate: The extent of kernel expansion, also called the dilation coefficient, is divided into dilate_x and dilate_y, which represent the extent of horizontal and vertical expansion of the kernel. Summary of the Invention

[0015] To address the aforementioned issues, the purpose of this application is as follows: General dilated convolution methods calculate the convolution result of each small convolution kernel after splitting the dilated convolution kernel with the corresponding input image. However, when the dilation coefficient is greater than 5, i.e. in some special scenarios (such as QR codes, voice applications, etc.), the general method is inefficient and slow. In order to improve the computation speed of dilated convolution in these special scenarios and make it more real-time in practical applications, this method is proposed.

[0016] NNA convolution is accelerated by splitting the convolution kernel. Specifically:

[0017] 1. Kernel splitting: For dilated convolution, when the size of the expanded convolution kernel exceeds the range of NNA register parameters, NNA convolution is accelerated by splitting the convolution kernel.

[0018] 2. NNA Register Configuration: To accelerate convolution using NNA, the relevant NNA registers must be configured correctly. Parameters such as the stride and kernel size of the convolution are configured through the NNA registers. These parameters must be within the range supported by NNA to obtain the correct convolution results.

[0019] Specifically, this invention provides a method for accelerating dilated convolution when the dilation coefficient is greater than 5. When the dilation coefficient is greater than 5, the dilated convolution kernel is split, and the kernel data of a portion of the smaller convolution kernels are all 0. At this time, these smaller convolution kernels are not convolved with the corresponding input image. The method includes:

[0020] Assuming a convolutional kernel size of 3x3, and dilate_x and dilate_y are greater than 5, the following table shows the results. Values ​​greater than 5 are listed in the table below. Ellipses in the table indicate values ​​greater than 6: A dilate_x of 6 means five zeros are padded between adjacent kernel elements; a dilate_y of 7 means six zeros are padded between adjacent kernel elements (ellipse represents one zero); a dilate_y of 8 means seven zeros are padded between adjacent kernel elements (ellipse represents two zeros), and so on. The convolutional kernel before dilation is as follows:

[0021] k11 k12 k13 K21 K22 K23 K31 K32 K33 ;

[0023] Dilated convolution kernels: The gray areas in the table represent smaller convolution kernels where the kernel data is not all zero after splitting.

[0024]

[0025] The dilation coefficients dilate_x and dilate_y are used to fill the gaps between adjacent horizontal true convolutional kernel data and between adjacent vertical true convolutional kernel data after kernel expansion. When the dilation coefficients dilate_x and dilate_y are greater than 5, the expanded convolutional kernel is split, with each split kernel being no larger than 3x3 and at least one split kernel containing all zeros. In this case, no convolution calculation is performed between this split kernel and the corresponding input image, thereby reducing the number of NNA activations and improving the speed of dilated convolution.

[0026] The method further includes the following steps:

[0027] S1. For dilated convolutions, due to the limitation of the NNA register parameter range, when the size of the dilated convolution kernel exceeds 3x3, the convolution kernel matrix needs to be split:

[0028] S1.1, Dilated Convolution Kernel: Input image size IH*IW, convolution kernel matrix size KH*KW, input image edge padding sizes pad_top, pad_bottom, pad_left, pad_right, convolution stride_x, stride_y, convolution kernel dilation coefficients dilate_x, dilate_y, and the dilated kernel sizes dilate_kh and dilate_kw are calculated using the following formulas:

[0029] dilate_kh=dilate_y*(KH-1)+1

[0030] dilate_kw=dilate_x*(KW-1)+1

[0031] The kernel sizes KH and KW are both 3, and the dilation coefficients dilate_y and dilate_x are both 6. The expanded kernel sizes dilate_kh and dilate_kw are both 13, obtained by formula. NNA supports a maximum of 3x3 kernels.

[0032] S1.2, split the convolution kernel matrix:

[0033] The expanded 13x13 convolutional kernel is split into smaller parts:

[0034] n = (dilate_kh + 2) / 3 = 5

[0035] m = (dilate_kw + 2) / 3 = 5

[0036] Where n is the count of the vertical splitting of the expanded convolutional kernel, m is the count of the horizontal splitting of the expanded convolutional kernel, and n*m=25 is the total number of split convolutional kernels. The size of each split convolutional kernel does not exceed 3*3.

[0037]

[0038] The kernel size for convolutions from K11 to K(n-1)(m-1) is 3*3, the kernel size for convolutions from K1m to K(n-1)m is 3*(dilate_kw%3), the kernel size for convolutions from Kn1 to Kn(m-1) is (dilate_kh%3)*3, and the kernel size for convolutions from Knm is (dilate_kh%3)*(dilate_kw%3), where % indicates the remainder.

[0039] That is, the expanded 13x13 convolutional kernel is split into 25 smaller convolutional kernels, of which K11 to K44 are 3x3, K15 to K45 are 3x1, K51 to K54 are 1x3, and K55 is 1x1.

[0040] Given an input image of size IH*IW, with no edge padding (pad_top, pad_bottom, pad_left, and pad_right are all 0), kernel sizes KH and KW both 3, stride_x and stride_y both 1, and dilate_x and dilate_y both 6, the resulting output image is size OH*OW.

[0041] dilate_kh=dilate_y*(KH-1)+1=13

[0042] dilate_kw=dilate_x*(KW-1)+1=13

[0043]

[0044]

[0045] S2, NNA dilated convolution process: When the dilation coefficient is greater than 5, the dilated convolution kernel is split. After splitting, the kernel data of some convolution kernels are all 0. Then, only the convolution kernels whose kernel data are not all 0 are written to WRAM. Then, the required input image data is written to FRAM. The convolution kernel size is set through the NNA hardware register, where KH and KW must be less than or equal to 3, and the pad and stride parameters are set. Then, NNA is started to perform convolution calculation and the convolution result is output.

[0046] The splitting in step S1.2 further includes:

[0047] The kernel size is 3x3 with a dilation coefficient of 6. The expanded kernel size is 13x13, and NNA supports a maximum of 3x3 kernels. The expanded 13x13 kernel is then split into 25 smaller kernels, with each kernel matrix having a dimension less than or equal to 3x3, as shown below:

[0048]

[0049] Pre-dilation convolutional kernel data arrangement;

[0050]

[0051]

[0052] The arrangement of the expanded convolutional kernel data.

[0053] In step S2, if NNA is activated for each of the 25 small convolution kernels after splitting, the calculation speed will be relatively slow and will not meet the requirements of practical applications. In order to ensure the calculation speed, NNA is activated only for the small convolution kernels whose data are not all 0 after splitting, i.e., the gray area. In this way, only 9 small convolution kernels need to be calculated.

[0054] In step S2, the convolution kernel data is written sequentially into the WRAM according to the size of the split convolution kernel, and the required input data is written into the FRAM all at once. This further includes:

[0055] S2.1, Configure the NNA register to set the convolution kernel size to 3x3, set the read addresses of WRAM and FRAM, and start NNA to calculate the 3x3 convolution;

[0056] S2.2, configure the NNA register to set the convolution kernel size to 3x3, set the read addresses of WRAM and FRAM, and start NNA to calculate the 3x3 convolution;

[0057] S2.3, configure the NNA register to set the convolution kernel size to 3x1, set the read addresses of WRAM and FRAM, and start NNA to calculate the 3x1 convolution;

[0058] S2.4, configure the NNA register to set the convolution kernel size to 3x3, set the read addresses of WRAM and FRAM, and start NNA to calculate the 3x3 convolution;

[0059] S2.5, configure the NNA register to set the convolution kernel size to 3x3, set the read addresses of WRAM and FRAM, and start NNA to calculate the 3x3 convolution;

[0060] S2.6, configure the NNA register to set the convolution kernel size to 3x1, set the read addresses of WRAM and FRAM, and start NNA to calculate the 3x1 convolution;

[0061] S2.7, configure the NNA register to set the convolution kernel size to 1x3, set the read addresses of WRAM and FRAM, and start NNA to calculate the 1x3 convolution;

[0062] S2.8, by configuring the NNA register, the convolution kernel size is set to 1x3, the read addresses of WRAM and FRAM are set, and the NNA is started to calculate the 1x3 convolution;

[0063] S2.9, by configuring the NNA register, set the convolution kernel size to 1x1, set the read addresses of WRAM and FRAM, and start NNA to calculate the 1x1 convolution.

[0064] The void coefficient greater than 5 includes at least coefficients 8, 10, 12, 24, and 36.

[0065] Therefore, the advantage of this application is that: when the dilation coefficient is greater than 5, some small convolution kernels after splitting have all data as 0. In this case, these small convolution kernels are not convolved with the corresponding input image, which can greatly reduce the number of operations and thus improve the operation speed. Improving the operation speed of dilated convolution when the dilation coefficient is large effectively solves the problem of efficiency reduction caused by large dilation coefficients in general methods. Attached Figure Description

[0066] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0067] Figure 1 This is a flowchart illustrating the method. Detailed Implementation

[0068] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0069] This application relates to an acceleration method for dilated convolution when the dilation coefficient is greater than 5. Due to the limitation of the NNA register parameter range, it can only handle convolution calculations with a kernel matrix dimension of less than or equal to 3x3, which has low versatility in practical applications. While the general dilated convolution implementation method used to address versatility can meet the general requirements, its efficiency is low and its computation speed is slow in some special scenarios (dilation coefficient greater than 5, such as QR codes, voice applications, etc.). In order to improve the computation speed of dilated convolution in these special scenarios and make it more real-time in practical applications, this method is proposed.

[0070] Given an input image size of IH*IW, a convolution kernel matrix size of KH*KW, input image edge padding sizes of pad_top, pad_bottom, pad_left, and pad_right, convolution strides of stride_x and stride_y, and convolution kernel dilation coefficients of dilate_x and dilate_y, the dilated kernel sizes of dilate_kh and dilate_kw can be calculated using the following formulas:

[0071] dilate_kh=dilate_y*(KH-1)+1

[0072] dilate_kw=dilate_x*(KW-1)+1

[0073] Specific implementation:

[0074] The kernel size is 3x3. If the dilation coefficient is 2, the kernel size after dilation is 5x5; if the dilation coefficient is 3, the kernel size after dilation is 7x7.

[0075]

[0076] 3x3 convolution kernel data arrangement

[0077]

[0078] The hole coefficient is 2, and the expanded convolution kernel

[0079]

[0080]

[0081] The hole coefficient is 3, and the expanded convolution kernel

[0082] The final output image size OH*OW can be calculated using the following formula:

[0083] dilate_kh=dilate_y*(KH-1)+1

[0084] dilate_kw=dilate_x*(KW-1)+1

[0085]

[0086]

[0087] NNA convolution process: Write the convolution kernel data to WRAM, then write the required input image data to FRAM, set the convolution kernel size (KH and KW must be less than or equal to 3), pad, stride and other parameters through the NNA hardware registers, then start NNA to perform convolution calculation and get the convolution result output.

[0088] For dilated convolution, due to the limitation of the NNA register parameter range, when the dilation coefficient is greater than 5, the size of the dilated convolution kernel exceeds 3x3, and the convolution kernel matrix needs to be split.

[0089] Implementation details: Assuming the kernel size is 3x3 and the dilation coefficient is 6, the expanded kernel size is 13x13. NNA can support a maximum of 3x3 kernels. The expanded 13x13 kernel is then split into 25 smaller kernels (each smaller kernel is no larger than 3x3).

[0090]

[0091] Pre-dilation kernel data arrangement

[0092]

[0093]

[0094] Dilated convolution kernel data arrangement

[0095] If NNA is activated for each of the 25 small convolution kernels after splitting, the computation speed will be relatively slow and will not meet the requirements of practical applications. In order to ensure the computation speed, NNA is activated only for the small convolution kernels (gray area) whose data is not all zero after splitting. This way, only 9 small convolution kernels need to be calculated, which greatly reduces the number of times NNA is activated and thus improves the computation speed.

[0096] Nine small convolutional kernels with non-zero kernel data are written into WRAM, and the required input data is written into FRAM all at once.

[0097] (1) Configure the NNA register to set the convolution kernel size to 3x3, set the read addresses of WRAM and FRAM, and start NNA to calculate 3x3 convolution;

[0098] (2) Configure the NNA register to set the convolution kernel size to 3x3, set the read addresses of WRAM and FRAM, and start NNA to calculate the 3x3 convolution;

[0099] (3) Configure the NNA register to set the convolution kernel size to 3x1, set the read addresses of WRAM and FRAM, and start NNA to calculate the 3x1 convolution;

[0100] (4) Configure the NNA register to set the convolution kernel size to 3x3, set the read addresses of WRAM and FRAM, and start NNA to calculate the 3x3 convolution;

[0101] (5) Configure the NNA register to set the convolution kernel size to 3x3, set the read addresses of WRAM and FRAM, and start NNA to calculate the 3x3 convolution;

[0102] (6) Configure the NNA register to set the convolution kernel size to 3x1, set the read addresses of WRAM and FRAM, and start NNA to calculate the 3x1 convolution;

[0103] (7) Configure the NNA register to set the convolution kernel size to 1x3, set the read addresses of WRAM and FRAM, and start NNA to calculate the 1x3 convolution;

[0104] (8) Configure the NNA register to set the convolution kernel size to 1x3, set the read addresses of WRAM and FRAM, and start NNA to calculate the 1x3 convolution;

[0105] (9) Configure the NNA register to set the convolution kernel size to 1x1, set the read addresses of WRAM and FRAM, and start NNA to calculate 1x1 convolution;

[0106] Finally, the dilated convolution output is obtained.

[0107] In summary, such as Figure 1 As shown, a specific embodiment of this method includes the following steps:

[0108] S1. For dilated convolutions, due to the limitation of the NNA register parameter range, when the size of the dilated convolution kernel exceeds 3x3, the convolution kernel matrix needs to be split:

[0109] S1.1, Dilated Convolution Kernel: Input image size IH*IW, convolution kernel matrix size KH*KW, input image edge padding sizes pad_top, pad_bottom, pad_left, pad_right, convolution stride_x, stride_y, convolution kernel dilation coefficients dilate_x, dilate_y, and the dilated kernel sizes dilate_kh and dilate_kw are calculated using the following formulas:

[0110] dilate_kh=dilate_y*(KH-1)+1

[0111] dilate_kw=dilate_x*(KW-1)+1

[0112] The kernel sizes KH and KW are both 3, and the dilation coefficients dilate_y and dilate_x are both 6. The expanded kernel sizes dilate_kh and dilate_kw are both 13, obtained by formula. NNA supports a maximum of 3x3 kernels.

[0113] S1.2, split the convolution kernel matrix:

[0114] The expanded 13x13 convolutional kernel is split into smaller parts:

[0115] n = (dilate_kh + 2) / 3 = 5

[0116] m = (dilate_kw + 2) / 3 = 5

[0117] Where n is the count of the vertical splitting of the expanded convolutional kernel, m is the count of the horizontal splitting of the expanded convolutional kernel, and n*m=25 is the total number of split convolutional kernels. The size of each split convolutional kernel does not exceed 3*3.

[0118]

[0119] The kernel size for convolutions from K11 to K(n-1)(m-1) is 3*3, the kernel size for convolutions from K1m to K(n-1)m is 3*(dilate_kw%3), the kernel size for convolutions from Kn1 to Kn(m-1) is (dilate_kh%3)*3, and the kernel size for convolutions from Knm is (dilate_kh%3)*(dilate_kw%3), where % indicates the remainder.

[0120] That is, the expanded 13x13 convolutional kernel is split into 25 smaller convolutional kernels, of which K11 to K44 are 3x3, K15 to K45 are 3x1, K51 to K54 are 1x3, and K55 is 1x1.

[0121] Given an input image of size IH*IW, with no edge padding (pad_top, pad_bottom, pad_left, and pad_right are all 0), kernel sizes KH and KW both 3, stride_x and stride_y both 1, and dilate_x and dilate_y both 6, the resulting output image is size OH*OW.

[0122] dilate_kh=dilate_y*(KH-1)+1=13

[0123] dilate_kw=dilate_x*(KW-1)+1=13

[0124]

[0125]

[0126] S2, NNA dilated convolution process: When the dilation coefficient is greater than 5, the dilated convolution kernel is split. After splitting, the kernel data of some convolution kernels are all 0. Then, only the convolution kernels whose kernel data are not all 0 are written to WRAM. Then, the required input image data is written to FRAM. The convolution kernel size is set through the NNA hardware register, where KH and KW must be less than or equal to 3, and the pad and stride parameters are set. Then, NNA is started to perform convolution calculation and the convolution result is output.

[0127] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for accelerating dilated convolution when the dilation coefficient is greater than 5, characterized in that, When the dilation coefficient is greater than 5, the dilated convolution kernel is split. After splitting, the kernel data of a portion of the smaller convolution kernels are all 0. At this time, these smaller convolution kernels are not convolved with the corresponding input image. The method includes: Before dilation, the kernel size is 3x3. When the dilation coefficients `dilate_x` and `dilate_y` are greater than 5, the following table shows the results, where ellipses indicate values ​​greater than 6: a dilation coefficient of 6 means five zeros are padded between adjacent kernel elements; a dilation coefficient of 7 means six zeros are padded between adjacent kernel elements (in this case, ellipses represent one zero); a dilation coefficient of 8 means seven zeros are padded between adjacent kernel elements (in this case, ellipses represent two zeros), and so on. The kernel size before dilation is: ； Dilated convolution kernels: The gray areas in the table represent smaller convolution kernels where the kernel data is not all zero after splitting. ； The dilation coefficients dilate_x and dilate_y are used to fill the gaps between adjacent horizontal true convolutional kernel data and between adjacent vertical true convolutional kernel data after kernel expansion. When the dilation coefficients dilate_x and dilate_y are greater than 5, the expanded convolutional kernel is split, and the size of each split convolutional kernel is no more than 3x3. At least one split convolutional kernel has all zero data. In this case, no convolution calculation is performed between the split convolutional kernel and the corresponding input image.

2. The accelerated method for dilated convolution when the dilation coefficient is greater than 5 according to claim 1, characterized in that, The method further includes the following steps: S1. For dilated convolutions, due to the limitation of the NNA register parameter range, when the size of the dilated convolution kernel exceeds 3x3, the convolution kernel matrix needs to be split: S1.1, Dilated Convolution Kernel: Input image size IH*IW, convolution kernel matrix size KH*KW, input image edge padding sizes pad_top, pad_bottom, pad_left, pad_right, convolution stride_x, stride_y, convolution kernel dilation coefficients dilate_x, dilate_y, and the dilated kernel sizes dilate_kh and dilate_kw are calculated using the following formulas: ； The kernel sizes KH and KW are both 3, and the dilation coefficients dilate_y and dilate_x are both 6. The expanded kernel size, dilate_kh, and dilate_kw are both 13, obtained from the formula. NNA supports a maximum of 3x3 kernel size. S1.2, split the convolution kernel matrix: The expanded 13x13 convolutional kernel is split into smaller parts: ； Where n is the count of the vertical splitting of the expanded convolutional kernel, m is the count of the horizontal splitting of the expanded convolutional kernel, and n*m=25 is the total number of split convolutional kernels. The size of each split convolutional kernel does not exceed 3*3. ； The kernel size for convolutions from K11 to K(n-1)(m-1) is 3*3, the kernel size for convolutions from K1m to K(n-1)m is 3*(dilate_kw%3), the kernel size for convolutions from Kn1 to Kn(m-1) is (dilate_kh%3)*3, and the kernel size for convolutions from Knm is (dilate_kh%3)*(dilate_kw%3), where % indicates the remainder. That is, the expanded 13x13 convolutional kernel is split into 25 smaller convolutional kernels, of which K11 to K44 are 3x3, K15 to K45 are 3x1, K51 to K54 are 1x3, and K55 is 1x1. Given an input image of size IH*IW, with no edge padding (pad_top, pad_bottom, pad_left, and pad_right are all 0), kernel sizes KH and KW both 3, stride_x and stride_y both 1, and dilate_x and dilate_y both 6, the resulting output image is size OH*OW. ； S2, NNA dilated convolution process: When the dilation coefficient is greater than 5, the dilated convolution kernel is split. After splitting, the kernel data of some convolution kernels are all 0. Then, only the convolution kernels whose kernel data are not all 0 are written to WRAM. Then, the required input image data is written to FRAM. The convolution kernel size, pad, and stride parameters are set through the NNA hardware register. KH and KW must be less than or equal to 3. Then, NNA is started to perform convolution calculation and the convolution result is output.

3. The accelerated method for dilated convolution when the dilation coefficient is greater than 5 according to claim 2, characterized in that, The splitting in step S1.2 further includes: The kernel size is 3x3 with a dilation coefficient of 6. The expanded kernel size is 13x13, and NNA supports a maximum of 3x3 kernels. The expanded 13x13 kernel is then split into 25 smaller kernels, with each kernel matrix having a dimension less than or equal to 3x3, as shown below: ； Pre-dilation convolutional kernel data arrangement; ； The arrangement of the expanded convolutional kernel data.

4. The accelerated method for dilated convolution when the dilation coefficient is greater than 5 according to claim 3, characterized in that, In step S2, if NNA is activated for each of the 25 small convolution kernels after splitting, the calculation speed will be relatively slow and will not meet the requirements of practical applications. In order to ensure the calculation speed, NNA is activated only for the small convolution kernels whose data are not all 0 after splitting, i.e., the gray area. In this way, only 9 small convolution kernels need to be calculated.

5. The accelerated method for dilated convolution when the dilation coefficient is greater than 5 according to claim 4, characterized in that, In step S2, the convolution kernel data is written sequentially into the WRAM according to the size of the split convolution kernel, and the required input data is written into the FRAM all at once. This further includes: S2.1, Configure the NNA register to set the convolution kernel size to 3x3, set the read addresses of WRAM and FRAM, and start NNA to calculate the 3x3 convolution; S2.2, configure the NNA register to set the convolution kernel size to 3x3, set the read addresses of WRAM and FRAM, and start NNA to calculate the 3x3 convolution; S2.3, configure the NNA register to set the convolution kernel size to 3x1, set the read addresses of WRAM and FRAM, and start NNA to calculate the 3x1 convolution; S2.4, configure the NNA register to set the convolution kernel size to 3x3, set the read addresses of WRAM and FRAM, and start NNA to calculate the 3x3 convolution; S2.5, configure the NNA register to set the convolution kernel size to 3x3, set the read addresses of WRAM and FRAM, and start NNA to calculate the 3x3 convolution; S2.6, configure the NNA register to set the convolution kernel size to 3x1, set the read addresses of WRAM and FRAM, and start NNA to calculate the 3x1 convolution; S2.7, configure the NNA register to set the convolution kernel size to 1x3, set the read addresses of WRAM and FRAM, and start NNA to calculate the 1x3 convolution; S2.8, by configuring the NNA register, the convolution kernel size is set to 1x3, the read addresses of WRAM and FRAM are set, and the NNA is started to calculate the 1x3 convolution; S2.9, by configuring the NNA register, set the convolution kernel size to 1x1, set the read addresses of WRAM and FRAM, and start NNA to calculate the 1x1 convolution.

6. The accelerated method for dilated convolution when the dilation coefficient is greater than 5 according to claim 1, characterized in that, The void coefficient greater than 5 includes at least coefficients 8, 10, 12, 24, and 36.

Images