An implementation method for channel-wise convolution

Abstract

This invention provides a method for implementing channel-wise convolution. When encountering channel-wise convolution, the method accelerates NNA convolution by expanding the convolution kernel. The expanded multi-channel convolution kernel has the number of channels in each kernel equal to the number of input channels, and the total number of convolution kernels equals the number of input channels. To ensure that the convolution result obtained after kernel expansion is consistent with the channel-wise convolution result, it is necessary to calculate the padding values ​​of the remaining channel elements after expansion, except for the corresponding channel.

Description

Technical Field

[0001] This invention relates to the field of neural network technology, and in particular to a method for implementing channel-wise convolution. Background Technology

[0002] With the advent of the big data era, the application of neural network technology is becoming increasingly widespread, with data processing technology becoming one of the most important application technologies. Convolutional neural networks (CNNs) have wide applications in image, video, and speech processing, especially as artificial intelligence is increasingly incorporated into consumer electronics and automotive electronics. Artificial intelligence requires massive computation in model training and inference. Artificial intelligence is developing at an unprecedentedly rapid pace, with deep learning and neural network technologies entering a peak development phase. As neural networks become increasingly complex, they require substantial resources for training and evaluation, while the performance growth and versatility improvement of hardware accelerators are slowing down. A CNN mainly consists of an input layer, convolutional layers, pooling layers, and fully connected layers. The convolutional layer is the core layer of the CNN, generating the vast majority of the computational load. Therefore, the operating speed of a CNN largely depends on the computational speed of its convolutional layers. However, limited by the characteristics of its algorithm and computation itself, traditional chips, which were widely used in the past, can no longer meet the demands of massive computations. This necessitates chip manufacturers to create dedicated chips for neural network algorithms, especially inference chips, i.e., neural network accelerators (NNAs). Currently, NNAs (NNA 1.0) only support fast multi-channel convolution operations. Furthermore, the depthwise convolution process involves a large number of multiplication and accumulation operations, which is slow.

[0003] In other words, the existing technology has the following drawbacks:

[0004] Convolution involves numerous multiplication and accumulation calculations, resulting in slow computation speed and lacking real-time capability in practical applications. NNA 1.0 only supports fast multi-channel convolution; it cannot accelerate depthwise convolution. The speed of these calculations directly impacts the performance of convolutional networks. While NNA essentially supports fast multiplication and accumulation of ordinary unsigned inputs and weights, its hardware accelerator limitations restrict its versatility.

[0005] In addition, the commonly used terms in the prior art are as follows:

[0006] 1. Neural Network: A mathematical model that imitates the structure and function of biological neural networks. By learning the inherent laws of training sample data, it gains the ability to analyze or express sample data. It can be applied to various application fields such as target detection, scene classification, and character recognition.

[0007] 2. Deep learning: The process and methods of training neural networks.

[0008] 3. Multi-channel convolution: For each input pixel, calculate the product of its neighboring pixels and the corresponding convolution kernel channel element, and then sum them up. The values ​​of each channel are summed to obtain the final convolution result.

[0009] 4. Depthwise Convolution: For each input pixel, the product of its neighboring pixels and the corresponding kernel channel element is calculated and summed to obtain the final convolution result. The number of kernel channels is equal to 1, and the number of kernels is equal to the number of input channels and the number of output channels.

[0010] 5. NNA (neural network accelerate unit): NNA is a hardware-based neural network accelerator. By configuring relevant register parameters, it can achieve fast multi-channel convolution operations, thereby significantly reducing the running time of neural networks. In practical applications, it offers higher real-time performance and a better user experience.

[0011] 6. FRAM: On-chip RAM that stores input image data inside the NNA.

[0012] 7. WRAM: On-chip RAM that stores convolution kernel data inside the NNA.

[0013] 8. ORAM: On-chip general purpose RAM.

[0014] 9. pixel: The smallest unit of input image.

[0015] 10. pad: Fills the edges of the input image. It is divided into pad_top, pad_bottom, pad_left, and pad_right, which represent the size of the padding on the top, bottom, left, and right edges of the image.

[0016] 11. stride: The stride of the convolution kernel matrix, divided into stride_x and stride_y, representing the stride of the convolution kernel matrix in the horizontal and vertical directions. Summary of the Invention

[0017] To address the aforementioned issues, this application aims to provide a method for implementing depthwise convolution based on NNA, enabling fast computation of depthwise convolution. NNA hardware limitations include a kernel size of at least 3, a stride of at least 2, and the number of input channels for the kernel being a multiple of 32. When these NNA hardware limitations are not met, special processing of the weight data is required to accelerate convolution using NNA. Specifically, how can the correct depthwise convolution result be obtained using an NNA hardware accelerator when the input image data is ordinary unsigned data and the weight data is ordinary signed data?

[0018] Specifically, this invention provides a method for implementing channel-wise convolution. When encountering channel-wise convolution, the method accelerates NNA convolution by expanding the convolution kernel. The expanded multi-channel convolution kernel has the same number of channels as the input channels, and the total number of convolution kernels equals the number of input channels. The NNA hardware limitations include that the kernel size must be less than or equal to 3, the stride must be less than or equal to 2, and the number of input channels of the convolution kernel must be a multiple of 32. When the NNA hardware limitations are not met, special processing of the weight data is required to accelerate convolution using NNA: when the input image data is ordinary unsigned data and the weight data is ordinary signed data, the padding values ​​for the remaining input channels, excluding the corresponding input channels, are calculated during depthwise convolution kernel expansion.

[0019] The NNA supports fast computation of ordinary unsigned input and ordinary unsigned weight multiplication and accumulation. Multichannel convolution is the result of ordinary unsigned input and ordinary signed weight multiplication and accumulation. In actual use, the correct convolution result can be obtained by configuring the relevant NNA registers. NNA supports ordinary convolution, and the number of input channels IC of the convolution kernel is a multiple of 32. To use NNA to calculate depthwise convolution, the input channels of the depthwise convolution kernel must be expanded, and then the depthwise convolution is calculated in the same way as NNA calculates ordinary convolution. In order to obtain the correct depthwise convolution result, the padding values ​​of the other input channels besides the corresponding input channels need to be calculated to ensure that the depthwise convolution formula is consistent with the NNA formula for calculating ordinary convolution. Assume that the convolution kernel size is size, the number of input channels is IC, and the number of bits of the weights is nw.

[0020] Normal unsigned input is F u The ordinary signed weight is W. s The ordinary unsigned weight is W. u The conversion formula is: W s =Wu -2 nw-1 ;

[0021] Multi-channel convolution process:

[0022]

[0023] Where K = size·IC.

[0024] For channel-wise convolution, since NNA 1.0 only supports multi-channel convolution, channel expansion is required for the convolution kernel of channel-wise convolution.

[0025] For the nth convolutional kernel: the element value of its nth channel is equal to the element value of the nth convolutional kernel before expansion, and the element values ​​of the remaining channels are all 2. nw -1, 1≤n≤IC, the expanded weights are Actual calculation process:

[0026]

[0027] Where K = size, M = size·IC.

[0028] The NNA convolution process:

[0029] The convolution kernel data is written to WRAM, and the input image data is written to FRAM. The read addresses of WRAM and FRAM, the input data bit width, the convolution kernel size, pad, and stride parameters are set through NNA hardware registers. Then, the NNA hardware instruction NNMACG is called to obtain the convolution result output. The input image size is: number of input channels IC * input image height IH * input image width IW. The convolution kernel size is: number of output channels OC * number of input channels IC * convolution kernel height KH * convolution kernel width KW. The output image size is: number of output channels OC * output image height OH * output image width OW. The pad is divided into: pad_top, pad_bottom, pad_left, pad_right. The stride is divided into stride_y and stride_x.

[0030] The formula for calculating the output image size is:

[0031]

[0032]

[0033] The NNA hardware limits the size of the convolution kernel data written to WRAM to be less than or equal to 3 (KH and KW), and the number of input channels of the convolution kernel must be a multiple of 32. Before calling the NNA for convolution acceleration, the input image data and convolution kernel data are processed to ensure that the number of input channels IC is a multiple of 32.

[0034] The NNA convolution acceleration process is as follows: the expanded convolution kernel data, i.e., the number of input channels equal to 32, is written into the WRAM; the input image data is written into the FRAM; the read address, input data bit width, convolution kernel size, pad, and stride parameters of the WRAM and FRAM are set through the NNA hardware registers; and then the NNA hardware instruction NNMACG is called to obtain the convolution result output.

[0035] The method further includes the following steps:

[0036] Let the kernel size be represented as KH*KW.

[0037] Let wbit represent the data width nw of the convolution kernel.

[0038] S1. Let the ordinary unsigned input be F. u The ordinary signed weight is W. S The ordinary unsigned weight is W. u The conversion formula is: W s =W u -2 wbit-1 Where wbit represents the bit width of the convolution kernel data;

[0039] Ordinary convolution: The convolution kernel size is KH*KW, the number of input image channels is IC, the number of input channels of the convolution kernel is equal to IC, and IC is a multiple of 32;

[0040] S2, NNA calculates ordinary convolution, the formula is as follows:

[0041]

[0042] W u Data is written to WRAM, F u Data is written to FRAM, and the NNA register is configured, including the read addresses of WRAM and FRAM, the kernel size, and the number of input channels for the kernel. The IC must be a multiple of 32. u Data bit width and W u The data bit width is then used to call the NNMACG instruction to obtain the convolution result, i.e.

[0043] S3, Calculate depthwise convolution: The kernel size is KH*KW, the number of input image channels is IC, where IC is a multiple of 32, and the kernel input channel number is 1; the convolution kernel after input channel expansion is... The expanded convolutional kernel has IC input channels, and the padding value for the remaining input channels is W. t ;

[0044] Derivation of the formula for calculating depthwise convolution in NNA:

[0045]

[0046] If W t =2 wbit-1 Then the formula can be further simplified to:

[0047]

[0048] Similar to the formula for calculating ordinary convolution using NNA, when the weight data is ordinary signed data, calculating the depthwise convolution result using NNA requires expanding the input channels of the depthwise convolution kernel. After expansion, the values ​​of all input channels except the corresponding channels are padded with 2. wbit-1 Where wbit is the bit width of the weight data;

[0049] S4, Data is written to WRAM, F u Data is written to FRAM, and the NNA register is configured, including the read addresses of WRAM and FRAM, the kernel size, and the number of input channels (IC) of the expanded kernel, where IC must be a multiple of 32. u Data bit width and The data bit width is then used to call the NNMACG instruction to obtain the depthwise convolution result, i.e.

[0050] Therefore, the advantage of this application lies in its ability to accelerate depthwise convolution using a simple method via NNA1.0. Specifically, when considering ordinary signed data as weight data, it simultaneously fills in the remaining input channels besides the corresponding input channels during the expansion process. Attached Figure Description

[0051] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0052] Figure 1 This is a flowchart illustrating the method described in this application. Detailed Implementation

[0053] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0054] Because NNA has some rigid limitations, it only supports multi-channel convolution, which limits its versatility in practical applications. To ensure versatility while improving convolution speed, NNA convolution is accelerated by expanding the channels of the convolution kernel when encountering depthwise convolution. In the expanded multi-channel convolution kernel, the number of channels in each kernel equals the number of input channels, and the total number of kernels equals the number of input channels.

[0055] NNA 1.0 actually supports fast computation of ordinary unsigned input and ordinary unsigned weights multiplied and accumulated, while multi-channel convolution is the result of ordinary unsigned input and ordinary signed weights multiplied and accumulated. In actual use, it is necessary to configure the relevant NNA registers to quickly obtain the correct convolution result.

[0056] Assume the convolution kernel size is size, the number of input channels is IC, and the number of bits for the weights is nw. Ordinary signed weights are W. s The ordinary unsigned weight is W. u The conversion formula is: W s =W u -2 nw-1 .

[0057] Multi-channel convolution process:

[0058]

[0059] Where K = size·IC.

[0060] For depthwise convolution, since NNA1.0 only supports multi-channel convolution, it is necessary to extend the channels of the convolution kernel for depthwise convolution.

[0061] For the nth convolutional kernel: the element value of its nth channel is equal to the element value of the nth convolutional kernel before expansion, and the element values ​​of the remaining channels are all 2. nw -1, 1≤n≤IC. The expanded weights are

[0062] Actual calculation process:

[0063]

[0064] Where K = size, M = size·IC

[0065] In summary, such as Figure 1 As shown, the method further includes the following steps:

[0066] Let the kernel size be represented as KH*KW.

[0067] Let wbit represent the data width nw of the convolution kernel.

[0068] S1. Let the ordinary unsigned input be F. u The ordinary signed weight is W. S The ordinary unsigned weight is W. u The conversion formula is: W s =W u -2 wbit-1 Where wbit represents the bit width of the convolution kernel data;

[0069] Ordinary convolution: The convolution kernel size is KH*KW, the number of input image channels is IC, the number of input channels of the convolution kernel is equal to IC, and IC is a multiple of 32;

[0070] S2, NNA calculates ordinary convolution, the formula is as follows:

[0071]

[0072] W u Data is written to WRAM, F u Data is written to FRAM, and the NNA register is configured, including the read addresses of WRAM and FRAM, the kernel size, and the number of input channels for the kernel. The IC must be a multiple of 32. u Data bit width and W u The data bit width is then used to call the NNMACG instruction to obtain the convolution result, i.e.

[0073] S3, Calculate depthwise convolution: The kernel size is KH*KW, the number of input image channels is IC, where IC is a multiple of 32, and the kernel input channel number is 1; the convolution kernel after input channel expansion is... The expanded convolutional kernel has IC input channels, and the padding value for the remaining input channels is W. t ;

[0074] Derivation of the formula for calculating depthwise convolution in NNA:

[0075]

[0076] If W t =2 wbit-1 Then the formula can be further simplified to:

[0077]

[0078] Similar to the formula for calculating ordinary convolution using NNA, when the weight data is ordinary signed data, calculating the depthwise convolution result using NNA requires expanding the input channels of the depthwise convolution kernel. After expansion, the values ​​of all input channels except the corresponding channels are padded with 2. wbit-1 Where wbit is the bit width of the weight data;

[0079] S4, Data is written to WRAM, F u Data is written to FRAM, and the NNA register is configured, including the read addresses of WRAM and FRAM, the kernel size, and the number of input channels (IC) of the expanded kernel, where IC must be a multiple of 32. u Data bit width and The data bit width is then used to call the NNMACG instruction to obtain the depthwise convolution result, i.e.

[0080] Therefore, the key point of this application is:

[0081] 1. Convolution kernel expansion: NNA 1.0 requires the number of channels in the convolution kernel to be equal to the number of channels in the input. For depthwise convolution, the number of channels in the convolution kernel is equal to 1, which is not supported by NNA 1.0. NNA convolution acceleration can be achieved by expanding the channels of the convolution kernel.

[0082] 2. To ensure that the convolution result obtained after kernel expansion is consistent with the depthwise convolution result, it is necessary to calculate the padding values ​​of the elements of the other channels except the corresponding channels after expansion.

[0083] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for implementing channel-wise convolution, characterized in that, When encountering channel-wise convolution, the method accelerates NNA convolution by expanding the convolution kernel. The expanded multi-channel convolution kernel has the same number of channels as the input channels, and the total number of convolution kernels equals the number of input channels. The NNA hardware limitations include a kernel size of at least 3, a stride of at least 2, and the number of input channels for the kernel being a multiple of 32. When these hardware limitations are not met, special processing of the weight data is required to accelerate convolution using NNA: when the input image data is ordinary unsigned data and the weight data is ordinary signed data, padding values ​​are calculated for all input channels except the corresponding input channels during depthwise convolution kernel expansion. The method further includes the following steps: Let the kernel size be represented as KH*KW. Let wbit represent the data width nw of the convolution kernel. S1, let the ordinary unsigned input be... Ordinary signed weights are Ordinary unsigned weights are ; The conversion formula is: Where wbit represents the bit width of the convolution kernel data; Ordinary convolution: The convolution kernel size is KH*KW, the number of input image channels is IC, the number of input channels of the convolution kernel is equal to IC, and IC is a multiple of 32; S2, NNA calculates ordinary convolution, the formula is as follows: ； Will Data is written to WRAM. Data is written to FRAM, and the NNA register is configured, including the read addresses of WRAM and FRAM, the kernel size, and the number of input channels for the kernel. The IC must be a multiple of 32. Data bit width and The data bit width is then used to call the NNMACG instruction to obtain the convolution result, i.e. ; S3, Calculate depthwise convolution: The kernel size is KH*KW, the number of input image channels is IC, where IC is a multiple of 32, and the kernel input channel number is 1; the convolution kernel after input channel expansion is... After expansion, the number of input channels in the convolution kernel is IC, and the padding value for the remaining input channels is [value missing]. ; Derivation of the formula for calculating depthwise convolution in NNA: ； ； like Then the formula can be further simplified to: ； Similar to the formula for calculating ordinary convolution using NNA, when the weight data is ordinary signed data, calculating the depthwise convolution result using NNA requires expanding the input channels of the depthwise convolution kernel. After expansion, the values ​​of all input channels except the corresponding channels are filled with null values. Where wbit is the bit width of the weight data; S4, Data is written to WRAM. Data is written to FRAM, and the NNA register is configured, including the read addresses of WRAM and FRAM, the kernel size, and the number of input channels (IC) of the expanded kernel. The IC must be a multiple of 32. Data bit width and The data bit width is then used to call the NNMACG instruction to obtain the depthwise convolution result, i.e. .

2. The method for implementing channel-wise convolution according to claim 1, characterized in that, The NNA supports fast computation of ordinary unsigned input and the accumulation of ordinary unsigned weights. Multi-channel convolution, however, is the result of multiplying and accumulating ordinary unsigned input and ordinary signed weights. In practice, configuring the relevant NNA registers is necessary to obtain the correct convolution result. NNA supports ordinary convolution, where the kernel input channel number IC is a multiple of 32. To calculate depthwise convolution using NNA, the depthwise convolution kernel must be expanded with additional input channels, and then the depthwise convolution must be calculated using the same method as NNA for ordinary convolution. To obtain the correct depthwise convolution result, padding values ​​for the remaining input channels (excluding the corresponding input channels) need to be calculated to ensure consistency between the depthwise convolution formula and the NNA formula for ordinary convolution. Assume the convolution kernel size is size, the number of input channels is IC, and the number of bits for the weights is nw; Ordinary unsigned input is Ordinary signed weights are Ordinary unsigned weights are The conversion formula is: ; Multi-channel convolution process: ； in .

3. The method for implementing channel-wise convolution according to claim 2, characterized in that, For channel-wise convolution, since NNA only supports multi-channel convolution, channel expansion is required for the convolution kernel of channel-wise convolution: For the nth convolutional kernel: the element value of its nth channel is equal to the element value of the nth convolutional kernel before expansion, and the element values ​​of the remaining channels are all equal to... , The expanded weights are ; Actual calculation process: ； in .

4. The method for implementing channel-wise convolution according to claim 3, characterized in that, The NNA convolution process: The convolution kernel data is written to WRAM, and the input image data is written to FRAM. The read address, input data bit width, convolution kernel size, pad, and stride parameters of WRAM and FRAM are set through NNA hardware registers. Then, the convolution result is output by calling the NNA hardware instruction NNMACG. The input image size is: number of input channels IC * input image height IH * input image width IW. The convolution kernel size is: number of output channels OC * number of input channels IC * convolution kernel height KH * convolution kernel width KW. The output image size is: number of output channels OC * output image height OH * output image width OW. The pad is divided into: pad_top, pad_bottom, pad_left, pad_right. The stride is divided into stride_y and stride_x. The formula for calculating the output image size is: ； 。 5. The method for implementing channel-wise convolution according to claim 4, characterized in that, The NNA hardware limits the size of the convolution kernel data written to WRAM to be less than or equal to 3 (KH and KW), and the number of input channels of the convolution kernel must be a multiple of 32. Before calling the NNA for convolution acceleration, the input image data and convolution kernel data are processed to ensure that the number of input channels IC is a multiple of 32.

6. The method for implementing channel-wise convolution according to claim 5, characterized in that, The NNA convolution acceleration process is as follows: the expanded convolution kernel data, i.e., the number of input channels equal to 32, is written into the WRAM; the input image data is written into the FRAM; the read address, input data bit width, convolution kernel size, pad, and stride parameters of the WRAM and FRAM are set through the NNA hardware registers; and then the NNA hardware instruction NNMACG is called to obtain the convolution result output.

Images