An implementation method of 1xN convolution based on NNA

Abstract

This invention provides a method for implementing 1xN convolution based on NNA. When the parameters of NNA are not within their supported range, NNA convolution acceleration is achieved by splitting, recombining, and accumulating the input image and the convolution kernel matrix. Specifically, kernel splitting is used: for 1xN convolution, when N is greater than 3, it exceeds the range of NNA register parameters; kernel splitting accelerates NNA convolution. The method also includes NNA register configuration: Correct configuration of NNA registers, the necessary parameters for convolution stride and kernel size, is crucial for NNA convolution acceleration. These parameters must be within the range supported by NNA to obtain correct convolution results. By configuring the relevant register parameters, convolution calculations are accelerated for kernel matrix dimensions ≤ 3x3, significantly reducing neural network runtime and resulting in higher real-time performance and a better user experience in practical applications. This method ensures versatility while improving convolution operation speed even when the NNA parameter range is not met.

Description

Technical Field

[0001] This invention relates to the field of neural network technology, and in particular to a method for implementing 1xN convolution based on NNA. Background Technology

[0002] In recent years, with the rapid development of science and technology, the era of big data has arrived, and the massive data processing technology in neural networks has increasingly become one of the important application technologies. Especially with the increasing adoption of artificial intelligence in consumer electronics, automotive electronics, and other products, artificial intelligence is developing at an unprecedented pace, with technologies such as deep learning and neural networks entering a period of rapid growth. Artificial intelligence requires a large amount of computation in model training and inference. However, due to the inherent characteristics of its algorithms and computations, traditional chips, which were widely used in the past, can no longer meet the demands of massive computations. This necessitates chip manufacturers to create dedicated chips for neural network algorithms, especially inference chips, namely neural network accelerators (NNAs). In current technology, NNA 1.0 supports fast computation for convolutional kernel matrix dimensions less than or equal to 3x3. 1xN: the dimension of the convolutional kernel matrix, where N is a positive integer.

[0003] However, the existing technology has the following drawbacks:

[0004] The convolution process involves a large number of multiplication and accumulation calculations, which is slow and not real-time in practical applications. Moreover, NNA1.0 only supports fast operation when the convolution kernel matrix dimension is less than or equal to 3x3. When N is large, for example, when N is greater than 3, NNA1.0 cannot accelerate the 1xN convolution process.

[0005] In addition, the commonly used terms in the prior art are as follows:

[0006] 1. Neural Network: A mathematical model that imitates the structure and function of biological neural networks. By learning the inherent laws of training sample data, it gains the ability to analyze or express sample data. It can be applied to various application fields such as target detection, scene classification, and character recognition.

[0007] 2. Deep learning: The process and methods of training neural networks.

[0008] 3. Image Convolution: For each pixel in the input image, the product of its neighboring pixels and the corresponding element of the convolution kernel matrix is ​​calculated and accumulated to obtain the value of that pixel position. By continuously sliding the convolution kernel matrix, the final convolution output result is obtained.

[0009] 4. Feature extraction: Using computers to extract image information, determining whether each pixel belongs to an image feature. The result is that the points on the image are divided into different subsets, which often belong to independent points, continuous curves, or continuous regions.

[0010] 5. NNA: Neural Network Accelerator.

[0011] 6. FRAM: On-chip RAM that stores input image data inside the NNA.

[0012] 7. WRAM: On-chip RAM that stores convolution kernel data inside the NNA.

[0013] 8. ORAM: On-chip general purpose RAM.

[0014] 9. pixel: The smallest unit of input image.

[0015] 10. pad: Fills the edges of the input image. It is divided into pad_top, pad_bottom, pad_left, and pad_right, which represent the size of the padding on the top, bottom, left, and right edges of the image.

[0016] 11. stride: The stride of the convolution kernel matrix, divided into stride_x and stride_y, representing the stride of the convolution kernel matrix in the horizontal and vertical directions. Summary of the Invention

[0017] To address the aforementioned issues, the purpose of this application is to provide a fast method for calculating 1xN convolutions based on NNA1.0, thereby improving the computation speed of 1xN convolutions.

[0018] Specifically, this invention provides a method for implementing 1xN convolution based on NNA 1.0. When the NNA-related parameter range is not met, this method accelerates NNA convolution by splitting, recombining, and accumulating the input image and the convolution kernel matrix.

[0019] Kernel splitting: For 1xN convolutions, when N is greater than 3, it exceeds the range of NNA register parameters. Kernel splitting is used to accelerate NNA convolutions.

[0020] This method also includes NNA register configuration: NNA-related registers must be configured correctly when using NNA for convolution acceleration; the stride of convolution, kernel size, input image size and data bit width, read and write addresses of FRAM and WRAM, and FRAM write address jump are all configured through NNA registers. These parameters must be within the range of parameters supported by NNA to obtain correct convolution results.

[0021] The method includes:

[0022] S1, input image size IH*IW, convolution kernel matrix size KH*KW, input image edge padding sizes are pad_top, pad_bottom, pad_left, pad_right, convolution stride is stride_x, stride_y, and output image size OH*OW is calculated using the following formula:

[0023]

[0024]

[0025] S2, NNA convolution process: The convolution kernel data is written to WRAM, and then the required input image data is written to FRAM. Four or eight pixels can be written at a time. The NNA hardware limits the number of pixels written to FRAM to be an integer multiple of 4. The convolution kernel size (KH and KW must be less than or equal to 3), pad, stride, input image size and data bit width, and read addresses of FRAM and WRAM are set through the NNA hardware registers. Then, the NNA hardware instructions are called to obtain the convolution result output, which can obtain 4 pixels of output data at a time.

[0026] The number of pixels that need to be written to FRAM each time is FP_H * FP_W:

[0027] FP_H=KH

[0028] FP_W = 3 * stride_x + KW

[0029] FP_W needs to be aligned to 4;

[0030] S3, for a 1xN convolution, when N≤3, the convolution result can be obtained directly through the above process; when N>3, due to the limitation of the NNA hardware register parameter range, the convolution kernel matrix needs to be split. Specifically, the convolution kernel size is 1*N (N>3). The NNA can support a maximum of 1*3 convolution kernels. The convolution kernel is split into multiple convolution kernels of 1*3, ..., 1*3, ..., 1*(N%3), where % represents the remainder. The number of convolution kernels K after splitting is:

[0031]

[0032] For the first K-1 convolution kernels (size 1*3), 1*fp_w pixels of input data need to be written to the FRAM, where fp_w = 3*stride_x + 3, and fp_w needs to be aligned to 4. The convolution kernel size (1*3), pad, stride, input image size and data bit width, and the read addresses of FRAM and WRAM are set by configuring the NNA register. Then, the NNA hardware instruction NNMACG is called to obtain 4 pixels of output data. The above process is repeated K-1 times.

[0033] For the last convolution kernel, which is 1*(N%3), 1*fp_w pixels of input data need to be written to the FRAM, where fp_w = 3*stride_x + N%3. fp_w needs to be aligned to 4. The convolution kernel size (1*(N%3), pad, stride, input image size and data bit width, and read addresses of FRAM and WRAM are set by configuring the NNA register. Then, the NNA hardware instruction NNMACG is called to obtain 4 pixels of output data. This result is accumulated with the results obtained in the previous K-1 iterations, that is, the corresponding positions of the 4 pixels of data obtained in the previous K-1 iterations are added to the 4 pixels of data obtained in this iteration, and the final 1*N convolution output is obtained.

[0034] In step S2, assuming the convolution kernel size is 3*3, the input image size is 64*64, and no edge padding is applied to the input image (i.e., pad_top, pad_bottom, pad_left, and pad_right are all 0), and the convolution stride_x and stride_y are both 1, the output image size is 62*62. To obtain 4 pixels of output data at once through NNA hardware instructions, 3*8 pixels of input data need to be written to the FRAM.

[0035] The more direct way to calculate K in step S3 is: K = (N+2) / 3, where the size of the first K-1 convolution kernels is 1*3, and the size of the last convolution kernel is 1*(N%3).

[0036] Therefore, the advantage of this application lies in the following: It accelerates convolution calculations for kernel matrices with dimensions less than or equal to 3x3 through a simple method—namely, configuring relevant register parameters—significantly reducing neural network runtime. This results in higher real-time performance and a better user experience in practical applications. Furthermore, it ensures versatility while improving convolution operation speed even when NNA parameters are not within their specified range. Attached Figure Description

[0037] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0038] Figure 1 This is a flowchart illustrating the method described in this application. Detailed Implementation

[0039] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0040] Convolutional neural networks (CNNs) have wide applications in image, video, and speech processing. As neural networks become increasingly complex, they require substantial resources for training and evaluation, while the performance and versatility of hardware accelerators are improving at a slower pace. A CNN primarily consists of an input layer, convolutional layers, pooling layers, and fully connected layers. The convolutional layer is the core of the CNN, generating the vast majority of the network's computation; therefore, the operating speed of a CNN largely depends on the computational speed of its convolutional layers.

[0041] NNA (neural network accelerate unit) is a hardware platform neural network accelerator. In this method, by configuring relevant register parameters, convolution calculations with a kernel matrix dimension of less than or equal to 3x3 can be accelerated, thereby significantly reducing the running time of the neural network. In practical applications, this results in higher real-time performance and a better user experience.

[0042] Due to certain rigid limitations, NNA has limited versatility in practical applications. To ensure versatility while improving convolution speed, when the relevant parameter range of NNA is not met, NNA convolution is accelerated by splitting, recombining, and accumulating the input image and the convolution kernel matrix.

[0043] Specifically, such as Figure 1 As shown, the method of this application includes:

[0044] S1, input image size IH*IW, convolution kernel matrix size KH*KW, input image edge padding sizes pad_top, pad_bottom, pad_left, pad_right, convolution stride length stride_x, stride_y, output image size OH*OW can be calculated using the following formula:

[0045]

[0046]

[0047] S2, NNA convolution process: The convolution kernel data is written to WRAM, and then the required input image data is written to FRAM. Four or eight pixels can be written at a time. The NNA hardware limits the number of pixels written to FRAM to be a multiple of four. The convolution kernel size (KH and KW must be less than or equal to 3), pad, stride, input image size and data bit width, and read addresses for FRAM and WRAM are set through NNA hardware registers. Then, NNA hardware instructions are called to obtain the convolution result output, which can obtain four pixels of output data at a time.

[0048] The number of pixels that need to be written to FRAM each time is FP_H * FP_W:

[0049] FP_H=KH

[0050] FP_W = 3 * stride_x + KW

[0051] FP_W needs to be aligned to 4.

[0052] For example, with a 3x3 kernel size, an input image size of 64x64, no edge padding (pad_top, pad_bottom, pad_left, pad_right are all 0), and stride_x and stride_y are both 1, the output image size is 62x62. To obtain 4 pixels of output data at once using NNA hardware instructions, 3x8 pixels of input data need to be written to FRAM.

[0053] S3. For a 1xN convolution, when N≤3, the convolution result can be obtained directly through the above process. When N>3, due to the limitation of the NNA hardware register parameter range, the convolution kernel matrix needs to be split.

[0054] Implementation details: The kernel size is 1*N (N>3), and NNA can support a maximum kernel size of 1*3. The kernel is split into multiple kernels of size 1*3, ..., 1*3, ..., 1*(N%3) (where % represents the remainder). The number of kernels K after splitting is:

[0055]

[0056] A more direct calculation method is: K = (N+2) / 3, where the size of the first K-1 convolution kernels is 1*3, and the size of the last convolution kernel is 1*(N%3);

[0057] For the first K-1 convolution kernels (size 1*3), 1*fp_w pixels of input data need to be written to the FRAM, where fp_w = 3*stride_x + 3, and fp_w needs to be aligned to 4. The convolution kernel size (1*3), pad, stride, input image size and data bit width, and the read addresses of FRAM and WRAM are set by configuring the NNA register. Then, the NNA hardware instruction NNMACG is called to obtain 4 pixels of output data. The above process is repeated K-1 times.

[0058] For the last convolution kernel (size 1*(N%3)), 1*fp_w pixels of input data need to be written to the FRAM, where fp_w = 3*stride_x + N%3. fp_w needs to be aligned to 4. The convolution kernel size (1*(N%3), pad, stride, input image size and data bit width, and the read addresses of FRAM and WRAM are set by configuring the NNA register. Then, the NNA hardware instruction NNMACG is called to obtain 4 pixels of output data. This result is accumulated with the results obtained in the previous K-1 iterations to obtain the final 1*N convolution output.

[0059] In summary, the key optimizations to the prior art in this application are as follows:

[0060] 1. Convolution kernel splitting: For 1xN convolution, when N is greater than 3, it exceeds the range of NNA convolution kernel parameters. Convolution kernel splitting is used to accelerate NNA convolution.

[0061] 2. NNA Register Configuration: To accelerate convolution using NNA, the relevant NNA registers must be configured correctly. Parameters such as the stride and kernel size of the convolution are configured through the NNA registers. These parameters must be within the range supported by NNA to obtain the correct convolution results.

[0062] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for implementing 1xN convolution based on NNA, characterized in that, When the NNA-related parameter range is not met, the method accelerates NNA convolution by splitting, recombining, and accumulating the input image and the convolution kernel matrix. Kernel splitting: For 1xN convolutions, when N is greater than 3, it exceeds the range of NNA register parameters. Kernel splitting is used to accelerate NNA convolutions. This method also includes NNA register configuration: When using NNA for convolution acceleration, the NNA-related registers, convolution stride, convolution kernel size, input image size and data bit width, FRAM and WRAM read / write addresses, and FRAM write address jumps are all configured through the NNA registers. The method includes the following steps: S1, input image size IH*IW, convolution kernel matrix size KH*KW, input image edge padding sizes are pad_top, pad_bottom, pad_left, pad_right, convolution stride is stride_x, stride_y, and output image size OH*OW is calculated using the following formula: ； ； S2, NNA convolution process: The convolution kernel data is written to WRAM, and then the required input image data is written to FRAM. Four or eight pixels can be written at a time. The NNA hardware limits the number of pixels written to FRAM to be an integer multiple of 4. The convolution kernel size (KH and KW must be less than or equal to 3), pad, stride, input image size and data bit width, and read address parameters of FRAM and WRAM are set through the NNA hardware registers. Then, the NNA hardware instruction is called to obtain the convolution result output, which can obtain 4 pixels of output data at a time. The number of pixels that need to be written to FRAM each time is FP_H * FP_W: ； FP_W needs to be aligned to 4; S3, for a 1xN convolution, when N≤3, the convolution result is obtained directly through the above process; when N>3, due to the limitation of the NNA hardware register parameter range, the convolution kernel matrix needs to be split. Specifically, the convolution kernel size is 1*N (N>3). The NNA supports a maximum of 1*3 convolution kernels. The convolution kernel is split into multiple convolution kernels of size 1*3, ..., 1*3, ..., 1*(N%3), where % represents the remainder. The number of convolution kernels K after splitting is: ； For the first K-1 convolutional kernels, 1*fp_w pixels of input data need to be written into the FRAM. fp_w needs to be aligned to 4. The convolution kernel size (1*3), pad, stride, input image size and data bit width, and read address parameters of FRAM and WRAM are set by configuring the NNA register. Then, the NNA hardware instruction is called to obtain the output data of 4 pixels. The above process is repeated K-1 times. For the last convolution kernel, which has a size of 1*(N%3), 1*fp_w pixels of input data need to be written to the FRAM. fp_w needs to be aligned to 4. The convolution kernel size (1*(N%3)), pad, stride, input image size and data bit width, and read address parameters of FRAM and WRAM are set by configuring the NNA register. Then, the NNA hardware instruction is called to obtain the output data of 4 pixels. This result is accumulated with the results obtained in the previous K-1 times. That is, the corresponding positions of the 4 pixels obtained in the previous K-1 times and the 4 pixels obtained in this time are added to obtain the final 1*N convolution output.

2. The method for implementing 1xN convolution based on NNA according to claim 1, characterized in that, In step S2 Assuming the convolution kernel size is 3*3, the input image size is 64*64, and no edge padding is applied to the input image (i.e., pad_top, pad_bottom, pad_left, and pad_right are all 0), and the convolution stride_x and stride_y are both 1, the output image size is 62*62. To obtain 4 pixels of output data at a time through NNA hardware instructions, 3*8 pixels of input data need to be written to FRAM.

3. The method for implementing 1xN convolution based on NNA according to claim 1, characterized in that, In step S3 A more direct way to calculate K is: The first K-1 convolutional kernels are 1*3 in size, and the last convolutional kernel is 1*(N%3).

4. The method for implementing 1xN convolution based on NNA according to claim 1, characterized in that, The NNA hardware instruction is the NNMACG instruction.

Images