A method for accelerating computation of add-to-convolution implementation
By converting ADD into a convolution operation and using NNA for computational acceleration, the problem of ADD becoming a bottleneck in neural networks is solved, thus improving network performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HEFEI JUNZHENG TECH CO LTD
- Filing Date
- 2024-12-23
- Publication Date
- 2026-06-23
AI Technical Summary
In existing technologies, neural network accelerators (NNAs) only support convolution acceleration, but not ADD operations, making ADD a bottleneck for network performance.
The ADD operation is converted into a convolution operation. By transforming ADD into a convolution calculation process with specific kernel size and stride values, NNA is used to accelerate the computation.
By converting ADD into convolution and utilizing NNA to accelerate the computation of ADD, the overall performance of the neural network is improved.
Smart Images
Figure CN122263985A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of neural network computing technology, and specifically relates to a method for accelerating computation based on ADD to convolution. Background Technology
[0002] In existing technologies, neural networks are mathematical models that mimic the structure and function of biological neural networks. By learning the inherent patterns in training sample data, they gain the ability to analyze or represent sample data, and can be applied to various fields such as object detection, scene classification, and character recognition. Among these, NNA (Neural Network Accelerator) is frequently used. Its single operation instruction can multiply and accumulate the input image data and the convolution kernel data, thus improving the speed of convolution operations. Convolution, for each pixel in the input image data, calculates the product of its neighboring pixels and the corresponding element of the convolution kernel, and accumulates the products to obtain the pixel value at the corresponding position in the output image. By continuously sliding the convolution kernel, the output image data is obtained. ADD, on the other hand, adds the corresponding elements of two input images to obtain the output image; that is, it adds the two input image data point-to-point, and the result is the pixel value at the corresponding position in the output image.
[0003] In existing technologies, since the input image resolution is not fixed while the FRAM size is fixed, in order to support situations where the FRAM cannot hold a large-resolution input, several rows of data from the input image are usually written to the ORAM at one time, and then written from the ORAM to the FRAM; the convolution kernel data is written to the WRAM, NNA is started, several rows of data from the output image can be obtained, and then all rows of data from the output image can be obtained through looping.
[0004] However, the shortcomings of existing technologies are:
[0005] NNA only supports convolution acceleration and does not support ADD acceleration. However, in real networks, there are not only convolutional layers but also ADD layers, and the operation speed of ADD can become a bottleneck for network performance.
[0006] In addition, the terminology commonly used in this technology includes:
[0007] 1. FRAM: The on-chip RAM inside the NNA that stores the input image data. Its size is N KB, where N is determined according to the specific chip definition and is generally less than or equal to 128KB.
[0008] 2. WRAM: On-chip RAM inside the NNA that stores convolution kernel data. Its size is M KB, where M is determined by the specific chip definition and is generally less than or equal to 64KB.
[0009] 3. ORAM: On-chip general purpose RAM.
[0010] 4. kernel: The size of the convolution kernel, divided into kernel_x and kernel_y, representing the horizontal and vertical sizes of the convolution kernel.
[0011] 5. stride: The stride of the convolution kernel, divided into stride_x and stride_y, representing the stride of the convolution kernel in the horizontal and vertical directions. Summary of the Invention
[0012] To address the aforementioned issues, the purpose of this application is to accelerate computation using NNA by converting ADD into convolution.
[0013] Specifically, this invention provides a method for accelerating computation by converting ADD to convolution. The method includes an ADD to convolution process: if the convolution kernel size is kh*kw, the stride is sh and sw, if kh is 2, kw is 1, sh is 2, and sw is 1, and there exists input image data a[1][1] = p, a[2][1] = q, and convolution kernel data b[1][1] = 1, b[2][1] = 1, then the computation process is as follows:
[0014]
[0015] The ADD is the addition of corresponding elements of two input images. The i-index indicates which input image, the j-index indicates which element of each input image, a[1][1] indicates the first point of the first input image, a[2][1] indicates the first point of the second input image, and a[1][1]+a[2][1] indicates the addition of the first points of the two input images to obtain the first point m of the output image.
[0016] In order to convert ADD into convolution, the ADD formula needs to be converted. Among them, b[1][1] and b[2][1] are the convolution kernels introduced in the process of converting ADD into convolution. The size of the convolution kernel is fixed at 2*1, and the data of the convolution kernel is always 1.
[0017] The ADD calculation process can be transformed into a convolution calculation process with kernel_x=1, kernel_y=2, stride_x=1, and stride_y=2, where all convolution kernel data are 1. The two input image data of ADD need to be written to ORAM in an interleaved manner, then to FRAM, and then the convolution kernel data is written to WRAM. Starting NNA will yield the convolution calculation result, which is also the ADD calculation result.
[0018] The method further includes:
[0019] S1, assuming both input images are of size NxN, then
[0020] The first input image data includes: x rows, x = 1, ..., N, and y columns: y = 1, ..., N;
[0021] The second input image data includes: x rows, x = 1, ..., N, and y columns, y = 1, ..., N;
[0022] The two input images for ADD must have exactly the same shape; otherwise, interleaving by row will not be possible.
[0023] The first and second input image data are written to the ORAM in an interleaved row pattern; the convolution kernel data are all set to 1.
[0024] S2, the result of the first row and first column of the output image data is the sum of the multiplication of the convolution kernel data (all values are 1) and the corresponding parts of the input image data in ORAM (i.e., the first row and first column of the first input image data and the first row and first column of the second input image data); the convolution kernel slides to the right with a stride of 1, and the result of the first row and second column of the output image data is the sum of the multiplication of the convolution kernel data (all values are 1) and the corresponding parts of the input image data in ORAM (i.e., the first row and second column of the first input image data and the second input image data); ... and so on, the result of the first row and nth column of the output image data is the sum of the multiplication of the convolution kernel data (all values are 1) and the corresponding parts of the input image data in ORAM (i.e., the first row and nth column of the first input image data and the second input image data);
[0025] The convolution kernel slides down with a stride of 2. The result of the second row and first column of the output image data is the multiplication and accumulation of the convolution kernel data (all values are 1) and the corresponding parts of the second row and first column of the first and second input image data in the ORAM; ... and so on. As the convolution kernel slides horizontally and vertically, the data of all points in the output image are obtained.
[0026] The method further includes: assuming that both input images are 8x8 in size:
[0027] S1, the first input image data, is shown in Table 1 below:
[0028] Table 1:
[0029]
[0030] The second input image data is shown in Table 2 below:
[0031] Table 2:
[0032]
[0033] The first and second input image data are written to the ORAM in an interleaved row format, as shown in Table 3 below:
[0034] Table 3:
[0035]
[0036]
[0037] The convolution kernel data are all set to 1, as shown in Table 4 below:
[0038] Table 4:
[0039]
[0040] S2, the first point of the output image data is the product and sum of the convolution kernel data (all values are 1) and the shaded areas of the corresponding squares in the first row and first column of the first input image data and the first row and first column of the second input image data in the ORAM. The convolution kernel slides to the right with a stride of 1, and the second point of the output image data horizontally is the product and sum of the convolution kernel data (all values are 1) and the shaded areas of the corresponding squares in the first row and second column of the first input image data and the first row and second column of the second input image data in the ORAM. The convolution kernel slides down with a stride of 2, and the second point of the output image data vertically is the product and sum of the convolution kernel data (all values are 1) and the shaded areas of the corresponding squares in the second row and first column of the first input image data and the second row and first column of the second input image data in the ORAM. As the convolution kernel slides horizontally and vertically, the data of all points in the output image is obtained.
[0041] The convolution is as follows: Given input image data 'a', convolution kernel data 'b', output image data 'y', kernel sizes 'kh' and 'kw', strides 'sh' and 'sw', the calculation process is as follows:
[0042]
[0043] Where i and j are the subscripts / indices of each pixel in the convolution kernel, and ordinary convolution is the multiplication and accumulation of the convolution kernel and the corresponding pixels in the input image.
[0044] The ADD (Additional Component) is defined as follows: Given two input image data points p and q, both of identical size, and an output image data point m, with the same size as the input images, the calculation process is as follows:
[0045] m = p + q.
[0046] Therefore, the advantage of this application lies in the following: The reason for converting ADD to convolution is that NNA only supports convolution acceleration. However, in real-world networks, there are not only convolutional layers but also ADD layers. The computation speed of ADD can become a bottleneck for network performance. To improve the computation speed of ADD, this application's method converts the ADD process into a convolutional process, thereby using NNA to accelerate ADD computation and further improve network performance. In short, by converting ADD to convolution and using NNA to accelerate convolution computation, ADD computation acceleration is achieved. Attached Figure Description
[0047] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.
[0048] Figure 1 This is a flowchart illustrating the method described in this application. Detailed Implementation
[0049] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.
[0050] First, convolution: Given input image data a, convolution kernel data b, output image data y, convolution kernel sizes kh and kw, strides sh and sw, the calculation process is as follows:
[0051]
[0052] Where i and j are the subscripts / indices of each pixel in the convolution kernel, and ordinary convolution is the multiplication and accumulation of the convolution kernel and the corresponding pixels in the input image.
[0053] For example: If kh = 2, kw = 1, sh = 2, sw = 1, and the input image size is 8x8, the input image data is shown in the table below:
[0054]
[0055] The convolution kernel data is shown in the table below:
[0056]
[0057] The first point of the output image data is the sum of the multiplication of the convolution kernel data and the input image data (the gray shaded area in the table above). As the convolution kernel slides to the right with a stride of 1, the second horizontal point of the output image data is the sum of the multiplication of the convolution kernel data and the input image data (the black shaded area in the table above). As the convolution kernel slides downwards with a stride of 2, the second vertical point of the output image data is the sum of the multiplication of the convolution kernel data and the input image data (the light gray shaded area in the table above). By sliding the convolution kernel horizontally and vertically, the data for all points in the output image are obtained.
[0058] Secondly, ADD: Given two input image data p and q, which are exactly the same size, and an output image data m, which is the same size as the input image, the calculation process is as follows:
[0059] m = p + q.
[0060] For example: If both input images are 8x8 in size,
[0061] The first input image data is shown in Table a:
[0062] Table a:
[0063]
[0064] The second input image data is shown in Table b:
[0065] Table b:
[0066]
[0067] The first point of the output image data is the sum of the first input image data (light gray shaded area in Table a above) and the second input image data (dark gray shaded area in Table 2 above). The two input image data points are added point-to-point to obtain the data for all points in the output image.
[0068] However, NNA only supports convolution calculations, and cannot accelerate ADD calculations that only support two input images of exactly the same size. Therefore, this invention proposes a method for accelerating calculations by converting ADD to convolution, the method comprising: converting ADD to convolution.
[0069] If the convolution kernel size is kh*kw, and the stride is sh and sw, and if kh is 2, kw is 1, sh is 2, and sw is 1, and there exists input image data a[1][1] = p, a[2][1] = q, and convolution kernel data b[1][1] = 1, b[2][1] = 1, then the calculation process is as follows:
[0070]
[0071] The ADD is the addition of corresponding elements of two input images. The i-index indicates which input image, the j-index indicates which element of each input image, a[1][1] indicates the first point of the first input image, a[2][1] indicates the first point of the second input image, and a[1][1]+a[2][1] indicates the addition of the first points of the two input images to obtain the first point m of the output image.
[0072] Since the formula for ordinary convolution is Where a[i][j] represents the input image data and b[i][j] represents the convolution kernel data. Ordinary convolution is the multiplication and accumulation of the convolution kernel data and the corresponding input image data (a total of kh*kw corresponding elements are multiplied and accumulated).
[0073] To convert ADD into a convolution, the ADD formula needs to be transformed into the form of a regular convolution formula:
[0074]
[0075] Where b[1][1] and b[2][1] are the convolution kernels introduced in the ADD to convolution process. Compared with the ordinary convolution formula, we can see that the convolution kernel size is fixed at 2*1 and the convolution kernel data is always 1. The reason for converting ADD to convolution is that NNA only supports convolution acceleration. However, in actual networks, the operation speed of ADD will also become a bottleneck of network performance. Therefore, by converting the ADD process into a convolution process, NNA can be used to accelerate ADD.
[0076] The ADD calculation process can be transformed into a convolution calculation process with kernel_x=1, kernel_y=2, stride_x=1, and stride_y=2, where all convolution kernel data are 1s. The two input image data for ADD need to be written to ORAM (Optical Random Access Memory) row-wise, then to FRAM (Optical Random Access Memory), and finally to WRAM (Optical Random Access Memory). Starting NNA (Non-Non-Analog Memory) yields the convolution calculation result, which is also the ADD calculation result. In other words, ADD adds corresponding elements of the two input images, while ordinary convolution multiplies and accumulates the convolution kernel data and the input image data. To convert ADD into ordinary convolution, corresponding elements of the two input images for ADD can be multiplied by 1 before addition. Here, the 1s are the introduced convolution kernel data and do not affect the ADD calculation result. The ADD process can be transformed into concatenating the two input images row-wise into one input image, and then performing element-wise multiplication and accumulation with a convolution kernel of size 2*1, where the convolution kernel data is always 1s. This is the ordinary convolution process.
[0077] like Figure 1 As shown, this method further includes:
[0078] S1, assuming both input images are of size NxN, then
[0079] The first input image data includes: x rows, x = 1, ..., N, and y columns: y = 1, ..., N;
[0080] The second input image data includes: x rows, x = 1, ..., N, and y columns, y = 1, ..., N;
[0081] The first and second input image data are written to the ORAM in an interleaved row pattern; the convolution kernel data are all set to 1.
[0082] S2, the result of the first row and first column of the output image data is the sum of the multiplication of the convolution kernel data (all values are 1) and the corresponding parts of the input image data in ORAM (i.e., the first row and first column of the first input image data and the first row and first column of the second input image data); the convolution kernel slides to the right with a stride of 1, and the result of the first row and second column of the output image data is the sum of the multiplication of the convolution kernel data (all values are 1) and the corresponding parts of the input image data in ORAM (i.e., the first row and second column of the first input image data and the second input image data); ... and so on, the result of the first row and nth column of the output image data is the sum of the multiplication of the convolution kernel data (all values are 1) and the corresponding parts of the input image data in ORAM (i.e., the first row and nth column of the first input image data and the second input image data);
[0083] The convolution kernel slides down with a stride of 2. The result of the second row and first column of the output image data is the convolution kernel data, which is all 1, and the sum of the corresponding parts of the input image data in ORAM, namely the second row and first column of the first input image data and the second row and first column of the second input image data; ... and so on; as the convolution kernel slides horizontally and vertically, the data of all points in the output image are obtained.
[0084] For example: If both input images are 8x8 in size:
[0085] S1, the first input image data, is shown in Table 1 below:
[0086] Table 1:
[0087]
[0088] The second input image data is shown in Table 2 below:
[0089] Table 2:
[0090]
[0091]
[0092] The first and second input image data are written to the ORAM in an interleaved row format, as shown in Table 3 below:
[0093] Table 3:
[0094]
[0095] The convolution kernel data are all set to 1, as shown in Table 4 below:
[0096] Table 4:
[0097]
[0098] S2, the first point of the output image data is the product and sum of the convolution kernel data (all values are 1) and the shaded areas of the corresponding squares in the first row and first column of the first input image data and the first row and first column of the second input image data in the ORAM. The convolution kernel slides to the right with a stride of 1, and the second point of the output image data horizontally is the product and sum of the convolution kernel data (all values are 1) and the shaded areas of the corresponding squares in the first row and second column of the first input image data and the first row and second column of the second input image data in the ORAM. The convolution kernel slides down with a stride of 2, and the second point of the output image data vertically is the product and sum of the convolution kernel data (all values are 1) and the shaded areas of the corresponding squares in the second row and first column of the first input image data and the second row and first column of the second input image data in the ORAM. As the convolution kernel slides horizontally and vertically, the data of all points in the output image is obtained.
[0099] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for accelerating computation by converting ADD to convolution, characterized in that, The method includes an ADD to convolution process: if the convolution kernel size is kh*kw, the stride is sh and sw, if kh is 2, kw is 1, sh is 2, and sw is 1, and there exists input image data a[1][1] = p, a[2][1] = q, and convolution kernel data b[1][1] = 1, b[2][1] = 1, then the calculation process is as follows: The ADD is the addition of corresponding elements of two input images. The i-index indicates which input image, the j-index indicates which element of each input image, a[1][1] indicates the first point of the first input image, a[2][1] indicates the first point of the second input image, and a[1][1]+a[2][1] indicates the addition of the first points of the two input images to obtain the first point m of the output image. In order to convert ADD into convolution, the ADD formula needs to be converted. Among them, b[1][1] and b[2][1] are the convolution kernels introduced in the process of converting ADD into convolution. The size of the convolution kernel is fixed at 2*1, and the data of the convolution kernel is always 1. The ADD calculation process can be transformed into a convolution calculation process with kernel_x=1, kernel_y=2, stride_x=1, and stride_y=2, where all convolution kernel data are 1. The two input image data of ADD need to be written to ORAM in an interleaved manner, then to FRAM, and then the convolution kernel data is written to WRAM. Starting NNA will yield the convolution calculation result, which is also the ADD calculation result.
2. The method for accelerating computation by converting ADD to convolution according to claim 1, characterized in that, The method further includes: S1, assuming both input images are of size NxN, then The first input image data includes: x rows, x = 1, ..., N, and y columns: y = 1, ..., N; The second input image data includes: x rows, x = 1, ..., N, y columns: y = 1, ..., N; the two input images for ADD must have exactly the same shape, otherwise the row-by-row interleaving cannot be performed; The first and second input image data are written to the ORAM in an interleaved row pattern; the convolution kernel data are all set to 1. S2, the result of the first row and first column of the output image data is the sum of the multiplication of the convolution kernel data (all values are 1) and the corresponding parts of the input image data in ORAM (i.e., the first row and first column of the first input image data and the first row and first column of the second input image data); the convolution kernel slides to the right with a stride of 1, and the result of the first row and second column of the output image data is the sum of the multiplication of the convolution kernel data (all values are 1) and the corresponding parts of the input image data in ORAM (i.e., the first row and second column of the first input image data and the second input image data); ... and so on, the result of the first row and nth column of the output image data is the sum of the multiplication of the convolution kernel data (all values are 1) and the corresponding parts of the input image data in ORAM (i.e., the first row and nth column of the first input image data and the second input image data); The convolution kernel slides down with a stride of 2. The result of the second row and first column of the output image data is the multiplication and accumulation of the convolution kernel data (all values are 1) and the corresponding parts of the second row and first column of the first and second input image data in the ORAM; ... and so on. As the convolution kernel slides horizontally and vertically, the data of all points in the output image are obtained.
3. The method for accelerating computation by converting ADD to convolution according to claim 2, characterized in that, The method further includes: assuming that both input images are 8x8 in size: S1, the first input image data, is shown in Table 1 below: Table 1: The second input image data is shown in Table 2 below: Table 2: The first and second input image data are written to the ORAM in an interleaved row format, as shown in Table 3 below: Table 3: The convolution kernel data are all set to 1, as shown in Table 4 below: Table 4: S2, the first point of the output image data is the product and sum of the convolution kernel data (all values are 1) and the shaded areas of the corresponding squares in the first row and first column of the first input image data and the first row and first column of the second input image data in the ORAM. The convolution kernel slides to the right with a stride of 1, and the second point of the output image data horizontally is the product and sum of the convolution kernel data (all values are 1) and the shaded areas of the corresponding squares in the first row and second column of the first input image data and the first row and second column of the second input image data in the ORAM. The convolution kernel slides down with a stride of 2, and the second point of the output image data vertically is the product and sum of the convolution kernel data (all values are 1) and the shaded areas of the corresponding squares in the second row and first column of the first input image data and the second row and first column of the second input image data in the ORAM. As the convolution kernel slides horizontally and vertically, the data of all points in the output image is obtained.
4. The method for accelerating computation by converting ADD to convolution according to claim 1, characterized in that, The convolution is as follows: Given input image data 'a', convolution kernel data 'b', output image data 'y', kernel sizes 'kh' and 'kw', strides 'sh' and 'sw', the calculation process is as follows: Where i and j are the subscripts / indices of each pixel in the convolution kernel, and ordinary convolution is the multiplication and accumulation of the convolution kernel and the corresponding pixels in the input image.
5. The method for accelerating computation by converting ADD to convolution according to claim 1, characterized in that, The ADD (Additional Component) is defined as follows: Given two input image data points p and q, both of identical size, and an output image data point m, with the same size as the input images, the calculation process is as follows: m = p + q.