A method for concatenation (concat) convolution implementation computing acceleration
By converting the Concat layer into a convolutional form and using NNA for acceleration, the problem of slow Concat layer operation speed is solved, and the overall performance of the neural network is improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HEFEI JUNZHENG TECH CO LTD
- Filing Date
- 2024-12-23
- Publication Date
- 2026-06-23
AI Technical Summary
In existing technologies, the Concat layer in neural networks operates slowly, becoming a bottleneck for network performance and making it impossible to effectively utilize NNA for acceleration.
The Concat layer is converted into a convolutional form and accelerated using NNA. By introducing a convolutional kernel of fixed size 1*1 and a stride of 1*1, the Concat computation process is transformed, leveraging the convolutional acceleration features of NNA.
By converting Concat into a convolution process and using NNA for computational acceleration, the performance of neural networks containing Concat layers is improved.
Smart Images
Figure CN122263986A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of neural network computing technology, and specifically relates to a method for accelerating computation based on concatenating concat to convolution. Background Technology
[0002] In existing technologies, neural networks are mathematical models that mimic the structure and function of biological neural networks. By learning the inherent patterns in training sample data, they gain the ability to analyze or represent sample data, and can be applied to various fields such as object detection, scene classification, and character recognition. Among them, NNA (Neural Network Accelerator) is frequently used. Its single operation instruction can perform the multiplication and accumulation of input image data and convolution kernel data, which can improve the operation speed of convolution. Convolution, on the other hand, calculates the product of each pixel in the input image data and the corresponding element of the convolution kernel, and then accumulates the products to obtain the pixel value of the corresponding position in the output image. By continuously sliding the convolution kernel, the output image data is obtained.
[0003] In existing technologies, input image data is written to FRAM, convolution kernel data is written to WRAM, and NNA is started to obtain output image data.
[0004] For example, the Ingenic T32 chip uses NNA for acceleration.
[0005] However, the shortcomings of existing technologies are:
[0006] NNA only supports convolution acceleration, but in real-world networks, in addition to convolutional layers, there are also concat layers. The speed of concat operations can also become a bottleneck for network performance. That is, in real-world networks, besides convolutional layers, the slow speed of concat operations also affects network performance.
[0007] In addition, the terminology commonly used in this technology includes:
[0008] FRAM: The on-chip RAM inside the NNA stores the input image data. Its size is N KB, where N is determined according to the specific chip definition and is generally less than or equal to 128KB.
[0009] WRAM: On-chip RAM inside the NNA that stores convolution kernel data. Its size is M KB, where M is determined by the specific chip definition and is generally less than or equal to 64KB.
[0010] kernel: The size of the convolution kernel, divided into kernel_x and kernel_y, representing the horizontal and vertical sizes of the convolution kernel.
[0011] stride: The stride of the convolution kernel, divided into stride_x and stride_y, representing the stride of the convolution kernel in the horizontal and vertical directions. Summary of the Invention
[0012] To address the aforementioned issues, the purpose of this application is to accelerate Concat computation by converting it into a convolutional form and using NNA for acceleration, thereby improving the performance of networks containing Concat. In other words, the solution proposed in this application is to convert Concat into a regular convolutional form and utilize the acceleration features of NNA convolution to achieve faster Concat computation.
[0013] Specifically, this invention provides a method for accelerating computation by concatenating and converting images using Concat to Convolution. The method includes the process of concatenating and converting images using Concat: if there are n input images, they are concatenated along a certain dimension to obtain an output image, where all dimensions except the concatenation dimension are of the same size.
[0014] Given input image data C and output image data D, the Concat calculation process is as follows:
[0015] D[m, ...] = C[m]
[0016] Where m = 1, ..., n,
[0017] To convert the Concat calculation process into a regular convolution calculation process, the Concat calculation formula needs to be transformed into the form of a regular convolution formula:
[0018]
[0019] To convert Concat into a regular convolution, a new convolution kernel is introduced: the kernel data is always 1, and the kernel size is fixed at 1*1.
[0020] The Concat process is transformed into a regular convolution process where each input image data point is multiplied and summed with the convolution kernel data to obtain the output image data. Specifically, the first point of the output image is the sum of the first point of the first input image and the convolution kernel data. The convolution kernel size is 1*1, and the data is always 1. As the convolution kernel slides sequentially across each input image, the data for all points in the output image is obtained. This involves writing n input image data points sequentially along a certain dimension into the FRAM, and then writing the convolution kernel data into the WRAM. By configuring the NNA register to set the convolution kernel size to 1*1 and the stride to 1*1, and starting NNA, the convolution calculation result, which is also the Concat calculation result, can be obtained.
[0021] The method assumes horizontal splicing and includes the following steps:
[0022] S1, assuming there are n input images, each of size m x m, which are concatenated horizontally to obtain an output image of size m x mn; that is, the first input image data, the second input image data, ..., the nth input image data are concatenated horizontally to obtain the output image data; the convolution kernel data are all 1;
[0023] S2, since the input images are stitched horizontally, the first horizontal point of the output image data is the product and sum of the convolution kernel data and the first horizontal point of the first input image data; the convolution kernel slides to the right with a stride of 1, and the (m+1)th horizontal point of the output image data is the product and sum of the convolution kernel data and the first horizontal point of the second input image data, and so on, until the (n-1)*m+1th horizontal point of the output image data is the product and sum of the convolution kernel data and the first horizontal point of the nth input image data; the convolution kernel slides downward with a stride of 1, and the second vertical point of the output image data is the product and sum of the convolution kernel data and the second vertical point of the first input image data; all the convolution kernel data are 1, and so on, as the convolution kernel slides horizontally and vertically, the data of all points of the output image are obtained.
[0024] The method further includes:
[0025] S1, if there are 3 input images, each 4x4 in size, they are stitched together horizontally to obtain an output image of size 4x12:
[0026] The first input image data is shown in Table 1 below:
[0027] Table 1:
[0028]
[0029] The second input image data is shown in Table 2 below:
[0030] Table 2:
[0031]
[0032] The third input image data is shown in Table 3 below:
[0033] Table 3:
[0034]
[0035] The output image data is shown in Table 4 below:
[0036] Table 4:
[0037]
[0038] The convolution kernel data are all set to 1, as shown in Table 5 below:
[0039] Table 5:
[0040]
[0041] S2, the first point of the output image data is the product and sum of the convolution kernel data (all values are 1) and the first input image data (i.e., the shaded area of the first square in the first row and first column of Table 4); the convolution kernel slides to the right with a stride of 1, the fifth point of the output image data horizontally is the product and sum of the convolution kernel data (all values are 1) and the second input image data (i.e., the shaded area of the first square in the fifth column of Table 4), the ninth point of the output image data horizontally is the product and sum of the convolution kernel data (all values are 1) and the third input image data (i.e., the shaded area of the first square in the ninth column of Table 4); the convolution kernel slides down with a stride of 1, the second point of the output image data vertically is the product and sum of the convolution kernel data (all values are 1) and the first input image data (i.e., the shaded area of the second square in the first row and first column of Table 4); as the convolution kernel slides horizontally and vertically, the data of all points of the output image are obtained.
[0042] The method assumes splicing along the longitudinal direction and includes the following steps:
[0043] S1, assuming there are n input images, each of size m x m, which are concatenated vertically to obtain an output image of size mn x m; that is, the first input image data, the second input image data, ..., the nth input image data are concatenated vertically to obtain the output image data; the convolution kernel data are all 1;
[0044] S2, since the input images are stitched along the vertical direction, the first point of the output image data in the vertical direction is the product and sum of the convolution kernel data and the first point of the first input image data in the vertical direction; the convolution kernel slides downward with a stride of 1, and the (m+1)th point of the output image data in the vertical direction is the product and sum of the convolution kernel data and the first point of the second input image data in the vertical direction, and so on, until the (m-1)*n+1th point of the output image data in the vertical direction is the product and sum of the convolution kernel data and the first point of the nth input image data in the vertical direction; the convolution kernel slides horizontally with a stride of 1, and the second point of the output image data in the horizontal direction is the product and sum of the convolution kernel data and the second point of the first input image data in the horizontal direction; all the convolution kernel data are 1, and so on, as the convolution kernel slides horizontally and vertically, the data of all points of the output image are obtained.
[0045] The convolution includes: input image data a, convolution kernel data b, output image data y, kernel sizes kh and kw, strides sh and sw, and the calculation process is as follows:
[0046]
[0047] i and j are the indices of each element of the convolution kernel. Convolution is the multiplication and summation of the convolution kernel and the corresponding elements of the input image.
[0048] The concat stitching process includes: given n input image data points C1, C2, ..., Cn, these n input image data points are stitched together horizontally, with all dimensions except the stitching dimension being identical in size. The output image data is D, with all dimensions except the stitching dimension being identical in size to the input image. The stitching dimension is the sum of the stitching dimensions of the input images. The calculation process is as follows:
[0049] D[m] = Cm, m = 1, ..., n.
[0050] The method is applicable to chips that use NNA for general convolution acceleration calculations, including the Ingenic T32 chip.
[0051] Therefore, the advantage of this application is:
[0052] By transforming the Concat process into a convolution process, NNA can be used to accelerate convolution calculations, thereby speeding up the Concat computation. Concat concatenates multiple input image data along a single axis to obtain the output image data. In other words, by converting Concat into convolution, NNA accelerates the computation. Attached Figure Description
[0053] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.
[0054] Figure 1 This is a schematic diagram of the longitudinal-to-horizontal splicing method in this application.
[0055] Figure 2 This is a schematic diagram illustrating an example of horizontal splicing in this application.
[0056] Figure 3 This is a schematic diagram illustrating an example of longitudinal splicing in this application. Detailed Implementation
[0057] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.
[0058] First, convolution includes: input image data a, convolution kernel data b, output image data y, kernel sizes kh and kw, strides sh and sw, and the calculation process is as follows:
[0059]
[0060] i and j are the subscripts / indices of each pixel in the convolution kernel. Convolution is the multiplication and accumulation of the convolution kernel and the corresponding pixels in the input image.
[0061] For example: If kh=1, kw=1, sh=1, sw=1, and the input image size is 4x4, the input image data is shown in the table below:
[0062]
[0063] The convolution kernel data is shown in the table below:
[0064]
[0065] The first point of the output image data is the sum of the multiplication of the convolution kernel data and the input image data (the first shaded area in the first row and first column of the table). As the convolution kernel slides to the right with a stride of 1, the second point of the output image data (horizontally) is the sum of the multiplication of the convolution kernel data and the input image data (the dark shaded area in the second row and second column of the table). As the convolution kernel slides downwards with a stride of 1, the second point of the output image data (vertically) is the sum of the multiplication of the convolution kernel data and the input image data (the light shaded area in the second row and first column of the table). By sliding the convolution kernel horizontally and vertically, the data for all points in the output image are obtained.
[0066] Secondly, concatenation includes: such as Figure 2 and Figure 3 As shown, this involves stitching along the horizontal and vertical directions. Assume we have n input image data points C1, C2, ..., Cn. We stitch these n input image data points horizontally, ensuring all dimensions except the stitching dimension are identical. The output image data is D, where all dimensions except the stitching dimension are identical to the input image size. The stitching dimension is the sum of the stitching dimensions of the input images. The calculation process is as follows:
[0067] D[m] = Cm, m = 1, ..., n
[0068] For example: If there are 3 input images, each 4x4 in size, and they are stitched together horizontally, the output image will be 4x12 in size.
[0069] The first input image data is shown in Table a below:
[0070] Table a:
[0071]
[0072] The second input image data is shown in Table b below:
[0073] Table b:
[0074]
[0075] The third input image data is shown in Table c below:
[0076] Table c:
[0077]
[0078] The output image data is shown in Table d below:
[0079] Table d:
[0080]
[0081] The first to fourth points horizontally in the output image data correspond to the first to fourth points horizontally in the first input image data (the shaded areas of the first four squares in the first row of Table d). The fifth to eighth points horizontally in the output image data correspond to the first to fourth points horizontally in the second input image data (the shaded areas of the four squares in the first row of Table d). The ninth to twelfth points horizontally in the output image data correspond to the first to fourth points horizontally in the third input image data (the shaded areas of the four squares in the first row of Table d). The second point vertically in the output image data corresponds to the second point vertically in the first input image data (the shaded area of the first square in the second row of Table d). This process continues until all points in the output image data are obtained.
[0082] However, NNA only supports convolution acceleration, but in real-world networks, the computation speed of concat can become a bottleneck for network performance. Therefore, by converting the concat process into a convolution process, NNA can be used to accelerate the concat process, thereby further improving network performance. How to accelerate the concat computation using NNA through convolution conversion and improve network performance is the problem this application aims to solve. Therefore, this application proposes a method for concat acceleration, accelerating the computation of layers containing concat in real-world networks and improving network performance. Specifically, it provides a method for accelerating computation by concatenating concat layers into convolution layers, including:
[0083] The process of concatenating (Concat) images to a convolutional image: If there are n input images, concatenating them along a certain dimension yields the output image. Except for the concatenation dimension, all other dimensions have the same size, i.e., as shown below. Figure 2 and Figure 3 As shown, the principle of Concat is: multiple input images are stitched together along a certain dimension, and the size of the other dimensions must be consistent. The size along this dimension is the sum of the sizes of each input image along this dimension (the input images can be regarded as cubes, and the cross-sections of the cubes of each input image along the stitching dimension must be consistent in size, otherwise the correct output shape cannot be obtained).
[0084] Given input image data C and output image data D, the Concat calculation process is as follows:
[0085] D[m, ...] = C[m]
[0086] Where m = 1,...,n, the calculation process of ordinary convolution is as follows: Where a[i][j] is the input image data, b[i][j] is the convolution kernel data, i and j are the indices of each element of the convolution kernel, and the kernel size is kh*kw. Ordinary convolution is simply the multiplication and summation of the convolution kernel and corresponding elements of the input image. To convert the Concat calculation process into the ordinary convolution calculation process, the Concat calculation formula needs to be transformed into the form of the ordinary convolution formula:
[0087]
[0088] To convert Concat into a regular convolution, a new convolution kernel is introduced: the kernel data is always 1, and the kernel size is fixed at 1*1.
[0089] The Concat process is transformed into a regular convolution process where each input image data point is multiplied and summed with the convolution kernel data to obtain the output image data. Specifically, the first point of the output image is the sum of the product of the first point of the first input image and the convolution kernel data. The convolution kernel size is 1*1, and the data is always 1. As the convolution kernel slides sequentially across each input image, the data for all points in the output image is obtained. In other words, n input image data points are sequentially written to the FRAM along a certain dimension, and the convolution kernel data is written to the WRAM. By configuring the NNA register to set the convolution kernel size to 1*1 and the stride to 1*1, and starting NNA, the convolution calculation result, which is also the Concat calculation result, can be obtained. Concat concatenates multiple input images along a single dimension, where all dimensions except the concatenation dimension must be identical in size. Concat only concatenates along a specific dimension, so it only applies to that dimension.
[0090] like Figure 1 As shown, the horizontal splicing in this method specifically includes:
[0091] S1, assuming there are n input images, each of size m x m, which are concatenated horizontally to obtain an output image of size m x mn; that is, the first input image data, the second input image data, ..., the nth input image data are concatenated horizontally to obtain the output image data; the convolution kernel data are all 1;
[0092] S2, the first horizontal point of the output image data is the product and sum of the convolution kernel data (all values are 1) and the first horizontal point of the first input image data; the convolution kernel slides to the right with a stride of 1, the (m+1)th horizontal point of the output image data is the product and sum of the convolution kernel data (all values are 1) and the first horizontal point of the second input image data, and so on, the (n-1)*m+1th horizontal point of the output image data is the product and sum of the convolution kernel data (all values are 1) and the first horizontal point of the nth input image data; the convolution kernel slides down with a stride of 1, the second vertical point of the output image data is the product and sum of the convolution kernel data (all values are 1) and the second vertical point of the first input image data; as the convolution kernel slides horizontally and vertically, the data of all points of the output image are obtained.
[0093] Similarly, this method concatenates along the vertical direction. Assuming there are n input images, each of size m x m, concatenating them vertically yields an output image of size mn x m; that is, the first input image data, the second input image data, ..., the nth input image data; concatenating them vertically yields the output image data; the convolution kernel data are all 1; the specific process of concatenation (Concat) changes from horizontal to vertical, and will not be elaborated here.
[0094] For example: If there are 3 input images, each 4x4 in size, and they are stitched together horizontally, the output image will be 4x12 in size.
[0095] The first input image data is shown in Table 1 below:
[0096] Table 1:
[0097]
[0098] The second input image data is shown in Table 2 below:
[0099] Table 2:
[0100]
[0101] The third input image data is shown in Table 3 below:
[0102] Table 3:
[0103]
[0104] The output image data is shown in Table 4 below:
[0105] Table 4:
[0106]
[0107]
[0108] The convolution kernel data are all set to 1, as shown in Table 5 below:
[0109] Table 5:
[0110]
[0111] The first point of the output image data is the sum of the multiplication of the convolution kernel data (all values are 1) and the first input image data (the shaded area of the first square in the first row and first column of Table 4). The convolution kernel slides to the right with a stride of 1. The fifth point in the horizontal direction of the output image data is the sum of the multiplication of the convolution kernel data (all values are 1) and the second input image data (the shaded area of the first row and fifth column of Table 4). The ninth point in the horizontal direction of the output image data is the sum of the multiplication of the convolution kernel data (all values are 1) and the third input image data (the shaded area of the first row and ninth column of Table 4). The convolution kernel slides downwards with a stride of 1. The second point in the vertical direction of the output image data is the sum of the multiplication of the convolution kernel data (all values are 1) and the first input image data (the shaded area of the second row and first column of Table 4). As the convolution kernel slides horizontally and vertically, the data for all points in the output image is obtained.
[0112] The method is applicable to chips that use NNA for general convolution acceleration calculations, including the Ingenic T32 chip.
[0113] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for accelerating computation by concatenating concat loops and converting them to convolutions, characterized in that, The method includes a concatenation-to-convolution process: if there are n input images, the output image is obtained by concatenating them along a certain dimension, with all dimensions except the concatenation dimension having the same size. Given input image data C and output image data D, the Concat calculation process is as follows: D[m, ...] = C[m] Where m = 1, ..., n, To convert the Concat calculation process into a regular convolution calculation process, the Concat calculation formula needs to be transformed into the form of a regular convolution formula: To convert Concat into a regular convolution, a new convolution kernel is introduced: the kernel data is always 1, and the kernel size is fixed at 1*1. The Concat process is transformed into a regular convolution process where each input image data point is multiplied and summed with the convolution kernel data to obtain the output image data. Specifically, the first point of the output image is the sum of the first point of the first input image and the convolution kernel data. The convolution kernel size is 1*1, and the data is always 1. As the convolution kernel slides sequentially across each input image, the data for all points in the output image is obtained. This involves writing n input image data points sequentially along a certain dimension into the FRAM, and then writing the convolution kernel data into the WRAM. By configuring the NNA register to set the convolution kernel size to 1*1 and the stride to 1*1, and starting NNA, the convolution calculation result, which is also the Concat calculation result, can be obtained.
2. The method for accelerating computation by concatenating concat loops to convolutions according to claim 1, characterized in that, The method assumes horizontal splicing and includes the following steps: S1, assuming there are n input images, each of size m x m, which are concatenated horizontally to obtain an output image of size m x mn; that is, the first input image data, the second input image data, ..., the nth input image data are concatenated horizontally to obtain the output image data; the convolution kernel data are all 1; S2, since the input images are stitched horizontally, the first horizontal point of the output image data is the product and sum of the convolution kernel data and the first horizontal point of the first input image data; the convolution kernel slides to the right with a stride of 1, and the (m+1)th horizontal point of the output image data is the product and sum of the convolution kernel data and the first horizontal point of the second input image data, and so on, until the (n-1)*m+1th horizontal point of the output image data is the product and sum of the convolution kernel data and the first horizontal point of the nth input image data; the convolution kernel slides downward with a stride of 1, and the second vertical point of the output image data is the product and sum of the convolution kernel data and the second vertical point of the first input image data; all the convolution kernel data are 1, and so on, as the convolution kernel slides horizontally and vertically, the data of all points of the output image are obtained.
3. The method for accelerating computation by concatenating concat loops to convolutions according to claim 2, characterized in that, The method further includes: S1, if there are 3 input images, each 4x4 in size, they are stitched together horizontally to obtain an output image of size 4x12: The first input image data is shown in Table 1 below: Table 1: The second input image data is shown in Table 2 below: Table 2: The third input image data is shown in Table 3 below: Table 3: The output image data is shown in Table 4 below: Table 4: The convolution kernel data are all set to 1, as shown in Table 5 below: Table 5: S2, the first point of the output image data is the product and sum of the convolution kernel data (all values are 1) and the first input image data (i.e., the shaded area of the first square in the first row and first column of Table 4); the convolution kernel slides to the right with a stride of 1, the fifth point of the output image data horizontally is the product and sum of the convolution kernel data (all values are 1) and the second input image data (i.e., the shaded area of the first square in the fifth column of Table 4), the ninth point of the output image data horizontally is the product and sum of the convolution kernel data (all values are 1) and the third input image data (i.e., the shaded area of the first square in the ninth column of Table 4); the convolution kernel slides down with a stride of 1, the second point of the output image data vertically is the product and sum of the convolution kernel data (all values are 1) and the first input image data (i.e., the shaded area of the second square in the first row and first column of Table 4); as the convolution kernel slides horizontally and vertically, the data of all points of the output image are obtained.
4. The method for accelerating computation by concatenating concat loops to convolutions according to claim 1, characterized in that, The method assumes splicing along the longitudinal direction and includes the following steps: S1, assuming there are n input images, each of size m x m, which are concatenated vertically to obtain an output image of size mn x m; that is, the first input image data, the second input image data, ..., the nth input image data are concatenated vertically to obtain the output image data; the convolution kernel data are all 1; S2, since the input images are stitched along the vertical direction, the first point of the output image data in the vertical direction is the product and sum of the convolution kernel data and the first point of the first input image data in the vertical direction; the convolution kernel slides down with a stride of 1, and the (m+1)th point of the output image data in the vertical direction is the product and sum of the convolution kernel data and the first point of the second input image data in the vertical direction, and so on, the (m*(n-1)+1)th point of the output image data in the vertical direction is the product and sum of the convolution kernel data and the first point of the nth input image data in the vertical direction; the convolution kernel slides to the right with a stride of 1, and the second point of the output image data in the horizontal direction is the product and sum of the convolution kernel data and the second point of the first input image data in the horizontal direction; all the convolution kernel data are 1, and so on, as the convolution kernel slides horizontally and vertically, the data of all points of the output image are obtained.
5. The method for accelerating computation by concatenating concat loops to convolutions according to claim 1, characterized in that, The convolution includes: input image data a, convolution kernel data b, output image data y, kernel sizes kh and kw, strides sh and sw, and the calculation process is as follows: i and j are the indices of each element of the convolution kernel. Convolution is the multiplication and summation of the convolution kernel and the corresponding elements of the input image.
6. The method for accelerating computation by concatenating concat loops to convolutions according to claim 1, characterized in that, The concat stitching process includes: given n input image data points C1, C2, ..., Cn, these n input image data points are stitched together horizontally, with all dimensions except the stitching dimension being identical in size. The output image data is D, with all dimensions except the stitching dimension being identical in size to the input image. The stitching dimension is the sum of the stitching dimensions of the input images. The calculation process is as follows: D[m] = Cm, m = 1, ..., n.
7. The method for accelerating computation by concatenating concat loops to convolutions according to claim 1, characterized in that, The method is applicable to chips that use NNA for general convolution acceleration calculations, including the Ingenic T32 chip.