A convolution implementation method based on NNA that supports focus operation
By merging the focus layer and the input layer and reordering the data, large convolutional kernels and large strides are transformed into small convolutional kernels and small strides, solving the problems of data bandwidth pressure and computational complexity, and improving the computing efficiency of the NNA hardware platform.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEFEI JUNZHENG TECH CO LTD
- Filing Date
- 2022-03-29
- Publication Date
- 2026-08-04
AI Technical Summary
In existing technologies, the focus layer and the input layer exist as two independent layers, which increases the pressure on data input and output bandwidth and computational complexity. Especially on the NNA hardware platform, convolution operations with a kernel size greater than 3 or a stride greater than 2 cannot be executed effectively.
The focus layer and input layer are merged into one layer. By reordering the data, convolutions with large kernels and large strides are equivalently transformed into convolutions with small kernels and small strides, which satisfies the operational characteristics of NNA, simplifies the computational logic, and reduces data transmission requirements.
It effectively reduces the number of network data inflows and outflows and bandwidth pressure, simplifies the calculation logic, and improves the computing efficiency of the NNA hardware platform.
Smart Images
Figure CN116957004B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of neural network technology, and in particular to a convolution implementation method based on NNA that supports focus operation. Background Technology
[0002] In recent years, with the rapid development of technology, the era of big data has arrived, and data processing techniques in neural networks have increasingly become one of the important application technologies. Chip manufacturers are creating dedicated chips for neural network algorithms, especially inference chips, also known as neural network accelerators (NNAs). Convolutional neural networks mainly consist of input layers, convolutional layers, pooling layers, and fully connected layers. For example... Figure 1 As shown, the function of the focus layer in the neural network is to use a 2*2 region as the basic operation unit and reorder the data in the region in a column-major manner on the input FeatureMap of the focus layer.
[0003] However, because the focus layer is simple and only sorts the data without performing any logical calculations, and the subsequent input layer also involves a similar data sorting process, having the focus and input layers as two separate layers increases bandwidth pressure and data flow.
[0004] In addition, the following are commonly used terms in the prior art:
[0005] 1. NNA: A hardware accelerator on the CPU's SIMD pipeline. Its operation is controlled by special CPU / SIMD instructions and runs on a single thread. It is used to solve most convolution multiplication and addition operations.
[0006] 2. Focus: The neural network layer. Based on basic spatial distribution information, high-resolution patches are gradually obtained from the original image and continuously fed into the network. This process gradually updates the prediction results and spatial distribution information, leading to more accurate judgments and the gradual search for key regions that the neural network has not yet seen. This stage is called Focus.
[0007] 3. BGRA: Pixel format. Summary of the Invention
[0008] To address the aforementioned issues, the purpose of this application is to propose an implementation method that combines the focus layer and the input layer into a single layer to reduce the frequency of network data input and output and the pressure on bandwidth.
[0009] Specifically, this invention provides a convolution implementation method based on NNA that supports focus operations, the method comprising the following steps:
[0010] S1. Assume there are two consecutive layers in the network. The focus layer of the neural network is followed by the input layer. The input layer performs convolution with kernelX*kernelY and strideX*strideY. Here, kernelX and kernelY are the sizes of the input layer's convolution kernel in the X and Y directions before convolution, and strideX and strideY are the convolution stride of the input layer in the X and Y directions before convolution.
[0011] S2, merge the focus layer and the input layer. After merging, it is equivalent to performing a convolution with a kernel size of (2*kernelX)*(2*kernelY) on the FeatureMap that has not undergone the focus operation, and the stride size of the convolution is (2*strideX)*(2*strideY). The focus operation is to reorder the data of each 2*2*C region of the FeatureMap in column-major order and integrate them into a 1*1*4C process, where C refers to the channels of the FeatureMap. After the focus operation, the width and height of the FeatureMap will become half of the original, and the number of channels will increase to 4 times the original.
[0012] S3. Since NNA does not support convolutions with a kernel size greater than 3 or a stride greater than 2, assuming the input layer data has only 4 channels, the input FeatureMap data corresponding to each convolution is reordered. This transforms the convolution with a kernel size of (2*kernelX)*(2*kernelY) and a stride of (2*strideX)*(2*strideY) into a convolution with a size of 1*1 and a stride of 1*1. This ensures that the parameters of the convolution after combining the focus layer and the input layer can meet the limitations of using NNA to implement convolution operations.
[0013] The reordering in step S3 refers to changing the data arrangement order of the convolution calculation to achieve an equivalent convolution that conforms to the MAC operation characteristics of NNA. Taking the input layer as an example, assuming a 3*3*4 channel convolution is performed, combined with the MAC operation characteristics of NNA, a 3*3*32 channel MAC operation will finally be performed in NNA. In this case, only 3 channels of valid data are available in every 32 channels of the MAC operation. By changing the arrangement order of the data required for the convolution operation, i.e., reordering, the 3*3*4 channel convolution can be equivalent to a 1*1*32 channel MAC operation in NNA. In this case, each 32 channels of the MAC operation will have 3*3*3=27 channels of valid data. After reordering, the efficiency of each MAC operation with 32 channels containing valid data is maximized.
[0014] The condition for reordering only needs to satisfy: ensure that the total amount of effective data for convolution operation remains unchanged before and after reordering, and that the correspondence between the input data and the corresponding weights for convolution operation remains unchanged. In other words, it can be guaranteed that the final result of multiplication and addition calculation obtained by convolution operation remains unchanged before and after reordering, which means that the two convolution operations before and after reordering can be regarded as equivalent.
[0015] The method further includes:
[0016] S1, Assume there are two consecutive layers in the network, with the focus layer followed immediately by the input layer, and the input layer performs a 3*3 convolution with a stride of 1.
[0017] S2, the focus layer and the input layer are merged. After merging, it is equivalent to performing a 6*6 convolution with a stride of 2 directly on the input feature map of the focus layer before merging. That is, after the focus layer and the input layer are combined, the input layer that originally performed a 3*3 convolution with a stride of 1*1 will be equivalent to an input layer that performs a 6*6 convolution with a stride of 2*2. The original two network layers, the focus layer and the input layer, are equivalent to a single input layer.
[0018] S3 transforms a 6*6 convolution with a stride of 2*2 into a 1*1 convolution with a stride of 1*1 by reordering the input FeatureMap data corresponding to each convolution.
[0019] The channel format in step S3 is BGRA format; the parameters of the convolution include the kernel size and the stride size.
[0020] Therefore, the advantage of this application is that it can be implemented by merging the Focus layer and the Input layer into one layer through a simple method, which reduces the number of network data inputs and outputs and the pressure on bandwidth. While reducing the number of network data inputs and outputs and the pressure on bandwidth, it also simplifies the logical calculation. Attached Figure Description
[0021] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.
[0022] Figure 1 This is a schematic diagram of the focus layer operation involved in this application.
[0023] Figure 2 This is a schematic diagram illustrating the correspondence between convolution before and after focusing in the method of this application.
[0024] Figure 3 This is a flowchart illustrating the method used in this application. Detailed Implementation
[0025] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.
[0026] Convolutional neural networks have wide applications in image, video, and speech processing. As neural networks become more complex, they require a lot of resources for training and evaluation, while the performance growth and versatility improvement of hardware accelerators are slowing down.
[0027] Convolutional neural networks (CNNs) mainly consist of an input layer, convolutional layers, pooling layers, and fully connected layers. Among them, the convolutional layer is the core layer of the CNN, generating the vast majority of the computational load. Therefore, the operating speed of a CNN basically depends on the computational speed of the convolutional layer.
[0028] NNA (neural network accelerate) is a hardware-based neural network accelerator that enables fast multi-channel convolution operations by configuring relevant register parameters, thereby significantly reducing the running time of neural networks. In practical applications, it offers higher real-time performance and a better user experience.
[0029] The following example illustrates the scheme of this application:
[0030] Assume there are two consecutive layers in the network, with the focus layer immediately followed by the input layer, and the input layer performs convolutions with a 3x3 kernel and a stride of 1.
[0031] Therefore, based on the principle of the focus layer, merging the two network layers is equivalent to performing a 6x6 convolution with a stride of 2 on the feature map that has not undergone the focus operation; for example... Figure 2 As shown.
[0032] Combining the Focus layer and the convolutional layer is equivalent to performing a convolution with a kernel size of (2*kernelX)*(2*kernelY) on the unfocused layer, with a stride size of (2*strideX)*(2*strideY). Here, kernelX and kernelY are the kernel sizes of the unfocused convolutional layer in the X and Y directions, respectively, and strideX and strideY are the strides of the unfocused convolutional layer in the X and Y directions, respectively.
[0033] Taking the combination of the focus layer and the input layer as an example, after their combination, the input layer that originally performed a 3*3 convolution with a stride of 1*1 is equivalently transformed into an input layer that performs a 6*6 convolution with a stride of 2*2. Since NNA does not support convolutions with more than 3 kernels, and the input layer data only has 4 channels (BGRA), by reordering the input FeatureMap data corresponding to each convolution, the 6*6 convolution with a stride of 2*2 can be equivalently transformed into a 1*1 convolution with a stride of 1*1, making it possible to implement the combination of the focus layer and the input layer on NNA.
[0034] In summary, the process of this application is as follows: Figure 3 As shown:
[0035] S1, Assume there are two consecutive layers in the network, with the focus layer immediately followed by the input layer. The input layer performs a convolution with kernel size X * kernel Y and a stride size X * stride Y, where kernel X and kernel Y are the sizes of the input layer's convolution kernel in the X and Y directions before convolution, and stride X and stride Y are the stride sizes of the input layer in the X and Y directions before convolution. S2, Merge the focus layer and the input layer. This merging is equivalent to performing a convolution with kernel size (2 * kernel X) * (2 * kernel Y) on the FeatureMap before the focus operation, with a stride size of (2 * stride X) * (2 * stride Y). Refer to the appendix of the specification. Figure 1 The focus operation reorders each 2*2*C region of the FeatureMap in column-major order, integrating them into a 1*1*4C structure, where C refers to the number of channels in the FeatureMap. After the focus operation, the width and height of the FeatureMap are halved, and the number of channels is increased to four times. S3, by reordering the input FeatureMap data corresponding to each convolution, the convolution with kernel size (2*kernelX)*(2*kernelY) and stride (2*strideX)*(2*strideY) is equivalently transformed into a convolution with size 1*1 and stride 1*1. This ensures that the parameters of the convolution after combining the focus layer and the input layer, such as kernel size and stride, can meet the limitations of using NNA to implement convolution operations. Specifically, NNA does not support convolutions with kernel size greater than 3 or stride greater than 2.
[0036] Reordering refers to changing the data arrangement order in convolution calculations to achieve an equivalent convolution that better conforms to the MAC operation characteristics of NNA. Taking the input layer as an example, if a 3x3x4 channel convolution is performed, combined with the MAC operation characteristics of NNA, it will ultimately result in a 3x3x32 channel MAC operation in NNA. In this case, only 3 out of every 32 channels in the MAC operation are valid data. By changing the arrangement order of the data required for the convolution operation, i.e., reordering, the 3x3x4 channel convolution can be equivalent to a 1x1x32 channel MAC operation in NNA. In this case, each 32 channels in the MAC operation will have 3x3x3 = 27 valid data channels. After reordering, the efficiency of each MAC operation is maximized by ensuring that the total amount of valid data in the convolution operation remains unchanged before and after reordering, and that the correspondence between the input data and the corresponding weights remains unchanged. This guarantees that the final result of the multiplication-addition calculation in the convolution operation remains the same before and after reordering, meaning that the two convolution operations before and after reordering can be considered equivalent.
[0037] Specifically, one embodiment of the method includes:
[0038] S1, Assume there are two consecutive layers in the network, with the focus layer followed immediately by the input layer, and the input layer performs a 3*3 convolution with a stride of 1.
[0039] S2, the focus layer and the input layer are merged. After merging, it is equivalent to performing a 6*6 convolution with a stride of 2 directly on the input feature map of the focus layer before merging. That is, after the focus layer and the input layer are combined, the input layer that originally performed a 3*3 convolution with a stride of 1*1 will be equivalent to an input layer that performs a 6*6 convolution with a stride of 2*2. The original two network layers, the focus layer and the input layer, are equivalent to a single input layer.
[0040] S3 transforms a 6*6 convolution with a stride of 2*2 into a 1*1 convolution with a stride of 1*1 by reordering the input FeatureMap data corresponding to each convolution.
[0041] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A convolution implementation method based on NNA that supports focus operation, characterized in that, The method includes the following steps: S1, Assume there are two consecutive layers in the network, with the focus layer immediately followed by the input layer, and the input layer undergoes convolution with a kernel of... The convolution of Y, with a stride of . Where kernelX and kernelY are the sizes of the convolutional kernels of the input layer in the X and Y directions before they are combined, and strideX and strideY are the convolutional strides of the input layer in the X and Y directions before they are combined; S2, merge the focus layer and the input layer. The merge is equivalent to performing a convolution on the FeatureMap without a focus operation. The convolution has a stride of 1. The focus operation is to focus each featureMap element... The data in each region is reordered and consolidated according to column priority. The process involves focusing a FeatureMap, where C refers to the number of channels. After focusing, the width and height of the FeatureMap will be halved, and the number of channels will increase to four times the original size. S3. Since NNA does not support convolutions with a kernel size greater than 3 or a stride greater than 2, assuming the input layer data only has 4 channels, the kernel size is adjusted by reordering the input FeatureMap data corresponding to each convolution. And step length The convolution is equivalent to the size transformation. Step length The convolution is such that the parameters of the convolution after combining the focus layer and the input layer can meet the constraints of implementing convolution operations using NNA; wherein, the reordering in step S3 refers to changing the data arrangement order of the convolution calculation to achieve an equivalent convolution that conforms to the MAC operation characteristics of NNA. Taking the input layer as an example, assuming that... Channel convolution, combined with the MAC operation characteristics of NNA, will finally be performed in NNA. In channel-wise MAC operations, only 3 out of every 32 channels in the MAC operation are valid data; this can be achieved by changing the order of the data required for the convolution operation, i.e., reordering. Channel convolution is performed in NNA. The equivalent MAC operation for each channel, in this case, every 32 channels in the MAC operation will... The effective data of each channel is reordered to maximize the efficiency of each MAC operation by including effective data in 32 channels. The reordering condition only needs to satisfy the following: ensure that the total amount of effective data for convolution operation remains unchanged before and after reordering, and that the correspondence between the input data and the corresponding weights for convolution operation remains unchanged. In other words, it can be guaranteed that the final result of the multiplication and addition calculation obtained by convolution operation remains unchanged before and after reordering, which means that the two convolution operations before and after reordering can be regarded as equivalent.
2. The convolution implementation method supporting focus operation based on NNA according to claim 1, characterized in that, The method further includes: S1, Assume there are two consecutive layers in the network, with the focus layer immediately followed by the input layer, and the input layer undergoes convolutional kernels. The convolution has a stride of 1. S2, merging the focus layer and the input layer is equivalent to directly performing a convolution kernel on the input feature map of the focus layer before merging. The convolution has a stride of 2; that is, after the focus layer and the input layer are combined, the original convolution with a stride of 2 is reduced to a size of 2. Step length The input layer of a convolution is equivalent to performing a convolution with a size of Step length The input layer of a convolution; The original two network layers, the focus layer and the input layer, are equivalently transformed into a single input layer. S3, by reordering the input FeatureMap data corresponding to each convolution, changes the size... Step length The convolution is equivalent to the size transformation. Step length The convolution.
3. The convolution implementation method supporting focus operation based on NNA according to claim 1, characterized in that, The channel format in step S3 is BGRA format; the parameters of the convolution include the kernel size and the stride size.