A winograd-based deep learning processing method

Abstract

The present disclosure belongs to the technical field of neural network processing methods, and in particular to a deep learning processing method based on Winograd, comprising: obtaining the size of a unit to be calculated; obtaining the sizes of a plurality of acceleration cores with different sizes; obtaining a calculation ratio according to the sizes of the plurality of acceleration cores; obtaining a division region of the unit to be calculated according to the size of the unit to be calculated and the calculation ratio, the division region of the unit to be calculated corresponding to the size of the acceleration core; performing calculation according to the division region, and then collecting and completing the acceleration calculation of the unit to be calculated. The present disclosure utilizes the difference in flexibility and efficiency of Winograd acceleration cores (Wino-Core) with different convolution kernel sizes and output sizes, combines a plurality of Wino-Cores with different sizes in a highly scalable deep learning processor architecture, forms a multi-core mixed heterogeneous deep learning processor, and realizes the overall improvement of the flexibility and efficiency of the Wino-DPU through the optimal allocation of the calculation power between the cores.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of neural network processing methods, and in particular relates to a Winograd-based deep learning processing method. BACKGROUND

[0002] With the performance of deep learning technology represented by the Convolutional Neural Network (CNN) continuously improving in the fields of image classification, target detection, etc., designing an efficient deep learning processor based on FPGA has always been a research hotspot in the industry. However, the DSP resources on FPGA are often limited, which limits the performance of the processor. The Winograd algorithm is one of the most effective ways to solve the above problems. It can greatly reduce the complexity of convolution calculation. Specifically, it can greatly reduce the number of multiplications by increasing a small number of additions. However, the deep learning processor designed based on the Winograd algorithm can only efficiently process convolution with a fixed size of convolution kernel, and has low efficiency for convolution with other sizes. The characteristics of the Winograd algorithm for reducing computational complexity often fail to fully play a role in deep learning processors. In addition, the 1x1 accelerator designed by nested loop unrolling and local parallelization of convolution layers has high flexibility, but has no equivalent speedup ratio and limited computing power. SUMMARY

[0003] The present disclosure is proposed based on the above needs of the prior art, and the technical problem to be solved by the present disclosure is to provide a Winograd-based deep learning processing method to improve the flexibility and efficiency of the deep learning processor.

[0004] To solve the above problems, the technical scheme provided by the present disclosure includes:

[0005] A Winograd-based deep learning processing method is provided, including: obtaining the size of a unit to be calculated; obtaining the sizes of a plurality of acceleration kernels with different sizes; obtaining a calculation ratio according to the sizes of the plurality of acceleration kernels; obtaining a division region of the unit to be calculated according to the size of the unit to be calculated and the calculation ratio, the division region of the unit to be calculated corresponding to the size of the acceleration kernel; performing calculation according to the division region, and then collecting the acceleration calculation of the unit to be calculated.

[0006] Preferably, the unit to be calculated includes a convolution kernel and an input feature map; the acceleration kernel includes F1(n1, m1), F2(n2, m2), …, F i (n i ,m i ), wherein F i (n i ,m i) represents the i-th accelerator core, whose size is m i ×m i , whose output feature size is n i ×n i .

[0007] Preferably, when the to-be-calculated unit is an input feature map, the calculation ratio according to the size of the plurality of accelerator cores is a peak computing power ratio, which is represented as: m1×m1×n1×n1:m2×m2×n2×n2:…:m i ×m i ×n i ×n i .

[0008] Preferably, when the to-be-calculated unit is an input feature map, the division region of the to-be-calculated unit according to the size of the to-be-calculated unit and the peak computing power ratio includes the following process:

[0009]

[0010] wherein N i is the number of rows of the input feature map corresponding to the i-th accelerator core, and s is the number of columns or rows of the input feature map.

[0011] Preferably, when the to-be-calculated unit is a convolution kernel, the calculation ratio according to the size of the plurality of accelerator cores includes: m1×m1:m2×m2:…:m i ×m i .

[0012] Preferably, when the to-be-calculated unit is a convolution kernel, the division region of the to-be-calculated unit according to the size of the to-be-calculated unit and the calculation ratio includes the following process: wherein N i ′ is the number of units of the area corresponding to the i-th accelerator core, and the unit area is 1×1, and the size of the convolution kernel is t×t.

[0013] Preferably, when the to-be-calculated unit is a convolution kernel, the division region of the to-be-calculated unit according to the size of the to-be-calculated unit and the calculation ratio further includes the following process: wherein Q i is the number of calculations of the area corresponding to the i-th accelerator core, q is the remainder generated in the above process, and P i is the actual number of the area corresponding to the i-th accelerator core.

[0014] Preferably, the size of the smallest accelerator core in the accelerator cores is 1×1.

[0015] Preferably, the sizes of the plurality of acceleration cores are arranged in order of size; when the calculation region is divided, the region corresponding to the largest acceleration core size is divided from the top corner position; and the division continues along the row direction or the column direction until the number of divided regions is equal to the actual number of regions corresponding to the largest acceleration core; and then the region division is performed in order of size corresponding to the size of the acceleration core and the actual number.

[0016] Preferably, when the to-be-calculated unit is smaller than the division region corresponding to the acceleration core, the part of the to-be-calculated unit smaller than the division region is supplemented with 0.

[0017] Compared with the prior art, the disclosure utilizes the difference in flexibility and efficiency of Winograd acceleration cores (Wino-Core) with different convolution kernel sizes and output sizes, combines a plurality of Wino-Cores with different sizes in a highly scalable deep learning processor architecture, forms a multi-core mixed heterogeneous deep learning processor, and realizes the overall improvement of the flexibility and efficiency of the Wino-DPU through the optimal allocation of the computing power between the cores. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present specification, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0019] Figure 1 A step flowchart of a deep learning processing method based on Winograd provided by the present application;

[0020] Figure 2 A system structure block diagram corresponding to a deep learning processing method based on Winograd provided by the present application;

[0021] Figure 3 A specific real-time manner of region division for a convolution kernel provided by the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0023] In the description of the embodiments of the present disclosure, it should be noted that, unless otherwise explicitly specified and limited, the term "connected" should be understood in a broad sense, for example, it can be fixedly connected, or detachably connected, or integrally connected, which can be mechanically connected, or electrically connected, which can be directly connected, or indirectly connected through an intermediate medium. For those skilled in the art, the specific meaning of the above-mentioned term in the present disclosure can be understood according to the specific circumstances.

[0024] The terms "top", "bottom", "above", "under" and "on" used throughout the description are relative positions of components of the device, for example, the relative positions of the top and bottom substrates inside the device. It can be understood that the device is multifunctional, regardless of their orientation in space.

[0025] In order to facilitate the understanding of the embodiments of the present application, the following will be further explained and described with specific embodiments in conjunction with the accompanying drawings, and the embodiments do not constitute a limitation on the embodiments of the present application.

[0026] The present embodiment provides a Winograd-based deep learning processing method, as shown in Figures 1-3 .

[0027] As shown in Figure 1 , the Winograd-based deep learning processing method comprises:

[0028] The size of the unit to be calculated is obtained.

[0029] The unit to be calculated includes a convolution kernel and an input feature map, and the size of the convolution kernel or the size of the input feature map is obtained.

[0030] The sizes of a plurality of acceleration kernels with different sizes are obtained.

[0031] The acceleration kernel is a convolution accelerator core corresponding to the Winograd algorithm, and the plurality of acceleration kernels have different kernel sizes and output sizes. When the unit to be calculated is a convolution kernel, the acceleration kernel can divide the convolution kernel into a plurality of sub-convolution kernels according to the acceleration kernel to process the input data, and when the unit to be calculated is an input feature map, the acceleration kernel will directly act on the input feature map and process it. The acceleration kernel is represented as F i (n i ,m i ), F i (n i ,m i ) represents the i-th acceleration kernel, whose size is m i ×m i , and the output feature size is n i ×n iFor ease of understanding, a specific implementation method is introduced here in advance, which is related to the following description, such as... Figure 2 As shown in the figure, this specific embodiment includes three parallel Wino-Cores: F1(2,3)Wino-Core, F2(2,2)Wino-Core, and F3(1,1)Wino-Core. F1(2,3) indicates that the first type of convolutional kernel has a size of 3×3 and an output feature size of 2×2; F2(2,2) indicates that the second type of convolutional kernel has a size of 2×2 and an output feature size of 2×2; and F3(1,1) indicates that the third type of convolutional kernel has a size of 1×1 and an output feature size of 1×1. By using a parallel, hybrid heterogeneous structure, acceleration kernels with different characteristics are aggregated, thereby simultaneously accelerating computation on the computing unit, comprehensively improving the flexibility and efficiency of deep learning processing methods.

[0032] The calculation ratio is obtained based on the size of multiple acceleration cores.

[0033] When the unit to be computed is the input feature map, the computation ratio obtained based on the size of multiple acceleration cores is the peak computing power ratio, specifically expressed as: m1×m1×n1×n1 : m2×m2×n2×n2 : … : m i ×m i ×n i ×n i Corresponding to the specific implementation described above, when the unit to be computed is the input feature map, the peak computing power ratio obtained by the acceleration core size is: 2×2×3×3:2×2×2×2:1×1×1×1, that is, the final peak computing power ratio is 36:16:1.

[0034] When the unit to be computed is a convolution kernel, the computational ratio obtained based on the size of multiple acceleration kernels can be expressed as: m1×m1 : m2×m2 : … : m i ×m i Corresponding to the specific implementation described above, when the computational unit is a convolutional kernel, the computational ratio obtained by adjusting the kernel size is 3×3:2×2:1×1, which means the final computational ratio is 9:4:1.

[0035] The division region of the unit to be computed is obtained based on the size of the unit to be computed and the computation ratio, and the division region of the unit to be computed corresponds to the size of the acceleration core.

[0036] When the unit to be computed is the input feature map, the process of dividing the unit into regions based on its size and peak computing power ratio includes the following steps:

[0037]

[0038] Where N iis the number of rows of the input feature map corresponding to the i-th acceleration core, and s is the number of columns or rows of the input feature map.

[0039] According to the above process, when the to-be-calculated unit is an input feature map and the size of the input feature map is 128x128, the division region corresponding to the acceleration core F1(2, 3) is that is, N1≈87; similarly, the division region corresponding to the acceleration core F2(2, 2) is that is, N2≈39; in addition, the division region corresponding to the acceleration core F3(1, 1) is that is, N3≈2. In other words, through the above calculation, the first 86 rows of the feature map or the first 87 columns of the feature map are allocated to F1(2, 3) for processing, the middle 39 rows of the feature map or the middle 38 columns of the feature map are allocated to F2(2, 2) for processing, and the remaining 4 rows of the feature map or the remaining 2 columns of the feature map are allocated to F3(1, 1) for processing.

[0040] When the to-be-calculated unit is a convolution kernel, the division region of the to-be-calculated unit is obtained according to the size and calculation ratio of the to-be-calculated unit, and the process includes the following steps:

[0041]

[0042] wherein, N i ′ is the number of unit areas of the division region corresponding to the i-th acceleration core, the unit area is 1x1, and the size of the convolution kernel is txt.

[0043] According to the above process, when the to-be-calculated unit is a convolution kernel and the size of the convolution kernel is 5x5, the number of unit areas of the division region corresponding to the acceleration core F1(2, 3) is that is, N1≈16; similarly, the number of unit areas of the division region corresponding to the acceleration core F2(2, 2) is that is, N1≈7; in addition, the number of unit areas of the division region corresponding to the acceleration core F3(1, 1) is that is, N1≈2.

[0044] Further, when the to-be-calculated unit is a convolution kernel, the process of dividing the region further includes:

[0045]

[0046]

[0047] wherein, Q i is the number of calculations of the division region corresponding to the i-th acceleration core, q is the remainder generated in the above process, and P i is the actual number of the division region corresponding to the i-th acceleration core.

[0048] In the above specific embodiment, the number of calculations corresponding to the region of the acceleration core F1(2, 3) is The above process generates a remainder, so that the actual number of regions corresponding to the first acceleration core is 2; the number of calculations corresponding to the region of the acceleration core F2(2, 2) is The above process generates a remainder, so that the actual number of regions corresponding to the first acceleration core is 2; the number of calculations corresponding to the region of the acceleration core F3(1, 1) is So the actual number of regions corresponding to the third acceleration core is 2.

[0049] When the to-be-calculated unit is a convolution kernel, the division region and the number of different division regions corresponding to the division region are obtained through the above process, and then the arrangement mode of the division region needs to be determined.

[0050] First, arrange the sizes of the plurality of acceleration cores in order of size, and in this specific embodiment, the acceleration cores are arranged in order of size from large to small: F1(2, 3), F2(2, 2), and F3(1, 1).

[0051] Next, start dividing from the top corner position, divide the region corresponding to the largest acceleration core size from the top corner position, and continue dividing along the row direction or column direction until the number of division regions is equal to the actual number of regions corresponding to the largest acceleration core; then divide the regions in order of size corresponding to the size of the acceleration core.

[0052] In this specific embodiment, as shown in Figure 3 the region corresponding to F1(2, 3) is set as the division region at the top left corner position of the convolution kernel, that is, a 3x3 region is divided from the top left corner in the 5x5 convolution kernel, and the region is continuously divided along the column direction. The number of 3x3 regions obtained above is 2, then a 3x3 region is divided below the 3x3 region at the top left corner, and since there is only one 3x2 region left at the left side at this time, a 0 is supplemented at the bottom to form another 3x3 region without affecting the final calculation result, thereby speeding up the calculation rate of convolution; at this time, the remaining division region of the convolution kernel is 5x2, which is divided into two 2x2 regions corresponding to F2(2, 2) stacked above and below and two 1x1 regions corresponding to F3(1, 1) stacked left and right.

[0053] According to the division region, the calculation is performed, and then the acceleration calculation of the to-be-calculated unit is completed.

[0054] The present application integrates acceleration cores with different characteristics by mixing heterogeneous ways, and realizes the comprehensive improvement of flexibility and efficiency of the deep learning processor.

[0055] The above detailed description of the specific implementation is further detailed for the purpose of the application, technical solutions and beneficial effects, and it should be understood that the above description is only for the specific implementation of the application and is not used to limit the protection scope of the application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A Winograd-based deep learning processing method, characterized in that, The method comprises the following steps: obtaining the size of a to-be-calculated unit; obtaining the sizes of a plurality of acceleration cores with different sizes; obtaining a calculation ratio according to the sizes of the plurality of acceleration cores; obtaining a division region of the to-be-calculated unit according to the size of the to-be-calculated unit and the calculation ratio, the division region of the to-be-calculated unit corresponding to the size of the acceleration core; performing calculation according to the division region, and then performing acceleration calculation of the to-be-calculated unit by summarizing; the to-be-calculated unit refers to convolution calculation of a convolution kernel and an input feature map; The acceleration cores include F1(n1, m1), F2(n2, m2), …, F i (n i ,m i ), where F i (n i ,m i ) represents the i-th acceleration core, the convolution kernel size of which is m i ×m i , and the output feature size is n i ×n i ; When the to-be-calculated unit is divided according to the input feature map, the calculation ratio according to the size of the plurality of acceleration cores is the peak calculation power ratio, which is represented as: m1xm1xn1xn1:m2xm2xn2xn2:…:m i xm i xn i xn i ; when the to-be-calculated unit is an input feature map, obtaining the division region of the to-be-calculated unit according to the size of the to-be-calculated unit and the peak calculation power ratio comprises the following steps: where N i is the number of rows of the input feature map corresponding to the i-th accelerator, and s is the number of columns or rows of the input feature map. 2.The Winograd-based deep learning processing method of claim 1, wherein, When the to-be-calculated unit is divided according to the convolution kernel, the calculation ratio obtained according to the sizes of the plurality of acceleration kernels comprises: m1×m1: m2×m2: …: m i ×m i . 3.The Winograd-based deep learning processing method of claim 2, wherein, when the to-be-calculated unit is a convolution kernel, obtaining the division region of the to-be-calculated unit according to the size of the to-be-calculated unit and the calculation ratio comprises the following steps: T = t 2 where N i is the number of regions corresponding to the i-th accelerator per unit area, and the size of the convolution kernel is t x t.

4. The Winograd-based deep learning processing method of claim 3, wherein, when the to-be-calculated unit is a convolution kernel, obtaining the division region of the to-be-calculated unit according to the size of the to-be-calculated unit and the calculation ratio further comprises the following steps: wherein Q i is the number of regions corresponding to the i-th acceleration core, q is the remainder generated by the above process, and P i is the actual number of regions corresponding to the i-th acceleration core.

5. The Winograd-based deep learning processing method of claim 1, wherein, the size of the smallest acceleration core in the acceleration core is 1*1.

6. The Winograd-based deep learning processing method according to claim 5, characterized in that, arranging the sizes of the plurality of acceleration cores in order of size; when dividing the calculation region, starting from the top corner position to divide the region corresponding to the size of the largest acceleration core; and continue to divide along the row direction or the column direction until the number of the division regions is equal to the actual number of the region corresponding to the largest acceleration core; then divide the regions in order of size corresponding to the size of the acceleration core and the actual number.

7. The Winograd-based deep learning processing method according to claim 1, characterized in that, when the to-be-calculated unit is smaller than the division region corresponding to the acceleration core, the part of the to-be-calculated unit smaller than the division region is supplemented with 0.

Images