Neural network processing device, neural network processing method, and neural network processing program

Abstract

To realize reduction of an operation amount while expanding an application range when winograd conversion processing is applied to a convolution neural network.SOLUTION: A neural network processing apparatus creates a plurality of sub input feature maps from an input feature map according to a value of an expansion rate for a two-dimensional convolution layer that is an expanded convolution layer in which an expansion rate is more than or equal to 2; creates a reduced kernel, in which a part being 0 is removed from an input kernel, according to the value of the expansion rate; performs winograd conversion processing for each of the plurality of sub input feature maps and the reduced kernel; obtains a plurality of sub convolution results corresponding respectively to the plurality of sub input feature maps; and synthesizes the plurality of sub convolution results to obtain a convolution result for the two-dimensional convolution layer.SELECTED DRAWING: Figure 2

Description

Technical Field

[0004] , , ,

[0001] The disclosed technology relates to a neural network processing device, a neural network processing method, and a neural network processing program.

Background Art

[0002] In a convolutional neural network (CNN), a network model is composed of multiple layers, and convolutional processing is performed in a convolutional layer. In the convolutional processing, as shown in FIG. 10, an input feature map (also referred to as input fmap) of a previous layer or the like and a kernel as a weight coefficient are input. A bias value is added to the convolutional operation result obtained by performing a convolutional operation (sum-of-products operation) on the input feature map and the input kernel, and activation function processing is performed. Thereby, an output feature map is obtained.

[0003] When implementing the inference processing of CNN on dedicated hardware such as LSI (Large Scale Integration) or FPGA (Field Programmable Gate Array), the convolutional operation occupies most of the computational amount in the inference processing. Since the convolutional operation is a repetition of the sum-of-products operation, the operation circuit is composed of a multiplier and an adder. In particular, since the required circuit area of the multiplier is larger than that of the adder, the multiplier of the convolutional operation unit occupies most of the arithmetic units of the CNN inference hardware. Therefore, the multiplier of the convolutional operation affects the area or power of the hardware, and whether the multiplier can be effectively utilized affects the throughput (inference processing performance) per area or power.

[0004] The winograd transform is a known technique for reducing the number of multiplications in convolution operations (see, for example, Non-Patent Document 1). As shown in Figure 11, the winograd transform involves performing a matrix transformation on both the input feature map and the input kernel beforehand, multiplying the transformed input feature map and the transformed input kernel by the Hadamard product, and then performing another matrix transformation afterward. This allows for obtaining results similar to those of a convolution operation. In this case, the number of multiplications required for the Hadamard product is reduced compared to the number of multiplications required for the original convolution operation, thus reducing the overall computational load. In the case of inference hardware, fewer multiplication circuits need to be implemented, improving throughput per unit of circuit area or power.

[0005] The winograd transformation shown in Figure 11 is expressed by the following equation (1), where I is the input feature map, I' is the transformed input feature map, K is the input kernel, K' is the transformed input kernel, O' is the Hadamard product result, O is the convolution result, and A, B, and C are the transformation matrices. Also, T is the transpose matrix.

[0006] JPEG0007909482000001.jpg11107(1)

[0007] For example, Figure 12 shows the case where the input feature map size is 4x4, the input kernel size is 3x3, and the convolution result size is 2x2.

[0008] In the case of the size configuration shown in Figure 12, the transformation matrices A, B, and C used for the winograd transformation are determined as shown in equation (2) below.

[0009] JPEG0007909482000002.jpg21127(2)

[0010] The size of the transformed input feature map is 4x4, and the size of the transformed input kernel is 3x3. The Hadamard product of these requires 16 multiplications. The transformation matrix A after the Hadamard product multiplication results in a 2x2 convolution of the 4x4 Hadamard product result O'.

[0011] In a standard convolution operation, (3×3)×(2×2)=36 multiplications are required, whereas by using the winograd transform, only 16 multiplications are needed, a reduction of 4 / 9 times. Similarly, for a 5×5 input kernel, if the size of the convolution result is 2×2, the number of multiplications is reduced to 9 / 25 times. [Prior art documents] [Non-patent literature]

[0012] [Non-Patent Document 1] “Fast Algorithms for Convolutional Neural Networks”, Andrew Lavin, Scott Gray. [Overview of the project] [Problems that the invention aims to solve]

[0013] On the other hand, transformation matrices A, B, and C differ for each kernel size. When implementing CNN inference processing in hardware, it is not practical to prepare transformation matrices A, B, and C for every kernel size, nor is it practical to implement a winograd transformation circuit that can perform matrix processing using all transformation matrices A, B, and C.

[0014] Therefore, it is common to prepare a winograd transformation circuit that is compatible only with specific kernel sizes (e.g., 3x3). In the case of 3x3, the winograd transformation circuit is used to improve processing speed, while for sizes other than 3x3, the winograd transformation circuit is not used, and the normal convolution operation is performed without improving processing speed. In this case, the range in which the winograd transformation circuit can be used is limited, and it lacks versatility.

[0015] Furthermore, applying winograd transforms to layers other than standard convolutional layers is difficult, and even if applied, they may not produce sufficient results.

[0016] Recent CNN inference models often include a special type of convolutional layer called a dilated convolution. The dilated convolution only enables discrete locations within the kernel. The interval between discrete locations is called the dilation rate (dr), and for a 3x3 kernel, dilation rates (dr) = 1, 2, and 4 are as shown in Figure 13. A dilation rate (dr) of 1 is the same as a normal convolution.

[0017] Extended convolutional layers allow convolution to be performed on a wide range of input feature maps without changing the number of sums of products in the convolution operation. For example, an input kernel with a size of 3x3 and an extension ratio of 2 is equivalent to a 5x5 input kernel where 16 out of 25 points have a value of 0, but the number of operations is the same as a normal 3x3 convolution, which is 9 / 25 times that of a 5x5 kernel.

[0018] However, the winograd transform cannot handle discrete kernels, so the only way to handle them is to fill in invalid kernels that are not discrete with zeros and treat them as normal convolutional layers. For example, a 3x3 kernel with an expansion ratio of 2 can only be handled by the winograd transform by treating it as a 5x5 layer.

[0019] By treating it as a 5x5 kernel and performing a winograd transform, the number of multiplications can be reduced compared to a normal 5x5 convolutional layer. However, since a normal 5x5 convolutional layer has a larger number of multiplications than a normal 3x3 convolutional layer, the combined effect of reducing the number of multiplications is negated. In other words, even if the number of multiplications is reduced to 9 / 25 times by the winograd transform, treating it as a 5x5 layer results in 25 / 9 times, effectively making it 1 time.

[0020] Thus, the range in which winograd transform circuits can be applied is limited, and furthermore, in special convolutional layers such as extended convolutional layers, the effect of reducing the number of multiplications is lost, and sufficient speed improvements in processing cannot be obtained by winograd transform circuits.

[0021] The disclosed technology was developed in view of the above points, and aims to provide a neural network processing device, a neural network processing method, and a neural network processing program that can reduce the amount of computation while expanding the scope of application when applying winograd transformation processing to a convolutional neural network. [Means for solving the problem]

[0022] A neural network processing device according to one aspect of the present disclosure includes: a sub-input feature map creation unit that creates a plurality of sub-input feature maps from an input feature map according to the value of the expansion ratio for a two-dimensional convolutional layer which is an expanded convolutional layer having an expansion ratio of 2 or more specified in the winograd transformation process of a convolutional neural network; a reduction kernel creation unit that creates a reduction kernel by removing the portion that is 0 from the input kernel according to the value of the expansion ratio; a winograd transformation processing unit that performs a winograd transformation on each of the plurality of sub-input feature maps and the reduction kernel to obtain a plurality of sub-convolution results corresponding to each of the plurality of sub-input feature maps; and a sub-convolution result synthesis unit that synthesizes the plurality of sub-convolution results to obtain a convolution result for the two-dimensional convolutional layer.

[0023] A neural network processing method according to one aspect of the present disclosure involves, for a two-dimensional convolutional layer which is an expanded convolutional layer whose expansion ratio is 2 or more as specified in the winograd transform process of a convolutional neural network, creating a plurality of sub-input feature maps from an input feature map according to the value of the expansion ratio, creating a reduced kernel by removing the portion that is 0 from the input kernel according to the value of the expansion ratio, performing a winograd transform process on each of the plurality of sub-input feature maps and the reduced kernel, obtaining a plurality of sub-convolution results corresponding to each of the plurality of sub-input feature maps, and synthesizing the plurality of sub-convolution results to obtain a convolution result for the two-dimensional convolutional layer.

[0024] A neural network processing program according to an aspect of the present disclosure causes a computer to execute a process of creating a plurality of sub-input feature maps from an input feature map according to a value of an expansion rate, creating a reduced kernel obtained by removing portions having a value of 0 from an input kernel according to the value of the expansion rate, performing a Winograd conversion process on each of the plurality of sub-input feature maps and the reduced kernel, obtaining a plurality of sub-convolution results corresponding to each of the plurality of sub-input feature maps, and synthesizing the plurality of sub-convolution results to obtain a convolution result for the two-dimensional convolutional layer, where the two-dimensional convolutional layer is an expanded convolutional layer in which the expansion rate specified in the Winograd conversion process of the convolutional neural network is 2 or more.

Advantages of the Invention

[0025] According to the disclosed technique, when applying the Winograd conversion process to a convolutional neural network, it is possible to expand the applicable range and reduce the amount of computation.

Brief Description of the Drawings

[0026] [Figure 1] It is a block diagram showing an example of the hardware configuration of a neural network processing apparatus according to an embodiment. [Figure 2] It is a block diagram showing an example of the configuration of a neural network processing apparatus according to an embodiment. [Figure 3] It is a diagram for explaining the convolution process according to an embodiment. [Figure 4] It is a diagram for explaining the process of creating a sub-input feature map from an input feature map. [Figure 5] It is a diagram for explaining the process of obtaining a sub-convolution result from a sub-input feature map and a reduced kernel. [Figure 6] It is a diagram for explaining the Winograd conversion process according to an embodiment. [Figure 7] It is a diagram for explaining the process of obtaining a convolution result for a two-dimensional convolutional layer from sub-convolution results. [Figure 8] This flowchart shows an example of the processing flow by the neural network processing device according to the embodiment. [Figure 9] This block diagram shows an example of the configuration of another neural network processing device according to the embodiment. [Figure 10] This is a diagram showing the conventional technology. [Figure 11] This is a diagram showing the conventional technology. [Figure 12] This is a diagram showing the conventional technology. [Figure 13] This is a diagram showing the conventional technology. [Modes for carrying out the invention]

[0027] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. Also, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from the actual ratios.

[0028] The neural network processing device according to this embodiment enables winograd transformation even with extended convolutional layers of the same kernel size, thereby expanding the scope of application of winograd transformation while reducing the amount of computation.

[0029] Figure 1 is a block diagram showing an example of the hardware configuration of the neural network processing unit 10 according to this embodiment.

[0030] As shown in Figure 1, the neural network processing unit 10 includes a CNN arithmetic circuit 100 and a memory 200. The memory 200 stores kernels and the like. The CNN arithmetic circuit 100 performs CNN processing such as convolution and activation function processing using kernels and the like transferred from the memory 200. The CNN arithmetic circuit 100 according to this embodiment is configured as hardware designed specifically for CNNs. The CNN arithmetic circuit 100 is configured as a dedicated electrical circuit, such as a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacturing, such as an FPGA (Field-Programmable Gate Array), or a processor having a circuit configuration specifically designed to execute a particular process, such as an ASIC (Application Specific Integrated Circuit).

[0031] Next, with reference to Figure 2, the specific configuration of the neural network processing unit 10 will be described.

[0032] Figure 2 is a block diagram showing an example of the configuration of the neural network processing unit 10 according to this embodiment.

[0033] As shown in Figure 2, the neural network processing unit 10 comprises a CNN processing circuit 100. The CNN processing circuit 100 includes a circuit for performing convolution operations for winograd transformation. Specifically, the CNN processing circuit 100 comprises a reduced kernel creation unit 101A, a sub-input feature map creation unit 101B, a winograd transformation processing unit 102, and a sub-convolution result synthesis unit 106. The winograd transformation processing unit 102 comprises a first matrix transformation unit 103A, a second matrix transformation unit 103B, a multiplication processing unit 104, and a third matrix transformation unit 105. Each of these components is implemented, for example, by an FPGA or ASIC designed specifically for CNNs.

[0034] The kernel reduction unit 101A creates a reduced kernel by removing the zeros from the input kernel according to the dilation rate value. Specifically, the kernel reduction unit 101A creates a reduced kernel smaller than the input kernel by removing zeros inserted at positions corresponding to the dilation rate value. For example, if the dilation rate is 2, the size of the input kernel is reduced from 5x5 to 3x3. Here, the dilation rate is a value specified in the winograd transformation process of the convolutional neural network, and as mentioned above, it represents the interval between discrete values ​​in the input kernel, that is, the interval between effective kernels in the input kernel. For example, as shown in Figure 13 above, if the dilation rate is 2, effective kernels are placed one space apart vertically and horizontally, and if the dilation rate is 4, effective kernels are placed two spaces apart vertically and horizontally. The dilation rate is a value predetermined for the current layer and is obtained, for example, from memory 200. The input kernel represents the kernel which is the weight coefficient obtained from the previous layer, etc., and is obtained, for example, from memory 200.

[0035] The sub-input feature map creation unit 101B creates multiple sub-input feature maps from the input feature map according to the value of the expansion ratio for a two-dimensional convolutional layer, which is an expanded convolutional layer with an expansion ratio of 2 or more. Specifically, the sub-input feature map creation unit 101B determines, for example, multiple separation units, which are units for separating each pixel of the input feature map, according to the value of the expansion ratio. For each of the multiple separation units, it selects pixels at the same position and combines the selected pixels to create a sub-input feature map. For example, if the value of the expansion ratio is 2, the input kernel has one pixel enabled for every two pixels in the vertical and horizontal directions. However, in the case of the input feature map, it separates into units of four pixels (two pixels vertically and two pixels horizontally), and combines pixels at the same position among the four pixels to create a sub-input feature map. Since the number of sub-input feature maps created is equal to the square of the value of the expansion ratio, four are created in this case.

[0036] The winograd transformation processing unit 102 performs winograd transformation on each of the multiple sub-input feature maps and the reduction kernel, obtaining multiple sub-convolution results corresponding to each of the multiple sub-input feature maps. Furthermore, the winograd transformation processing unit 102 performs winograd transformation on two-dimensional convolutional layers that are not expansion convolutional layers with an expansion ratio of 1, obtaining convolution results corresponding to the input feature map. In other words, when the expansion ratio is 1, the input feature map is not separated into multiple sub-input feature maps, and normal convolution processing is performed.

[0037] Specifically, the winograd transformation processing unit 102 includes a first matrix transformation unit 103A, a second matrix transformation unit 103B, a multiplication processing unit 104, and a third matrix transformation unit 105.

[0038] The first matrix transformation unit 103A performs a matrix transformation on the reduced kernel using, for example, the transformation matrix B shown in equation (2) above. The second matrix transformation unit 103B performs a matrix transformation on each of the multiple sub-input feature maps using, for example, the transformation matrix C shown in equation (2) above. The multiplication processing unit 104 derives the Hadamard product of the reduced kernel transformed by the first matrix transformation unit 103A and each of the multiple sub-input feature maps transformed by the second matrix transformation unit 103B. The third matrix transformation unit 105 performs a matrix transformation on the Hadamard product using, for example, the transformation matrix A shown in equation (2) above, to obtain multiple sub-convolution results. Note that transformation matrices A, B, and C can be the same matrix even if the expansion ratio is different, as long as the kernel size is constant.

[0039] The sub-convolution result synthesis unit 106 synthesizes multiple sub-convolution results obtained by winograd transformation processing by the winograd transformation processing unit 102 to obtain a convolution result for the two-dimensional convolution layer.

[0040] The following example will provide a concrete explanation using the case where the input feature map size is 8x8, the expansion ratio is 2, the input kernel size is 5x5 (the reduced kernel size is 3x3), and the convolution result is 4x4.

[0041] Figure 3 is a diagram illustrating the convolution process according to this embodiment.

[0042] As shown in Figure 3, the convolution result C1 is obtained by augmented convolution of the input feature map M1 and the input kernel K1. The pixels 1 to 16 of the input feature map M1 required to obtain each pixel A to D of the convolution result C1 are as follows: Pixel A requires pixels 1, 2, 3, 5, 6, 7, 9, 10, and 11; pixel B requires pixels 2, 3, 4, 6, 7, 8, 10, 11, and 12; pixel C requires pixels 5, 6, 7, 9, 10, 11, 13, 14, and 15; and pixel D requires pixels 6, 7, 8, 10, 11, 12, 14, 15, and 16.

[0043] Figure 4 illustrates the process of creating sub-input feature maps SM1 to SM4 from the input feature map M1. This process is performed by the sub-input feature map creation unit 101B.

[0044] As shown in Figure 4, a 2x2 pixel in the input feature map M1 is treated as one separation unit. Each pixel in the input feature map M1 is separated into multiple separation units (16 in this case), and a number (1 to 16 in this case) is assigned to each pixel in the multiple separation units. Then, pixels at the same position are selected for each of the multiple separation units, and the selected pixels are grouped together to create sub-input feature maps SM1 to SM4. Each of the sub-input feature maps SM1 to SM4 is 4x4 in size. In other words, for the separation units (2x2 pixels) of the input feature map M1, pixels at the same position in the top left, top right, bottom left, and bottom right are selected, and the selected pixels are grouped together to create sub-input feature maps SM1 to SM4.

[0045] When sub-input feature maps SM1 to SM4 are created in this way, each of the sub-input feature maps SM1 to SM4 and the input kernel K1 are at similarly discrete positions. Therefore, the convolution operation between each of the sub-input feature maps SM1 to SM4 and the input kernel K1 is essentially the same as a normal convolution operation without dilation. This normal convolution operation yields the sub-convolution result.

[0046] Figure 5 illustrates the process of obtaining sub-convolution results SC1 to SC4 from sub-input feature maps SM1 to SM4 and reduced kernel SK1. This process is performed by the winograd transformation processing unit 102.

[0047] As described above, if each kernel of the 5x5 input kernel K1 has 9 valid kernels and 16 invalid kernels (=0), then, as shown in Figure 5, the invalid kernels can be skipped and only the 9 valid kernels can be combined into a 3x3 normal kernel (=reduced kernel SK1). In other words, by applying the 3x3 normal reduced kernel SK1 to each of the 4x4 sub-input feature maps SM1 to SM4 and performing a normal convolution operation, the same result as the original dilated convolution operation can be obtained. The pixels 1 to 16 of the input feature map M1 required to obtain pixels A to D of the sub-convolution results SC1 to SC4 are as follows: Pixel A requires pixels 1, 2, 3, 5, 6, 7, 9, 10, and 11, and pixel B requires pixels 2, 3, 4, 6, 7, 8, 10, 11, and 12. Pixel C requires pixels 5, 6, 7, 9, 10, 11, 13, 14, and 15, while pixel D requires pixels 6, 7, 8, 10, 11, 12, 14, 15, and 16.

[0048] Figure 6 is a diagram illustrating the winograd conversion process according to this embodiment.

[0049] As shown in Figure 6, the reduced kernel SK1, which is convolved into the sub-input feature maps SM1 to SM4, is a 3x3 normal kernel, not a 5x5 kernel, so the winograd transformation can be applied directly. In other words, as mentioned above, the amount of computation can be reduced by performing a 3x3 winograd transformation instead of a 5x5 winograd transformation. The winograd transformation process according to this embodiment follows the same procedure as the normal 3x3 convolution process. The winograd transformation process shown in Figure 6 is performed using equations (1) and (2) above to obtain sub-convolution results SC1 to SC4 from the sub-input feature maps SM1 to SM4 and the reduced kernel SK1.

[0050] Figure 7 illustrates the process of obtaining the convolution result C1 for a two-dimensional convolutional layer from the sub-convolution results SC1 to SC4. This process is performed by the sub-convolution result synthesis unit 106.

[0051] As shown in Figure 7, the convolution result C1 is obtained by performing the reverse process on the multiple sub-convolution results SC1 to SC4 that were ultimately created, as was done when the sub-input feature maps SM1 to SM4 were created. Depending on the dilation rate, each pixel of the sub-convolution results SC1 to SC4 is nested alternately to create a single convolution result C1 from the multiple sub-convolution results SC1 to SC4. The convolution result C1 obtained from these processes is the same as the convolution result that would be obtained if it were created by processing it as a dilated convolution layer without separation.

[0052] Next, with reference to Figure 8, the operation of the neural network processing device 10 according to this embodiment will be described.

[0053] Figure 8 is a flowchart illustrating an example of the processing flow by the neural network processing unit 10 according to this embodiment. The processing by the neural network processing unit 10 is realized by the CNN calculation circuit 100 of the neural network processing unit 10.

[0054] In step S101 of Figure 8, the CNN arithmetic circuit 100 acquires, for example, the input feature map M1 and input kernel K1 shown in Figure 3 above.

[0055] In step S102, the CNN arithmetic circuit 100 determines whether the specified expansion ratio value for the current layer is 2 or greater. If it determines that the expansion ratio value is 2 or greater (positive determination), it proceeds to step S103. If it determines that the expansion ratio value is less than 2, i.e., 1 (negative determination), it proceeds to step S107.

[0056] In step S103, the CNN arithmetic circuit 100 creates multiple sub-input feature maps SM1 to SM4 from the input feature map M1, as shown in Figure 4 above, as an example. Specifically, it determines multiple separation units, which are units for separating each pixel of the input feature map M1, according to the value of the expansion ratio, selects pixels at the same position for each of the multiple separation units, and combines the selected pixels to create the sub-input feature maps SM1 to SM4.

[0057] In step S104, the CNN arithmetic circuit 100 creates a reduced kernel SK1 from the input kernel K1 according to the value of the expansion ratio, as shown in Figure 5 above, for example. Note that the order of processing in steps S103 and S104 may be reversed.

[0058] In step S105, the CNN arithmetic circuit 100 performs a winograd transform on each of the sub-input feature maps SM1 to SM4 created in step S103 and the reduced kernel SK1 created in step S104, as shown in Figure 6 above, for example, to obtain multiple sub-convolution results SC1 to SC4.

[0059] In step S106, the CNN processing circuit 100 synthesizes the multiple sub-convolution results SC1 to SC4 obtained in step S105, as shown in Figure 7 above, to obtain a single convolution result C1. This completes the series of processes performed by the neural network processing unit 10.

[0060] Meanwhile, in step S107, the CNN processing circuit 100 performs winograd transformation on the input feature map M1 and the input kernel K1 to obtain a single convolution result. In other words, when the expansion ratio is 1, the input kernel K1 becomes a normal 3x3 kernel, so the input feature map M1 is not separated into multiple sub-input feature maps, and the normal convolution process is performed. This completes the series of processes by the neural network processing unit 10.

[0061] Furthermore, even when the expansion ratio is not 2, i.e., 3 or higher, by creating sub-input feature maps by selectively choosing pixels from the input feature map according to the expansion ratio, the winograd transformation can be applied to each sub-input feature map. Additionally, if a winograd transformation circuit for a standard 3x3 convolutional layer is provided within the device, the winograd transformation can be applied to 3x3 kernels of any expansion ratio using the method described above, thereby expanding the scope of winograd transformation application while reducing the computational load caused by the application of winograd transformation.

[0062] Figure 9 is a block diagram showing an example of the configuration of another neural network processing unit 10A according to this embodiment.

[0063] The neural network processing unit 10A shown in Figure 9 is a circuit that supports winograd transformation processing for multiple kernel sizes. In this case, the sub-input feature map creation unit 101B separates the input feature map into multiple sub-input feature maps according to the expansion ratio value, regardless of the kernel size. The sub-convolution result synthesis unit 106 synthesizes multiple sub-convolution results obtained from the multiple sub-input feature maps and the reduced kernel to obtain one convolution result. Multiple sets of transformation matrices A, B, and C corresponding to the kernel size are prepared, and the transformation matrix corresponding to the kernel size of the current layer is selected and provided to the first matrix transformation unit 103A, the second matrix transformation unit 103B, and the third matrix transformation unit 105, respectively.

[0064] Thus, according to this embodiment, the portion where the apparent kernel size becomes larger due to the extended convolutional layer can be treated as the actual kernel size by creating a sub-input feature map. Therefore, the effect of reducing the computational load by the winograd transform can be obtained in the same way as with a normal convolutional layer that is not an extended convolutional layer.

[0065] For example, even in the case of a 3x3 grid with an expansion factor of 2, it is possible to perform the winograd transformation as a 3x3 grid instead of a 5x5 grid, thereby reducing the computational load by 4 / 9 times.

[0066] Furthermore, the original winograd transform circuit can only be used in standard convolutional operations with a fixed kernel size. For example, if a winograd transform circuit for a 3x3 layer is provided within the AI ​​inference hardware, the circuit can only be used for standard 3x3 convolutional layers, and the winograd transform will not provide the speedup for other layers.

[0067] In contrast, according to this embodiment, as long as the kernel size matches, any extension convolutional layer with any extension ratio can be accelerated by using the winograd transformation circuit, thereby improving versatility.

[0068] As described above, by using a winograd transformation circuit designed for conventional convolutional layers to enable winograd transformation even for extended convolutional layers of the same kernel size, it is possible to expand the scope of winograd transformation while simultaneously reducing the computational load caused by applying winograd transformation.

[0069] In the above embodiment, the neural network processing performed by the CNN calculation circuit 100 may be realized by a processor such as a CPU (Central Processing Unit) executing a neural network processing program. For example, the neural network processing program may be pre-stored (also called "installed") in ROM (Read Only Memory) or storage. The neural network processing program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), and USB (Universal Serial Bus) memory. Alternatively, the neural network processing program may be downloaded from an external device via a network.

[0070] Furthermore, neural network processing may be performed on one of these various processors, or on a combination of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). More specifically, the hardware structure of these various processors is an electrical circuit made up of circuit elements such as semiconductor elements.

[0071] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0072] The following additional information is disclosed regarding the embodiments described above.

[0073] (Additional note 1) Memory and At least one processor connected to the memory, Includes, The aforementioned processor, For a two-dimensional convolutional layer, which is an extended convolutional layer with an extension ratio of 2 or more specified in the winograd transform process of a convolutional neural network, multiple sub-input feature maps are created from the input feature map according to the value of the extension ratio. Depending on the value of the expansion ratio, a reduced kernel is created by removing the portion that becomes zero from the input kernel. A winograd transform is performed on each of the multiple sub-input feature maps and the reduction kernel to obtain multiple sub-convolution results corresponding to each of the multiple sub-input feature maps. A neural network processing device configured to synthesize the aforementioned multiple sub-convolution results to obtain a convolution result for the two-dimensional convolution layer.

[0074] (Additional note 2) A non-temporary storage medium that stores a program executable by a computer to perform neural network processing, The aforementioned neural network processing is, For a two-dimensional convolutional layer, which is an extended convolutional layer with an extension ratio of 2 or more specified in the winograd transform process of a convolutional neural network, multiple sub-input feature maps are created from the input feature map according to the value of the extension ratio. Depending on the value of the expansion ratio, a reduced kernel is created by removing the portion that becomes zero from the input kernel. A winograd transform is performed on each of the multiple sub-input feature maps and the reduction kernel to obtain multiple sub-convolution results corresponding to each of the multiple sub-input feature maps. The above-mentioned sub-convolution results are combined to obtain the convolution result for the two-dimensional convolution layer. Non-transitory storage medium. [Explanation of Symbols]

[0075] 10, 10A Neural Network Processing Unit 100 CNN computing circuits 101A Miniaturized Kernel Creation Section 101B Sub-input Feature Map Creation Unit 102 Winograd Conversion Processing Unit 103A First Matrix Transformation Section 103B Second Matrix Transformation Section 104 Multiplication Processing Unit 105 Third Matrix Transformation Section 106 Subconvolution Result Synthesis Unit 200 memory

Claims

1. For a two-dimensional convolutional layer which is an extended convolutional layer with an extension ratio of 2 or more specified in the winograd transformation process of a convolutional neural network, a sub-input feature map creation unit creates multiple sub-input feature maps from the input feature map according to the value of the extension ratio, A reduced kernel creation unit creates a reduced kernel by removing the portion that becomes zero from the input kernel according to the value of the expansion ratio, A winograd transformation processing unit performs winograd transformation on each of the plurality of sub-input feature maps and the reduction kernel to obtain a plurality of sub-convolution results corresponding to each of the plurality of sub-input feature maps, A sub-convolution result synthesis unit that synthesizes the plurality of sub-convolution results to obtain a convolution result for the two-dimensional convolution layer, A neural network processing unit equipped with [a specific feature].

2. The winograd conversion processing unit performs winograd conversion on a two-dimensional convolutional layer that is not an extended convolutional layer with an expansion ratio of 1, and obtains a convolution result corresponding to the input feature map. The neural network processing apparatus according to claim 1.

3. The sub-input feature map creation unit determines a plurality of separation units, which are units for separating each pixel of the input feature map, according to the value of the expansion ratio, selects pixels at the same position for each of the plurality of separation units, and creates the sub-input feature map by combining the selected pixels. The reduced kernel creation unit creates a reduced kernel that is smaller in size than the input kernel, excluding zeros inserted at positions corresponding to the expansion ratio value. The neural network processing apparatus according to claim 1.

4. The aforementioned winograd conversion processing unit is A first matrix transformation unit performs a matrix transformation on the reduced kernel, A second matrix transformation unit performs a matrix transformation on each of the aforementioned plurality of sub-input feature maps, A multiplication processing unit that derives the Hadamard product of the reduced kernel, which has been matrix-transformed by the first matrix transformation unit, and each of the plurality of sub-input feature maps, which have been matrix-transformed by the second matrix transformation unit, A third matrix transformation unit performs a matrix transformation on the Hadamard product to obtain the plurality of sub-convolution results, including, The neural network processing apparatus according to claim 3.

5. The neural network processing unit For a two-dimensional convolutional layer, which is an extended convolutional layer with an extension ratio of 2 or more specified in the winograd transform process of a convolutional neural network, multiple sub-input feature maps are created from the input feature map according to the value of the extension ratio. Depending on the value of the expansion ratio, a reduced kernel is created by removing the portion that is zero from the input kernel. A winograd transform process is performed on each of the plurality of sub-input feature maps and the reduction kernel to obtain a plurality of sub-convolution results corresponding to each of the plurality of sub-input feature maps. The above-mentioned sub-convolution results are combined to obtain the convolution result for the two-dimensional convolution layer. Neural network processing methods.

6. For a two-dimensional convolutional layer, which is an extended convolutional layer with an extension ratio of 2 or more specified in the winograd transform process of a convolutional neural network, multiple sub-input feature maps are created from the input feature map according to the value of the extension ratio. Depending on the value of the expansion ratio, a reduced kernel is created by removing the portion that is zero from the input kernel. A winograd transform process is performed on each of the plurality of sub-input feature maps and the reduction kernel to obtain a plurality of sub-convolution results corresponding to each of the plurality of sub-input feature maps. The process of combining the above-mentioned sub-convolution results to obtain a convolution result for the two-dimensional convolution layer is as follows: A neural network processing program designed to be executed by a computer.

Images