Neural network processing device, neural network processing method, and neural network processing program

Abstract

This neural network processing device including a convolutional neural network using the Winograd transform comprises: a first transform unit that obtains a first matrix by matrix-transforming a kernel for each input channel; a second transform unit that obtains a second matrix by matrix-transforming an input feature map for each of the input channels; a multiplication unit that calculates a Hadamard product between the first matrix and the second matrix for each of the input channels; a holding unit that holds at least one of occurrence rates of clipping generated in at least one of a step for acquiring the first matrix and a step for acquiring the second matrix; and a switching unit that switches, on the basis of the holding result from the holding unit, whether to bypass the Winograd transform.

Description

Neural network processing device, neural network processing method, and neural network processing program

[0001] The disclosed technology relates to a neural network processing device, a neural network processing method, and a neural network processing program.

[0002] In a convolutional neural network (CNN), the network model is composed of multiple layers, and convolution processing is performed in the convolutional layer. The convolution processing uses an input feature map (also called input fmap) from a previous layer, etc., and a kernel, which is a weighting coefficient, as input. A convolution operation is performed for one input channel (ich) by repeatedly performing a product-sum operation on the input feature map and kernel in a two-dimensional plane, and the operation results are cumulatively added across all input channels to generate the convolution operation result for one output channel. A bias value is added to the result, and then activation function processing is performed. This results in an output feature map for one output channel. By performing these processes on all output channels, an output feature map (also called output fmap) for one convolutional layer is generated. The created output feature map becomes the input feature map for the next layer (see Figure 1).

[0003] When CNN inference processing is implemented using dedicated hardware such as an LSI (Large Scale Integration) or an FPGA (Field Programmable Gate Array), the convolution operation accounts for the majority of the computational load in the inference processing. Therefore, the computation circuit of the convolution operation unit accounts for the majority of the computing unit of the CNN inference hardware, and the computation circuit of the convolution operation unit determines the circuit area and power consumption of the entire dedicated CNN hardware. Whether the computation circuit of the convolution operation unit can be effectively utilized determines the throughput (inference processing performance) per circuit area and power consumption.

[0004] To reduce the size of the convolution calculation circuit, the bit width of the input data for the convolution calculation is often reduced. For example, when convolution calculations are processed in software using a general-purpose CPU, they are often calculated using 32-bit floating-point numbers, but hardware dedicated to CNN inference often uses 8-bit fixed-point numbers. In this case, the input feature map, kernel, and bias value are 8 bits, and the product-sum calculation for the convolution calculation is repeated using 8 bits * 8 bits = 16 bits. During the cumulative addition in the ith direction, for example, 16 bits are maintained while the cumulative addition is performed sequentially. When the cumulative addition is finally completed for all ith channels, the 16 bits are reduced to 8 bits using some step width (decimal point position) (or reduced to 8 bits after bias addition). After that, activation function processing is performed to obtain an 8-bit output feature map (see Figure 2).

[0005] In this case, the step width (decimal point position) per bit of the output feature map is determined when converting 16 bits to 8 bits when creating the convolution calculation result.

[0006] The lower bits are truncated (or rounded off and then truncated). For example, if the step width for 8 bits is 2 times the step width for 16 bits, ５ =) If you decide to multiply by 32, the lower 5 bits will be truncated when converting from 16 bits to 8 bits (the decimal point position will move 5 places to the left in binary).

[0007] If the value exceeds the limit, it will be clipped. For example, if the value after truncating the lowest 5 bits when converting from 16 bits to 8 bits is 130 in int, 130 will be clipped to 127 because an 8-bit int can range from -128 to 127 (see the top right of Figure 3).

[0008] When adjusting this step size, if the step size is too large, the calculation will be performed with low accuracy overall, resulting in a decrease in inference accuracy.

[0009] If the step width is too small, the value will exceed the maximum value and be clipped more frequently, resulting in localized convolution calculation results that are completely different from the actual value, reducing inference accuracy.

[0010] Furthermore, if the step width is too small, even if the value does not exceed the limit when the convolution calculation result is created, the value will often exceed the limit and be clipped when cumulatively adding the values ​​in the input channel direction of the next layer.

[0011] When accumulating in the input channel direction, even if the result of the convolution operation in 1 inch fits within a specified bit, the value becomes larger as it accumulates in the channel direction, and it will no longer fit. For example, in the case of multiplication of 8 bits, it fits within 16 bits, but if the number of input channels is 1024, it will be +10 bits (= log ２ A total of 1024 channels are required, and a 26-bit temporary storage memory is required in the hardware. Since it is not realistic to have memory capable of holding intermediate results of cumulative addition for all input channels due to the scale of the hardware, intermediate results are held at a fixed number of bits (16 bits in the example of Figure 4). In this case, the value may exceed the limit during cumulative addition in the i-ch direction, and in such cases clipping is required, just as when creating the convolution calculation result.

[0012] Since the output feature map of a layer is the input feature map of the next layer, the step width of the output feature map is directly linked to the frequency of clipping during the accumulation of the next layer (see the long circle in Figure 4).

[0013] From the above, if the step width is too large, calculations cannot be performed with high precision, and if the step width is too small, values ​​are likely to exceed the limit when creating the convolution calculation result in the current layer and when cumulatively adding in the channel direction in the next layer. Therefore, taking these factors into consideration, an appropriate step width is selected that is as small as possible and that does not cause too much excess when creating the convolution calculation result and when cumulatively adding in the channel direction in the next layer.

[0014] On the other hand, as a method for reducing the amount of calculation in convolution calculations from another perspective, the winograd transform is known, which can reduce the number of multiplications in convolution calculations (see, for example, Non-Patent Document 1).

[0015] As shown in FIG. 5 , the Winograd transform performs a matrix transformation on each of the input feature map and the input kernel beforehand, multiplies the transformed input feature map by the transformed input kernel with a Hadamard product, and then performs a post-transformation matrix transformation. This allows for the same results as a convolution operation. Since the number of multiplications required for the Hadamard product is less than the number of multiplications required for the original convolution operation, the overall amount of calculation can be reduced. In the case of inference hardware, fewer multiplication circuits need to be implemented, improving throughput per circuit area or power.

[0016] 5 shows the process of Winograd transform of one input channel among one output channel. The convolution operation (product-sum operation) of one input channel in FIG. 4 can be replaced with FIG.

[0017] The transformation formula for the winograd transformation shown in FIG. 5 is expressed by the following formula (1).

[0018] where I is the input feature map, I' is the transformed input feature map, K is the input kernel, K' is the transformed input kernel, O' is the Hadamard product result, O is the convolution result, A, B, and C are transformation matrices, and T is the transposed matrix.

[0019] (1)

[0020] For example, FIG. 6 shows a case where the size of the input feature map is 4×4, the size of the input kernel is 3×3, and the size of the convolution result is 2×2.

[0021] In the case of the size configuration shown in FIG. 6, the transformation matrices A, B, and C used for the winograd transformation are determined as shown in the following equation (2).

[0022] (2)

[0023] The size of the transformed input feature map is 4x4, and the size of the transformed input kernel is 3x3. These Hadamard products require 16 multiplications. The transformation matrix A after the Hadamard product multiplication turns the 4x4 Hadamard product result O' into a 2x2 convolution result O.

[0024] While a normal convolution operation requires (3 × 3) × (2 × 2) = 36 multiplications, using the Winograd transform reduces the number to 16, a 4 / 9 multiplication factor. Similarly, for a 5 × 5 input kernel, if the size of the convolution result is 2 × 2, the number of multiplications required is 9 / 25.

[0025] If the same number of multiplication circuits are used, it can be said that the calculation speed when Winograd is applied is faster than the calculation speed when Winograd is not applied. For example, in the case of the above 3 × 3 Winograd transformation, the calculation speed when Winograd is applied is 9 / 4 times the calculation speed when Winograd is not applied.

[0026] By combining these techniques, it is possible to reduce the number of calculation bits for the convolution operation to 8-bit fixed point, etc., and further reduce the number of multiplications for the convolution operation by using Winograd transform, thereby significantly reducing the overall amount of calculation required for the convolution operation.

[0027] Andrew Lavin, Scott Gray: Fast Algorithms for Convolutional Neural Networks, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Pages: 4013-4021.

[0028] On the other hand, the number of bits between the kernel and the input feature map increases due to the first and second matrix transformations in the winograd transformation.

[0029] For example, in the example of Figure 7, the kernel after matrix transformation requires an additional +4 bits compared to before matrix transformation, and the input feature map after matrix transformation requires an additional +2 bits compared to before matrix transformation. As a result, the two inputs to the multiplication processor in Winograd have a larger number of bits than the two inputs in the multiplication process in convolution processing. Although Winograd can reduce the number of multiplications compared to convolution operations, if the number of bits of the multiplication input increases even though the number of multiplications is reduced, the overall effect of reducing the circuit size / area of ​​the multiplication circuit is significantly reduced (Figure 7).

[0030] For this reason, it is possible to reduce the number of bits after the first and second matrix transformations, and clip the value if it exceeds the limit, thereby suppressing the number of bits of the multiplication input and maintaining the effect of reducing the size of the multiplication circuit (see the long circle in Figure 8). Figure 8 shows an example where both are reduced to 8 bits.

[0031] However, the step width determined conventionally takes into consideration only two factors: when generating the convolution calculation result and when accumulating, and is determined as a value appropriate for normal convolution calculation processing, not Winograd transformation.

[0032] In particular, the input feature map undergoes a second matrix transformation with the step size determined by the convolution result of the previous layer.

[0033] For this reason, if the step size in generating the convolution calculation results in the previous layer is set too large, it directly leads to an increased frequency of clipping in the WinoGrad matrix transformation in the next layer.Even if the step size is appropriate and no clipping occurs when the WinoGrad transformation is not used, when the WinoGrad transformation is used, the value may exceed the limit during matrix transformation, resulting in frequent clipping (see the long circle in Figure 9).

[0034] In this way, when bit reduction and Winograd transformation are used in combination to reduce the calculation scale of AI inference hardware, the inference accuracy is likely to decrease compared to when Winograd transformation is not used due to the process of reducing the number of bits in the matrix transformation section of the Winograd transformation.

[0035] The disclosed technology has been made in consideration of the above points, and aims to provide a neural network processing device, a neural network processing method, and a neural network processing program that are capable of determining whether or not to use Winograd transformation on a tile-by-tile basis to suppress a decrease in inference accuracy when applying Winograd transformation processing to a convolutional neural network.

[0036] A first aspect of the present disclosure is a neural network processing device including a convolutional neural network using a Winograd transform, comprising: a first transformation unit that performs matrix transformation on a kernel for each input channel to obtain a first matrix; a second transformation unit that performs matrix transformation on an input feature map for each input channel to obtain a second matrix; a multiplication unit that calculates a Hadamard product of the first matrix and the second matrix for each input channel; a storage unit that stores at least one of clipping incidence rates generated in at least one of the process of obtaining the first matrix and the process of obtaining the second matrix; and a switching unit that switches whether to bypass the Winograd transform based on the result stored by the storage unit.

[0037] A second aspect of the present disclosure is a neural network processing method, comprising: a neural network processing device including a convolutional neural network using a Winograd transform;

[0038] A kernel is matrix-transformed for each input channel to obtain a first matrix, an input feature map is matrix-transformed for each input channel to obtain a second matrix, a Hadamard product of the first matrix and the second matrix is ​​calculated for each input channel, at least one of a clipping incidence rate occurring in at least one of the steps of obtaining the first matrix and obtaining the second matrix is ​​retained, and whether or not to bypass the Winograd transform is switched based on the retained result of the clipping incidence rate.

[0039] A third aspect of the present disclosure is a neural network processing program for causing a computer to function as the neural network processing device of the first aspect.

[0040] According to the above aspects, the neural network processing device, neural network processing method, and neural network processing program of the present disclosure can determine whether or not to use the Winograd transform on a tile-by-tile basis in order to suppress a decrease in inference accuracy when applying the Winograd transform processing to a convolutional neural network.

[0041] 1 is a diagram illustrating a method for obtaining an output feature map using a convolutional neural network. It is a diagram illustrating the number of bits in a method for obtaining an output feature map using a convolutional neural network. It is a diagram illustrating clipping processing. It is a diagram illustrating clipping processing. It is a diagram illustrating Winograd transform. It is a diagram illustrating convolution processing. It is a diagram illustrating the number of bits in Winograd transform. It is a diagram illustrating clipping processing. It is a diagram illustrating clipping processing. It is a block diagram showing an example of the hardware configuration of a neural network processing device. It is a block diagram showing an example of the functional configuration of a CNN arithmetic circuit of a neural network processing device. It is a diagram illustrating functions of a neural network processing device according to the first embodiment. It is a flowchart showing the flow of bypass switching processing. It is a flowchart showing the flow of arithmetic processing of a convolutional layer. It is a diagram showing an example of a tile. It is a diagram illustrating a method for determining a motion vector. It is a diagram illustrating a method for switching bypass of Winograd transform. It is a diagram illustrating a method for switching bypass of Winograd transform. It is a diagram illustrating a method for switching bypass of Winograd transform. It is a diagram illustrating functions of a neural network processing device according to a second embodiment.

[0042] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0043] [First Embodiment] [Configuration of a Neural Network Processing Device According to a First Embodiment] A neural network processing device according to this embodiment determines an appropriate step size that can reduce the amount of calculation while suppressing a decrease in inference accuracy when applying the Winograd transformation to a convolutional neural network, thereby suppressing a decrease in inference accuracy. The neural network processing device 10 according to this embodiment will be described below.

[0044] First, the hardware configuration of a neural network processing device 10 according to this embodiment will be described with reference to Fig. 10. Fig. 10 is a block diagram showing the hardware configuration of the neural network processing device 10. As shown in Fig. 10, the neural network processing device 10 includes a CNN arithmetic circuit 50, a CPU 91, a ROM (Read Only Memory) 92, a RAM (Random Access Memory) 93, a storage 94, an input unit 95, a display unit 96, and a communication I / F (Interface) 97. Each component is connected to each other via a bus 99 so as to be able to communicate with each other.

[0045] The CPU 91 is a central processing unit that executes various programs and controls each part. That is, the CPU 91 reads programs from the ROM 92 or the storage 94 and executes the programs using the RAM 93 as a work area. The CPU 91 controls each of the above components and performs various arithmetic processing in accordance with the programs stored in the ROM 92 or the storage 94.

[0046] The ROM 92 stores various data and programs. The RAM 93 temporarily stores programs or data as a working area. The storage 94 is configured with a storage device such as a hard disk drive (HDD) or a solid state drive (SSD), and stores various programs including an operating system and various data. The ROM 92 or the storage 94 stores input / output data used in inference processing by the CNN, such as an input feature map, an output feature map, a kernel, and a bias value (details will be described later).

[0047] The input unit 95 includes, for example, a pointing device such as a mouse, a keyboard, etc., and is used to input various types of information. The display unit 96 is, for example, a liquid crystal display, and displays various types of information. The display unit 96 may also function as the input unit 95 by employing a touch panel system.

[0048] The communication I / F 97 is an interface for communicating with other devices, for example, using a wired communication standard such as Ethernet (registered trademark) or FDDI (Fiber Distributed Data Interface), or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark).

[0049] The CNN arithmetic circuit 50 performs CNN processing such as convolution processing and activation function processing using an input feature map, kernel, bias value, etc. transferred from the ROM 92 or the storage 94. The CNN arithmetic circuit 50 according to this embodiment is configured as hardware designed specifically for CNN. The CNN arithmetic circuit 50 is configured as a dedicated electrical circuit that is a processor having a circuit configuration designed specifically for executing specific processing, such as a programmable logic device (PLD) whose circuit configuration can be changed after manufacturing, such as an FPGA, or an application specific integrated circuit (ASIC).

[0050] Next, the functional configuration of the CNN arithmetic circuit 50 will be described with reference to Fig. 11. Fig. 11 is a block diagram showing an example of the functional configuration of the CNN arithmetic circuit 50.

[0051] As shown in FIG. 11 , the CNN arithmetic circuit 50 includes a circuit for performing convolution operations including the Winograd transform, and performs neural network processing on each tile for each layer of one frame. Specifically, the CNN arithmetic circuit 50 includes a convolution unit 40 and an activation function processing unit 51 as functional components. The convolution unit 40 includes a Winograd transform unit 30, an adder unit 41, and a result generation unit 42. The Winograd transform unit 30 includes a first transform unit 31, a second transform unit 32, a multiplier unit 33, and a third transform unit 34. Each of these components is realized, for example, by an FPGA or ASIC designed specifically for CNN.

[0052] The convolution unit 40 is configured to be able to switch, for each convolution layer, whether to perform a convolution operation with or without applying the Winograd transform. Whether to perform the Winograd transform is determined by receiving an instruction from, for example, the CPU 91.

[0053] [Convolution Operation Applying Winograd Transform] First, the convolution operation when the winograd transform is applied will be described.

[0054] The first conversion unit 31 performs matrix transformation on the kernel for each input channel to obtain a first matrix. Specifically, the first conversion unit 31 acquires the kernel stored in the ROM 92 or the like, and performs matrix transformation on the kernel using the transformation matrix B shown in the above equation (2). At this time, if the value of the obtained first matrix exceeds the upper limit of the representable bit width (overflows), the clipping processing unit 31A of the first conversion unit 31 performs clipping and saturation processing so that the upper limit is not exceeded.

[0055] For example, if the kernel is 8 bits and the transformation matrix B shown in the above formula (2) is used, 12 bits are required to accurately represent the first matrix. On the other hand, when the first matrix is ​​converted to 8 bits to reduce the amount of calculation, if the transformation result is −128 or less, it is clipped to −128, and if the transformation result is 127 or more, it is clipped to 127, so that the size of the first matrix fits within 8 bits.

[0056] The kernel is weight data and is determined in advance through learning. That is, before the inference process by the CNN begins, the kernel value, including the decimal point position (step size), is already determined. Therefore, the first conversion unit 31 does not need to determine the decimal point position (step size) or calculate the first matrix in real time during the inference process. For example, the first conversion unit 31 may perform matrix transformation on the kernel for each input channel in advance to obtain the first matrix and store the obtained first matrix in the ROM 92, storage 94, or the like. In this case, the amount of calculation required for the matrix transformation can be reduced, thereby improving the throughput (inference processing performance) per circuit area and power consumption of the CNN arithmetic circuit 50. On the other hand, since the data size of the kernel is smaller than the first matrix, it is preferable to perform matrix transformation on the kernel in real time during the inference process in order to reduce the amount of data transfer between the CNN arithmetic circuit 50 and an external memory, such as the ROM 92, in which the kernel or the first matrix is ​​stored. Therefore, whether to perform matrix transformation on the kernel in advance or perform matrix transformation in real time during the inference process may be selected as appropriate.

[0057] Alternatively, for example, the inference process by the CNN arithmetic circuit 50 may be performed several times in advance with the Winograd transform applied, and the decimal point position with the best inference accuracy may be set as the decimal point position of the kernel when the Winograd transform is applied. Alternatively, for example, a kernel (e.g., 8 bits) corresponding to a predetermined decimal point position may be created by an external computer or the like, and the kernel may be stored in advance in the ROM 92, storage 94, or the like. In this case, the first conversion unit 31 may read and use the kernel stored in the ROM 92, storage 94, or the like as appropriate when the inference process is started.

[0058] The second conversion unit 32 performs matrix transformation on the input feature map for each input channel to obtain a second matrix. Specifically, the second conversion unit 32 acquires the input feature map stored in the ROM 92, the storage 94, or the like, and performs matrix transformation on the input feature map using the transformation matrix C shown in the above equation (2). At this time, if the value of the obtained second matrix exceeds the upper limit of the representable bit width (overflows), the clipping processing unit 32A of the second conversion unit 32 performs clipping and saturation processing so that the upper limit is not exceeded.

[0059] For example, if the input feature map is 8 bits and the transformation matrix C shown in the above formula (2) is used, 10 bits are required to correctly represent the second matrix. On the other hand, when the second matrix is ​​converted to 8 bits to reduce the amount of calculation, if the transformation result is −128 or less, it is clipped to −128, and if the transformation result is 127 or more, it is clipped to 127, so that the size of the second matrix fits within 8 bits.

[0060] Note that since the input feature map is generated in real time during inference processing by the CNN based on input data such as images, it is difficult to determine the decimal point position and second matrix in advance, as with the kernel.

[0061] The multiplication unit 33 calculates the Hadamard product of the first matrix and the second matrix for each input channel. Specifically, the multiplication unit 33 acquires the first matrix obtained by the first conversion unit 31 and the second matrix obtained by the second conversion unit 32, and calculates the Hadamard product using the above formula (1).

[0062] The third transformation unit 34 performs matrix transformation on the Hadamard product for each input channel to obtain a third matrix. Specifically, the third transformation unit 34 obtains the Hadamard product calculated by the multiplication unit 33, and performs matrix transformation on the Hadamard product using the transformation matrix A shown in the above equation (2).

[0063] The adder 41 calculates the cumulative sum of the third matrix for each input channel. Specifically, the adder 41 obtains the third matrix for each input channel obtained by the third conversion unit 34 and performs cumulative addition for all input channels to obtain the cumulative sum. For example, the adder 41 stores the cumulative sum during the cumulative addition in the RAM 93 or the like, and when the adder 41 obtains the third matrix for another input channel, it reads the cumulative sum during the cumulative addition from the RAM 93 or the like, adds it, updates it, and stores the updated cumulative sum again in the RAM 93 or the like. Furthermore, when the adder 41 completes the cumulative addition for all channels, it reads the cumulative sum from the RAM 93 or the like and outputs it to the result creation unit 42.

[0064] At this time, if the cumulative sum during the cumulative addition exceeds the upper limit of the expressible bit width (overflows), the clipping processing unit 41A of the adding unit 41 performs clipping and saturation processing so that the upper limit is not exceeded. For example, if a 16-bit temporary storage area for cumulative addition is provided, if the cumulative sum during the cumulative addition is −32768 or less, it is clipped to −32768, and if the conversion result is 32767 or more, it is clipped to 32767, so that the size of the cumulative sum fits within 16 bits.

[0065] Each of the clipping processing units 31A, 32A, and 41A counts the number of clippings for each tile. Here, the two-dimensional plane of the frame is divided into several units, which are called tiles.

[0066] The clipping processing units 31A, 32A, and 41A count the total number of times clipping has occurred in one tile, and calculate the clipping occurrence rate as a ratio.

[0067] Specifically, the clipping processing unit 31A calculates the clipping occurrence rate by dividing the frequency of clipping that occurred when performing matrix transformation on the kernel for each input channel to obtain the first matrix by the number of input channels.

[0068] The clipping processing unit 32A calculates the clipping occurrence rate by dividing the frequency of clipping that occurred when performing matrix transformation on the target tile of the input feature map for each input channel to obtain the second matrix by the number of input channels.

[0069] The clipping processor 41A calculates the clipping occurrence rate by dividing the frequency of clipping that occurs when calculating the cumulative sum of the third matrix for each input channel for the target tile by the number of input channels. The clipping occurrence rate for each tile is stored in the storage 94 or the like, similar to the output feature map.

[0070] The result creation unit 42 receives the convolution operation results accumulated for all inputs, creates a final convolution operation result, and outputs it to the activation function processing unit 51. Specifically, the result creation unit 42 adds a bias value to the convolution operation results accumulated for all inputs.

[0071] The activation function processing unit 51 generates an output feature map by performing predetermined nonlinear processing on the final convolution operation result (i.e., the cumulative sum of the third matrix to which the bias value has been added) output from the result generation unit 42. The output feature map is stored in the ROM 92, RAM 93, storage 94, etc., and serves as an input feature map for the next layer (convolution layer, pooling layer, fully connected layer, etc.). As the nonlinear processing, processing using a known activation function can be applied as appropriate.

[0072] In this way, whether or not to use the Winograd transform can be switched for each tile in order to suppress a decrease in inference accuracy, depending on the clipping occurrence rate for each tile. This makes it possible to maintain the accuracy of the convolution operation and inference processing.

[0073] [Convolution Operation Without Applying Winograd Transform] Next, a convolution operation without applying Winograd transform will be described. When the clipping occurrence rate is high, it may be preferable to perform the convolution operation without applying Winograd transform.

[0074] Therefore, the CNN arithmetic circuit 50 according to this embodiment also has the function of performing a convolution operation without performing a Winograd transform. In Fig. 11, the dotted line indicates the data flow when a convolution operation is performed without performing a Winograd transform. As shown in Fig. 11, when a Winograd transform is not performed, the first conversion unit 31, the second conversion unit 32, and the third conversion unit 34 are not used. Some of the explanations overlapping with the case when a Winograd transform is performed will be omitted.

[0075] First, the switching unit 66 (described later) may determine not to perform Winograd transform in the next convolutional layer when the clipping incidence rate for each tile satisfies a predetermined condition. Hereinafter, the functions of each functional unit when the switching unit 66 determines not to perform Winograd transform in the previous convolutional layer will be described.

[0076] The multiplication unit 33 calculates the sum of products between the kernel and the input feature map for each input channel instead of the Hadamard product. The addition unit 41 calculates the cumulative sum of products between the kernel and the input feature map for each input channel calculated by the multiplication unit 33 instead of the cumulative sum of the third matrix. That is, the multiplication unit 33 and the addition unit 41 calculate the sum of products between the kernel and the input feature map and perform cumulative addition, similar to a general CNN.

[0077] The result generation unit 42 adds the bias value to the cumulative sum of the products of the kernel and the input feature map. Furthermore, the clipping unit 41A of the adder 41 also performs clipping if an overflow occurs.

[0078] The activation function processing unit 51 generates an output feature map by performing a predetermined nonlinear processing on the final convolution operation result (the cumulative sum of the products of the kernel to which the bias value has been added and the input feature map) output from the result generation unit 42.

[0079] As shown in FIG. 12, the neural network processing device 10 functionally comprises an acquisition unit 60, a processing unit 62, a holding unit 64, and a switching unit 66.

[0080] The acquisition unit 60 acquires the input target image to be processed.

[0081] The processing unit 62 processes the target image using a neural network including convolution processing using the Winograd algorithm. When performing the convolution processing, the processing unit 62 calculates the Hadamard product of the result of the kernel transformation processing based on the Winograd algorithm and the kernel transformation matrix, and uses the calculation result of the Hadamard product for multiplication to obtain the result of the convolution processing.

[0082] The processing using the neural network is executed by using the CNN calculation circuit 50. At this time, the target image and the kernel are input to the CNN calculation circuit 50, and the processing result is output from the CNN calculation circuit 50.

[0083] The storage unit 64 stores at least one of the clipping incidence rates generated in at least one of the process of obtaining the first matrix and the process of obtaining the second matrix. In this embodiment, the storage unit 64 stores, for each tile, the clipping incidence rates generated in the process of obtaining the first matrix by the first conversion unit 31, the process of obtaining the second matrix by the second conversion unit 32, and the process of calculating the cumulative sum of the third matrix by the adder 41.

[0084] The switching unit 66 switches, for each tile, whether or not to bypass the winograd transformation based on the result stored by the storage unit 64 .

[0085] Specifically, it is determined on a tile-by-tile basis whether clipping occurs frequently in the winograd matrix transformation, and for tiles where clipping is likely to occur frequently, the winograd transformation is switched to be bypassed, and if not, the winograd transformation is not bypassed.

[0086] The clipping incidence rate of the tile in the previous frame is used to determine whether to apply the Winograd transformation. For example, the average value of the clipping incidence rates calculated by the clipping processing units 31A, 32A, and 41A is used. This allows for the reduction in inference accuracy to be suppressed by performing a normal convolution operation for tiles where the inference accuracy is reduced when Winograd is used. On the other hand, for tiles where the inference accuracy is not likely to be reduced even when Winograd transformation is used, the Winograd processing can improve throughput and processing speed. Note that, although the above description has been given using an example in which the average value of the clipping incidence rates calculated by the clipping processing units 31A, 32A, and 41A is used, this is not limiting. For example, the minimum or maximum value of the clipping incidence rates calculated by the clipping processing units 31A, 32A, and 41A may also be used.

[0087] The result of processing the target image using the neural network is displayed on the display unit 96.

[0088] [Operation of the Neural Network Processing Device According to the First Embodiment] Next, the operation of the neural network processing device 10 according to this embodiment will be described.

[0089] First, a target image is input to the neural network processing device 10. The acquisition unit 60 acquires the input target image. Then, the processing unit 62 processes the target image using a pre-trained neural network, using the CNN arithmetic circuit 50. The results of processing the target image using the neural network are then displayed on the display unit 96.

[0090] An example of the bypass switching process by the switching unit 66 will now be described with reference to Fig. 13. The bypass switching process is repeatedly performed for each layer of the convolution process for the target image. The bypass switching process and the arithmetic process for each convolution layer are an example of a neural network processing method.

[0091] First, in step S50, the switching unit 66 determines whether to bypass the Winograd transformation for the target tile. For example, if the clipping incidence rate held for the target tile is equal to or greater than a threshold, the switching unit 66 determines to bypass the Winograd transformation for the target tile, and proceeds to step S52. On the other hand, if the clipping incidence rate held for the target tile is less than the threshold, the switching unit 66 determines not to bypass the Winograd transformation for the target tile, and proceeds to step S54. Note that, although the case where whether to bypass the Winograd transformation is determined depending on whether the clipping incidence rate (average, minimum, or maximum) held for the target tile is equal to or greater than a threshold has been described as an example, the present invention is not limited thereto. For example, a threshold may be specified for each of the clipping incidence rates calculated by the clipping processing units 31A, 32A, and 41A, and whether to bypass the Winograd transformation may be determined depending on the number of clipping incidence rates that exceed the threshold.

[0092] In step S52, the switching unit 66 instructs the CNN calculation circuit 50 to turn off the winograd transform for the target tile.

[0093] In step S54, the switching unit 66 instructs the CNN calculation circuit 50 to turn on the winograd transform for the target tile.

[0094] In step S56, the switching unit 66 determines whether steps S50 to S54 have been repeatedly executed for all tiles. If there is a tile for which steps S50 to S54 have not been repeatedly executed, the process returns to step S50, and step S50 is executed for that tile as the target tile.

[0095] On the other hand, if steps S50 to S54 have been repeatedly executed for all tiles, the bypass switching process for the layer is terminated.

[0096] The arithmetic processing of each convolution layer by the neural network processing device 10 is realized by the processing routine shown in Fig. 14. The processing routine of Fig. 14 is realized by the CNN arithmetic circuit 50 of the neural network processing device 10.

[0097] In step S10, the convolution unit 40 checks whether or not the Winograd transform is to be applied to the current convolution layer. If the determination in step S10 is affirmative, the process proceeds to step S12, where the first transformation unit 31 performs matrix transformation on the kernel to obtain a first matrix. In step S14, the second transformation unit 32 performs matrix transformation on the input feature map to obtain a second matrix. In step S16, the multiplication unit 33 calculates the Hadamard product of the first matrix obtained in step S12 and the second matrix obtained in step S14. In step S18, the third transformation unit 34 performs matrix transformation on the Hadamard product calculated in step S16 to obtain a third matrix.

[0098] On the other hand, if the determination in step S10 is negative, the process proceeds to step S20, where the multiplication unit 33 calculates the sum of products of the kernel and the input feature map. In step S22, the convolution unit 40 determines whether or not the matrix transformation (third matrix) of the Hadamard product in step S18 or the calculation of the sum of products of the kernel and the input feature map in step S20 has been completed for all input channels. If the processing for all input channels has not been completed, the process returns to step S10 and is repeated until the processing for all input channels has been completed.

[0099] On the other hand, if the result of step S22 is a positive judgment, in step S24, the adder 41 calculates the cumulative sum of the third matrix obtained in step S18 or the cumulative sum of the products of the kernel and the input feature map obtained in step S20.

[0100] In step S28, the result generation unit 42 adds the bias value to the cumulative sum of the third matrix obtained in step S24 or the cumulative sum of the products of the kernel and the input feature map obtained in step S20.

[0101] In step S30, the activation function processing unit 51 generates an output feature map by performing a predetermined nonlinear process on the final convolution operation result (the cumulative sum to which the bias value has been added) obtained in step S28. In step S32, the activation function processing unit 51 outputs the output feature map generated in step S30 to the next convolution layer, and then ends the processing routine.

[0102] As described above, the neural network processing device 10 of this embodiment stores at least one of the clipping incidence rates generated in at least one of the steps of performing matrix transformation on a kernel for each input channel to obtain a first matrix and the step of performing matrix transformation on an input feature map for each input channel to obtain a second matrix in a convolutional neural network using the Winograd transform, and switches between bypassing and bypassing the Winograd transform based on the stored clipping incidence rate. This makes it possible to determine whether to use the Winograd transform on a tile-by-tile basis in order to prevent a decrease in inference accuracy when applying the Winograd transform to a convolutional neural network.

[0103] According to the neural network processing device 10 of this embodiment, when the Winograd transformation is applied to a convolutional neural network, the amount of calculation can be reduced while suppressing a decrease in inference accuracy.

[0104] Second Embodiment Next, a neural network processing device according to a second embodiment will be described. Note that parts having the same configuration as those in the first embodiment will be given the same reference numerals and descriptions thereof will be omitted.

[0105] The second embodiment differs from the first embodiment in that the motion vector obtained by the motion search is used to switch between bypassing the winograd transform for each tile.

[0106] [Outline of the Second Embodiment] Motion estimation is performed once per frame, and motion between frames is determined based on the current frame and the immediately preceding frame.

[0107] As shown in Figure 15, the current frame is divided into several units on a two-dimensional plane. These units are called tiles. A motion vector is determined for each tile.

[0108] To determine a motion vector, for example, the sum of absolute differences (SAD) is used. Specifically, for each pixel in the tile and for each channel ch (R, G, B), the difference value between the target tile of the current frame and an arbitrary tile of the immediately preceding frame is calculated, and the absolute values ​​of these are summed to obtain the SAD. The tile with the smallest SAD is considered to be the most similar to the target tile of the current frame, and therefore this vector is determined to be the motion vector (see FIG. 16).

[0109] In addition, when determining whether to bypass the Winograd transform, the decision as to whether to bypass the Winograd transform for each tile of the current frame is made based on the tile-by-tile motion vector of the current frame and the clipping occurrence rate for each tile of the previous frame, and the result is output to the CNN calculation circuit.

[0110] For example, as shown in Figure 17, for a tile (tile A) to be processed in the current frame, attention is focused on a specific tile (tile B) in the immediately preceding frame, which is the destination of the tile-by-tile motion vector. The clipping incidence rate of tile B is measured during the inference process of the immediately preceding frame and stored in storage 94, etc. Because tile A and tile B are similar, it can be assumed that their clipping incidence rates also tend to be similar. Therefore, if the clipping incidence rate during matrix transformation in the Winograd transform of tile B is high, it is highly likely that the clipping incidence rate during Winograd matrix transformation of tile A will also be high, and the inference accuracy of tile A is highly likely to decrease due to clipping during matrix transformation in the Winograd transform.

[0111] Therefore, if the clipping occurrence rate of the matrix transform in the Winograd transform of tile B is equal to or greater than a threshold, the Winograd transform of tile A is bypassed to prevent a decrease in inference accuracy due to the Winograd transform of tile A. The CNN calculation circuit receives a command to turn off the Winograd transform and performs a normal convolution operation without using the Winograd transform. On the other hand, if the clipping occurrence rate of the matrix transform in the Winograd transform of tile B is less than the threshold, the Winograd transform of tile A is not bypassed.

[0112] By the above processing, it is determined on a tile-by-tile basis whether clipping occurs frequently in the matrix transformation of winograd, and if there is a high possibility that clipping will occur frequently, winograd is bypassed, and if not, winograd is not bypassed.

[0113] To determine whether to use the winograd transform, the clipping occurrence rate of the previous frame is used, and the one that is closest to the target tile through motion search is used.

[0114] As a result, for tiles where the inference accuracy decreases when the Winograd transform is used, the decrease in inference accuracy can be suppressed by performing a normal convolution operation, while for tiles where the inference accuracy is less likely to decrease even when the Winograd transform is used, the throughput and processing speed can be improved by performing the Winograd transform.

[0115] In addition, tiles ahead of the motion vector may have already bypassed WinoGrad transformation. In the example on the left side of Fig. 18, the top center and bottom right of the previous frame bypassed WinoGrad transformation (see the diagonally shaded tiles on the left side of Fig. 18). In the current frame, the top center, top right, and bottom right indicate tiles that bypassed WinoGrad transformation in the previous frame (see the dotted-line tiles on the left side of Fig. 18).

[0116] When the winograd transformation is bypassed, the matrix transformation of the winograd transformation is not performed, and therefore, data on the clipping occurrence rate does not exist.

[0117] Therefore, the data on the clipping occurrence rate of the immediately preceding frame cannot be referenced in the top center, top right, and bottom right of the current frame, and it is not possible to determine whether or not the winograd transformation should be bypassed in the current frame.

[0118] In such a case, the tiles can be determined as in (1) to (3) below, for example.

[0119] (1) Always avoid bypassing Winograd transformation. (2) Always bypass Winograd transformation. (3) Do not select tiles that bypassed Winograd transformation in the immediately preceding frame as the destination of the motion vector (see the right side of Figure 18). By selecting the tile with the smallest SAD with the target tile of the current frame from among the tiles that did not bypass Winograd transformation in the immediately preceding frame, it is possible to ensure that the clipping occurrence rate data for the immediately preceding frame cannot be referenced.

[0120] Furthermore, there may be a case where the number of tiles in a frame that do not bypass the winograd transformation is reduced, making it impossible to correctly determine whether or not the winograd transformation should be bypassed in the current frame.

[0121] In this case, it may be possible to avoid bypassing the winograd transform for all tiles in periodic frames.

[0122] For example, once every three frames, the Winograd transform is not bypassed for all tiles at all times. This prevents the number of tiles that do not bypass the Winograd transform from decreasing too much within a frame, and enables proper switching of the Winograd transform bypass to be determined.

[0123] This is particularly effective in the case of "(2) Always bypass the winograd conversion."

[0124] Furthermore, rather than bypassing the Winograd transform for all tiles in periodic frames, it is possible to prevent the Winograd transform from being bypassed for all tiles when the number of tiles bypassing the Winograd transform reaches a certain level.

[0125] For example, if the number of tiles bypassing the Winograd transformation is three or more and the Winograd transformation is not bypassed for all tiles, in the example of Figure 19, there are four tiles bypassing the Winograd transformation at the time of frame 2 (see the tiles with diagonal lines in Figure 19), so in frame 3 the Winograd transformation is changed so that it is not bypassed for all tiles.

[0126] [Configuration of Neural Network Processing Device According to Second Embodiment] As shown in FIG. 10, the hardware configuration of the neural network processing device 210 according to the second embodiment is the same as that of the first embodiment.

[0127] As shown in FIG. 20, the neural network processing device 210 according to the second embodiment functionally comprises an acquisition unit 60, a motion search unit 261, a processing unit 62, a holding unit 64, and a switching unit 266.

[0128] The acquisition unit 60 acquires a plurality of frames of input target images to be processed.

[0129] The motion search unit 261 determines a motion vector representing the motion between frames for each pair of a current frame and an immediately preceding frame for a target image of multiple frames.

[0130] The processing unit 62 processes the target image using a neural network including convolution processing using the Winograd algorithm. When performing the convolution processing, the processing unit 62 calculates the Hadamard product of the result of the kernel transformation processing based on the Winograd algorithm and the kernel transformation matrix, and uses the calculation result of the Hadamard product for multiplication to obtain the result of the convolution processing.

[0131] The processing using the neural network is executed by using the CNN calculation circuit 50. At this time, the target image and the kernel are input to the CNN calculation circuit 50, and the processing result is output from the CNN calculation circuit 50.

[0132] The storage unit 64 stores at least one of the clipping incidence rates occurring in at least one of the process of obtaining the first matrix and the process of obtaining the second matrix. In this embodiment, the storage unit 64 stores, for each tile, the clipping incidence rates occurring in the process of obtaining the first matrix by the first conversion unit 31 and the process of obtaining the second matrix by the second conversion unit 32.

[0133] The switching unit 266 switches, for each tile, whether or not to bypass the winograd transformation based on the motion vector for each tile of the current frame and the clipping incidence rate for each tile of the immediately preceding frame held by the holding unit 64 .

[0134] Specifically, for a tile to be processed in the current frame, if the clipping incidence rate of a specific tile in the immediately preceding frame pointed to by a tile-by-tile motion vector is equal to or greater than a threshold, the Winograd transform is switched to be bypassed; and if the clipping incidence rate of a specific tile in the immediately preceding frame is less than the threshold, the Winograd transform is not bypassed. In this way, the clipping incidence rate of the tile in the immediately preceding frame that is closest to the tile to be processed by motion estimation is used to determine whether to apply the Winograd transform. As a result, for tiles where the inference accuracy would decrease if Winograd were used, taking motion estimation into consideration, the decrease in inference accuracy can be suppressed by performing a normal convolution operation. On the other hand, for tiles where the inference accuracy is not likely to decrease even if Winograd transform is used, the throughput and processing speed can be improved by performing Winograd processing.

[0135] The other configurations and operations of the neural network processing device 210 according to the second embodiment are the same as those of the first embodiment, and therefore will not be described again.

[0136] In the above embodiment, the CPU 11 switches between bypassing and bypassing the Winograd transform for each tile as the switching unit 66, 266. However, the present invention is not limited to this. For example, the CNN arithmetic circuit 50 may switch between bypassing and bypassing the Winograd transform for each tile as the switching unit 66, 266.

[0137] The neural network processing performed by the CNN arithmetic circuit 50 in the above embodiment may be realized by a processor such as a CPU executing a neural network processing program. For example, the neural network processing program may be pre-stored (also referred to as "installed") in the ROM 92 or the storage 94. The neural network processing program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The neural network processing program may also be downloaded from an external device via a network.

[0138] Furthermore, neural network processing may be performed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., a plurality of FPGAs, or a combination of a CPU and an FPGA, etc.).More specifically, the hardware structure of these various processors is an electric circuit that combines circuit elements such as semiconductor elements.

[0139] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0140] The following additional notes are provided regarding the above-described embodiments.

[0141] (Supplementary Item 1) A neural network processing device including a convolutional neural network using a Winograd transform, comprising: a memory; and at least one processor connected to the memory, wherein the processor: performs matrix transformation on a kernel for each input channel to obtain a first matrix; performs matrix transformation on an input feature map for each input channel to obtain a second matrix; calculates a Hadamard product of the first matrix and the second matrix for each input channel; holds at least one of clipping incidence rates generated in at least one of the process of obtaining the first matrix and the process of obtaining the second matrix; and switches whether to bypass the Winograd transform based on the held result of the clipping incidence rate.

[0142] (Supplementary Item 2) A non-transitory storage medium storing a program executable by a computer to perform neural network processing including a convolutional neural network using a Winograd transform, wherein the neural network processing: performs matrix transformation on a kernel for each input channel to obtain a first matrix; performs matrix transformation on an input feature map for each input channel to obtain a second matrix; calculates a Hadamard product of the first matrix and the second matrix for each input channel; stores at least one of clipping incidence rates generated in at least one of the process of obtaining the first matrix and the process of obtaining the second matrix; and switches whether to bypass the Winograd transform based on the stored result of the clipping incidence rate.

[0143] 10, 210 Neural network processing device 30 Winograd conversion unit 31 First conversion unit 31A, 32A, 41A Clipping processing unit 32 Second conversion unit 33 Multiplication unit 34 Third conversion unit 40 Convolution unit 41 Addition unit 42 Result creation unit 50 CNN arithmetic circuit 51 Activation function processing unit 60 Acquisition unit 62 Processing unit 64 Holding unit 66, 266 Switching unit 91 CPU 94 Storage 95 Input unit 96 Display unit 210 Neural network processing device 261 Motion search unit

Claims

1. A neural network processing device including a convolutional neural network using a Winograd transform, comprising: a first transformation unit that performs matrix transformation on a kernel for each input channel to obtain a first matrix; a second transformation unit that performs matrix transformation on an input feature map for each input channel to obtain a second matrix; a multiplication unit that calculates a Hadamard product of the first matrix and the second matrix for each input channel; a storage unit that stores at least one of a clipping occurrence rate generated in at least one of the process of obtaining the first matrix and the process of obtaining the second matrix; and a switching unit that switches whether to bypass the Winograd transform based on a result stored by the storage unit.

2. The neural network processing device according to claim 1, further comprising: a third transformation unit that performs a matrix transformation on the Hadamard product for each of the input channels to obtain a third matrix; and an addition unit that calculates a cumulative sum of the third matrix for each of the input channels, wherein the holding unit further holds a clipping occurrence rate that occurs in the process of calculating the cumulative sum of the third matrix.

3. A neural network processing method in which a neural network processing device including a convolutional neural network using Winograd transform performs matrix transformation on a kernel for each input channel to obtain a first matrix, performs matrix transformation on an input feature map for each input channel to obtain a second matrix, calculates a Hadamard product of the first matrix and the second matrix for each input channel, retains at least one of the clipping occurrence rates generated in at least one of the process of obtaining the first matrix and the process of obtaining the second matrix, and switches whether or not to bypass the Winograd transform based on the retained result of the clipping occurrence rate.

4. A neural network processing program for causing a computer to function as the neural network processing device according to claim 1 or 2.

Images