Method for improving computation of convolutional neural networks

Abstract

The application discloses a method for improving calculation of a convolutional neural network (CNN). The CNN comprises a plurality of convolutional layers, each of which uses N convolutional kernels, N being an integer greater than 1, and the method comprises the following steps. First, according to the number of convolutional kernels used by all the convolutional layers, N multipliers and N adders are determined. Then, when the CNN runs to the i-th convolutional layer and the N convolutional kernels of the i-th convolutional layer are all of the size of Kx1x1, the N multipliers and the N adders are used to respectively calculate one multiplication operation and one addition operation of the N convolutional kernels of the i-th convolutional layer in each cycle, so that N output results of the N convolutional kernels of the i-th convolutional layer are obtained after K cycles, i is an integer greater than or equal to 1, and K is an integer greater than 1.

Description

Technical Field

[0001] This invention relates to Convolutional Neural Networks (CNNs), and particularly to an improved method for performing computations in CNNs. Background Technology

[0002] CNNs excel in speech recognition, and Mel-Frequency Cepstral Coefficients (MFCCs) are widely used as input data for CNNs in this field. However, CNN implementations require significant storage and computational resources, and the kernels (also known as filters) of each convolutional layer can vary in size. For example, a CNN processing MFCCs might use four convolutional layers, each with 16 kernels. However, the kernels in the first layer are typically 10×1×1, the second layer's kernels are typically 10×1×16, and the third and fourth layers' kernels are typically 6×1×16. This necessitates complex storage control and intermediate buffers, resulting in relatively large implementation area and power consumption. Summary of the Invention

[0003] Therefore, this invention provides an improved method for computation in a CNN. The CNN includes multiple convolutional layers, each using N convolutional kernels, where N is an integer greater than 1. The method includes the following steps: First, based on the number of convolutional kernels used in each convolutional layer, N multipliers and N adders are determined. Next, when the CNN reaches the i-th convolutional layer and all N convolutional kernels of the i-th convolutional layer are of size K×1×1, the N multipliers and N adders are used to perform one multiplication operation and one addition operation on the N convolutional kernels of the i-th convolutional layer in each cycle, resulting in N output results from the N convolutional kernels of the i-th convolutional layer after K cycles, where i is an integer greater than or equal to 1, and K is an integer greater than 1.

[0004] Preferably, the method further includes, when the CNN runs to the j-th convolutional layer and all N convolutional kernels of the j-th convolutional layer are of size P×1×N, using these N multipliers and N adders to calculate N multiplication operations and N addition operations of the target convolutional kernel among the N convolutional kernels of the j-th convolutional layer in each cycle, so that an output result of the target convolutional kernel is obtained after P cycles, where j is an integer greater than or equal to 1 and P is an integer greater than 1.

[0005] Preferably, the CNN further includes multiple fully connected layers, and the method further includes using these N multipliers and N adders to perform the transformation calculation of N data points of the input data in each cycle when the CNN runs to the k-th fully connected layer and the total number of input data of the k-th fully connected layer is M*N, so that the output result of the k-th fully connected layer is obtained after M cycles, where k and M are integers greater than or equal to 1.

[0006] To further understand the features and technical content of the present invention, please refer to the following detailed description and accompanying drawings. However, the drawings provided are for reference and illustration only and are not intended to limit the present invention. Attached Figure Description

[0007] Figure 1 This is a flowchart illustrating the steps of an improved CNN method for using convolutional kernels of different sizes in different convolutional layers, as provided in an embodiment of the present invention.

[0008] Figure 2 This is a schematic diagram of CNN processing MFCC according to an embodiment of the present invention.

[0009] Figures 3A to 3C yes Figure 1 The method is applied in Figure 2 This is a schematic diagram of the first convolutional layer. Detailed Implementation

[0010] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can understand the advantages and effects of the present invention from the content provided in this specification. The present invention can be implemented or applied through other different specific embodiments, and the details in this specification can also be modified and changed based on different viewpoints and applications without departing from the concept of the present invention. Furthermore, it should be understood that the accompanying drawings are for simple illustrative purposes only and are not depictions based on actual sizes. The following embodiments will further describe the relevant technical content of the present invention in detail, but the content provided is not intended to limit the scope of protection of the present invention.

[0011] It should be understood that while terms such as "first," "second," and "third" may be used in this document to describe various components or signals, these components or signals should not be limited by these terms. These terms are primarily used to distinguish one component from another, or one signal from another. Furthermore, the term "or" as used in this document should be interpreted to include, as appropriate, any combination of any one or more of the related listed items.

[0012] Please also refer to Figure 1 and Figure 2 , Figure 1This is a flowchart illustrating the steps of an improved CNN method using convolutional kernels of different sizes in different convolutional layers, as provided in this embodiment of the invention. Figure 2 This is a schematic diagram of a CNN processing MFCC according to an embodiment of the present invention. As mentioned above, a CNN includes multiple convolutional layers, and each convolutional layer can use N convolutional kernels, where N is an integer greater than 1. For the convenience of the following description, this embodiment only uses... Figure 2 The example shown is a CNN that processes MFCCs, consisting of four convolutional layers, each using 16 convolutional kernels. However, this invention does not limit the input data of the CNN to MFCCs, nor does it limit the number of convolutional layers or the number of convolutional kernels used in each layer. Typically, an MFCC can be a parameter matrix of size 1×13×1, and 99 parameters are input to the CNN. Figure 2 The input data can be a matrix of size 99×13×1, but this invention is not limited to this.

[0013] Since the kernels of the first convolutional layer are all 10×1×1 in size, existing techniques perform 10 multiplications and 9 additions with the kernel of the first convolutional layer to obtain a single output. Furthermore, since the kernels of the second convolutional layer are all 10×1×16 in size, existing techniques perform 160 multiplications and 159 additions with the kernel of the second convolutional layer to obtain a single output. Similarly, since the kernels of the 3rd and 4th convolutional layers are both 6×1×16 in size, the existing technology performs 96 multiplications and 95 additions on the 6*16 elements with one kernel of the 3rd convolutional layer to obtain an output result, and performs the same on the 4th convolutional layer.

[0014] Therefore, existing technologies require 10 multipliers in the first convolutional layer, 160 multipliers in the second convolutional layer, and 96 multipliers each in the third and fourth convolutional layers, making circuit merging difficult. Furthermore, each convolutional layer requires independent control and access circuits, especially the data storage section, where the number of elements read varies each time, leading to complex storage control and intermediate buffering. To address these technical problems, in Figure 1In step S110, based on the number of convolutional kernels used in these convolutional layers, the present invention determines N multipliers and N adders. Then, in step S120, when the CNN reaches the i-th convolutional layer and all N convolutional kernels of the i-th convolutional layer are of size K×1×1, the present invention uses these N multipliers and N adders to perform one multiplication operation and one addition operation on the N convolutional kernels of the i-th convolutional layer in each epoch, so that after K epochs, N output results of the N convolutional kernels of the i-th convolutional layer are obtained, where i is an integer greater than or equal to 1, and K is an integer greater than 1.

[0015] In other words, i, N, and K can be 1, 16, and 10 respectively in this embodiment, but the present invention is not limited thereto. Therefore, the present invention... Figure 2 In the first convolutional layer, each element is simultaneously multiplied and added once with each kernel of that layer. Please refer to the following: Figures 3A to 3C , Figures 3A to 3C yes Figure 1 The method is applied in Figure 2 This is a schematic diagram of the first convolutional layer. (See diagram below.) Figure 3A As shown, in the first cycle, the present invention can input data element A 1,1 With the first convolution kernel CK of the first convolutional layer 1,1 Element B 1,1 The result of this stage, C, is obtained by multiplying the result and then adding it to the result of the previous stage. 1,1 And also element A 1,1 With the second convolution kernel CK of the first convolutional layer 1,2 Element B 2,1 The result of this stage, C, is obtained by multiplying the result and then adding it to the result of the previous stage. 2,1 Similarly, this invention also includes element A. 1,1 With the 16th convolutional kernel CK of the first convolutional layer 1,16 Element B 16,1 The result of this stage, C, is obtained by multiplying the result and then adding it to the result of the previous stage. 16,1 Since the calculation results from the previous stage are not yet available at this point, this invention can involve element A. 1,1 Individual and element B 1,1 ~B 16,1 After multiplying, add 0 to get the result C. 1,1 ~C 16,1 .

[0016] Similarly, such as Figure 3B As shown, in the second cycle, the present invention will input data element A. 1,2 With the first convolution kernel CK of the first convolutional layer1,1 Element B 1,2 After multiplying, add the result C from the previous stage. 1,1 To obtain the calculation result C at this level. 1,2 And also element A 1,2 With the second convolution kernel CK of the first convolutional layer 1,2 Element B 2,2 After multiplying, add the result C from the previous stage. 2,1 To obtain the calculation result C at this level. 2,2 Similarly, this invention also includes element A. 1,2 With the 16th convolutional kernel CK of the first convolutional layer 1,16 Element B 16,2 After multiplying, add the result C from the previous stage. 16,1 To obtain the calculation result C at this level. 16,2 Therefore, as Figure 3C As shown, in the 10th cycle, the present invention can obtain a result equal to A. 1,1 *B r,1 +A 1,2 *B r,2 +A 1,3 *B r,3 +A 1,4 *B r,4 +A 1,5 *B r,5 +A 1,6 *B r,6 +A 1,7 *B r,7 +A 1,8 *B r,8 +A 1,9 *B r,9 +A 1,10 *B r,10 The calculation result C r,10 r are integers from 1 to 16 in sequence, which means that the output results of 16 convolution kernels are obtained at the same time.

[0017] Therefore, it can be seen that... Figure 2 For example, this invention can use 16 multipliers and 16 adders to perform one multiplication and one addition operation on the 16 convolutional kernels of the first convolutional layer in each cycle, so that after 10 cycles, 16 output results of the 16 convolutional kernels of the first convolutional layer are obtained. This simplifies storage control, and the intermediate cache only requires a separate storage block. Furthermore, in Figure 1In step S130, when the CNN runs to the j-th convolutional layer and all N convolutional kernels of the j-th convolutional layer are of size P×1×N, the present invention uses these N multipliers and N adders to calculate N multiplication operations and N addition operations of the target convolutional kernel in the N convolutional kernels of the j-th convolutional layer in each cycle, so that an output result of the target convolutional kernel is obtained after P cycles, where j is an integer greater than or equal to 1 and P is an integer greater than 1.

[0018] In other words, j and P can initially be 2 and 10 respectively in this embodiment, but the present invention is not limited thereto. Therefore, the present invention... Figure 2 In the second convolutional layer, 16 elements are multiplied 16 times and added 16 times with one of the convolutional kernels (i.e., the target convolutional kernel) of the second convolutional layer, resulting in an output of the target convolutional kernel after 10 cycles. Similarly, j and P can also be 3 and 6 respectively in this embodiment, therefore, the present invention... Figure 2 In the third convolutional layer, 16 elements are multiplied 16 times and added 16 times with one of the convolutional kernels (i.e., the target convolutional kernel) of the third convolutional layer, so that the output result of the target convolutional kernel is obtained after 6 cycles; or j and P can be 4 and 6 respectively in this embodiment, therefore, the present invention... Figure 2 In the fourth convolutional layer, 16 elements are multiplied 16 times and added 16 times with one of the convolutional kernels (i.e., the target convolutional kernel) of the fourth convolutional layer, so that the output result of the target convolutional kernel is obtained after 6 cycles.

[0019] It should be understood that the present invention does not limit the execution order or the number of executions of steps S120 and S130. Furthermore, the CNN may also include multiple fully connected layers for classification. However, since the working principle of fully connected layers is well known to those skilled in the art, its details will not be elaborated further. In summary, in Figure 1 In step S140, when the CNN runs to the k-th fully connected layer and the total number of input data for the k-th fully connected layer is M*N, the present invention will use these N multipliers and these N adders to complete the conversion calculation of N data points of the input data in each cycle, so that the output result of the k-th fully connected layer is obtained after M cycles, where k and M are integers greater than or equal to 1.

[0020] like Figure 2 As shown, k and M can initially be 1 and 13 respectively, therefore, in this invention... Figure 2 In the first fully connected layer, these 16 multipliers and 16 adders are used to perform transformation calculations on 16 input data entries in each cycle, resulting in the output of the first fully connected layer after 13 cycles. Similarly, k and M can both be 2 in this embodiment, therefore, the present invention... Figure 2 The second fully connected layer uses these 16 multipliers and 16 adders to perform the transformation calculation of 16 input data in each cycle, so that the output result of the second fully connected layer is obtained after 2 cycles.

[0021] In summary, compared with the prior art, the present invention provides an improved method for CNN computation that does not require complex storage control and intermediate cache, and results in a smaller area and lower power consumption.

[0022] The above-described content is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Therefore, all equivalent technical changes made based on the description and drawings of the present invention are included within the scope of protection of the present invention.

[0023] [Symbol Explanation]

[0024] S110~S140: Process Steps

[0025] A 1,1 A 1,2 A 1,10 A 1,11 A 1,99 A 2,1 A 2,2 A 2,10 A 2,11 A 2,99 A 13,1 A 13,2 A 13,10 A 13,11 A 13,99 : Elements of input data

[0026] CK 1,1 CK 1,2 CK 1,16 Convolution kernel

[0027] B 1,1 B 1,2 B 1,10 B 2,1 B 2,2 B 2,10 B 16,1 B 16,2 B 16,10 Elements of the convolution kernel

[0028] C 1,1 C 1,2 C 1,9 C 1,10 C 2,1 C 2,2 C 2,9 C 2,10C 16,1 C 16,2 C 16,9 C 16,10 : Calculation result.

Claims

1. A method for improving computation in a convolutional neural network, the convolutional neural network comprising multiple convolutional layers, each of the multiple convolutional layers using N convolutional kernels, where N is an integer greater than 1, and each of the multiple convolutional layers using convolutional kernels of different sizes, the method comprising: Based on the number of convolutional kernels used in all the convolutional layers, determine N multipliers and N adders; as well as When the convolutional neural network runs to the i-th convolutional layer and all N convolutional kernels of the i-th convolutional layer are of size K×1×1, the N multipliers and N adders are used to calculate one multiplication operation and one addition operation of the N convolutional kernels of the i-th convolutional layer in each cycle, and a single storage device is used to store the intermediate buffers generated by the N multipliers and N adders during the calculation, so that after K cycles, N output results of the N convolutional kernels of the i-th convolutional layer are obtained, where i is an integer greater than or equal to 1 and K is an integer greater than 1.

2. The method according to claim 1, further comprising: When the convolutional neural network runs to the j-th convolutional layer and all N convolutional kernels of the j-th convolutional layer are of size P×1×N, the N multipliers and N adders are used to calculate N multiplication operations and N addition operations of the target convolutional kernel among the N convolutional kernels of the j-th convolutional layer in each cycle, so that an output result of the target convolutional kernel is obtained after P cycles, where j is an integer greater than or equal to 1, and P is an integer greater than 1.

3. The method according to claim 2, wherein the convolutional neural network further comprises a plurality of fully connected layers, and the method further comprises: When the convolutional neural network runs to the kth fully connected layer and the total number of input data of the kth fully connected layer is M N, the conversion calculation of N pieces of data of the input data is completed at each cycle using the N multipliers and the N adders, so that the output result of the kth fully connected layer is obtained after M cycles, wherein k and M are integers greater than or equal to 1.

Images