A convolution operation circuit and operation method thereof

Abstract

The present invention proposes a convolution operation circuit and an operation method thereof. The convolution operation circuit is composed of a PE array and supports matrix multiplication operation and two-dimensional convolution operation at the same time. When performing matrix multiplication operation, the PE receives row input from a left matrix in the row direction and column input from a right matrix in the column direction respectively, multiplies the two and accumulates them into a local accumulation register. When performing two-dimensional convolution operation, the PE receives input feature map input from the row direction, multiplies the input feature map input with the convolution kernel weight stored internally, and then accumulates the partial sum outputted by the previous row of PE, and then outputs the accumulated partial sum to the next row. The present invention sets a plurality of weight registers inside the PE, stores the convolution kernel weights of multiple channels in a folded manner inside a column of PE, sets a first-in-first-out memory at the bottom of the PE array for storing intermediate calculation results, avoids repeated transmission of weights and intermediate calculation results between the memory and the PE, and improves operation efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of convolution operations, and in particular to a convolution operation circuit and an operation method thereof which simultaneously supports matrix multiplication operations and two-dimensional convolution operations. Background Art

[0002] The convolution layer is one of the basic layers of the convolutional neural network. The convolution operation is the most important type of operation in the field of deep learning, accounting for more than 90% of the operations of the convolutional neural network. Among the remaining calculations, the calculation of the fully connected layer accounts for a large part. The calculation of the fully connected layer can be regarded as a matrix multiplication calculation. Therefore, most of the calculations in deep learning are convolution operations and matrix multiplication calculations.

[0003] One convolution operation method is to convert the convolution operation into a matrix multiplication operation through the im2col operation. This method expands the convolution kernel into a matrix and converts the input feature map into a matrix. The weight of a window in the convolution kernel is converted into a column of another matrix, and the weights of all convolution kernels are combined to form a new convolution kernel matrix; each sliding window in the input feature map is converted into a row of the matrix, and the data of all sliding windows are combined to form a new feature map matrix; the convolution kernel matrix and the feature map matrix are matrix multiplied to obtain the convolution result.

[0004] If the im2col operation is implemented by software, the same feature map data will appear in multiple sliding windows, resulting in a large number of repeated elements in the matrix, which takes up a lot of extra memory and bandwidth; if the im2col operation is implemented by hardware, it is necessary to design additional complex hardware circuits. For convolution operations, there are already special hardware circuits to support them. Such hardware circuits usually use a systolic array method to fix the weights on the operation units, and the partial sums generated by the convolution flow and accumulate between the operation units until all partial sums are accumulated, and the convolution result is output on the last operation unit.

[0005] However, to date, there is no circuit structure in the field of deep learning that can simultaneously support matrix multiplication operations and two-dimensional convolution operations. Summary of the invention

[0006] In view of the gap in the circuit structure that simultaneously supports matrix multiplication operations and two-dimensional convolution operations in the field of deep learning, the present invention proposes a convolution operation circuit and an operation method thereof that simultaneously supports matrix multiplication operations and two-dimensional convolution operations.

[0007] The present invention protects a convolution operation circuit, which is composed of m rows and n columns of operation units (PE: Processing Element), and supports matrix multiplication operation and two-dimensional convolution operation at the same time;

[0008] When performing a matrix multiplication operation, the operation unit receives a row input from a left matrix in the row direction and a column input from a right matrix in the column direction, respectively, multiplies the two and then accumulates them into a local accumulation register;

[0009] When performing a two-dimensional convolution operation, the operation unit receives an input feature map input in the row direction, multiplies it with the internally stored convolution kernel weight, accumulates it with the partial sum output by the operation unit in the previous row, and then outputs the accumulated partial sum to the next row.

[0010] Furthermore, the operation unit is mainly composed of a multiplier, an accumulator and a weight register; the row direction input side of the multiplier inputs the input in the row direction; the column direction input side of the multiplier is provided with two inputs that can be selected, which are the input in the column direction and the convolution kernel weight input from the weight register; the partial sum input side of the accumulator is provided with two inputs that can be selected, which are the partial sum from the local accumulator register and the output of the operation unit in the previous row.

[0011] Furthermore, each operation unit is provided with a plurality of weight registers, and the weights are folded and arranged on the operation unit in units of sliding window size. An address selection controller is provided between the weight register and the column-direction input side of the multiplier. The address selection controller selects the convolution kernel weights stored in the plurality of weight registers according to the value of the address selection register to participate in the convolution operation.

[0012] Furthermore, the last row of operation units in each column of operation units is connected to a FIFO (First In First Out) memory, and the output of the FIFO memory is connected to the first row of operation units.

[0013] The present invention also protects the operation method of the above-mentioned convolution operation circuit. When performing matrix multiplication operation, the corresponding row data of the input left matrix is ​​broadcast to the row direction input side of the operation unit multiplier, and the corresponding column data of the input right matrix is ​​broadcast to the column direction input side of the operation unit multiplier. Each operation unit multiplies the data received in the row direction and the column direction in each cycle and accumulates them into the local accumulation register.

[0014] When performing a two-dimensional convolution operation, the weight of each convolution kernel is sequentially stored in the operation unit weight register of the corresponding column of the operation circuit. If the width of the convolution kernel sliding window is C and the height is R, when calculating the convolution operation result of the convolution kernel sliding on the corresponding row of the input feature map matrix, the 1-R rows of input feature map row data where the convolution kernel currently performs the convolution operation are broadcasted in sequence to the 1st to Cth rows, the C+1th to 2Cth rows, ..., the R*CCth to R*Cth rows of the input operation circuit; the input feature map row data of the i+1th row is input into the PE array C cycles later than the input feature map row data of the i-th row, where 1≤i≤R; each row of input feature map row data is sequentially input into the PE unit, and one data is input in each cycle; each operation unit multiplies the data received in the row direction with the convolution kernel weight selected by the address selection controller in each cycle, and accumulates the partial sum output by the operation unit in the previous row, and outputs the accumulated partial sum to the next row, until the convolution kernel calculation result of the column is output in the last row.

[0015] Furthermore, when the sliding window of the convolution kernel is too large and a column of PEs in the PE array cannot hold all the weight data of a convolution kernel at once, the weight distribution is performed according to the following scheme:

[0016] For the convolution kernel with single-channel input, the remaining weight data is folded down from the first row of the PE array and placed one column at a time. The weights of the convolution kernels are placed in The i-th fold places the weight in the i-th weight register inside the computational unit, where 1≤i≤ After the convolution kernel weights are arranged in this way, when the input feature map is input into the PE array, the previous convolution kernel sliding along the row direction is first input. The address selection controller selects the value of the first weight register to participate in the multiplication operation. The last line outputs the intermediate result of the convolution operation and is temporarily stored in the FIFO memory. After completing the row data entry, enter the second The address selection controller selects the value of the second weight register to participate in the multiplication operation, and the intermediate result temporarily stored in the FIFO is input from the first row to participate in the accumulation operation; the input feature map data of the remaining rows are input into the PE array in a similar manner until all the row input feature map data that the convolution kernel slides through along the row direction are fully input. At this time, the final convolution result is stored in the FIFO.

[0017] For the convolution kernel with multi-channel input, let the number of input channels be I, and fold and place the convolution kernel weights of each input channel according to the weight distribution scheme of the convolution kernel with single-channel input; first place the weight of the first input channel, and each column still places The weights of the convolution kernels are arranged After that, the weight of the second channel is folded down in sequence until the weight of the first channel is folded down in total. The i-th fold places the weight in the i-th weight register inside the PE, where In this case, when the input feature map is input into the PE array, the R rows of data that the convolution kernel slides through in the row direction for each channel of the input feature map are input into the PE array in the following order. From the 1st row of data participating in the operation of the 1st channel to the Rth row of data participating in the operation of the last channel, a total of I*R rows of data participate in the multi-channel two-dimensional convolution operation, which are respectively recorded as 1 to I*R rows of input feature map data participating in the operation. First, input the first row of data that the convolution kernel slides through in the row direction. The address selection controller selects the value of the first weight register to participate in the multiplication operation. The last line outputs the intermediate result of the convolution operation and is temporarily stored in the FIFO memory. After completing the row data entry, enter the second The address selection controller selects the value of the second weight register to participate in the multiplication operation, and the intermediate result temporarily stored in the FIFO is input from the first row to participate in the accumulation operation; the input feature map data of the remaining rows are input into the PE array in a similar manner until all the row input feature map data that the convolution kernel slides through along the row direction are fully input. At this time, the final convolution result is stored in the FIFO.

[0018] The beneficial effects of the present invention are as follows: 1. It can efficiently perform both matrix multiplication operations and two-dimensional convolution operations; 2. It supports common two-dimensional convolution operations with multiple input channels and multiple convolution kernels; 3. When the number of input channels is large, and the number of input channels multiplied by the number of weights in the sliding window of the convolution kernel is greater than the height of the PE array, the intermediate results of the convolution calculation are cached through the FIFO of the PE array, so that the intermediate results of the convolution calculation do not need to be moved out of the PE array to complete all convolution calculations; 4. By folding the weights and storing them in multiple weight registers inside the PE, the weight data of a convolution kernel can be imported into one column of the PE array at one time, so that the weight data does not need to be repeatedly imported into the PE array when participating in convolution operations multiple times. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the convolution operation circuit;

[0020] Figure 2 Schematic diagram of the internal structure of the computing unit;

[0021] Figure 3 It is the internal working circuit of the operation unit when performing matrix multiplication operation;

[0022] Figure 4 A schematic diagram of data flow when the operation unit performs a matrix multiplication operation;

[0023] Figure 5 Schematic diagram of left and right matrix multiplication;

[0024] Figure 6 A schematic diagram of the arrangement of the row data of the left matrix and the column data of the right matrix into the operation circuit;

[0025] Figure 7 It is the internal working circuit of the operation unit when performing a two-dimensional convolution operation;

[0026] Figure 8 Schematic diagram of input feature map and convolution kernel;

[0027] Fig. 9 Schematic diagram of data flow when performing a two-dimensional convolution operation;

[0028] Fig.10 This is the data status diagram of the preparation stage before the calculation starts;

[0029] Figure 11-Figure 21 They are the data state diagrams from the first cycle to the eleventh cycle of the two-dimensional convolution operation. DETAILED DESCRIPTION

[0030] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention are provided for the purpose of illustration and description, and are not intended to be exhaustive or to limit the present invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention, and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

[0031] Example 1

[0032] A convolution operation circuit is composed of m rows and n columns of operation units PE (for example Figure 1 The convolution operation circuit shown is composed of 4 rows and 8 columns of operation units PE), and supports two working modes: matrix multiplication operation and two-dimensional convolution operation.

[0033] When performing a matrix multiplication operation, the operation unit receives row inputs from the left matrix in the row direction and column inputs from the right matrix in the column direction, respectively, multiplies the two, and then accumulates them into a local accumulation register.

[0034] When performing a two-dimensional convolution operation, the operation unit receives an input feature map input in the row direction, multiplies it with the internally stored convolution kernel weight, accumulates it with the partial sum output by the operation unit in the previous row, and then outputs the accumulated partial sum to the next row.

[0035] To implement the above solution, in this embodiment, the operation unit is mainly composed of a multiplier, an accumulator and a weight register, such as Figure 2 shown.

[0036] The row direction input side of the multiplier receives input in the row direction.

[0037] The column-direction input side of the multiplier is provided with two inputs for selection, namely, the column-direction input and the convolution kernel weight input from the weight register. When performing a matrix multiplication operation, the column-direction input is selected; when performing a two-dimensional convolution operation, the convolution kernel weight input from the weight register is selected.

[0038] The partial sum input side of the accumulator is provided with two inputs that can be selected, which are respectively from the local accumulator register and the partial sum output from the previous row of operation units. When performing matrix multiplication operation, the local accumulator register is selected; when performing two-dimensional convolution operation, the partial sum output from the previous row of operation units is selected.

[0039] The accumulation result selector controls whether the accumulated result is output to the local accumulation register or to the next row of operation units. When performing matrix multiplication operations, the accumulated result is output to the local accumulation register; when performing two-dimensional convolution operations, the accumulated result is output to the next row of adjacent operation units.

[0040] The convolution operation method is described below in conjunction with the convolution operation circuit disclosed in the present invention and its data flow mode.

[0041] When the convolution operation circuit works in the matrix multiplication operation mode, the internal working circuit of the operation unit is as follows Figure 3 As shown, the corresponding row data of the left matrix is ​​broadcast to the row direction input side of the operation unit multiplier, and the corresponding column data of the right matrix is ​​broadcast to the column direction input side of the operation unit multiplier. Each operation unit multiplies the data received in the row direction and the column direction in each cycle and accumulates them into the local accumulation register. The overall data flow is as follows Figure 4 shown.

[0042] by Figure 5 Take the multiplication operation of the 8×8 left matrix A and the 8×8 right matrix B as an example. When performing matrix multiplication, the row data of the left matrix and the column data of the right matrix are calculated according to Figure 6 Arrange as shown and enter Figure 1The operation unit corresponding to the convolution operation circuit shown in the figure. Each operation unit multiplies the broadcast input data in the row direction with the broadcast input data in the column direction, then adds the value in the local accumulation register, and finally stores the operation result in the local accumulation register. At this time, the value stored in each local accumulation register in the PE array is the value of each element of the result matrix.

[0043] When the number of rows of the left matrix is ​​greater than the number of rows of the PE array or the number of columns of the right matrix is ​​greater than the number of columns of the PE array, the matrix needs to be processed in blocks so that the two block matrices to be multiplied satisfy that the number of rows of the left block matrix is ​​less than or equal to the number of rows of the PE array and the number of columns of the right block matrix is ​​less than or equal to the number of columns of the PE array. Perform block matrix multiplication on each block matrix after block processing, then temporarily store the operation results, and finally combine the operation results of each block matrix to obtain the matrix multiplication result of the left matrix and the right matrix. Figure 6 For example, the left matrix and the right matrix are both 8×8 matrices, and the PE array is 4 rows and 8 columns. The 8 columns of the right matrix can be input into the PE array at one time, while the left matrix needs to be processed in blocks and divided into two 4×8 block matrices for processing.

[0044] When the convolution operation circuit works in the two-dimensional convolution operation mode, the internal working circuit of the operation unit is as follows Figure 7 As shown. Generally speaking, a convolution layer in a deep learning model has multiple convolution kernels. The calculations of these convolution kernels have no data correlation and can be executed in parallel. In this embodiment, each column of PE stores the data of a convolution kernel to complete the convolution calculation of a convolution kernel. The weight data of a convolution window is arranged row by row in the weight register of the corresponding column of PE. After the weights of the convolution window for the first input channel are arranged, the weights of the convolution window for the second input channel are arranged downward on the column. Fig. 9 The bottom row of the PE array is shown. Therefore, the weight of each convolution kernel is stored in the weight register of the operation unit in the corresponding column of the operation circuit in sequence, which not only means that the weights of the convolution windows of the same input channel are arranged in sequence, but also the weights of the convolution windows of different input channels are arranged in sequence.

[0045] Let the width of the convolution kernel sliding window be C and the height be R. When calculating the convolution operation result of the convolution kernel sliding on the corresponding row of the input feature map matrix, the 1-R rows of input feature map row data where the convolution kernel currently performs the convolution operation are broadcasted in sequence to the 1st to Cth rows, the C+1th to 2Cth rows, ..., the R*CCth to R*Cth rows of the input operation circuit; the input feature map row data of the i+1th row is input into the PE array C cycles later than the input feature map row data of the i-th row, where 1≤i≤R; each row of input feature map row data is input into the PE unit in sequence, and one data is input in each cycle; each operation unit multiplies the data received in the row direction with the convolution kernel weight selected by the address selection controller in each cycle, and accumulates the partial sum output by the operation unit in the previous row, and outputs the accumulated partial sum to the next row, until the convolution kernel calculation result of the column is output in the last row.

[0046] by Figure 8 As shown in the figure, the input feature map is a 6×6 matrix and the convolution kernel is a 3×3 matrix. The distribution of the convolution kernel weights in the PE array is as follows: Fig. 9 shown. Fig. 9 Xn,c,h,w in the formula represents the hth row and wth column data of the cth feature map of the nth batch of data input from the row direction, and Wo,i,x,y represents the xth row and yth column weight of the oth convolution kernel stored in the weight register for the i-th input channel convolution window. Fig. 9 As shown in the figure, the weight of the first convolution kernel is arranged in the first column of the PE array; the weight of the second convolution kernel is arranged in the second column of the PE array, and so on. When the number of convolution kernels is greater than the number of columns in the PE array, a similar split can be performed. In this way, each column of PE works in parallel, and different convolution kernels for the same input feature map can perform convolution operations in parallel at the same time.

[0047] Reference Fig. 9 , the convolution kernel storage is as follows: the 3 weights of the first row of the convolution kernel for the 0th input channel are stored in the 1st-3rd operation unit weight registers from top to bottom along the column direction of the PE array, the 3 weights of the second row of the convolution kernel are stored in the 4th-6th operation unit weight registers from top to bottom along the column direction of the PE array, and the 3 weights of the third row of the convolution kernel are stored in the 7th-9th weight registers from top to bottom along the column direction of the PE array. If the number of PE array rows is sufficient, after the weights of the convolution window for the 0th input channel are arranged, the weights of the convolution window for the 1st input channel can be arranged downward on the column, refer to Fig. 9 The bottom row of the PE array is shown.

[0048] When calculating the convolution operation result of sliding three rows of the convolution kernel on the input feature map matrix, the data is Fig. 9The input PE array is shown. If there are multi-channel inputs, the weights of the convolution window for the 0th input channel are arranged in rows 1-9, the weights of the convolution window for the 1st input channel are arranged in rows 10-18, and so on. The input feature maps of different input channels are input into the PE array in the corresponding rows of the weights of the convolution window for the corresponding input channel.

[0049] Figure 8 In the example, since the convolution operation result of the convolution kernel sliding three rows on the input feature map matrix is ​​calculated, the first row of data of the input feature map is broadcasted to the 1st to 3rd rows of the PE array, the second row of data of the input feature map is broadcasted to the 4th to 6th rows of the PE array, and the third row of data of the input feature map is broadcasted to the 7th to 9th rows of the PE array. Similarly, when calculating the convolution operation result of the convolution kernel sliding three rows under the input feature map matrix, the fourth row of data of the input feature map is broadcasted to the 1st to 3rd rows of the PE array, the fifth row of data of the input feature map is broadcasted to the 4th to 6th rows of the PE array, and the sixth row of data of the input feature map is broadcasted to the 7th to 9th rows of the PE array.

[0050] In order to improve the applicability of the operation circuit, multiple weight registers can be set inside each operation unit, and the weights are folded and arranged on the operation unit.

[0051] For a convolution kernel with a single input channel, let the convolution kernel sliding window width be C and height be R. When the sliding window is too large and one column of PE array cannot hold all weight data at once, that is, when R*C>m, the remaining weight data will be folded down from the first row of the PE array in sequence; the weights should be rearranged from the first row to ensure that a row of weights is placed on consecutive PE computing units. The weights of the convolution kernels are placed in The i-th fold places the weight in the i-th weight register inside the PE, where 1≤i≤

[0052] For the convolution kernel with multiple input channels, let the width of the convolution kernel sliding window be C and the height be R. There are I input channels in total. First, the weight of the first channel is arranged in a single-channel manner. Each column is still arranged The weights of the convolution kernels are arranged After that, the weight of the second channel is folded down in sequence until the weight of the first channel is folded down in total. The i-th fold places the weight in the i-th weight register inside the PE, where

[0053] Based on the above convolution kernel weight distribution scheme, all weights of a convolution kernel can be input into the PE array at one time. Compared with calculating the convolution results of each channel one by one and then accumulating them, it avoids the repeated export or import of the intermediate results of the convolution into the PE array, improves efficiency and reduces power consumption.

[0054] Below Fig. 9 The specific workflow of the two-dimensional convolution operation is described by taking the first column of the PE array shown as an example.

[0055] Fig.10 This is the data state diagram before the calculation starts. The dotted line indicates the propagation direction of the input data, and the solid line indicates the propagation direction of the convolution operation part. Figure 11-Figure 21 They are the data state diagrams from the first cycle to the eleventh cycle of the two-dimensional convolution operation. Fig.10 X in 1,7 -X 1,11 In the input feature Figure X There is no corresponding data in , so it can be treated as zero.

[0056] The first cycle of two-dimensional convolution operation: X 1,1 The first row and first column of the input operation circuit PE is compared with the weight W stored in the PE. 1,1 Multiply, and then output W to the next line 1,1* X 1,1 , refer to Fig.11 .

[0057] The second cycle of two-dimensional convolution operation: X 1,2 The second row and first column of the input operation circuit PE is compared with the weight W stored in the PE. 1,2 Multiply and then add to the W output in the previous line 1,1* X 1,1 Accumulate and output W to the next line 1,1* X 1,1 +W 1,2* X 1,2 , refer to Fig.12 .

[0058] And so on, until the ninth cycle of the two-dimensional convolution: X 3,3 The input circuit is in the 9th row and 1st column PE, and the weight W stored in the PE 3,3 Multiply, then add to the previous line output, and output W to the next line 1,1* X 1,1 +W 1,2* X 1,2 +W 1,3* X 1,3 +W 2,1* X 2,1 +W 2,2* X 2,2+W 2,3* X 2,3 +W 3,1* X 3,1 +W 3,2* X 3,2 +W 3,3* X 3,3 , which is the result of the convolution operation when the 3×3 convolution kernel W is located in the upper left corner of the 6×6 input feature map, refer to Fig.19 .

[0059] The tenth cycle of the two-dimensional convolution: X 3,4 The input circuit is in the 9th row and 1st column PE, and the weight W stored in the PE 3,3 Multiply, then add to the previous line output, and output W to the next line 1,1* X 1,2 +W 1,2* X 1,3 +W 1,3* X 1,4 +W 2,1* X 2,2 +W 2,2* X 2,3 +W 2,3* X 2,4 +W 3,1* X 3,2 +W 3,2* X 3,3 +W 3,3* X 3,4 , which is the 3×3 convolution kernel W from Fig.19 The convolution result after sliding one grid to the right is shown in the figure. Fig. 20 .

[0060] By analogy, the eleventh cycle of the two-dimensional convolution: X 3,5 The input circuit is in the 9th row and 1st column PE, and the weight W stored in the PE 3,3 Multiply, then add to the previous line output, and output W to the next line 1,1* X 1,3 +W 1,2* X 1,4 +W 1,3* X 1,5 +W 2,1* X 2,3 +W 2,2* X 2,4 +W 2,3* X 2,5 +W 3,1* X 3,3 +W 3,2* X 3,4 +W 3,3* X 3,5 , which is the 3×3 convolution kernel W from Fig. 20 The convolution result after sliding one grid to the right again is shown in Figure 1. Fig.21 .

[0061] In each cycle, each operation unit multiplies the data received in the row and column directions and accumulates the partial sum output by the operation unit in the previous row. The next row outputs the accumulated partial sum until the last row outputs the convolution kernel calculation result of the column.

[0062] The last row of PEs in each column is connected to a FIFO memory, and the output of the FIFO memory is connected to the first row of PEs for storing the partial sums of the convolution calculations. When the number of input channels is large and the number of input channels multiplied by the number of weights in the sliding window of the convolution kernel is greater than the height of the PE array, the intermediate results of the convolution calculation are cached through the FIFO of the PE array, so that the intermediate results of the convolution calculation do not need to be moved out of the PE array to complete the entire convolution calculation.

[0063] When performing a two-dimensional convolution operation on a single input channel, if the sliding window is too large and the PE array cannot hold all the weight data at once, that is, when R*C>m, the weight data is folded down from the first row of the PE array in sequence; the weights are rearranged from the first row to ensure that the weights of a row of convolution kernels are placed on consecutive PE operation units; and the weights are placed one row at a time on one column of the PE array in sequence. The weights of the convolution kernels are placed in The i-th fold places the weight in the i-th weight register inside the PE, where In this case, when the input feature map is input into the PE array, the convolution kernel first slides through the previous The address selection controller selects the value of the first weight register to participate in the multiplication operation. The last line outputs the intermediate result of the convolution operation and is temporarily stored in the FIFO memory. After completing the row data entry, enter the second The address selection controller selects the value of the second weight register to participate in the multiplication operation, and the intermediate result temporarily stored in the FIFO is input from the first row to participate in the accumulation operation; the input feature map data of the remaining rows are input into the PE array in a similar manner until all the input feature map data of all rows that the convolution kernel slides along the row direction are input. At this time, the final convolution result is stored in the FIFO.

[0064] When performing a two-dimensional convolution operation on multiple input channels, the number of input channels is recorded as I. When there are too many convolution kernel weights for the feature maps of multiple input channels, that is, I*R*C>m, and a column of PEs in the PE array cannot hold all the weight data at once, all the convolution kernel weights are placed in a similar manner to the above; the weights are rearranged from the first row to ensure that a row of weights of the convolution kernel is placed on consecutive PE operation units. Similarly, when the input feature maps of each channel are transmitted to the PE array, they are also folded and arranged in a corresponding manner and transmitted to the PE array in a manner corresponding to the weights involved in the calculation; the R rows of data that each channel of the input feature map slides through along the row direction of the convolution kernel are input into the PE array in the following order. A total of I*R rows of data participate in the multi-channel two-dimensional convolution operation, from the first row of data participating in the operation of the first channel to the Rth row of data participating in the operation of the last channel, respectively recorded as the 1st to I*R rows of input feature map data participating in the operation. If R*C>m, that is, the weights of the convolution kernel for the feature map of the first channel cannot be placed in sequence in a column of PEs, first input the feature map of the first input channel in the aforementioned order. Row data. If R*C≤m, but I*R*C>m, that is, the weights of the convolution kernel for the first channel feature map can be placed in sequence inside a column of PE, but the weights for all input channels cannot be placed in sequence inside a column of PE; input the R rows of data of the first input channel feature map and the first X rows of data of the second input channel feature map in the aforementioned order, where X is the largest integer that makes (R+X)*C≤m. This batch of data input in sequence is input one by one in the order of the row direction until all the data in the row direction are input into the PE array. This batch of data input in sequence is multiplied by the first register storing weights in each PE. When the first batch of data is all involved in the calculation, the FIFO stores the intermediate results of the first batch of row data of the input feature map after the convolution operation. Afterwards, the second batch of data from the 1 to I*R rows of data involved in the operation of the multi-channel input feature map is selected according to the above rules, and input into the PE array in the aforementioned order. This batch of input data is multiplied by the second register storing weights in each PE. At the same time, the intermediate results of the convolution calculation stored in the FIFO are input from the first row of PEs in sequence, and then passed down in the form of partial sums to participate in the convolution calculation process of this batch. After each batch of input feature map data is calculated, the FIFO stores the convolution results of all input feature maps that participated in the calculation before. When the R rows of data (a total of I*R rows of data) of all input channels have been input into the PE array, the FIFO memory stores the convolution calculation results of the convolution kernel for the R rows of data of the multi-channel input feature map. The two-dimensional convolution calculation of the n columns of PE is executed in parallel. When the R rows of data (a total of I*R rows of data) of all input channels have been input into the PE array, the convolution calculation results of the n convolution kernels stored in the PE array for the multi-channel input feature map are calculated and output from the FIFO or directly output to the convolution result memory when calculating the convolution calculation results of the last batch of input feature maps.

[0065] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field and related fields without creative work should fall within the scope of protection of the present invention.

Claims

1. A convolution operation circuit, characterized in that: It consists of m rows and n columns of PEs, supporting both matrix multiplication and two-dimensional convolution operations. When performing a matrix multiplication operation, the operation unit receives a row input from a left matrix in the row direction and a column input from a right matrix in the column direction, respectively, multiplies the two and then accumulates them into a local accumulation register; When performing a two-dimensional convolution operation, the operation unit receives an input feature map input in the row direction, multiplies it with the internally stored convolution kernel weight, and then accumulates it with the partial sum output by the operation unit in the previous row, and then outputs the accumulated partial sum to the next row; The arithmetic unit is mainly composed of a multiplier, an accumulator and a weight register; The row direction input side of the multiplier receives input in the row direction; The column direction input side of the multiplier is provided with two inputs which are selected one from the other, namely, the input in the column direction and the convolution kernel weight input from the weight register; The partial sum input side of the accumulator is provided with two inputs which are selected one from the local accumulator register and the partial sum output from the previous row of operation units; Each operation unit is internally provided with a plurality of weight registers, and the weights are folded and arranged on the operation unit in units of sliding window size, and the weight registers are provided with an address selection controller, and the address selection controller selects the convolution kernel weights stored in the plurality of weight registers according to the value of the address selection register to participate in the convolution operation; The operation method of the convolution operation circuit is as follows: when performing matrix multiplication operation, the corresponding row data of the input left matrix is ​​broadcast to the row direction input side of the operation unit multiplier, and the corresponding column data of the input right matrix is ​​broadcast to the column direction input side of the operation unit multiplier, and each operation unit multiplies the data received in the row direction and the column direction in each cycle and accumulates them into the local accumulation register; When performing a two-dimensional convolution operation, the weight of each convolution kernel is stored in the weight register of the operation unit of the corresponding column of the operation circuit in sequence; if the width of the convolution kernel sliding window is C and the height is R, then when calculating the convolution operation result of the convolution kernel sliding on the corresponding row of the input feature map matrix, the 1-R rows of input feature map row data where the convolution kernel currently performs the convolution operation are broadcasted in sequence to the 1st to Cth rows, the C+1th to 2Cth rows, ..., the R*CCth to R*Cth rows of the input operation circuit; the input feature map row data of the i+1th row is input into the PE array C cycles later than the input feature map row data of the i-th row, and each row of input feature map row data is input into the PE unit in sequence, with one data input in each cycle; each operation unit multiplies the data received in the row direction with the convolution kernel weight selected by the address selection controller in each cycle, and then accumulates the partial sum output by the operation unit of the previous row, and outputs the accumulated partial sum to the next row, until the convolution kernel calculation result of the column is output in the last row.

2. The convolution operation circuit according to claim 1, characterized in that: The last row of operation units in each column of operation units is connected to a FIFO memory, and the output of the FIFO memory is connected to the first row of operation units.

3. The convolution operation circuit according to claim 1, characterized in that: When the convolution kernel sliding window is too large and a column of PEs in the PE array cannot hold all the weight data of a convolution kernel at once, the weight distribution is performed according to the following scheme: For the convolution kernel with single-channel input, the remaining weight data is folded down from the first row of the PE array and placed one column at a time. The weights of the convolution kernels are placed in The i-th fold places the weight in the i-th weight register inside the computational unit, where 1≤i≤ For the convolution kernel with multi-channel input, let the number of input channels be I, and fold and place the convolution kernel weights of each input channel according to the weight distribution scheme of the convolution kernel with single-channel input; first place the weight of the first input channel, and each column still places The weights of the convolution kernels are placed After that, the weight of the second channel is folded down in sequence until the weight of the first channel is folded down in total. The i-th fold places the weight in the i-th weight register inside the PE, where

Images