A hardware acceleration method and hardware acceleration circuit for convolution

Abstract

The present invention discloses a convolution hardware acceleration method and a hardware acceleration circuit, and belongs to the field of convolutional neural network and chip design. According to the step length of the convolution operation in two directions, the weight cube and the feature data cube are divided into multiple sub-blocks, and the width and height of each sub-block are equal to the step length of the convolution kernel in two directions; the data blocks in each sub-block are rearranged for the first time and the second time; according to the number of multiplication and addition units, the rearranged data matrix and the weight matrix are divided into blocks along the channel direction to obtain multiple sub-matrices; the convolution operation is performed on each sub-matrix respectively to obtain multiple parts and matrices; the multiple parts and the corresponding position elements of the matrix are accumulated to obtain the convolution operation result. The present invention improves the utilization rate of the multiplier-adder in the convolution array, reduces the number of calculations, and improves the efficiency and speed of the convolution calculation; it can save the storage resources of the storage module and save the data reading bandwidth during the convolution operation.

Description

Technical Field

[0001] The present invention relates to the technical field of convolutional neural networks and chip design, and in particular to a convolution hardware acceleration method and a hardware acceleration circuit. Background Art

[0002] In recent years, artificial intelligence technology has developed rapidly. Convolutional neural networks, as a key technology, have also been widely used in target detection, autonomous driving and other fields.

[0003] Convolutional neural networks contain a large number of convolution operations, which consume a lot of time and limit the speed of convolutional neural networks. In the hardware circuit used to accelerate neural networks, the convolution array responsible for convolution operations is the most critical part. The speed and efficiency of the convolution array determine the performance of the neural network. Since the size of the convolution array is fixed, while the data size of the convolution operation is not fixed, especially when the number of channels of the data is small, the resources of the convolution array are often wasted, which restricts the performance of the convolutional neural network. Summary of the invention

[0004] The purpose of the present invention is to provide a convolution hardware acceleration method and a hardware acceleration circuit to solve the problems in the background technology.

[0005] In order to solve the above technical problems, the present invention provides a convolution hardware acceleration method and a hardware acceleration circuit, comprising the following steps:

[0006] The weight cube and feature data cube are divided into multiple sub-blocks according to the step length of the convolution operation in two directions, and the width and height of each sub-block are equal to the step length of the convolution kernel in two directions;

[0007] Performing a first rearrangement and a second rearrangement on the data blocks in each sub-block;

[0008] According to the number of multiplication and addition units, the rearranged data matrix and the weight matrix are divided into blocks along the channel direction to obtain multiple sub-matrices;

[0009] A convolution operation is performed on each submatrix to obtain multiple parts and matrices; elements at corresponding positions of the multiple parts and matrices are accumulated to obtain a convolution operation result.

[0010] In one embodiment, the first rearrangement is to arrange the data blocks of each row in sequence from left to right in the channel direction; the second rearrangement is to arrange the data blocks in each sub-block in sequence from top to bottom in the channel direction to obtain the rearranged data cube and weight cube.

[0011] In one embodiment, when the step length of the convolution operation in both directions is 1, the weight cube and the data cube are not rearranged; when the step length of the convolution operation in any direction is not 1, the step length of the rearranged convolution operation is 1.

[0012] In one embodiment, when the width or height of the weight cube or the feature data cube cannot be divided by the convolution step, the edges of the weight cube and the feature data cube are padded so that their width and height are both divisible by the convolution step, and the values ​​of the padded parts are all 0.

[0013] The present invention also provides a hardware acceleration circuit, the hardware acceleration circuit is connected to an external storage module, and the external storage module is used to store multiple feature cubes, multiple convolution kernels and multiple operation results;

[0014] The hardware acceleration circuit comprises:

[0015] A data reading module, which reads the convolution kernel and feature data required for this convolution operation from the external storage module;

[0016] A data cache module, caching the convolution kernel and feature data read from the external storage module;

[0017] A data loading module, which reads the convolution kernel and feature data from the data cache module in sequence and sends them to the convolution unit;

[0018] A convolution array, composed of convolution units, receives the convolution kernel and data of the data loading module, and performs multiplication and addition operations on the convolution kernel and data to obtain a plurality of convolution partial sums;

[0019] The convolution accumulation module accumulates the sum of the convolution parts to obtain the final convolution result, and sends the convolution result to the data transformation module or outputs it to the external storage module;

[0020] The data transformation module transforms the data and weights of the next convolution operation and outputs the results to the external memory.

[0021] In one embodiment, the data reading module reads the feature data in sequence from left to right and from top to bottom. The feature data read each time is the same as the number of convolution units of the convolution array in the channel direction. When the number of feature data channels is less than the number of convolution units, padding is performed in the channel direction, and the value of the padding part is 0; when the number of feature data channels is greater than the number of convolution units, it is read in multiple times in order from front to back.

[0022] In one embodiment, the data cache module is configured into two parts according to the size of the feature data and the weight, which are used to store the weight and the feature data respectively.

[0023] In one embodiment, the data loading module loads a weight element and the feature data that needs to be convolved with the weight element in a sequence from left to right, from top to bottom, and from front to back according to the size of the feature data, the size of the weight, and the size of the convolution result, and sends a flag signal to inform the operation module whether all the data to be convolved with the current weight element has been loaded.

[0024] In one embodiment, the convolution array includes multiple convolution units, each of which is composed of multiple multipliers and adders. Each multiplier multiplies the weight and feature data of a channel, and then adds the products of all multipliers to obtain a partial sum data, and then calculates the convolution of the next feature data and the weight element.

[0025] In one embodiment, the convolution accumulation module is used to cache the convolution partial sum of the same weight element, and accumulate it with the partial sum of the next weight element to obtain the final convolution result.

[0026] In one embodiment, the data transformation module includes a data reading module, an internal storage module, a data splicing module, and a data output module; the data reading module reads weights and feature data from the external storage module in a sequence from left to right and from top to bottom, or receives the output result of the convolution accumulation module; the internal storage module stores the read weights and feature data in sequence to different storage locations according to the convolution step size; the data splicing module is connected to the internal storage module, reads data in sequence, and performs splicing; the data output module outputs the spliced ​​data to the external storage module.

[0027] In one embodiment, the internal storage module is composed of multiple first-in-first-out memories, and the number of first-in-first-out memories used at the same time is equal to the size of the divided sub-blocks. The data of the same sub-block will be stored in different first-in-first-out memories in sequence, and the data at the same position in different sub-blocks will be stored in the same first-in-first-out memory in sequence.

[0028] In one implementation, the data concatenation module calculates the remaining data size in the register through a counter, which is obtained according to the number of multipliers in each convolution unit, the remaining size of the previous data, and the size of the next data.

[0029] The present invention provides a convolution hardware acceleration method and a hardware acceleration circuit, which divide weights and feature data into blocks according to the convolution step size and rearrange them, thereby reducing the width and height of the weights and feature data, increasing the number of channels, improving the utilization rate of multipliers and adders in the convolution array, reducing the number of calculations, and improving the efficiency and speed of convolution calculations. The present invention rearranges the weights and data of subsequent convolution operations through a data conversion module, and writes them into an external storage module to wait for calculation, thereby saving storage resources of the storage module and saving data reading bandwidth during convolution operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of the principle of convolution operation;

[0031] Figure 2 It is a schematic diagram of the existing convolution operation method;

[0032] Figure 3 Schematic diagram of convolution hardware acceleration method;

[0033] Figure 4 It is a schematic diagram of the hardware acceleration circuit structure;

[0034] Figure 5 Schematic diagram of loading weights and feature data in the convolution array;

[0035] Figure 6 It is a structural diagram of the data conversion module;

[0036] Figure 7 Schematic diagram of the data storage order in the internal storage module. DETAILED DESCRIPTION

[0037] The following is a further detailed description of a convolution hardware acceleration method and a hardware acceleration circuit proposed by the present invention in combination with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become clearer according to the following description. It should be noted that the accompanying drawings are all in a very simplified form and are not in precise proportions, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0038] like Figure 1 As shown in the figure, W, H, and C represent the width, height, and number of channels of the feature data cube, respectively; K, R, S, and C represent the number, width, height, and number of channels of the weight cube, respectively; stride_x and stride_y represent the step lengths of the convolution in the horizontal and vertical directions, respectively; the K weight cubes are convolved with the data of size R*S, and a result is obtained, and then the result is moved horizontally by stride_x to perform convolution operation with the second group of data of size R*S; after the horizontal operation is completed, the result is moved vertically by stride_y to perform convolution operation with the next group of data of size R*S.O 、H O 、C O represent the width, height, and number of channels of the convolution result respectively, then there is:

[0039]

[0040]

[0041]

[0042] The number of multiply-accumulators in each convolution unit is c. When performing convolution operations, data of size 1*1*c is sent into the convolution unit for calculation. When C < c, a cube of size 1*1*C will be expanded to 1*1*c, and the expanded part has a value of 0; when C > c, each cube of size 1*1*C will be divided into n cubes of size 1*1*c and a cube of size 1*1*(C - n*c). The size of n needs to satisfy the following formula:

[0043] n*c < C < (n + 1)*c

[0044] A large number of multiplications are involved in the convolution operation process, and the operation speed is slow. Therefore, it is necessary to make the convolution array run as efficiently as possible. When C < c or the calculation data in the convolution array is 1*1*(C - n*c), the multipliers in the convolution array are not fully utilized. Especially when C is very small, the efficiency of the convolution array is very low, which greatly restricts the performance of the convolutional neural network.

[0045] Exemplarily, the weight cube size is 4*4*C, the data cube size is 6*6*C, and c = 4C. The convolution strides in the horizontal and vertical directions are both 2, and the size of the convolution result is 2*2. Existing convolution operation methods such as Figure 2 shown, the weight cube contains a total of 16 cubes of size 1*1*C, and each cube needs to be multiplied by 4 data cubes of size 1*1*C respectively, requiring a total of 64 convolution operations. Only C multipliers are utilized in each convolution unit, and the efficiency of the convolution array is only 25%. Let the output of the convolution be O, the weight be w, and the input data be d, then there is:

[0046]

[0047]

[0048]

[0049]

[0050]

[0051]

[0052]

[0053] A convolution acceleration method proposed in the present invention divides the weight matrix and the data matrix into blocks according to the convolution step size to obtain 4 weight sub-blocks of 2*2*C size and 9 data sub-blocks of 2*2*C size. Each weight sub-block and each data sub-block contains 4 cubes of 1*1*C size respectively.

[0054] ,

[0055] ,

[0056] ,

[0057] , ,

[0058] , ,

[0059] , ,

[0060] Transform the weight matrix and data matrix, and arrange the cubes in each sub-block (including weight sub-block and data sub-block) in the channel direction from left to right and from top to bottom to obtain:

[0061] ,

[0062] ,

[0063] , ,

[0064] , ,

[0065] , ,

[0066] After the conversion, the size of the weight matrix becomes 2*2*4C, the data matrix becomes 3*3*4C, the convolution step becomes 1, and the size of the output matrix is ​​2*2:

[0067]

[0068]

[0069]

[0070]

[0071] Among them, the convolution of each 1*1*4C weight and data needs to be multiplied element by element and then accumulated, and we can get:

[0072]

[0073]

[0074]

[0075]

[0076] From this we can get:

[0077]

[0078] By comparing the formula, we can see , similarly we can get: , , , the output matrix remains unchanged before and after the transformation, and the transformed weight cube contains 4 1*1*4C cubes in total. Each cube needs to be multiplied with 4 1*1*4C data cubes, which only requires 16 convolution operations. The efficiency of the convolution array is 100% after the transformation. The efficiency of the convolution array is increased by four times, and the speed of the convolution operation is also increased by 4 times. The convolution operation after the transformation is as follows Figure 3 shown.

[0079] Based on the above convolution acceleration method, the present invention also proposes a convolution hardware acceleration circuit for implementing the above convolution acceleration method, and its functional block diagram is as follows: Figure 4As shown. Structurally, the convolution hardware acceleration circuit includes a first data reading module 42, a data cache module 43, a data loading module 44, a convolution array 45, a convolution accumulation module 47, and a data transformation module 48. When performing a convolution operation, the first data reading module 42, the data cache module 43, the data loading module 44, the convolution array 45, and the data accumulation module 47 need to work together, and the data must flow through these five modules; however, the data transformation module 48 can choose to receive data from the convolution accumulation module 47, or directly read data from the external storage module 41; the output result of the convolution accumulation module 47 can be sent to the data transformation module 48, or directly output to the external storage module 41.

[0080] The first data reading module 42 reads the feature data in sequence from left to right and from top to bottom. The feature data read each time is the same as the number of multipliers in each convolution unit 46 of the convolution array 45 in the channel direction. When the number of feature data channels is less than the number of multiplication and addition units, padding will be performed in the channel direction, and the value of the padding part is 0; when the number of feature data channels is greater than the number of multiplication and addition units, it will be read in multiple times in sequence from front to back; the first data reading module 42 will store these data in the data cache module 43, and inform the data loading module 44 of the amount of saved data.

[0081] The data cache module 43 can be configured into two parts according to the size of the feature data and the weight, which are used to store the weight and the feature data respectively.

[0082] The data loading module 44 will load the weights and feature data of the convolution in a certain order. When loading weights, the first weight cube of the K weight cubes will be loaded in sequence, and the K cubes will be loaded into the K multiplication and addition units respectively, and then the second weight cube of the different weight cube will be loaded until all weights are loaded; within the same weight cube, the order of loading weights is C→W→H. When loading data, the feature data for the operation with the first weight cube will be loaded in sequence, and these data will be loaded into the K multiplication and addition units at the same time for operation until all data are loaded, and then the feature data for the operation with the next weight cube will be loaded. When the first weight cube is loaded, the feature data is not loaded. After the first weight cube is loaded, the second weight cube and the feature data for the operation with the first weight cube are loaded at the same time. After all the weights and feature data are loaded, the nth weight cube and the feature data for the operation with the n-1th weight cube are loaded. After all the feature data corresponding to a certain weight cube are loaded, a flag will be sent to update the weights in the convolution multiplication and addition unit.

[0083] For example, Figure 5The figure shows that the weight of 3*3*c*4 is convolved with the feature data of 4*4*c, and the output size is 2*2*4. Each weight cube needs to be calculated with the data of 2*2 size. The data loading module 44 will first load K0_0, K1_0, K2_0, K3_0. This part of the weight will be temporarily stored in the register, and then the feature data D00, D01, D10, D11 will be loaded and sent to the convolution unit together with the data for calculation.

[0084] After completing the loading of all weights and data, the data loading module 44 will notify the first data reading module 42 of the amount of released data so as to store the data for subsequent convolution operations.

[0085] The convolution array 45 is composed of k multiplication and addition units, each of which includes c multipliers. Each multiplication and addition unit first loads a 1*1*c cube of K weight cubes, and performs operations with a feature data cube of the same size of 1*1*c. Each multiplier performs a 1*1*1 multiplication operation, and then adds all the multiplier results in a multiplication and addition unit to obtain a partial sum data, and then calculates the convolution of the next feature data and the weight element. During this period, the convolution array still receives weights from the data loading module 44, but these weights will be stored in registers until the flag of the data loading module 44 is received, and then these weights are loaded into the multiplication and addition unit for convolution operation.

[0086] After receiving the convolution partial sum data, the convolution accumulation module 47 will store these data in the fifo in order. When receiving the mark indicating the end of the current weight, it will read out the partial sum in the fifo in turn, accumulate them with the partial sum of the next weight element respectively, and then write them into the fifo in turn. After completing the accumulation of the partial sums between all weight elements, the final convolution result is obtained.

[0087] Figure 6 The data transformation module 48 is shown, which includes a second data reading module 63, an internal storage module 64, a data splicing module 66, a data output module 67, etc. Among them, the second data reading module 63 can receive the output result of the convolution accumulation module 47, or directly read the weight and feature data from the external storage module 41, and store the data in the internal storage module 64; the internal storage module 64 stores the read weight and feature data in different storage locations in sequence according to the convolution step; the data splicing module 66 can read the data of the internal storage module 64 in a certain order and splice it; the data output module 67 is responsible for outputting the spliced ​​data to the external storage module 41.

[0088] The internal storage module 64 is composed of multiple fifos (first-in-first-off, first-in first-out storage unit 65). The number of fifos used at the same time is equal to the size of the divided sub-blocks. The data of the same sub-block will be stored in different fifos in sequence, and the data at the same position in different sub-blocks will be stored in the same fifo in sequence.

[0089] For example, in a convolution with a step size of 2, the first data of the first row read is stored in fifo0, the second data of the first row is stored in fifo1, the third data of the first row is stored in fifo0, and the fourth data of the first row is stored in fifo1; the first data of the second row is stored in fifo2, the second data of the second row is stored in fifo3, the third data of the second row is stored in fifo2, and the fourth data of the second row is stored in fifo3; the storage order of the third row of data is the same as that of the first row, the storage order of the fourth row of data is the same as that of the second row, and so on. Figure 7 shown.

[0090] The data splicing module 66 will read the data in the fifo in sequence according to the sequence number of the fifo, and splice the read data into a size of 1*1*c in sequence. The spliced ​​part will be written into the memory, and the unspliced ​​part will be stored in the register, waiting for the next splicing.

[0091] Furthermore, the data splicing module 66 calculates the size of the remaining data in the register through a counter, which is obtained according to the size of the multiplication unit, the remaining size of the previous data and the size of the next data. The counter is used to calculate the number of bytes of the data to be spliced ​​stored in the register, and by comparing the count value with the number of multiplication and addition units c of the convolution array, it is determined whether the next batch of data needs to be read from the fifo.

[0092] The above description is only a description of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention. Any changes or modifications made by a person skilled in the art in the field of the present invention based on the above disclosure shall fall within the scope of protection of the claims.

Claims

1. A convolution hardware acceleration method, characterized in that: The following steps are involved: The weight cube and feature data cube are divided into multiple sub-blocks according to the step length of the convolution operation in two directions, and the width and height of each sub-block are equal to the step length of the convolution kernel in two directions; Performing a first rearrangement and a second rearrangement on the data blocks in each sub-block; According to the number of multiplication and addition units, the rearranged data matrix and the weight matrix are divided into blocks along the channel direction to obtain multiple sub-matrices; Perform convolution operation on each sub-matrix to obtain multiple parts and matrices; accumulate the elements at corresponding positions of the multiple parts and matrices to obtain the convolution operation result; The first rearrangement is: arranging the data blocks of each row in order from left to right in the channel direction; the second rearrangement is: arranging the data blocks in each sub-block in order from top to bottom in the channel direction, to obtain the rearranged data cube and weight cube; When the step length of the convolution operation in both directions is 1, the weight cube and the data cube are not rearranged; when the step length of the convolution operation in any direction is not 1, the step length of the rearranged convolution operation is 1; When the width or height of the weight cube or feature data cube cannot be divided by the convolution step, the edges of the weight cube and feature data cube are padded so that their width and height are both divisible by the convolution step, and the values ​​of the padded parts are all 0.

2. A hardware acceleration circuit, characterized in that: The hardware acceleration circuit is connected to an external storage module, and the external storage module is used to store multiple feature cubes, multiple convolution kernels and multiple operation results; The hardware acceleration circuit comprises: A data reading module, which reads the convolution kernel and feature data required for this convolution operation from the external storage module; A data cache module, caching the convolution kernel and feature data read from the external storage module; A data loading module, which reads the convolution kernel and feature data from the data cache module in sequence and sends them to the convolution unit; A convolution array, composed of convolution units, receives the convolution kernel and data of the data loading module, and performs multiplication and addition operations on the convolution kernel and data to obtain a plurality of convolution partial sums; The convolution accumulation module accumulates the sum of the convolution parts to obtain the final convolution result, and sends the convolution result to the data transformation module or outputs it to the external storage module; The data transformation module transforms the data and weights of the next convolution operation and outputs the results to the external memory; The data reading module reads the feature data in sequence from left to right and from top to bottom. The feature data read each time is the same as the number of convolution units of the convolution array in the channel direction. When the number of feature data channels is less than the number of convolution units, padding will be performed in the channel direction, and the value of the padding part is 0; when the number of feature data channels is greater than the number of convolution units, it will be read multiple times in order from front to back; The data cache module is configured into two parts according to the size of the feature data and the weight, respectively used to store the weight and the feature data; The data loading module loads a weight element and the feature data that needs to be convolved with the weight element in a sequence from left to right, top to bottom, and front to back according to the size of the feature data, the size of the weight, and the size of the convolution result, and sends a flag signal to inform the operation module whether all the data to be convolved with the current weight element has been loaded.

3. The hardware acceleration circuit according to claim 2, characterized in that: The convolution array includes multiple convolution units, each of which is composed of multiple multipliers and adders. Each multiplier multiplies the weight and feature data of a channel, and then adds the products of all multipliers to obtain a partial sum data, and then calculates the convolution of the next feature data and the weight element.

4. The hardware acceleration circuit according to claim 2, characterized in that: The convolution accumulation module is used to cache the convolution partial sum of the same weight element and accumulate it with the partial sum of the next weight element to obtain the final convolution result.

5. The hardware acceleration circuit according to claim 2, characterized in that: The data transformation module includes a data reading module, an internal storage module, a data splicing module, and a data output module; the data reading module reads weights and feature data from the external storage module in a sequence from left to right and from top to bottom, or receives the output result of the convolution accumulation module; the internal storage module stores the read weights and feature data in sequence to different storage locations according to the convolution step size; the data splicing module is connected to the internal storage module, reads data in sequence, and performs splicing; the data output module outputs the spliced ​​data to the external storage module.

6. The hardware acceleration circuit according to claim 5, characterized in that: The internal storage module is composed of multiple first-in-first-out memories, and the number of first-in-first-out memories used at the same time is equal to the size of the divided sub-blocks. The data of the same sub-block will be stored in different first-in-first-out memories in sequence, and the data at the same position in different sub-blocks will be stored in the same first-in-first-out memory in sequence.

7. The hardware acceleration circuit according to claim 5, characterized in that: The data splicing module calculates the remaining data size in the register through a counter, which is obtained according to the number of multipliers in each convolution unit, the remaining size of the previous data and the size of the next data.

Images