A method to improve the efficiency of hardware circuits performing single-batch two-dimensional convolution calculations
By parallel computation of 2D convolution results in the MAC array and quadruple loop control of sliding window movement, the problem of low computation efficiency of single-batch 2D convolution is solved, thereby improving the computational efficiency and data processing capability of the hardware circuit.
Patent Information
- Application Number
- CN202311712355.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-13
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2043-12-13
AI Technical Summary
Existing neural network hardware computing devices are inefficient when processing single-batch two-dimensional convolution calculations, and cannot effectively utilize computing power.
By implementing parallel computation of two-dimensional convolution results in different regions within the MAC array, the on-chip memory is divided into three main blocks to provide data, and the sliding window is moved on the two-dimensional feature map through a quadruple loop control, thus achieving parallel computation of all processing units.
It improves the efficiency of single-batch 2D convolution computation, enhances the utilization of the MAC array, realizes multi-region parallel computation of 2D feature maps, adapts to convolution kernels of different sizes and shapes, and reduces data rearrangement steps.
Smart Images

Figure CN117592522B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to two-dimensional convolution calculation, and in particular to a method for improving the efficiency of hardware circuits in performing single-batch two-dimensional convolution calculations. Background Art
[0002] In existing neural network hardware computing devices, the main internal computing unit is called MAC (multiply-accumulate), and the MAC array is generally arranged in a regular rectangle in the form of a two-dimensional computing array. A general MAC array includes several PEs (Processing Elements), and each PE contains one or more MACs. From the perspective of rows, the MAC array can be seen as including several rows of PEs, and each row in the MAC array contains the same number of PEs; from the perspective of columns, the MAC array can also be seen as including several columns of PEs, and each column in the MAC array contains the same number of PEs.
[0003] When processing two-dimensional convolution calculations, parallel processing is generally performed at two levels: output channel and batch. At the output channel level, since the calculation processes of different output channels are completely unrelated, they can be processed in parallel. One method for parallel calculation of output channel calculation results is: the input feature map is broadcast to all PEs in a row in the row direction, and each column of PE calculates an output channel calculation result to achieve data reuse of the input feature map. At the batch level, since the calculations of different batches are also completely unrelated, they can also be processed in parallel. One method for parallel calculation of calculation results of different batches is: the input feature map of each batch is broadcast from the row direction to all PEs in each row, and different rows of PE process different batches of data. The weight data is broadcast from the column direction to each column of PE, and two-dimensional convolution calculations are performed with the input feature map of each batch to achieve data reuse of weights.
[0004] The above method is a common parallel processing method used by existing neural network hardware computing devices to process two-dimensional convolution calculations. Generally, the two methods are used at the same time. However, the above method is only effective when processing multiple batches of input feature maps at the same time. When a single batch of input feature maps needs to be processed, the above method can only utilize one row of PE, which greatly affects the computing power of the neural network hardware computing device. Summary of the invention
[0005] 1. Technical issues to be resolved
[0006] In view of the above-mentioned shortcomings of the prior art, the present invention provides a method for improving the efficiency of hardware circuits in performing single-batch two-dimensional convolution calculations, which can effectively overcome the defect of low efficiency in processing single-batch two-dimensional convolution calculations in the prior art.
[0007] (II) Technical solution
[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:
[0009] A method for improving the efficiency of hardware circuits performing single-batch two-dimensional convolution calculations. Based on the characteristic that the two-dimensional convolution results of elements in different regions in the plane of a two-dimensional feature map can be calculated in parallel, all processing units PE in a MAC array can calculate the two-dimensional convolution results of each region in the two-dimensional feature map in parallel when processing a single-batch two-dimensional convolution calculation.
[0010] Preferably, the MAC array is provided with data by an on-chip memory, and the on-chip memory is divided into three blocks, which are called blocks. One block of memory provides data to the MAC array from the row direction, which is recorded as input_buffer; one block of memory provides data to the MAC array from the column direction, which is recorded as weight_buffer; one block of memory receives the calculation result of the MAC array, which is recorded as output_buffer;
[0011] Each memory block is divided into multiple small blocks, and each small memory block transmits data to a row of processing units PE or a column of processing units PE. These small memory blocks can transmit data to the MAC array at the same time, which is recorded as a bank.
[0012] Preferably, the data of each bank of the input_buffer is broadcast to each row of the MAC array, the data of each bank of the weight_buffer is broadcast to each column of the MAC array, each of the processing units PE performs multiplication and accumulation calculations of the input data in the row direction and the input data in the column direction, and the calculation results of the processing units PE in a row are spliced and output to the corresponding bank of the output_buffer;
[0013] Among them, the number of banks of input_buffer and output_buffer is the same as the number of rows R of processing units PE in the MAC array, and the number of banks of weight_buffer is the same as the number of columns C of processing units PE in the MAC array.
[0014] Preferably, when processing a single batch of two-dimensional convolution calculations, the data of each region in the two-dimensional feature map is stored in each bank of input_buffer, the convolution kernel weight for each output channel is stored in each bank of weight_buffer, and the two-dimensional convolution result of each region is stored in each bank of output_buffer;
[0015] The two-dimensional feature map is divided into multiple regions. The data of each region is stored in each bank of input_buffer respectively, and broadcasted to all processing units PE in each row of the MAC array respectively. Each processing unit PE in each row calculates the two-dimensional convolution result of one region respectively, so that all processing units PE in the MAC array can calculate the two-dimensional convolution results of each region in the two-dimensional feature map in parallel when processing a single batch of two-dimensional convolution calculations;
[0016] When the calculation results are output, the calculation results of each row of processing units PE are spliced and output to the corresponding bank of output_buffer to achieve the output feature Figure 1 The calculation results of the regions are still in one bank in the output_buffer, so when the two-dimensional convolution calculation is performed on the calculation results again, there is no need to rearrange the data;
[0017] Among them, when filling the regional data of the two-dimensional feature map in each bank of input_buffer, they are arranged in the order of channel, column, and row.
[0018] Preferably, the two-dimensional feature map is evenly divided into R blocks along the column direction. The width of the two-dimensional feature map, that is, the number of columns of the two-dimensional feature map, is FC, and the number of banks in the input_buffer is R. Then each bank of the input_buffer can store at most Column data, and only the first Bank storage Column data, the next bank stores the rest Column data, the following banks are not filled with data;
[0019] in, To round up, To round down.
[0020] Preferably, when processing a single batch of two-dimensional convolution calculations, use uop to specify the address of taking data from input_buffer and weight_buffer, and the address of storing data in output_buffer;
[0021] Among them, one uop is used to specify the address of taking data from input_buffer and weight_buffer during a calculation, as well as the address stored in output_buffer. Multiple uops are combined to complete the calculation of a two-dimensional convolution sliding window. Multiple uops are stored in uop_buffer, and the addresses specified by multiple uops may not be continuous.
[0022] Preferably, based on the characteristic that the sliding window moves with a fixed step size in the row direction or column direction during the two-dimensional convolution process, a quadruple loop is used to control the movement of the two-dimensional convolution sliding window on the two-dimensional feature map.
[0023] Preferably, the adopting of a quadruple loop to control the movement of the two-dimensional convolution sliding window on the two-dimensional feature map includes:
[0024] The innermost loop traverses multiple input channel data corresponding to the same two-dimensional feature map position. The second inner loop flexibly specifies the address of each element in the sliding window participating in the convolution calculation through uop. The number of uops in the second inner loop is the same as the number of elements in the two-dimensional convolution sliding window. The innermost loop and the second inner loop complete the two-dimensional convolution result calculation of an output feature map position.
[0025] The outermost two loops control traversal of the two-dimensional feature map in the row direction or column direction respectively;
[0026] The above traversal method targets the R two-dimensional feature map regions stored in each bank of input_buffer, and controls input_buffer to retrieve the same two-dimensional feature map region in each bank in the same manner.
[0027] Preferably, the innermost loop traverses multiple input channel data corresponding to the same two-dimensional feature map position, the second inner loop flexibly specifies the address of each element in the sliding window participating in the convolution calculation through uop, and the outermost two loops respectively control the traversal of the two-dimensional feature map from the row direction or the column direction, including:
[0028] Use the innermost loop count inner_loop_cnt and the innermost loop step inner_input_stride to specify multiple input channel data addresses corresponding to the same two-dimensional feature map position;
[0029] Use multiple uops to flexibly specify the address of each element in the sliding window participating in the convolution calculation;
[0030] Use the outer_loop_cnt, the Outer_input_stride, the Top_loop_cnt, and the Top_Input_stride to control the traversal of the two-dimensional feature map in the row or column direction.
[0031] Preferably, the use of the sub-outer loop number outer_loop_cnt, the sub-outer loop step Outer_input_stride, the outermost loop number top_loop_cnt, and the outermost loop step Top_input_stride to control traversing the two-dimensional feature map in the row direction or the column direction includes:
[0032] Use the outer_loop_cnt and the Outer_input_stride of the outer loop to control the movement of the two-dimensional convolution sliding window in the row direction, and use the top_loop_cnt and the Top_Input_stride of the outer loop to control the movement of the two-dimensional convolution sliding window in the column direction;
[0033] Alternatively, use the outermost loop number top_loop_cnt and the outermost loop step size Top_Input_stride to control the movement of the two-dimensional convolution sliding window in the row direction, and use the second outer loop number outer_loop_cnt and the second outer loop step size Outer_input_stride to control the movement of the two-dimensional convolution sliding window in the column direction.
[0034] (III) Beneficial effects
[0035] Compared with the prior art, the method provided by the present invention for improving the efficiency of hardware circuits in performing single-batch two-dimensional convolution calculations has the following beneficial effects:
[0036] 1) This application enables the neural network hardware computing device to simultaneously calculate the two-dimensional convolution results of each region in the two-dimensional feature map when processing a single batch of two-dimensional convolution calculations, thereby improving the parallelism of data processing by improving the utilization rate of the MAC array, and effectively improving the efficiency of processing a single batch of two-dimensional convolution calculations;
[0037] 2) This application uses the outermost two layers of loops to traverse multiple regions of the two-dimensional feature map at the same time, and efficiently realizes the parallel calculation of the two-dimensional convolution results of multiple regions of the two-dimensional feature map. If the two-dimensional feature map is completely stored in input_buffer, and the weight is completely stored in weight_buffer, the entire two-dimensional convolution calculation can be completed with one instruction;
[0038] 3) This application uses uop to specify the position of each convolution kernel corresponding to the input feature map point in input_buffer, which is used to flexibly specify the position of each element in the two-dimensional convolution sliding window with discontinuous storage positions, and can flexibly adapt to convolution kernels of different sizes and shapes, as well as convolution calculations with different characteristics such as dilated convolution and deconvolution;
[0039] 4) A row of processing units PE executes two-dimensional features Figure 1 The two-dimensional convolution calculation of each region is performed. Each processing unit PE performs the two-dimensional convolution calculation for different output channels. The two-dimensional convolution results of different output channels are spliced and output to the corresponding bank of output_buffer, so that one bank of output_buffer can store output features. Figure 1 The effect of the calculation results of each region is that when the two-dimensional convolution calculation is performed on the calculation results again later, there is no need to rearrange the data, and the next two-dimensional convolution calculation can be started directly according to the method proposed in this application. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0041] Figure 1 A schematic diagram of data transmission between an on-chip memory and a MAC array in the present invention;
[0042] Figure 2 A schematic diagram of filling the regional data of the two-dimensional feature map in the bank of input_buffer in the present invention;
[0043] Figure 3 is a multi-channel two-dimensional feature map in the present invention;
[0044] Figure 4 A weight map for the multi-input channel and multi-output channel two-dimensional convolution calculation in the present invention;
[0045] Figure 5 for Figure 3 The two-dimensional feature graph shown is arranged in the order of channel, column, and row priorities to obtain the data layout of the first bank in input_buffer;
[0046] Figure 6 This is a schematic diagram of the first uop specifying to fetch data from input_buffer for a 3*3 convolution kernel in the present invention;
[0047] Figure 7 This is a schematic diagram of the second uop specifying to fetch data from input_buffer for a 3*3 convolution kernel in the present invention;
[0048] Figure 8 This is a schematic diagram of the third uop specifying to fetch data from input_buffer for a 3*3 convolution kernel in the present invention;
[0049] Fig. 9 This is a schematic diagram of the fourth uop specifying to fetch data from input_buffer for a 3*3 convolution kernel in the present invention;
[0050] Fig.10This is a schematic diagram of the ninth uop specifying to fetch data from input_buffer for a 3*3 convolution kernel in the present invention;
[0051] Fig.11 A schematic diagram of the outermost two layers of loop control traversing a two-dimensional feature graph in the present invention;
[0052] Fig.12 Another schematic diagram of the outermost two layers of loop control traversing the two-dimensional feature graph in the present invention. DETAILED DESCRIPTION
[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0054] A method for improving the efficiency of hardware circuits performing single-batch two-dimensional convolution calculations. Based on the characteristic that the two-dimensional convolution results of elements in different regions in the plane of a two-dimensional feature map can be calculated in parallel, all processing units PE in a MAC array can calculate the two-dimensional convolution results of each region in the two-dimensional feature map in parallel when processing a single-batch two-dimensional convolution calculation.
[0055] like Figure 1 As shown, the MAC array is provided with data by the on-chip memory, which is divided into three blocks, called blocks. One block of memory provides data to the MAC array from the row direction, denoted as input_buffer; one block of memory provides data to the MAC array from the column direction, denoted as weight_buffer; one block of memory receives the calculation results of the MAC array, denoted as output_buffer;
[0056] Each memory block is divided into multiple small blocks, and each small memory block transmits data to a row of processing units PE or a column of processing units PE. These small memory blocks can transmit data to the MAC array at the same time, which is recorded as a bank.
[0057] The data of each bank of input_buffer is broadcast to each row of the MAC array, and the data of each bank of weight_buffer is broadcast to each column of the MAC array. Each processing unit PE performs multiplication and accumulation calculations of the input data in the row direction and the input data in the column direction. The calculation results of a row of processing units PE are spliced and output to the corresponding bank of output_buffer;
[0058] Among them, the number of banks of input_buffer and output_buffer is the same as the number of rows R of processing units PE in the MAC array, and the number of banks of weight_buffer is the same as the number of columns C of processing units PE in the MAC array.
[0059] As pointed out in the background technology, a specific method for parallel computing output channel calculation results is:
[0060] When processing a single batch of two-dimensional convolution calculations, the input feature map is stored in input_buffer, the convolution kernel weight is stored in weight_buffer, and the two-dimensional convolution result is stored in output_buffer;
[0061] Among them, in weight_buffer, each bank stores the convolution kernel weight of an output channel, and these weight data are broadcast to all processing units PE in each column of the MAC array;
[0062] The input feature map is broadcast to all processing elements PE in a row of the MAC array. Each processing element PE in a row calculates the two-dimensional convolution result of an output channel, realizing parallel computing at the output channel level.
[0063] As pointed out in the background technology, a specific method for parallel computing of different batches of calculation results is:
[0064] When processing multiple batches of two-dimensional convolution calculations, the input feature maps of each batch are stored in each bank of input_buffer, the convolution kernel weights are stored in weight_buffer, and the two-dimensional convolution results of each batch are stored in output_buffer;
[0065] Among them, in weight_buffer, each bank stores the convolution kernel weight of an output channel, and these weight data are broadcast to all processing units PE in each column of the MAC array;
[0066] Each batch of input feature maps is broadcast to all processing units PE in each row of the MAC array. Each row of processing units PE calculates the two-dimensional convolution results of a batch to achieve batch-level parallel computing.
[0067] Generally, the above two methods mentioned in the background technology are used simultaneously to achieve simultaneous parallel calculations at the output channel level and the batch level, but this method cannot efficiently handle single-batch two-dimensional convolution calculations.
[0068] When processing a single batch of two-dimensional convolution calculations, the technical solution of the present application stores the data of each region in the two-dimensional feature map in each bank of input_buffer, the convolution kernel weight for each output channel in each bank of weight_buffer, and the two-dimensional convolution result of each region in each bank of output_buffer.
[0069] The two-dimensional feature map is divided into multiple regions. The data of each region is stored in each bank of input_buffer respectively, and broadcasted to all processing units PE in each row of the MAC array respectively. Each processing unit PE in each row calculates the two-dimensional convolution result of one region respectively, so that all processing units PE in the MAC array can calculate the two-dimensional convolution results of each region in the two-dimensional feature map in parallel when processing a single batch of two-dimensional convolution calculations;
[0070] When the calculation results are output, the calculation results of each row of processing units PE are spliced and output to the corresponding bank of output_buffer to achieve the output feature Figure 1 The calculation results of the regions are still in one bank in the output_buffer, so when the two-dimensional convolution calculation is performed on the calculation results again, there is no need to rearrange the data;
[0071] Among them, when filling the regional data of the two-dimensional feature map in each bank of input_buffer, they are arranged in the order of channel, column, and row, such as Figure 3-5 shown.
[0072] A bank of weight_buffer stores the convolution kernel weight for an output channel, realizing parallel calculation of multiple output channels, and the calculation results of a row of processing units PE are spliced and output to the corresponding bank of output_buffer, realizing that a bank of output_buffer stores output features Figure 1 The effect of the calculation results of each region is that when the two-dimensional convolution calculation is performed on the calculation results again later, there is no need to rearrange the data, and the next two-dimensional convolution calculation can be started directly according to the method proposed in this application.
[0073] like Figure 2 As shown, the two-dimensional feature map is evenly divided into R blocks along the column direction. The width of the two-dimensional feature map, that is, the number of columns of the two-dimensional feature map, is FC, and the number of banks in the input_buffer is R. Then each bank of the input_buffer can store at most Column data, and only the first Bank storage Column data, the next bank stores the rest Column data, the following banks are not filled with data;
[0074] in, To round up, To round down.
[0075] Figure 2 In , N represents the number of columns of the input feature map stored in a bank, and N is obtained by dividing the number of input feature map columns FC by the number of banks in input_buffer, that is, CI represents the number of input channels that can be stored in one depth position of a bank of input_buffer.
[0076] When processing a single batch of two-dimensional convolution calculations, the technical solution of the present application uses uop to specify the address of taking data from input_buffer and weight_buffer, and the address of storing data in output_buffer;
[0077] Among them, one uop is used to specify the address of taking data from input_buffer and weight_buffer during a calculation, as well as the address stored in output_buffer. Multiple uops are combined to complete the calculation of a two-dimensional convolution sliding window. Multiple uops are stored in uop_buffer, and the addresses specified by multiple uops may not be continuous.
[0078] Since the data placement proposed in this application is carried out according to the three priorities of channel, column and row, in the two-dimensional feature map plane, one column is placed before the next column. The addresses of the data in the same two-dimensional convolution sliding window in the input_buffer may not be continuous. By using uop to specify the address, the address of the data can be flexibly selected.
[0079] Taking a 3*3 size ordinary convolution kernel as an example, multiple uops can be used to complete the calculation of a two-dimensional convolution sliding window. The way to get the input_buffer and weight_buffer under each uop is as follows Figure 6-9 As shown, the rest of the uops are deduced in the same way until the ninth uop, such as Fig.10 shown. Figure 6-10 In the figure, B represents the output port width of each bank (in units of 8 bits), and the single data width in the figure is also 8 bits, so the data of B input channels can be read in one cycle. Note that at this time, in the two-dimensional convolution sliding window, only the same column of data is placed continuously in the input_buffer, and the placement of different columns of data in the input_buffer is not continuous and may be far apart, but these discontinuous input feature map data can be flexibly specified by combining multiple uops.
[0080] In addition, for dilated convolution and deconvolution, even if the same column of data is in the two-dimensional convolution sliding window, its position in the two-dimensional feature map plane is discontinuous, and its placement in the input_buffer is also discontinuous. The input feature map data in these sliding windows can also rely on the uop in the uop_buffer to flexibly specify its address in the input_buffer.
[0081] At the same time, the technical solution of the present application is based on the characteristic that the sliding window moves with a fixed step size in the row direction or column direction during the two-dimensional convolution process, and uses one instruction to control a quadruple loop to control the movement of the two-dimensional convolution sliding window on the two-dimensional feature map.
[0082] This application proposes to use an instruction to control the hardware state machine to execute the following pseudo code to drive the MAC array to complete the entire two-dimensional convolution calculation:
[0083]
[0084]
[0085] Specifically, a quadruple loop is used to control the movement of the two-dimensional convolution sliding window on the two-dimensional feature map, including:
[0086] The innermost loop traverses multiple input channel data corresponding to the same two-dimensional feature map position. The second inner loop flexibly specifies the address of each element in the sliding window participating in the convolution calculation through uop. The number of uops in the second inner loop is the same as the number of elements in the two-dimensional convolution sliding window. The innermost loop and the second inner loop complete the two-dimensional convolution result calculation of an output feature map position.
[0087] The outermost two loops control traversal of the two-dimensional feature map in the row direction or column direction respectively;
[0088] The above traversal method targets the R two-dimensional feature map regions stored in each bank of input_buffer, and controls input_buffer to retrieve the same two-dimensional feature map region in each bank in the same manner.
[0089] Specifically, the innermost loop traverses multiple input channel data corresponding to the same two-dimensional feature map position, the second inner loop flexibly specifies the address of each element in the sliding window participating in the convolution calculation through uop, and the outermost two loops control the traversal of the two-dimensional feature map from the row direction or column direction, including:
[0090] Use the innermost loop count inner_loop_cnt and the innermost loop step inner_input_stride to specify multiple input channel data addresses corresponding to the same two-dimensional feature map position;
[0091] Use multiple uops to flexibly specify the address of each element in the sliding window participating in the convolution calculation;
[0092] Use the outer_loop_cnt, the Outer_input_stride, the Top_loop_cnt, and the Top_Input_stride to control the traversal of the two-dimensional feature map in the row or column direction.
[0093] Specifically, the outer loop number outer_loop_cnt, the outer loop step size Outer_input_stride, the outermost loop number top_loop_cnt, and the outermost loop step size Top_Input_stride are used to control traversal of the two-dimensional feature map from the row direction or the column direction, including:
[0094] Use the outer_loop_cnt and the outer_input_stride to control the movement of the two-dimensional convolution sliding window in the row direction, and use the outer_loop_cnt and the outer_input_stride to control the movement of the two-dimensional convolution sliding window in the column direction. Fig.11 As shown;
[0095] Or use the outermost loop count top_loop_cnt and the outermost loop step size Top_Input_stride to control the movement of the two-dimensional convolution sliding window in the row direction, and use the second outer loop count outer_loop_cnt and the second outer loop step size Outer_input_stride to control the movement of the two-dimensional convolution sliding window in the column direction, such as Fig.12 shown.
[0096] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for improving the efficiency of hardware circuits performing single-batch two-dimensional convolution calculations, characterized in that: When processing a single batch of two-dimensional convolution calculations, the data of each region in the two-dimensional feature map is stored in each bank of input_buffer, the convolution kernel weights for each output channel are stored in each bank of weight_buffer, and the two-dimensional convolution results of each region are stored in each bank of output_buffer; The two-dimensional feature map is divided into multiple regions. The data of each region is stored in each bank of input_buffer respectively, and broadcasted to all processing units PE in each row of the MAC array respectively. Each processing unit PE in each row calculates the two-dimensional convolution result of one region respectively, so that all processing units PE in the MAC array can calculate the two-dimensional convolution results of each region in the two-dimensional feature map in parallel when processing a single batch of two-dimensional convolution calculations; When the calculation results are output, the calculation results of each row of processing units PE are spliced and output to the corresponding bank of output_buffer, so that the calculation results of one area of the output feature map are still in one bank in output_buffer. When performing two-dimensional convolution calculation on the calculation results again, there is no need to rearrange the data; Among them, when filling the regional data of the two-dimensional feature map in each bank of input_buffer, it is placed in the order of channel, column, and row; The MAC array is provided with data by the on-chip memory, which is divided into three blocks, which are called blocks. One block of memory provides data to the MAC array from the row direction, which is recorded as input_buffer; one block of memory provides data to the MAC array from the column direction, which is recorded as weight_buffer; one block of memory receives the calculation results of the MAC array, which is recorded as output_buffer; Each memory block is divided into multiple small blocks, each small memory block transmits data to a row of processing units PE or a column of processing units PE, and these small memory blocks transmit data to the MAC array at the same time, which is recorded as a bank; The data of each bank of the input_buffer is broadcast to each row of the MAC array, and the data of each bank of the weight_buffer is broadcast to each column of the MAC array. Each of the processing units PE performs multiplication and accumulation calculations of the input data in the row direction and the input data in the column direction. The calculation results of the processing units PE in a row are spliced and output to the corresponding bank of the output_buffer; Among them, the number of banks of input_buffer and output_buffer is the same as the number of rows R of processing units PE in the MAC array, and the number of banks of weight_buffer is the same as the number of columns C of processing units PE in the MAC array.
2. The method for improving the efficiency of hardware circuits performing single-batch two-dimensional convolution calculations according to claim 1, characterized in that: Divide the two-dimensional feature map into R blocks along the column direction. Let the width of the two-dimensional feature map, that is, the number of columns of the two-dimensional feature map, be FC, and the number of banks in the input_buffer be R. Then each bank of the input_buffer can store at most Column data, and only the first Bank storage Column data, the next bank stores the rest Column data, the following banks are not filled with data; in, To round up, To round down.
3. The method for improving the efficiency of hardware circuits performing single-batch two-dimensional convolution calculations according to claim 1, characterized in that: When processing a single batch of two-dimensional convolution calculations, use uop to specify the address of taking data from input_buffer and weight_buffer, and the address stored in output_buffer; Among them, one uop is used to specify the address of taking data from input_buffer and weight_buffer during a calculation, as well as the address stored in output_buffer. Multiple uops are combined to complete the calculation of a two-dimensional convolution sliding window. Multiple uops are stored in uop_buffer, and the addresses specified by multiple uops are not continuous.
4. The method for improving the efficiency of hardware circuits performing single-batch two-dimensional convolution calculations according to claim 3, characterized in that: Based on the characteristic that the sliding window moves with a fixed step size in the row or column direction during the two-dimensional convolution process, a quadruple loop is used to control the movement of the two-dimensional convolution sliding window on the two-dimensional feature map.
5. The method for improving the efficiency of hardware circuits performing single-batch two-dimensional convolution calculations according to claim 4, characterized in that: The use of a quadruple loop to control the movement of the two-dimensional convolution sliding window on the two-dimensional feature map includes: The innermost loop traverses multiple input channel data corresponding to the same two-dimensional feature map position. The second inner loop flexibly specifies the address of each element in the sliding window participating in the convolution calculation through uop. The number of uops in the second inner loop is the same as the number of elements in the two-dimensional convolution sliding window. The innermost loop and the second inner loop complete the two-dimensional convolution result calculation of an output feature map position. The outermost two loops control traversal of the two-dimensional feature map in the row direction or column direction respectively; The above traversal method targets the R two-dimensional feature map regions stored in each bank of input_buffer, and controls input_buffer to retrieve the same two-dimensional feature map region in each bank in the same manner.
6. The method for improving the efficiency of hardware circuits performing single-batch two-dimensional convolution calculations according to claim 5, characterized in that: The innermost loop traverses multiple input channel data corresponding to the same two-dimensional feature map position, the second inner loop flexibly specifies the address of each element in the sliding window participating in the convolution calculation through uop, and the outermost two loops respectively control the traversal of the two-dimensional feature map from the row direction or the column direction, including: Use the innermost loop count inner_loop_cnt and the innermost loop step inner_input_stride to specify multiple input channel data addresses corresponding to the same two-dimensional feature map position; Use multiple uops to flexibly specify the address of each element in the sliding window participating in the convolution calculation; Use the outer_loop_cnt, the Outer_input_stride, the Top_loop_cnt, and the Top_Input_stride to control the traversal of the two-dimensional feature map in the row or column direction.
7. The method for improving the efficiency of hardware circuits performing single-batch two-dimensional convolution calculations according to claim 6, characterized in that: The method uses the sub-outer loop number outer_loop_cnt, the sub-outer loop step Outer_input_stride, the outermost loop number top_loop_cnt, and the outermost loop step Top_input_stride to control traversing the two-dimensional feature map in the row direction or the column direction, including: Use the outer_loop_cnt and the Outer_input_stride of the outer loop to control the movement of the two-dimensional convolution sliding window in the row direction, and use the top_loop_cnt and the Top_Input_stride of the outer loop to control the movement of the two-dimensional convolution sliding window in the column direction; Alternatively, use the outermost loop number top_loop_cnt and the outermost loop step size Top_Input_stride to control the movement of the two-dimensional convolution sliding window in the row direction, and use the second outer loop number outer_loop_cnt and the second outer loop step size Outer_input_stride to control the movement of the two-dimensional convolution sliding window in the column direction.
Citation Information
Patent Citations
Apparatus and method for processing reciprocal square root operations
CN109947475A
Convolution operation circuit and operation method thereof
CN113869498A
Two-dimensional convolution calculation hardware framework and pipelined two-dimensional convolution calculation method based on two-dimensional convolution calculation hardware framework
CN115545157A
Convolution operation circuit and method, neural network accelerator and electronic equipment
CN115759212A
Safety protection method and system of neural network, accelerator and chip
CN116150784A
