Universal neural network acceleration method based on three-dimensional loop expansion

Abstract

The invention belongs to the technical field of neural network acceleration and processing unit design, and provides a universal neural network acceleration method based on three-dimensional loop expansion. The method comprises the following steps: receiving an instruction and sending decoded instruction information to configure each control module and a core computing unit; enabling after receiving a start signal; reading input data and weight data; reading the input data and the weight from the zero-level buffer area and writing the input data and the weight into a core calculation unit; the read-write control module is used for continuously generating pre-read enable signals to respectively control part and data read-write, quantization weight and output data by the core calculation unit; and the output data read-write control module reads the output data from the core calculation unit and writes the output data into the first-stage buffer area, and after all write-in work is completed, a calculation ending signal is sent. According to the method, parallel computing is carried out in the three dimensions of the input channel, the output channel and the output feature map, higher computing performance and data reuse rate are achieved, and power consumption caused by memory access is reduced.

Description

Technical Field

[0001] The invention belongs to the technical field of neural network acceleration and processing unit design, and proposes a universal neural network acceleration method based on three-dimensional loop unrolling. Background Art

[0002] The excellent performance of deep convolutional neural networks in many fields such as computer vision and speech recognition has made it the mainstream direction of machine learning today. However, with the increasing complexity and computing requirements of deep networks, traditional central processing units (CPUs) and graphics processing units (GPUs) face efficiency and performance bottlenecks when processing these computationally intensive tasks. Therefore, heterogeneous computing platforms such as FPGAs and ASICs are widely used in the design of hardware accelerators for deep networks.

[0003] As the main operation in neural networks, convolution is a key factor that significantly affects the efficiency and performance of accelerators. However, the limited computing resources and storage capacity on the hardware platform make the acceleration of convolution calculations a complex multi-dimensional optimization problem. In addition, the energy consumption costs associated with the large amount of data movement and memory access involved in convolution operations even exceed the energy consumption of calculations. Based on the above problems, achieving energy-efficient hardware acceleration of convolutional neural networks requires maximizing data reuse and resource utilization while minimizing data communication, which places high demands on the optimization and acceleration of convolution.

[0004] In response to the above problems, this application is dedicated to proposing a general neural network acceleration method based on three-dimensional loop unrolling. Summary of the invention

[0005] The purpose of the present invention is to address the problems of low computational efficiency and high memory access energy consumption of convolution computing units in current neural network accelerators, and to propose a universal neural network acceleration method with three-dimensional loop expansion.

[0006] In order to achieve the above object, the present invention adopts the following technical solution.

[0007] As a first aspect of the present invention, a general neural network computing unit connected to an external memory and an external controller is proposed, wherein the general neural network computing unit includes a central control module, a core computing unit, a primary buffer, and a primary buffer read-write controller;

[0008] The external controller sends a start signal after the central control module has configured each buffer and control module;

[0009] The central control module controls the first-level reading and writing of input data, the first-level reading and writing of weight data, the reading and writing of output data, the reading and writing of partial sum data and the reading and writing of quantized weight data through the sending, receiving and decoding of instructions, and controls the zero-level reading and writing of input data and the zero-level reading and writing of weight data to realize the configuration of each buffer and control module; and enables each control module when the start signal arrives;

[0010] The first-level buffer includes a quantized weight data buffer, a partial sum data buffer, an input data first-level buffer, a weight data first-level buffer, and an output data buffer; the general neural network calculation unit performs first-level buffering on the input feature map data, weight data, and quantized weight data required before the neural network calculation, the partial sum data generated during the neural network calculation, and the output data generated after the neural network calculation, and adjusts the read-write control logic in real time through instructions;

[0011] The universal neural network computing unit further includes a zero-level buffer and a zero-level buffer read-write controller; the zero-level buffer includes an input data zero-level buffer and a weight data zero-level buffer; the zero-level buffer read-write controller includes an input data zero-level read-write control module and a weight data zero-level read-write control module; the zero-level buffer is respectively connected to the first-level buffer read-write controller and the zero-level buffer read-write controller;

[0012] The first-level buffer read-write controller includes a quantized weight data read-write control module, a partial data read-write control module, an input data first-level read-write control module, a weight data first-level read-write control module, and an output data read-write control module; the input data first-level read-write control module and the weight data first-level read-write control module respectively detect the state of the input data and weight data first-level buffer and the state of the zero-level buffer, and enter a blocking state when the buffer state does not meet the reading and writing requirements, and wait in the blocking state until the reading and writing requirements are met before performing data reading and writing operations;

[0013] The input data zero-level read-write control module and the weight data zero-level read-write control module read data from the zero-level buffer and write it into the core computing unit for convolution calculation, and the output data obtained by the convolution is sent to the output data buffer;

[0014] After the output data buffer receives the output data obtained by convolution, the core computing unit sends a calculation end signal to the central control module; after receiving the calculation end signal, the central control module sends a clear instruction to each control module and the core computing unit to complete the calculation.

[0015] As a second aspect of the present invention, a three-dimensional loop unrolling general neural network acceleration method is proposed, comprising the following steps:

[0016] S1. The central control module controls the first-level reading and writing of input data, the first-level reading and writing of weight data, the reading and writing of output data, the reading and writing of partial sum data and the reading and writing of quantized weight data through the sending, receiving and decoding of instructions, and controls the zero-level reading and writing of input data and the zero-level reading and writing of weight data to realize the configuration of each buffer and control module. After the configuration is completed, the external controller sends a start signal;

[0017] S2, the central control module enables each control module when the start signal arrives;

[0018] S3, the input data level read-write control module and the weight data level read-write control module detect the state of the input data and weight data level buffer and the state of the zero-level buffer respectively, and enter the blocking state when the buffer state does not meet the reading and writing requirements, and wait in the blocking state until the reading and writing requirements are met before performing data reading and writing operations;

[0019] S4, the input data zero-level read-write control module and the weight data zero-level read-write control module read data from the zero-level buffer and write it into the core computing unit for convolution calculation, and the output data obtained by the convolution is sent to the output data buffer;

[0020] S5. When all the output data obtained by the convolution are sent to the output data buffer, the core computing unit sends a computing end signal;

[0021] S6. After receiving the calculation end signal, the central control module sends a clear instruction to each control module and core computing unit to complete the convolution calculation.

[0022] The configuration of each buffer and control module in S1 is as follows: the central control module sends instructions to each first-level buffer to inform the operator type of this operation and the storage location, form and size of each data in the buffer, as well as the outer calculation order; sends instructions to each zero-level buffer to inform it of the inner calculation order of this calculation; sends instructions to the zero-level buffer and core computing unit to configure registers related to the calculation.

[0023] The S2 central control module enables each control module when the start signal arrives. Specifically, after receiving the start signal, the central control module samples the rising edge of the start signal and pulls up the enable signal of each control module when the rising edge of the start signal arrives.

[0024] S3 The buffer state does not meet the reading and writing requirements, specifically: when the buffer state is empty, it indicates that the buffer state does not meet the reading requirements; when the buffer state is full, it indicates that the buffer state does not meet the writing requirements.

[0025] In the S3 blocking state, the read and write requirements are met before performing data reading and writing operations. Specifically, in the blocking state, when the buffer state is empty, wait until the buffer state is not empty before reading data. Specifically, read the data from the first-level buffer and rearrange the data and partially fill in zeros according to the instruction information, and then send the data to the zero-level buffer; when the buffer state is full, wait until the buffer state is not full before writing data. Specifically, write the data after data rearrangement and partial zero filling operations into the zero-level buffer.

[0026] The convolution calculation described in S4 specifically relies on the core computing unit to continuously generate a pre-read enable signal to complete the pre-read operation of partial sum data, quantized weight data and output data.

[0027] S4 describes reading data from the zero-level buffer and writing it into the core computing unit, specifically: the input data zero-level read-write control module and the weight data zero-level read-write control module respectively detect the status of the zero-level buffer and the status of the core computing unit corresponding to the input data and the weight data. If the corresponding read and write requirements are not met, a blocking state is entered, and data reading and writing operations are performed after the requirements are met; if the corresponding read and write requirements are met, the data is read out from the zero-level buffer, and after the remaining zero-filling operation of the data is completed according to the instruction information, the input data and weight data are sent to the core computing unit.

[0028] The convolution calculation in S4 is implemented by a core computing unit including a broadcast control module, a multiplier array, an adder tree array, an accumulator array, and a quantization array. The specific calculation process includes the following sub-steps:

[0029] S41, the core computing unit receives input data and weight data, and completes data broadcasting, multiplication and reduction operations according to the instruction information;

[0030] S42, the multiplier-adder array in the core computing unit starts computing and generates an enable signal for partial and data reading and writing;

[0031] S43, after receiving the enable signal of partial sum data reading and writing, the partial sum data reading and writing control module reads the corresponding partial sum data in the primary buffer required for calculation, and then sends the partial sum data to the core computing unit;

[0032] S44, after receiving the partial sum data, the core computing unit completes the accumulation operation between the reduced data and the partial sum data;

[0033] S45, the accumulator array in the core computing unit starts calculation and generates an enable signal for reading and writing quantized weight data;

[0034] S46, after receiving the enable signal for reading and writing the quantized weight data, the quantized weight data read and write control module reads the corresponding quantized weight data in the primary buffer required for calculation, and then sends the quantized weight data to the core calculation unit;

[0035] S47, after receiving the quantization weight data, the core computing unit completes the quantization operation;

[0036] S48, the quantization array calculation in the core calculation unit starts and generates an enable signal for reading and writing output data;

[0037] S49, after receiving the output data read and write enable signal, the output data read and write control module sends the calculated output data to the output buffer.

[0038] Beneficial Effects

[0039] The present invention proposes a general neural network acceleration method based on three-dimensional loop unrolling, which has the following beneficial effects compared with the existing neural network acceleration methods:

[0040] 1. The method proposes a method for input channels, output channels and output features. Figure 3 It can expand the loop in multiple dimensions for parallel computing. Compared with the conventional loop expansion method, it has higher computing performance and data reuse rate, and reduces the power consumption caused by memory access.

[0041] 2. The method proposes an address signal generation mechanism based on multiple loop unrolling. By configuring the cascade counters in real time, a module can be adapted to various address signal generation methods based on loop unrolling at the same time, which greatly improves the flexibility and configurability of the neural network accelerator.

[0042] 3. The method proposes an efficient zero-padding and data rearrangement mechanism for neural networks. With the help of a multi-level buffer, it realizes phased zero-padding of the input feature map, realizes partial real-time data rearrangement and zero-padding, simplifies the zero-padding and data rearrangement logic, reduces external bandwidth requirements, and improves the data utilization of the on-chip buffer. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of the composition and connection relationship of the acceleration unit in a general neural network acceleration method based on three-dimensional loop unrolling according to the present invention;

[0044] Figure 2 A convolution calculation principle diagram of a three-dimensional loop expansion according to the present invention;

[0045] Figure 3 A flow control mechanism diagram of a three-dimensional loop unrolling general neural network acceleration method according to the present invention;

[0046] Figure 4 A flowchart of a buffer controller of a three-dimensional loop unrolling general neural network acceleration method according to the present invention;

[0047] Figure 5 This is a data rearrangement principle diagram of a three-dimensional loop unrolling general neural network acceleration method described in the present invention. DETAILED DESCRIPTION

[0048] The practical application process and advantages (beneficiary effects) of a general neural network acceleration method based on three-dimensional loop unrolling of the present invention are described in detail below in conjunction with the accompanying drawings and embodiments.

[0049] Example 1

[0050] This embodiment describes a three-dimensional loop unrolling general neural network acceleration method, which belongs to the technical field of neural network acceleration and processing unit design. The method relies on a general neural network computing unit connected to an external memory and an external controller, such as Figure 1 As shown, the general neural network computing unit includes a central control module, a first-level buffer, a first-level buffer controller and a core computing unit; the first-level buffer includes an input data first-level buffer, a weight data first-level buffer, a quantized weight data buffer, a partial sum data buffer and an output data buffer; the first-level buffer controller, the first-level buffer is connected to an external controller and a first-level buffer controller, and the computing unit also includes a zero-level buffer and a zero-level buffer controller; the zero-level buffer includes an input data zero-level buffer and a weight data zero-level buffer; the zero-level buffer controller, the zero-level buffer is respectively connected to the first-level buffer controller and the zero-level buffer controller. The method includes: receiving instructions and sending decoded instruction information to configure each control module and the core computing unit; enabling after receiving a start signal; reading input data and weight data; reading input data and weight from the zero-level buffer and writing them into the core computing unit; the core computing unit continuously generates a pre-read enable signal to control the reading and writing of partial sum data, the reading and writing of quantized weights and output data respectively; after the output data is written into the zero-level buffer, the output data read and write control module reads the output data from the zero-level buffer and writes it into the first-level buffer, and after completing all writing work, sends a calculation end signal. The method has input channels, output channels and output features Figure 3 Dimensional parallel computing has higher computing performance and data reuse rate, and reduces the power consumption caused by memory access.

[0051] A general neural network acceleration method based on three-dimensional loop unrolling, in which the input channel, output channel and output feature Figure 3The expanded dimensions are stored in the buffer in sequence. The expanded dimensions are set to PC, PK and PE, all of which are configurable parameters. For the three-dimensional loop expansion, each loop calculates the matrix multiplication of PC×PK×PE input pixels and the corresponding position weights. After the matrix multiplication is completed, an addition tree with a fan-in of PC is input to sum the multiplication results; each loop generates PK×PE partial sums for cyclic accumulation. The specific implementation includes the following steps:

[0052] Step 1: The central control module receives the instruction, generates configuration information of each control module and the core computing unit according to the received instruction, and sends the decoded instruction information to each control module and the core computing unit.

[0053] The specific contents are as follows:

[0054] The central control module sends instructions to each first-level buffer controller, informing it of the operator type of this operation and the storage location, form and size of each data in the buffer, as well as the outer calculation order; sends instructions to each zero-level buffer, informing it of the inner calculation order of this calculation; sends instructions to the zero-level buffer and computing unit to configure its calculation-related registers.

[0055] Step 2: The central control module receives the start signal and samples the rising edge of the start signal. When the rising edge of the start signal arrives, the enable signals of each control module and the core computing unit are pulled high.

[0056] Step 3: Read input data and weight data. The reading process includes the following sub-steps:

[0057] Step 31, the input data read-write control module and the weight data read-write control module in the first-level buffer controller first detect the status of the first-level buffer and the status of the zero-level buffer. If the corresponding read and write requirements are not met, they enter the blocking state and wait until the requirements are met before performing data reading and writing operations;

[0058] Step 32: If the corresponding read and write requirements are met, the data is read out from the first-level buffer, and after data rearrangement and partial zero filling operations are performed according to the instruction information, the data is sent to the zero-level buffer.

[0059] Step 4: The input data read / write control module and the weight data read / write control module in the zero-level buffer controller read data from the zero-level buffer and write it into the core computing unit. The reading process includes the following sub-steps:

[0060] Step 41. The two modules in step 4 respectively detect the status of their corresponding zero-level buffer and the status of the computing unit. If the corresponding reading and writing requirements are not met, they enter a blocking state and wait until the requirements are met before performing data reading and writing operations.

[0061] Step 42: If the corresponding read and write requirements are met, the data is read out from the zero-level buffer, and after the remaining zero-filling operation of the data is completed according to the instruction information, the data is sent to the core computing unit.

[0062] Step 5, during the flow of data input into the core computing unit in the core computing unit, a pre-read enable signal is continuously generated, and the partial sum data read-write control module, the quantized weight data read-write control module and the output data read-write control module are respectively controlled to generate enable signals in advance, so as to realize the pre-read operation of partial sum data, quantized weight data and output data, so as to complete the calculation process of the core computing unit, and the calculation process includes the following sub-steps:

[0063] Step 51, after receiving the input data and weight data, the core computing unit completes the data broadcasting, multiplication and reduction operations according to the instruction information.

[0064] Step 52: When the multiplier array calculation in the core computing unit starts, an enable signal for the partial and data read and write control module is generated.

[0065] Step 53, after receiving the enable signal, the partial sum data read and write control module reads the corresponding partial sum and quantization weights in the first-level buffer required for calculation in real time according to the previously received instruction information, and then sends the partial sum data to the core computing unit.

[0066] Step 54, after receiving the partial sum data, the core computing unit completes the accumulation operation between the reduced data and the partial sum data according to the instruction information.

[0067] Step 55: When the accumulator array calculation in the core computing unit starts, an enable signal for the quantization weight data read and write control module is generated.

[0068] Step 56, after receiving the enable signal, the quantization weight data read and write control module reads the corresponding quantization weight data in the first-level buffer required for calculation in real time according to the previously received instruction information, and then sends the quantization weight data to the core computing unit.

[0069] Step 57, after receiving the quantization weight data, the core computing unit completes the quantization operation according to the instruction information.

[0070] Step 58: When the quantization array calculation in the core computing unit starts, an enable signal for the output data read and write control module is generated.

[0071] Step 59, after receiving the enable signal, the output data read and write control module sends the calculated output data to the zero-level buffer in real time according to the previously received instruction information.

[0072] Step 6. After the output data is written into the zero-level buffer, the output data read-write control module in the first-level buffer controller reads the output data from the zero-level buffer and writes it into the first-level buffer. After completing all writing work, it sends a calculation end signal to the central control module.

[0073] Step 7. After receiving the calculation end signal, the central control module sends a reset instruction to each control module and core computing unit to complete the reset operation of this calculation.

[0074] In the specific implementation, the characteristics of the convolutional neural network are analyzed to determine the three unfolded loops, and the specific calculation order is determined according to the selected three-dimensional loop:

[0075] 1. Analysis of convolution kernel dimension (loop1). The sizes of convolution kernels commonly used in convolutional neural networks range from 1×1 to 13×13, which is a relatively small range. In modern neural network accelerators, except for a few special scenarios, it is rare to perform parallel computing on convolution kernels, because doing so will lead to a significant reduction in the types of convolution kernel sizes and step sizes that the accelerator can adapt, thereby limiting the scope of application of the accelerator. Therefore, the loop unrolling method will not perform parallel computing on the convolution kernel (loop 1);

[0076] 2. Analysis of input channel dimension (loop 2) and output channel dimension (loop 4); Common input channels and output channels in convolutional neural networks range from 1 to 1024. Since the coupling between input channels and output channels is low, the control logic is simple, and it is compatible with convolutions of various sizes and steps, modern neural network accelerators will choose to perform parallel calculations on input channels or output channels. The loop unrolling method chooses to perform parallel calculations on both input channels and output channels;

[0077] 3. Analysis of output feature map dimension (loop 3): The size of the output feature map ranges from 20×20 to 512×512, which can provide rich parallelism without worrying about low hardware utilization. However, since the output feature map dimension is a double loop, if both loops are expanded at the same time, problems such as complex data paths and poor adaptability will be faced; therefore, the loop expansion method chooses to perform parallel calculations on one of the dimensions of the output feature map.

[0078] In summary, the three-dimensional expansion method selects the input channel, output channel and output feature Figure 3 It can perform parallel computing in multiple dimensions, thereby improving the performance ceiling and simplifying the data path.

[0079] Figure 2This is a schematic diagram of a three-dimensional loop expansion according to the present invention; based on the three-dimensional loop expansion method proposed in this embodiment, the output channel, the vertical direction of the output feature map and the input channel are expanded in parallel, and the degrees of parallelism are recorded as PK, PE and PC respectively. At this time, the convolution operation can be mapped to the loop expansion shown below:

[0080]

[0081] In the above formula, the loop can be divided into two parts. Loop4, loop3-h, loop3-w, loop2, loop1-h and loop1-w are the outer loop expansion and inner loop expansion, corresponding to the process of data flowing from each level 1 buffer into the computing unit; the innermost three loops are parallel loop expansion, corresponding to the process of data being calculated in the computing unit. At this time, if the loops in the outer loop expansion and inner loop expansion are closer to the parallel loop expansion, it is called the calculation is closer to the inside; the farther away from the parallel loop expansion, it is called the calculation is closer to the outside.

[0082] On this basis, we split the outer loop and the inner loop and swap the loop order, determine the data format corresponding to the outer loop and the inner loop, and map the input data and weight data to the zero-level buffer respectively.

[0083] In order to determine the calculation order of the outer loop expansion and the six-fold remaining loop of the inner loop expansion part, we first analyze the impact of different loop layers on the neural network calculation process:

[0084] 1. Output channel (loop 4): The data of different output channels are generated by convolution of different convolution kernels, and the calculations between different convolution kernels are independent of each other. Therefore, in modern neural network accelerators, the output channel is usually placed at the outermost layer of the loop, that is, the output channel is calculated last.

[0085] 2. Input channel (loop 2): When performing convolution calculations, if the input channel dimension is switched, the input feature map and convolution kernel will change at the same time. Because the input channel has a low coupling degree with the other three dimensions, adjusting its calculation order in loop unrolling does not increase the complexity of the data path and control signals, but the calculation order of the input channel will greatly affect the scale of the partial sums. The specific impact is as follows:

[0086] The closer the input channel calculation is to the outside, the larger the scale of the partial sum will be, but a simpler data path can be easily obtained, because without involving the expansion of the outer input and output channels, the data of the convolution kernel will remain unchanged for a long time, thereby reducing the memory access bandwidth requirements of the storage subsystem; when the input channel calculation is closer to the inside, the smaller the scale of the partial sum will be, but this also means that the feature map and convolution kernel of the input computing unit need to be switched more frequently. At this time, in order to achieve data reuse and reduce the power consumption overhead of accessing the storage subsystem, a larger on-chip memory access bandwidth is required.

[0087] Therefore, in order to meet the bandwidth requirements of the storage subsystem in different application scenarios, the computing unit can freely adjust the order of input channels in the loop unrolling, thereby balancing the memory access bandwidth and the partial and buffer sizes.

[0088] 3. Output feature map (loop 3) and convolution kernel (loop 1): Convolution operation is localized. During the sliding window process, there is a lot of data reuse. This reuse can be divided into inter-row data reuse and intra-row data reuse. Inter-row reuse means that the same data may be reused during the sliding window process of the convolution kernel between different rows. Intra-row reuse means that the same data may be reused during the sliding window process of the convolution kernel within the same row.

[0089] Take the convolution kernel size of 3×3, and the input feature map and output feature map being the same as an example. Ideally, if the data reuse generated in the sliding window process can be fully utilized, the on-chip buffer size will be reduced by 9 times compared to directly converting the convolution into matrix multiplication; if the on-chip register file can be used to cache all the data that needs to be reused, the number of times data is read from the global buffer will also be reduced by 9 times.

[0090] The output feature map and convolution kernel both have two layers of loops in the horizontal direction and the vertical direction. When the calculation order of the output feature map and the convolution kernel is closer to the inside, the data reuse is higher, but the complexity of the data path will also increase significantly. When the calculation order of the output feature map and the convolution kernel is closer to the outside, the complexity of the data path is greatly reduced, but due to the reduced data reuse, it brings greater bandwidth and area pressure to the storage subsystem. Therefore, it is necessary to properly loop these four layers and these two loops to balance the contradiction between data reuse and data path complexity.

[0091] When the horizontal and vertical loops are close together, it means that the computing unit needs to frequently read data located in different rows in order to complete a convolution operation in a short time, which makes the design of the data path very difficult. Therefore, it is necessary to separate the horizontal and vertical directions as much as possible and insert other loops between these two loops. This can effectively utilize the data reuse within the row without sacrificing too many hardware resources and control logic to achieve data reuse between rows.

[0092] In summary, the calculation sequence corresponding to the serial loop unrolling of the computing unit is as follows:

[0093]

[0094] Among them, the loop expansion can be divided into 4 parts. The position of the output channel (loop 4) is absolutely fixed and located at the outermost side of the loop; the positions of the output feature map (loop 3) and the convolution kernel (loop 1) can be swapped, corresponding to the convolution operations expanded by row and column respectively; the position of the input channel (loop2) is absolutely flexible. By adjusting the position of the loop of this layer, different calculation orders can be activated. A total of 2×5=10 different calculation orders can be obtained, thereby obtaining different power consumption and on-chip cache overhead. This shows the flexibility and configurability of the computing unit.

[0095] Assuming that the input channel is at the innermost side, when the above serial loop expansion is used for convolution calculation, it can not only ensure a certain data reuse, but also greatly reduce the complexity of the data path. Taking the convolution size of 3×3 and the input feature map and the output feature map of equal size as an example, directly using the loop calculation and using the zero-level buffer in the computing unit for data buffering can save 3 times the on-chip buffer size and reduce the number of times of reading data from the global buffer by 3 times.

[0096] Step 2, the method performs data reading, single-stage data rearrangement, and double-stage zero padding according to the three dimensions of loop expansion. Specifically, the feature map data is read from the first-level buffer, and zero padding and data rearrangement are performed on it according to the three-dimensional loop expansion before writing it into the zero-level buffer; the feature map data is read from the zero-level buffer, and zero padding is performed on it according to the three-dimensional loop expansion before writing it into the convolution calculation unit.

[0097] The process of mapping the method to the convolutional computing unit is analyzed.

[0098] When loading data from the global buffer into the convolutional compute unit, there are two mismatches: the mismatch between the read and write bandwidth of the global buffer and the read and write bandwidth of the compute unit; and the mismatch between the data storage format in the global buffer and the data storage format required by the compute unit.

[0099] The former mismatch is caused by the uncontrollable computational intensity of the operator. For example, the convolution operator has a high computational intensity and the data has a high data reuse rate, so the read and write bandwidth requirements for the global buffer are relatively low. To address this mismatch, we add a zero-level buffer and implement it through data flow control and transmission mechanisms.

[0100] Figure 3 This is a flow control mechanism diagram of a three-dimensional loop unrolling general neural network acceleration method described in the present invention. The flow control mechanism consists of three parts: an upstream control module, a buffer, and a downstream control module. The upstream control module sends a write enable and a write pointer to the buffer, and the downstream control module sends a read enable and a read pointer to the buffer.

[0101] Take the convolution kernel size of 3*3 and the input channel (loop 2) in the innermost layer as an example. At this time, the upstream control module is a first-level buffer controller, the buffer is a zero-level buffer, and the downstream control module is a zero-level buffer controller. At this time, the upstream will write data of one address in each clock cycle when the buffer is not full, and increase the write pointer by 1; the downstream will read data of one address in each clock cycle when the buffer is not empty, and increase the read pointer by 3 after an interval of 3 clock cycles. When the input channel moves outward, the downstream will increase 3× the number of rows after an interval of 3× the number of rows of clock cycles.

[0102] After adjusting the calculation order of the convolution loop expansion, the mechanism can automatically receive instruction information and adjust the behavior logic to start calculation immediately without too much waiting. The traditional ping-pong structure has to wait for the data to fill at least one buffer before it can start, reaching the speed limit of data reading in the process.

[0103] There are two reasons for the latter mismatch: on the one hand, due to the loop expansion, the data that the computing unit can receive needs to be an integer multiple of the parallelism. Taking the loop expansion as 4×4×4, when the input data is 6×6, it has to be padded with zeros to 8×8 before it can be sent to the computing unit; on the other hand, since the convolution operator has specific zero-padding logic, additional data rearrangement and zero-padding logic need to be added to achieve it. The computing unit solves this mismatch problem by working together with the first-level buffer read-write controller and the zero-level buffer read-write controller.

[0104] Figure 4 This is a flowchart of the buffer controller of the three-dimensional loop unrolling general neural network acceleration method described in the present invention.

[0105] The functions of the first-level buffer read-write controller are as follows: on the one hand, it completes the data rearrangement operation and converts the data format in the global buffer into the data format required for the convolution operation; on the other hand, it completes the zero-filling operation between rows.

[0106] The functions of the zero-level buffer read and write controller are as follows: sequentially read the data in the zero-level buffer and complete the zero-filling operation between columns.

[0107] For example, when the convolution kernel size is 3×3 and the step size is 1, the input data first-level read-write control module pads the input feature map in the global buffer from 60×60 to 64×60, with one row of zeros on the top and three rows of zeros on the bottom, and sends it to the zero-level buffer; the input data zero-level read-write control module pads the 64*60 in the zero-level buffer to 64×62, and sends it to the computing unit for calculation.

[0108] Through single-stage data rearrangement and double-stage zero-filling operation, the original complex zero-filling logic is decoupled, the number of combinational logic levels required for zero-filling operation is reduced by 50%, and the upper limit of the system's operating frequency is increased. Through the above structure, a total of 9 times the on-chip buffer size can be saved, while reducing the number of times data is read from the global buffer by 9 times, reaching the theoretical upper limit.

[0109] Step 3: The convolution calculation unit reads the feature map data from the zero-level buffer and performs convolution calculation after three-dimensional loop expansion.

[0110] Example 2

[0111] This embodiment specifically describes the data rearrangement and zero padding process of the method.

[0112] In convolution calculations, the input feature map is usually a multi-channel multi-dimensional data. In order to use memory and cache more efficiently, the input data can be rearranged into a continuous storage form. When rearranging, the input channel dimension is usually placed in the innermost layer, so that the data can be accessed sequentially during the calculation process, reducing the randomness of memory access, reducing the number of times data is moved from memory to the processing unit, and improving calculation efficiency. Similarly, weights can also be rearranged, merging the weight size and channel into a larger continuous block, so that they can be accessed and calculated faster in convolution operations.

[0113] In order to match the read and write bandwidth of the global buffer with the read and write bandwidth of the computing unit, the present invention performs ping-pong caching and data rearrangement in the first-level buffer control unit. Figure 5 Shown is the schematic diagram of data rearrangement.

[0114] For example, when the read and write bandwidths of the global buffer and the computing unit are both PE×PC, due to the locality of data storage, the data read from the first-level buffer is PE×PC data along the input channel, while the computing unit requires PE×PC data expanded in the vertical direction of the feature map and the input channel direction. The present invention opens two buffers in the first-level buffer controller for ping-pong caching. When PE×PC data is written to one of the buffers by row, PE×PC data is read out by column from the other buffer at the same time. In this way, read bandwidth matching and efficient data reading are achieved.

[0115] Zero padding can keep the spatial size of the input and output feature maps relatively unchanged, avoiding the problem of the output feature map size shrinking due to convolution kernel sliding. The size of the zero padding is specified when configuring the convolution layer, and is dynamically implemented during the loading and calculation of the input feature map to adapt to convolution kernels of different sizes and strides.

[0116] The present invention realizes row zero padding and column zero padding of a feature map through double-stage zero padding: a first-level buffer controller performs row zero padding after reading data from a first-level buffer and then sends the data to a zero-level buffer; a zero-level buffer controller performs column zero padding after reading data from a zero-level buffer and then sends the data to a convolution calculation unit.

[0117] For example, when the input feature map size is 60×60, the convolution kernel size is 3×3, and the step size is 1, the input data first-level buffer controller pads the feature map to 64×60, with one row of zeros on the top and three rows of zeros on the bottom. In the first-level buffer controller, the feature map vertical expansion counter, input channel counter, feature map row counter, convolution kernel column counter, feature map column counter, and output channel counter work in cascade order to generate address signals and zero padding signals in real time. When the sum of the feature map vertical expansion counter, convolution kernel column counter, and feature map column counter is less than 1, the zero padding signal is valid and the first row is padded with zeros; when the sum of the feature map vertical expansion counter, convolution kernel column counter, and feature map column counter is greater than 60, the zero padding signal is valid and the last three rows are padded with zeros.

[0118] The input data buffer controller pads the feature map in the zero-level buffer from 64*60 to 64×62, and sends it to the calculation unit for calculation. In the zero-level buffer controller, the input channel counter, convolution kernel row counter, feature map row counter, convolution kernel column counter, feature map column counter, and output channel counter work in cascade order to generate address signals and zero padding signals in real time. When the sum of the convolution kernel row counter and the feature map row counter is less than 1, the zero padding signal is valid and the first column is padded with zeros; when the sum of the convolution kernel row counter and the feature map row counter is greater than 60, the zero padding signal is valid and the last column is padded with zeros.

[0119] Step 1: The method reads, rearranges and fills data with zeros according to the three dimensions of loop expansion. Specifically, feature map data is read from the first-level buffer, and zero-filled and data rearranged according to the three-dimensional loop expansion before writing into the zero-level buffer.

[0120] Step 11. Configure convolution related parameters, including: convolution kernel size, feature map size, number of input channels, number of output channels, zero padding size, cascade counter configuration, and buffer address offset.

[0121] Step 12. Cascade counters in sequence: feature map vertical expansion counter, input channel counter, feature map row counter, convolution kernel column counter, feature map column counter, output channel counter, and increment and clear them according to the configured parameters.

[0122] Step 13, synchronously generate the data reading address of the first-level buffer according to the counter value.

[0123] Step 14. Determine whether to fill with zeros based on the configuration parameters and the counter value. If not, read the data of the corresponding address from the first-level buffer, otherwise replace it with zero.

[0124] Step 15, rearrange the read data or zero-filled data; the specific process is: cache the read data and zero-filled data into the register in the form of a matrix, and then transpose the output.

[0125] Step 16. Write the rearranged data into the zero-level buffer.

[0126] Step 2: The method reads and fills data with zeros according to the three dimensions of loop expansion. Specifically, the feature map data is read from the zero-level buffer, and zero-filled between columns according to the three-dimensional loop expansion before writing into the convolution calculation unit.

[0127] Step 21. Configure convolution related parameters, including: convolution kernel size, feature map size, number of input channels, number of output channels, zero padding size, cascade counter configuration, and buffer address offset.

[0128] Step 22. Cascade counters in sequence: input channel counter, convolution kernel row counter, feature map row counter, convolution kernel column counter, feature map column counter, output channel counter, and increment and clear them according to the configured parameters.

[0129] Step 23, synchronously generate the data reading address of the zero-level buffer according to the counter value.

[0130] Step 24, determine whether to fill with zeros according to the configuration parameters and the counter value. If not, read the data of the corresponding address from L0_cache, otherwise replace it with zero;

[0131] Step 25, write the zero-filled data into the calculation unit;

[0132] The above is only a preferred embodiment of the present invention, and the present invention should not be limited to the contents disclosed in the embodiment and the drawings. Any equivalent or modification completed without departing from the spirit disclosed in the present invention shall fall within the scope of protection of the present invention.

Claims

1. A general neural network acceleration method based on three-dimensional loop unrolling, relying on a general neural network computing unit connected to an external memory and an external controller, including a central control module, a core computing unit, a primary buffer, and a primary buffer read-write controller; the primary buffer includes a quantized weight data buffer, a partial sum data buffer, an input data primary buffer, a weight data primary buffer, and an output data buffer; the general neural network computing unit performs primary buffering on the input feature map data, weight data, and quantized weight data required before the neural network calculation, the partial sum data generated during the neural network calculation, and the output data generated after the neural network calculation, and adjusts the read-write control logic in real time through instructions; It is characterized in that The general neural network computing unit further includes a zero-level buffer and a zero-level buffer read-write controller; the zero-level buffer includes an input data zero-level buffer and a weight data zero-level buffer; the zero-level buffer read-write controller includes an input data zero-level read-write control module and a weight data zero-level read-write control module; the zero-level buffer is connected to the first-level buffer read-write controller and the zero-level buffer read-write controller respectively, and the general neural network acceleration method includes the following steps: S1. The central control module controls the first-level reading and writing of input data, the first-level reading and writing of weight data, the reading and writing of output data, the reading and writing of partial sum data and the reading and writing of quantized weight data through the sending, receiving and decoding of instructions, and controls the zero-level reading and writing of input data and the zero-level reading and writing of weight data to realize the configuration of each buffer and control module. After the configuration is completed, the external controller sends a start signal; S2, the central control module enables each control module when the start signal arrives; S3, the input data level read-write control module and the weight data level read-write control module detect the state of the input data and weight data level buffer and the state of the zero-level buffer respectively, and enter the blocking state when the buffer state does not meet the reading and writing requirements, and wait in the blocking state until the reading and writing requirements are met before performing data reading and writing operations; S4, the input data zero-level read-write control module and the weight data zero-level read-write control module read data from the zero-level buffer and write it into the core computing unit for convolution calculation, and the output data obtained by the convolution is sent to the output data buffer; S5. When all the output data obtained by the convolution are sent to the output data buffer, the core computing unit sends a computing end signal; S6. After receiving the calculation end signal, the central control module sends a clear instruction to each control module and core computing unit to complete the convolution calculation.

2. A general neural network calculation method based on three-dimensional loop unrolling according to claim 1, characterized in that: The configuration of each buffer and control module in S1 is as follows: the central control module sends instructions to each first-level buffer to inform the operator type of this operation and the storage location, form, size and outer calculation order of each data in the buffer; sends instructions to each zero-level buffer to inform it of the inner calculation order of this calculation; sends instructions to the zero-level buffer and core computing unit to configure registers related to the calculation.

3. A general neural network calculation method based on three-dimensional loop unrolling according to claim 1, characterized in that: The S2 central control module enables each control module when the start signal arrives. Specifically, after receiving the start signal, the central control module samples the rising edge of the start signal and pulls up the enable signal of each control module when the rising edge of the start signal arrives.

4. A general neural network calculation method based on three-dimensional loop unrolling according to claim 1, characterized in that: S3 The buffer state does not meet the reading and writing requirements, specifically: when the buffer state is empty, it indicates that the buffer state does not meet the reading requirements; when the buffer state is full, it indicates that the buffer state does not meet the writing requirements.

5. A general neural network calculation method based on three-dimensional loop unrolling according to claim 4, characterized in that: In the S3 blocking state, the read and write requirements are met before performing data reading and writing operations. Specifically, in the blocking state, when the buffer state is empty, wait until the buffer state is not empty before reading data. Specifically, read the data from the first-level buffer and rearrange the data and partially fill in zeros according to the instruction information, and then send the data to the zero-level buffer; when the buffer state is full, wait until the buffer state is not full before writing data. Specifically, write the data after data rearrangement and partial zero filling operations into the zero-level buffer.

6. A general neural network calculation method based on three-dimensional loop unrolling according to claim 1, characterized in that: The convolution calculation described in S4 specifically relies on the core computing unit to continuously generate a pre-read enable signal to complete the pre-read operation of partial sum data, quantized weight data and output data.

7. A general neural network acceleration method based on three-dimensional loop unrolling according to claim 1, characterized in that: S4 describes reading data from the zero-level buffer and writing it into the core computing unit, specifically: the input data zero-level read-write control module and the weight data zero-level read-write control module respectively detect the status of the zero-level buffer and the status of the core computing unit corresponding to the input data and the weight data. If the corresponding read and write requirements are not met, a blocking state is entered, and data reading and writing operations are performed after the requirements are met; if the corresponding read and write requirements are met, the data is read out from the zero-level buffer, and after the remaining zero-filling operation of the data is completed according to the instruction information, the input data and weight data are sent to the core computing unit.

8. A general neural network acceleration method based on three-dimensional loop unrolling according to claim 1, characterized in that: The convolution calculation in S4 is implemented by a core computing unit including a broadcast control module, a multiplier array, an adder tree array, an accumulator array, and a quantization array. The specific calculation process includes the following sub-steps: S41, the core computing unit receives input data and weight data, and completes data broadcasting, multiplication and reduction operations according to the instruction information; S42, the multiplier-adder array in the core computing unit starts computing and generates an enable signal for partial and data reading and writing; S43, after receiving the enable signal of partial sum data reading and writing, the partial sum data reading and writing control module reads the corresponding partial sum data in the primary buffer required for calculation, and then sends the partial sum data to the core computing unit; S44, after receiving the partial sum data, the core computing unit completes the accumulation operation between the reduced data and the partial sum data; S45, the accumulator array in the core computing unit starts calculation and generates an enable signal for reading and writing quantized weight data; S46, after receiving the enable signal for reading and writing the quantized weight data, the quantized weight data read and write control module reads the corresponding quantized weight data in the primary buffer required for calculation, and then sends the quantized weight data to the core calculation unit; S47, after receiving the quantization weight data, the core computing unit completes the quantization operation; S48, the quantization array calculation in the core calculation unit starts and generates an enable signal for reading and writing output data; S49, after receiving the output data read and write enable signal, the output data read and write control module sends the calculated output data to the output buffer.

Images