Low-power convolutional neural network accelerator based on layer fusion and its acceleration method

Abstract

The present invention discloses a low-power convolutional neural network accelerator based on layer fusion and an acceleration method thereof, which belongs to the field of neural network accelerators. The layer fusion design method is introduced on the basis of the traditional convolutional neural network accelerator design. Through the layer fusion computing architecture, the intermediate feature map data output by the previous convolution layer can be directly used as the input feature map data for the calculation of the next convolution layer, avoiding the transmission of the intermediate feature map to the off-chip memory, effectively reducing the overall power consumption of the system, alleviating bandwidth pressure, and reducing inference delay; at the same time, the layer fusion design can realize the pipeline design of the convolution level without sacrificing too many on-chip storage resources; each convolution layer has a dedicated convolution computing module, and each convolution array can be designed for array scale according to the corresponding convolution layer. The present invention introduces the design space exploration of the computing architecture parameters to maximize the computing efficiency of the accelerator.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network accelerators, and in particular to a low-power convolutional neural network accelerator based on layer fusion and an acceleration method thereof. Background Art

[0002] With the rapid development of deep learning and the continuous improvement of convolutional neural networks, computer vision research has reached a new peak. Object detection, a fundamental task in computer vision, is one of the areas that has achieved fruitful results. It is widely used in various fields of modern life and has broad application prospects in areas such as road monitoring, pedestrian detection, autonomous vehicles, medical image analysis, drone navigation, and national defense security. Convolutional neural networks, with their advantages in feature learning and transfer learning, have achieved a series of breakthroughs in image processing, making convolutional neural network-based object detection algorithms the current mainstream target detection algorithm.

[0003] Convolutional neural networks are computationally and memory-intensive. The significant computational time and data access energy costs inherent in their derivation process are critical considerations in hardware deployment. To address these challenges, network design can be optimized for large neural networks by compressing them. Furthermore, hardware design often employs methods such as loop reordering and unrolling to achieve efficient hardware architectures.

[0004] However, typical FPGA implementations of convolutional neural network accelerators typically perform convolution calculations layer by layer, and traditional optimization methods can only reduce energy consumption to a certain extent. As each convolution layer progresses, the intermediate feature maps generated are stored off-chip. After a convolution layer is completed, they are transferred back on-chip as input data for the next convolution layer, resulting in higher energy consumption and bandwidth requirements. As the network scale increases, the amount of data required to be transferred between the accelerator and memory also increases. Summary of the Invention

[0005] The present invention provides a low-power convolutional neural network accelerator based on layer fusion and an acceleration method thereof. The accelerator computing architecture based on the layer fusion method eliminates the transmission process of intermediate image data and off-chip memory, reduces the overall power consumption of the system by reducing the off-chip transmission energy consumption, and introduces a design method of design space exploration to explore and find the design scheme with the optimal computing performance and operation intensity under different hardware platforms, thereby ensuring the maximum performance and resource utilization of the accelerator design.

[0006] A first aspect of the present invention provides a low-power convolutional neural network accelerator based on layer fusion, comprising: an input feature map cache module for storing input feature map data transmitted from off-chip to on-chip;

[0007] The first convolution calculation module to the Nth convolution calculation module, the input end of the first convolution calculation module is connected to the output end of the input feature map cache module, the output end of the N-1th convolution calculation module is connected to the input end of the Nth convolution calculation module, N is greater than or equal to 2, the first convolution calculation module to the Nth convolution calculation module is used to complete the convolution calculation of each convolution layer according to the input feature map data, and complete the activation operation and the maximum pooling operation according to the configuration of each convolution layer, and the intermediate feature map data obtained after the calculation is temporarily stored in the module;

[0008] The input end of the activation cache module is connected to the output end of the Nth convolution calculation module, one output end is connected to the input end of the first convolution calculation module, and the other output end is connected to the off-chip memory, which is used to cache the complete output feature map data after the convolution calculation of one stage is completed, and according to the configuration of the state of the control module, choose to transmit the output feature map data to the first convolution calculation module to continue the next round of convolution calculation, or end the convolution calculation and transmit the output feature map data to the off-chip memory for storage;

[0009] The control module is respectively connected to the input feature map cache module, the first to Nth convolution calculation modules, and the activation cache module, and is used to control the jump between different calculation stages during the operation of the accelerator, and parse the control instructions sent by the CPU to complete the parameter configuration of the accelerator calculation;

[0010] The weight cache module is connected to the first convolution calculation module to the Nth convolution calculation module respectively, and is used to cache the weight data required for the convolution calculation at each stage.

[0011] In one embodiment of the present invention, the first convolution calculation module to the Nth convolution calculation module each include a convolution calculation array, an activation unit, a pooling unit and an intermediate data buffer module connected in sequence;

[0012] The convolution calculation array is used to perform convolution calculation on the input feature map data and the weight data to complete the multiplication and accumulation operation within the convolution sliding window and the accumulation operation of the calculation results between different input channels;

[0013] The activation unit is used to perform an activation operation on the feature map data after the convolution calculation is completed;

[0014] The pooling unit is used to complete the maximum pooling operation after the convolution layer according to the network structure;

[0015] The intermediate data buffer module is used to store the intermediate feature map data obtained after calculation.

[0016] In one embodiment of the present invention, the convolution calculation array is composed of Tm calculation channels, each corresponding to the calculation of a convolution output channel; the calculation structure of the Tm calculation channels is the same, including Tn convolution processing units, accumulators and row buffers, which are used to multiply and accumulate the input feature map data of the Tn input channels and the corresponding weight data, and add the results of the Tn input channels through the accumulator and send them to the row buffer. After the convolution sliding window slides one row on the input feature map, a partial sum of one row of output feature map is obtained. After the multiplication and accumulation results of the next batch of Tn input channels are obtained, they are added to the partial sum in the row buffer and stored again in the row buffer. The cycle is repeated multiple times until a complete row of output feature map data is obtained.

[0017] In one embodiment of the present invention, the pooling unit is used to perform maximum pooling operations with different step lengths according to the configuration of the control module. When performing a pooling operation with a step length of 2, the odd-bit data and the even-bit data of a row of data output by the convolution calculation array are compared in sequence, and the larger of the two values ​​is taken out and cached in a register. The above operation is repeated when the next row of data arrives, and the two rows of maximum value data are compared in pairs to obtain the maximum pooling result when the step length is 2; when performing a pooling operation with a step length of 1, the adjacent data of a row of data output by the convolution calculation array are compared in sequence, and the larger of the two values ​​is taken out and cached in a register. The above operation is repeated when the next row of data arrives, and the two rows of maximum value data are compared in pairs to obtain the maximum pooling result when the step length is 1.

[0018] In one embodiment of the present invention, the intermediate data buffer module includes multiple memories for storing the output feature map data obtained by convolution row by row, and each row of data is stored separately in a memory. The memory size is the output feature map width multiplied by the number of output channels. The memory is composed of multiple storage units, and each storage unit stores one feature map data; the number of memories required for the intermediate data buffer module is the convolution kernel size plus one. When the corresponding intermediate feature map data is placed in the memory with a number equal to the convolution kernel size, it is sent to the next convolution calculation module as the input data of the next convolution layer, and the remaining memory is used to continue to receive the convolution calculation output of the convolution layer.

[0019] In one embodiment of the present invention, the computing structure of the first convolution computing module to the Nth convolution computing module adopts a row buffer design, and each convolution computing module slides the convolution window on the corresponding convolution layer to obtain output feature map data row by row; after the previous convolution layer obtains the feature map data required for the lower layer to perform a row of convolution calculation, the intermediate feature map data enters the next convolution computing module, and the lower layer convolution starts the same computing process, outputs the feature map data row by row, and puts each row of data into the memory of the corresponding intermediate data buffer module.

[0020] In one embodiment of the present invention, in a layer fusion accelerator built on different hardware platforms, in order to maximize hardware resource utilization and throughput, the optimal design parameters of the first convolution computing module to the Nth convolution computing module are determined by a space exploration method, including: sequentially selecting the input data, weight data and output data of each layer of the convolutional neural network, the output data size is R rows, C columns and M output channels, the weight data size is K rows, K columns, N input channels and M output channels, and the total number of computing operations of the network model is:

[0021]

[0022] Among them, L is the number of layers of the selected convolutional neural network;

[0023] Max_cycles i is the maximum cycle required by each convolution calculation module in a round of convolution, is the height of the output feature map of the last convolution layer in a round of convolution. P represents the number of convolution calculation modules. The total number of execution cycles for each round of convolution calculation is:

[0024] Execution cycles=Max_cycles×R last

[0025] The data access amount for each convolution calculation is:

[0026] Data access = α in ×B in +α weight ×B weight +α out ×B out

[0027] Among them, α is respectively in , α weight , α out The number of times the input data, weight data, and output data of each convolution layer are transferred. The amount of data transferred between the accelerator and the off-chip memory in each inference is a constant value, which is the sum of the input feature map size of the first convolution layer and the output feature map size of the last convolution layer. The data transfer refers to the data transfer from the on-chip memory to the convolution calculation array. Tm is the output channel block size, Tn is the input channel block size, Tr is the row block size, and Tc is the column block size. The specific number of times is calculated as:

[0028]

[0029]

[0030]

[0031] Record B separately in 、B weight 、B out The total amount of input data, weight data, and output data for each convolution layer is calculated as follows:

[0032] B in =T n (ST r +KS)(ST c +KS)

[0033] B weight =T m T n K 2

[0034] B out =T m T r T c

[0035] Among them, S is the convolution step size of each convolutional layer;

[0036] Based on the roofline model, the computing performance and operation intensity CTC of the convolutional neural network under the accelerator architecture are obtained as follows:

[0037]

[0038]

[0039] The hardware computing efficiency is:

[0040]

[0041] Note T mi and T ni is the number of output channel blocks and input channel blocks of the i-th layer convolution, MACs PCU The number of multiplications and accumulations required for each convolution calculation unit, DSPs total The total amount of DSP resources on the hardware platform is used to calculate the resource constraints:

[0042]

[0043] Remember Buffer weight Buffer input Buffer inter Buffer activationThe on-chip storage capacity required for weights, input feature map data, intermediate data, and activation data, respectively, BRAM total is the total BRAM resource of the hardware platform, and the on-chip storage resource constraint is:

[0044] Buffer weight +Buffer input +Buffer inter +Buffer activation ≤BRAM total

[0045] The computing performance Tm, Tn and P in the operation intensity CTC are enumerated to obtain the accelerator performance indicators under different design spaces, and the optimal design parameters of the layer fusion accelerator of the network model are obtained under the constraints of the hardware computing efficiency Efficiency and computing resource constraints.

[0046] A second embodiment of the present invention provides an acceleration method for a low-power convolutional neural network accelerator based on layer fusion, comprising the following steps: calculating the total number of operations of a network model and the number of execution cycles required for each round of convolution, accumulating the number of execution cycles required for multiple rounds of convolution, and dividing the sum by the total number of operations of the network model to obtain the computing performance of the accelerator; obtaining the operational intensity of the accelerator design based on the data access volume of each layer of convolution and the total number of network operations; obtaining the hardware computing efficiency based on the computing performance and parallelism; and obtaining the optimal design solution in the feasible design space based on the constraint relationship of hardware resources;

[0047] Design the size of the input feature map cache module, activation cache module, weight cache module, as well as the parameters and control logic of the control module through the optimal design scheme in the feasible design space;

[0048] Build a complete layer fusion accelerator and use a layer fusion-based accelerator to accelerate convolutional neural networks. The input feature map in the accelerator's external memory is transferred to the input feature map cache module through the AXI4-Stream bus. The weight data is transferred to the weight cache module through the AXI4-Stream bus. Then, the control module is configured to start convolution calculations. After the complete convolution calculation of an image is completed, it is stored in the activation cache module. The accelerator is then controlled to transfer the output data obtained by convolution to the off-chip memory for further processing and output.

[0049] The low-power convolutional neural network accelerator based on layer fusion and the acceleration method thereof according to the embodiments of the present invention have the following beneficial effects:

[0050] 1. By adopting a layer-fusion computing architecture, multiple convolution computing modules are designed on the chip to calculate different convolutional layers respectively. The intermediate feature map data output by the previous convolutional layer can be directly used as the input feature map data for the calculation of the next convolutional layer, avoiding the transmission of intermediate feature maps to off-chip memory, effectively reducing the overall power consumption of the system, alleviating bandwidth pressure, and reducing inference latency.

[0051] 2. The layer-fusion-based design does not need to store the complete intermediate feature maps between convolutional layers. Instead, it only stores the data required for the first row of convolution at the next convolutional layer. Therefore, it can implement a pipeline design at the convolutional level without sacrificing too much on-chip storage resources, thereby improving computational parallelism with less on-chip storage cost.

[0052] 3. Introducing the design space exploration of computing architecture parameters, traversing the design parameters to obtain a feasible design space, and performing hardware resource constraints according to the deployed hardware platform. This can maximize the computing performance, operation intensity, and hardware computing efficiency of the low-power convolutional neural network accelerator based on layer fusion according to the resource conditions of different platforms.

[0053] In summary, the low-power convolutional neural network accelerator based on layer fusion and its acceleration method of the present invention, from the perspective of reducing off-chip transmission power consumption, eliminates the transmission process of intermediate image data and off-chip memory by placing the image inference process completely on the FPGA using a layer fusion structure. At the same time, the design method of design space exploration is adopted to find the design scheme with the optimal computing performance and operation intensity under the constraints of the limited resources of the hardware platform, ensuring the maximum performance and resource utilization of the accelerator design.

[0054] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0056] Figure 1 A schematic structural diagram of a low-power convolutional neural network accelerator based on layer fusion according to an embodiment of the present invention;

[0057] Figure 2 A schematic structural diagram of an intermediate data buffer module provided according to an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of the structure of a convolutional computing module based on a layer fusion computing architecture according to an embodiment of the present invention;

[0059] Figure 4A schematic diagram illustrating the relationship between computing performance and operation intensity based on design space exploration of a layer fusion accelerator according to an embodiment of the present invention;

[0060] Figure 5 A schematic diagram illustrating the relationship between computing performance and hardware computing efficiency based on design space exploration of a layer fusion accelerator provided according to an embodiment of the present invention;

[0061] Figure 6 This is a flow chart of an acceleration method for a low-power convolutional neural network accelerator based on layer fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0063] To address the issues mentioned in the background, an optimization method for layer fusion was proposed. In a network, each point in the output layer corresponds to a well-defined region of the network's initial input. When the top-level data corresponding to an output point enters the accelerator, the data undergoes convolutional calculations at each layer within the accelerator until the desired output is achieved. Intermediate feature map data does not need to be stored off-chip.

[0064] Figure 1 This is a structural diagram of a low-power convolutional neural network accelerator based on layer fusion provided according to an embodiment of the present invention.

[0065] like Figure 1 As shown, the low-power convolutional neural network accelerator based on layer fusion includes: an input feature map cache module 101, a convolution calculation module 102, an activation cache module 103, a control module 104 and a weight cache module 105.

[0066] The input feature map cache module 101 is used to store the input feature map data transmitted from off-chip to on-chip. Since the layer fusion design avoids repeated transmission of intermediate feature maps to off-chip memory, the input feature map data only includes the input image data of the first layer of convolution.

[0067] The first convolution calculation module to the Nth convolution calculation module 102, the input end of the first convolution calculation module is connected to the output end of the input feature map cache module 101, the output end of the N-1th convolution calculation module is connected to the input end of the Nth convolution calculation module, N is greater than or equal to 2, the first convolution calculation module to the Nth convolution calculation module are used to complete the convolution calculation of each convolution layer according to the input feature map data, and complete the activation operation and the maximum pooling operation according to the configuration of each convolution layer, and the intermediate feature map data obtained after the calculation is temporarily stored in the module.

[0068] In the same calculation period, a convolution calculation module 102 corresponds to the calculation of a convolution layer and completes all the calculation requirements of the convolution layer; different convolution calculation modules perform convolution operations of multiple convolution layers in parallel to realize the pipeline design of the convolution level.

[0069] The intermediate feature map data generated by convolution is no longer transmitted to the off-chip memory, but is directly used as the input feature map data for the next convolution layer calculation, avoiding the transmission of the intermediate feature map to the off-chip memory, reducing the off-chip transmission energy consumption and the overall power consumption of the system.

[0070] The input end of the activation cache module 103 is connected to the output end of the Nth convolution calculation module, one output end is connected to the input end of the first convolution calculation module, and the other output end is connected to the off-chip memory, which is used to cache the complete output feature map data after the convolution calculation of one stage is completed, and according to the configuration of the state of the control module 104, it is selected to transmit the output feature map data to the first convolution calculation module to continue the next round of convolution calculation, or to end the convolution calculation and transmit the output feature map data to the off-chip memory for storage.

[0071] The control module 104 is connected to the input feature map cache module 101, the first convolution calculation module to the Nth convolution calculation module 102 and the activation cache module 103 respectively, and is used to control the jump between different calculation stages during the operation of the accelerator, and parse the control instructions sent by the CPU to complete the parameter configuration of the accelerator calculation, which is specifically implemented by a finite state machine.

[0072] The weight cache module 105 is connected to the first through Nth convolutional calculation modules 102, respectively, to cache the weight data required for each convolutional calculation stage. If on-chip storage allows, the weight data for multiple stages can be stored on-chip simultaneously to reduce energy consumption caused by off-chip transmission of weight data.

[0073] The above structure is used to accelerate the convolutional neural network in a low-power convolutional neural network accelerator based on layer fusion.

[0074] In one embodiment of the present invention, the first convolution calculation module to the Nth convolution calculation module 102 each include a convolution calculation array 106, an activation unit 107, a pooling unit 108 and an intermediate data buffer module 109 connected in sequence.

[0075] The convolution calculation array 106 is used to perform convolution calculation on the input feature map data and the weight data to complete the multiplication and accumulation operation within the convolution sliding window and the accumulation operation of the calculation results between different input channels.

[0076] The activation unit 107 is used to perform an activation operation on the feature map data after the convolution calculation is completed. In order to facilitate the implementation of nonlinear functions on hardware, the ReLU activation function is used in both network training and hardware implementation.

[0077] The pooling unit 108 is used to complete the maximum pooling operation after the convolution layer according to the network structure.

[0078] The intermediate data buffer module 109 is used to store the intermediate feature map data obtained after the calculation is completed. It serves as the output buffer of the previous convolution layer calculation. Since the convolution calculation modules responsible for different convolution layer operations are connected in sequence, it also serves as the input buffer of the next convolution layer calculation. Figure 2 shown.

[0079] In one embodiment of the present invention, the convolution calculation array 106 is composed of Tm calculation channels 301, each corresponding to the calculation of a convolution output channel; the calculation structure of the Tm calculation channels is the same, including Tn convolution processing units 302, accumulators 303 and row buffers 304, which are used to multiply and accumulate the input feature map data of the Tn input channels and the corresponding weight data, and add the results of the Tn input channels through the accumulator 303 and send them to the row buffer. After the convolution sliding window slides one row on the input feature map, a partial sum of a row of output feature maps is obtained. After the multiplication and accumulation results of the next batch of Tn input channels are obtained, they are added to the partial sum in the row buffer and stored again in the row buffer. This cycle is repeated multiple times until a complete row of output feature map data is obtained. Figure 3 shown.

[0080] In one embodiment of the present invention, the pooling unit 108 is used to perform maximum pooling operations with different step sizes according to the configuration of the control module 104. When performing a pooling operation with a step size of 2, the odd-bit data and the even-bit data of a row of data output by the convolution calculation array 106 are compared in sequence, and the larger of the two values ​​is taken out and cached in a register. The above operation is repeated when the next row of data arrives, and the two rows of maximum value data are compared pairwise to obtain the maximum pooling result when the step size is 2; when performing a pooling operation with a step size of 1, the adjacent data of a row of data output by the convolution calculation array 106 are compared in sequence, and the larger of the two values ​​is taken out and cached in a register. The above operation is repeated when the next row of data arrives, and the two rows of maximum value data are compared pairwise to obtain the maximum pooling result when the step size is 1.

[0081] In one embodiment of the present invention, the intermediate data buffer module 109 includes multiple memories 202, which are used to store the output feature map data obtained by convolution line by line. Each line of data is stored separately in a memory. The memory size is the output feature map width multiplied by the number of output channels. The memory is composed of multiple storage units 201, and each storage unit stores one feature map data; the number of memories required by the intermediate data buffer module 109 is the convolution kernel size plus one. When the corresponding intermediate feature map data is placed in the memory with a number equal to the convolution kernel size, it is sent to the next convolution calculation module as the input data of the next convolution layer, and the remaining memory is used to continue to receive the convolution calculation output of the convolution layer.

[0082] In one embodiment of the present invention, the computational structure of the first through Nth convolution computation modules 102 utilizes a row buffer design. Each convolution computation module slides a convolution window on the corresponding convolution layer to obtain output feature map data row by row. After the previous convolution layer obtains the feature map data required for a row of convolution computations in the lower layer, the intermediate feature map data enters the next convolution computation module, and the lower layer convolution begins the same computational process, outputting feature map data row by row. Each row of data is then stored in the memory of the corresponding intermediate data buffer module. This structure enables input feature map reuse between computation channels and weight reuse within adjacent convolution windows.

[0083] In one embodiment of the present invention, in a layer fusion accelerator built on different hardware platforms, in order to maximize hardware resource utilization and throughput, the optimal design parameters of the first convolution computing module to the Nth convolution computing module are determined by a space exploration method, including: sequentially selecting the input data, weight data and output data of each layer of the convolutional neural network, the output data size is R rows, C columns and M output channels, the weight data size is K rows, K columns, N input channels and M output channels, and the total number of computing operations of the network model is:

[0084]

[0085] Among them, L is the number of layers of the selected convolutional neural network;

[0086] Max_cycles i is the maximum cycle required by each convolution calculation module in a round of convolution, is the height of the output feature map of the last convolution layer in a round of convolution. P represents the number of convolution calculation modules. The total number of execution cycles for each round of convolution calculation is:

[0087] Execution cycles=Max_cycles×R last

[0088] The data access amount for each convolution calculation is:

[0089] Data access = α in ×B in +α weight ×B weight +α out ×B out

[0090] Among them, α is respectively in , α weight , α out = is the number of times the input data, weight data, and output data of each convolution layer are transferred. Since the design of the accelerator of the present invention avoids the transmission between the intermediate feature map data and the off-chip memory, the data transfer amount between the accelerator and the off-chip memory in each inference is a constant value, which is the sum of the input feature map size of the first convolution layer and the output feature map size of the last convolution layer. Data transfer refers to the data transfer from the on-chip memory to the convolution calculation array. Let Tm be the output channel block size, Tn be the input channel block size, Tr be the row block size, and Tc be the column block size. The specific number of times is calculated as:

[0091]

[0092]

[0093]

[0094] Record B separately in 、B weight 、B out The total amount of input data, weight data, and output data for each convolution layer is calculated as follows:

[0095] B in =T n (STr +KS)(ST c +KS)

[0096] B weight =T m T n K 2

[0097] B out =T m T r T c

[0098] Among them, S is the convolution step size of each convolutional layer;

[0099] Based on the roofline model, the computing performance and operation intensity CTC of the convolutional neural network under the accelerator architecture are obtained as follows:

[0100]

[0101]

[0102] The hardware computing efficiency is:

[0103]

[0104] Different hardware platforms have different hardware resource constraints. The relationship between the resources required for accelerator design and hardware resources is as follows. mi and T ni is the number of output channel blocks and input channel blocks of the i-th layer convolution, MACs PCU The number of multiplications and accumulations required for each convolution calculation unit, DSPs total The total amount of DSP resources on the hardware platform is used to calculate the resource constraints:

[0105]

[0106] Remember Buffer weight Buffer input Buffer inter Buffer activation The on-chip storage capacity required for weights, input feature map data, intermediate data, and activation data, respectively, BRAM total is the total BRAM resource of the hardware platform, and the on-chip storage resource constraint is:

[0107] Buffer weight +Buffer input +Buffer inter +Bufferactivation ≤BRAM total

[0108] The computing performance Tm, Tn and P in the operation intensity CTC are enumerated to obtain the accelerator performance indicators under different design spaces, and the optimal design parameters of the layer fusion accelerator of the network model are obtained under the constraints of the hardware computing efficiency Efficiency and computing resource constraints.

[0109] After determining the scale of the convolutional computing module, design the size of the input feature map cache module, activation cache module, weight cache module, as well as the parameters and control logic of the control module.

[0110] After completing the above work, build a complete layer fusion accelerator, use the layer fusion-based accelerator to implement YOLOv2-Tiny convolutional neural network acceleration, transfer the input feature map in the accelerator's external memory to the input feature map cache module through the AXI4-Stream bus, and transfer the weight data to the weight cache module through the AXI4-Stream bus. Then configure the control module to start the convolution calculation. After the complete convolution calculation is completed on an image, it is stored in the activation cache module, and then control the accelerator to transfer the output data obtained by the convolution to the off-chip memory for further processing and output.

[0111] like Figure 4 and Figure 5 As shown, the low-power convolutional neural network accelerator based on layer fusion in an embodiment of the present invention introduces a layer fusion design method on the basis of the traditional convolutional neural network accelerator design. Through the layer fusion computing architecture, the intermediate feature map data output by the previous convolution layer can be directly used as the input feature map data for the next convolution layer calculation, avoiding the transmission of the intermediate feature map to the off-chip memory, effectively reducing the overall power consumption of the system, alleviating bandwidth pressure, and reducing inference delay; at the same time, the layer fusion design can realize the pipeline design of the convolution level without sacrificing too many on-chip storage resources; each convolution layer has a dedicated convolution computing module, and each convolution array can be designed for array scale according to the corresponding convolution layer. The present invention introduces the design space exploration of the computing architecture parameters to maximize the computing efficiency of the accelerator.

[0112] Next, an acceleration method for a low-power convolutional neural network accelerator based on layer fusion proposed in an embodiment of the present invention is described with reference to the accompanying drawings.

[0113] Figure 6 This is a flow chart of an acceleration method for a low-power convolutional neural network accelerator based on layer fusion according to an embodiment of the present invention.

[0114] like Figure 6As shown, the acceleration method of the low-power convolutional neural network accelerator based on layer fusion includes the following steps:

[0115] Step S101, calculate the total number of operations of the network model and the number of execution cycles required for each round of convolution, accumulate the number of execution cycles required for multiple rounds of convolution, and divide it by the total number of operations of the network model to obtain the computing performance of the accelerator; obtain the operational intensity of the accelerator design based on the data access volume of each layer of convolution and the total number of network operations; obtain the hardware computing efficiency based on the computing performance and parallelism; and obtain the optimal design solution in the feasible design space based on the constraints of hardware resources.

[0116] Step S102, through the optimal design scheme in the feasible design space, design the size of the input feature map cache module, the activation cache module, the weight cache module, and the parameters and control logic of the control module.

[0117] Step S103: Build a complete layer fusion accelerator and use the layer fusion-based accelerator to accelerate the convolutional neural network. The input feature map in the accelerator's external memory is transferred to the input feature map cache module through the AXI4-Stream bus. The weight data is transferred to the weight cache module through the AXI4-Stream bus. Then, the control module is configured to start the convolution calculation. After the complete convolution calculation of an image is completed, it is stored in the activation cache module. The accelerator is then controlled to transfer the output data obtained by the convolution to the off-chip memory for further processing and output.

[0118] It should be noted that the aforementioned explanation of the embodiment of the low-power convolutional neural network accelerator based on layer fusion is also applicable to the acceleration method of the low-power convolutional neural network accelerator based on layer fusion in this embodiment, and will not be repeated here.

[0119] According to the acceleration method of the low-power convolutional neural network accelerator based on layer fusion proposed in an embodiment of the present invention, from the perspective of reducing off-chip transmission power consumption, the image inference process is completely placed on the FPGA using a layer fusion structure, thereby eliminating the transmission process of intermediate image data and off-chip memory. At the same time, a design space exploration design method is adopted to find the design solution with the optimal computing performance and operation intensity under the constraints of the limited resources of the hardware platform, thereby ensuring the maximum performance and resource utilization of the accelerator design.

[0120] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0121] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "N" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0122] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or N executable instructions for implementing a custom logical function or step of a process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

Claims

1. A low-power convolutional neural network accelerator based on layer fusion, characterized in that: include: The input feature map cache module is used to store input feature map data transmitted from off-chip to on-chip; The first convolution calculation module to the Nth convolution calculation module, the input end of the first convolution calculation module is connected to the output end of the input feature map cache module, the output end of the N-1th convolution calculation module is connected to the input end of the Nth convolution calculation module, N is greater than or equal to 2, the first convolution calculation module to the Nth convolution calculation module is used to complete the convolution calculation of each convolution layer according to the input feature map data, and complete the activation operation and the maximum pooling operation according to the configuration of each convolution layer, and the intermediate feature map data obtained after the calculation is temporarily stored in the module; The input end of the activation cache module is connected to the output end of the Nth convolution calculation module, one output end of the activation cache module is connected to the input end of the first convolution calculation module, and the other output end of the activation cache module is connected to the off-chip memory, so as to cache the complete output feature map data after the convolution calculation of one stage is completed, and according to the configuration of the state of the control module, select to transmit the output feature map data to the first convolution calculation module to continue the next round of convolution calculation, or end the convolution calculation and transmit the output feature map data to the off-chip memory for storage; The control module is respectively connected to the input feature map cache module, the first to Nth convolution calculation modules, and the activation cache module, and is used to control the jump between different calculation stages during the operation of the accelerator, and parse the control instructions sent by the CPU to complete the parameter configuration of the accelerator calculation; The weight cache module is connected to the first convolution calculation module to the Nth convolution calculation module respectively, and is used to cache the weight data required for the convolution calculation at each stage.

2. The low-power convolutional neural network accelerator based on layer fusion according to claim 1, characterized in that The first convolution calculation module to the Nth convolution calculation module each include a convolution calculation array, an activation unit, a pooling unit and an intermediate data buffer module connected in sequence; The convolution calculation array is used to perform convolution calculation on the input feature map data and the weight data to complete the multiplication and accumulation operation within the convolution sliding window and the accumulation operation of the calculation results between different input channels; The activation unit is used to perform an activation operation on the feature map data after the convolution calculation is completed; The pooling unit is used to complete the maximum pooling operation after the convolution layer according to the network structure; The intermediate data buffer module is used to store the intermediate feature map data obtained after calculation.

3. The low-power convolutional neural network accelerator based on layer fusion according to claim 2, characterized in that: The convolution calculation array is composed of Tm calculation channels, each corresponding to the calculation of a convolution output channel; the calculation structure of the Tm calculation channels is the same, including Tn convolution processing units, accumulators and row buffers, which are used to multiply and accumulate the input feature map data of the Tn input channels and the corresponding weight data, and add the results of the Tn input channels through the accumulator and send them to the row buffer. After the convolution sliding window slides one row on the input feature map, a partial sum of one row of output feature map is obtained. After the multiplication and accumulation results of the next batch of Tn input channels are obtained, they are added to the partial sum in the row buffer and stored again in the row buffer. This cycle is repeated multiple times until a complete row of output feature map data is obtained.

4. The low-power convolutional neural network accelerator based on layer fusion according to claim 2, characterized in that The pooling unit is used to perform maximum pooling operations with different step lengths according to the configuration of the control module. When performing a pooling operation with a step length of 2, the odd-bit data and the even-bit data of a row of data output by the convolution calculation array are compared in sequence, and the larger value of the two is taken out and cached in a register. The above operation is repeated when the next row of data arrives, and the two rows of maximum value data are compared in pairs to obtain the maximum pooling result when the step length is 2; when performing a pooling operation with a step length of 1, the adjacent data of a row of data output by the convolution calculation array are compared in sequence, and the larger value of the two is taken out and cached in a register. The above operation is repeated when the next row of data arrives, and the two rows of maximum value data are compared in pairs to obtain the maximum pooling result when the step length is 1.

5. The low-power convolutional neural network accelerator based on layer fusion according to claim 2, characterized in that: The intermediate data buffer module includes multiple memories for storing the output feature map data obtained by convolution row by row, and each row of data is stored separately in a memory. The memory size is the output feature map width multiplied by the number of output channels. The memory is composed of multiple storage units, and each storage unit stores one feature map data; the number of memories required for the intermediate data buffer module is the convolution kernel size plus one. When the corresponding intermediate feature map data is placed in the memory of the convolution kernel size, it is sent to the next convolution calculation module as the input data of the next convolution layer, and the remaining memory is used to continue to receive the convolution calculation output of the convolution layer.

6. The low-power convolutional neural network accelerator based on layer fusion according to claim 1, characterized in that The calculation structure of the first convolution calculation module to the Nth convolution calculation module adopts a row buffer design. Each convolution calculation module slides the convolution window on the corresponding convolution layer to obtain the output feature map data row by row; after the previous convolution layer obtains the feature map data required for the lower layer to perform a row of convolution calculation, the intermediate feature map data enters the next convolution calculation module, and the lower layer convolution starts the same calculation process, outputs the feature map data row by row, and puts each row of data into the memory of the corresponding intermediate data buffer module.

7. The low-power convolutional neural network accelerator based on layer fusion according to claim 1, characterized in that: In the layer fusion accelerator built on different hardware platforms, in order to maximize hardware resource utilization and throughput, the optimal design parameters of the first convolution computing module to the Nth convolution computing module are determined through a spatial exploration method, including: sequentially selecting the input data, weight data, and output data of each layer of the convolutional neural network, the output data size is R rows, C columns, and M output channels, the weight data size is K rows, K columns, N input channels, and M output channels, and the total number of computing operations of the network model is: Among them, L is the number of layers of the selected convolutional neural network; Max_cycles i is the maximum cycle required by each convolution calculation module in a round of convolution, is the height of the output feature map of the last convolution layer in a round of convolution. P represents the number of convolution calculation modules. The total number of execution cycles for each round of convolution calculation is: Execution cycles=Max_cycles×R last The data access amount for each convolution calculation is: Data access = a in ×B in +a weight ×B eight +a out ×B out Among them, α is respectively in , α weight , α out The number of times the input data, weight data, and output data of each convolution layer are transferred. The amount of data transferred between the accelerator and the off-chip memory in each inference is a constant value, which is the sum of the input feature map size of the first convolution layer and the output feature map size of the last convolution layer. The data transfer refers to the data transfer from the on-chip memory to the convolution calculation array. Tm is the output channel block size, Tn is the input channel block size, Tr is the row block size, and Tc is the column block size. The specific number of times is calculated as: Record B separately in 、B weight 、B out The total amount of input data, weight data, and output data for each convolution layer is calculated as follows: B in =T n (ST r +K-S)(ST c +K-S) B weight =T m T n K 2 B out =T m T r T c Among them, S is the convolution step size of each convolutional layer; Based on the roofline model, the computing performance and operation intensity CTC of the convolutional neural network under the accelerator architecture are obtained as follows: The hardware computing efficiency is: Note T mi and T ni is the number of output channel blocks and input channel blocks of the i-th layer convolution, MACs PCU The number of multiplications and accumulations required for each convolution calculation unit, DSPs total The total amount of DSP resources on the hardware platform is used to calculate the resource constraints: Remember Buffer weight Buffer input Buffer inter Buffer activation The on-chip storage capacity required for weights, input feature map data, intermediate data, and activation data, respectively, BRAM total is the total BRAM resource of the hardware platform, and the on-chip storage resource constraint is: Buffer weight +Buffer input +Buffer inter +Buffer activation ≤BRAM total The computing performance Tm, Tn and P in the operation intensity CTC are enumerated to obtain the accelerator performance indicators under different design spaces, and the optimal design parameters of the layer fusion accelerator of the network model are obtained under the constraints of the hardware computing efficiency Efficiency and computing resource constraints.

8. An acceleration method for a low-power convolutional neural network accelerator based on layer fusion according to any one of claims 1 to 7, characterized in that: The following steps are involved: Calculate the total number of operations of the network model and the number of execution cycles required for each round of convolution. Add the execution cycles required for multiple rounds of convolution and divide it by the total number of operations of the network model to obtain the computing performance of the accelerator. Calculate the operational intensity of the accelerator design based on the data access volume of each convolution layer and the total number of network operations. Get hardware computing efficiency based on computing performance and parallelism; According to the constraints of hardware resources, the optimal design solution in the feasible design space is obtained; Design the size of the input feature map cache module, activation cache module, weight cache module, as well as the parameters and control logic of the control module through the optimal design scheme in the feasible design space; Build a complete layer fusion accelerator and use a layer fusion-based accelerator to accelerate convolutional neural networks. The input feature map in the accelerator's external memory is transferred to the input feature map cache module through the AXI4-Stream bus. The weight data is transferred to the weight cache module through the AXI4-Stream bus. Then, the control module is configured to start convolution calculations. After the complete convolution calculation of an image is completed, it is stored in the activation cache module. The accelerator is then controlled to transfer the output data obtained by convolution to the off-chip memory for further processing and output.

Images