A convolution hardware accelerator based on RS data stream and method thereof

Abstract

The present invention discloses a convolution hardware accelerator based on RS data stream and its method. The accelerator includes an off-chip DDR memory; an on-chip cache module for storing raw image data, convolution kernel weight data, and intermediate convolution calculation results read from the off-chip DDR memory; a DDR controller for controlling data exchange between the off-chip DDR memory and the on-chip cache module; a data distribution module for transmitting required weights and excitation data to the optoelectronic calculation module based on current calculation configuration information and the calculation rules of the optoelectronic calculation module; the optoelectronic calculation module for completing convolution operations at each layer of the network; and a result collection module for receiving the convolution results from the optoelectronic calculation module and splicing the results to obtain the input for the next layer, or storing the spliced ​​results in the off-chip DDR memory. The present invention performs convolution operations based on the RS data stream method, maximizes data reuse, reduces access to the off-chip DDR, and improves energy efficiency.

Description

Technical Field

[0001] The present invention relates to a convolution hardware accelerator architecture based on row-fixed (RS) data stream and a method thereof, belonging to the technical field of hardware accelerated neural networks. Background Art

[0002] In recent years, with the development of computer science and internet technology, the amount of global data has exploded. Artificial intelligence has also evolved from early manual feature engineering to the ability to learn from vast amounts of data. Inspired by neuroscience, neural network models, developed over many years, have achieved significant success in machine learning and are widely used in fields such as computer vision, speech recognition, and natural language processing.

[0003] To handle complex problems or improve model accuracy, neural network models are becoming increasingly large, requiring extensive computing resources. Large-scale neural network algorithms pose challenges to the underlying hardware in terms of both energy efficiency and throughput. According to statistics, convolutional computations account for over 90% of the computational load in convolutional neural networks (CNNs). While CNN models can be deployed on highly parallel graphics processing units (GPUs) for training or inference, offering significant speed advantages, practical deployment of neural network algorithms remains limited due to factors such as power consumption and cost.

[0004] In order to better deploy neural network models on hardware systems, it is necessary to optimize computing resource consumption and the throughput loss caused by large-scale data access. How to reasonably design data flow and computing module architecture has become the focus of optimization. Summary of the Invention

[0005] To address the above technical issues, the present invention aims to provide a convolution hardware accelerator based on RS data streams to maximize data reuse during convolution calculations, reduce data handling, and improve system throughput and energy efficiency. The present invention also provides a method for utilizing this hardware accelerator.

[0006] The technical solution adopted in the present invention is:

[0007] A convolution hardware accelerator based on RS data stream, comprising:

[0008] Off-chip DDR memory, used to store raw image data and neural network inference result data;

[0009] On-chip cache module, used to store raw image data, convolution kernel weight data, and intermediate convolution calculation results read from off-chip DDR memory;

[0010] DDR controller, used to control data interaction between off-chip DDR memory and on-chip cache module;

[0011] The data distribution module is used to transmit the required weight and incentive data to the optoelectronic calculation module according to the current calculation configuration information and the calculation rules of the optoelectronic calculation module;

[0012] Optoelectronic computing module, used to complete the convolution operation of each layer of the network;

[0013] The result collection module is used to receive the convolution results of the optoelectronic computing module and splice the results to obtain the input of the next layer, or store the spliced ​​results in the off-chip DDR memory;

[0014] The controller is used to store configuration information and coordinate the operation of the DDR controller, data distribution module, optoelectronic computing module and result collection module according to the current operation status.

[0015] Furthermore, the data distribution module includes a data reading module FIFO buffer, a ping-pong register group and an address calculation unit. The data reading module FIFO buffer is connected to the on-chip cache module, and the address calculation unit is respectively connected to the ping-pong register group and the on-chip cache module; the data reading module FIFO buffer is used to realize buffering and bit width matching during data interaction between the off-chip DDR memory and the on-chip cache module, the ping-pong register group is used to store the weight data and incentive data read from the on-chip cache module, assemble and splice the data into the form required by the calculation module and output it, and the address calculation unit is used to control the address of reading and writing data to the on-chip cache module.

[0016] Furthermore, the photoelectric calculation module includes multiple photoelectric calculation arrays and accumulation modules; the photoelectric calculation array is used to complete the multiplication and accumulation operations of the weight window and the corresponding excitation window, and the accumulation module is used to accumulate the calculation results of the photoelectric calculation array.

[0017] Furthermore, the photoelectric computing array includes M convolution filter groups, each convolution filter group is an array composed of a group of computing units, each convolution filter group includes C channel groups, and each channel group contains N computing units; the M convolution filter groups share the same feature map data and input different convolution kernels; the C channel groups correspond to C input channels, and each channel inputs the corresponding channel data of the feature map and the convolution kernel; each computing unit in the channel group shares the same weight data.

[0018] The present invention also provides an acceleration method for a convolution hardware accelerator based on RS data stream, comprising the following steps:

[0019] 1) Before starting the calculation, the controller sends a read weight and read stimulus request to the DDR controller. The DDR controller moves the feature map data and convolution kernel data from the off-chip DDR memory to the on-chip cache module in the data distribution module.

[0020] 2) According to the current convolution calculation stage, the data distribution module calculates the storage address of the next batch of required excitation data and weight data in the on-chip cache module, and reads the next batch of excitation data and weight data into the data distribution module respectively;

[0021] 3) Expand and splice the weight and incentive data stored in the data distribution module into the format required by the optoelectronic calculation module interface, and distribute them to the optoelectronic calculation module according to the calculation order;

[0022] 4) After the optoelectronic computing module completes the calculation, the result collection module receives and stores the convolution calculation result; if the received result is the last layer of convolution, the controller sends a write result request to the DDR controller to save the final result to the off-chip DDR memory.

[0023] Furthermore, the feature map is stored in the on-chip cache module in the order of column-channel-row, that is, one address stores the excitation of one row of one channel; the data of all channels of one row of a block is stored first, and then the data of the next row is stored; the excitation of each layer is stored in a different on-chip cache module, and the convolution kernel is stored in the on-chip cache module according to the convolution filter, one address stores the weights of C channels of a convolution filter, and the weights of all channels of a convolution filter are stored first, and then the next convolution filter is stored.

[0024] Furthermore, in step 2), the data distribution module includes a data reading module FIFO buffer, a ping-pong register group and an address calculation unit, the ping-pong register group is divided into an excitation ping-pong register group and a weight ping-pong register group, the excitation ping-pong register group is used to store excitation data, and the weight ping-pong register group is used to store weight data; wherein, the weight data needs to update C channels of M convolution kernels each time, and the weight data of M addresses needs to be read; assuming that the convolution kernel size is K, the excitation data requires excitation data of K*C addresses each time it is updated.

[0025] Furthermore, in step 3), the photoelectric computing module includes multiple photoelectric computing arrays and accumulation modules, the photoelectric computing array includes M convolution filter groups, each convolution filter group is an array composed of a group of computing units, the data distribution module distributes data from the ping-pong register group to the photoelectric computing array, and the specific distribution order is: for weight data, the data distribution module first distributes C channel weight data of M convolution kernels of a layer of convolution, and then distributes the next C channel weight data of the current M convolution kernels; after the distribution of all channel weight data of the current M convolution kernels is completed, the weight data of the next M convolution kernels are distributed; wherein the distributed M convolution kernel weight data are respectively input into the convolution filter group corresponding to the photoelectric computing module; the weight data of the C channels in a distributed convolution kernel weight are respectively input into the corresponding channel group in the convolution filter group; all computing units in the same channel group of the convolution filter group share the same weight data; for excitation data, the data distribution module first distributes feature Figure 1 The C channels of the first row are distributed, and then the next C channels are distributed; when all channels of the output result row are calculated, the excitation data of the next row are distributed; the distributed excitation data are copied M times and input into the M convolution filter groups of the photoelectric calculation module respectively; the C channels of the distributed excitation data are input into the C channel groups in the convolution filter group respectively

[0026] The proposed convolution hardware accelerator based on RS data streams allows for the rational design of convolutional computing hardware architectures based on hardware resources and the parameters of the deployed neural network model. By leveraging data reusability to varying degrees, different neural network algorithms can be flexibly mapped to the hardware. This approach improves system throughput while ensuring high hardware resource utilization, enabling hardware acceleration of neural network models. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a structural block diagram of a convolution hardware accelerator based on RS fixed data stream of the present invention;

[0028] Figure 2 It is an operation flow chart of the convolutional neural network accelerator of the present invention;

[0029] Figure 3 It is the process of one-dimensional convolution;

[0030] Figure 4 It is a schematic diagram of the data flow of a two-dimensional convolution operation of a logical PE set;

[0031] Figure 5 : is a schematic diagram of the CNN network model in this embodiment;

[0032] Figure 6 Schematic diagram of the PE computing array structure in this embodiment;

[0033] Figure 7 It is a diagram of data distribution. DETAILED DESCRIPTION

[0034] The present invention will be described in detail below with reference to the accompanying drawings.

[0035] like Figure 1 The block diagram of the RS-based fixed-data stream convolution hardware accelerator shown in the figure includes a controller, a DDR controller, an on-chip data cache unit (RAM), an off-chip DDR memory, a data distribution module, an optoelectronic computing module, and a result collection module. The off-chip DDR memory is used to store raw image data and neural network inference results. The on-chip data cache unit is used to store raw image data, convolution kernel weight data, and intermediate convolution calculation results read from the off-chip DDR memory. The DDR controller is used to control data exchange between the off-chip DDR memory and the on-chip data cache unit. The optoelectronic computing module is used to perform convolution operations at each layer of the network and includes multiple optoelectronic computing arrays (PE computing arrays), each of which includes multiple PE computing units. The controller is connected to the DDR controller, the data distribution module, the PE computing array module, and the result collection module to store configuration information and coordinate the operation of each module based on the current computing status. The data distribution module is connected to the PE computing array and the on-chip data cache unit to transmit the required weight and excitation data to the optoelectronic computing module based on the current computing configuration information and the calculation rules of the optoelectronic computing module. The result collection module is connected to the PE computation array and the on-chip data cache unit. It receives the convolution results from the optoelectronic computation module and concatenates them to form the input for the next layer, or stores the concatenated results in off-chip DDR memory. The on-chip data cache unit includes a weight cache unit and an excitation cache unit. The DDR controller accesses the off-chip DDR memory via a bus and reads the weight and excitation data into the corresponding cache units. The data distribution module includes a data read module FIFO buffer, a ping-pong register set, and an address calculation unit. The data read module FIFO buffer is used to implement buffering and bit width matching when data is exchanged between the off-chip DDR memory and the on-chip data cache unit. The ping-pong register set is used to store the weight and excitation data read from the on-chip data cache unit, assemble and concatenate the data into the format required by the computation module, and output it. The address calculation unit controls the addresses for reading and writing data to and from the on-chip data cache unit. The data read module FIFO buffer is connected to the write data line of the on-chip data cache unit, the ping-pong register set is connected to the read data line of the on-chip data cache unit, and the address calculation unit is connected to the read and write address lines of the on-chip data cache unit.

[0036] Figure 2The figure shows the operation flow chart of the above-mentioned convolutional neural network accelerator. After the system is started, the controller controls the entire calculation process through state machine state transitions based on the configuration information to ensure the coordinated operation of each module. First, after the controller configuration is completed, it enters the preparation state. The DDR controller transfers all weight data and a block of the input image to the on-chip data cache unit. The controller controls the data distribution module to send excitation and weight data to the PE calculation array according to the calculation order of the PE calculation array. The PE calculation array performs convolution operations in sequence. At the same time, the result collection module receives and stores the calculation results of the PE calculation array. After all the convolution calculations of the current layer are completed, it is determined whether it is the last layer of convolution. If not, the convolution results integrated by the result collection module will be used as input to start the next layer of convolution calculation; if so, the results will be transferred to the off-chip DDR memory.

[0037] Figure 3 The figure shows the processing process of a one-dimensional convolution. Convolution operations in neural network models for computer vision are generally three-dimensional convolution operations. They can be decomposed into a combination of multiple two-dimensional convolution operations, and the basic unit of a two-dimensional convolution operation is a one-dimensional convolution operation. Figure 4 The figure shows a schematic diagram of the data flow for a two-dimensional convolution operation within a logical PE set. First, a one-dimensional convolution operation is mapped to a PE computing unit within the logical PE set. Multiple PE units form a logical PE set. A row of weight data is reused horizontally within the logical PE set, a row of input excitations is reused diagonally within the logical PE set, and the logical PE sets are accumulated vertically to produce a row of convolution output results. In a neural network model, the input feature map of a convolution layer has C channels and M convolution kernels. Therefore, completing the convolution operation for a single feature map requires C*M logical PE sets. The M sets can share the same feature map, and the multiplication and addition results of the C sets are then added together through an accumulation module. Based on the specific parameters of the deployed neural network model and the design register size, the logical PE sets are folded and spatially mapped in the M and C dimensions to determine the design size of the physical PE computing array.

[0038] Figure 5This is a schematic diagram of the CNN network model in this embodiment. There are four layers of convolution. The first layer has an input feature map size of W1*H1*4 channels, a convolution kernel of 3*3*4 channels, and 16 convolution kernels. The second layer has an input feature map size of W2*H2*16 channels, a convolution kernel of 3*3*16 channels, and 32 convolution kernels. The third layer has an input feature map size of W3*H3*32 channels, a convolution kernel of 3*3*32 channels, and 32 convolution kernels. The fourth layer has an input feature map size of W4*H4*32 channels, a convolution kernel of 3*3*32 channels, and 32 convolution kernels. According to the above-mentioned convolution mapping method based on RS data stream, calculating one layer of convolution requires at most C*N logical PE sets, where C is the number of input feature map channels and N is the number of convolution kernels. Combined with hardware computing resources and register resources, the logical PE sets are folded and spatially mapped. Figure 6 The figure shows a schematic diagram of the PE computing array structure in this embodiment. The PE array is primarily responsible for convolution operations in multi-layer networks. In the PE array design, the C*N logical PE sets are folded into 4 in the C dimension and folded into 4 in the N dimension. The PE array includes 4 convolution filter groups, each of which includes 4 channels. The 4 convolution filter groups input different convolution kernels and share the same feature map data. The feature map data is reused 4 times. The 4 channels are accumulated in the channel direction. Each channel is divided into 8 parts, each part shares a part of a row of convolution, and 8 parallel operations are performed to reduce calculation time. The 8 parts share the same weights, and the weight data is reused 8 times. During calculation, the 3 rows of excitation data for the first 4 channels of the input convolution feature map layer and the weights of the first 4 channels of the first 4 groups of convolution kernels are first calculated. The accumulation modules of the 4 convolution filter groups respectively accumulate and store the convolution results of the 4 channels. Then, the data of the 5th to 8th channels of the feature map and the 5th to 8th channels of the first 4 groups of convolution kernels are calculated. Similarly, after all channels are calculated, the accumulation module of the four convolution filter groups outputs the convolution results for one row of output channels 1 to 4. Then, the convolution kernels of the fifth to eighth groups are replaced. After calculating the convolution results for one row of all output channels, the next row of convolution is calculated. This process continues in this way to complete all convolution calculations for this layer.

[0039] Figure 7The figure shows a schematic diagram of the data distribution module, which includes the weight distribution module and the stimulus distribution module. The weight distribution module consists of a weight cache unit, a weight address calculation module, and a weight ping-pong register bank. When configured, the controller transfers all weights from the DDR memory to the weight cache unit. Weights are stored in the cache unit by filter, with the weights for all channels of the first filter stored first, followed by the weights for the second filter, and so on. During the computation process, the weights used in each calculation of the PE compute array are converted into the weight matrix required by the PE compute array and sent to the PE compute array in parallel. The next batch of required weights is read from RAM and stored in the weight ping-pong register bank. For the second layer, for example, the input feature map is W2*H2*16, the convolution kernel has 3*3*16 channels, and the number of convolution kernels is 32. The weight distribution module first distributes the weights for channels 1 to 4 of the first to fourth convolution kernels, then distributes the weights for channels 5 to 8, and so on. Once all channel weights have been distributed, distribution begins for the fifth to eighth convolution kernels, and so on until all convolution kernel data has been distributed. The excitation distribution module includes an excitation cache unit, an excitation address calculation module, and an excitation ping-pong register group. The excitation distribution module is mainly responsible for retrieving the excitation data updated in an excitation update from the excitation cache unit according to the calculation order of the calculation module during the calculation process, and expanding the excitation data into the excitation matrix form required for the PE calculation array operation. The excitation distribution uses a ping-pong reg group to match the PE calculation speed. The four convolution filter groups share the same excitation data. The eight parts in the same channel group in a convolution filter group are jointly responsible for the calculation of a row of excitation data and convolution weights. First, channels 1 to 4 of a row of excitation are distributed, followed by channels 5 to 8, until the calculation of the four output channels of a row is completed. The above distribution process is repeated. After all the output channels of a row are calculated, the next row of excitation data is distributed.

[0040] The present invention performs convolution operations based on the RS data stream mode, maximizes data reuse, reduces access to off-chip DDR, and improves energy efficiency.

Claims

1. An acceleration method for a convolution hardware accelerator based on RS data stream, wherein the accelerator includes an off-chip DDR memory, an on-chip cache module, a DDR controller, a data distribution module, an optoelectronic computing module, a result collection module, and a controller, wherein the optoelectronic computing module includes M convolution filter groups, each of which is an array consisting of a group of computing units, each of which includes C channel groups, each of which contains N computing units; characterized in that: The method comprises the following steps: 1) Before starting calculations, the controller sends read weight and read stimulus requests to the DDR controller. The DDR controller moves the feature map data and convolution kernel data from the off-chip DDR memory to the on-chip cache module in the data distribution module. The feature map is stored in the on-chip cache module in the order of column-channel-row, that is, one address stores one row of stimulus for one channel. The data of all channels in one row of a block is stored first, and then the data of the next row is stored. The stimulus of each layer is stored in a different on-chip cache module. The convolution kernel is stored in the on-chip cache module according to the convolution filter. One address stores the weights of the C channels of a convolution filter. The weights of all channels of a convolution filter are stored before the next convolution filter. 2) Based on the current convolution calculation stage, the data distribution module calculates the storage address of the next batch of required excitation data and weight data in the on-chip cache module, and reads the next batch of excitation data and weight data into the data distribution module respectively; 3) Expand and splice the weight and incentive data stored in the data distribution module into the format required by the optoelectronic calculation module interface, and distribute them to the optoelectronic calculation module according to the calculation order; the specific distribution order is: For weight data, first distribute the C channel weight data of the M convolution kernels of a layer of convolution, and then distribute the next C channel weight data of the current M convolution kernels; after the distribution of all channel weight data of the current M convolution kernels is completed, distribute the weight data of the next M convolution kernels; the distributed M convolution kernel weight data are respectively input into the convolution filter group corresponding to the optoelectronic calculation module; the weight data of the C channels in a distributed convolution kernel weight are respectively input into the corresponding channel group in the convolution filter group; all calculation units in the same channel group of the convolution filter group share the same weight data; For the excitation data, the C channels of a row of the feature map are distributed first, and then the next C channels are distributed. After all channels of a row of the output result are calculated, the excitation data of the next row is distributed. The distributed excitation data is copied M times and input into the M convolution filter groups of the optoelectronic calculation module respectively. The C channels of the distributed excitation data are input into the C channel groups in the convolution filter group respectively. 4) After the optoelectronic computing module completes the calculation, the result collection module receives and stores the convolution calculation result. If the received result is the last layer of convolution, the controller sends a write result request to the DDR controller to save the final result to the off-chip DDR memory.

2. The acceleration method of the convolution hardware accelerator based on RS data stream according to claim 1, characterized in that: In step 2), the data distribution module includes a data reading module FIFO buffer, a ping-pong register group and an address calculation unit. The ping-pong register group is divided into an excitation ping-pong register group and a weight ping-pong register group. The excitation ping-pong register group is used to store excitation data, and the weight ping-pong register group is used to store weight data. The weight data needs to update C channels of M convolution kernels each time, and the weight data of M addresses needs to be read. Assuming that the convolution kernel size is K, the excitation data needs to be updated with excitation data of K*C addresses each time.

Images