A convolutional neural network acceleration computing system for bandwidth adaptive data transfer

Abstract

The application relates to a bandwidth-adaptive data transmission convolutional neural network acceleration computing system, comprising a convolutional neural network accelerator for performing specific data calculation; and a bandwidth-adaptive data transmission device arranged between the convolutional neural network accelerator and an external storage device, which senses data consumption and generation rate of the convolutional neural network accelerator, senses occupation of the external storage device bandwidth, determines a data transmission strategy between the convolutional neural network accelerator and the external storage device according to the sensed data, and comprises a data loading module and a data unloading module. Through the above structure, the off-chip storage bandwidth can be dynamically adapted, the continuity of data transmission can be greatly improved, and thus the off-chip data transmission bandwidth of the convolutional neural network accelerator can be effectively improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of special algorithm hardware circuit design, and particularly relates to a bandwidth adaptive data transmission convolutional neural network acceleration computing system. BACKGROUND

[0002] Deep learning algorithms represented by convolutional neural networks (CNN) have continuously improved performance in image classification, target detection and other fields. The massive computation and storage of deep learning algorithms have given rise to many deep learning processors based on ASIC or FPGA or more specialized CNN accelerators.

[0003] The performance of CNN accelerators is mainly limited by the upper limit of off-chip storage bandwidth and the upper limit of on-chip computing resources. However, due to the long-term lag of the development speed of storage technology behind the development speed of processor technology, the former is often the main factor limiting the performance of CNN accelerators.

[0004] Taking a single convolutional layer of a CNN as an example, multiple read and write operations of input feature data, weight bias data and output feature data are required. These read and write operations usually have strong burstiness and aggregation, which can easily cause the off-chip storage bandwidth to be blocked at times and idle at times, resulting in the limited bandwidth not being fully utilized and thus slowing down the overall performance of the system. SUMMARY

[0005] The present application is proposed based on the above-mentioned needs of the prior art, and the technical problem to be solved by the present application is to provide a bandwidth adaptive data transmission convolutional neural network acceleration computing system to solve the problem of low utilization efficiency of off-chip storage bandwidth in a convolutional neural network acceleration computing system.

[0006] To solve the above problems, the technical solution provided by the present application comprises:

[0007] A bandwidth adaptive data transmission convolutional neural network acceleration computing system, characterized in that it comprises a convolutional neural network accelerator for performing specific data computation, and a bandwidth adaptive data transmission device arranged between the convolutional neural network accelerator and an external storage device, which senses the data consumption of the convolutional neural network accelerator and generates a rate, and senses the occupancy of the bandwidth of the external storage device; determines the data transmission strategy between the convolutional neural network accelerator and the external storage device according to the sensed data; the bandwidth adaptive data transmission device comprises a data loading module for loading data in a specified position in the off-chip external storage unit to the convolutional neural network accelerator; and a data unloading module for unloading data generated by the convolutional neural network accelerator to a specified position in the external storage unit.

[0008] Preferably, a FIFO data interface is adopted between the convolutional neural network accelerator and the bandwidth adaptive data transmission device; and an AXI data interface is adopted between the convolutional neural network accelerator and the external storage device.

[0009] Preferably, the convolutional neural network accelerator comprises a calculation unit for performing specific data calculation, an on-chip cache unit for obtaining data for calculation from the external storage device, and a register group arranged between the on-chip cache unit and the calculation unit and configured to store the data for calculation.

[0010] Preferably, the data loading unit comprises an instruction configuration module configured to receive a control instruction from an external host, parse and calculate instruction information, and generate configuration information of each other module; an address automatic generation module configured to generate address information for guiding the driving module to read a data block from the external storage unit according to the configuration information of the instruction module; a data sensing module configured to sense a data change in a data channel between the bandwidth adaptive data transmission unit and the convolutional neural network accelerator and the external storage unit according to the configuration information of the instruction configuration module; an automatic interpolation module configured to receive data from the external storage unit and insert a predetermined value into a predetermined position of a data stream according to the configuration information generated by the instruction configuration module; and a data driving module configured to drive a data interface to transmit data according to the configuration information of the instruction configuration module; when the driving module receives data, the FULL signal state of a communication line between the convolutional neural network accelerator and the bandwidth adaptive data transmission unit is judged, and the data stream with the predetermined value inserted is transmitted to the convolutional neural network accelerator when the FULL signal is not high.

[0011] Preferably, the data loading unit comprises an instruction configuration module configured to receive a control instruction from an external host, parse and calculate instruction information, and generate configuration information of each other module; an address automatic generation module configured to generate address information for guiding the driving module to read a data block from the external storage unit according to the configuration information of the instruction module; a data sensing module configured to sense a data change in a data channel between the bandwidth adaptive data transmission unit and the convolutional neural network accelerator and the external storage unit according to the configuration information of the instruction configuration module; and a data driving module configured to drive a data interface to transmit data according to the configuration information of the instruction configuration module.

[0012] Preferably, the data driving module of the loading unit comprises an AXI data sensing module configured to sense an AXI bus data transmission bandwidth occupation situation according to the configuration information of the instruction configuration module and inform the AXI driving module; in the data loading mode, the duty cycle D R of an AXI bus RVALID signal and a change rate of the duty cycle are detected.

[0013] Preferably, the data driving module of the unloading unit comprises an AXI data sensing module, which senses the data transmission bandwidth occupation on the AXI bus according to the configuration information of the instruction configuration module, and informs the AXI driving module; in the data unloading mode, the duty cycle D of the AXI bus WREADY signal is detected W , and the change rate of the duty cycle.

[0014] Preferably, the data sensing module of the loading unit comprises an AXI data sensing module, which senses the data transmission bandwidth occupation on the AXI bus according to the configuration information of the instruction configuration module, and informs the AXI driving module; in the data loading mode, the duty cycle D of the AXI bus RVALID signal is detected R , and the change rate of the duty cycle; a FIFO data sensing module senses the change of data in the FIFO according to the configuration information of the instruction configuration module, and informs the data AXI driving module and the FIFO driving module; in the data loading mode, the duty cycle F of the FIFO is sensed e , and the change rate of the duty cycle.

[0015] Preferably, the data sensing module of the unloading unit comprises an AXI data sensing module, which senses the data transmission bandwidth occupation on the AXI bus according to the configuration information of the instruction configuration module, and informs the AXI driving module; in the data unloading mode, the duty cycle D of the AXI bus WREADY signal is detected W , and the change rate of the duty cycle; a FIFO data sensing module senses the change of data in the FIFO according to the configuration information of the instruction configuration module, and informs the data AXI driving module and the FIFO driving module; in the data unloading mode, the duty cycle F of the FIFO is sensed f , and the change rate of the duty cycle.

[0016] Preferably, the external storage device comprises a DDR.

[0017] By sensing the data consumption (generation) rate of the CNN accelerator, sensing the occupation of the off-chip storage bandwidth, improving the continuity of data transmission, and other means, the off-chip data transmission efficiency of the convolutional neural network accelerator is improved. In addition, the data transmission module improves the calculation continuity by automatic interpolation, which can reduce the efficiency loss caused by the first delay of the convolutional neural network accelerator calculation unit. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is the structure diagram of the convolutional neural network acceleration calculation system in the embodiment;

[0019] Figure 2 is the structure diagram of the data loading unit in the embodiment;

[0020] Figure 3 is a structure diagram of a data offloading unit in the embodiment;

[0021] Figure 4 is a size annotation and block diagram of a convolutional layer input feature map. DETAILED DESCRIPTION

[0022] The embodiment provides a convolutional neural network acceleration computing system with bandwidth adaptive data transmission. The convolutional neural network acceleration computing system is mainly used for computing a large amount of data in a convolutional neural network, for example, computing a convolution kernel of the convolutional neural network, and plays a role of accelerating the convolutional neural network.

[0023] As shown in Figure 1 , the convolutional neural network acceleration computing system with bandwidth adaptive data transmission in the embodiment includes a convolutional neural network accelerator and a bandwidth adaptive data transmission unit.

[0024] The convolutional neural network accelerator includes a computing unit configured to perform specific data computation, for example, computation of a convolution kernel, and the data computation unit can include one or more computing modules using different kinds of data computation methods.

[0025] An on-chip cache unit is configured to obtain data for computation from an external storage device, for example, from a DDR memory. For example, the data includes but is not limited to feature map data, convolution kernel data, weight data, and the like.

[0026] A register group is arranged between the on-chip cache unit and the computing unit, and registers various data for computation.

[0027] A bandwidth adaptive data transmission device is connected to the computing unit through a FITO data bus, and includes a data loading unit and a data offloading unit. The bandwidth adaptive data transmission device is connected to a DDR through an AXI data bus. The data loading unit is configured to load data at a specified position in the off-chip DDR to the convolutional neural network accelerator. The data offloading unit is configured to offload data generated by the convolutional neural network accelerator to the specified position in the off-chip DDR.

[0028] The overall architecture of the data loading unit is shown in Figure 2 , and includes an instruction configuration module, an address automatic generation module, a FIFO data sensing module, an AXI data sensing module, an AXI driving module, a FIFO driving module, and an automatic interpolation module.

[0029] The data offloading unit has an overall architecture as shown in Figure 3As shown, it mainly consists of an instruction configuration module, an automatic address generation module, a FIFO data sensing module, an AXI data sensing module, an AXI driver module, and a FIFO driver module. The automatic address generation module can operate in output feature offloading mode, while the FIFO data sensing module, AXI data sensing module, AXI driver module, and FIFO driver module operate in data offloading mode.

[0030] The instruction configuration module is used to receive control instructions from the external master controller, parse and process the instruction information, configure the base address A0, data block length generation information and offset address generation information of the automatic address generation module, configure the data burst read / write length B and transmission interval unit time t of the AXI driver module, configure the interpolation point and interpolation value of the automatic interpolation module, and configure the sensing signal type and signal fusion method of the AXI data sensing module and FIFO data sensing module.

[0031] The address auto-generation module generates the starting address and data block length for each data transfer based on the configuration information of the instruction module, guiding the AXI driver module to read data blocks from specific locations in DDR. In convolutional neural network accelerator applications, the address auto-generation module has multiple modes, including input feature loading mode, weight data loading mode, and output feature unloading mode. Specific modes can be added or modified according to application requirements.

[0032] For example, in a typical input feature loading mode, the length L of each transmitted data block will be determined based on the width W of the input feature map and the number of blocks T in the height direction of the input feature map. h The default height H of the input feature map row blocks. t The final height H of the input feature map row block t The data block starting address A is calculated using Formula 1; the starting address A will be determined based on the base address A0, the width W of the input feature map, the height H of the input feature map, and the height H of the overlapping region of the input feature map. o The data width *m* is calculated using Formula 2. The specific meanings of the above parameters and symbols are as follows: Figure 4 As shown.

[0033]

[0034] A = A0 + ((id) ch -1)×W×H+(id h -1)×W×(H t -H o ))×m (Formula 2)

[0035] Among them, id ch and ID h These are the block IDs for the input feature map in the channel direction and height direction, respectively.ch Count from 1, id ch is incremented by 1; when id ch reaches the number of blocks T in the input feature map channel direction, id ch is reset to 1, and idh ch is incremented by 1; when id h reaches the number of blocks T in the input feature map height direction, idh h is reset to 1, and the entire layer of input feature data transmission is completed.

[0036] The calculation method of the data block length L and the data block head address A in other modes is similar to the above example.

[0037] The AXI data sensing module senses the data transmission bandwidth occupation on the AXI bus according to the configuration information of the instruction configuration module, and informs the AXI drive module. In the data loading mode, the module detects the duty cycle D R of the AXI bus RVALID signal and the change rate of the duty cycle; in the data unloading mode, the module detects the duty cycle D W of the AXI bus WREADY signal and the change rate of the duty cycle. The smaller the RVALID signal duty cycle, the more the DDR is in the read data busy state; the smaller the WREADY signal duty cycle, the more the DDR is in the write data busy state. The change rate of the duty cycle refers to the change degree of the duty cycle per unit time, for the sake of simplifying the hardware logic, only the duty cycle D1 of the first k clock cycles and the duty cycle D2 of the last k clock cycles after the data starts to be transmitted are counted, and the change rate V D of the duty cycle can be represented by 1 bit, and the calculation method is obtained by the following formula 3.

[0038]

[0039] V D equals 1 indicates that the bandwidth of the AXI bus is gradually idle, and vice versa.

[0040] The FIFO data sensing module senses the change of the data in the FIFO according to the configuration information of the instruction configuration module, and informs the data AXI drive module and the FIFO drive module. In the data loading mode, the module senses the occupancy rate F e of the FIFO and the change rate of the occupancy rate; in the data unloading mode, the module senses the occupancy rate F f of the FIFO and the change rate of the occupancy rate. The occupancy rate represents the proportion of the depth of the un-stored data to the total depth of the FIFO; the occupancy rate represents the proportion of the data storage depth to the total depth of the FIFO. The change rate V FThe duty cycle F1 of the first f clock cycles and the duty cycle F2 of the last f clock cycles can be calculated according to the following formula 4.

[0041]

[0042] V F 1 indicates that the data transmission rate is less than the calculated rate, and vice versa.

[0043] The AXI driving module drives the AXI interface to transmit data according to the configuration information of the instruction configuration module, the data transmission start address A and the data block length L of the address automatic generation module, the AXI bus bandwidth occupation (the duty cycle D of the RVALID / WVALID signal) of the AXI data sensing module, the calculation of the transmission time N of the data transmission request, and the calculation of the transmission time N of the data transmission request. R and D W , and the duty cycle change rate V D ), the occupation of the FIFO data sensing module (the duty cycle F f or the duty cycle F e , the change rate V F of the duty cycle), the calculation of the transmission time N of the data transmission request, and the calculation of the transmission time N of the data transmission request.

[0044]

[0045] wherein, is a rounding up operation, B is the AXI burst read / write length, and the upper limit is 256. Generally, the larger the value of B is, the higher the data transmission efficiency is. In the above formula, L is usually much larger than B, so the efficiency of most data transmission requests can be guaranteed. That is, the data transmission unit can improve the continuity of data transmission by this means, thereby improving the transmission efficiency.

[0046] In the data loading mode, the time interval T L between two data transmission requests can be calculated according to formula 6.

[0047] T L = t x (2-D R x V D -F e x V F ) (formula 6)

[0048] The more idle the AXI bus is and the emptier the FIFO is, the shorter the transmission interval time is; otherwise, the longer the time is.

[0049] In the data unloading mode, the time interval T D between two data transmission requests can be calculated according to formula 7.

[0050] T D = t x (2-D W x V D -F f x V F ) (Formula 7)

[0051] The more idle the AXI bus is, the fuller the FIFO receiving data is, and the shorter the transmission interval time is; otherwise, the time is longer.

[0052] The automatic interpolation module only exists in the data loading unit. Data is obtained from the AXI driving module, and according to the configuration information of the instruction configuration unit, the specified value is inserted into the specified position, and the interpolated data is punched into the FIFO driving module.

[0053] The FIFO driving module is driven by data, and does not need configuration information. In the data loading mode, the data of the automatic interpolation module is received, and the data is punched into the FIFO under the condition that the FULL signal of the FIFO is not high. In the data unloading mode, the data is read under the condition that the EMPTY signal of the FIFO is not high, and punched into the AXI driving module.

[0054] The working method of the convolutional neural network acceleration computing system with the above bandwidth adaptive data transmission is as follows:

[0055] Firstly, the basic information of the input feature map is written into the instruction configuration module by the host, including the height H of the feature map, the width W of the feature map, etc.; the basic information of the feature map blocking, including the number of blocking T h in the height direction, the number of blocking T ch in the channel direction, the overlap region height H o of the input feature map, the data width m, etc.; the data transmission related information, including the DDR base address A0, the AXI burst read-write length B, the transmission interval unit time t, etc.

[0056] Secondly, the address automatic generation module generates the data block length L, the first address A, the ID id ch of the data block in the channel direction and the ID id h of the data block in the height direction according to the configuration information, and informs the AXI driving module of the information.

[0057] Thirdly, the AXI driving module initiates the read operation of the first data block according to the configuration information and the information of the address automatic generation module, and punches the data into the automatic interpolation module.

[0058] Fourthly, after the AXI interface starts data transmission, the AXI data sensing module changes the duty cycle D R of the RVALID signal according to the configuration information, and the change rate VD The system performs a detection and sends the information back to the AXI driver module.

[0059] The fifth step is that after receiving the data, the automatic interpolation module inserts a specific value into a specific position in the data stream according to the configuration information, and then pushes the interpolated data into the FIFO driver module.

[0060] Step 6: After receiving the data, the FIFO driver module will push the data into the FIFO if the FULL signal of the FIFO is low.

[0061] Step 7: Simultaneously with the FIFO driver module starting to work, the FIFO data sensing module, based on the configuration information, determines the FIFO's duty cycle F. e and the rate of change of duty cycle V F The system performs a test and then feeds this information back to the AXI driver module and the FIFO driver module.

[0062] Step 8: After the AXI driver module completes the initial data read, it determines the data based on the configuration information and the FIFO duty cycle F. e The rate of change of duty cycle V F Duty cycle DR and duty cycle rate of change V of AXI RVALID signal D Calculate the time interval T between transmission requests. L And according to the interval time T L Initiate the next data transfer until the transfer of N data blocks is complete.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention based on the concept of the present invention, without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A bandwidth-adaptive data transmission acceleration system for convolutional neural networks, characterized in that, Including convolutional neural network accelerators for performing specific data computations; and A bandwidth-adaptive data transmission device is disposed between the convolutional neural network accelerator and the external storage device. It senses the data consumption and generation rate of the convolutional neural network accelerator and the bandwidth occupancy of the external storage device. Based on the sensed data, it determines the data transmission strategy between the convolutional neural network accelerator and the external storage device. The bandwidth adaptive data transmission device includes: a data loading unit for loading data from a specified location in an off-chip external storage unit to the convolutional neural network accelerator; and a data unloading module for unloading data generated by the convolutional neural network accelerator to a specified location in the external storage unit. The data loading unit includes: an instruction configuration module, which receives control instructions from an external master controller, parses and processes the instruction information, and generates configuration information for each other module; an address automatic generation module, which generates address information to guide the driver module to read data blocks from the external storage unit based on the configuration information of the instruction module; a data perception module, which senses the data changes in the data channel between the bandwidth adaptive data transmission unit, the convolutional neural network accelerator, and the external storage unit based on the configuration information of the instruction configuration module; an automatic interpolation module, which receives data from the external storage unit and inserts predetermined values ​​into predetermined positions in the data stream according to the configuration information generated by the instruction configuration module; and a data driving module, which drives the data interface to transmit data according to the configuration information of the instruction configuration module. When the driving module receives data, it determines the FULL signal status of the communication line between the convolutional neural network accelerator and the bandwidth adaptive data transmission unit. When the FULL signal is not high, it sends the data stream with the inserted predetermined values ​​to the convolutional neural network accelerator. The data driving module includes an AXI data sensing module, which senses the data transmission bandwidth occupancy on the AXI bus based on the configuration information from the instruction configuration module and informs the AXI driving module; in data loading mode, it detects the duty cycle D of the AXI bus RVALID signal. R And the rate of change of the duty cycle.

2. The convolutional neural network accelerated computing system with bandwidth adaptive data transmission according to claim 1, characterized in that, The convolutional neural network accelerator and the bandwidth adaptive data transmission device are connected via a FIFO data interface; the bandwidth adaptive data transmission device and the external storage device are connected via an AXI data interface.

3. The convolutional neural network accelerated computing system with bandwidth adaptive data transmission according to claim 1, characterized in that, The convolutional neural network accelerator includes: The computing unit is used to perform specific data calculations; the on-chip cache unit is used to cache data obtained from external storage devices or generated by the computing unit; the register group is located between the on-chip cache unit and the computing unit to store data used for calculation.

4. The convolutional neural network accelerated computing system with bandwidth adaptive data transmission according to claim 1, characterized in that, The data offloading unit includes: The instruction configuration module receives control instructions from the external master controller, parses and processes the instruction information, and then generates configuration information for each other module. The address automatic generation module generates address information to guide the driver module in reading data blocks from external storage units based on the configuration information of the instruction module. The data perception module senses the data changes in the data channel between the bandwidth adaptive data transmission unit, the convolutional neural network accelerator, and the external storage unit based on the configuration information of the instruction configuration module. The data-driven module drives the data interface to transmit data based on the configuration information of the instruction configuration module.

5. The convolutional neural network accelerated computing system with bandwidth adaptive data transmission according to claim 4, characterized in that, The data driving module of the unloading unit includes an AXI data sensing module, which senses the data transmission bandwidth occupancy on the AXI bus according to the configuration information of the instruction configuration module and informs the AXI driving module; in data unloading mode, this module detects the duty cycle D of the AXI bus WREADY signal. W And the rate of change of the duty cycle.

6. The convolutional neural network accelerated computing system with bandwidth adaptive data transmission according to claim 1, characterized in that, The data sensing module of the loading unit includes: The AXI data sensing module, based on the configuration information from the instruction configuration module, senses the data transmission bandwidth occupancy on the AXI bus and informs the AXI driver module; in data loading mode, it detects the duty cycle D of the AXI bus RVALID signal. R And the rate of change of the duty cycle; The FIFO data sensing module, based on the configuration information from the instruction configuration module, senses changes in the data within the FIFO and informs the data AXI driver module and the FIFO driver module; in data loading mode, it senses the FIFO's duty cycle F. e And the rate of change of duty cycle.

7. The convolutional neural network accelerated computing system with bandwidth adaptive data transmission according to claim 4, characterized in that, The data sensing module of the unloading unit includes: The AXI data sensing module, based on the configuration information from the instruction configuration module, senses the data transmission bandwidth occupancy on the AXI bus and informs the AXI driver module; in data offload mode, it detects the duty cycle D of the AXI bus WREADY signal. W And the rate of change of the duty cycle; The FIFO data sensing module, based on the configuration information from the instruction configuration module, senses changes in the data within the FIFO and informs the data AXI driver module and the FIFO driver module; in data offloading mode, it senses the FIFO occupancy rate F. f And the rate of change of the duty cycle.

8. The convolutional neural network accelerated computing system with bandwidth adaptive data transmission according to claim 1, characterized in that, The external storage device includes DDR.

Images