A reconfigurable sparse neural network accelerator

Abstract

The application provides a reconfigurable sparse neural network accelerator, which is characterized by comprising an input feature map storage module, an instruction storage module, a weight storage module, a bias storage module, an output feature map storage module, a feature map Bitmap storage module, a control module, a calculation control module, an address mapping module, a PE array module and a PU general data processing module. The control module is connected with the input feature map storage module, the instruction storage module, the weight storage module, the bias storage module and the feature map Bitmap storage module; the calculation control module is connected with the input feature map storage module, the instruction storage module, the weight storage module, the bias storage module, the output feature map storage module, the feature map Bitmap storage module and the PE array operation module; the address mapping module is connected with the input feature map storage module, the feature map Bitmap storage module and the output feature map storage module; the PE array operation module is connected with the input feature map storage module, the output feature map storage module, the weight storage module and the bias storage module; and the PU general data processing module is connected with the instruction storage module, the output feature map storage module and a DDR module. The application can support multiple convolution calculations with less resources to improve the versatility of the neural network accelerator.

Description

Technical Field

[0001] The present invention relates to the field of neural network accelerators, and in particular to a reconfigurable sparse neural network accelerator. Background Art

[0002] With the development of convolutional neural networks, they have been widely used in fields such as computer vision, speech recognition, natural language processing, and autonomous driving. Convolutional neural networks contain a large number of parameters, all of which are required to compute the neural network model. However, some parameters become zero as the number of neural network layers increases. These zero values, when multiplied by the weights during the convolution process, always result in zero, making them unnecessary. To accelerate the computation of neural network models, it is necessary to consider how to skip the computation of these zero values ​​to improve efficiency. Furthermore, convolutional neural network computations are divided into numerous types, such as pointwise convolution, deconvolution (or transposed convolution), depthwise convolution, and dilated convolution. Each of these different convolutions requires different computational processes, including varying data traversal ranges, data storage offsets, and data mapping calculation methods. Therefore, how to simultaneously support multiple convolution computations with fewer resources to improve the versatility of neural network accelerators remains an open question. Summary of the Invention

[0003] The present invention aims to at least solve the technical problems existing in the prior art, and in particular innovatively proposes a reconfigurable sparse neural network accelerator.

[0004] In order to achieve the above-mentioned object of the present invention, the present invention provides a reconfigurable sparse neural network accelerator, comprising: an input feature map storage module, an instruction storage module, a weight storage module, a bias storage module, an output feature map storage module, a feature map bitmap storage module, a control module, a calculation control module, an address mapping module, a PE array module and a PU general data processing module;

[0005] The control module is connected to the input feature map storage module, instruction storage module, weight storage module, bias storage module, and feature map bitmap storage module. The control module reads and decodes instruction data from the instruction storage module. Depending on the instruction data, it maintains different accelerator states, such as starting and stopping calculations, and reading data. It also controls the reading and writing of DDR memory by each storage unit.

[0006] The calculation control module is connected to the input feature map storage module, instruction storage module, weight storage module, bias storage module, output feature map storage module, feature map Bitmap storage module, and PE array operation module; the calculation control module controls the working state of the PE operation array according to the configuration of the instructions, and selects the input data, weight data, bias data and output feature map data input to the PE operation array.

[0007] The address mapping module is connected to the input feature map storage module, the feature map Bitmap storage module, and the output feature map storage module, and is used to generate the reading address of the input feature map data of the current data to be calculated and the storage address of the output feature map data.

[0008] The PE array operation module is connected to the input feature map storage module, the output feature map storage module, the weight storage module and the bias storage module; the module can perform multiplication and accumulation operations on the input data in parallel. Each PE operation result represents the data of M output channels, and the calculated result is written into the output data storage unit.

[0009] The PU general data processing module is connected to the instruction storage module, output feature map storage module, and DDR module. This module performs post-processing on the data completed by the PE array operation in the output storage module according to the configuration of the instruction;

[0010] The input feature image storage module is connected to the control module, the calculation control module, the address mapping module and the PE array operation module; the input feature image storage module is used to store the feature image to be calculated this time.

[0011] Furthermore, the operating method of the reconfigurable sparse neural network accelerator includes:

[0012] S1, input feature map acquisition, storage and feature map bitmap generation: For a convolution layer of a sparse or non-sparse neural network model, the accelerator first obtains the current feature map size to be calculated and the DDR memory address from the instruction, then obtains the input feature map to be convolution calculated from the DDR memory through the AXI protocol, and transfers it in batches to the feature map storage module of this accelerator; the feature map bitmap is automatically generated when the input feature map is transferred from the DDR memory to this accelerator.

[0013] S2, feature map bitmap analysis: After the input feature map is transmitted, the address mapping module starts working and maps the binary matrix data to generate the read address of the feature map storage module and the write address of the output feature map storage module; the binary matrix is ​​the feature map bitmap.

[0014] S3, PE array obtains weights and input feature map data: loads a set of weight data of size M×N and saves it to the PE array module. For M PEs, each PE can store N weight data values; when a set of non-all-zero data is read out from the input feature map module and sent to the PE array for operation, different processing is performed to support Depthwise convolution and other convolution types; the set of data has M channels.

[0015] The overall workflow is as follows: For the input feature map data (size H*W*N) transmitted to the accelerator, a binary matrix (size H*W) is generated based on whether its N channels are all 0. If all N channels are 0, 0 is written; otherwise, 1 is written. When parsing the feature map binary matrix, only the positions with a value of 1 are found, and 0 is skipped. The address mapping module then calculates the input feature map read address corresponding to this 1 value, and then reads the feature map data based on this address for calculation.

[0016] When a set of data consisting entirely of zeros is transmitted and stored in the accelerator's input feature map module, it is not read out of the input feature map storage module because it is skipped during the feature map bitmap parsing process. Because it is a set of all zeros, even if it is not read out for calculation, it will not affect the final calculation result (calculation accuracy).

[0017] S4, accelerator performs PE operations;

[0018] S5, output data: The PE operation results are stored in the output feature map module for storage, and then the PU general data processing module reads the data from the output feature map module for calculation, and then performs activation, upsampling and maximum pooling operations, and finally sends the PU calculation results to the DDR memory for storage.

[0019] Upsampling operation: Send the same data to multiple DDR addresses for storage to achieve the purpose of data replication;

[0020] Maxpooling operation: Take the maximum value of each sub-region containing data in the output feature map, and the sub-region size is usually 2x2.

[0021] Furthermore, the S1 further includes:

[0022] When the input feature map data is transferred from the DDR memory on the FPGA board to the accelerator, the accelerator generates a binary matrix with the same width and height as the final input feature map. The binary matrix has only two dimensions, HW, and no channel C direction. If the data in the N-channel direction of a transmitted point is all 0, the corresponding position in the binary matrix is ​​stored as 0, indicating that the value is empty. Otherwise, the corresponding position in the binary matrix is ​​set to 1, indicating that the value is not empty.

[0023] Furthermore, the address mapping module includes:

[0024] Take out a row of data from the binary matrix in a row-by-row manner, find the position where each value is 1 in the row, and find a position where the value is 1 in each cycle;

[0025] The data storage location of the corresponding input feature map is calculated through a mapping method, and then the data is read out from the input feature map storage module and sent to the subsequent PE array module for convolution calculation; while calculating the input feature map reading address, the address mapping module also calculates the target address where the current data calculation result needs to be stored to obtain the convolution result, thereby skipping the calculation of all-zero value data in the data dimension of N channels and accelerating the neural network calculation.

[0026] Furthermore, the mapping method of the address mapping module includes:

[0027] SA, pre-defines some parameters: the width value W1 of the input feature map, the width value W2 of the output feature map;

[0028] SB, calculates the current output feature map size width W2 based on the current input feature map size width W1;

[0029] SC, for the read address generation of the input feature map storage module, uses the following offset formula:

[0030] Read address A = the position of the number with the current bit value of 1 in a row (starting from 0) + offset value A1 + offset value A2.

[0031] The offset value A1 is independent of the current calculation weight group. When the Bitmap switches to the next row of data, the offset value will increase and the increase value (relative to the original value) is equal to the width value W1 of the input size.

[0032] The offset value A2 is independent of the number of rows in the bitmap and remains unchanged within the same set of weight calculation cycles. It will change when the weights are switched, and the amount of change depends on the convolution type.

[0033] For the write address generation of the output feature map storage module, the following offset formula is used:

[0034] Write address B = the position of the number with the current bit value of 1 in a row (starting from 0) + offset value B1 + offset value B2.

[0035] The offset value B1 has nothing to do with the current calculation weight group. When the Bitmap switches to the next row of data, the offset value will increase and the amount of increase depends on the convolution type.

[0036] The offset value B2 is independent of the number of rows in the bitmap and remains unchanged within the same set of weight calculation cycles. It will change when the weights are switched, and the amount of change depends on the convolution type.

[0037] The address mapping module can configure address offsets through instructions to support various convolutional neural network operations, including but not limited to normal convolution, deconvolution, depthwise convolution, and dilated convolution.

[0038] If configurable address offsets are not used to generate read and write addresses, for the four types of convolution supported in this article, namely normal convolution, deconvolution, depthwise convolution, and dilated convolution, three sets of read and write addresses need to be generated simultaneously (equivalent to an additional 2 times the address mapping module overhead of this accelerator) due to the different ways of traversing the input and output feature maps, which are divided into three categories (normal convolution and depthwise convolution, and the other two categories). Then, one of them is selected for use based on the current convolution type.

[0039] Furthermore, S2 also includes:

[0040] The operations of fetching the position with a value of 1 in a row of feature map bitmap data and calculating the read and write addresses are all pipelined to avoid idle cycles and waste of hardware resources. The calculation of read and write addresses is performed simultaneously.

[0041] The read address value of the generated feature map storage module and the write address value of the output feature map storage module are sent to two FIFO storages respectively for temporary storage before the PE array calculation begins. The use of this FIFO can alleviate the problem of idle calculation cycles caused by some rows of Bitmap data being all 0 values ​​to a certain extent.

[0042] Furthermore, the different processing to support depthwise convolution and non-depthwise convolution types includes:

[0043] For DepthWise convolution calculations, the N channel data read out are transferred as a whole to the next-level M registers for storage. If N is less than M, the data is expanded to M by padding with zeros. If N is equal to M, no zero padding is required. Each register then sends the temporarily stored data to the corresponding PE for calculation, resulting in a set of outputs.

[0044] Registers are the fundamental storage resources on FPGA boards. The memory modules in the accelerator and the data storage components in the FIFO are all built using registers. Data transmitted from DDR memory to the accelerator is stored in registers. Beyond this, registers have no other relationship with DDR.

[0045] For non-Depthwise convolution, if the N channel data read out is less than M, it will be expanded to M by padding with 0. If N is equal to M, there is no need to pad with 0. Then an operation similar to generating the feature map Bitmap will be performed again, that is, a set of M Bit BitRow data will be generated, M channels correspond to M Bit data, and each Bit represents whether the corresponding single channel value is 0. Then another set of address mapping modules starts working, obtains M Bit BitRow data, searches for the position x of the next non-zero channel value in order from low to high channel position, reads out the single channel data stored in position x, and saves it in a separate register. Then the register will be copied M times and sent to M PEs to perform convolution calculations until all non-zero data in the BitRow are traversed. At this time, the output of each PE represents the data of 1 channel at a certain point in the output feature map, and the results of M PEs represent the data of M channels at a certain point in the output feature map.

[0046] Where M represents the number of PEs, N≤M.

[0047] Furthermore, when the data stream module is in Weight Stationary mode, the weight data of the Depthwise convolution needs to be transformed in advance as follows:

[0048] For a set of weight data of HWC, first fix the direction of the C channel. At this time, the data is arranged as a two-dimensional matrix of C groups of HW; for each group of the C group of two-dimensional matrices, the weights are traversed in the order of first traversing W and then traversing H, and the two-dimensional matrix of HW size is stretched into a one-dimensional array of H×W. The one-dimensional array is used as a set of data in the C channel direction of the transformed weight. The one-dimensional array will be expanded to N data, and the part less than N data will be filled with 0 values ​​until all data complete the above operations.

[0049] Furthermore, the PE operation includes:

[0050] For Depthwise convolution, when a channel data of the input feature map is sent to the current PE, the PE will read the y-th value of a pre-stored set of weight values, and the y value is determined by the number of times the input feature map is traversed; the input feature map channel data and the read y-th value complete the convolution calculation. Due to the particularity of the Depthwise convolution calculation, the number of channels of the output feature map is consistent with the number of channels of the input feature map, that is, the output data in the channel direction is not accumulated, so after completing a convolution calculation, it is directly sent to the output feature map module for storage.

[0051] For non-Depthwise convolution, the value of each channel of the input feature map readout data is traversed before being sent to the PE for calculation. Each time, a non-zero channel value is selected and sent to the next-level register, and M copies of the data are copied from the separate register and sent to the PE respectively. At the same time, each channel data sent to the PE is accompanied by a channel position i data. The i value is used to read the corresponding i-th data in a set of weight data temporarily stored in the PE. The weight data is then convolved with the feature point channel data and the convolution result is accumulated until all channels of the set of input feature map data are traversed and the convolution calculation is completed. The final result is sent to the output feature map module for storage.

[0052] In summary, due to the adoption of the above technical solution, the present invention can use fewer resources to simultaneously support multiple convolution calculations to improve the versatility of the neural network accelerator.

[0053] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0055] Figure 1 It is a schematic diagram of the traversal of the input feature graph of the present invention.

[0056] Figure 2 It is a schematic diagram of the composition of the data graph of the present invention.

[0057] Figure 3 It is a working diagram of generating the feature map Bitmap of the present invention.

[0058] Figure 4 It is a working diagram of the address mapping module of the present invention.

[0059] Figure 5 This is a schematic diagram of the processing process of a group of non-all-0 data before being sent to the PE array for operation.

[0060] Figure 6 It is a schematic diagram of the two-dimensional weight transformation of the present invention.

[0061] Figure 7 It is a schematic diagram of three-dimensional weight transformation of the present invention.

[0062] Figure 8 It is a schematic diagram of the PE structure of the present invention.

[0063] Figure 9 It is a schematic diagram of the PE array and related modules of the present invention.

[0064] Figure 10 It is a schematic diagram of the overall system architecture of the present invention. DETAILED DESCRIPTION

[0065] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0066] The present invention provides a reconfigurable sparse neural network accelerator based on FPGA to solve the problems of low processing efficiency of non-sparse neural network accelerators in the process of calculating neural networks due to the large amount of sparse data and irregular distribution, as well as the accelerator's lack of support for various convolutional neural networks.

[0067] The applicable models of the reconfigurable sparse neural network accelerator proposed in the present invention include both sparse neural network models and non-sparse neural network models.

[0068] The present invention fixes the traversal order of the input feature map as follows: first extract the N-channel data of a point (C direction), then traverse the data points in the direction of a row, extract the N-channel data of the next point in row order (W direction), until all the data of the row are extracted, then switch to the next row (H direction) and repeat the above steps. If the total number of channels of the input feature map is greater than N (i.e., 2 times or higher of N), then after traversing all the data of the first N channels, traverse all the data of the last N channels, such as Figure 1 As shown:

[0069] The data of the present invention is shown in FIG Figure 2 As shown in the figure, data is usually composed of three dimensions HWC, where H represents height, W represents width, and C represents channels; each channel represents a computable data value. In addition, a point in the text represents a basic operation data composed of one or more channel data.

[0070] The acceleration rate of the sparse neural network proposed in the present invention depends on the sparsity of each row of data, and the maximum rate is an equal multiple of the width of the input feature map, that is, a row has at most only one valid data.

[0071] The present invention accelerates the calculation of sparse data by using a bitmap method, which mainly consists of the following two parts:

[0072] 1. The first part applies to the case where a group of 32 channels at a certain point is all 0. In this case, the accelerator skips reading the data for that group of channels and directly selects the next group of 32 channels with non-all 0 data. This part will be described in detail later.

[0073] 2. The second part applies to the case where all 32 channels are not 0. This part is located in the PE array module and will be covered later.

[0074] The sparse neural network accelerator proposed in the present invention includes an input feature map storage module, an instruction storage module, a weight storage module, a bias storage module, an output feature map storage module, a control module, a calculation control module, a feature map bitmap storage module, an address mapping module, a PE array module and a PU general data processing module.

[0075] The sizes of the following memory module units can be pre-configured based on the target FPGA onboard resources.

[0076] The input feature image storage module is used to store the feature image to be calculated this time;

[0077] The weight storage module is used to store the weight data required to complete the convolution operation;

[0078] The bias storage module is used to store the bias data that needs to be added after the convolution operation;

[0079] The output feature map storage module is used to store the image after the current calculation is completed. The image will be used as the image input of the next layer (if the current calculation is not the last layer of the neural network).

[0080] The feature map Bitmap storage module is used to store the binary matrix generated by parsing the input feature map. The effective size is the same as the input feature map. Figure 1 To.

[0081] The DDR memory module is not an internal module of the accelerator. It is only connected to the periphery of the accelerator and is used to store data.

[0082] All storage modules except the output feature map module are connected to the DDR memory module.

[0083] In addition, the input feature map storage module is connected to the control module, the calculation control module, the address mapping module and the PE array operation module.

[0084] The feature map Bitmap storage module is connected to the input feature map storage module, the control module, the calculation control module and the address mapping module.

[0085] The weight storage module is connected to the control module, the calculation control module and the PE array operation module.

[0086] The bias storage and control module, the calculation control module and the PE array operation module are connected.

[0087] The output feature map module is connected to the control module, the calculation control module, the PE array operation module, the address mapping module and the PU general data processing module.

[0088] The instruction storage module is connected to the DDR memory module and is mainly used to store instruction data generated by the software compiler. The accelerator will perform corresponding operations based on the instruction content.

[0089] The control module is connected to each storage unit. This module reads and decodes the instruction data from the instruction storage module. Depending on the instruction data, it maintains different states of the accelerator, such as starting calculations, stopping calculations, and reading data, and controls the reading and writing of DDR memory by each storage unit.

[0090] The computation control module is connected to each storage unit and the PE array computation module. Based on the configuration of the instructions, this module controls the working state of the PE computation array and selects the input data, weight data, bias data, and output feature map data to be input to the PE computation array.

[0091] The address mapping module is connected to the input feature map storage module, the feature map Bitmap storage module, and the output feature map storage module. This module is used to generate the reading address of the input feature map data of the current data to be calculated and the storage address of the output feature map data. This module can configure the address offset through instructions to support a variety of convolutional neural network operations, including but not limited to ordinary convolution (Pointwise convolution), deconvolution (Deconvolution also known as Transposed convolution), Depthwise convolution and dilated convolution.

[0092] The PE array operation module is connected to the input feature map storage module, output feature map storage module, weight storage module, and bias storage module. It configures M PE operation arrays based on the M value, where M represents the number of filters. This module can perform multiplication and accumulation operations on the input data in parallel. Each PE operation result represents the data of M output channels, and the calculated results are written to the output data storage unit.

[0093] The PU general data processing module is connected to the instruction storage module, output feature map storage module, and DDR module. Based on the instruction configuration, this module post-processes the data from the PE array operations in the output storage module. This module supports different activation functions (Linear, LeakyReLU), multi-size pooling operations, upsampling operations, and Typecast processing. Finally, this module writes the processed data back to DDR memory for storage.

[0094] The specific embodiments are as follows:

[0095] In the present invention, M is the number of filters involved in a single round of calculation, and N is the number of input feature map channels involved in a single round of calculation. Both M and N are configurable parameters. In the embodiment, both M and N are 32.

[0096] The PE array module size M included in the embodiment of the present invention is 32.

[0097] The data flow included in the PE array of the present invention is in Weight Stationary mode. That is, a set of (M*N points) weight data is pre-stored in the PE array, and the next set of weight data is switched after the input feature graph is traversed and calculated, thereby saving data transmission.

[0098] The Weight in Weight Stationary is the weight value required for calculation, and Stationary means stationary. This mode means that before the PE array starts calculation, a set of weight data (32*32 data) in the weight storage module is pre-read into the PE. Since the PE array size is 32 PEs, each PE can store 32 data. After a complete traversal (also understood as a complete read) of the input feature Figure 1 Before that, the set of weight data will remain unchanged in the PE array until the input feature map is completely traversed once, at which point the PE array module will load another set of weight data from the weight storage module. In summary, a layer of neural network calculation will traverse the input feature map multiple times, and a set of weight data remains static between two traversals of the input feature map, which is called stationary.

[0099] The main implementation steps are as follows:

[0100] 1. Input feature map acquisition, storage and feature map Bitmap generation:

[0101] The input feature map is first stored in the DDR memory on the FPGA board. Each time, a set of data of 32 channels is transferred from the DDR to this accelerator for storage until all the data are transferred. For a certain convolution layer of a sparse or non-sparse neural network model, this neural network accelerator first obtains the current feature map size to be calculated and the memory address of the DDR from the instruction, and then obtains the input feature map to be convolved from the DDR memory through the AXI protocol, and transfers it in batches to the feature map storage module of this accelerator. When the input feature map data is transferred from the DDR memory on the FPGA board to this accelerator, the accelerator generates a binary matrix with the same width and height as the final input feature map. Here, the size of the binary matrix has only two dimensions HW and no channel C direction. If the data in the 32-channel direction of a certain point of the transmission is all 0, the corresponding position of the binary matrix is ​​stored as 0, indicating that the value is empty. Otherwise, the value of the corresponding position of the binary matrix is ​​set to 1, indicating that the value is not empty, such as Figure 3 shown.

[0102] 2. Feature map Bitmap analysis:

[0103] After the input feature map is transmitted, the address mapping module starts working. This module is mainly used to map the binary matrix data to generate the read address of the feature map storage module and the write address of the output feature map storage module. The process is as follows: Figure 4 As shown:

[0104] This module reads a row of data from the binary matrix of the feature map Bitmap in a row-by-row manner, and searches for each position in the row where the value is 1. Each cycle (i.e., Cycle in the figure) can find a position where the value is 1. The data storage location of the corresponding input feature map is calculated through a mapping method, and then the data is read out from the input feature map storage module and sent to the subsequent PE array module for convolution calculation. While calculating the input feature map read address, the address mapping module also calculates the target address where the current data calculation result needs to be stored to obtain the convolution result. In this way, the calculation of all 0-valued data is skipped in the entire 32-channel data dimension, accelerating the neural network calculation.

[0105] 3. Address generation:

[0106] like Figure 4 As shown, while parsing Btimap, the read address of the input feature map and the write address of the output feature map are generated.

[0107] The mapping method of the address mapping module is as follows:

[0108] Some parameters are defined in advance: W1: the width of the input feature map, W2: the width of the output feature map.

[0109] First, the current output feature map size width W2 is calculated based on the current input feature map size width W1. In ordinary convolution, W2 is usually smaller than W1; in deconvolution, W2 is usually larger than W1.

[0110] For the read address generation of the input feature map storage module, the following offset formula is used:

[0111] Read address A = the position of the number with the current bit value of 1 in a row (starting from 0) + offset value A1 + offset value A2.

[0112] The offset value A1 is independent of the current calculation weight group. When the Bitmap switches to the next row of data, the offset value will increase and the increase value (relative to the original value) is equal to the width value W1 of the input size.

[0113] The offset value A2 is independent of the number of rows in the bitmap and remains unchanged within the same set of weight calculation cycles. It will change when the weights are switched, and the amount of change depends on the convolution type.

[0114] For ordinary convolution, when the weights are switched, the increase in offset A2 (relative to the initial value) is 0, 1, 2, W1, W1+1, W1+2, W1×2, W1×2+1, W1×2+2... and so on.

[0115] For deconvolution calculations, the value of A2 is fixed to 0.

[0116] For the write address generation of the output feature map storage module, the following offset formula is used:

[0117] Write address B = the position of the number with the current bit value of 1 in a row (starting from 0) + offset value B1 + offset value B2.

[0118] The offset value B1 is independent of the current calculation weight group. When the Bitmap switches to the next row of data, the offset value increases and the amount of increase is determined by the convolution type.

[0119] For ordinary convolution, the increase in B1 (relative to its own value) is equal to the width W2 of the output size;

[0120] For deconvolution, the increase in B1 (relative to its own value) is equal to the width of the output size W2×2;

[0121] The offset value B2 is independent of the number of rows in the bitmap and remains unchanged within the same set of weight calculation cycles. It will change when the weights are switched, and the amount of change depends on the convolution type.

[0122] For ordinary convolution, the value of B2 is fixed to 0.

[0123] For ordinary convolution, when the weights are switched, the increase in offset B2 (relative to the initial value) is 0, 1, 2, W2, W2+1, W2+2, W2×2, W2×2+1, W2×2+2... and so on.

[0124] The situation of writing address B2 is exactly the opposite of the situation of reading address A2, and most of the computing resources can be reused.

[0125] The corresponding relationship of the above formula can be derived for other convolutions. Here we only use ordinary convolution and deconvolution as examples.

[0126] 4. Supplement of feature map Bitmap parsing and address generation:

[0127] To ensure that subsequent data calculations do not idle, the operations of fetching the positions with values ​​of 1 in a row of the feature map bitmap and calculating the read and write addresses are pipelined to eliminate idle cycles and waste hardware resources. The read address value of the generated feature map storage module and the write address value of the output feature map storage module are respectively sent to two FIFOs for temporary storage before the PE array calculation begins. The use of these FIFOs can alleviate the problem of idle calculation cycles caused by certain rows of bitmap data being all 0.

[0128] 5.PE array obtains weights and input feature map data:

[0129] Before the PE array starts convolution calculation, the accelerator will first load a set of M×N (32×32) size weight data and save it to the PE array module. For M (32) PEs, each PE can store N (32) weight data values.

[0130] When a set of non-all-zero data (32 channels) is read out from the input feature map storage module and sent to the PE array for operation, different processing is required to support Depthwise convolution and other convolution types. The process is as follows: Figure 5 As shown:

[0131] For DepthWise convolution calculations, the 32 channel data read out will be transferred as a whole to the next level 32 registers for storage. Then each register sends the temporarily stored data to the corresponding PE for calculation to obtain a set of outputs.

[0132] As mentioned above, the data flow module used in this invention is in Weight Stationary mode (weight stationary, i.e., a set of weights are loaded before the operation, and then the set of data is kept unchanged for a certain period of time for calculation). For Depthwise convolution, the weight data used needs to be transformed in advance to support the structure of this accelerator. The transformations that need to be completed for the weight data are as follows: Figure 7 As shown:

[0133] For a set of weight data of HWC, first fix the direction of C channel. At this time, the arrangement of data can be understood as a two-dimensional matrix of C group HW (HW value is usually 3 and 3). For a set of HW weight transformations, Figure 6 As shown: for each group of C two-dimensional matrices, the weights are traversed in the order of first traversing W and then traversing H, and the two-dimensional matrix of HW size is stretched into an H×W one-dimensional array (9 in the case of 3×3). The one-dimensional array is used as a set of data in the C channel direction of the transformed weight. The one-dimensional array will then be expanded to 32 data, and the part less than 32 data will be filled with 0 values ​​until all data complete the above operation.

[0134] Figure 6 、 Figure 7 All the values ​​in represent only serial numbers, not the values ​​of the data themselves.

[0135] For non-Depthwise convolution, the 32 channel data read out will perform an operation similar to the feature map Bitmap, that is, after the data is read out, a set of 32-bit BitRow data is generated at the same time. 32 channels correspond to 32 Bit data, and each Bit represents whether the corresponding single channel value is 0. Then another set of address mapping modules starts working, obtains the 32-bit BitRow data, and searches for the position x of the next non-zero channel value in order from low to high channel position (from right to left in the figure), reads out the single channel data stored in position x, and saves it in a separate register. The register will then be copied 32 times and sent to 32 PEs to perform convolution calculations until all non-zero data in the BitRow are traversed. At this time, the output of each PE represents the data of 1 channel at a certain point in the output feature map, and the results of the 32 PEs represent the data of 32 channels at a certain point in the output feature map.

[0136] 6.PE operations:

[0137] PE structure such as Figure 8As shown: The PE operation process is also divided into two cases: Depthwise convolution and other convolution. For the Depthwise convolution operation, when a channel data of the input feature map is sent to the current PE, the PE will read the yth value of a set of pre-stored weight values. The y value is determined by the number of times the input feature map is traversed. For example, for a set of weight data that is 3×3×C before the transformation, it is C×32 after the transformation and expansion (the actual valid data is only C×9). When traversing the input feature map for the first time, M=0, and M=1 for the second time, until the last time M=8 (because there are only 9 valid data in each PE). The input feature map channel data and the read yth value complete the convolution calculation. Due to the particularity of the Depthwise convolution calculation, that is, the number of channels of the output feature map is consistent with the number of channels of the input feature map, that is, the output data in the channel direction is not accumulated, so after completing a convolution calculation, it is directly sent to the output feature map module for storage.

[0138] For the remaining convolution operation, according to the above content, non-Depthwise convolution will traverse the value of each channel of the input feature map readout data before sending it to PE for calculation. Each time, a non-zero channel value is selected and sent to the next level register, and 32 copies of the register are copied and sent to PE respectively. At the same time, each channel data sent to PE will be accompanied by a channel position i data. The i value will be used to read the corresponding i-th data in a set of weight data temporarily stored in PE. The weight data is then convolved with the feature point channel data and the convolution result is accumulated until all channels of the set of input feature map data are traversed and the convolution calculation is completed. The final result is sent to the output feature map module for storage.

[0139] PE array and related modules such as Figure 9 As shown. After the PE obtains the data read from the input feature map storage module, it determines the current operation mode, which is divided into two cases: depthwise convolution and non-depthwise convolution. After the data is processed, it is sent to each PE for calculation according to the relationship described above, that is, the relationship described in the first two paragraphs. The final calculation result of the PE is transmitted to the subsequent ACC accumulator for accumulation to obtain the complete output feature map result. The result is then sent to the output feature map module for storage, waiting to be read by the subsequent module.

[0140] 7. Output data:

[0141] The data in the final output feature map module is activated, up-sampled, and max-pooled by the PU general data processing module and then sent to the DDR memory for storage.

[0142] The overall system architecture is as follows Figure 10As shown. The system performs two operations on part of the input feature map data transmitted from the DDR, namely, storing it in the input feature map storage module and parsing to generate feature map Bitmap data. The feature map Bitmap is then read again to obtain the position of the non-zero data value, and the read address generation and write address generation are completed in the address mapping module according to the position and mapping relationship. The read address is used to send to the feature map storage module to read the corresponding address data, and the write address is used to send to the output feature map module to store the corresponding address data. For the data read from the input feature map, it will be parsed according to the above relationship and then sent to the PE array to complete the convolution calculation. The final calculation result of the PE will be stored in the output feature map module. The PU will then take out the result for activation, upsampling and maximum pooling operations. Among these three operations, activation is used in each layer, and upsampling or maximum pooling operations are not necessarily performed every time; the data after the operation is then sent back to the DDR memory for storage, that is, the calculation of an entire layer of the neural network is completed.

[0143] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A reconfigurable sparse neural network accelerator, characterized in that: include: Input feature map storage module, instruction storage module, weight storage module, bias storage module, output feature map storage module, feature map Bitmap storage module, control module, calculation control module, address mapping module, PE array module and PU general data processing module; The control module is connected to the input feature map storage module, the instruction storage module, the weight storage module, the bias storage module, and the feature map Bitmap storage module; The calculation control module is connected to the input feature map storage module, the instruction storage module, the weight storage module, the bias storage module, the output feature map storage module, the feature map Bitmap storage module, and the PE array operation module; The address mapping module is connected to the input feature map storage module, the feature map Bitmap storage module, and the output feature map storage module; The PE array operation module is connected to the input feature map storage module, the output feature map storage module, the weight storage module and the bias storage module; The PU general data processing module is connected to the instruction storage module, the output feature map storage module, and the DDR module; The input feature map storage module is connected to the control module, the calculation control module, the address mapping module and the PE array operation module; The operation method of the reconfigurable sparse neural network accelerator includes: S1, input feature map acquisition, storage and feature map bitmap generation: For a convolution layer of a sparse or non-sparse neural network model, the accelerator first obtains the current feature map size to be calculated and the DDR memory address from the instruction, then obtains the input feature map to be convolutionally calculated from the DDR memory via the AXI protocol and transfers it in batches to the feature map storage module of this accelerator; S2, feature map bitmap analysis: After the input feature map is transmitted, the address mapping module starts working to map the binary matrix data to generate the read address of the feature map storage module and the write address of the output feature map storage module; S3, PE array obtains weights and input feature map data: loads a set of weight data of size M×N and saves it to the PE array module. For M PEs, each PE can store N weight data values. When a set of non-all-zero data is read from the input feature map module and sent to the PE array for operation, different processing is performed to support depthwise convolution and other convolution types. S4, accelerator performs PE operations; S5, output data: The PE operation results are stored in the output feature map module for storage, and then the PU general data processing module reads data from the output feature map module for calculation, and then performs activation, upsampling and maximum pooling operations, and finally sends the PU calculation results to the DDR memory for storage.

2. The reconfigurable sparse neural network accelerator according to claim 1, characterized in that: Said S1 further comprises: When the input feature map data is transmitted from the DDR memory on the FPGA board to this accelerator, the accelerator will generate a binary matrix with the same width and height as the final input feature map. If the data in the N-channel direction of a certain point of the transmission is all 0, the corresponding position of the binary matrix is ​​stored as 0, indicating that the value is empty. Otherwise, the value of the corresponding position of the binary matrix is ​​set to 1, indicating that the value is not empty.

3. The reconfigurable sparse neural network accelerator according to claim 1, characterized in that: The address mapping module includes: Take out a row of data from the binary matrix in a row-by-row manner, find the position where each value is 1 in the row, and find a position where the value is 1 in each cycle; The data storage location of the corresponding input feature map is calculated through the mapping method, and then the data is read out from the input feature map storage module and sent to the subsequent PE array module for convolution calculation; while calculating the input feature map reading address, the address mapping module also calculates the target address where the current data calculation result needs to be stored to obtain the convolution result.

4. The reconfigurable sparse neural network accelerator according to claim 3, characterized in that: The mapping methods of the address mapping module include: SA, pre-defines some parameters: the width value W1 of the input feature map, the width value W2 of the output feature map; SB, calculates the current output feature map size width W2 based on the current input feature map size width W1; SC, for the read address generation of the input feature map storage module, uses the following offset formula: Read address A = the position of the number with the current bit value of 1 in a row + offset value A1 + offset value A2; For the write address generation of the output feature map storage module, the following offset formula is used: Write address B = the position of the number with the current bit value of 1 in a row + offset value B1 + offset value B2.

5. The reconfigurable sparse neural network accelerator according to claim 1, characterized in that: S2 also includes: The operations of obtaining the position with a value of 1 in a row of data in the feature map Bitmap and calculating the read address and write address are all run in a pipeline manner, and the read address value of the generated feature map storage module and the write address value of the output feature map storage module are sent to two FIFO storages respectively.

6. The reconfigurable sparse neural network accelerator according to claim 1, characterized in that: The different processing to support Depthwise convolution and non-Depthwise convolution types include: For DepthWise convolution calculations, the N channel data read out are transferred as a whole to the next-level M registers for storage. If N is less than M, the data is expanded to M by padding with zeros. If N is equal to M, no zero padding is required. Each register then sends the temporarily stored data to the corresponding PE for calculation, resulting in a set of outputs. For non-Depthwise convolution, if the N channel data read out is less than M, it will be expanded to M by padding with 0. If N is equal to M, there is no need to pad with 0. Then an operation similar to generating the feature map Bitmap will be performed again, and then another set of address mapping modules will start working to obtain M Bit BitRow data, and search for the position x of the next non-zero channel value in order from low to high channel position, read out the single channel data stored in position x, and save it in a separate register. Then the register will be copied M times and sent to M PEs to perform convolution calculations until all non-zero data in the BitRow are traversed. At this time, the output of each PE represents the data of 1 channel at a certain point in the output feature map, and the results of M PEs represent the data of M channels at a certain point in the output feature map.

7. The reconfigurable sparse neural network accelerator according to claim 6, characterized in that: When the data stream module is in Weight Stationary mode, the weight data of the Depthwise convolution needs to be transformed in advance as follows: For a set of weight data of HWC, first fix the direction of the C channel. At this time, the data is arranged as a two-dimensional matrix of C groups of HW; for each group of the C group of two-dimensional matrices, the weights are traversed in the order of first traversing W and then traversing H, and the two-dimensional matrix of HW size is stretched into a one-dimensional array of H×W. The one-dimensional array is used as a set of data in the C channel direction of the transformed weight. The one-dimensional array will be expanded to N data, and the part less than N data will be filled with 0 values ​​until all data complete the above operations.

8. The reconfigurable sparse neural network accelerator according to claim 1, characterized in that: The PE operations include: For Depthwise convolution, when a channel data of the input feature map is sent to the current PE, the PE will read the y-th value of a set of pre-stored weight values. The y value is determined by the number of times the input feature map is traversed. The input feature map channel data and the read y-th value complete the convolution calculation. For non-Depthwise convolution, the value of each channel of the input feature map read-out data will be traversed before being sent to PE for calculation. Each time, a non-zero channel value will be selected and sent to the next level register, and M copies of the data will be copied from the separate register and sent to PE respectively. At the same time, each channel data sent to PE will be accompanied by a channel position i data. The i value will be used to read the corresponding i-th data in a set of weight data temporarily stored in PE. The weight data will then be convolved with the feature point channel data and the convolution result will be accumulated until all channels of the set of input feature map data are traversed and the convolution calculation is completed.

Images