A binary weighted convolutional neural network accelerator and RISC-V system-on-a-chip

Abstract

This invention discloses a binary weighted convolutional neural network accelerator and a RISC-V system-on-a-chip. The binary weighted convolutional neural network accelerator includes an instruction parsing module, an address generation module, a finite state machine, a shift window module, a feature map storage module, a weight storage module, a multi-channel shared computing array, an accumulation module, and batch normalization and pooling modules. The RISC-V system-on-a-chip includes a FLASH module, a DDR module, an E203 RISC-V soft core module, an AXI Interconnect module, an AXI data transmission path, a cross-clock domain module, and the binary weighted convolutional neural network accelerator. This invention can significantly improve data reuse and storage retrieval efficiency, reduce resource consumption, and thus improve the overall operating efficiency of the accelerator.

Description

Technical Field

[0001] This invention relates to the field of convolutional neural network accelerator technology, specifically to a binary weighted convolutional neural network accelerator and a RISC-V system-on-a-chip. Background Technology

[0002] Convolutional neural networks (CNNs) are fundamental to biological vision and natural language processing (NLP), with LeNet-5 laying the foundation for future CNN development. VGG-16, using 3x3 convolutional kernels, built a very deep network model, demonstrating that increasing network depth can significantly impact performance. Since late 2011, Yu Dong, Deng Li, and others at Microsoft Research have applied Deep Neural Network (DNN) technology to large-vocabulary speech recognition tasks, constructing a DNN-HMM model that significantly reduced speech recognition error rates. As more complex neural network models have been proposed, the high computational cost and frequent memory access of neural networks present significant challenges for deployment on edge devices. While high-performance computing devices such as Graphics Processing Units (GPUs) are typically used for training neural networks, their high power consumption makes them less ideal for forward inference. In recent years, Field Programmable Gate Arrays (FPGAs) have received increasing attention from academia and industry in CNN accelerators due to their large number of computing units and memory blocks.

[0003] Limited storage and computing resources on FPGAs, coupled with the constant innovation of network models, pose challenges to the versatility and energy efficiency of neural networks. Currently, many neural network accelerators improve speed and performance by building lightweight network models and new hardware architectures. The Roofine model can improve overall system throughput under different constraints by increasing data transfer bandwidth and the number of computations per unit memory access. Ma et al. studied loop unrolling, tiling, and transformation in convolution operations, utilizing data utilization and parallelism between convolution kernels to improve accelerator throughput. Neural network accelerators incorporate reconfigurable designs to adapt to different network sizes. OPUs use ping-pong storage for feature maps, weights, and instructions to improve computational efficiency, but this consumes too many resources, leading to high system power consumption. Due to the sparsity of neural networks, using the sparsity of weights or feature maps to skip zero-value computations can improve efficiency. Methods such as weight encoding and non-zero bitmaps can also compress matrix size to reduce computation, but this introduces additional storage space for the encoding matrix and complicates computational control. Huang et al. designed a multi-engine pipelined computing accelerator architecture, setting dedicated parameters for each network layer, which lacks versatility. As can be seen, the above studies have optimized the computing storage efficiency, versatility, parallelism, and quantization compression through various methods.

[0004] Fixed-point high-precision neural network accelerators have achieved considerable computational efficiency, but the core computation of high-precision neural networks still relies heavily on costly and power-intensive multiplication operations, requiring frequent accesses to off-chip memory. Binary neural networks, on the other hand, have low computational complexity and low storage requirements, making them ideal for multiplier-free operations and reducing the need for extensive off-chip memory accesses. Therefore, hardware acceleration of binary neural networks has been extensively studied.

[0005] FPGA-based binary neural network hardware accelerators are exemplified by the FINN architecture proposed by Xilinx Research Lab. FINN is a fast and flexible heterogeneous dataflow neural network architecture. Through a series of architectural optimizations, it achieves throughputs of 9 TOPS and 2.5 TOPS on the MNIST and CIFAR-10 datasets, respectively, with performance of 1032 GOPS / W and 685 GOPS / W. Fu et al. proposed a fast and efficient binary neural network inference architecture that, by reusing previous computation results, can skip a large number of data buffer accesses and computation cycles, reducing computation by 80% and buffer access by 40%. The throughput achieved on an FPGA reached 975 GOPS, with resource efficiency 1.9 times higher than advanced designs of the same type and 10 times higher than 16-bit accelerators. BinaryEye is an FPGA-based streaming media camera system that uses a binary neural network to classify regions of interest within a frame, achieving significant data reduction and a throughput of 20,000 frames per second (FPS). The binary neural network accelerator developed by Qiao et al. can fuse batch normalization layers and reduces computational resources and memory requirements by 84.2% and 96.4%, respectively. FracBNN uses fractional convolution to quantize data to 2 bits to improve model accuracy and decomposes the computation process, allowing for hardware acceleration using xnor and popcount. FracBNN achieves a Top-1 accuracy of 71.8% on ImageNet.

[0006] Existing technologies typically employ non-open-source ARM architectures and suffer from problems such as excessively large network parameters and high GPU operating costs, failing to meet the demands of increasing data requirements and improved hardware computing performance. Summary of the Invention

[0007] To address the aforementioned shortcomings in the prior art, this invention provides a binary weighted convolutional neural network accelerator and a RISC-V system-on-a-chip.

[0008] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0009] In a first aspect, the present invention proposes a binary weighted convolutional neural network accelerator, comprising:

[0010] The instruction parsing module is used to parse the input configuration instructions and send the instruction parsing results to the finite state machine and address generation module;

[0011] The address generation module is used to generate the base address of feature map, weight, and bias data based on the instruction parsing results.

[0012] A finite state machine is used to control the transition process between layers during the operation of the accelerator and to issue the configuration parameters required for the accelerator calculation process based on the instruction parsing results.

[0013] The shift window module is used to store and update the input feature map data of the feature map storage module by using multiple shift random access memories to form a serial-in parallel-out first-in-first-out memory.

[0014] The feature map storage module is used to store feature map data of the intermediate operation process of the convolutional neural network using multiple sets of feature map ping-pong block random access memory, and to exchange feature map data with external transmission channels. The stored feature map data is also input into the multi-channel shared computing array through a data selector and feature map base address.

[0015] The weight storage module is used to store the weight data mapped and transmitted through the external transmission channel using multiple sets of weight block random access memory, and input the stored weight data into the multi-channel shared computing array through the data selector and the feature map base address.

[0016] A multi-channel shared computing array is used to construct a computing array using multiple sets of multiply-accumulate units, and to share each set of feature map data with the corresponding multiple sets of weight data, perform multiply-accumulate operations and output the data to the accumulation module;

[0017] The accumulation module is used to accumulate the computation results of the multi-channel shared computing array, add bias and perform ReLU operation to obtain the output result of the convolutional neural network, and input it into the batch normalization and pooling modules;

[0018] The batch normalization and pooling modules are used to perform batch normalization and max pooling on the output of the convolutional neural network before outputting it to the feature map storage module.

[0019] Alternatively, the sliding window module may specifically include:

[0020] Three shift random access memories are used to form a serial-in parallel-out first-in-first-out memory, and the first three columns of feature map data of the input feature map data of the feature map storage module are all stored in the three shift random access memories;

[0021] Set the kernel size and stride, and slide the four kernels in parallel on the feature map data stored in three shift random access memories while performing convolution calculations simultaneously.

[0022] After all three columns of feature map data have been calculated, the next column of feature map data from the input feature map data of the feature map storage module is stored in three shift random access memories, and the four convolution kernels continue to slide in parallel on the feature map data stored in the three shift random access memories.

[0023] Optionally, the feature map storage module specifically includes:

[0024] Four sets of feature map ping-pong block random access memory, each set of feature map ping-pong block random access memory includes two parallel feature map ping-pong block random access memory.

[0025] Optionally, the weight storage module specifically includes:

[0026] Four sets of weighted block random access memory, each set of weighted block random access memory includes four parallel weighted ping-pong block random access memory.

[0027] Alternatively, the multi-channel shared computing array may specifically include:

[0028] The sixteen groups of multiply-accumulate units are arranged into a column of four groups to form a calculation array, including the first multiply-accumulate array, the second multiply-accumulate array, the third multiply-accumulate array and the fourth multiply-accumulate array;

[0029] Each multiply-accumulate array corresponds to a set of feature map ping-pong block random access memory, and each set of multiply-accumulate units in each multiply-accumulate array corresponds to a feature map ping-pong block random access memory.

[0030] The feature map data output from the four sets of feature map ping-pong block random access memory are respectively entered into the column-first multiply-accumulate unit of the corresponding multiply-accumulate array. At the same time, the weight data output from the sixteen weight ping-pong block random access memory are respectively entered into the corresponding multiply-accumulate unit of the corresponding multiply-accumulate array for multiply-accumulate operation.

[0031] The column tail multiply-accumulate unit of the fourth multiply-accumulate array is output to the column tail multiply-accumulate unit of the first multiply-accumulate array via the column tail multiply-accumulate units of the third and second multiply-accumulate arrays for multiply-accumulate operation; the operation result of the column tail multiply-accumulate unit of the first multiply-accumulate array is output to the feature map storage module.

[0032] Optionally, the multiply-accumulate unit specifically includes:

[0033] The system comprises a first sparse binary convolution computation unit, a second sparse binary convolution computation unit, a third sparse binary convolution computation unit, and a fourth sparse binary convolution computation unit, as well as an addition tree unit.

[0034] The first sparse binary convolution calculation unit, the second sparse binary convolution calculation unit, the third sparse binary convolution calculation unit, and the fourth sparse binary convolution calculation unit perform binary convolution calculations on the input feature map data and weight data, respectively, and output all the calculation results to the addition tree unit for accumulation operation.

[0035] Optionally, the first sparse binary convolution calculation unit, the second sparse binary convolution calculation unit, the third sparse binary convolution calculation unit, and the fourth sparse binary convolution calculation unit have the same structure, each including:

[0036] First data selector, second data selector, and adder;

[0037] The first data selector determines whether the input feature map data is zero; if so, it directly enters the adder; otherwise, the feature map data is output to the second data selector.

[0038] The second data selector performs binary convolution multiplication on the input feature map data according to the input weight data, and outputs the result to the adder;

[0039] The adder performs multiple summation operations on the calculation results and then outputs the result.

[0040] Secondly, this invention also proposes a RISC-V on-chip system, comprising:

[0041] The FLASH module is used for weight data in convolutional neural networks;

[0042] The DDR module is used to store feature map data during the intermediate operations of the convolutional neural network, and to exchange feature map data with the feature map storage module.

[0043] The E203RISC-V soft core module is used to establish a custom instruction set and send custom instructions through the AXIInterconnect module to accelerate the binary weighted convolutional neural network accelerator.

[0044] The AXIInterconnect module is used to interconnect and forward the FLASH module, DDR module, and E203RISC-V soft core module with the binary weighted convolutional neural network accelerator through the AXI data transmission path;

[0045] The AXI data transmission path is used to establish a weight and feature map data transmission path with 4 AXIDMA and 1 AXICDMA, and to issue custom commands using the AXI instruction conversion bridge interface.

[0046] A cross-clock domain module is used to provide a 200MHz clock frequency to the binary weighted convolutional neural network accelerator;

[0047] And the binary weighted convolutional neural network accelerator described in any one of claims 1 to 7.

[0048] Alternatively, the AXI data transmission path may specifically include:

[0049] Four feature map transmission channels, one weight transmission channel, and one custom instruction transmission channel;

[0050] The four feature map transmission channels exchange feature map data with four sets of weight block random access memory through a set of input feature map first-in-first-out memory, an output feature map first-in-first-out memory, and four data selectors, respectively. The storage controller controls each data selector to send feature map configuration parameters and fills the feature map input to the input feature map first-in-first-out memory with the address.

[0051] One weight transmission channel exchanges weight data with four sets of weight block random access memory via a data selector;

[0052] A custom instruction transfer channel exchanges instruction data with a custom instruction register via the AXI instruction transfer module.

[0053] Optionally, the custom instruction set specifically includes:

[0054] Accelerator reset instruction, clear storage flag, padding mode, network row and column and channel number instructions, feature map ping-pong block random access memory full flag instruction, weight block random access memory full flag instruction, input feature map first-in-first-out memory empty flag instruction, block random access memory ready to receive data flag instruction, output feature map first-in-first-out memory empty flag instruction, feature map ping-pong block random access memory empty flag instruction and weight block random access memory empty flag instruction.

[0055] At the start of the first convolutional computation, the system enters the write-ready state based on the block random access memory (RAM) readiness flag, and begins transmitting weight data. After weight data transmission is complete, the system enters the feature map data transmission state based on the weight block RAM full flag, and begins transmitting feature map data. After feature map data transmission is complete, the system enters the convolution computation state based on the feature map ping-pong block RAM full flag. After convolution computation is complete, the system enters the feature map data return state based on the input and output feature map FIFO memory empty flags. After the feature map data is returned to the DDR module, the system returns to the initial state based on the clear storage flags, padding mode, network row / column and channel number instructions.

[0056] The present invention has the following beneficial effects:

[0057] This invention, through the design of a binary weighted convolutional neural network accelerator and a RISC-V system-on-a-chip based on the binary weighted convolutional neural network accelerator, can significantly improve data reuse and storage retrieval efficiency, reduce resource consumption, and thus improve the overall working efficiency of the accelerator. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the structure of a binary weighted convolutional neural network accelerator in Embodiment 1.

[0059] Figure 2 This is a schematic diagram of the sliding window module in Embodiment 1.

[0060] Figure 3 This is a schematic diagram of the data ping-pong storage and pipeline transmission process in Embodiment 1.

[0061] Figure 4 This is a schematic diagram of the multiply-accumulate unit in Embodiment 1.

[0062] Figure 5 This is a schematic diagram of the sparse binary convolution calculation unit in Embodiment 1.

[0063] Figure 6 This is a schematic diagram of the RISC-V on-chip system in Embodiment 2.

[0064] Figure 7 This is a schematic diagram of the AXI data transmission path in Embodiment 2.

[0065] Figure 8 This is a schematic diagram illustrating the state transition principle of convolution operation in Embodiment 2. Detailed Implementation

[0066] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0067] Example 1

[0068] like Figure 1 As shown, this embodiment of the invention provides a binary weighted convolutional neural network accelerator, comprising:

[0069] The instruction parsing module is used to parse the input configuration instructions and send the instruction parsing results to the finite state machine and address generation module;

[0070] The address generation module is used to generate the base address of feature map, weight, and bias data based on the instruction parsing results.

[0071] A finite state machine is used to control the transition process between layers during the operation of the accelerator and to issue the configuration parameters required for the accelerator calculation process based on the instruction parsing results.

[0072] The shift window module is used to store and update the input feature map data of the feature map storage module by using multiple shift random access memories to form a serial-in parallel-out first-in-first-out memory.

[0073] The feature map storage module is used to store feature map data of the intermediate operation process of the convolutional neural network using multiple sets of feature map ping-pong block random access memory, and to exchange feature map data with external transmission channels. The stored feature map data is also input into the multi-channel shared computing array through a data selector and feature map base address.

[0074] The weight storage module is used to store the weight data mapped and transmitted through the external transmission channel using multiple sets of weight block random access memory, and input the stored weight data into the multi-channel shared computing array through the data selector and the feature map base address.

[0075] A multi-channel shared computing array is used to construct a computing array using multiple sets of multiply-accumulate units, and to share each set of feature map data with the corresponding multiple sets of weight data, perform multiply-accumulate operations and output the data to the accumulation module;

[0076] The accumulation module is used to accumulate the computation results of the multi-channel shared computing array, add bias and perform ReLU operation to obtain the output result of the convolutional neural network, and input it into the batch normalization and pooling modules;

[0077] The batch normalization and pooling modules are used to perform batch normalization and max pooling on the output of the convolutional neural network before outputting it to the feature map storage module.

[0078] In this embodiment, a data selector is used to input four sets of feature map data and sixteen sets of weight data into the corresponding computation modules, and to output four sets of feature map data into the corresponding output feature map storage FIFO. A finite state machine is used to control the jump process between layers during the entire accelerator operation. Based on the data parsed by the instruction parsing module, the configuration parameters required for the accelerator computation process are issued. The configuration parameters include the base address of the feature map and weight data, the base address of the bias data, the empty / full flags of the internal FIFO and BRAM storage modules, the length, width, and number of channels of the feature map. Finally, the batch normalization and pooling modules are used to batch normalize the convolutional layer results obtained after accumulation to make the data distribution stable. The batch normalized results are then subjected to max pooling to extract the features containing the most information, effectively reducing the scale of the output features, thereby reducing the number of parameters required for subsequent model calculations.

[0079] In an optional embodiment of the present invention, a CNN neural network comprises multiple convolutional layers, normalization layers, activation layers, and pooling layers, each playing a different role in the model. Using convolutional layers to extract features from the input data is an abstract representation of the correlation between the source data.

[0080] The input for convolution calculation is the feature map Fin(H,W,C) i ), for C i The input feature map is a two-dimensional matrix with dimensions H and W. The padding is applied to the matrix Finp(H+2,W+2,C). i The parameters of the convolution filter are ω(Q,P,C). i C o ), where Q and P are the width and height of the convolution kernel, and 3*3 is used as the kernel size in this paper. C i C represents the number of input convolution kernel filters. o This specifies the number of output convolution kernel filters. The output of the convolution calculation is the feature map Fout(H,W,C). i ), for C i The data consists of a two-dimensional matrix with dimensions H and W. The bias term is Bias(K). The output Fout(i,j,k) of the k-th feature can be described by the following formula:

[0081]

[0082] The asterisk (*) represents the convolution operation. The essence of convolution computation is a sliding window multiplication-addition and vertical summation operation of a two-dimensional filter, resulting in a local feature information matrix extracted from the original matrix.

[0083] CNN models typically perform floating-point calculations during training or inference in software. However, performing floating-point calculations on an FPGA requires significant resources. Therefore, the model needs to be quantized to reduce the resource consumption of FPGA computation and storage. Convolutional neural networks are highly robust, and resource overhead can be reduced by decreasing the bit width while maintaining accuracy.

[0084] Traditional binary neural networks quantize all weights and activation functions to 1 bit. While this method can greatly reduce the number of parameters, it cannot effectively guarantee the accuracy of the network. Therefore, this embodiment designs a Binary Weighted Convolutional Neural Network (BWN) that limits the weights of the convolutional layers to 1 bit, thereby avoiding the need for DSP multiplication operations in hardware and reducing resource overhead.

[0085] In a binary weighted convolutional neural network, the convolution weights are constrained to 1 bit according to the following formula:

[0086]

[0087] Among them W fp This represents the original, full-precision weights of the convolutional layer. Network training typically includes three stages: forward propagation, backward propagation, and parameter update. In binary-weighted convolutional neural networks, the aforementioned binary weights W are used in both the forward and backward propagation stages. b However, full-precision weights W are used when updating parameters. fp This embodiment uses transfer learning to retrain a binary weighted convolutional neural network using the weights of a pre-trained full-precision model.

[0088] In an optional embodiment of the present invention, the sliding window module specifically includes:

[0089] Three shift random access memories are used to form a serial-in parallel-out first-in-first-out memory, and the first three columns of feature map data of the input feature map data of the feature map storage module are all stored in the three shift random access memories;

[0090] Set the kernel size and stride, and slide the four kernels in parallel on the feature map data stored in three shift random access memories while performing convolution calculations simultaneously.

[0091] After all three columns of feature map data have been calculated, the next column of feature map data from the input feature map data of the feature map storage module is stored in three shift random access memories, and the four convolution kernels continue to slide in parallel on the feature map data stored in the three shift random access memories.

[0092] Specifically, in this embodiment, the convolution kernel size is 3*3, and the stride is 1. The convolution operation is essentially the convolution kernel sliding across the input feature map, performing convolution operations on different local data. To update the feature map data requiring convolution in real time, this embodiment is designed as follows: Figure 2 The diagram shows a flowing sliding window module. The feature map data width and height of each layer of the neural network are W and H, respectively, and the number of convolutional channels is Ci.

[0093] The shift window module uses three shift RAMs to form a serial-in, parallel-out FirstInputFirstOutput (FIFO) to store and update the three columns of data in the input feature map. Among them, W, H, and C... i This specifies the number of rows, columns, and channels of the input feature map. For example... Figure 2 As shown in (a), the input feature map first stores the first three columns of data into three shift RAMs respectively. Because the fixed-point width of the feature map is 8 bits, while the bus data width is 32 bits, therefore... Figure 2 As shown in (b), the data is output in a 3x4 format. After all three columns of feature map data are output, as shown... Figure 2 As shown in (c), the next column of feature map data is input into the shift RAM, and the convolution matrix continues to slide and output data. That is, the first column of data is stored in the first shift RAM, the second column of data is stored in the second shift RAM, and the third column of data is stored in the third shift RAM. While outputting in parallel, data from the second shift RAM is stored in the first shift RAM, data from the third shift RAM is stored in the second shift RAM, and new data from the next column is stored in the third shift RAM. This process is repeated until a feature map is calculated. To match the transmission bandwidth with the computation bandwidth, this embodiment reuses the overlapping data of the sliding window, employing methods such as... Figure 2 The four convolutional matrices shown in (d) slide in parallel and are computed simultaneously. Each feature map data storage module can output 36 feature map data per clock cycle during the convolution computation process, so the four feature map data storage modules can output 144 feature map data per clock cycle.

[0094] In an optional embodiment of the present invention, the feature map storage module specifically includes:

[0095] Four sets of feature map ping-pong block random access memory, each set of feature map ping-pong block random access memory includes two parallel feature map ping-pong block random access memory.

[0096] Since data transmission accounts for a significant portion of the overall system runtime, this embodiment designs a pipelined data transmission method based on ping-pong storage to improve computational efficiency. The pipelined architecture is as follows: Figure 3 As shown, data is first sent to BRAM0. After BRAM0 is full, data is sent to BRAM1, and calculation of the data already stored in BRAM0 begins simultaneously. When the data in BRAM0 is exhausted, data continues to be input to BRAM0 while waiting for BRAM1 to be full before starting calculation of the data in BRAM1. During operation, the idle time of the entire system depends on the difference between the input storage time and the convolution calculation time.

[0097] The weight storage module specifically includes:

[0098] Four sets of weighted block random access memory, each set of weighted block random access memory includes four parallel weighted ping-pong block random access memory.

[0099] In addition, this embodiment also includes an intermediate cache storage module for storing feature map data that has not been fully accumulated during the accumulation calculation process.

[0100] The multi-channel shared computing array specifically includes:

[0101] The sixteen groups of multiply-accumulate units are arranged into a column of four groups to form a calculation array, including the first multiply-accumulate array, the second multiply-accumulate array, the third multiply-accumulate array and the fourth multiply-accumulate array;

[0102] Each multiply-accumulate array corresponds to a set of feature map ping-pong block random access memory, and each set of multiply-accumulate units in each multiply-accumulate array corresponds to a feature map ping-pong block random access memory.

[0103] The feature map data output from the four sets of feature map ping-pong block random access memory are respectively entered into the column-first multiply-accumulate unit of the corresponding multiply-accumulate array. At the same time, the weight data output from the sixteen weight ping-pong block random access memory are respectively entered into the corresponding multiply-accumulate unit of the corresponding multiply-accumulate array for multiply-accumulate operation.

[0104] The column tail multiply-accumulate unit of the fourth multiply-accumulate array is output to the column tail multiply-accumulate unit of the first multiply-accumulate array via the column tail multiply-accumulate units of the third and second multiply-accumulate arrays for multiply-accumulate operation; the operation result of the column tail multiply-accumulate unit of the first multiply-accumulate array is output to the feature map storage module.

[0105] Specifically, in the design of convolutional neural network accelerator architectures, the data transmission system determines the data transmission bandwidth, while the parallel computing structure determines the data computation speed, which is also one of the determining factors of CNN acceleration performance. The multi-channel shared computing array designed in this embodiment is as follows: Figure 1 As shown, it consists of 16 multiply-accumulate units (MACs), and the structure of the multiply-accumulate unit is as follows: Figure 4 As shown. During the calculation process, the four sets of feature map ping-pong block random access memory can output 144 feature map data in one clock cycle. Each set of feature map data is shared by four weight ping-pong block random access memory. Each set of weight multiply-accumulate units corresponds to one weight ping-pong block random access memory. After the feature map data and weight data enter the multiply-accumulate array, 36*4*4 (576) multiplication calculations can be completed in one clock cycle. Finally, each multiply-accumulate array outputs the calculation result to the addition tree for accumulation operation. After accumulation, it is sent to the feature map storage module, and the feature map data is exchanged with the external transmission channel through the output feature map storage FIFO.

[0106] In an optional embodiment of the present invention, the multiply-accumulate unit specifically includes:

[0107] The system comprises a first sparse binary convolution computation unit, a second sparse binary convolution computation unit, a third sparse binary convolution computation unit, and a fourth sparse binary convolution computation unit, as well as an addition tree unit.

[0108] The first sparse binary convolution calculation unit, the second sparse binary convolution calculation unit, the third sparse binary convolution calculation unit, and the fourth sparse binary convolution calculation unit perform binary convolution calculations on the input feature map data and weight data, respectively, and output all the calculation results to the addition tree unit for accumulation operation.

[0109] The first sparse binary convolution calculation unit, the second sparse binary convolution calculation unit, the third sparse binary convolution calculation unit, and the fourth sparse binary convolution calculation unit have the same structure and all include:

[0110] First data selector, second data selector, and adder;

[0111] The first data selector determines whether the input feature map data is zero; if so, it directly enters the adder; otherwise, the feature map data is output to the second data selector.

[0112] The second data selector performs binary convolution multiplication on the input feature map data according to the input weight data, and outputs the result to the adder;

[0113] The adder performs multiple summation operations on the calculation results and then outputs the result.

[0114] Specifically, the structure of the sparse binary convolution computation unit is as follows: Figure 5 As shown, since 1 bit cannot represent the actual weight -1, this embodiment uses 0 instead of -1 when storing the weight. Also, because the feature map data from the previous layer has been processed by the ReLU function, it contains many zero values. During convolution calculation, these zero values ​​can skip the calculation and directly enter the summation stage. For feature map data with numerical values, the calculation process is as follows:

[0115]

[0116] Where x represents the input feature map data. After the binary convolution multiplication is completed, the nine data points resulting from the 3*3 convolution are summed eight times to form a single data point, which is then fed into the accumulation module for further processing.

[0117] Example 2

[0118] like Figure 6 As shown, based on Embodiment 1, this invention also proposes a RISC-V on-chip system, including:

[0119] The FLASH module is used for weight data in convolutional neural networks;

[0120] The DDR module is used to store feature map data during the intermediate operations of the convolutional neural network, and to exchange feature map data with the feature map storage module.

[0121] The E203RISC-V soft core module is used to establish a custom instruction set and send custom instructions through the AXIInterconnect module to accelerate the binary weighted convolutional neural network accelerator.

[0122] The AXIInterconnect module is used to interconnect and forward the FLASH module, DDR module, and E203RISC-V soft core module with the binary weighted convolutional neural network accelerator through the AXI data transmission path;

[0123] The AXI data transmission path is used to establish a weight and feature map data transmission path with 4 AXIDMA and 1 AXICDMA, and to issue custom commands using the AXI instruction conversion bridge interface.

[0124] A cross-clock domain module is used to provide a 200MHz clock frequency to the binary weighted convolutional neural network accelerator;

[0125] And the binary weighted convolutional neural network accelerator described in Example 1.

[0126] Specifically, the FLASH module is a storage chip on the FPGA board, which is responsible for storing network weights in the entire neural network accelerator.

[0127] The DDR module is a storage chip on the FPGA board. Because the feature map data in the intermediate operation process of the neural network is large and cannot be stored using the on-chip resources, it is necessary to transfer the feature map data to the DDR for caching during the calculation process.

[0128] The E203RISC-V soft core module adopts the RISC-V open-source instruction set architecture. The E203 open-source processor soft core acts as the CPU to control the operation of the hardware accelerator in the entire neural network accelerator SoC system. This embodiment achieves hardware acceleration through custom instructions. The E203 soft core sends instructions via the AXI interface to control the neural network accelerator to accelerate the recognition algorithm and configure different parameters to achieve a configurable design. The RISC-V has a clock frequency of 32MHz, and the control commands issued by the E203 are connected to various peripherals through the AXIInterconnect module.

[0129] The AXIInterconnect module is a forwarding module for interconnecting the control bus and data bus of various peripherals.

[0130] The AXI data transmission path establishes a weighted and feature map data transmission system with 4 AXIDMA and 1 AXICDMA, and designs an AXI instruction conversion bridge interface to enable the issuance of custom configuration instructions.

[0131] The cross-clock domain module is designed to ensure a high-efficiency accelerator design. It increases the original 100MHz clock frequency to 200MHz through the cross-clock domain design, enabling the accelerator to operate at a 200MHz clock frequency and further improving computing performance.

[0132] The binary weighted convolutional neural network accelerator is the core component of the entire SoC system. The FSM finite state machine parses the instructions issued by the E203 and generates control commands to configure the entire accelerator's workflow. Multi-channel weight, feature map, and cache data storage further improves data reuse and storage retrieval efficiency. The multi-channel weight shared computing array reduces resource consumption and improves the overall MAC efficiency. After accumulation, pooling, and activation functions are completed, the data is moved back to DDR via AXIFIFO.

[0133] In an optional embodiment of the present invention, the AXI data transmission path specifically includes:

[0134] Four feature map transmission channels, one weight transmission channel, and one custom instruction transmission channel;

[0135] The four feature map transmission channels exchange feature map data with four sets of weight block random access memory through a set of input feature map first-in-first-out memory, an output feature map first-in-first-out memory, and four data selectors, respectively. The storage controller controls each data selector to send feature map configuration parameters and fills the feature map input to the input feature map first-in-first-out memory with the address.

[0136] One weight transmission channel exchanges weight data with four sets of weight block random access memory via a data selector;

[0137] A custom instruction transfer channel exchanges instruction data with a custom instruction register via the AXI instruction transfer module.

[0138] Specifically, due to the large size of CNN models, the weights and feature maps of all intermediate layers cannot be fully stored in on-chip BRAM. Therefore, when designing a CNN acceleration system, it is necessary to utilize external double data rate (DDR) and on-chip BRAM for large-scale data exchange. The design and construction of the data transmission system affects the data transmission bandwidth and is one of the determining factors of CNN acceleration performance. This embodiment constructs a system as follows... Figure 7 The AXI data transmission path of the CNN accelerator shown includes four feature map transmission channels, one weight transmission channel, and one custom instruction transmission channel.

[0139] For the feature map transmission channel, each feature map channel consists of two feature map ping-pong storage BRAMs, one output feature map storage FIFO, and one input feature map storage FIFO. The MUX is a data selector. The storage controller controls the MUX to send feature map configuration parameters and fills the feature map with addresses, thus realizing the padding operation in convolution without additional mapping units.

[0140] For the weight transmission channel, AXI_CDMA is used to map the weights from the source address to the destination address, and it consists of 16 weight storage BRAMs.

[0141] In an optional embodiment of the present invention, the custom instruction set specifically includes:

[0142] Accelerator reset instruction, clear storage flag, padding mode, network row and column and channel number instructions, feature map ping-pong block random access memory full flag instruction, weight block random access memory full flag instruction, input feature map first-in-first-out memory empty flag instruction, block random access memory ready to receive data flag instruction, output feature map first-in-first-out memory empty flag instruction, feature map ping-pong block random access memory empty flag instruction and weight block random access memory empty flag instruction.

[0143] At the start of the first convolutional computation, the system enters the write-ready state based on the block random access memory (RAM) readiness flag, and begins transmitting weight data. After weight data transmission is complete, the system enters the feature map data transmission state based on the weight block RAM full flag, and begins transmitting feature map data. After feature map data transmission is complete, the system enters the convolution computation state based on the feature map ping-pong block RAM full flag. After convolution computation is complete, the system enters the feature map data return state based on the input and output feature map FIFO memory empty flags. After the feature map data is returned to the DDR module, the system returns to the initial state based on the clear storage flags, padding mode, network row / column and channel number instructions.

[0144] Specifically, to achieve forward computation hardware acceleration for networks of varying sizes, this embodiment designs a general instruction set for the accelerator. Custom instructions are issued via an E203 soft-core CPU, each with a fixed width of 32 bits. Different instructions contain different information, including reset, convolution size, number of channels, and storage state. The CPU sends instructions to the PL, which is then parsed by the instruction processor and distributed to various modules by the controller. The modules work collaboratively to improve efficiency. The meanings of the custom instructions are shown in Table 1.

[0145] Table 1. Meaning of Accelerator Commands

[0146]

[0147]

[0148] like Figure 8 The diagram illustrates a first-layer convolution calculation as an example, with a total of five valid states. S0 indicates the BRAM is ready to be written; S1 indicates weights have been transferred; S2 indicates input feature map data has been transferred; S3 indicates convolution calculation begins; and S4 indicates the completed data is transferred to the DDR. The first-layer convolution calculation begins in state S0. Once the BRAM at the PL is ready to receive data, it enters state S1 to begin transferring weight data. After the weight data transfer is complete, it enters state S2 to transfer feature map data. After the feature map data is transferred, it enters state S3 to perform convolution calculation. When the output feature map FIFO is full, it enters state S4 to transfer the calculated feature map data back to the DDR. If the layer is not yet complete, it returns to state S3 to continue convolution. After all calculations are complete, the parameter configuration information for the next layer is sent, and the process returns to state S0.

[0149] In FPGAs, lookup tables (LUTs), flip-flops (FFs), and DSPs are crucial resources. In Vivado 2018.2, implementing common 8W / 8A quantization of data consumes 72 LUTs or 1 DSP for the multiplication of a feature map data and a weight. However, the 1W / 8A quantization method designed in this embodiment consumes only 7 LUTs for the multiplication of a feature map data and a weight, significantly reducing hardware resources.

[0150] When performing multiplication calculations using convolutional computation units based on binary weights, no multiplier DSP is required, so the calculation can be performed on any number of data points that come in.

[0151] In terms of storage resources, binary weights significantly reduce the number of weights. This embodiment uses a network with 16 convolutional layers as an example, constraining the weights of all convolutional layers to 1 bit. Table 2 shows a comparison of the model structure and related parameters of the full-precision network and the binary weight network. After binarization, the number of parameters in all convolutional layers is only 3.63% of the full-precision parameters. Specifically, the total number of parameters in all convolutional layers is only 0.68M.

[0152] Table 2 Analysis of Convolutional Layer Network Parameters

[0153]

[0154]

[0155]

[0156] MP stands for MaxPooling.

[0157] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0158] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0159] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0160] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0161] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A binary weight convolutional neural network accelerator, comprising: The application relates to a convolutional neural network accelerator, which comprises the following parts: an instruction analysis module, which is used for analyzing input configuration instructions and sending instruction analysis results to a finite state machine and an address generation module; the address generation module, which is used for generating base addresses of feature maps and weight data and base addresses of bias data according to the instruction analysis results; the finite state machine, which is used for controlling the jump flow of layers in the accelerator running process and issuing configuration parameters required in the accelerator calculation process according to the instruction analysis results; a shift sliding window module, which is used for storing and updating input feature map data of the feature map storage module by adopting a plurality of shift random access memories to form a first-in-first-out memory of series-in-parallel-out; the feature map storage module, which is used for storing feature map data in the intermediate operation process of the convolutional neural network by adopting a plurality of feature map ping-pong block random memories, exchanging the feature map data with an external transmission channel, and inputting the stored feature map data into a multi-channel shared calculation array through a data selector and a feature map base address; the weight storage module, which is used for storing weight data mapped and transmitted through the external transmission channel by adopting a plurality of weight block random memories, and inputting the stored weight data into the multi-channel shared calculation array through a data selector and a feature map base address; the multi-channel shared calculation array, which is used for constituting a calculation array by adopting a plurality of multiply-accumulate units, sharing each group of feature map data with a plurality of corresponding groups of weight data, and outputting the multiply-accumulate operation result to an accumulation module; the accumulation module, which is used for accumulating the calculation result of the multi-channel shared calculation array, adding bias, and performing a RELU operation to obtain the output result of the convolutional neural network, and inputting the output result into a batch normalization and pooling module; the batch normalization and pooling module, which is used for performing batch normalization processing and maximum value pooling processing on the output result of the convolutional neural network and outputting the result to the feature map storage module. 2.The binary weight convolutional neural network accelerator of claim 1, wherein, The shift sliding window module specifically comprises the following parts: three shift random access memories are adopted to form a first-in-first-out memory of series-in-parallel-out, and the first three columns of feature map data of the input feature map data of the feature map storage module are all stored in the three shift random access memories; convolution kernel sizes and sliding steps are set, four convolution kernels are slid on the feature map data stored in the three shift random access memories in parallel and convolution calculation is simultaneously performed; when the three columns of feature map data are all calculated, the next column of feature map data of the input feature map data of the feature map storage module is all stored in the three shift random access memories, and the four convolution kernels continue to slide on the feature map data stored in the three shift random access memories in parallel. 3.The binary weight convolutional neural network accelerator of claim 1, wherein, The feature map storage module specifically comprises the following parts: four groups of feature map ping-pong block random memories, and each group of feature map ping-pong block random memory comprises two parallel feature map ping-pong block random memories. 4.The binary weight convolutional neural network accelerator of claim 1, wherein, The weight storage module specifically comprises the following parts: four groups of weight block random memories, and each group of weight block random memory comprises four parallel weight ping-pong block random memories.

5. The binary weight convolutional neural network accelerator of claim 1, wherein, The multi-channel shared calculation array specifically comprises the following parts: each four groups of multiply-accumulate units in the sixteen groups of multiply-accumulate units form a column to constitute a calculation array, including a first multiply-accumulate array, a second multiply-accumulate array, a third multiply-accumulate array and a fourth multiply-accumulate array. Each multiply-accumulate array corresponds to a group of feature map ping-pong block random memories, and each group of multiply-accumulate units in each multiply-accumulate array corresponds to a feature map ping-pong block random memory; The feature map data output by the four groups of feature map ping-pong block random memories enter the column head multiply-accumulate units of the corresponding multiply-accumulate arrays, and the weight data output by the sixteen weight ping-pong block random memories enter the multiply-accumulate units of the corresponding multiply-accumulate arrays, respectively, to perform multiply-accumulate operation; The column tail multiply-accumulate unit of the fourth multiply-accumulate array sequentially outputs to the column tail multiply-accumulate unit of the first multiply-accumulate array through the column tail multiply-accumulate units of the third multiply-accumulate array and the second multiply-accumulate array to perform multiply-accumulate operation; and the operation result of the column tail multiply-accumulate unit of the first multiply-accumulate array is output to the feature map storage module.

6. The binary weight convolutional neural network accelerator of claim 5, wherein, The multiply-accumulate unit specifically comprises: Parallel first, second, third and fourth sparse binary convolution calculation units, and an addition tree unit; The first, second, third and fourth sparse binary convolution calculation units respectively perform binary convolution calculation on the input feature map data and weight data, and output the calculation results to the addition tree unit for accumulation operation.

7. The binary weight convolutional neural network accelerator of claim 6, wherein, The first, second, third and fourth sparse binary convolution calculation units are identical in structure and each comprises: A first data selector, a second data selector and an adder; The first data selector determines whether the input feature map data is zero value; if yes, the first data selector directly enters the adder; otherwise, the first data selector outputs the feature map data to the second data selector; The second data selector performs binary convolution multiplication operation on the input feature map data according to the input weight data, and outputs the operation result to the adder; The adder outputs the operation result after multiple addition operations.

8. A RISC-V system on chip, characterized in that, Comprise: A FLASH module for weight data of a convolutional neural network; A DDR module for storing feature map data in an intermediate operation process of the convolutional neural network, and exchanging the feature map data with the feature map storage module; An E203 RISC-V soft core module for establishing a custom instruction set and sending a custom instruction to the binary weight convolutional neural network accelerator through an AXI Interconnect module to control acceleration of the binary weight convolutional neural network accelerator; An AXI Interconnect module for interconnecting and forwarding the FLASH module, the DDR module and the E203 RISC-V soft core module with the binary weight convolutional neural network accelerator through an AXI data transmission path; An AXI data transmission path for establishing a weight and feature map data transmission path with four AXI DMAs and one AXI CDMA, and issuing a custom instruction through an AXI instruction conversion bridge interface; A cross-clock domain module for providing a 200MHz clock frequency to the binary weight convolutional neural network accelerator; And the binary weight convolutional neural network accelerator of any one of claims 1-7.

9. The RISC-V system on chip of claim 8, wherein, The AXI data transmission path specifically comprises: Four feature map transmission channels, one weight transmission channel and one custom instruction transmission channel; The four feature map transmission channels respectively exchange feature map data with four groups of weight block random memories through a group of input feature map first-in-first-out memories and output feature map first-in-first-out memories and four data selectors, and the storage controller controls each data selector to issue feature map configuration parameters and fill the input feature map first-in-first-out memories with feature maps through addresses; One weight transmission channel exchanges weight data with four groups of weight block random memories through one data selector; One custom instruction transmission channel exchanges instruction data with custom instruction registers through an AXI instruction transmission module.

10. The RISC-V system on chip of claim 8, wherein, The custom instruction set specifically comprises: Accelerator reset instruction, clear storage flag, padding mode, network row and channel number instruction, feature map ping-pong block random memory full flag instruction, weight block random memory full flag instruction, input feature map first-in-first-out memory empty flag instruction, block random memory ready to receive data flag instruction, output feature map first-in-first-out memory empty flag instruction, feature map ping-pong block random memory empty flag instruction and weight block random memory empty flag instruction; When the first layer convolution calculation starts, the block random memory ready to receive data flag instruction is entered into the state of being ready to be written, and the weight data transmission starts; when the weight data transmission is completed, the feature map data transmission state is entered according to the weight block random memory full flag instruction, and the feature map data transmission starts; when the feature map data transmission is completed, the convolution calculation state is entered according to the feature map ping-pong block random memory full flag instruction; when the convolution calculation is completed, the feature map data back transmission state is entered according to the input feature map first-in-first-out memory empty flag instruction and the output feature map first-in-first-out memory empty flag instruction; when the feature map data back transmission to the DDR module is completed, the initial state is returned according to the clear storage flag, padding mode, network row and channel number instruction.

Images