Deep neural network accelerator system and computing method based on inter-layer and intra-layer mixed precision operations

Abstract

The present invention discloses a deep neural network accelerator system and calculation method based on inter-layer and intra-layer mixed-precision operations, including a main controller, an instruction memory, a weight cache address controller, a feature map cache address controller, a high-order data encoding index table, a weight cache module group, a feature map cache module group, a multiply-accumulate array, and a quantizer group; the instruction memory is used to store instruction codes; the main controller controls other modules according to the instruction codes; the weight cache address controller and the feature map cache address controller are used for data handling; the high-order data encoding index table is used to store information of mixed-precision calculations within the layer; the weight cache module group and the feature map cache module group are used to cache weights and feature maps; the multiply-accumulate array is used to implement multiply-accumulate operations; and the quantizer group is used to quantize transition and activation functions. The present invention can improve parallelism, reduce the number of memory accesses, and reduce quantization errors, and is suitable for the deployment of quantized models of deep neural networks on edge devices.

Description

Technical Field

[0001] The present invention relates to the fields of deep neural networks and edge computing, and in particular to a deep neural network accelerator system and computing method based on inter-layer and intra-layer mixed precision operations. Background Art

[0002] Deep neural networks are increasingly being used in various scenarios, including image processing and industrial control. As deep neural network models grow in size, the demands for computing and memory access increase significantly, severely constraining their deployment on edge devices due to power consumption and resource constraints.

[0003] Deep neural networks primarily rely on a large number of multiplication-accumulation operations, making them both memory-intensive and compute-intensive. The sheer number of operands not only places significant demands on storage space, but also consumes orders of magnitude more energy than computation, creating a power bottleneck for the entire system. Furthermore, the high-volume, high-precision multiplication operations consume significant hardware resources and increase latency in the overall neural network's inference calculations.

[0004] To address the high computational and memory requirements of large-scale neural network models, a growing number of researchers are turning to quantized deployment of deep neural networks. Leveraging the high noise tolerance of neural networks, weights and feature maps in computations are processed with low precision before deployment and computation. Currently, mainstream neural network models use single-precision floating-point numbers for inference and training. After quantization, individual operands can be converted to 16-bit, 8-bit integers, or even binary values. This not only eliminates resource-intensive and power-intensive floating-point operations but also exponentially reduces storage and memory requirements. However, quantization inevitably results in a loss of model accuracy. In low-latency, high-precision scenarios such as autonomous driving, while quantization offers low power consumption and fast response times, it also introduces safety concerns.

[0005] Research has shown that larger numbers in deep neural networks are increasingly important to the network. However, if left unprocessed, the limited bit width after quantization can only express a limited range of values. Some operands that contribute significantly to the system are truncated, resulting in a sharp drop in network accuracy. The structures and data characteristics of different deep neural network models vary significantly, and even the data distribution varies between different layers of the same network model. This makes deployment on traditional neural network accelerators difficult and inefficient. Summary of the Invention

[0006] In order to address the shortcomings of the above-mentioned existing technologies, the present invention proposes a deep neural network accelerator system and calculation method based on inter-layer and intra-layer mixed precision operations, in order to improve the parallelism of operations, reduce the number of memory accesses, reduce calculation delays and reduce quantization errors, thereby being suitable for the deployment of quantized models of deep neural networks on edge devices.

[0007] The present invention adopts the following technical solutions to achieve the above-mentioned purpose:

[0008] The present invention is characterized by a deep neural network accelerator system based on inter-layer and intra-layer mixed precision operations, comprising: a main control block, a storage block, a storage control block and a computing block;

[0009] The main control block includes: a main controller and an instruction memory;

[0010] The storage block includes: a weight cache module group and a feature map cache module group;

[0011] The storage control block includes: a weight cache address controller, a feature map cache address controller and a high-order data encoding index table;

[0012] The calculation block includes: a weight shared cache, a feature map shared cache, a multiplication-accumulation array and a quantizer group;

[0013] The instruction memory is used to store instruction codes sent by the host computer, and the instruction codes include: mapping instructions, bias loading instructions, calculation instructions, quantization instructions and end instructions;

[0014] The main controller starts up under the command sent by the host computer and reads the current instruction from the instruction memory in sequence;

[0015] If the read current instruction is parsed as the mapping instruction, the main controller sends the mapping configuration information to the computing block according to the instruction information in the mapping instruction, and the computing block reconstructs the data path between the weight shared cache and the feature map shared cache, the feature map shared cache and the multiply-accumulate array, the weight cache module group and the feature map cache module group, and the feature map cache module group and the multiply-accumulate array according to the mapping configuration information;

[0016] If the read current instruction is parsed as the biased load instruction, the main controller sends the bias information in the biased load instruction to the weight cache address controller, so that the weight cache address controller generates an access signal according to the bias information and sends it to the weight cache module group, and the weight cache module group loads the bias information into the multiply-accumulate array;

[0017] If the current instruction read is parsed as the calculation instruction, the main controller parses and sends the calculation configuration information in the calculation instruction to the multiplication-accumulation array, and sends the operand information in the calculation instruction to the weight cache address controller and the feature map cache address controller respectively; the weight cache address controller and the feature map cache address controller both generate storage control signals and send them to the weight cache module group and the feature map cache module group respectively; the weight cache module group and the feature map cache module group move their own calculation data to the weight shared cache and the feature map shared cache respectively; the multiplication-accumulation array reconstructs the calculation accuracy according to the calculation configuration information, receives the operands sent by the weight shared cache and the feature map shared cache respectively, and completes the calculation;

[0018] If the read current instruction is parsed as the quantization instruction, the main controller sends the quantization configuration information and write-back information in the quantization instruction to the quantizer group; the quantizer group quantizes the calculation result sent by the multiplication-accumulation array according to the quantization configuration information, and then writes the quantization result back to the feature map cache module group according to the write-back information;

[0019] If the current instruction read is parsed as the end instruction, the main controller sends a completion signal to the host computer and enters an idle state; the storage layer sends the final calculation result to the external memory to wait for the host computer to read it.

[0020] The deep neural network hardware accelerator system of the present invention is also characterized in that the multiply-accumulate array includes N×N MAC units for reconstructing a grouping pattern with dimensions of (N×2)×(N / 2), (N / 2)×(N×2), or 1×(N×N); wherein N represents the one-dimensional length of the multiply-accumulate array;

[0021] The feature map cache module group includes 2×N feature map cache modules for reorganizing the grouping mode with dimensions of N×2, N / 2 or 1;

[0022] The weight cache module group includes N×N feature map cache modules, which are used to reorganize the grouping mode with a dimension of N / 2, N×2 or N×N.

[0023] The MAC unit is reconfigured into a 4-way M×M, 2-way (2×M)×M, or 1-way (2×M)×(2×M) multiplication and accumulation unit under the configuration of the calculation configuration information; wherein M represents the bit width;

[0024] The MAC unit includes: four multipliers, four adders, a shift unit, a multiplexer and an output buffer; each multiplier is used to calculate M×M data;

[0025] When the MAC unit is configured as a 4-way M×M multiplication and accumulation unit, the 4-way feature maps and weights are sent to both ends of the four multipliers respectively, the product results of the first and second multipliers are added through the first adder, the product results of the third and fourth multipliers are added through the second adder, the calculation results of the first and second adders are added again through the third adder, and the obtained calculation results are accumulated through the fourth adder and written into the output buffer;

[0026] When the MAC unit is configured as a 2-way (2×M)×M multiplication and accumulation unit, the upper M bits of the 2×M operands are sent to the first and second multipliers, and the lower M bits are sent to the third and fourth multipliers; the first-way M-bit operand is sent to the first and third multipliers, and the second-way M-bit operand is sent to the second and fourth multipliers; the product of the first and second multipliers is added by the first adder; the product of the third and fourth multipliers is added by the second adder; the result of the first adder is selected and shifted by the multiplexer and then added to the result of the second adder by the third adder; the result of the third adder is accumulated by the fourth adder and written to the output buffer;

[0027] When the MAC unit is configured as a 1-way (2×M)×(2×M) multiplication and accumulation unit, the first and second multipliers receive the lower M bits of the first operand, the third and fourth multipliers receive the upper M bits of the first operand, the first and fourth multipliers receive the upper M bits of the second operand, and the second and third multipliers receive the lower M bits of the second operand; the products of the first and third multipliers are added through the first adder, the result of the fourth multiplier is selected and shifted through the multiplexer, and then added to the product of the second multiplier through the second adder; the result of the first adder is selected and shifted through the multiplexer, and then added to the result of the second adder through the third adder, and the result of the third adder is accumulated through the fourth adder and written to the output buffer.

[0028] The high-order data encoding index table is used to store the number of high-order data of each block in the input feature map;

[0029] If the main controller interprets the calculation instruction as an intra-layer mixed precision mode, the main controller configures the multiplication-accumulation array to a 4-way M×M precision mode, and then instructs the feature map cache address controller to initiate an intra-layer mixed feature map transmission;

[0030] The feature map cache address controller first transfers the low-order data of the current block to the multiply-accumulate array for calculation, then accesses the high-order data encoding index table to obtain the number of high-order data of the current block, and then encodes the high-order data and transmits it to the multiply-accumulate array according to the number of high-order data; wherein the code length of the high-order data encoding is (I+W+O+M) bits, wherein the high (I+W+O) bits are encoding information, I bit is used to store input feature map information, W bits are used to store weight information, O bits are used to store output feature map information, and the low M bits are encoding data;

[0031] After receiving the high-order data code, the MAC unit first parses the code information and then calculates the code data according to the code information.

[0032] The quantizer group includes 2×N quantizers, each quantizer includes: a multiplier, a shift unit, a multiplexer, a comparator and a high-bit data code generation unit;

[0033] Any j-th quantizer receives the quantization coefficient sent by the main controller and the calculation result sent by the j-th group of MAC units, and completes the quantization operation through its own multiplier and shift unit to obtain a quantization result; <j≤2×N;

[0034] If the quantization configuration information sent by the main controller is an intra-layer hybrid scheme, the j-th quantizer judges the quantization result through its own comparator and then selects through the multiplexer whether to use the high-order data code generation unit to generate the high-order data code and the high-order data code index;

[0035] The j-th quantizer writes the quantization result and the high-order data code back to the feature map cache module group according to the write-back information sent by the main controller, and writes the high-order data code index into the high-order data code index table.

[0036] The present invention discloses a calculation method for a deep neural network accelerator system based on inter-layer and intra-layer mixed-precision operations, characterized by being applied to a deep neural network accelerator system consisting of a main control block, a storage block, a storage control block, and a calculation block; the main control block includes: a main controller and an instruction memory; the storage block includes: a weight cache module group and a feature map cache module group; the storage control block includes: a weight cache address controller, a feature map cache address controller, and a high-order data encoding index table; the calculation block includes: a weight shared cache, a feature map shared cache, a multiply-accumulate array, and a quantizer group; the calculation method is performed according to the following steps:

[0037] Step 1: The host computer sends the instruction code to the instruction memory, and sends the weights and input feature map to the external memory, and then starts the main controller; the instruction code includes: mapping instruction, bias loading instruction, calculation instruction, quantization instruction and end instruction;

[0038] Step 2: The external memory loads the input feature map and weights into the feature map cache module group and the weight cache module group respectively;

[0039] Step 3: The main controller reads a mapping instruction and remaps the data path between the storage block and the computing block according to instruction information in the mapping instruction;

[0040] Step 4: The main controller reads the bias load instruction and sends the bias information in the bias load instruction to the weight cache address controller; the weight cache address controller loads the data in the weight cache module group into the multiply-accumulate array;

[0041] Step 5: The main controller reads the calculation instruction, sends the calculation configuration information in the calculation instruction to the feature map cache address controller and the weight cache address controller, and sends the calculation configuration information to the multiplication-accumulation array; the feature map cache address controller accesses the feature map cache module group and the high-order data encoding index to transfer the feature map to the multiplication-accumulation array, and the weight cache address controller accesses the weight cache module group to transfer the weight to the multiplication-accumulation array and start calculation to obtain the output feature map;

[0042] Step 6: Repeat step 5 until all accumulations of the current output feature map are completed, the main controller reads the quantization instruction, sends the quantization configuration information to the quantizer group, and then controls the multiplication-accumulation array to output the result; the quantizer group quantizes the result output by the multiplication-accumulation array according to the quantization configuration information, writes the quantization result back to the feature map cache module group, and writes the high-order coding index back to the high-order coding index table;

[0043] Step 7: Repeat steps 4 to 6 until the single-layer network calculation is completed;

[0044] Step 8: Repeat steps 3 to 7 until all network calculations are completed.

[0045] Step 9: The main controller reads the end instruction and sends an output signal to the storage layer so that the storage layer moves the output feature map to the external memory; after sending the completion signal to the host computer, the main controller enters the idle state to wait for the host computer to read the result.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. The present invention can regroup the multiplication-accumulation arrays according to the computing characteristics, and remap the interconnection relationship between them and the cache according to the forms of different computing layers, thereby changing the proportion of input feature maps and weight reuse to optimize the data flow of each layer of calculation, thereby improving resource utilization while optimizing data access during calculation.

[0048] 2. The present invention utilizes the spatial repetitive calculation characteristics of convolution operations within the multiplication and accumulation units through multiplexing between computing units and multiplexing within computing units, further improving the reuse of feature maps and weights, improving the utilization rate of operands, and reducing the energy consumption caused by memory access.

[0049] 3. The present invention can support calculations of different precisions by subdividing and reconstructing the internal computing unit, thereby improving the flexibility of the accelerator, enabling it to adapt to model calculations of different precision modes, and reducing the difficulty of deploying hybrid quantization models.

[0050] 4. The present invention retains the large numerical feature maps that are truncated due to bit width limitations during the calculation process through a high-bit data encoding index table and a quantizer, thereby avoiding the truncation caused by the quantization accuracy of the original model and improving the calculation accuracy of the software quantization algorithm from the hardware level. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0052] Figure 2a It is a schematic diagram of the calculation data flow of the present invention;

[0053] Figure 2b is a schematic diagram of the multiplication-accumulation array reconstruction of the present invention;

[0054] Figure 3 It is a structural diagram of the mixed precision computing unit of the present invention;

[0055] Figure 4 It is a schematic diagram of the intra-layer hybrid computing mode of the present invention;

[0056] Figure 5 It is a structural diagram of the quantizer of the present invention. DETAILED DESCRIPTION

[0057] In this embodiment, a deep neural network accelerator system based on mixed precision operations between layers and within layers is provided. Figure 1 As shown, it includes: main control block, storage block, storage control block and computing block;

[0058] The main control block includes: main controller and instruction memory;

[0059] The storage block includes: a weight cache module group and a feature map cache module group;

[0060] The storage control block includes: a weight cache address controller, a feature map cache address controller and a high-order data encoding index table;

[0061] The computational block includes: weight shared cache, feature map shared cache, multiply-accumulate array, and quantizer group;

[0062] Based on the structure and quantization scheme of the network to be deployed, operators program the mapping from the software model to the hardware on the host computer. The host computer interprets the mapping code as executable instructions for the main controller and sends it to the instruction memory. The host computer then sends the network weights to the off-chip memory. Each inference is initiated by the host computer, which sends the input feature map to the off-chip memory and then activates the main controller. The main controller then sequentially reads the instructions from the instruction memory and controls and schedules the entire system based on the information in the instruction code.

[0063] The main controller mainly contains five types of executable instructions. The instruction memory is used to store the instruction codes sent by the host computer. The instruction codes include: mapping instructions, bias loading instructions, calculation instructions, quantization instructions and end instructions;

[0064] Mapping instructions: Remap the data paths between storage blocks, storage control blocks, and computing blocks to implement different data flows to optimize memory access modes during operations at different levels.

[0065] Biased load instruction: sent to the storage control block before calculation. The storage control block will parse the instruction, take out the bias from the storage block and load it into the output cache register inside the computing unit.

[0066] Compute instructions: These contain the computation mode, weights, and address starting information for the feature map, as well as the burst length. These instructions configure the storage control block and the computation block. The storage control block then moves operands according to data rules, and the computation block reconstructs the data according to the computation mode, receives the operands, and performs the computation.

[0067] Quantization instructions: After configuring the quantizer module, control the multiplication and accumulation array to output the calculation results to the quantizer module. The quantizer module quantizes the data and writes it back to the feature map cache module.

[0068] End instruction: When the end instruction is read, it means that the current reasoning has been completed and there is no subsequent computing task. At this time, the main controller will send a calculation completion message to the host computer and then wait for the host computer to read the result.

[0069] The main controller starts up under the command sent by the host computer and reads the current instruction from the instruction memory in sequence;

[0070] If the current instruction read is parsed as a mapping instruction, the main controller sends the mapping configuration information to the computing block according to the instruction information in the mapping instruction, and the computing block reconstructs the data path between the weight shared cache to the feature map shared cache, the feature map shared cache to the multiplication-accumulation array, the weight cache module group to the feature map cache module group, and the feature map cache module group to the multiplication-accumulation array according to the mapping configuration information;

[0071] If the current instruction read is parsed as a biased load instruction, the main controller sends the bias information in the biased load instruction to the weight cache address controller, so that the weight cache address controller generates an access signal according to the bias information and sends it to the weight cache module group, and the weight cache module group loads the bias information into the multiply-accumulate array;

[0072] If the current instruction read is parsed as a calculation instruction, the main controller parses and sends the calculation configuration information in the calculation instruction to the multiplication and accumulation array, and sends the operand information in the calculation instruction to the weight cache address controller and the feature map cache address controller respectively; the weight cache address controller and the feature map cache address controller both generate storage control signals and send them to the weight cache module group and the feature map cache module group respectively; the weight cache module group and the feature map cache module group move their own calculation data to the weight shared cache and the feature map shared cache respectively; the multiplication and accumulation array reconstructs the calculation accuracy according to the calculation configuration information, and then receives the operands sent by the weight shared cache and the feature map shared cache respectively and completes the calculation;

[0073] If the current instruction read is parsed as a quantization instruction, the main controller sends the quantization configuration information and write-back information in the quantization instruction to the quantizer group; the quantizer group quantizes the calculation result sent by the multiplication and accumulation array according to the quantization configuration information, and then writes the quantization result back to the feature map cache module group according to the write-back information;

[0074] If the current instruction read is interpreted as an end instruction, the main controller sends a completion signal to the host computer and enters the idle state; the storage layer sends the final calculation result to the external memory to wait for the host computer to read it.

[0075] In this embodiment, the multiply-accumulate array includes N×N MAC units for reconstructing a packet pattern of dimensions (N×2)×(N / 2), (N / 2)×(N×2), or 1×(N×N); where N represents the one-dimensional length of the multiply-accumulate array;

[0076] The feature map cache module group contains 2×N feature map cache modules, which are used to reorganize the grouping mode with dimensions of N×2, N / 2 or 1;

[0077] The weight cache module group contains N×N feature map cache modules, which are used to reorganize the grouping mode with dimensions of N / 2, N×2 or N×N.

[0078] The present invention calculates data flow such as Figure 2a As shown, in order to optimize the memory access mode of each layer according to the difference in the calculation method of each layer, the present invention proposes remapping of storage blocks and calculation blocks to realize the block division of calculation blocks, so as to realize weight sharing and feature map sharing of different modes. Each solid box in the figure is a feature map sharing calculation unit group, and the multiplication and accumulation units in the same group share the data sent by the same feature map cache module; each dotted box in the figure is a weight sharing calculation unit group, and the multiplication and accumulation units in the same group share the data sent by the same weight cache module. Figure 2b By remapping and adjusting the number of feature map shared computing unit groups and weight shared computing unit groups, as well as the number of computing units within each group, the 64 MAC units can support three multiplexing modes: I16-W4 (16-bit input, 4-bit weight, the same below), I4-W16, and I1-W64. This increases the number of output feature maps calculated simultaneously for different layer structures and reduces the number of accesses to weights and input feature maps.

[0079] Figure 2a Take a 16×16 input feature map, a 3×3 convolution kernel, and a convolution stride of 2 as an example. The multiply-accumulate array is mapped in an I4-W16 pattern, with the 64 multiply-accumulate units divided into 4 (×16) feature map sharing groups and 16 (×4) weight sharing groups. In this mapping pattern, the 16 RAMs in the weight cache module are mapped to the 16 weight sharing computation units to provide the required bandwidth for the weights. The feature map cache module is reorganized into 4 RAMs and mapped to the feature map sharing groups via a multiplexer. Each weight sharing group is responsible for calculating the convolution of the four parts of a complete output feature map from top to bottom. The 16 weight sharing groups can simultaneously calculate 16 output channels. In this mapping pattern, the 2D convolution operation is split into 1D convolutions, with each calculation performed on a row basis. The weights for each row are stored in the weight cache registers within the computation unit upon initial read-in. Overlapping feature maps within a row are selectively stored in the overlapping locations based on the multiplexing signal generated by the feature map address control module. If the current convolutional layer has 256 output channels, the input feature map only needs to be accessed 16 times to complete all calculations, while the weights and output feature map only need to be accessed and written back once. Since the computing unit designed in the present invention can support 4-way 8-bit-8-bit, 2-way 16-bit-8-bit, and 1-way 16-bit-16-bit multiplication and accumulation operations, the above-mentioned input feature map cache module and weight cache module are both stored with a 32-bit bit width. One 32-bit data store 4 channels of 8-bit operands or 2 channels of 16-bit operands.

[0080] In this embodiment, the MAC unit is reconfigured into a 4-way M×M, 2-way (2×M)×M, or 1-way (2×M)×(2×M) multiplication and accumulation unit under the configuration of the calculation configuration information; where M represents the bit width;

[0081] The MAC unit contains: four multipliers, four adders, a shift unit, a multiplexer, and an output buffer; each multiplier is used to calculate M×M data;

[0082] When the MAC unit is configured as a 4-way M×M multiplication and accumulation unit, the 4-way feature maps and weights are sent to the two ends of the four multipliers respectively. The product results of the first and second multipliers are added through the first adder, and the product results of the third and fourth multipliers are added through the second adder. The calculation results of the first and second adders are then added through the third adder. The obtained calculation results are accumulated through the fourth adder and written to the output buffer;

[0083] When the MAC unit is configured as a 2-way (2×M)×M multiplication and accumulation unit, the upper M bits of the 2×M operands are sent to the first and second multipliers, and the lower M bits are sent to the third and fourth multipliers; the first-way M-bit operand is sent to the first and third multipliers, and the second-way M-bit operand is sent to the second and fourth multipliers; the products of the first and second multipliers are added through the first adder; the products of the third and fourth multipliers are added through the second adder, the result of the first adder is shifted through the multiplexer, and then added to the result of the second adder through the third adder, and the result of the third adder is accumulated through the fourth adder and written to the output buffer;

[0084] When the MAC unit is configured as a 1-way (2×M)×(2×M) multiplication and accumulation unit, the first and second multipliers receive the lower M bits of the first operand, the third and fourth multipliers receive the upper M bits of the first operand, the first and fourth multipliers receive the upper M bits of the second operand, and the second and third multipliers receive the lower M bits of the second operand; the products of the first and third multipliers are added through the first adder, the result of the fourth multiplier is selected and shifted through the multiplexer, and then added to the product of the second multiplier through the second adder; the result of the first adder is selected and shifted through the multiplexer, and then added to the result of the second adder through the third adder, and the result of the third adder is accumulated through the fourth adder and written to the output cache.

[0085] The mixed precision computing unit of the present invention is as follows Figure 3As shown, each MAU consists of four 8-bit×8-bit multipliers and four adders, configured via configuration instructions sent by the main controller. These units can be configured in four precision modes: I8×W8 (8-bit input, 8-bit weight, same below), I16×W8, I8×W16, and I16×W16. When the precision mode is 8-bit×8-bit, a single MAU supports parallel computation of four input feature maps. When the number of input feature maps is less than four, clock gating is used to disable a certain number of multipliers to reduce power consumption. The products of the four-way convolutions are summed via three adders, and the resulting result is accumulated with the partial sum temporarily stored in the output register. When the precision modes are 16×8-bit and 8×16-bit, a single MAU can simultaneously compute the convolutions of two input feature maps. Each 8-bit operand is sent to two multipliers simultaneously, with the upper and lower 8 bits of the two 16-bit operands fed into four multipliers, respectively. The output results of the two multipliers responsible for calculating the upper 8 bits are added together, shifted left by 8 bits, and then added to the output sum of the two multipliers responsible for calculating the lower 8 bits to obtain the partial sum of the two convolutions. When the precision mode is 16bit×16bit, the upper 8 bits and lower 8 bits of the two 16-bit operands are assigned to four multipliers for cross-multiplication. The two upper 8-bit product results are shifted left by 16 bits, and the product results of the upper 8 bits and lower 8 bits are shifted left by 8 bits. The two lower 8-bit product results remain unchanged, and then the four products are added to obtain the correct result. Through this reconstruction of the multiplication and accumulation unit, different data precisions can be deployed in different layers of the same network, thereby realizing inter-layer hybrid computing.

[0086] In this embodiment, the high-order data encoding index table is used to store the number of high-order data of each block in the input feature map;

[0087] If the main controller interprets the calculation instruction as intra-layer mixed precision mode, the main controller configures the multiplication and accumulation array to 4-way M×M precision mode, and then instructs the feature map cache address controller to initiate an intra-layer mixed feature map transmission;

[0088] The feature map cache address controller first transfers the low-order data of the current block to the multiply-accumulate array for calculation, then accesses the high-order data encoding index table to obtain the number of high-order data of the current block, and then encodes the high-order data and transmits it to the multiply-accumulate array based on the number of high-order data. The code length of the high-order data encoding is (I+W+O+M) bits, where the high (I+W+O) bits are encoding information, I bits are used to store input feature map information, W bits are used to store weight information, O bits are used to store output feature map information, and the low M bits are encoding data.

[0089] After receiving the high-order data code, the MAC unit first parses the code information and then calculates the code data based on the code information.

[0090] like Figure 4 As shown, the intra-layer mixed-precision calculation flow of the present invention introduces a high-order data encoding (MSB-DC) index table compared to the single-precision data flow. In the intra-layer mixed-precision calculation mode, input feature maps with irregular bit widths are stored as blocks with one row, and each block is divided into an LSB portion and an MSB portion. The LSB portion is stored in the same 8-bit regular order as the intra-layer single-precision calculation. When each mixed calculation is initiated, the feature map address controller accesses and calculates the LSB portion according to the single-precision calculation method. The partial sum of the low-order calculation results is temporarily stored in the output register of the multiplication and accumulation unit. The feature map address controller then accesses the location pointed to by the current block in the MSB-DC index table and reads the number of MSB-DCs in the current block. If the read value is 0, the MSB portion is skipped and the calculation of the current block ends. If the value is not 0, the feature map address controller initiates a burst access to the feature map cache module with a length equal to the MSB-DC value, transferring the MSB-DC of the current block to the multiplication and accumulation unit. The MSB-DC data format is 16 bits, with bits [15:14] indicating the input channel corresponding to the current MSB, bits [13:12] indicating the corresponding weight, bits [11:8] indicating the part and position of the calculation result, and bits [7:0] storing the MSB value. The multiply-accumulate module parses the MSB-DC after receiving it. Based on the input channel and weight information, it indexes the operation weight in the internal weight register, multiplies the MSB value with it, shifts it, and accumulates it into the output buffer corresponding to the output information.

[0091] In this embodiment, the quantizer group includes 2×N quantizers, each of which includes: a multiplier, a shift unit, a multiplexer, a comparator, and a high-bit data code generation unit;

[0092] Any j-th quantizer receives the quantization coefficient sent by the main controller and the calculation result sent by the j-th group of MAC units, and completes the quantization operation through its own multiplier and shift unit to obtain the quantization result; <j≤2×N;

[0093] If the quantization configuration information sent by the main controller is an intra-layer hybrid scheme, the jth quantizer judges the quantization result through its own comparator and then selects through the multiplexer whether to use the high-order data code generation unit to generate the high-order data code and the high-order data code index;

[0094] The jth quantizer writes the quantization result and the high-order data code back to the feature map cache module group according to the write-back information sent by the main controller, and writes the high-order data code index into the high-order data code index table.

[0095] Figure 5The quantizer structure of the present invention is shown. In the offline pre-quantized network model, the weights and input feature maps of each layer have corresponding quantization resolutions q Wi and q Ii (Quantization resolution: Assuming that a parameter is distributed between [-1,1] and quantized to 8 bits, the quantization resolution ), so the quantization resolution of each layer’s bias and output feature map In order to make the next layer calculate correctly, it is necessary to perform transition quantization on the feature map before writing it back to the cache module. The transition quantization coefficient between the two layers is It is saved in the form of single-precision floating-point numbers and compiled into the instructions after being generated according to the quantization network model. When the multiplication and accumulation array completes the calculation, the main controller initiates a quantization write-back instruction, initiates an output instruction for the multiplication and accumulation array, and loads the quantization coefficients to the quantizer. The multiplication and accumulation array vectorizer then transmits the integer calculation results. The quantizer internally splits the quantizer coefficients in floating-point format and multiplies and shifts them with the calculation results to obtain an output feature map in integer format. If the quantization scheme of the next layer is single-precision calculation, the overflow is determined based on the bit width of the input feature map of the next layer and the output is truncated. If the quantization scheme of the next layer is mixed-precision calculation, the output feature map with overflow is encoded with high-bit data and written back to the feature map cache module.

[0096] In this embodiment, a computing method for a deep neural network accelerator system based on inter-layer and intra-layer mixed-precision operations is applied to a deep neural network accelerator system composed of a main control block, a storage block, a storage control block and a computing block; the main control block includes: a main controller and an instruction memory; the storage block includes: a weight cache module group and a feature map cache module group; the storage control block includes: a weight cache address controller, a feature map cache address controller and a high-order data encoding index table; the computing block includes: a weight shared cache, a feature map shared cache, a multiply-accumulate array and a quantizer group; the computing method includes the following steps.

[0097] Step 1: The host computer sends the instruction code to the instruction memory, and sends the weights and input feature map to the external memory, and then starts the main controller; wherein, the instruction code is programmed and compiled by combining the trained model parameters, network structure and target quantization scheme, and the instruction types include: mapping instructions, bias loading instructions, calculation instructions, quantization instructions and end instructions.

[0098] Step 2: The external memory loads the input feature map and weights into the feature map cache module group and the weight cache module group respectively.

[0099] Step 3: The main controller reads the mapping instruction and remaps the data path between the storage block and the computation block according to the instruction information in the mapping instruction. The specific process is as follows: After the main controller parses the MAP instruction to obtain the mapping mode parameters of the weight and feature map, it configures the multiplexer to reconstruct the control path between the weight address controller and the weight cache module group, and the feature map address controller and the feature map cache module group. It also reconstructs the data path between the weight cache module group and the multiply-accumulate array, and the feature map cache module group and the multiply-accumulate array, dividing the multiply-accumulate array into a weight sharing group and a feature map sharing group.

[0100] Step 4: The main controller reads the bias load instruction and sends the bias information in the bias load instruction to the weight cache address controller; the weight cache address controller loads the data in the weight cache module group into the multiplication and accumulation array. The specific process is: the main controller parses the BIAS instruction to obtain the bias address pointed to by the output feature map to be calculated, sends it to the weight address controller, and informs the weight address controller to initiate a bias loading task. After the weight address controller broadcasts the bias address to the weight cache module group in the current mapping mode, each weight cache module will broadcast the bias to each computing unit under the weight sharing group it is mounted on.

[0101] Step 5: The main controller reads the calculation instruction and sends the calculation configuration information in the calculation instruction to the feature map cache address controller and the weight cache address controller, and at the same time sends the calculation configuration information to the multiplication and accumulation array; the feature map cache address controller accesses the feature map cache module group and the high-order data encoding index to transfer the feature map to the multiplication and accumulation array, and the weight cache address controller accesses the weight cache module group to transfer the weights to the multiplication and accumulation array and start the calculation to obtain the output feature map. The specific process is as follows: the main controller sends the calculation configuration information parsed from the instruction to the multiplication and accumulation array, and sends the operand information to the address control module. The weight address control module receives the starting address and size of the convolution kernel, initiates a read operation of the weight cache module with a length equal to the number of weights in the convolution kernel, and loads the entire convolution kernel into the shared cache belonging to each weight sharing group. The feature map address control module receives the starting address and burst length of the feature map, initiates a read of the feature map row to the feature map cache module group, and sends it to the feature map sharing group mounted under it. If the current row is an overlapping row, it is also loaded into the feature map shared cache of the adjacent feature map shared group. When the adjacent shared group calculates this row of data, it skips accessing the feature map cache module group and directly obtains the data from the shared cache. Each multiplication and accumulation unit extracts a row of weights from its weight shared cache and loads them into the internal weight register. Each multiplication and accumulation unit receives feature maps from the feature map cache module or the feature map shared cache. Combined with the multiplexing signal generated by the address control module, the weights in the weight register are cyclically sent to the multiplication unit according to the calculation mode. The feature map will also selectively write the overlapping portion into the feature map register for later retrieval. If the current calculation is a single-precision calculation within the layer, the next step is entered. Otherwise, if the current calculation is a mixed-precision calculation within the layer, the main controller accesses the MSB-DC index table based on the current calculation position, retrieves the number of MSB-DCs in the current calculation block, and then sends it to the feature map address controller to initiate an MSB partial transfer. At the same time, after the multiplication and accumulation array is reconfigured to the mixed computing mode, the multiplication and accumulation unit will receive the MSB-DC data and perform calculations based on its information and then accumulate it into the output register.

[0102] Step 6: Repeat Step 5 until all accumulations of the current output feature map are complete. The main controller then reads the quantization instruction and sends the quantization configuration information to the quantizer group, which then controls the output of the multiply-accumulate array. The quantizer group quantizes the output of the multiply-accumulate array according to the quantization configuration information, writes the quantized result back to the feature map cache module group, and writes the high-order encoding index back to the high-order encoding index table. The specific process is as follows: the main controller sends the write-back address, quantization coefficient, and configuration information for the next layer contained in the quantization instruction to the quantizer group, and simultaneously sends an output instruction to the multiply-accumulate array. The multiply-accumulate unit transfers the output from the output register to the quantizer. The quantizer multiplies the output by the quantization coefficient to achieve quantization, and writes the result back to the feature map cache module in 16-bit or 8-bit format, depending on the precision mode of the next layer. If the precision mode of the next layer is non-hybrid computing, if the current result exceeds the maximum value that can be expressed by the specified bit width, it is truncated. If the precision mode of the next layer is hybrid computing, if there is overflow, the MSB-DC generation unit converts the overflowed high-order bits into a feature map in MSB-DC format.

[0103] Step 7. Repeat steps 4 to 6 until the single-layer network calculation is completed.

[0104] Step 8. Repeat steps 3 to 7 until all network calculations are completed.

[0105] Step 9. The main controller reads the end instruction and sends an output signal to the storage layer so that the storage layer moves the output feature map to the external memory; after sending the completion signal to the host computer, the main controller enters the idle state to wait for the host computer to read the result.

Claims

1. A deep neural network accelerator system based on mixed-precision operations between layers and within layers, characterized by: Main control block, storage block, storage control block and computing block; The main control block includes: a main controller and an instruction memory; The storage block includes: a weight cache module group and a feature map cache module group; The storage control block includes: a weight cache address controller, a feature map cache address controller and a high-order data encoding index table; The calculation block includes: a weight shared cache, a feature map shared cache, a multiplication-accumulation array and a quantizer group; The instruction memory is used to store instruction codes sent by the host computer, and the instruction codes include: mapping instructions, bias loading instructions, calculation instructions, quantization instructions and end instructions; The main controller starts up under the command sent by the host computer and reads the current instruction from the instruction memory in sequence; If the read current instruction is parsed as the mapping instruction, the main controller sends the mapping configuration information to the computing block according to the instruction information in the mapping instruction, and the computing block reconstructs the data path between the weight shared cache and the feature map shared cache, the feature map shared cache and the multiply-accumulate array, the weight cache module group and the feature map cache module group, and the feature map cache module group and the multiply-accumulate array according to the mapping configuration information; If the read current instruction is parsed as the bias load instruction, the main controller sends the bias information in the bias load instruction to the weight cache address controller, so that the weight cache address controller generates an access signal according to the bias information and sends it to the weight cache module group, and the weight cache module group loads the bias information into the multiply-accumulate array; If the current instruction read is parsed as the calculation instruction, the main controller parses and sends the calculation configuration information in the calculation instruction to the multiplication-accumulation array, and sends the operand information in the calculation instruction to the weight cache address controller and the feature map cache address controller respectively; the weight cache address controller and the feature map cache address controller both generate storage control signals and send them to the weight cache module group and the feature map cache module group respectively; the weight cache module group and the feature map cache module group move their own calculation data to the weight shared cache and the feature map shared cache respectively; the multiplication-accumulation array reconstructs the calculation accuracy according to the calculation configuration information, receives the operands sent by the weight shared cache and the feature map shared cache respectively, and completes the calculation; If the read current instruction is parsed as the quantization instruction, the main controller sends the quantization configuration information and write-back information in the quantization instruction to the quantizer group; the quantizer group quantizes the calculation result sent by the multiplication-accumulation array according to the quantization configuration information, and then writes the quantization result back to the feature map cache module group according to the write-back information; If the current instruction read is parsed as the end instruction, the main controller sends a completion signal to the host computer and enters an idle state; the storage layer sends the final calculation result to the external memory to wait for the host computer to read it.

2. The deep neural network hardware accelerator system according to claim 1, characterized in that: The multiply-accumulate array comprises N×N MAC units for reconstructing a packet pattern of dimensions (N×2)×(N / 2), (N / 2)×(N×2), or 1×(N×N); wherein N represents a one-dimensional length of the multiply-accumulate array; The feature map cache module group includes 2×N feature map cache modules for reorganizing the grouping mode with dimensions of N×2, N / 2 or 1; The weight cache module group includes N×N feature map cache modules, which are used to reorganize the grouping mode with a dimension of N / 2, N×2 or N×N.

3. The deep neural network hardware accelerator system according to claim 2, characterized in that: The MAC unit is reconfigured into a 4-way M×M, 2-way (2×M)×M, or 1-way (2×M)×(2×M) multiplication and accumulation unit under the configuration of the calculation configuration information; wherein M represents the bit width; The MAC unit includes: four multipliers, four adders, a shift unit, a multiplexer and an output buffer; each multiplier is used to calculate M×M data; When the MAC unit is configured as a 4-way M×M multiplication and accumulation unit, the 4-way feature maps and weights are sent to both ends of the four multipliers respectively, the product results of the first and second multipliers are added through the first adder, the product results of the third and fourth multipliers are added through the second adder, the calculation results of the first and second adders are added again through the third adder, and the obtained calculation results are accumulated through the fourth adder and written into the output buffer; When the MAC unit is configured as a 2-way (2×M)×M multiplication and accumulation unit, the upper M bits of the 2×M operands are sent to the first and second multipliers, and the lower M bits are sent to the third and fourth multipliers; the first-way M-bit operand is sent to the first and third multipliers, and the second-way M-bit operand is sent to the second and fourth multipliers; the product of the first and second multipliers is added through the first adder; the product of the third and fourth multipliers is added through the second adder, the result of the first adder is selected and shifted by the multiplexer, and then added to the result of the second adder through the third adder, and the result of the third adder is accumulated through the fourth adder and written into the output buffer; When the MAC unit is configured as a 1-way (2×M)×(2×M) multiplication and accumulation unit, the first and second multipliers receive the lower M bits of the first operand, the third and fourth multipliers receive the upper M bits of the first operand, the first and fourth multipliers receive the upper M bits of the second operand, and the second and third multipliers receive the lower M bits of the second operand; the products of the first and third multipliers are added through the first adder, the result of the fourth multiplier is selected and shifted through the multiplexer, and then added to the product of the second multiplier through the second adder; the result of the first adder is selected and shifted through the multiplexer, and then added to the result of the second adder through the third adder, and the result of the third adder is accumulated through the fourth adder and written to the output buffer.

4. The deep neural network hardware accelerator system according to claim 2, characterized in that: The high-order data encoding index table is used to store the number of high-order data of each block in the input feature map; If the main controller interprets the calculation instruction as an intra-layer mixed precision mode, the main controller configures the multiplication-accumulation array to a 4-way M×M precision mode, and then instructs the feature map cache address controller to initiate an intra-layer mixed feature map transmission; The feature map cache address controller first transfers the low-order data of the current block to the multiply-accumulate array for calculation, then accesses the high-order data encoding index table to obtain the number of high-order data of the current block, and then encodes the high-order data and transmits it to the multiply-accumulate array according to the number of high-order data; wherein the code length of the high-order data encoding is (I+W+O+M) bits, wherein the high (I+W+O) bits are encoding information, I bit is used to store input feature map information, W bits are used to store weight information, O bits are used to store output feature map information, and the low M bits are encoding data; After receiving the high-order data code, the MAC unit first parses the code information and then calculates the code data according to the code information.

5. The deep neural network hardware accelerator system according to claim 4, characterized in that: The quantizer group includes 2×N quantizers, each quantizer includes: a multiplier, a shift unit, a multiplexer, a comparator and a high-bit data code generation unit; Any j-th quantizer receives the quantization coefficient sent by the main controller and the calculation result sent by the j-th group of MAC units, and completes the quantization operation through its own multiplier and shift unit to obtain a quantization result; <j≤2×N; If the quantization configuration information sent by the main controller is an intra-layer hybrid scheme, the j-th quantizer judges the quantization result through its own comparator and then selects through the multiplexer whether to use the high-order data code generation unit to generate the high-order data code and the high-order data code index; The j-th quantizer writes the quantization result and the high-order data code back to the feature map cache module group according to the write-back information sent by the main controller, and writes the high-order data code index into the high-order data code index table.

6. A computing method for a deep neural network accelerator system based on mixed precision operations between layers and within layers, characterized by Applied to a deep neural network accelerator system consisting of a main control block, a storage block, a storage control block, and a computing block; The main control block includes: a main controller and an instruction memory; the storage block includes: a weight cache module group and a feature map cache module group; the storage control block includes: a weight cache address controller, a feature map cache address controller and a high-order data encoding index table; the calculation block includes: a weight shared cache, a feature map shared cache, a multiply-accumulate array and a quantizer group; the calculation method is performed according to the following steps: Step 1: The host computer sends the instruction code to the instruction memory, and sends the weights and input feature map to the external memory, and then starts the main controller; the instruction code includes: mapping instruction, bias loading instruction, calculation instruction, quantization instruction and end instruction; Step 2: The external memory loads the input feature map and weights into the feature map cache module group and the weight cache module group respectively; Step 3: The main controller reads a mapping instruction and remaps the data path between the storage block and the computing block according to instruction information in the mapping instruction; Step 4: The main controller reads the bias load instruction and sends the bias information in the bias load instruction to the weight cache address controller; the weight cache address controller loads the data in the weight cache module group into the multiply-accumulate array; Step 5: The main controller reads the calculation instruction, sends the calculation configuration information in the calculation instruction to the feature map cache address controller and the weight cache address controller, and sends the calculation configuration information to the multiplication-accumulation array; the feature map cache address controller accesses the feature map cache module group and the high-order data encoding index to transfer the feature map to the multiplication-accumulation array, and the weight cache address controller accesses the weight cache module group to transfer the weight to the multiplication-accumulation array and start calculation to obtain the output feature map; Step 6: Repeat step 5 until all accumulations of the current output feature map are completed, the main controller reads the quantization instruction, sends the quantization configuration information to the quantizer group, and then controls the multiplication-accumulation array to output the result; the quantizer group quantizes the result output by the multiplication-accumulation array according to the quantization configuration information, writes the quantization result back to the feature map cache module group, and writes the high-order coding index back to the high-order coding index table; Step 7: Repeat steps 4 to 6 until the single-layer network calculation is completed; Step 8: Repeat steps 3 to 7 until all network calculations are completed. Step 9: The main controller reads the end instruction and sends an output signal to the storage layer so that the storage layer moves the output feature map to the external memory; after sending the completion signal to the host computer, the main controller enters the idle state to wait for the host computer to read the result.

Images