Neural network acceleration method, device and accelerator based on algorithm and hardware collaboration

Abstract

The present invention relates to a neural network acceleration method, device and accelerator based on algorithm and hardware collaboration, the method comprising: based on a sparsity threshold, processing the value-level index of a quantized tensor to generate layer metadata, and dynamically adjusting the sparsity mode; identifying the position and number of non-zero bits according to a symbol conversion and a bit lookup table, and placing the processed data into a bit grouping to eliminate non-contributing bits; processing the bit grouping, and allocating the relevant inputs and tensors of the adaptive indexing mechanism extracted from the layer metadata to the bit elastic processing unit, and performing shift addition calculations according to the number of non-zero bits; based on a column-based calculation array, instructing all arrays to complete multiplication operations within a preset cycle, and storing partial products and data streams corresponding to their respective bit widths. The invention enables the accelerator controller to efficiently skip these zero bits during the calculation process, and at the same time, improves the performance and applicability of the bit-level accelerator.

Description

Technical Field

[0001] The present invention relates to the field of neural networks, and in particular to a neural network acceleration method, device and accelerator based on algorithm and hardware collaboration. Background Art

[0002] Traditional bit-level accelerators can be divided into three categories: bit-interleaved, bit-parallel, and bit-serial. Most of them focus on bit-level sparsity within values. Some bit-level accelerators propose partially adopted hardware-efficient approximation strategies, which will lead to a corresponding decrease in inference accuracy. In order to balance tensor precision and model accuracy, more and more accelerators explore and apply bit-level sparsity to achieve significant energy efficiency improvements. Although bit sparsity is used for acceleration, the random distribution of zero bits makes it impossible for the accelerator controller to efficiently skip these zero bits during the calculation process, thus affecting the efficient utilization of hardware. This challenge is particularly significant in fine-grained parallel tasks, where this imbalance may limit the performance and wider applicability of bit-level accelerators, so it needs to be improved. Summary of the invention

[0003] Based on this, it is necessary to provide a neural network acceleration method, device and accelerator based on the collaboration of algorithm and hardware to address the problem that although traditional bit-level accelerators utilize bit sparsity for acceleration, the random distribution of zero bits makes it impossible for the accelerator controller to efficiently skip these zero bits during the calculation process, thereby affecting the efficient utilization of the hardware. This imbalance may limit the performance and wider applicability of bit-level accelerators.

[0004] The present invention provides a neural network acceleration method based on algorithm and hardware collaboration, the method comprising:

[0005] Based on a sparsity threshold, processing a value-level index of a quantized tensor to generate layer metadata and dynamically adjust a sparsity mode, wherein the layer metadata includes a start address, an end address, an index address, and a dense or sparse flag;

[0006] identifying the location and number of non-zero bits based on the sign conversion and the bit lookup table, and placing the processed data into bit groups to eliminate non-contributing bits;

[0007] Processing bit grouping and distributing relevant inputs and tensors of the adaptive indexing mechanism extracted from the layer metadata to the bit elastic processing units, and performing shift-add operations according to the number of non-zero bits;

[0008] Based on the column-based computing array, all arrays are instructed to complete multiplication operations within a preset cycle, and store partial products and data streams corresponding to their respective bit widths.

[0009] In one embodiment, the processing of value-level indexes of quantized tensors based on sparsity thresholds to generate layer metadata and dynamically adjust the sparsity mode includes:

[0010] Obtain the starting address, ending address, index address, and density or sparsity flag of each layer of the neural network, and identify the sparsity ratio of each layer of the neural network;

[0011] In response to when the sparsity of the layer neural network is less than a sparsity threshold, storing the tensor value in a dense format to generate a continuous index range;

[0012] In response to when the sparsity of the layer neural network exceeds a sparsity threshold, the value of the non-zero element and its index position are stored.

[0013] In one embodiment, identifying the sparsity ratio of each layer of the neural network includes:

[0014] Get the number of non-zero elements in each layer of the neural network;

[0015] The number of non-zero elements is input into a neural network model, and a sparse ratio result is output. The neural network model is trained using samples of the number of non-zero elements and sparse ratio samples in each layer of the neural network as training data.

[0016] In one embodiment, the step of identifying the position and number of non-zero bits according to the sign conversion and the bit lookup table, and placing the processed data into bit groups to eliminate non-contributing bits, comprises:

[0017] Extract the sign flag from the value-level tensor and convert the signed value of the current tensor to an unsigned integer;

[0018] Based on the high-order lookup table and the low-order lookup table, determining the number of non-zero bits and their positions to generate a bit grouping structure;

[0019] In the process of generating the bit grouping, all zero bit positions are removed, and the generated bit grouping data is stored in the pre-buffer area.

[0020] In one embodiment, the processing bit grouping and distributing the relevant input and tensor of the adaptive indexing mechanism extracted from the layer metadata to the bit elastic processing unit, and performing shift-addition calculation according to the number of non-zero bits, comprises:

[0021] Dynamically adjust the size of the bit grouping based on the sparsity threshold extracted from the layer metadata and the computational power of the column-based computation array;

[0022] Based on the adaptive indexing mechanism, the index information is extracted from the layer metadata and pre-stored in a dedicated buffer of the elastic processing unit;

[0023] Extract the bit grouping information and the corresponding input and weight tensors from the pre-buffer area and load them into a dedicated buffer of the bit elastic processing unit;

[0024] Parse the non-zero bit positions in the bit grouping and locate the specific position of each bit in the calculation;

[0025] Perform sign addition on the intermediate results based on the sign flag;

[0026] An accumulation operation is performed on the non-zero bits within the group to generate a partial product result.

[0027] In one embodiment, performing sign addition on the intermediate result based on the sign flag includes:

[0028] In response to two input values ​​of the intermediate result having the same sign, calculating the sum of the two;

[0029] In response to two input values ​​of the intermediate result having different signs, a difference between the two is calculated and the sign of the result is determined based on the larger absolute value.

[0030] The present invention also provides a neural network acceleration device based on algorithm and hardware collaboration, comprising:

[0031] An adjustment module, configured to process the value-level index of the quantized tensor based on a sparsity threshold to generate layer metadata and dynamically adjust the sparsity mode, wherein the layer metadata includes a start address, an end address, an index address, and a density or sparsity flag;

[0032] an identification module for identifying the position and number of non-zero bits based on the sign conversion and the bit lookup table, and placing the processed data into bit groups to eliminate non-contributing bits;

[0033] an allocation module, configured to process bit grouping and allocate relevant inputs and tensors of the adaptive indexing mechanism extracted from the layer metadata to the bit elastic processing units, and perform shift-addition calculations according to the number of non-zero bits;

[0034] The storage module is used to instruct all arrays to complete multiplication operations within a preset cycle based on column-based calculation arrays, and to store partial products and data streams corresponding to respective bit widths.

[0035] The present invention also provides a neural network accelerator based on algorithm and hardware collaboration, including a top-level controller, a pre-cache scheduling manager, a memory, a bit-elastic processing unit and a column-based processing array. The top-level controller processes instructions and distributes corresponding control signals to the pre-cache scheduling manager, the memory and the bit-elastic processing unit. The pre-cache scheduling manager is used to allocate inputs and tensors to the bit-elastic processing unit, and retain layer metadata and bit grouping in the pre-cache module and the scheduler module until the processing array is fully configured. The input buffer and tensor buffer in the pre-cache scheduling manager transmit elements in a column manner and perform iterations of the model level. The column-based processing array uses the bit-elastic processing unit to perform multiplication operations and uses a signed adder to process intermediate results. The memory is used to calculate arrays based on the column, instruct all arrays to complete multiplication operations within a preset cycle, and store partial products and data streams corresponding to their respective bit widths.

[0036] In one of the embodiments, the bit elastic processing unit includes a shift unit, an accumulation unit, a multiplexer and a signed adder, wherein the shift unit is used to extract non-zero bit configurations, the accumulation unit is used to aggregate position information, the multiplexer is used to dynamically select an input path according to a control signal, and the signed adder is used to perform a final accumulation operation.

[0037] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described neural network acceleration methods based on the collaboration of algorithms and hardware.

[0038] The above-mentioned neural network acceleration method, device and accelerator based on algorithm and hardware collaboration, on the one hand, adopt static sparse and dense storage formats, combined with dynamic bit sparsity utilization schemes, aim to minimize non-contributing bits, so that the accelerator controller can efficiently skip these zero bits during the calculation process. At the same time, the random distribution of non-zero bits is retained to maintain tensor accuracy, providing a basis for solving computational dependence and low hardware utilization problems, and improving the performance and applicability of bit-level accelerators. On the other hand, through the bit-elastic processing unit with bit grouping to achieve balanced workload distribution across different bit widths, the column-based computing array further routes sparse or dense workloads to the bit-elastic processing unit, realizing scalable pre-cache scheduling management to cope with random non-zero bit operations and inputs, thereby improving computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 A flowchart of a neural network acceleration method based on collaboration between algorithms and hardware in one embodiment;

[0041] Figure 2 A schematic diagram of a process for dynamically adjusting a sparsity mode in an embodiment;

[0042] Figure 3 is a flowchart of a bit sparsity and threshold adjustment algorithm in one embodiment;

[0043] Figure 4 A schematic diagram of a process for identifying the sparsity ratio of each layer of a neural network in one embodiment;

[0044] Figure 5 is a schematic diagram of a process of eliminating non-contributing bits in one embodiment;

[0045] Figure 6 A schematic diagram of a process of allocating inputs and tensors to elastic processing units in one embodiment;

[0046] Figure 7 It is a schematic diagram of a process of performing symbol addition on an intermediate result based on a symbol flag in an embodiment;

[0047] Figure 8 A schematic diagram of the structure of a neural network acceleration device based on the collaboration of algorithms and hardware in one embodiment;

[0048] Fig. 9 A schematic diagram of the structure of a neural network accelerator based on collaboration between algorithms and hardware in one embodiment;

[0049] Fig.10 A schematic diagram of a neural network accelerator principle framework based on algorithm and hardware collaboration in one embodiment;

[0050] Fig.11 A schematic diagram of a pre-cache-scheduler management and bit elasticity processing unit in one embodiment;

[0051] Fig.12 FIG. 4 is a schematic diagram of a processing unit array arranged in columns in one embodiment.

[0052] Reference numerals:

[0053] 810, adjustment module; 820, identification module; 830, allocation module; 840, storage module; 910, top-level controller; 920, pre-cache scheduling manager; 930, memory; 940, bit elastic processing unit; 941, shift unit; 942, accumulation unit; 943, multiplexer; 944, signed adder; 950, column processing array. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Quantized deep learning models effectively compress model size and runtime overhead by representing quantized tensors as low-precision integers. However, the research community has mainly focused on improving the expressiveness of the models (e.g., the tensor size of GPT-4 reaches 170 trillion, which is more than 1,000 times larger than GPT-3.5), which poses challenges for memory- and energy-constrained devices. Traditional deep learning accelerators provide a potential solution to these challenges by exploiting computational parallelism and numerical sparsity. It is worth noting that numerical sparsity is limited and these accelerators cannot fully exploit the scalability of dynamic bit width, thus limiting the optimal utilization of the hardware. To address this limitation, previous studies have shown that bit-level accelerators that skip "zero bits" in operands are competitive in performance. Therefore, deep learning accelerators are moving towards bit-level research to support model advancement and improve computational efficiency.

[0056] Traditional bit-level accelerators can be divided into three categories: bit-interleaved, bit-parallel, and bit-serial. Most of them focus on bit-level sparsity within values. Some bit-level accelerators propose partially adopted hardware-efficient approximation strategies, which will lead to a corresponding decrease in inference accuracy. In order to balance tensor precision and model accuracy, more and more accelerators explore and apply bit-level sparsity to achieve significant energy efficiency improvements. Although bit sparsity is used for acceleration, the random distribution of zero bits makes it impossible for the accelerator controller to efficiently skip these zero bits during the calculation process, thus affecting the efficient utilization of hardware. This challenge is particularly prominent in fine-grained parallel tasks, where this imbalance may limit the performance and wider applicability of bit-level accelerators.

[0057] To address this issue, a potential solution is to more effectively combine value-level sparsity and bit-level sparsity to reduce the number of non-contributing bits in deep learning models. However, bit-level sparsity varies between computing units, resulting in a large amount of ineffective computation during parallel processing. Therefore, precision scalable accelerators developed through algorithm-hardware co-design represent a promising approach to address this irregularity. Quantization of deep learning models with more than 6.7 billion parameters faces the challenge of maintaining accuracy during inference. This makes it particularly important to ensure computational efficiency while maintaining model accuracy during inference. The algorithm design should manage the irregular distribution of zero bits in the operands, while the hardware simplifies the control logic and develops an efficient sparse indexing mechanism. The algorithm-hardware co-design ensures wider applicability and minimizes computational overhead.

[0058] Combine the following Figure 1-Figure 12 The present invention describes a neural network acceleration method, device and accelerator based on algorithm and hardware collaboration.

[0059] like Figure 1 As shown, in one embodiment, a neural network acceleration method based on algorithm and hardware collaboration includes the following steps:

[0060] Step S100, based on the sparsity threshold, processing the value-level index of the quantized tensor to generate layer metadata and dynamically adjust the sparsity mode, the layer metadata includes a start address, an end address, an index address, and a density or sparsity flag.

[0061] It should be noted that the start address and the end address are used to identify the address range of the current layer in the memory, the index address is used to quickly access the non-zero bit information in the group, and the density or sparsity flag indicates whether the sparsity exceeds the sparsity threshold.

[0062] The start address, end address, and index address are gradually updated as the base address of each layer is adjusted to ensure adaptability across network layers. The access of each layer of the neural network distinguishes non-zero elements in the memory and the adaptive indexing mechanism, thereby reducing invalid memory access. This address-based hierarchical access reduces the additional delay introduced by the conversion between neural network layers and reduces access conflicts across neural network layers.

[0063] By automatically determining the sparsity pattern of tensors with the help of sparsity thresholds and efficiently switching between sparse and dense processing, it supports dynamic adaptation of different levels and sparsity distributions. At the same time, with the help of layer metadata, the sparsity status, storage location and index range of each layer of tensors are recorded to achieve dynamic adjustment of sparsity across network layers while maximizing memory access efficiency.

[0064] By combining dynamic sparsity patterns with value-level index processing, efficient use of storage and computing resources is achieved, laying a technical foundation for high-performance neural network inference acceleration.

[0065] Step S200, identifying the position and number of non-zero bits according to the sign conversion and the bit lookup table, and placing the processed data into bit groups to eliminate non-contributing bits.

[0066] In neural network reasoning, especially when calculating quantized models, tensor data is usually represented by a finite bit width. Due to sparsity and the sparse distribution of non-zero data, many bits may not participate in effective calculations, forming non-contributory bits. If these non-contributory bits are processed directly, it will lead to unnecessary computational overhead and waste of hardware resources. By identifying non-zero bits and their positions, and organizing the processed data into efficient bit groupings, the computational efficiency can be significantly improved and the hardware resource usage can be reduced, while at the same time, the randomness of zero bits can be retained.

[0067] Step S300 , processing bit grouping, and distributing the relevant input and tensor of the adaptive indexing mechanism extracted from the layer metadata to the bit elasticity processing unit, and performing shift-addition calculation according to the number of non-zero bits.

[0068] By processing bit grouping and combining the adaptive indexing mechanism of layer metadata, with the bit elastic processing unit and shift-add calculation, it can produce significant effects in the following aspects:

[0069] 1. Maximize the utilization of hardware resources: improve the parallelism of computing units and reduce the idle rate.

[0070] 2. Reduce power consumption and latency: avoid redundant calculations and reduce complex operations.

[0071] 3. Improved data efficiency: Optimize storage and transmission, and reduce bandwidth requirements.

[0072] 4. Enhanced adaptability: Dynamically adapt to the sparsity of different layers to maintain stable performance.

[0073] 5. Computational accuracy guarantee: Quickly generate accurate partial product results and support efficient reasoning for deep learning.

[0074] Step S400 , based on the column-based calculation array, instruct all arrays to complete the multiplication operation within a preset period, and store partial products and data streams corresponding to their respective bit widths.

[0075] By storing partial products and data streams corresponding to their respective bit widths, not only the storage efficiency and data transmission performance are optimized, but also the system energy consumption is effectively reduced while the computing performance and flexibility are improved.

[0076] This neural network acceleration method based on the collaboration of algorithms and hardware, on the one hand, aims to minimize non-contributing bits by adopting static sparse and dense storage formats and combining them with dynamic bit sparsity utilization schemes, so as to facilitate the accelerator controller to efficiently skip these zero bits during the calculation process. At the same time, the random distribution of non-zero bits is retained to maintain tensor accuracy, providing a basis for solving the problems of computational dependence and low hardware utilization, and improving the performance and applicability of bit-level accelerators. On the other hand, through the bit-elastic processing unit with bit grouping to achieve balanced workload distribution across different bit widths, the column-based computing array further routes sparse or dense workloads to the bit-elastic processing unit, realizing scalable pre-cache scheduling management to cope with random non-zero bit operations and inputs, thereby improving computing efficiency.

[0077] In this embodiment, see Figure 2 and Figure 3 , the value-level indexes of the quantized tensor are processed separately based on the sparsity threshold to generate layer metadata and dynamically adjust the sparsity mode, including the following steps:

[0078] Step S110, obtaining the starting address, ending address, index address and density or sparsity flag of each layer of the neural network, and identifying the sparsity ratio of each layer of the neural network.

[0079] The sparse ratio refers to the ratio of non-zero elements to the total elements in a neural network layer, which can be expressed as:

[0080]

[0081] in, The higher the value, the stronger the sparsity and the lower the computational complexity.

[0082] Step S120, in response to when the sparsity of the layer neural network is less than a sparsity threshold, storing the tensor value in a dense format to generate a continuous index range.

[0083] To adapt to the sparsity changes in model hierarchies and levels, value-level processing adaptively adjusts the indexing method according to the density or sparsity flag, which can improve the efficiency of numerical retrieval and minimize the inherent numerical indexing overhead by aligning the address arrangement with the indexing mechanism determined by the sparsity threshold.

[0084] Step S130, in response to the sparsity of the layer neural network exceeding the sparsity threshold, the value of the non-zero element and its index position are stored.

[0085] It should be noted that for neural network layers whose sparsity exceeds the sparsity threshold, the indexes of zero elements are stored between the end address and the index address. In contrast, the indexes of non-zero elements are stored in the range of the difference between the end address and the index address.

[0086] In this embodiment, see Figure 4 , identifying the sparseness ratio of each layer of the neural network, including the following steps:

[0087] Step S111, obtaining the number of non-zero elements in each layer of the neural network.

[0088] By collecting the number of non-zero elements in each layer of the neural network, it is convenient to identify the sparse ratio of each layer of the neural network.

[0089] Step S112, inputting the number of non-zero elements into the neural network model and outputting the sparse ratio result, the neural network model is trained using the number of non-zero elements samples and sparse ratio samples in each layer of the neural network as training data.

[0090] By continuously training the neural network model and continuously optimizing it, it is convenient to improve the accuracy of the neural network model.

[0091] In this embodiment, see Figure 5 , identifying the position and number of non-zero bits according to the sign conversion and the bit lookup table, and placing the processed data into bit groups to eliminate the non-contributing bits, including the following steps:

[0092] Step S210, extracting the sign flag from the value-level tensor and converting the signed value of the current tensor into an unsigned integer.

[0093] The current tensor is converted to an unsigned integer and stored with the sign flag set to 1. During the conversion, the sign flag is retained for subsequent addition and accumulation operations.

[0094] Step S220 , determining the number of non-zero bits and their positions based on the high-bit lookup table and the low-bit lookup table to generate a bit grouping structure.

[0095] In order to effectively determine the position of non-zero bits, the bit matching stage uses a high-order lookup table and a low-order lookup table. The high-order lookup table contains the number of high-order non-zero bits and their corresponding bit positions, and the low-order lookup table contains the number of low-order non-zero bits and their corresponding bit positions. It avoids frequent memory reconfiguration or mode adjustment under different sparsity configurations.

[0096] Non-zero bit count: For each tensor, the total number of non-zero bits is determined as the number of high-order non-zero bits and the number of low-order non-zero bits from the corresponding high-order lookup table and low-order lookup table, respectively.

[0097] It should be noted that the bit grouping includes the number of non-zero bits and the positions of the non-zero bits.

[0098] Step S230: in the process of generating the bit grouping, all zero bit positions are removed, and the generated bit grouping data is stored in the pre-buffer area.

[0099] By removing all zero bit positions during the bit grouping process, it is convenient to ensure that only non-zero bits are retained. At the same time, the numerical accuracy of the neural network weights is effectively maintained, the bit-level flexibility is enhanced, and a solid foundation is laid for optimizing hardware resource utilization.

[0100] In this embodiment, see Figure 6 , processing bit grouping, and distributing the relevant inputs and tensors of the adaptive indexing mechanism extracted from the layer metadata to the bit elastic processing unit, and performing shift-addition calculations according to the number of non-zero bits, including the following steps:

[0101] Step S310 , dynamically adjusting the size of the bit grouping based on the sparsity threshold extracted from the layer metadata and the computing capability of the column-based computing array.

[0102] Sparse layer processing: Divide the non-zero bits into small groups according to the sparsity threshold to avoid wasting hardware resources on zero bits.

[0103] Dense layer processing: In the case of high density, groups are organized according to fixed bit width (such as 32 or 64 bits) to improve the parallel efficiency of hardware computing.

[0104] Step S320: extracting index information from the layer metadata based on the adaptive indexing mechanism, and pre-storing the index information in a dedicated buffer of the elastic processing unit.

[0105] The principle of the adaptive indexing mechanism is: when the sparsity is high, the sparse indexing mechanism is used to index only non-zero elements to reduce memory access overhead; when the sparsity is low, it switches to a dense indexing mechanism to improve data retrieval efficiency.

[0106] By pre-storing the index information in a dedicated buffer of the bit elastic processing unit, it is convenient to ensure that the bit elastic processing unit can quickly access required data.

[0107] Step S330 , extracting the bit grouping information and the corresponding input and weight tensors from the pre-cache area, and loading them into a dedicated buffer of the bit elastic processing unit.

[0108] During the loading process, sparse indexes are used to avoid inefficient data transfer.

[0109] Step S340, parse the non-zero bit positions in the bit grouping, and locate the specific position of each bit in the calculation.

[0110] By parsing the non-zero bit positions in the bit grouping, we can focus on the bits that really contribute and ignore the zero bits, which significantly reduces the computational overhead, improves the efficiency of sparse computing, and optimizes the utilization of hardware resources.

[0111] Step S350, performing sign addition on the intermediate result based on the sign flag.

[0112] Symbolic addition can help improve computational efficiency, reduce energy consumption, and ensure the correctness of intermediate results.

[0113] Step S360, performing an accumulation operation of the non-zero bits in the group to generate a partial product result.

[0114] By skipping non-contributing bits and focusing on accumulating valid non-zero bits, the computing efficiency is greatly improved. At the same time, compared with the traditional bit-by-bit calculation method, this method increases the effective calculation ratio in a single operation to more than 80%, especially in highly sparse networks.

[0115] In this embodiment, see Figure 7 , perform symbolic addition on the intermediate results based on the symbol flag, including the following steps:

[0116] Step S351, in response to the two input values ​​of the intermediate result having the same signs, the sum of the two is calculated.

[0117] By detecting in advance whether the signs of the two input values ​​are the same, the path for sum calculation can be quickly determined, avoiding additional logical judgments and calculations. At the same time, when the signs are the same, the addition operation is performed directly, reducing the error of dynamic branch prediction and improving pipeline performance.

[0118] Step S352, in response to the two input values ​​of the intermediate result having different signs, calculate the difference between the two and determine the sign of the result according to the larger absolute value.

[0119] By comparing the absolute values ​​of the two input values, the result sign is always guaranteed to be consistent with the value with the larger absolute value. This avoids the problem of incorrect sign assignment caused by calculation errors. Compared with direct signed addition and subtraction operations, this method maintains high accuracy of the result, especially when dealing with values ​​close to zero or with low dynamic range.

[0120] The following is a description of the neural network acceleration device based on the collaboration of algorithm and hardware provided by the present invention. The neural network acceleration device based on the collaboration of algorithm and hardware described below and the neural network acceleration method based on the collaboration of algorithm and hardware described above can be referenced to each other.

[0121] like Figure 8 As shown, in one embodiment, a neural network acceleration device based on the collaboration of algorithms and hardware includes an adjustment module 810, an identification module 820, an allocation module 830 and a storage module 840.

[0122] The adjustment module 810 is used to process the value-level index of the quantized tensor based on the sparsity threshold to generate layer metadata and dynamically adjust the sparsity mode. The layer metadata includes a start address, an end address, an index address, and a density or sparsity flag.

[0123] The identification module 820 is used to identify the position and number of non-zero bits according to the sign conversion and the bit lookup table, and put the processed data into bit groups to eliminate non-contributing bits.

[0124] The allocation module 830 is used to process bit grouping, and allocate the relevant inputs and tensors of the adaptive indexing mechanism extracted from the layer metadata to the bit elastic processing units, and perform shift addition calculations according to the number of non-zero bits.

[0125] The storage module 840 is used to instruct all arrays to complete multiplication operations within a preset period based on column-based calculation arrays, and to store partial products and data streams corresponding to respective bit widths.

[0126] In this embodiment, the adjustment module 810 is specifically used to:

[0127] Obtain the starting address, ending address, index address, and density or sparsity flag of each layer of the neural network, and identify the sparsity ratio of each layer of the neural network;

[0128] In response to when the sparsity of the layer neural network is less than a sparsity threshold, storing the tensor value in a dense format to generate a continuous index range;

[0129] In response to when the sparsity of the layer neural network exceeds a sparsity threshold, the value of the non-zero element and its index position are stored.

[0130] In this embodiment, the sparseness ratio of each layer of the neural network is identified, specifically for:

[0131] Get the number of non-zero elements in each layer of the neural network;

[0132] The number of non-zero elements is input into the neural network model, and the sparse ratio result is output. The neural network model is trained using the non-zero element number samples and sparse ratio samples in each layer of the neural network as training data.

[0133] In this embodiment, the identification module 820 is specifically used for:

[0134] Extract the sign flag from the value-level tensor and convert the signed value of the current tensor to an unsigned integer;

[0135] Based on the high-order lookup table and the low-order lookup table, determining the number of non-zero bits and their positions to generate a bit grouping structure;

[0136] In the process of generating the bit grouping, all zero bit positions are removed, and the generated bit grouping data is stored in the pre-buffer area.

[0137] In this embodiment, the allocation module 830 is specifically used for:

[0138] Dynamically adjust the size of the bit grouping based on the sparsity threshold extracted from the layer metadata and the computational power of the column-based computation array;

[0139] Based on the adaptive indexing mechanism, the index information is extracted from the layer metadata and pre-stored in a dedicated buffer of the elastic processing unit;

[0140] Extract the bit grouping information and the corresponding input and weight tensors from the pre-buffer area and load them into a dedicated buffer of the bit elastic processing unit;

[0141] Parse the non-zero bit positions in the bit grouping and locate the specific position of each bit in the calculation;

[0142] Perform sign addition on the intermediate results based on the sign flag;

[0143] An accumulation operation is performed on the non-zero bits within the group to generate a partial product result.

[0144] In this embodiment, the intermediate result is added with a sign based on the sign flag, specifically for:

[0145] In response to two input values ​​of the intermediate result having the same sign, calculating the sum of the two;

[0146] In response to two input values ​​of the intermediate result having different signs, a difference between the two is calculated and the sign of the result is determined based on the larger absolute value.

[0147] This neural network acceleration device based on the collaboration of algorithms and hardware, on the one hand, aims to minimize non-contributing bits by adopting static sparse and dense storage formats combined with dynamic bit sparsity utilization schemes, so as to facilitate the accelerator controller to efficiently skip these zero bits during the calculation process. At the same time, it retains the random distribution of non-zero bits to maintain tensor accuracy, provides a basis for solving the problems of computational dependence and low hardware utilization, and improves the performance and applicability of bit-level accelerators. On the other hand, through the bit-elastic processing unit with bit grouping to achieve balanced workload distribution across different bit widths, the column-based computing array further routes sparse or dense workloads to the bit-elastic processing unit, realizing scalable pre-cache scheduling management to cope with random non-zero bit operations and inputs, thereby improving computing efficiency.

[0148] like Fig. 9 and Fig.10As shown, in one embodiment, the present invention also provides a neural network accelerator based on algorithm and hardware collaboration, the accelerator includes a top-level controller 910, a pre-cache scheduling manager 920, a memory 930, a bit elastic processing unit 940 and a column processing array 950, the top-level controller 910 processes instructions and distributes corresponding control signals to the pre-cache scheduling manager 920, the memory 930 and the bit elastic processing unit 940, the pre-cache scheduling manager 920 is used to allocate inputs and tensors to the bit elastic processing unit 940, and retain layer metadata and bit grouping in the pre-cache module and the scheduler module until the processing array is fully configured, the input buffer and tensor buffer in the pre-cache scheduling manager 920 transmit elements in a column manner, and perform iterations of the model level, the column processing array 950 uses the bit elastic processing unit 940 to perform multiplication operations, and uses a signed adder to process intermediate results, the memory 930 is used to calculate arrays based on columns, instruct all arrays to complete multiplication operations within a preset cycle, and store partial products and data streams corresponding to their respective bit widths.

[0149] Among them, the pre-caching module utilizes the layer metadata from the value-level format and the corresponding numerical indexing mechanism, while the scheduler module processes the bit groups generated by the bit-level pre-processing, which include the number of non-zero bits and their non-zero bit positions.

[0150] See also Fig.11 The scheduler module and the pre-cache module are based on the dimensions of the processing unit array and the number of operations per processing unit (Q p比特位宽 ) to make preliminary allocations to prepare for subsequent computing tasks. After applying the dynamic bit processing scheme, the pre-caching module uses the total number of non-zero bits (N nzb_num ) optimizes the distribution of weights among processing units. Subsequently, the pre-caching module evaluates the availability of processing units and dynamically allocates arrays to the corresponding tensors. The scheduler module extracts the adaptive indexing mechanism (Ψ idx ), and loads the relevant inputs into the input and weight dedicated buffers to ensure efficient utilization of the processing units during the calculation process. In addition, the scheduler module also rearranges the intermediate results in the processing unit array to further alleviate the load imbalance problem. Although the initial pre-allocation phase introduces latency due to the complexity of resource configuration, subsequent operations are performed synchronously with the calculation, and this latency is effectively hidden by overlapping the calculation with the data movement, thereby supporting the computing needs of multiple cores or multiple layers, allowing the processing units to perform operations with consistent latency.

[0151] The column processing array 950 allocates each column to perform an entire calculation chain, ensuring that all arrays complete multiplication operations in a given cycle.

[0152] 1. Array value reading

[0153] Fig.12 The numerical representation of W0(-3), W1(-8), W2(-14) and W3(-5) in the algorithmic level - bit sparsity and threshold adjustment algorithm format is shown. The column processing array 950 extracts the non-zero bit positions of the weights (W bitpos ) and symbol mark (S flag ), and stores it in a dedicated buffer (read module A). At the same time, the bit sparsity and threshold adjustment algorithm is calculated by the corresponding numerical index Ψ idx Loaded into the read B module. Due to the column arrangement, the non-zero bit position (W bitpos ) buffers provide independent access to the processing unit array for multiplication operations.

[0154] 2. Array numerical multiplication

[0155] The A and B read modules use two buffers to transfer the bit stream (from the array value reading step) to the processing unit array through the corresponding communication interface, effectively reducing the hardware overhead.

[0156] The intermediate result calculation formula is:

[0157] ,

[0158] in, represents the weight, Represents an input element.

[0159] After the multiplication operation is completed, the bit-elastic processing unit (BEP) is deactivated, freeing up hardware resources for subsequent computational operations.

[0160] 3. Accumulation of array numerical calculation results

[0161] Each processing unit array contains multiple intermediate results, which are then shifted and accumulated with adjacent processing units. idx , determine the number of non-zero elements (NNZ num ) to perform the subsequent addition operation. The column identifier (column, c) is used to extract the processing unit array data, and then the fully pipelined accumulator performs the operation. Each processing unit is equipped with an S-based flag The adders are used to ensure computational accuracy after the signed to unsigned conversion of the tensor. Therefore, the addition operation is performed in a column-wise manner and the adders are pipelined, aligned to the number of processing units specified in the kernel or layer configuration.

[0162] 4. Write array numerical calculation results

[0163] The processing unit array consists of rows and columns, where each column processes the output in the same cycle. To optimize output access, each neural network layer is allocated in a columnar manner, which facilitates data transfer of the computation array and improves parallel processing capabilities. Each column is equipped with a data stream with the corresponding addition bit width, and the elements are transferred to the memory in a streaming manner.

[0164] In this embodiment, the bit elastic processing unit 940 includes a shift unit 941, an accumulation unit 942, a multiplexer 943 and a signed adder 944, the shift unit 941 is used to extract the non-zero bit configuration, the accumulation unit 942 is used to aggregate position information, the multiplexer 943 is used to dynamically select the input path according to the control signal, and the signed adder 944 is used to perform the final accumulation operation.

[0165] The bit elastic processing unit (BEP) retrieves multiple weights (W Idx ) and its corresponding input element (I Idx ). Each processing unit integrates multiple adaptive sub-multipliers working in parallel to optimize computing efficiency. Fig.11 The right figure in the figure shows the configuration as an operand (Q p比特位宽 ) bit elastic processing unit, which performs shift addition calculation according to the number of non-zero bits. This unit effectively manages different tensor characteristics and ensures calculation efficiency through the coordinated operation of the shift unit 941, the accumulation unit 942, the multiplexer 943 and the signed adder 944, where the sign flag (S flag ) plays a key role. The bit elastic processing unit uses the symbol mark (S flag ) to identify whether the weight is positive (P) or negative (N), and performs signed addition on the intermediate results for each weight.

[0166] At the algorithm level, a novel bit-level sparsity pattern is proposed, a static sparse and dense storage format is adopted, and a dynamic bit sparsity utilization scheme is combined to eliminate non-contributing bits while maintaining tensor accuracy. At the hardware level, a bit-level accelerator is developed, including a bit-elastic processing unit, a pre-cache scheduling manager, and a column-based computing array, which is specially optimized for scalable precision computing. Experimental results show that compared with the most advanced bit-level accelerator, the present invention improves speed, area efficiency, and energy efficiency by 2.92 times, 1.74 times, and 3.59 times, respectively. At the same time, the present invention reduces the number of zero bits in the model by 84% on average, and improves hardware utilization by 4.68 to 8.51 times in OPT, MobileNet-V2, ResNet50, VGG19, ViT_B, and AlexNet models.

[0167] On the other hand, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements a neural network acceleration method based on algorithm and hardware collaboration, the method comprising:

[0168] Based on the sparsity threshold, the value-level index of the quantized tensor is processed to generate layer metadata and dynamically adjust the sparsity mode, the layer metadata including the start address, the end address, the index address, and the density or sparsity flag;

[0169] identifying the location and number of non-zero bits based on the sign conversion and the bit lookup table, and placing the processed data into bit groups to eliminate non-contributing bits;

[0170] Processing bit grouping and distributing relevant inputs and tensors of the adaptive indexing mechanism extracted from the layer metadata to the bit elastic processing units, and performing shift-add operations according to the number of non-zero bits;

[0171] Based on the column-based computing array, all arrays are instructed to complete multiplication operations within a preset cycle, and store partial products and data streams corresponding to their respective bit widths.

[0172] In another aspect, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, a neural network acceleration method based on algorithm and hardware collaboration is implemented, the method comprising:

[0173] Based on the sparsity threshold, the value-level index of the quantized tensor is processed to generate layer metadata and dynamically adjust the sparsity mode, the layer metadata including the start address, the end address, the index address, and the density or sparsity flag;

[0174] identifying the location and number of non-zero bits based on the sign conversion and the bit lookup table, and placing the processed data into bit groups to eliminate non-contributing bits;

[0175] Processing bit grouping and distributing relevant inputs and tensors of the adaptive indexing mechanism extracted from the layer metadata to the bit elastic processing units, and performing shift-add operations according to the number of non-zero bits;

[0176] Based on the column-based computing array, all arrays are instructed to complete multiplication operations within a preset cycle, and store partial products and data streams corresponding to their respective bit widths.

[0177] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0178] By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0179] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0180] The above-described embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A neural network acceleration method based on algorithm and hardware collaboration, characterized in that: The method comprises: Based on a sparsity threshold, processing a value-level index of a quantized tensor to generate layer metadata and dynamically adjust a sparsity mode, wherein the layer metadata includes a start address, an end address, an index address, and a dense or sparse flag; identifying the location and number of non-zero bits based on the sign conversion and the bit lookup table, and placing the processed data into bit groups to eliminate non-contributing bits; Processing bit grouping, and assigning relevant inputs and tensors of the adaptive indexing mechanism extracted from the layer metadata to the bit elastic processing unit, and performing shift-addition calculation according to the number of non-zero bits, wherein the bit elastic processing unit includes a shift unit, an accumulation unit, a multiplexer and a signed adder, the shift unit is used to extract the non-zero bit configuration, the accumulation unit is used to aggregate the position information, the multiplexer is used to dynamically select the input path according to the control signal, and the signed adder is used to perform the final accumulation operation; Based on the column-based computing array, instruct all arrays to complete the multiplication operation within a preset cycle, and store partial products and data streams corresponding to their respective bit widths; The processing bit grouping and distributing the relevant input and tensor of the adaptive indexing mechanism extracted from the layer metadata to the bit elastic processing unit, and performing shift addition calculation according to the number of non-zero bits, including: Dynamically adjust the size of the bit grouping based on the sparsity threshold extracted from the layer metadata and the computational power of the column-based computation array; Based on the adaptive indexing mechanism, the index information is extracted from the layer metadata and pre-stored in a dedicated buffer of the elastic processing unit; Extract the bit grouping information and the corresponding input and weight tensors from the pre-buffer area and load them into a dedicated buffer of the bit elastic processing unit; Parse the non-zero bit positions in the bit grouping and locate the specific position of each bit in the calculation; Perform sign addition on the intermediate results based on the sign flag; An accumulation operation is performed on the non-zero bits within the group to generate a partial product result.

2. The neural network acceleration method based on algorithm and hardware collaboration according to claim 1, characterized in that: The method of processing the value-level indexes of the quantized tensor based on the sparsity threshold to generate layer metadata and dynamically adjust the sparsity mode includes: Obtain the starting address, ending address, index address, and density or sparsity flag of each layer of the neural network, and identify the sparsity ratio of each layer of the neural network; In response to when the sparsity of the layer neural network is less than a sparsity threshold, storing the tensor value in a dense format to generate a continuous index range; In response to when the sparsity of the layer neural network exceeds a sparsity threshold, the value of the non-zero element and its index position are stored.

3. The neural network acceleration method based on algorithm and hardware collaboration according to claim 2 is characterized in that: The identifying the sparse ratio of each layer of the neural network includes: Get the number of non-zero elements in each layer of the neural network; The number of non-zero elements is input into a neural network model, and a sparse ratio result is output. The neural network model is trained using samples of the number of non-zero elements and sparse ratio samples in each layer of the neural network as training data.

4. The neural network acceleration method based on algorithm and hardware collaboration according to claim 3 is characterized in that: The method of identifying the position and number of non-zero bits according to the sign conversion and the bit lookup table, and placing the processed data into bit groups to eliminate non-contributory bits, includes: Extract the sign flag from the value-level tensor and convert the signed value of the current tensor to an unsigned integer; Based on the high-order lookup table and the low-order lookup table, determining the number of non-zero bits and their positions to generate a bit grouping structure; In the process of generating the bit grouping, all zero bit positions are removed, and the generated bit grouping data is stored in the pre-buffer area.

5. The neural network acceleration method based on algorithm and hardware collaboration according to claim 1, characterized in that: The performing sign addition on the intermediate result based on the sign flag includes: In response to two input values ​​of the intermediate result having the same sign, calculating the sum of the two; In response to two input values ​​of the intermediate result having different signs, a difference between the two is calculated and the sign of the result is determined based on the larger absolute value.

6. A neural network acceleration device based on algorithm and hardware collaboration, characterized in that: include: An adjustment module, configured to process the value-level index of the quantized tensor based on a sparsity threshold to generate layer metadata and dynamically adjust the sparsity mode, wherein the layer metadata includes a start address, an end address, an index address, and a density or sparsity flag; an identification module for identifying the position and number of non-zero bits based on the sign conversion and the bit lookup table, and placing the processed data into bit groups to eliminate non-contributing bits; an allocation module, configured to process bit grouping, and allocate relevant inputs and tensors of the adaptive indexing mechanism extracted from the layer metadata to the bit elastic processing unit, and perform shift-addition calculation according to the number of non-zero bits; wherein the bit elastic processing unit includes a shift unit, an accumulation unit, a multiplexer and a signed adder, the shift unit is configured to extract the non-zero bit configuration, the accumulation unit is configured to aggregate the position information, the multiplexer is configured to dynamically select the input path according to the control signal, and the signed adder is configured to perform the final accumulation operation; A storage module, for instructing all arrays to complete multiplication operations within a preset cycle based on column-based computing arrays, and storing partial products and data streams corresponding to respective bit widths; The processing bit grouping and distributing the relevant input and tensor of the adaptive indexing mechanism extracted from the layer metadata to the bit elastic processing unit, and performing shift addition calculation according to the number of non-zero bits, including: Dynamically adjust the size of the bit grouping based on the sparsity threshold extracted from the layer metadata and the computational power of the column-based computation array; Based on the adaptive indexing mechanism, the index information is extracted from the layer metadata and pre-stored in a dedicated buffer of the elastic processing unit; Extract the bit grouping information and the corresponding input and weight tensors from the pre-buffer area and load them into a dedicated buffer of the bit elastic processing unit; Parse the non-zero bit positions in the bit grouping and locate the specific position of each bit in the calculation; Perform sign addition on the intermediate results based on the sign flag; An accumulation operation is performed on the non-zero bits within the group to generate a partial product result.

7. A neural network accelerator based on algorithm and hardware collaboration, characterized in that: It includes a top-level controller, a pre-cache scheduling manager, a memory, a bit-elastic processing unit and a column-based processing array. The top-level controller processes instructions and distributes corresponding control signals to the pre-cache scheduling manager, the memory and the bit-elastic processing unit. The pre-cache scheduling manager is used to allocate inputs and tensors to the bit-elastic processing unit, and retain layer metadata and bit grouping in the pre-cache module and the scheduler module until the processing array is fully configured. The input buffer and tensor buffer in the pre-cache scheduling manager transmit elements in a column manner and perform iterations of the model level. The column-based processing array uses the bit-elastic processing unit to perform multiplication operations and uses a signed adder to process intermediate results. The memory is used to calculate arrays based on the column, instruct all arrays to complete multiplication operations within a preset cycle, and store partial products and data streams corresponding to their respective bit widths.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the neural network acceleration method based on the collaboration of algorithm and hardware as described in any one of claims 1 to 5 are implemented.

Images