Accelerator generation tool for underlying optic neural network algorithm

Abstract

The invention belongs to the technical field of integrated circuits, and particularly relates to an accelerator generation tool for an underlying optic neural network algorithm. According to the invention, the generation of the underlying visual accelerator is realized through software and hardware collaborative design; comprising three modules: an activation-weight perception hybrid precision quantification module, a hardware architecture parameter optimization module and a bottom layer visual hardware template; according to the tool, firstly, activation-weight perception hybrid precision quantization is carried out on a given bottom-layer vision algorithm to determine the bit width of activation and weight of each layer, and quantized weight is generated; then a hardware architecture parameter optimization module searches for appropriate hardware template parameters for the algorithm; and finally, endowing the searched bit width and architecture parameters to a hardware template to generate the high-energy-efficiency high-throughput accelerator special for the underlying vision algorithm. According to the method, high-energy-efficiency and high-throughput hardware can be quickly generated according to different underlying vision algorithms, and the development period is shortened; the method can be widely applied to IoT, edge side and other intelligent devices.

Description

Technical Field

[0001] The present invention belongs to the technical field of integrated circuits, and in particular relates to an accelerator generation tool for underlying visual neural network algorithms. Background Art

[0002] Low-level vision algorithms continue to expand their application scenarios in fields such as biomedical imaging, industrial inspection, security monitoring, and autonomous driving. Their core tasks include pre-processing operations such as image denoising, super-resolution, and style transfer, which directly affect the accuracy and reliability of subsequent high-level vision tasks. These algorithms are often deployed in resource-constrained embedded devices, placing stringent demands on the energy efficiency of hardware accelerators. At the same time, the rapid iteration of algorithm architectures and hyperparameters, coupled with the diverse demands of scenarios, pose severe challenges to traditional hardware design models. On the one hand, low-level vision algorithm models are constantly upgraded with technological advances, and new network architecture parameters are constantly being proposed for new scenarios, making it difficult for previous accelerators to adapt efficiently. On the other hand, the hardware development process is seriously out of sync with the speed of algorithm evolution. Traditional customized accelerator design requires a significant investment of time to complete algorithm analysis, architectural exploration, and optimization, making it difficult for the hardware design cycle to match the frequency of algorithm updates.

[0003] In existing solutions, the design of dedicated integrated circuits must go through multiple stages of algorithm-specific design, making it difficult to flexibly adapt to algorithm changes. This fragmented R&D model leads to inefficient collaborative optimization of algorithms and hardware, and a large amount of engineering resources are consumed in repetitive debugging. Therefore, there is an urgent need to build an intelligent automatic design framework that can achieve rapid transformation from algorithm models to high-performance accelerators by mapping the relationship between abstract algorithm features and hardware architecture. Such a framework should have core functions such as algorithm parsing, architecture template matching, and resource optimization. It should be able to adapt to the energy efficiency constraints of different scenarios and be compatible with algorithm iteration through modular design. By lowering the development threshold through automation technology and shortening the path from algorithm innovation to hardware implementation, it can provide infrastructure support for the application of underlying visual technologies. Summary of the Invention

[0004] The purpose of the present invention is to propose a high-energy-efficiency, high-throughput accelerator generation tool for underlying visual neural network (CNN) algorithms, which can quickly generate RTL designs based on a given underlying visual CNN algorithm.

[0005] The accelerator generation tool for underlying visual neural network algorithms provided by the present invention achieves high-efficiency, high-throughput underlying visual accelerator generation through software-hardware collaborative design. It includes three modules: an activation-weight-aware mixed-precision quantization module, an underlying visual hardware template, and a hardware architecture parameter optimization module. During the accelerator's RTL (Register Transfer Level) generation process, for a given underlying visual network, the activation-weight-aware mixed-precision quantization module first quantizes the activations and weights of each layer of the network. Taking advantage of the fact that the number of activations in the underlying visual network is much larger than the weights, it searches for the bit width of each layer and quantizes the weights to integers within the corresponding bit width, significantly reducing its computation and storage requirements. The underlying visual hardware template adopts laminar flow, multiple parallelisms, configurable data bit widths, and a multi-level storage architecture with activation reuse distance perception based on the characteristics of the underlying visual network to achieve efficient adaptation to different underlying visual algorithms. The hardware architecture parameter optimization module uses a gradient descent algorithm to optimize the parameters in the hardware template based on the hyperparameters of the underlying visual algorithm network, ultimately generating a high-efficiency, high-throughput underlying visual accelerator.

[0006] Further:

[0007] The activation-weight-aware mixed precision quantization module adopts the DARTS (Differentiable Architecture Search) method to search for the optimal bit width for the activations and weights of each layer while meeting the accuracy requirements, taking into account the number of activations and weights. Since the number of activations of the underlying visual network is much larger than the weights, the DARTS technology reduces the overall storage and bandwidth requirements as much as possible by compressing the bit width of the activations and increasing the bit width of the weights, thereby achieving friendly deployment of the underlying visual algorithm. Specifically, the specific steps of using the DARTS method for mixed precision quantization are as follows: First, the given underlying visual network is reconstructed into a new super network. The super network is an extension of the original network, that is, each layer of the original network is transformed into a multi-branch parallel structure, and each branch corresponds to a different activation-weight bit width combination. The input V of each layer in and weights W with different precision combinations (remember the precision combination as E ij , where i represents the input bit width and j represents the weight bit width). Subsequently, the calculation results under different bit widths are calculated through the trainable coefficient P ij Scaling is performed and the output of this layer is obtained by summing (denoted as V out ). Then the super network is trained to obtain P ij The large branch is the searched quantitative configuration.

[0008] The loss function Loss defined during the hypernetwork training process is:

[0009] Loss=Loss val +L prob-1 +λ∑ layer E(SN), (1)

[0010] Among them, Loss val is the network accuracy loss function, L prob-1 is the regularized loss function, and E(SN) is the proposed activation-weight-aware loss function:

[0011]

[0012] Among them, N W and N A Representing the number of weights and activations in the layer, respectively. This loss function considers the difference in the number of weights and activations during the quantization bit width search: for network architectures with more activations than weights, the function prioritizes lower data bit widths for activations and higher data bit widths for weights while maintaining accuracy. Conversely, if the number of weights is predominant, a lower bit width is assigned to weights while a higher bit width is assigned to activations. This design optimizes model performance and computational efficiency by dynamically balancing the quantization accuracy of both.

[0013] The detailed structure of the underlying visual hardware template is described below. It consists of multiple computing units connected in series, each responsible for computing a layer within the underlying visual network. The entire computational flow is performed in a layer-by-layer pipeline. After each layer calculates a portion of the data, the completed portion is fed into the computing unit of the next layer for further calculation, thus achieving computational pipeline and parallel computation between layers.

[0014] Each computing unit contains a multi-parallel computing array and a two-level storage for data reuse perception. The computing unit array has an activation space direction T H , input channel T N , output channel T M Three directions of parallelism; the computing unit has an input bit width B I , weight bit width B w , output bit width B OThree configurable bit widths; a data reuse-aware two-level storage consisting of a RAM-based row cache and a register-based shift cache for storing activation data. When the underlying visual network performs convolution operations in column-major order, the reuse distance of data at the right boundary is short, while the reuse distance of data at the bottom boundary is long. Based on this characteristic, the shift cache stores data with a short reuse distance and releases it after being reused K times (K is the convolution kernel size). After each computation, it shifts left to discard the boundary data and loads new data from the row cache. The row cache not only stores data with short reuse distances but also retains data with long reuse distances for subsequent computations. This hierarchical storage design based on data reuse characteristics reduces overall memory access times and energy consumption through differentiated data retention strategies. Furthermore, this template introduces Line Buffer Parallelism (LBP) for the row cache. By adjusting the LBP, the read and write power consumption of the row cache changes, optimizing it for the most energy-efficient data read and write.

[0015] The configurable architecture parameters introduced by the underlying visual hardware template are mainly shown in Table 1.

[0016] Table 1 Configurable architecture parameters for the underlying visual hardware template

[0017]

[0018] Among them, the bit width parameter is determined by the activation-weight-aware mixed precision quantization module, and the remaining parameters are determined by the hardware architecture parameter optimization module.

[0019] The hardware architecture parameter optimization module, the above template introduces a huge search space. In order to effectively generate high-efficiency hardware for a specific underlying visual algorithm, it is necessary to determine appropriate parameters for the hardware template to balance the computational load between layers and optimize the fine-grained power consumption of inference. However, these parameters are in a discrete space, and it often takes a long time to optimize using a discrete solution. This module transforms these parameters from a discrete space into a differentiable continuous space, and uses gradient descent for rapid optimization. We use the NNLUT (Neural Network Look up Table) method to fit the power consumption and resource occupancy of each component in the template, so that it is transferred to a continuous space, and then gradient descent is used for optimization, thereby determining the computational parallelism and memory access parallelism of each layer in the large search space, and completing the generation of a high-efficiency accelerator for a specific underlying visual algorithm.

[0020] The process of the NNLUT method is as follows:

[0021] (1) Traverse the template component variables and comprehensively evaluate resources and power consumption: Traverse the variable parameters related to the above template components, perform synthesis on the target backend, and obtain resource consumption and power consumption data.

[0022] (2) Fully connected network fitting resource and power consumption model: A simple fully connected neural network is used to fit resource consumption and power consumption. To ensure accurate fitting of each data point (to avoid the impact of absolute value differences on training), MAPE (mean absolute percentage error) is used as the loss function during training.

[0023] (3) Freezing the fitted network weights for architecture parameter search: After fitting is complete, the network weights are fixed. In the subsequent gradient descent-based architecture parameter optimization process, these weights are only used for forward propagation and backpropagation gradient calculations and do not participate in parameter updates. This mechanism ensures the stability of the resource and power consumption model while supporting end-to-end optimization of architecture parameters.

[0024] The objective function constructed by searching the hardware architecture parameters using gradient descent is as follows:

[0025] Objective=TCycle+α*Energy+β*Balance+λ*(χ all +ξ), (3)

[0026] Among them, TCycle is the total inference cycle, which is calculated by hardware parameters; Energy is the total inference delay, which is calculated by NNLUT and hardware parameters; Balance is the balance term, which is the balance term of the computing time of different pipeline stages; all is a value constraint item, which ensures that the values ​​of the architecture parameters are within a reasonable range; ξ is a resource constraint item, which ensures that the generated RTL resources are within a reasonable range, and it is also estimated and calculated using NNLUT; α, β, λ are weight coefficients used to adjust the importance of different goals.

[0027] Compared with the prior art, the beneficial technical effects of the present invention are mainly reflected in:

[0028] (1) The activation-weight-aware mixed precision quantization module fully considers the huge difference in the number of activations and weights of the underlying visual network when performing quantization, and gives activations a smaller bit width and weights a larger bit width while meeting the accuracy requirements. This approach can reduce data handling during model inference, speed up the calculation process, reduce functionality, and reduce the consumption of generated hardware resources.

[0029] (2) The underlying visual hardware template has a layer pipeline architecture, parallelism between layers, activation space directions, input channels, and output channels, flexible and configurable computing unit data bit width, and a two-level storage architecture that reuses distance perception; this architecture can adapt to various CNN-based underlying visual algorithms and has high energy efficiency.

[0030] (3) The hardware architecture parameter optimization module is based on the gradient descent algorithm and NNLUT, which converts the discrete optimization problem into a continuous space optimization problem; this method can quickly find suitable hardware architecture parameters in a complex search space.

[0031] (4) The tool can generate energy-efficient, high-throughput hardware accelerators for the underlying visual network. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Generate a flow chart for the low-level vision-oriented accelerator.

[0033] Figure 2 Diagram of the activation-weight-aware mixed-precision quantization module workflow.

[0034] Figure 3 Illustration of the underlying visual hardware template.

[0035] Figure 4 Illustration of reuse distance-aware two-level storage behavior. DETAILED DESCRIPTION

[0036] The overall block diagram of the accelerator generation tool for underlying visual algorithms proposed in this invention is as follows: Figure 1 It includes an activation-weight-aware mixed-precision quantization module, an underlying visual hardware template, and a hardware architecture parameter optimization module. The specific workflow is as follows:

[0037] First, the given network is first passed through the activation-weight-aware mixed precision quantization module to quantize the weights and determine the activation-bitwidth parameters of each layer. It uses the DARTS (Differentiable Architecture Search) method to search the bitwidth of each layer. For a given underlying visual network, first, the given underlying visual network is reconstructed into a new super network, and different data precisions are sampled in the process. The input V of each layer is in and weights W will be combined with different precisions (denoted as E ij , where i represents the input bit width and j represents the weight bit width) is operated, such as Figure 2 Then, the calculation results under different bit widths are calculated through the trainable coefficient P ij Scaling is performed and the output of this layer is obtained by summing (denoted as V out). Then the super network is trained to obtain P ij The large branch is the searched quantitative configuration.

[0038] The loss function Loss defined during the hypernetwork training process is:

[0039]

[0040] Among them, Loss val is the network accuracy loss function, L prob-1 is the regularized loss function, and E(SN) is the activation-weight-aware loss function:

[0041]

[0042] Among them, N W and N A Representing the number of weights and activations in the layer, respectively. This loss function considers the difference in the number of weights and activations during the quantization bit width search process: for network architectures with more activations than weights, the function prioritizes using a lower data bit width for activations and a higher data bit width for weights while maintaining accuracy. Conversely, if the number of weights is greater, a lower bit width is assigned to the weights and a higher bit width is assigned to the activations.

[0043] After determining the bit width of the weights and activations of each layer, the network model is fine-tuned to restore the network accuracy, thereby obtaining the quantized network and the bit width configuration of each layer.

[0044] Then, we need to search the architecture parameters in the hardware template. Figure 3 As shown in the figure, it is composed of multiple layers of computing units connected in series, and each layer of computing unit is responsible for the calculation of one layer in the network. The entire computing flow is carried out in a layer-by-layer flow. When each layer calculates part of the data, it sends the completed part to the computing unit of the next layer for subsequent calculation, thereby realizing the computation flow and parallel computation between layers. Each computing unit contains a multi-parallel computing array and a two-level storage for data reuse perception. Among them, the computing unit array has an activation space direction T H , input channel T N , output channel T M Three directions of parallelism; the computing unit has an input bit width B I , weight bit width B w , output bit width B OThree configurable bit widths; the two-level storage for data reuse awareness consists of a RAM-based row cache and a register-based shift cache for storing activation data. When the underlying visual network performs convolution operations in column-first order, the reuse distance of the right boundary data is short, while the reuse distance of the bottom boundary data is long. Based on this feature: the shift cache stores data with short reuse distance; the row cache retains data with long reuse distance for subsequent calculations. Cache interaction behavior is as follows Figure 4 As shown in the figure, when the results of the previous layer are transferred to the current layer, the data is first stored in the shift buffer. The data in the shift buffer consists of three parts: the data remaining in the shift buffer after the previous calculation (the two left columns); data read from the row buffer (the black portion in the right column), which is used to overwrite the historical data required for the convolution kernel shift; and the new data transferred from the previous layer (the gray portion in the right column). When the current calculation is completed, the shift buffer is shifted left to make room for the next calculation. The row buffer discards the first data and caches the previous layer's results at the end of the row buffer. Once the result is calculated from the computation array, it is fed into the accumulator to be added to the previous result, and the intermediate result is stored in the output buffer. After the accumulation is complete, the result is fed into the post-processing module for biasing and quantization scaling. The output result is quantized to the input bit width of the next layer, and the quantized result is stored back in the output buffer. Finally, the result is output from the output buffer to the computation unit of the next layer.

[0045] The architecture parameter optimization module uses the gradient descent method and NNLUT to optimize the architecture parameters. NNLUT is used to fit power consumption and resource consumption. The construction method of NNLUT is as follows:

[0046] (1) Traverse the template component variables and comprehensively evaluate resources and power consumption: Traverse the variable parameters related to the above template components, perform synthesis on the target backend, and obtain resource consumption and power consumption data.

[0047] (2) Fully connected network fitting resource and power consumption model: A simple fully connected neural network is used to fit resource consumption and power consumption. To ensure accurate fitting of each data point (to avoid the impact of absolute value differences on training), MAPE (mean absolute percentage error) is used as the loss function during training.

[0048] (3) Freezing the fitted network weights for architecture parameter search: After fitting is complete, the network weights are fixed. In the subsequent gradient descent-based architecture parameter optimization process, these weights are only used for forward propagation and backpropagation gradient calculations and do not participate in parameter updates. This mechanism ensures the stability of the resource and power consumption model while supporting end-to-end optimization of architecture parameters.

[0049] When the gradient descent method is used for optimization, the objective function constructed is as follows:

[0050] Objective=TCycle+α*Energy+β*Balance+λ*(χ all +ξ)

[0051] Among them, TCycle is the total inference cycle, which is calculated by hardware parameters; Energy is the total inference delay, which is calculated by NNLUT and hardware parameters; Balance is the balance term, which is the balance term of the computing time of different pipeline stages; all is a value constraint item, which ensures that the values ​​of the architecture parameters are within a reasonable range; ξ is a resource constraint item, which ensures that the generated RTL resources are within a reasonable range, and is also calculated using NNLUT.

[0052] After setting the objective function, gradient descent is used for search. In the configuration using gradient descent, since the actual architecture parameters are in discrete space, the objective function will inevitably introduce rounding operations, which are non-differentiable. For this operation, this module introduces the following assumptions in backpropagation:

[0053]

[0054] When the search process converges, the architectural parameters of each layer are obtained. These architectural parameters are configured into the hardware template to generate an accelerator that is energy-efficient and high-throughput for the underlying visual layer.

Claims

1. An accelerator generation tool for underlying visual neural network algorithms, characterized by: The generation of the underlying visual accelerator is achieved through software-hardware collaborative design; it includes three modules: an activation-weight-aware mixed precision quantization module, an underlying visual hardware template, and a hardware architecture parameter optimization module; during the RTL generation process of the accelerator, for a given underlying visual network, the activation-weight-aware mixed precision quantization module first quantizes the activations and weights of each layer of the network. It utilizes the characteristic that the number of activations of the underlying visual network is much larger than the weights. While meeting the accuracy conditions, it searches for the bit width of each layer and quantizes the weights to integers under the corresponding bit width, thereby greatly reducing its calculation and storage; the underlying visual hardware template adopts laminar flow, multiple parallelism, configurable data bit width, and a multi-level storage architecture with activation reuse distance perception according to the characteristics of the underlying visual network to achieve efficient adaptation to different underlying visual algorithms; the hardware architecture parameter optimization module adopts a gradient descent algorithm to optimize the parameters in the hardware template according to the hyperparameters of the underlying visual algorithm network, and finally generates a high-energy-efficiency, high-throughput underlying visual accelerator.

2. The accelerator generation tool according to claim 1, characterized in that The activation-weight-aware mixed precision quantization module adopts the differentiable search (DARTS) method to perform mixed precision quantization. The specific steps are as follows: first, the given underlying visual network is reconstructed to create a super network. The super network is an extension of the original network, that is, each layer of the original network is transformed into a multi-branch parallel structure, and each branch corresponds to a different activation-weight bit width combination; in this process, different data precisions are sampled; the input V of each layer is in The weight W is calculated with different precision combinations, and the precision combination is recorded as E ij , where i represents the input bit width and j represents the weight bit width; Subsequently, the calculation results at different bit widths are calculated by the trainable coefficient P ij Scaling is performed and the output of this layer is obtained by summing, which is recorded as V out ; Then train the super network and get P ij The large branch is the searched quantitative configuration; The loss function Loss defined during the hypernetwork training process is: Loss=Loss val +L prob-1 +λ∑ layer E(SN), (1) Among them, Loss val is the network accuracy loss function, L prob-1 is the regularized loss function, and E(SN) is the proposed activation-weight-aware loss function: Among them, N W and N A They represent the number of weights and activation values ​​in the layer respectively; the loss function takes into account the impact of the difference in the number of weights and activation values ​​during the quantization bit width search process: for network architectures where the number of activation values ​​is greater than the number of weights, the E(SN) function prioritizes using lower data bit widths for activation values ​​and higher data bit widths for weights while maintaining accuracy; conversely, if the number of weights dominates, a lower bit width is assigned to the weights and a higher bit width is assigned to the activation values; this design optimizes model performance and computational efficiency by dynamically balancing the quantization accuracy of the two.

3. The accelerator generation tool according to claim 2, characterized in that The bottom-level visual hardware template is composed of multiple computing units connected in series, and one computing unit is responsible for the calculation of one layer in the bottom-level visual network; the entire computing flow is carried out in a layer-by-layer flow, and when each layer calculates part of the data, the calculated part is sent to the computing unit of the next layer for subsequent calculation, thereby realizing the computation flow and the parallel computation between layers; each computing unit contains a multi-parallel computing array and a two-level storage for data reuse perception; wherein the computing unit array has an activation space direction T H , input channel T N , output channel T M Three directions of parallelism; the computing unit has an input bit width B I , weight bit width B w , output bit width B O Three configurable bit widths; the data reuse-aware two-level storage consists of a RAM-based row cache and a register-based shift cache for storing activation data; when the underlying visual network performs convolution operations in column-priority order, the reuse distance of the right boundary data is shorter, while the reuse distance of the bottom boundary data is longer. Based on this feature: the shift cache stores data with a short reuse distance and releases it after being reused K times, where K is the convolution kernel size. After each calculation is completed, the boundary data is shifted to the left and new data is loaded from the row cache; the row cache not only stores short reuse distance data, but also retains long reuse distance data for subsequent calculations; this hierarchical storage design based on data reuse characteristics ultimately reduces the overall memory access times and energy consumption through differentiated data retention strategies; in addition, for the row cache, the template introduces row cache read and write parallelism LBP; by adjusting the LBP, the read and write power consumption of the row cache changes, and it is optimized to achieve the most energy-efficient data reading and writing.

4. The accelerator generation tool according to claim 3, characterized in that The configurable architecture parameters in the underlying visual hardware template include: activation of spatial direction parallelism T H , the input channel parallelism of the l-th layer computing unit Output channel parallelism of the computing unit in layer l The input activation bit width of the l-th layer computing unit The weight bit width of the l-th layer computing unit The output activation bit width of the computational unit in layer l The read and write parallelism of the l-th layer row cache LBP l Among them, the bit width parameter is determined by the activation-weight-aware mixed precision quantization module, and the remaining parameters are determined by the hardware architecture parameter optimization module.

5. The accelerator generation tool according to claim 4, characterized in that The hardware architecture parameter optimization module determines appropriate parameters for the hardware template to balance computational loads across layers and optimize inference power consumption in a fine-grained manner. Since these parameters are in a discrete space, they are first transformed into a differentiable continuous space and then rapidly optimized using gradient descent. Specifically: The NNLUT method is used to fit the power consumption and resource usage of each component in the template, so that it is transferred into a continuous space. Then, gradient descent is used for optimization, so that the computational parallelism and memory access parallelism of each layer are determined in the large search space, completing the generation of a high-efficiency accelerator for specific underlying visual algorithms.

6. The accelerator generation tool according to claim 5, characterized in that In the hardware architecture parameter optimization module, the NNLUT process is as follows: (1) Traverse the template component variables and comprehensively evaluate resources and power consumption: Traverse the variable parameters related to the above template components, perform synthesis on the target backend, and obtain resource consumption and power consumption data; (2) Fully connected network fitting resource and power consumption model: A fully connected neural network is used to fit resource consumption and power consumption. To ensure accurate fitting of each data point, MAPE is used as the loss function during training. (3) Freezing the fitted network weights for architecture parameter search: After fitting is completed, the network weights are fixed; in the subsequent gradient descent-based architecture parameter optimization process, these weights are only used for forward propagation calculations and backpropagation gradient calculations and do not participate in parameter updates; this mechanism ensures the stability of the resource and power consumption model while supporting end-to-end optimization of architecture parameters; The objective function constructed by searching the hardware architecture parameters using gradient descent is as follows: Objective=TCycle+α*Energy+β*Balance+λ*(x all +ξ), (3) Among them, TCycle is the total inference cycle, which is calculated by hardware parameters; Energy is the total inference delay, which is calculated by NNLUT and hardware parameters; Balance is the balance item, which is the balance item of computing time of different pipeline stages; x all is a value constraint item, which ensures that the values ​​of the architecture parameters are within a reasonable range; ξ is a resource constraint item, which ensures that the generated RTL resources are within a reasonable range, and it is also estimated and calculated using NNLUT; α, β, and λ are weight coefficients.

Images