Neural network lightweight and efficient FPGA deployment method based on Bayesian optimization

Abstract

The invention provides a neural network lightweight and FPGA efficient deployment method based on Bayesian optimization. Firstly, an edge application data set and a reference model are selected, L1 regularization sparse parameters are adopted, and model performance is restored after pruning. Determining the quantization range and bit width of a convolutional layer and a full-connection layer, and finding an optimal quantization scheme through Bayesian optimization; the lightweight model is converted into an HLS project by means of an hls4ml library, and rapid conversion from the model to hardware design is achieved. For different FPGA models, reuse factor configuration is calculated, the HLS engineering resource utilization rate is controlled to be 70%-95%, and key operator hardware logic is optimized. And finally, constructing a full-process automatic deployment scheme which covers light weight, model conversion and comprehensive implementation stages. According to the method, a full-precision CNN reference model can be converted into a lightweight model meeting the requirements of low delay and low power consumption of an edge end, and efficient deployment of target hardware is realized.

Description

Technical Field

[0001] The present invention relates to the field of edge intelligence, and specifically to a neural network lightweight and efficient FPGA deployment method based on Bayesian optimization. Background Art

[0002] In the rapid development of information technology, breakthroughs in artificial intelligence, the Internet of Things, communication technology, and edge computing have jointly spawned edge intelligence, a key research field in modern intelligent systems. Edge intelligence combines two cutting-edge technologies, edge computing and artificial intelligence. By deploying AI models such as deep learning neural networks at the edge of the network, it realizes localized data processing and decision-making, thereby promoting technological innovation in the fields of intelligent driving, smart cities, industrial intelligence, and industrial Internet of Things.

[0003] Although the traditional centralized cloud computing architecture has significant advantages in computing and storage capabilities, its limitations are gradually becoming apparent when processing massive distributed data with high real-time requirements, such as latency and consumption of large amounts of bandwidth. Edge computing has emerged as a complementary technology to cloud computing.

[0004] In today's digital age, neural networks have become the core force driving the development of artificial intelligence, especially in the fields of image recognition and natural language processing. However, as neural network models become increasingly complex, their demand for computing resources and energy consumption has also increased dramatically, which is particularly prominent on resource-constrained edge devices. For example, traditional deep learning models such as AlexNet have more than 60 million parameters, which makes their deployment on edge devices face huge challenges. Summary of the invention

[0005] The purpose of the present invention is to provide a neural network lightweight and efficient FPGA deployment method based on Bayesian optimization to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions: A Bayesian optimization-based neural network lightweight and efficient FPGA deployment method, comprising: Step 1: Select the edge application dataset and choose a 4-layer convolutional structure as the benchmark model; Step 2: Use L1 regularization to perform parameter sparsification; Step 3: After pruning, keep the sparse connection structure unchanged, retrain the remaining sparse connections, restore the performance of the baseline model, and obtain a quantized pruning model; Step 4: Clearly quantize the weights and activations of the convolutional and fully connected layers. Layers that do not change the input data type do not need to be quantized. Step 5, select the quantization bit width; Step 6, using Bayesian optimization to search for the optimal quantization solution; Step 7: With the help of the hls4ml library, the lightweight quantized pruning model is automatically converted into an HLS project to achieve rapid mapping from the model to the hardware design. Step 8: For different FPGA models, the reuse factor of the layer is calculated as the conversion configuration by combining the configuration file and hardware constraints, the number of parameters of each layer of the quantized pruning model, and the number of DSPs of the target FPGA, to ensure that the resource utilization rate of the generated HLS project does not exceed q; where the value range of q is between 70% and 95%; Step 9: Optimize the hardware logic of key operators; Step 10: Establish a full-process automated deployment plan including three stages: lightweighting, model conversion, and comprehensive implementation, to achieve a one-stop solution from model design to hardware deployment.

[0007] Furthermore, the benchmark model in step 1 includes 4 convolution blocks, a fully connected layer and an activation layer, each convolution block is composed of a convolution layer, a batch normalization layer and an activation layer; the fully connected layer and the activation layer are used to learn the features extracted by convolution and classify the output results; in the entire benchmark model, two activation functions, ReLU and Sigmoid, are used.

[0008] Furthermore, the step 2 comprises: Step 2.1, use L1 regularization in the unstructured pruning method; Step 2.2, add the L1 norm of the weight as a penalty term in the loss function, and the modified L1 regularized loss function L λ (W) is: Among them, L is the original loss function, W is the weight matrix, represents the L1 norm of the weight matrix, that is, the sum of the absolute values ​​of the weights. λ is a hyperparameter of the regularization strength, which is used to weigh the weight of data fitting and regularization. λ > 0; In step 2.3, a dynamic pruning method is used to set the weight connections below the pruning threshold τ to zero during the training process, and the remaining connections with non-zero weights form sparse connections to achieve the sparsification of the baseline model parameters.

[0009] Furthermore, the step 2.3 includes: Step 2.3.1, pruning execution interval N t Execute the algorithm for the cycle; Step 2.3.2, determine the number of training steps t and the number of pruning start steps T i If t<T i, then only training without pruning; if t≥T i , then enter the dynamic pruning process; Step 2.3.3, at T i ≤t≤T e In the interval, count the zero values ​​in the benchmark model parameters, update the current sparsity S(t), and calculate the pruning threshold , where percentile(·) represents the percentile function, It means that the weight matrix W of the selected benchmark model is a set of absolute values ​​of each element; the weight matrix W is sparsified according to the pruning threshold, the weights below the threshold are reset to zero, and then t is incremented by 1.

[0010] Furthermore, the step 5 comprises: Step 5.1, select 4, 8, or 16 bits as the optional quantization bit width; In step 5.2, the maximum quantization bit width of the activation value is limited to 16 bits, and the maximum quantization bit width of the weight is limited to 8 bits.

[0011] Furthermore, the step 6 comprises: Step 6.1, using the Bayesian optimization algorithm, use the Horowitz energy consumption estimation method to construct the energy consumption objective function; the core formula of the Horowitz energy consumption estimation method is: ,in, They are the execution frequency, equivalent capacitance, operating voltage, and activity factor of each operation, which are refined into the following for the deep learning model: , where E compute is the energy consumption of the computational operation, E memory is the energy consumption of storage operation, E data_movement is the energy consumption of data transmission between storage unit and computing unit, E val is the overall energy consumption evaluation value of the deep learning model; Step 6.2, for the specific implementation of optimization objectives and constraints, introduce the concept of tolerance factor FF, which is defined as: , where △ acc represents the tolerable percentage of accuracy drop, R represents the energy consumption reduction ratio that the optimized model must achieve compared to the original model, S is an adjustment factor used to reduce the numerical weight of the reference model, thereby forcing the optimizer to select a smaller model, and parameter C ref and C trial They represent the cost of the reference model and the cost of the quantitative model being tested, respectively. The values ​​of both are derived from the Horowitz energy consumption evaluation method; Step 6.3, combined with FF, the objective function f(x) of the automatic quantization search process is determined as: f(x)=C trial +α·Lacc FF, where L acc is the accuracy loss function of the normal training process, α is the L acc The weighting factor of Step 6.4, the acquisition function u(x) corresponding to f(x) uses the expected improvement based on the predicted value of the objective function and the current optimal value f best To select the next evaluation point, the expression is: Where u(x) is represented by EI(x), EI(x) is the expected improvement acquisition function, E[·] represents mathematical expectation, max(·) represents the maximum value, and the expected improvement acquisition function is used to evaluate the expected degree of improvement of the function value when sampling a point near the current optimal value; if the improvement of the function value is less than the expected expected value after the Bayesian optimization quantized space search algorithm is run, it means that the current optimal value point may be a local optimal solution. At this time, the Bayesian optimization quantized space search algorithm will turn to other areas in the search definition domain to find the global optimal solution; In step 6.5, the automatic quantization search is performed using the quantization space search algorithm based on Bayesian optimization. Before the iterative search begins, a random configuration is selected to initialize, and a Gaussian process is used to fit the objective function as a proxy model. Then, in a limited number of trials, the quantization pruning model is trained to obtain the prior distribution and update the proxy model, so as to select the next trial configuration according to the acquisition function. After such iteration, the results of automatic quantization are given and the final quantization model configuration is saved.

[0012] Furthermore, the requirements of the quantization space search algorithm based on Bayesian optimization in step 6.5 are: define the quantization space V, the objective function f(x), and the acquisition function u(x); The purpose of the quantitative space search algorithm based on Bayesian optimization is to search for the maximum value of the acquisition function within the number of trials k < the maximum number of trials max_t ,in, is a combination of mathematical symbols used to find the function The value of x when the maximum value is obtained, , represents a data set consisting of k-1 observations of the function f(x), x n is a function A series of values ​​of the independent variable in the data set In, x n The corresponding function value f(x n ) together constitute the observation value, n is the index value from 1 to k-1, and each x n Represents the value of the independent variable when observing the objective function f(x); The process of the quantized space search algorithm based on Bayesian optimization includes: S201, randomly select several configurations to test as initialization; S202, using Gaussian process to fit the objective function as a surrogate model; S203, iterative search according to the following steps: S203.1, training a quantization pruning model for the current quantization configuration; S203.2, update the prior distribution of the objective function f(x), and t ) for sampling, x t It is the data set composed of the results of the previous t-1 trials at the t-th iteration search. , so that the acquisition function The value of the independent variable x when the maximum value is obtained is used to sample the objective function f(x); S203.3, adding the results of this experiment to a data set consisting of past observations of the objective function f(x), expanding the data set, and then updating the proxy model; S203.4, use the updated proxy model to calculate the acquisition function and select the point x with the largest acquisition function value from the search space t+1 , and use it as the next test configuration; S203.5, increase the value of the number of iterations k by 1; S203.6, determine whether the number of iterations k is less than the maximum number of trials max_t. If so, return to S203.1 to continue the subsequent operations; if not, terminate the iterative search; S204, saving the final quantization configuration, and retraining the quantization pruning model to obtain complete performance; S205, testing the energy consumption and accuracy of the retrained quantized pruning model.

[0013] Furthermore, the step 9 comprises: Step 9.1, re-optimize the hardware logic of key operators, and use the sparse acceleration method to further improve efficiency by taking advantage of the sparse characteristics of lightweight model parameters; Step 9.2, configure the reuse factor reasonably. In hls4ml, configure the reuse factor for each layer to set the number of times the multiplier is used when calculating the output value of the layer to balance the contradiction between inference latency and resource consumption.

[0014] Compared with the prior art, the present invention aims to transform a full-precision CNN benchmark model into a lightweight model suitable for low latency and low power consumption requirements at the edge through a series of interrelated steps, and realize its efficient deployment on the target hardware. The specific beneficial effects include: 1) Significant model lightweighting: L1 regularization in the unstructured pruning method is adopted, and the L1 norm of the weight is added as a penalty term in the loss function. The weights below the threshold are dynamically reset to zero, which effectively achieves the sparsification of the baseline model parameters, removes redundant connections, and significantly reduces the number of model parameters and calculations, creating conditions for subsequent deployment on the resource-constrained edge.

[0015] 2) Quantization scheme optimization: Using the Bayesian optimization algorithm, combined with the Horowitz energy consumption estimation method, with model accuracy and energy consumption as the objective function, search for the optimal mixed precision quantization scheme. Introduce the tolerance factor FF quantization to the decline of indicators such as model accuracy, and incorporate it into the objective function to guide the Bayesian optimization search, while ensuring a certain accuracy, minimize energy consumption and improve the operating efficiency of the model on edge devices.

[0016] 3) Strong hardware adaptability: For different FPGA models, through configuration files and hardware constraints, combined with the number of parameters of each layer of the neural network and the number of DSPs of the target FPGA, the reuse factor of the layer is calculated as the conversion configuration to ensure that the resource utilization rate of the generated HLS project is reasonable (such as not exceeding the set threshold q), give full play to the computational parallelism of FPGA, improve the utilization of hardware resources, and achieve efficient hardware-model adaptation.

[0017] 4) Full-process automation: Establish a full-process automated deployment solution including three stages: lightweight, model conversion, and comprehensive implementation. With the help of relevant tool libraries (such as the hls4ml library), rapid mapping from model to hardware design can be achieved, reducing manual intervention, improving deployment efficiency, and realizing a one-stop solution from model design to hardware deployment, thereby reducing development costs and cycles.

[0018] 5) Performance guarantee: Re-optimize the hardware logic of key operators, combine sparse acceleration methods and reasonable reuse factor configuration to further improve the model inference speed, and retrain the remaining sparse connections after pruning to effectively restore the model performance and ensure the reliability and effectiveness of the lightweight model in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a process framework diagram of the present invention.

[0020] Figure 2-Figure 5 The following is a distribution diagram of weight parameters under different pruning effects, taking the Conv_0 layer as an example. The vertical axis is in exponential form.

[0021] Figure 2 It is the weight distribution diagram of the unpruned model of the Conv_0 layer, showing the weight distribution of the Conv_0 layer when it is not pruned, with high values ​​on both sides around 0 and low values ​​in the middle.

[0022] Figure 3 It is the weight distribution diagram of the Conv_0 layer 50% pruning model, which shows the weight distribution after 50% pruning of the Conv_0 layer. The zero-value weight accounts for 50.00%, and the distribution is more sparse.

[0023] Figure 4 This is the weight distribution diagram of the Conv_0 layer 75% pruning model, which shows the weight distribution after 75% pruning of the Conv_0 layer. The zero-value weight accounts for 75.00%, and the sparsity is further improved.

[0024] Figure 5 This is the weight distribution diagram of the Conv_0 layer 90% pruning model, which reflects the weight distribution of the Conv_0 layer after 85% pruning. The zero-value weight accounts for 90.05%, and the distribution is highly sparse.

[0025] Figure 6 This is the distribution diagram of the weight values ​​of the Conv_0 layer of the quantized model after pruning, and the vertical axis in the figure is in exponential form.

[0026] Figure 7 This is a diagram of the deployment solution based on hls4ml.

[0027] Figure 8 is a schematic diagram of reuse factors. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] See also Figure 1-Figure 8 , a neural network lightweight and efficient FPGA deployment method based on Bayesian optimization, including: Step 1: Select the edge application dataset and select a 4-layer convolution structure as the benchmark model. The benchmark model specifically includes 4 convolution blocks, a fully connected layer and an activation layer. Each convolution block consists of a convolution layer, a batch normalization layer and an activation layer. The fully connected layer and the activation layer are used to learn the features extracted by the convolution and classify the output results. In the entire network, two activation functions, ReLU and Sigmoid, are used.

[0030] Step 2: Use L1 regularization to perform parameter sparsification, which includes: Step 2.1, use L1 regularization in the unstructured pruning method.

[0031] Step 2.2, add the L1 norm of the weight as a penalty term in the loss function, and the modified L1 regularized loss function L λ (W) is: Among them, L is the original loss function, W is the weight matrix, represents the L1 norm of the weight matrix, that is, the sum of the absolute values ​​of the weights. λ is a hyperparameter of the regularization strength, which is used to weigh the weight of data fitting and regularization. λ > 0.

[0032] In step 2.3, a dynamic pruning method is used to set the weight connections below the pruning threshold τ to zero during the training process, and the remaining connections with non-zero weights form sparse connections, thus achieving the sparsification of the baseline model parameters.

[0033] Among them, the pruning threshold τ is determined according to the model training situation, and the algorithm The calculation results are the values ​​at the percentile of the sparsity S(t) in the distribution of the absolute value of the weight. Among them, percentile(·) represents the percentile function, which is a function used to calculate percentiles. It returns the value at the corresponding percentile in the data set. Specifically, given a data set and a percentage, the percentile function finds the number in the data set that makes the data of that percentage less than or equal to this value. Refers to the weight matrix of the selected benchmark model, W is a set of absolute values ​​of each element, That is, find the value at the sparsity S(t) percentile in the absolute value distribution of the weight, which is the pruning threshold T(t).

[0034] The requirements of the dynamic pruning method are: define the pruning execution interval N t , the number of steps for pruning to start is T i , the number of steps for pruning to end is T e , the target sparsity is S e .

[0035] The goal of the dynamic pruning method is: compared with the initial sparsity S i , the sparsity S(t) at step number t should meet the following requirements:

[0036] The process of dynamic pruning method includes: S101, N t Execute the algorithm for intervals.

[0037] S102, if t<T i , then only training without pruning, otherwise go to S103.

[0038] S103, when the training step number t satisfies T i ≤t≤T e When , the following operations are performed. Where T i is the number of steps to start pruning, T e The number of steps for pruning to end.

[0039] S104, counting the zero values ​​in the benchmark model parameters, and updating the current sparsity S(t).

[0040] S105, calculate pruning threshold , that is, the value at the sparsity S(t) percentile in the distribution of the absolute value of the weight.

[0041] S106, applying sparseness to the weights W according to the threshold, and resetting the weights below the threshold to zero.

[0042] S107, increase the value of the training step number t by 1.

[0043] S108, end the above when T i ≤t≤T e The loop operation performed when

[0044] S109, retrain the remaining sparse connections to restore model performance.

[0045] S110, check whether the sparsity of the current model has reached the target sparsity S e .

[0046] Step 3: After pruning, keep the sparse connection structure unchanged. Retrain the remaining sparse connections to restore the performance of the baseline model and obtain a quantized pruned model.

[0047] Step 4: Clearly quantize the weights and activations of the convolutional and fully connected layers. Layers that do not change the input data type do not need to be quantized.

[0048] Step 5: Select the quantization bit width, including: In step 5.1, considering the superiority of power-of-two quantization in terms of hardware efficiency and resource consumption, 4, 8, and 16 bits are selected as the optional quantization bit widths.

[0049] In step 5.2, the maximum quantization bit width of the activation value is limited to 16 bits, and the maximum quantization bit width of the weight is limited to 8 bits. After this step of clarifying the quantization bit width, combined with the previously completed pruning steps, L1 regularization pruning and quantization configuration settings are fully implemented, and a quantized pruning model is truly obtained, which provides a clear object for subsequent training and optimization using this model in Bayesian optimization.

[0050] Step 6: Use Bayesian optimization to search for the optimal quantization solution, including: Step 6.1, using the Bayesian optimization algorithm, use the Horowitz energy consumption estimation method to construct the energy consumption objective function. The core formula of the Horowitz energy consumption evaluation method is: ,in, They are the execution frequency, equivalent capacitance, operating voltage, and activity factor (probability of the operation occurring) of each operation, which are refined into the following for the deep learning model: , where E compute is the energy consumption of the computational operation, E memory is the energy consumption of storage operation, E data_movement is the energy consumption of data transmission between storage unit and computing unit, E val is the overall energy consumption estimate of the deep learning model.

[0051] Step 6.2, for the specific implementation of optimization objectives and constraints, introduce the concept of tolerance factor FF, which is defined as: , where △ acc represents the tolerable percentage of accuracy drop, R represents the energy consumption reduction ratio that the optimized model must achieve compared to the original model, S is an adjustment factor used to reduce the numerical weight of the reference model, thereby forcing the optimizer to select a smaller model, and parameter C ref and C trial They represent the cost of the reference model and the cost of the quantitative model being tested, respectively, and their values ​​are derived from the Horowitz energy consumption evaluation method.

[0052] Step 6.3, combined with FF, the objective function f(x) of the automatic quantization search process is determined as: f(x)=C trial +α·L acc FF, where L acc is the accuracy loss function of the normal training process, α is the L acc The weighting factor of .

[0053] Step 6.4, the acquisition function u(x) corresponding to f(x) uses the expected improvement based on the predicted value of the objective function and the current optimal value f best To select the next evaluation point, the expression is: Where u(x) is represented by EI(x), EI(x) is the expected improvement acquisition function, E[·] represents mathematical expectation, max(·) represents the maximum value, and the expected improvement acquisition function is used to evaluate the expected degree of improvement of the function value when sampling a point near the current optimal value; if the improvement of the function value is less than the expected value after the Bayesian optimization quantized space search algorithm is run, it means that the current optimal value point may be a local optimal solution. At this time, the Bayesian optimization quantized space search algorithm will turn to other areas in the search domain to find the global optimal solution.

[0054] In step 6.5, the automatic quantization search is performed using the quantization space search algorithm based on Bayesian optimization. Before the iterative search begins, a random configuration is selected to initialize, and a Gaussian process is used to fit the objective function as a proxy model. Then, in a limited number of trials, the quantization pruning model is trained to obtain the prior distribution and update the proxy model, so as to select the next trial configuration according to the acquisition function. After such iteration, the results of automatic quantization are given and the final quantization model configuration is saved.

[0055] The requirements of the quantization space search algorithm based on Bayesian optimization are: define the quantization space V, the objective function f(x), and the acquisition function u(x).

[0056] The purpose of the quantitative space search algorithm based on Bayesian optimization is to search for the maximum value of the acquisition function within the number of trials t < the maximum number of trials max_t ,in, is a combination of mathematical symbols used to find the function The value of x when the maximum value is obtained, , represents a data set consisting of t-1 observations of the function f(x), x n is a function A series of values ​​of the independent variable in the data set In, x n The corresponding function value f(x n ) together constitute the observation value, n is the index value from 1 to k-1, and each x n Represents the value of the independent variable when observing the objective function f(x) once.

[0057] The process of the quantized space search algorithm based on Bayesian optimization includes: S201, randomly select several configurations to test as initialization.

[0058] S202, using a Gaussian process to fit the objective function as a proxy model.

[0059] S203, iterative search according to the following steps: S203.1, train the model for the current quantization configuration.

[0060] S203.2, update the prior distribution of the objective function f(x), and t ) for sampling, x t It is the data set composed of the results of the previous t-1 trials at the t-th iteration search. , so that the acquisition function The value of the independent variable x when the maximum value is obtained is used to sample the objective function f(x).

[0061] S203.3, adding the results of this experiment to the data set consisting of past observations of the function f(x), expanding the data set, and then updating the proxy model.

[0062] S203.4, use the updated proxy model to calculate the acquisition function and select the point x with the largest acquisition function value from the search space t+1 , and use it as the next test configuration.

[0063] S203.5, increase the value of the number of iterations t by 1.

[0064] S203.6, determine whether the number of iterations t is less than the maximum number of trials max_t. If so, return to S203.1 to continue subsequent operations; if not, end the iterative search.

[0065] S204, saving the final quantization configuration, and retraining the quantization pruning model to obtain complete performance.

[0066] S205, testing the energy consumption and accuracy of the quantized pruning model after the above retraining.

[0067] Step 7: With the help of the hls4ml library, the lightweight quantized pruning model is automatically converted into an HLS project to achieve rapid mapping from the model to the hardware design. Figure 7 The efficient deployment solution generation process proposed in this paper is shown, including three stages: lightweight (yellow box), model conversion (blue box) and comprehensive implementation (black box). The solution aims to automate the entire process and can optimize engineering configuration for different FPGA models to avoid resource overload or inefficiency while ensuring that the model inference accuracy is not affected.

[0068] Step 8: For different FPGA models such as xczu3eg, xczu5eg, and xqzu7ev, the reuse factor of each layer is calculated as the conversion configuration by combining the parameter quantity of each layer of the quantized pruning model neural network and the number of DSPs of the target FPGA through configuration files and hardware constraints to ensure that the resource utilization rate of the generated HLS project is between 70% and 95%, preferably 90%.

[0069] In hls4ml, you can configure the reuse factor of each layer and set the multiplier to calculate the number of times the output value of the layer is used, such as Figure 8 (Schematic diagram of reuse factor). The figure simplifies the multiplication and accumulation operation of the neural network, and the formula is {a1[0], a1[1]} ={a0[0] · w0 + a0[1] · w1, a0[0] · w3 + a0[1] · w4}, where a and w represent activation and weight respectively. When the reuse factor is 1, the calculation is completely parallel and multiplication is completed in one clock; when it is 2, each multiplier is used twice, requiring 2 clocks; when it is 4, 4 clocks are required, reflecting the idea of ​​"trading speed for area", that is, reusing hardware resources, allocating computing tasks, sacrificing total latency to reduce resource requirements.

[0070] Step 9: Optimize the hardware logic of key operators, including: Step 9.1, re-optimize the hardware logic of key operators, combine sparse acceleration methods such as sparse matrix multiplication, sparse convolution, and reasoning acceleration after pruning, and use the sparse characteristics of lightweight model parameters to further improve efficiency. Sparse acceleration methods aim to use the sparsity of data or models to improve computing efficiency and reduce memory usage. They are widely used in scientific computing (such as numerical simulation, quantum chemistry calculations, etc.), machine learning (such as neural network model training and reasoning, recommendation systems, etc.), signal processing (such as image processing, speech recognition, etc.) and other fields.

[0071] Step 9.2, configure the reuse factor reasonably. In hls4ml, configure the reuse factor for each layer to set the number of times the multiplier is used when calculating the output value of the layer to balance the contradiction between inference latency and resource consumption.

[0072] Step 10: Establish a full-process automated deployment solution with three stages: lightweight (implemented through automatic quantization search and regularization pruning functions), model conversion (exporting model structure and parameter files with the help of deep learning library, and converting to HLS project using hls4ml library), and comprehensive implementation (synthesis and layout and routing according to HLS internal configuration, and using Vivado tool chain to generate usable bitstream files), to achieve a one-stop solution from model design to hardware deployment.

[0073] Example According to the above detailed implementation method, the embodiment of the present invention adopts pruning and quantization technology of deep learning model to improve the sparsity and reasoning efficiency of the model. The implementation focuses on two aspects: The first is the implementation of the model pruning strategy. Figure 2-Figure 5 It is a weight distribution effect diagram of the model after training under different pruning degrees, and the distribution diagram of the unpruned benchmark model is introduced as a reference. By comparing the performance of the model under different pruning degrees, the pruning degree that achieves higher sparsity while maintaining the model performance is selected. Specifically, the L1 regularization pruning method is adopted in the embodiment, and the pruning degrees are 50%, 75% and 85% respectively. The experimental results show that even at a high pruning degree of 85%, the accuracy of the model can still be maintained at 99.78%, the AUC is 0.9928, the F1-Score is 0.9718, and the performance loss is small.

[0074] Another aspect is the implementation of the model quantization strategy. In order to prevent the problem of excessive accuracy loss and difficulty in model training caused by pruning after quantization, the automatic quantization strategy of the present invention is relatively conservative and has a low tolerance for accuracy loss of only 1%. The quantization configuration is as follows: for the convolutional layer (Conv), the weight bit widths are 8, 4, 8, 4, and the activation bit widths are 8, 16, 16, 16, respectively; for the fully connected layer (FC), the weight bit widths are 16, 8, and the activation bit widths are 8. The output layer does not participate in the quantization search and is fixed at a bit width of 16 bits. Under this optimal configuration, the estimated total inference energy consumption of the model after quantization is reduced by 72.49%, while the accuracy and AUC are almost unchanged, and the F1-Score is only reduced by 0.96%.

[0075] Combining pruning and quantization strategies, in the embodiment, after performing regularized pruning with 75% sparsity, the above quantization configuration is used for automatic quantization, and the theoretical compression ratio of the model reaches 14.5 times, while still maintaining an accuracy of 99.64% and only reducing the F1-Score by 1.25%. Figure 6 It is the weight value distribution diagram after pruning and automatic quantization. Figure 2-Figure 5 The unquantized parameter distribution in the quantized model has a clearer distribution of weights due to the reduction of the bit width after quantization, and all non-zero weight values ​​near zero are eliminated. The method of the present invention can significantly improve the sparsity and reasoning efficiency of the model while maintaining the performance of the model.

[0076] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A Bayesian optimization-based neural network lightweight and efficient FPGA deployment method, characterized in that: include: Step 1: Select the edge application dataset and choose a 4-layer convolutional structure as the benchmark model; Step 2: Use L1 regularization to perform parameter sparsification; Step 3: After pruning, keep the sparse connection structure unchanged, retrain the remaining sparse connections, restore the performance of the baseline model, and obtain a quantized pruned model; Step 4: Clearly quantize the weights and activations of the convolutional and fully connected layers. Layers that do not change the input data type do not need to be quantized. Step 5, select the quantization bit width; Step 6, using Bayesian optimization to search for the optimal quantization solution; Step 7: With the help of the hls4ml library, the lightweight quantized pruning model is automatically converted into an HLS project to achieve rapid mapping from the model to the hardware design. Step 8: For different FPGA models, the reuse factor of the layer is calculated as the conversion configuration by combining the configuration file and hardware constraints, the number of parameters of each layer of the quantized pruning model, and the number of DSPs of the target FPGA, to ensure that the resource utilization rate of the generated HLS project does not exceed q; where the value range of q is between 70% and 95%; Step 9: Optimize the hardware logic of key operators; Step 10: Establish a full-process automated deployment plan including three stages: lightweighting, model conversion, and comprehensive implementation, to achieve a one-stop solution from model design to hardware deployment.

2. According to the Bayesian optimization-based neural network lightweight and efficient FPGA deployment method of claim 1, it is characterized in that: The benchmark model in step 1 includes 4 convolution blocks, a fully connected layer and an activation layer, each convolution block consists of a convolution layer, a batch normalization layer and an activation layer; the fully connected layer and the activation layer are used to learn the features extracted by convolution and classify the output results; in the entire benchmark model, two activation functions, ReLU and Sigmoid, are used.

3. According to the Bayesian optimization-based neural network lightweight and efficient FPGA deployment method of claim 1, it is characterized in that: The step 2 comprises: Step 2.1, use L1 regularization in the unstructured pruning method; Step 2.2, add the L1 norm of the weight as a penalty term in the loss function, and the modified L1 regularized loss function L λ (W) is: Among them, L is the original loss function, W is the weight matrix, represents the L1 norm of the weight matrix, that is, the sum of the absolute values ​​of the weights. λ is a hyperparameter of the regularization strength, which is used to weigh the weight of data fitting and regularization. λ > 0; In step 2.3, a dynamic pruning method is used to set the weight connections below the pruning threshold τ to zero during the training process, and the remaining connections with non-zero weights form sparse connections to achieve the sparsification of the baseline model parameters.

4. According to claim 3, a Bayesian optimization-based neural network lightweight and efficient FPGA deployment method is characterized in that: The step 2.3 includes: Step 2.3.1, pruning execution interval N t Execute the algorithm for the cycle; Step 2.3.2, determine the number of training steps t and the number of pruning start steps T i If t<T i , then only training without pruning; if t≥T i , then enter the dynamic pruning process; Step 2.3.3, at T i ≤t≤T e In the interval, count the zero values ​​in the benchmark model parameters, update the current sparsity S(t), and calculate the pruning threshold , where percentile(·) represents the percentile function, It means that the weight matrix W of the selected benchmark model is a set of absolute values ​​of each element; the weight matrix W is sparsified according to the pruning threshold, the weights below the threshold are reset to zero, and then t is incremented by 1.

5. According to the Bayesian optimization-based neural network lightweight and efficient FPGA deployment method of claim 1, it is characterized in that: The step 5 comprises: Step 5.1, select 4, 8, or 16 bits as the optional quantization bit width; In step 5.2, the maximum quantization bit width of the activation value is limited to 16 bits, and the maximum quantization bit width of the weight is limited to 8 bits.

6. According to the Bayesian optimization-based neural network lightweight and efficient FPGA deployment method of claim 1, it is characterized in that: The step 6 comprises: Step 6.1, using the Bayesian optimization algorithm, use the Horowitz energy consumption estimation method to construct the energy consumption objective function; the core formula of the Horowitz energy consumption estimation method is: ,in, They are the execution frequency, equivalent capacitance, operating voltage, and activity factor of each operation; for the deep learning model, they are refined into , where E compute is the energy consumption of the computational operation, E memory is the energy consumption of storage operation, E data_movement is the energy consumption of data transmission between storage unit and computing unit, E val is the overall energy consumption evaluation value of the deep learning model; Step 6.2, for the specific implementation of optimization objectives and constraints, introduce the concept of tolerance factor FF, which is defined as: , where △ acc represents the tolerable percentage of accuracy drop, R represents the energy consumption reduction ratio that the optimized model must achieve compared to the original model, S is an adjustment factor used to reduce the numerical weight of the reference model, thereby forcing the optimizer to select a smaller model, and parameter C ref and C trial They represent the cost of the reference model and the cost of the quantitative model being tested, respectively. The values ​​of both are derived from the Horowitz energy consumption evaluation method; Step 6.3, combined with FF, the objective function f(x) of the automatic quantization search process is determined as: f(x)=C trial +α·L acc FF, where L acc is the accuracy loss function of the normal training process, α is the L acc The weighting factor of Step 6.4, the acquisition function u(x) corresponding to f(x) uses the expected improvement based on the predicted value of the objective function and the current optimal value f best To select the next evaluation point, the expression is: Where u(x) is represented by EI(x), EI(x) is the expected improvement acquisition function, E[·] represents mathematical expectation, max(·) represents the maximum value, and the expected improvement acquisition function is used to evaluate the expected degree of improvement of the function value when sampling a point near the current optimal value; if the improvement of the function value is less than the expected expected value after the Bayesian optimization quantized space search algorithm is run, it means that the current optimal value point may be a local optimal solution. At this time, the Bayesian optimization quantized space search algorithm will turn to other areas in the search definition domain to find the global optimal solution; In step 6.5, the automatic quantization search is performed using the quantization space search algorithm based on Bayesian optimization. Before the iterative search begins, a random configuration is selected to initialize, and a Gaussian process is used to fit the objective function as a proxy model. Then, in a limited number of trials, the quantization pruning model is trained to obtain the prior distribution and update the proxy model, so as to select the next trial configuration according to the acquisition function. After such iteration, the results of automatic quantization are given and the final quantization model configuration is saved.

7. The method for lightweight and efficient FPGA deployment of a neural network based on Bayesian optimization according to claim 6 is characterized in that: The requirements of the quantization space search algorithm based on Bayesian optimization in step 6.5 are: define the quantization space V, the objective function f(x), and the acquisition function u(x); The purpose of the quantitative space search algorithm based on Bayesian optimization is to search for the maximum value of the acquisition function within the number of trials k < the maximum number of trials max_t ,in, is a combination of mathematical symbols used to find the function The value of x when the maximum value is obtained, , represents a data set consisting of k-1 observations of the function f(x), x n is a function A series of values ​​of the independent variable in the data set In, x n The corresponding function value f(x n ) together constitute the observation value, n is the index value from 1 to k-1, each x n Represents the value of the independent variable when observing the objective function f(x) once; The process of the quantized space search algorithm based on Bayesian optimization includes: S201, randomly select several configurations to test as initialization; S202, using Gaussian process to fit the objective function as a surrogate model; S203, iterative search according to the following steps: S203.1, training a quantization pruning model for the current quantization configuration; S203.2, update the prior distribution of the objective function f(x), and t ) for sampling, x t It is the data set composed of the results of the previous t-1 trials at the t-th iteration search. , so that the acquisition function The value of the independent variable x when the maximum value is obtained is used to sample the objective function f(x); S203.3, adding the results of this experiment to a data set consisting of past observations of the objective function f(x), expanding the data set, and then updating the proxy model; S203.4, use the updated proxy model to calculate the acquisition function and select the point x with the largest acquisition function value from the search space t+1 , and use it as the next test configuration; S203.5, increase the value of the number of iterations k by 1; S203.6, determine whether the number of iterations k is less than the maximum number of trials max_t. If so, return to S203.1 to continue the subsequent operations; if not, terminate the iterative search; S204, saving the final quantization configuration, and retraining the quantization pruning model to obtain complete performance; S205, testing the energy consumption and accuracy of the retrained quantized pruning model.

8. According to the Bayesian optimization-based neural network lightweight and efficient FPGA deployment method of claim 1, it is characterized in that: The step 9 comprises: Step 9.1, re-optimize the hardware logic of key operators, and use the sparse acceleration method to further improve efficiency by taking advantage of the sparse characteristics of lightweight model parameters; Step 9.2, configure the reuse factor reasonably. In hls4ml, configure the reuse factor for each layer to set the number of times the multiplier is used when calculating the output value of the layer to balance the contradiction between inference latency and resource consumption.

Images