A lightweight and efficient FPGA deployment method for neural networks based on Bayesian optimization

Abstract

The present invention proposes a method for lightweight neural networks and efficient FPGA deployment based on Bayesian optimization. First, an edge application dataset and a benchmark model are selected, and L1 regularization is used to sparsely parameterize the model. Model performance is restored after pruning. The quantization range and bit width of the convolutional layer and fully connected layer are then determined, and the optimal quantization scheme is found using Bayesian optimization. The lightweight model is converted into an HLS project with the help of the hls4ml library, enabling rapid model-to-hardware design conversion. For different FPGA models, the reuse factor configuration is calculated, the HLS project resource utilization rate is controlled at 70%-95%, and the key operator hardware logic is optimized. Finally, a full-process automated deployment solution is constructed, covering the lightweighting, model conversion, and comprehensive implementation stages. This method can convert a full-precision CNN benchmark model into a lightweight model that meets the low latency and low power consumption requirements of the edge, enabling efficient deployment of the target hardware.

Description

Technical Field

[0001] The present invention relates to the field of edge intelligence, and specifically to a lightweight and efficient FPGA deployment method for neural networks based on Bayesian optimization. Background Art

[0002] Amidst the rapid development of information technology, breakthroughs in artificial intelligence, the Internet of Things (IoT), communications technologies, and edge computing have given rise to edge intelligence, a key research area in modern intelligent systems. Edge intelligence integrates cutting-edge technologies, edge computing and AI. By deploying AI models such as deep learning neural networks at the edge of the network, it enables localized data processing and decision-making, driving technological innovation in areas such as intelligent driving, smart cities, industrial intelligence, and the Industrial Internet of Things.

[0003] While traditional centralized cloud computing architecture offers significant advantages in computing and storage capabilities, its limitations are becoming increasingly apparent when processing massive amounts of distributed data with high real-time requirements, such as latency and bandwidth consumption. Edge computing has emerged as a complementary technology to cloud computing.

[0004] In today's digital age, neural networks have become a core force driving the development of artificial intelligence, achieving remarkable success in areas such as image recognition and natural language processing. However, as neural network models become increasingly complex, their demands for computing resources and energy consumption have increased dramatically. This is particularly true on resource-constrained edge devices. For example, traditional deep learning models such as AlexNet have over 60 million parameters, making their deployment on edge devices a significant challenge. Summary of the Invention

[0005] The purpose of the present invention is to provide a lightweight and efficient FPGA deployment method for neural networks based on Bayesian optimization to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A Bayesian optimization-based method for lightweight and efficient FPGA deployment of neural networks, including:

[0008] Step 1: Select an edge application dataset and choose a 4-layer convolutional structure as the baseline model;

[0009] Step 2: Use L1 regularization to perform parameter sparsification;

[0010] Step 3: After pruning, the sparse connection structure is kept unchanged, and the remaining sparse connections are retrained to restore the baseline model performance to obtain a quantized pruned model.

[0011] Step 4: Clearly quantize the weights and activations of the convolutional and fully connected layers. Layers that do not change the input data type do not need to be quantized.

[0012] Step 5, select the quantization bit width;

[0013] Step 6: Use Bayesian optimization to search for the optimal quantization solution;

[0014] Step 7: With the help of the hls4ml library, the lightweight quantized pruned model is automatically converted into an HLS project to achieve rapid mapping from the model to the hardware design.

[0015] Step 8: For different FPGA models, the reuse factor of each layer is calculated based on the configuration file and hardware constraints, the number of parameters in each layer of the quantized pruning model, and the number of DSPs in the target FPGA. This is used as the conversion configuration to ensure that the resource utilization rate of the generated HLS project does not exceed q. The value of q ranges from 70% to 95%.

[0016] Step 9: Optimize the hardware logic of key operators;

[0017] Step 10: Establish a full-process automated deployment plan including three stages: lightweighting, model conversion, and comprehensive implementation, to achieve a one-stop solution from model design to hardware deployment.

[0018] Furthermore, the benchmark model in step 1 includes 4 convolution blocks, a fully connected layer and an activation layer, each convolution block consists of a convolution layer, a batch normalization layer and an activation layer; the fully connected layer and the activation layer are used to learn the features extracted by convolution and classify the output results; in the entire benchmark model, two activation functions, ReLU and Sigmoid, are used.

[0019] Furthermore, the step 2 includes:

[0020] Step 2.1, use L1 regularization in the unstructured pruning method;

[0021] Step 2.2, add the L1 norm of the weight as a penalty term in the loss function, and the modified L1 regularization loss function L λ (W) is:

[0022]

[0023] Among them, L is the original loss function, W is the weight matrix, represents the L1 norm of the weight matrix, that is, the sum of the absolute values ​​of the weights. λ is a hyperparameter of the regularization strength, which is used to weigh the proportion of data fitting and regularization. λ > 0;

[0024] In step 2.3, a dynamic pruning method is used to set the weight connections below the pruning threshold τ to zero during the training process, and the remaining connections with non-zero weights form sparse connections to achieve the sparsification of the baseline model parameters.

[0025] Furthermore, the step 2.3 includes:

[0026] Step 2.3.1, pruning is performed with interval N t Execute the algorithm for the period;

[0027] Step 2.3.2, determine the number of training steps t and the number of pruning start steps T i The size relationship, if t<T i , then only training without pruning; if t≥T i , then enter the dynamic pruning process;

[0028] Step 2.3.3, at T i ≤t≤T e In the interval, count the zero values ​​in the benchmark model parameters, update the current sparsity S(t), and calculate the pruning threshold , where percentile(·) represents the percentile function, The weight matrix W of the selected benchmark model is a set of the absolute values ​​of each element; sparsification is applied to the weight matrix W according to the pruning threshold, the weights below the threshold are reset to zero, and then t is incremented by 1.

[0029] Furthermore, the step 5 includes:

[0030] Step 5.1, select 4, 8, or 16 bits as the optional quantization bit width;

[0031] In step 5.2, the maximum quantization bit width of the activation value is limited to 16 bits, and the maximum quantization bit width of the weight is limited to 8 bits.

[0032] Furthermore, the step 6 includes:

[0033] Step 6.1: Use the Bayesian optimization algorithm and the Horowitz energy consumption estimation method to construct the energy consumption objective function. The core formula of the Horowitz energy consumption estimation method is: ,in, They are the execution frequency, equivalent capacitance, operating voltage and activity factor of each operation, which are refined into , where E compute is the energy consumption of computing operations, E memory is the energy consumption of storage operation, E data_movement is the energy consumption of data transmission between storage unit and computing unit, E val is the overall energy consumption evaluation value of the deep learning model;

[0034] Step 6.2, for the specific implementation of optimization objectives and constraints, introduce the concept of tolerance factor FF, which is defined as: , where △ acc Indicates the tolerable percentage of accuracy drop, R indicates the energy consumption reduction ratio that the optimized model must achieve compared to the original model, S is an adjustment factor used to reduce the numerical weight of the reference model, thereby forcing the optimizer to select a smaller model, and the parameter C ref and C trial Represent the cost of the reference model and the cost of the quantitative model being tested, respectively. The values ​​of both are derived from the Horowitz energy consumption evaluation method;

[0035] Step 6.3, combined with FF, the objective function f(x) of the automatic quantization search process is determined as: f(x)=C trial +α·L acc FF, where L acc is the accuracy loss function of the normal training process, α is the L acc The weighting factor of

[0036] In step 6.4, the acquisition function u(x) corresponding to f(x) is improved using expectation, based on the predicted value of the objective function and the current optimal value f best To select the next evaluation point, the expression is:

[0037]

[0038] Where u(x) is represented by EI(x), EI(x) is the expected improvement acquisition function, E[·] represents the mathematical expectation, and max(·) represents the maximum value. The expected improvement acquisition function is used to evaluate the expected degree of improvement of the function value when sampling a point near the current optimal value. If the improvement of the function value is less than the expected value after the Bayesian optimization quantized space search algorithm is run, it means that the current optimal value point may be a local optimal solution. At this time, the Bayesian optimization quantized space search algorithm will turn to other areas within the search domain to find the global optimal solution.

[0039] In step 6.5, an automatic quantization search is performed using a quantization space search algorithm based on Bayesian optimization. Before the iterative search begins, a random configuration is selected for initialization, and a Gaussian process is used to fit the objective function as a proxy model. Then, within a limited number of trials, the quantization pruning model is trained to obtain the prior distribution and update the proxy model, thereby selecting the next trial configuration according to the acquisition function. After this iteration, the results of the automatic quantization are given, and the final quantization model configuration is saved.

[0040] Furthermore, the requirements of the quantization space search algorithm based on Bayesian optimization in step 6.5 are as follows: define the quantization space V, the objective function f(x), and the acquisition function u(x);

[0041] The purpose of the quantitative space search algorithm based on Bayesian optimization is to search for the maximum value of the acquisition function within the number of trials k < the maximum number of trials max_t ,in, is a combination of mathematical symbols used to find the function The value of x when the maximum value is obtained, , represents a data set consisting of k-1 observations of the function f(x), x n is a function A series of values ​​of the independent variable in the data set In, x n The corresponding function value f(x n ) together constitute the observation value, n is the index value from 1 to k-1, each x n Represents the value of the independent variable when observing the objective function f(x);

[0042] The process of the quantized space search algorithm based on Bayesian optimization includes:

[0043] S201, randomly select several configurations to test as initialization;

[0044] S202, using Gaussian process to fit the objective function as a surrogate model;

[0045] S203, iterative search according to the following steps:

[0046] S203.1, train the quantization pruning model for the current quantization configuration;

[0047] S203.2, update the prior distribution of the objective function f(x), and t ) for sampling, x t It is the data set based on the results of the previous t-1 trials at the t-th iteration search , so that the acquisition function The value of the independent variable x when the maximum value is obtained is used to sample the objective function f(x);

[0048] S203.3, adding the results of this experiment to the data set consisting of past observations of the objective function f(x), thereby expanding the data set and updating the surrogate model;

[0049] S203.4, use the updated proxy model to calculate the acquisition function and select the point x with the largest acquisition function value from the search space. t+1, and use it as the next test configuration;

[0050] S203.5, increase the value of the number of iterations k by 1;

[0051] S203.6, determine whether the number of iterations k is less than the maximum number of trials max_t. If so, return to S203.1 and continue with the subsequent operations. If not, terminate the iterative search.

[0052] S204, saving the final quantization configuration and retraining the quantization pruning model to obtain full performance;

[0053] S205 , testing the energy consumption and accuracy of the retrained quantized pruning model.

[0054] Furthermore, the step 9 includes:

[0055] Step 9.1: Re-optimize the hardware logic of key operators and combine sparse acceleration methods to further improve efficiency by leveraging the sparse characteristics of lightweight model parameters.

[0056] Step 9.2: Configure the reuse factor reasonably. In hls4ml, configure the reuse factor for each layer to set the number of times the multiplier is used when calculating the output value of the layer to balance the contradiction between inference latency and resource consumption.

[0057] Compared with the existing technology, this invention aims to transform a full-precision CNN baseline model into a lightweight model suitable for the low-latency and low-power requirements of the edge through a series of interrelated steps, and realize its efficient deployment on the target hardware. Specific benefits include:

[0058] 1) Significant model lightweighting: L1 regularization in the unstructured pruning method is adopted, and the L1 norm of the weight is added to the loss function as a penalty term. The weights below the threshold are dynamically reset to zero, effectively achieving sparseness of the baseline model parameters, removing redundant connections, and significantly reducing the number of model parameters and computational complexity, creating conditions for subsequent deployment on resource-constrained edge devices.

[0059] 2) Quantization Scheme Optimization: Utilizing a Bayesian optimization algorithm combined with the Horowitz energy estimation method, we search for the optimal mixed-precision quantization scheme, using model accuracy and energy consumption as objective functions. We introduce a tolerance factor (FF) to measure the tolerance of quantization to declines in model accuracy and other metrics, and incorporate this into the objective function to guide the Bayesian optimization search. This minimizes energy consumption while maintaining a certain level of accuracy, thereby improving the efficiency of the model on edge devices.

[0060] 3) Strong hardware adaptability: For different FPGA models, the reuse factor of each layer is calculated based on the configuration file and hardware constraints, combined with the number of parameters in each neural network layer and the number of DSPs in the target FPGA. This is used as the conversion configuration to ensure that the resource utilization rate of the generated HLS project is reasonable (for example, it does not exceed the set threshold q). This fully utilizes the computational parallelism of the FPGA, improves hardware resource utilization, and achieves efficient hardware-model adaptation.

[0061] 4) Full-process automation: Establish a full-process automated deployment solution that includes three stages: lightweighting, model conversion, and comprehensive implementation. Use relevant tool libraries (such as the hls4ml library) to achieve rapid mapping from model to hardware design, reduce manual intervention, improve deployment efficiency, and achieve a one-stop solution from model design to hardware deployment, reducing development costs and cycles.

[0062] 5) Performance assurance: We re-optimize the hardware logic of key operators, combine sparse acceleration methods with reasonable reuse factor configuration, and further improve model inference speed. We also retrain the remaining sparse connections after pruning to effectively restore model performance and ensure the reliability and effectiveness of lightweight models in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a process framework diagram of the present invention.

[0064] Figure 2-Figure 5 The following is a graph showing the distribution of weight parameters under different pruning effects, taking the Conv_0 layer as an example. The vertical axis is in exponential form.

[0065] Figure 2 This is the weight distribution diagram of the unpruned model of the Conv_0 layer, showing the weight distribution of the Conv_0 layer when it is not pruned, with high values ​​on both sides of 0 and low values ​​in the middle.

[0066] Figure 3 This is the weight distribution diagram of the model with 50% pruning of the Conv_0 layer. It shows the weight distribution after 50% pruning of the Conv_0 layer. The zero-valued weight accounts for 50.00%, and the distribution is more sparse.

[0067] Figure 4 This is the weight distribution diagram of the model with 75% pruning of the Conv_0 layer. It shows the weight distribution after 75% pruning of the Conv_0 layer. The zero-value weight accounts for 75.00%, and the sparsity is further improved.

[0068] Figure 5 This is the weight distribution diagram of the Conv_0 layer 90% pruning model, which reflects the weight distribution after 85% pruning of the Conv_0 layer. The zero-value weight accounts for 90.05%, and the distribution is highly sparse.

[0069] Figure 6This is the weight distribution diagram of the Conv_0 layer of the quantized model after pruning, where the vertical axis is in exponential form.

[0070] Figure 7 This is a diagram of the deployment solution based on hls4ml.

[0071] Figure 8 is a schematic diagram of reuse factors. DETAILED DESCRIPTION

[0072] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0073] See also Figures 1-8 , a Bayesian optimization-based neural network lightweight and efficient FPGA deployment method, including:

[0074] Step 1: Select an edge application dataset and a four-layer convolutional architecture as the baseline model. This baseline model specifically consists of four convolutional blocks, fully connected layers, and activation layers. Each convolutional block consists of a convolutional layer, a batch normalization layer, and an activation layer. The fully connected and activation layers are used to learn the features extracted by the convolution and output classification results. Throughout the network, ReLU and Sigmoid activation functions are used.

[0075] Step 2: Use L1 regularization to perform parameter sparsification, which includes:

[0076] In step 2.1, L1 regularization is used in the unstructured pruning method.

[0077] Step 2.2, add the L1 norm of the weight as a penalty term in the loss function, and the modified L1 regularization loss function L λ (W) is:

[0078]

[0079] Among them, L is the original loss function, W is the weight matrix, represents the L1 norm of the weight matrix, that is, the sum of the absolute values ​​of the weights. λ is a hyperparameter of the regularization strength, which is used to weigh the proportion of data fitting and regularization. λ > 0.

[0080] In step 2.3, a dynamic pruning method is used to set the weight connections below the pruning threshold τ to zero during the training process. The remaining connections with non-zero weights form sparse connections, achieving the sparsification of the baseline model parameters.

[0081] Among them, the pruning threshold τ is determined according to the model training situation, and the algorithm The calculated value is the percentile value of the weighted absolute value distribution at the sparsity S(t) percentile. Here, percentile(·) represents the percentile function, which is used to calculate percentiles and returns the value at the corresponding percentile in the dataset. Specifically, given a dataset and a percentage, the percentile function finds the number in the dataset that makes the percentage of data less than or equal to this value. Refers to the weight matrix of the selected benchmark model, W is the set of absolute values ​​of each element, That is, find the value at the sparsity S(t) percentile in the distribution of the absolute value of the weight. This value is the pruning threshold T(t).

[0082] The requirements of the dynamic pruning method are: define the pruning execution interval N t , the number of steps for pruning to start is T i , the number of steps for pruning to end is T e , the target sparsity is S e .

[0083] The goal of the dynamic pruning method is to: i , the sparsity S(t) at step number t should meet the following requirements:

[0084]

[0085]

[0086] The process of dynamic pruning method includes:

[0087] S101, N t Execute the algorithm for the interval.

[0088] S102, if t<T i , then only training without pruning, otherwise go to S103.

[0089] S103, when the number of training steps t satisfies T i ≤t≤T e When , the following operations are performed. Where T i is the starting step number of pruning, T e The number of steps at which pruning ends.

[0090] S104, counting the zero values ​​in the benchmark model parameters, and updating the current sparsity S(t).

[0091] S105, calculate the pruning threshold , that is, the value at the sparsity S(t) percentile in the distribution of the absolute value of the weight.

[0092] S106 , applying sparsification to the weights W according to the threshold, and resetting the weights below the threshold to zero.

[0093] S107, increase the value of the training step number t by 1.

[0094] S108, end the above when T i ≤t≤T e The loop operation performed when

[0095] S109, retrain the remaining sparse connections to restore model performance.

[0096] S110, check whether the sparsity of the current model has reached the target sparsity S e .

[0097] Step 3: After pruning, keep the sparse connection structure unchanged. Retrain the remaining sparse connections to restore the baseline model performance and obtain the quantized pruned model.

[0098] Step 4: Clearly quantize the weights and activations of convolutional and fully connected layers. Layers that do not change the input data type do not need to be quantized.

[0099] Step 5: Select the quantization bit width, including:

[0100] In step 5.1, considering the superiority of power-of-two quantization in terms of hardware efficiency and resource consumption, 4, 8, and 16 bits are selected as the optional quantization bit widths.

[0101] In step 5.2, the maximum quantization bit width for activation values ​​is limited to 16 bits, and the maximum quantization bit width for weights is limited to 8 bits. This step, combined with the previously completed pruning steps, fully implements L1 regularization pruning and quantization configuration settings, resulting in a truly quantized and pruned model. This provides a clear target for subsequent training and optimization using this model in Bayesian optimization.

[0102] Step 6: Use Bayesian optimization to search for the optimal quantization solution, which includes:

[0103] Step 6.1: Use the Bayesian optimization algorithm and the Horowitz energy consumption estimation method to construct the energy consumption objective function. The core formula of the Horowitz energy consumption evaluation method is: ,in, They are the execution frequency, equivalent capacitance, operating voltage, and activity factor (probability of the operation occurring) of each operation, which are refined into , where E compute is the energy consumption of computing operations, Ememory is the energy consumption of storage operation, E data_movement is the energy consumption of data transmission between storage unit and computing unit, E val is the overall energy consumption evaluation value of the deep learning model.

[0104] Step 6.2, for the specific implementation of optimization objectives and constraints, introduce the concept of tolerance factor FF, which is defined as: , where △ acc Indicates the tolerable percentage of accuracy drop, R indicates the energy consumption reduction ratio that the optimized model must achieve compared to the original model, S is an adjustment factor used to reduce the numerical weight of the reference model, thereby forcing the optimizer to select a smaller model, and the parameter C ref and C trial They represent the cost of the reference model and the cost of the quantitative model being tested, respectively. The values ​​of both are derived from the Horowitz energy consumption evaluation method.

[0105] Step 6.3, combined with FF, the objective function f(x) of the automatic quantization search process is determined as: f(x)=C trial +α·L acc FF, where L acc is the accuracy loss function of the normal training process, α is the L acc The weighting factor of .

[0106] In step 6.4, the acquisition function u(x) corresponding to f(x) is improved using expectation, based on the predicted value of the objective function and the current optimal value f best To select the next evaluation point, the expression is:

[0107]

[0108] Where u(x) is represented by EI(x), EI(x) is the expected improvement acquisition function, E[·] represents the mathematical expectation, and max(·) represents the maximum value. The expected improvement acquisition function is used to evaluate the expected degree of improvement of the function value when sampling a point near the current optimal value. If the improvement of the function value is less than the expected value after the Bayesian optimization quantized space search algorithm is run, it means that the current optimal value point may be a local optimal solution. At this time, the Bayesian optimization quantized space search algorithm will turn to other areas within the search domain to find the global optimal solution.

[0109] In step 6.5, an automatic quantization search is performed using a quantization space search algorithm based on Bayesian optimization. Before the iterative search begins, a random configuration is selected for initialization, and a Gaussian process is used to fit the objective function as a proxy model. Then, within a limited number of trials, the quantization pruning model is trained to obtain the prior distribution and update the proxy model, thereby selecting the next trial configuration according to the acquisition function. After this iteration, the results of the automatic quantization are given, and the final quantization model configuration is saved.

[0110] The requirements of the quantization space search algorithm based on Bayesian optimization are: defining the quantization space V, the objective function f(x), and the acquisition function u(x).

[0111] The purpose of the quantitative space search algorithm based on Bayesian optimization is to search for the maximum value of the acquisition function within the number of trials t < the maximum number of trials max_t ,in, is a combination of mathematical symbols used to find the function The value of x when the maximum value is obtained, , represents a data set consisting of t-1 observations of the function f(x), x n is a function A series of values ​​of the independent variable in the data set In, x n The corresponding function value f(x n ) together constitute the observation value, n is the index value from 1 to k-1, each x n Represents the value of the independent variable when observing the objective function f(x) once.

[0112] The process of the quantized space search algorithm based on Bayesian optimization includes:

[0113] S201, randomly select several configurations to test as initialization.

[0114] S202, using a Gaussian process to fit the objective function as a surrogate model.

[0115] S203, iterative search according to the following steps:

[0116] S203.1, train the model based on the current quantization configuration.

[0117] S203.2, update the prior distribution of the objective function f(x), and t ) for sampling, x t It is the data set based on the results of the previous t-1 trials at the t-th iteration search , so that the acquisition function The value of the independent variable x when the maximum value is obtained is used to sample the objective function f(x).

[0118] S203.3, add the results of this experiment to the data set consisting of past observations of the function f(x), expand the data set, and then update the proxy model.

[0119] S203.4, use the updated proxy model to calculate the acquisition function and select the point x with the largest acquisition function value from the search space. t+1 , and use it as the next test configuration.

[0120] S203.5, increase the value of the number of iterations t by 1.

[0121] S203.6, determine whether the number of iterations t is less than the maximum number of trials max_t. If so, return to S203.1 to continue with subsequent operations; if not, terminate the iterative search. S204: Save the final quantization configuration and retrain the quantization pruning model to obtain complete performance.

[0122] S205 , testing the energy consumption and accuracy of the retrained quantized pruning model.

[0123] Step 7: With the help of the hls4ml library, the lightweight quantized pruned model is automatically converted into an HLS project to achieve rapid mapping from the model to the hardware design. Figure 7 This article demonstrates the efficient deployment solution generation process proposed in this paper, which consists of three phases: lightweighting (yellow box), model conversion (blue box), and comprehensive implementation (black box). Aiming to automate the entire process, this solution optimizes engineering configuration for different FPGA models, avoiding resource overload and inefficiency while ensuring model inference accuracy.

[0124] In step 8, for different FPGA models such as xczu3eg, xczu5eg, and xqzu7ev, the reuse factor of each layer is calculated as the conversion configuration based on the configuration file and hardware constraints, the number of parameters of each layer of the quantized pruning model neural network, and the number of DSPs in the target FPGA. This ensures that the resource utilization rate of the generated HLS project is between 70% and 95%, preferably 90%.

[0125] In hls4ml, you can configure the reuse factor of each layer and set the multiplier to calculate the number of times the output value of the layer is used, such as Figure 8(Reuse factor diagram) The diagram simplifies the multiplication-accumulation operation of a neural network, using the formula {a1[0], a1[1]} = {a0[0] · w0 + a0[1] · w1, a0[0] · w3 + a0[1] · w4}, where a and w represent activation and weight, respectively. When the reuse factor is 1, computation is fully parallelized, completing the multiplication in one clock cycle. When the reuse factor is 2, each multiplier is used twice, requiring two clock cycles. When the reuse factor is 4, four clock cycles are required, embodying the "speed-for-area" principle: reusing hardware resources and distributing computational tasks, sacrificing overall latency to reduce resource requirements.

[0126] Step 9: Optimize the hardware logic of key operators, including:

[0127] Step 9.1: Re-optimize the hardware logic of key operators, combining sparse acceleration methods such as sparse matrix multiplication, sparse convolution, and pruned inference acceleration to further improve efficiency by leveraging the sparsity of lightweight model parameters. Sparse acceleration methods aim to leverage the sparsity of data or models to improve computational efficiency and reduce memory usage. They are widely used in scientific computing (such as numerical simulation and quantum chemistry calculations), machine learning (such as neural network model training and inference, recommendation systems), and signal processing (such as image processing and speech recognition).

[0128] Step 9.2: Configure the reuse factor reasonably. In hls4ml, configure the reuse factor for each layer to set the number of times the multiplier is used when calculating the output value of the layer to balance the contradiction between inference latency and resource consumption.

[0129] Step 10: Establish a full-process automated deployment solution with three stages: lightweighting (implemented through automatic quantization search and regularization pruning functions), model conversion (exporting model structure and parameter files with the help of deep learning libraries, and converting them into HLS projects using the hls4ml library), and comprehensive implementation (performing synthesis and layout and routing based on HLS internal configurations, and using the Vivado tool chain to generate usable bitstream files), to achieve a one-stop solution from model design to hardware deployment.

[0130] Example

[0131] According to the above detailed implementation method, the embodiment of the present invention adopts pruning and quantization technology of deep learning models to improve the sparsity and inference efficiency of the model. The implementation focuses on two aspects:

[0132] The first is the implementation of the model pruning strategy. Figure 2-Figure 5It is a diagram of the weight distribution of the model after training is completed under different pruning degrees, and the distribution diagram of the unpruned benchmark model is introduced as a reference. By comparing the performance of the model under different pruning degrees, the pruning degree that achieves higher sparsity while maintaining the model performance is selected. Specifically, the L1 regularization pruning method is adopted in the embodiment, and the pruning degrees are 50%, 75% and 85% respectively. The experimental results show that even at a high pruning degree of 85%, the accuracy of the model can still be maintained at 99.78%, the AUC is 0.9928, and the F1-Score is 0.9718, with a small performance loss.

[0133] Another aspect is the implementation of the model quantization strategy. To prevent excessive precision loss and model training difficulties caused by post-quantization pruning, the automatic quantization strategy of this invention is relatively conservative, with a low tolerance for accuracy loss of only 1%. The quantization configuration is as follows: for the convolutional layers (Conv), the weight bit widths are 8, 4, 8, and 4, respectively, and the activation bit widths are 8, 16, 16, and 16, respectively; for the fully connected layers (FC), the weight bit widths are 16 and 8, respectively, and the activation bit width is 8. The output layer does not participate in the quantization search and is fixed at a 16-bit bit width. Under this optimal configuration, the total estimated inference energy consumption of the quantized model is reduced by 72.49%, while the accuracy and AUC are barely reduced, and the F1-Score is only reduced by 0.96%.

[0134] Combining pruning and quantization strategies, in the embodiment, after performing regularized pruning with 75% sparsity, the above quantization configuration is used for automatic quantization, and the theoretical compression ratio of the model reaches 14.5 times, while still maintaining an accuracy of 99.64% and only reducing the F1-Score by 1.25%. Figure 6 It is the weight value distribution diagram after pruning and automatic quantization. Figure 2-Figure 5 The unquantized parameter distribution in the quantized model is more distinct due to the reduction of the bit width after quantization, and all non-zero weight values ​​near zero are eliminated. The method of the present invention can significantly improve the sparsity and inference efficiency of the model while maintaining the performance of the model.

[0135] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A Bayesian optimization-based neural network lightweight and efficient FPGA deployment method, characterized in that: include: Step 1: Select an edge application dataset and choose a 4-layer convolutional structure as the baseline model; Step 2: Use L1 regularization to perform parameter sparsification; Step 3: After pruning, the sparse connection structure is kept unchanged, and the remaining sparse connections are retrained to restore the baseline model performance to obtain a quantized pruned model. Step 4: Clearly quantize the weights and activations of the convolutional and fully connected layers. Layers that do not change the input data type do not need to be quantized. Step 5, select the quantization bit width; Step 6: Use Bayesian optimization to search for the optimal quantization solution, including: Step 6.1: Use the Bayesian optimization algorithm and the Horowitz energy estimation method to construct an energy consumption objective function. The energy consumption objective function considers the energy consumption of computing operations, storage operations, and data transmission operations in the deep learning model. Step 6.2, introduce the concept of tolerance factor FF, which is defined as: , where △ acc represents the tolerable percentage of accuracy drop, R represents the energy consumption reduction ratio that the optimized model must achieve compared to the original model, S is an adjustment factor used to reduce the numerical weight of the reference model, and the parameter C ref and C trial represent the cost of the reference model and the cost of the quantitative model being tested, respectively; Step 6.3, combined with FF, the objective function f(x) of the automatic quantization search process is determined as: f(x)=C trial +α·L acc FF, where L acc is the accuracy loss function of the normal training process, α is the L acc The weighting factor of Step 6.4, select the next evaluation point using the expected improved acquisition function u(x), which is expressed as: Where u(x) is represented by EI(x), EI(x) is the expected improved acquisition function, E[·] represents the mathematical expectation, and max(·) represents the maximum value; Step 6.5, performing automatic quantitative search using a quantitative space search algorithm based on Bayesian optimization; Step 7: With the help of the hls4ml library, the lightweight quantized pruned model is automatically converted into an HLS project to achieve rapid mapping from the model to the hardware design. Step 8: For different FPGA models, the reuse factor of each layer is calculated based on the configuration file and hardware constraints, the number of parameters in each layer of the quantized pruning model, and the number of DSPs in the target FPGA. This is used as the conversion configuration to ensure that the resource utilization rate of the generated HLS project does not exceed q. The value of q ranges from 70% to 95%. Step 9: Optimize the hardware logic of key operators; Step 10: Establish a full-process automated deployment plan including three stages: lightweighting, model conversion, and comprehensive implementation, to achieve a one-stop solution from model design to hardware deployment.

2. The Bayesian optimization-based neural network lightweight and efficient FPGA deployment method according to claim 1 is characterized in that: The benchmark model described in step 1 includes 4 convolution blocks, a fully connected layer and an activation layer. Each convolution block consists of a convolution layer, a batch normalization layer and an activation layer. The fully connected layer and the activation layer are used to learn the features extracted by convolution and classify the output results. In the entire benchmark model, two activation functions, ReLU and Sigmoid, are used.

3. The method for lightweight and efficient FPGA deployment of a neural network based on Bayesian optimization according to claim 1, characterized in that: The step 2 includes: Step 2.1, use L1 regularization in the unstructured pruning method; Step 2.2, add the L1 norm of the weight as a penalty term in the loss function, and the modified L1 regularization loss function L λ (W) is: Among them, L is the original loss function, W is the weight matrix, represents the L1 norm of the weight matrix, that is, the sum of the absolute values ​​of the weights. λ is a hyperparameter of the regularization strength, which is used to weigh the proportion of data fitting and regularization. λ > 0; In step 2.3, a dynamic pruning method is used to set the weight connections below the pruning threshold τ to zero during the training process, and the remaining connections with non-zero weights form sparse connections to achieve the sparsification of the baseline model parameters.

4. The method for lightweight and efficient FPGA deployment of a neural network based on Bayesian optimization according to claim 3, characterized in that: The step 2.3 includes: Step 2.3.1, pruning is performed with interval N t Execute the algorithm for the period; Step 2.3.2, determine the number of training steps t and the number of pruning start steps T i The size relationship, if t<T i , then only training without pruning; if t≥T i , then enter the dynamic pruning process; Step 2.3.3, at T i ≤t≤T e In the interval, count the zero values ​​in the benchmark model parameters, update the current sparsity S(t), and calculate the pruning threshold , where percentile(·) represents the percentile function, The weight matrix W of the selected benchmark model is a set of the absolute values ​​of each element; sparsification is applied to the weight matrix W according to the pruning threshold, the weights below the threshold are reset to zero, and then t is incremented by 1.

5. The method for lightweight and efficient FPGA deployment of a neural network based on Bayesian optimization according to claim 1, characterized in that: The step 5 comprises: Step 5.1, select 4, 8, or 16 bits as the optional quantization bit width; In step 5.2, the maximum quantization bit width of the activation value is limited to 16 bits, and the maximum quantization bit width of the weight is limited to 8 bits.

6. The method for lightweight and efficient FPGA deployment of a neural network based on Bayesian optimization according to claim 1, characterized in that: The requirements of the quantization space search algorithm based on Bayesian optimization in step 6.5 are as follows: define the quantization space V, the objective function f(x), and the acquisition function u(x); The purpose of the quantitative space search algorithm based on Bayesian optimization is to search for the maximum value of the acquisition function within the number of trials k < the maximum number of trials max_t ,in, is a combination of mathematical symbols used to find the function The value of x when the maximum value is obtained, , represents a data set consisting of k-1 observations of the function f(x), x n is a function A series of values ​​of the independent variable in the data set In, x n The corresponding function value f(x n ) together constitute the observation value, n is the index value from 1 to k-1, each x n Represents the value of the independent variable when observing the objective function f(x); The process of the quantized space search algorithm based on Bayesian optimization includes: S201, randomly select several configurations to test as initialization; S202, using Gaussian process to fit the objective function as a surrogate model; S203, iterative search according to the following steps: S203.1, train the quantization pruning model for the current quantization configuration; S203.2, update the prior distribution of the objective function f(x), and t ) for sampling, x t It is the data set based on the results of the previous t-1 trials at the t-th iteration search , so that the acquisition function The value of the independent variable x when the maximum value is obtained is used to sample the objective function f(x); S203.3, adding the results of this experiment to the data set consisting of past observations of the objective function f(x), thereby expanding the data set and updating the surrogate model; S203.4, use the updated proxy model to calculate the acquisition function and select the point x with the largest acquisition function value from the search space. t+1 , and use it as the next test configuration; S203.5, increase the value of the number of iterations k by 1; S203.6, determine whether the number of iterations k is less than the maximum number of trials max_t. If so, return to S203.1 and continue with the subsequent operations. If not, terminate the iterative search. S204, saving the final quantization configuration and retraining the quantization pruning model to obtain full performance; S205 , testing the energy consumption and accuracy of the retrained quantized pruning model.

7. The method for lightweight and efficient FPGA deployment of a neural network based on Bayesian optimization according to claim 1, characterized in that: The step 9 comprises: Step 9.1: Re-optimize the hardware logic of key operators and combine sparse acceleration methods to further improve efficiency by leveraging the sparse characteristics of lightweight model parameters. Step 9.2: Configure the reuse factor reasonably. In hls4ml, configure the reuse factor for each layer to set the number of times the multiplier is used when calculating the output value of the layer to balance the contradiction between inference latency and resource consumption.

Images