Neural network multi-objective optimization and FPGA hardware acceleration collaborative design method

Abstract

The invention provides a neural network multi-objective optimization and FPGA hardware acceleration collaborative design method, and belongs to the field of deep learning model compression and hardware collaborative design. The method comprises the following steps: constructing a joint optimization space containing a neural network compression parameter and an FPGA hardware design parameter; a multi-target Bayesian optimization search strategy is adopted, iterative search is carried out in the joint optimization space, model precision, FPGA resource occupation and reasoning delay are synchronously optimized, and optimal candidate configuration is obtained; matching the compressed network structure with the FPGA parallel architecture by using a hardware-perceived pruning and quantification strategy; a multi-task performance prediction model is adopted to quickly predict the precision, resource occupation and delay of the optimal candidate configuration so as to accelerate the search process; according to the optimal configuration, a hardware accelerator code facing the target FPGA is automatically generated, and integration and implementation are completed. According to the method, collaborative optimization of neural network compression and hardware design is achieved, FPGA resource occupation can be remarkably reduced, the reasoning speed can be increased, and meanwhile the model precision is kept.

Description

Technical Field

[0001] This invention relates to the field of deep learning model compression and hardware co-design technology, and more specifically, to a method for co-designing neural network multi-objective optimization and FPGA hardware acceleration. Background Technology

[0002] With the rapid development of scenarios such as the Internet of Things, smart manufacturing, and autonomous driving, a massive number of artificial intelligence tasks require the real-time execution of deep neural networks on resource-constrained embedded devices and terminals (such as smart cameras, drone controllers, and mobile terminal accelerator cards). Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and attention mechanism networks (such as Transformers) have demonstrated outstanding performance in tasks such as image recognition, speech recognition, and object detection. However, these network models typically have millions to hundreds of millions of parameters, resulting in enormous computational demands and requiring high-performance GPUs for real-time response. This contradicts the hardware limitations of embedded and edge devices, such as power consumption, heat dissipation, and area constraints.

[0003] To deploy efficient neural networks in constrained hardware environments, researchers have proposed various model compression techniques: network pruning reduces the number of model parameters by removing redundant weights or neurons; quantization reduces storage and accumulation computation overhead by decreasing the representation width of weights and activations; knowledge distillation simplifies the structure by training small models to mimic larger models; and Neural Architecture Search (NAS) automatically searches for lightweight model architectures at the network structure level. While these techniques have achieved success in improving model efficiency, most remain at the algorithmic level, neglecting the specific architecture of the target hardware platform.

[0004] The existing technology has the following main problems: First, the pruning pattern is mismatched with the hardware parallelism. Traditional pruning typically aims to reduce the number of parameters or computational load, without considering how the sparse pattern of the pruned network maps to specific hardware. On parallel accelerators such as FPGAs, computational tasks often need to be vectorized or matrixed to fully utilize DSPs and multiply-accumulate units. When the sparsity resulting from pruning does not match the distribution of hardware resources, even if the number of parameters is reduced, the actual speedup may not improve. For example, arbitrarily pruning certain channels may result in the remaining parameters being logically unpacked for parallel computation, leading to a waste of hardware unit resources.

[0005] Second, uniform quantization ignores inter-layer differences. Existing quantization methods often use a uniform fixed bit width, such as 8-bit fixed-point. However, different layers in a neural network have different sensitivities to precision. The first and last layers are often more sensitive to quantization, while intermediate layers can tolerate more quantization errors. On the hardware side, different bit widths correspond to different resource overheads: increasing the weight bit width linearly increases DSP usage, while increasing the activation bit width increases multiply-accumulate logic and storage overhead. In mixed-precision designs, bit widths should be flexibly allocated based on the precision sensitivity of each layer and the hardware cost to achieve better overall performance.

[0006] Third, algorithm optimization and hardware design are disconnected. Many compression algorithms are designed independently without hardware feedback, while hardware accelerators are often optimized for fixed network structures. This separate design process makes it difficult to simultaneously achieve the optimization goals of low latency, high throughput, and high accuracy. In the traditional workflow, neural network compression engineers focus only on model compression metrics, while FPGA design engineers independently optimize the accelerator structure. Both optimize according to their respective goals, lacking a collaborative feedback mechanism. Some research has attempted to combine model compression with hardware design, such as hardware-aware neural architecture search methods based on reinforcement learning and quantization strategy optimization for specific hardware platforms. However, these methods typically optimize only a single goal or consider only some hardware constraints, failing to achieve comprehensive joint optimization of compression parameters and hardware design parameters. Furthermore, the optimization process often requires extensive practical hardware evaluation, which is time-consuming.

[0007] Therefore, there is an urgent need for a method that can collaboratively consider the characteristics of neural network compression algorithms and target FPGA hardware architecture, jointly optimize algorithm compression strategies and hardware design parameters during the search process, form a closed-loop collaborative optimization process, and thus obtain a solution that achieves the optimal balance under multiple objectives such as accuracy, resource consumption, and latency. Summary of the Invention

[0008] In view of this, this invention proposes a collaborative design method for neural network multi-objective optimization and FPGA hardware acceleration, which unifies neural network compression parameters and FPGA hardware design parameters into a joint optimization space. By using algorithm parameters such as pruning sparsity and quantization bit width, along with hardware parameters such as parallelism and buffer depth, as unified optimization variables, the search process can perceive the mutual influence between the two, avoiding the performance loss caused by the separation of algorithm and hardware optimization in traditional methods. Through key technologies such as constructing a joint optimization space, a multi-objective Bayesian optimization search strategy, a hardware-aware compression strategy, multi-level performance analysis, and automatic accelerator generation, synchronous optimization of neural network compression and hardware design is achieved.

[0009] To achieve the above objectives, this invention proposes a collaborative design method for neural network multi-objective optimization and FPGA hardware acceleration, including neural network compression and FPGA hardware accelerator design, such as... Figure 1 As shown, it includes the following steps: A joint optimization space is constructed, which uses neural network compression parameters and FPGA hardware design parameters as unified optimization variables. The neural network compression parameters include pruning parameters and quantization bit width configuration of each layer, and the FPGA hardware design parameters include computational parallelism parameters and resource configuration parameters. Based on a multi-objective Bayesian optimization search strategy, an iterative search is performed in the joint optimization space to simultaneously optimize model accuracy, FPGA resource consumption, and inference latency to construct optimization objectives and obtain the optimal candidate configuration. During the iterative search process, a multi-task performance prediction model is used to quickly predict the accuracy, resource consumption, and latency of candidate configurations to replace the complete hardware comprehensive evaluation, thereby accelerating the search convergence and finally obtaining the optimal configuration from the Pareto optimal solution set. The iterative search process employs a hardware-aware compression strategy to match the compressed network structure with the parallel computing architecture of the target FPGA. Based on the optimal configuration, hardware accelerator code for the target FPGA is automatically generated, and synthesis and implementation are completed.

[0010] Furthermore, the neural network compression parameters include the pruning sparsity of each layer and the mixed precision quantization bit width; the mixed precision quantization parameters include the total network bit width, the independent bit width allocation of each layer's weights and activations, and the ratio of integer bits to decimal bits in the fixed-point number. The FPGA hardware design parameters include data bus width, computational parallelism multiplexing factor, on-chip memory buffer depth, and pipeline stages.

[0011] Furthermore, the multi-objective Bayesian optimization search strategy uses a surrogate model to predict the performance of candidate configurations and selects subsequent candidate configurations based on the acquisition function. The iterative search process includes: In each iteration, based on the current agent model and existing Pareto front information, a multi-objective acquisition function is used to select the next or a batch of candidate configurations; For each selected candidate configuration, a multi-objective evaluation is performed, including model accuracy evaluation, FPGA resource consumption evaluation, and inference latency evaluation. Based on the evaluation results, it is determined whether the candidate configuration should be added to the Pareto optimal solution set. The iterative process is optimized using an objective function, and the optimal candidate configuration is obtained after the frontier set converges.

[0012] Furthermore, the objective function is as follows:

[0013] in, These are the weighting coefficients for each performance indicator. Acc ( x ) indicates model accuracy. LUT ( x ), DSP ( x ), BRAM ( x This indicates FPGA resource usage. Latency ( x ) indicates a delay in reasoning.

[0014] Furthermore, the iterative search process also includes: A hardware-aware model compression strategy is adopted, which guides the pruning sparsity and quantization bit width allocation of each layer based on the parallel computing and memory access characteristics of the target FPGA, so that the compressed network structure matches the FPGA hardware architecture. Specifically, this includes: Hardware-aware pruning: Based on the sensitivity analysis of the pruning effect on model accuracy at each layer and the requirements of FPGA parallel architecture for data packaging, the pruning sparsity of each layer is determined, and structured pruning mode is preferred. Hardware-aware quantization: Based on the sensitivity analysis of the accuracy of each layer of quantization and the impact of each layer's bit width on different FPGA resource types, appropriate bit widths are allocated to the weights and activations of each layer.

[0015] Furthermore, the iterative search process also includes multi-level performance profiling and search space guidance, specifically including: In the early stages of the search, multiple sets of experiments were performed on each layer of the neural network to compare the accuracy decrease under different pruning rates and quantization bit widths. The accuracy change trend of each layer under different compression intensities was recorded as a sensitivity curve to determine the tolerance of each layer to pruning and quantization, and soft constraints were set for the parameters of each layer. Meanwhile, the dependency relationship between global hardware performance indicators and design parameters is analyzed. Before or in the early stage of the search, different configurations are randomly sampled to obtain FPGA resource usage and latency data. The impact of each hardware parameter on resources and latency is evaluated, and reasonable upper and lower limits are set for hardware parameters, or inefficient areas are removed from the search space.

[0016] Furthermore, the multi-task performance prediction model adopts a hybrid expert network structure, including multiple expert sub-networks and a gating network; The expert subnetwork is used to learn the nonlinear mapping relationship between configuration parameters and various performance indicators. The gating network is used to assign weights to each expert subnetwork according to the input configuration. The total output of the model is the weighted sum of the outputs of each expert subnetwork.

[0017] Furthermore, the training data for the multi-task performance prediction model is obtained through the following methods: Multiple configurations are sampled within the joint optimization space. Model compression training is performed on each configuration to obtain accuracy metrics. FPGA synthesis simulation is then performed on the same configuration to obtain resource usage and latency metrics, thus forming a training sample set.

[0018] Furthermore, the process of generating the hardware accelerator code includes: Export the compressed network model corresponding to the optimal configuration as an intermediate representation format; The intermediate representation is parsed using a high-level synthesis tool, and hardware description code containing a specified degree of parallelism is automatically generated, wherein each layer of operation adopts a mixed-precision fixed-point arithmetic module corresponding to the quantization scheme. Configure inter-layer data flow interfaces to establish pipeline data transmission channels; The FPGA vendor's toolchain is used to complete synthesis, placement and routing, and generate bitstream files.

[0019] On the other hand, to achieve the above objectives, the present invention provides a collaborative design system for neural network multi-objective optimization and FPGA hardware acceleration, comprising: A joint optimization space construction module is used to integrate neural network compression parameters and FPGA hardware design parameters into the joint optimization space. A multi-objective optimization search module is used to perform iterative search within the joint optimization space based on a multi-objective Bayesian optimization strategy. The multi-task performance prediction module is used to quickly predict the accuracy, resource consumption, and latency of candidate configurations. The accelerator generation module is used to automatically generate hardware accelerator code for the target FPGA based on the optimal configuration. Compared with the prior art, the beneficial effects of the present invention are as follows: (1) Multi-objective collaborative optimization effect: Through joint optimization space, the accuracy, resources and latency are optimized synchronously. Compared with the traditional separate method, DSP resources are significantly reduced (e.g., more than 50%), BRAM resources are reduced (e.g., more than 50%), inference latency is reduced by more than two orders of magnitude, and accuracy loss is controlled within 1%.

[0020] (2) Efficient use of hardware resources: Hardware-aware pruning and quantization strategies make the compression model highly compatible with the FPGA architecture, greatly improving the utilization rate of computing units (e.g., more than 25%), and eliminating the resource idle problem in traditional methods.

[0021] (3) Significantly improved optimization efficiency: The multi-task performance prediction model replaces the actual hardware comprehensive evaluation, reducing the single configuration evaluation time from several hours to seconds, and shortening the overall optimization cycle by several times.

[0022] (4) End-to-end automation: The entire process from model compression to FPGA accelerator generation is automated, eliminating the need for manual parameter adjustment, thus reducing the development threshold and engineering cycle. Attached Figure Description

[0023] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings: Figure 1 This is a flowchart illustrating the neural network and FPGA collaborative optimization method of the present invention. Figure 2 This is a schematic diagram of the network structure of the multi-task performance prediction model used in this invention; Figure 3 This is a schematic diagram of the hybrid expert network model in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the hardware implementation and deployment in an embodiment of the present invention; Figure 5 This is a diagram showing the deployment result of the FPGA in an embodiment of the present invention. Detailed Implementation

[0024] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] This embodiment proposes a collaborative design method for neural network multi-objective optimization and FPGA hardware acceleration, such as... Figure 1 As shown, it includes the following steps: Joint optimization design space construction: By unifying the neural network compression algorithm parameters and FPGA hardware design parameters into the optimized joint space, synchronous adjustment of the algorithm and hardware can be achieved.

[0026] The neural network parameters include the pruning sparsity of each convolutional or fully connected layer (which can be the pruning ratio of unstructured weights or structured filters / channels), and mixed-precision quantization parameters (such as the total bit width of the network, the bit width allocation of each layer or activation / weight, and the ratio of integer to decimal places in the fixed-point number). These parameters determine the network structure, weight data type, and computation format of the compressed model.

[0027] FPGA hardware parameters include adjustable parameters used when designing an accelerator, such as the data bus width used in the High-Level Synthesizer (HLS), the fixed-point integer bit allocation ratio, the reuse factor (representing the number of multiplexed multiply-accumulate units instantiated in the same layer), the size of on-chip memory buffers (such as inter-layer FIFOs and BRAM buffer depths), and the pipeline depth. These parameters control the parallelism and resource allocation of the accelerator.

[0028] By treating the aforementioned algorithm and hardware parameters as overall optimization variables, the method simultaneously determines the sparsity, quantization bit width, and FPGA design selection for each network layer during the search process. In a single joint search, the proportion to be pruned for multiple convolutional layers is specified, the quantization bit width of the layer's weights is set, and the multiply-accumulate unit multiplexing factor and data channel width of the layer on the FPGA are determined accordingly. This ensures that the pruned and quantized network is precisely allocated to an appropriate number of DSPs and lines in the FPGA design. In this way, the final compressed model and accelerator implementation maintain intrinsic consistency, avoiding the contradiction of traditional methods struggling to efficiently map hardware after compression.

[0029] Multi-objective Bayesian optimization search strategy: Model accuracy, FPGA resource utilization (LUT, DSP, BRAM, registers, etc.), and inference latency are optimized simultaneously as multiple objectives. Multi-Objective Bayesian Optimization is used to progressively search for the optimal configuration. Specifically: Surrogate Model: Within the Bayesian optimization framework, a surrogate model, such as Gaussian process regression or tree-structured Parzen estimation, is constructed for the objective function to approximate the complex functional relationship between configuration and performance metrics (including validation accuracy, resource usage, and latency). The surrogate model can be updated round by round, continuously learning from existing data, thereby predicting the possible performance of unknown configurations.

[0030] Acquisition Function: In each iteration, based on the current surrogate model and existing Pareto front information, a multi-objective acquisition function (such as the desired hypervolume improvement of EHVI) is used to select the next or a batch of candidate configurations. This acquisition function balances the need to explore unexplored areas with the need to utilize the current best region to find configurations that may improve the Pareto front.

[0031] Performance evaluation: For each selected candidate configuration, perform the following process: Accuracy evaluation: Apply the pruning and mixed precision quantization strategy of this configuration to the target network on a neural network training platform (such as PyTorch, TensorFlow, etc.) for quantization-aware fine-tuning training. After the model converges to a stable state, measure the prediction accuracy Acc(x) on the validation set.

[0032] Resource and latency estimation: Input the same configuration into a pre-trained multi-task performance prediction model (e.g.) Figure 2 As shown, the predicted values ​​for FPGA resource usage (LUT(x), DSP(x), BRAM(x), FF(x), etc.) and inference latency (x) corresponding to this configuration are quickly output. To ensure prediction accuracy, a real FPGA synthesis simulation is performed on some key configurations to calibrate and update the prediction model.

[0033] Pareto Front Update: Combining measured or predicted metrics, determine whether a candidate configuration should be added to the Pareto optimal solution set: If, for any existing solution in the current Pareto set, the new configuration is not inferior to that existing solution in all objective metrics, and is strictly superior in at least one objective, then the new configuration dominates the existing solution, and the original solution should be removed and the new configuration added to the front; otherwise, if it is dominated by an existing solution or has no advantage, the configuration is discarded. Through multiple iterations, the front set converges to the optimal trade-off between accuracy, resources, and latency.

[0034] Objective function definition: In practical optimization, a weighted comprehensive objective function can also be used to simplify the search:

[0035] in These are the weighting coefficients for each performance indicator, which can be dynamically adjusted according to the application scenario. This represents the upper limit of hardware resource specifications. The composite objective L(x) can be directly fitted using a Gaussian process or other surrogate model to guide the next round of data acquisition.

[0036] Hardware-aware model compression strategies: Hardware design feedback is introduced during pruning and quantization to ensure that the compression strategy is guided by actual hardware acceleration efficiency. Specific implementations include: Hardware-Aware Pruning: This involves analyzing the parallel computing resources and memory access characteristics of the target FPGA to guide optimization of the pruning strategy. First, sensitivity tests are conducted on different pruning rates for each layer of the network: the network is trained and evaluated under various conditions including no pruning, moderate pruning, and high-intensity pruning, and the impact curve of each layer's pruning rate on overall accuracy is statistically analyzed. This determines accuracy sensitivity: the pruning intensity is limited for layers with high accuracy sensitivity (layers prone to accuracy degradation), while the pruning ratio is increased for layers with low accuracy sensitivity. For example, if pruning layers 2 and 3 leads to a significant decrease in accuracy, higher pruning ratios are prioritized for less sensitive layers such as layers 1 and 4. Furthermore, based on the FPGA's parallel architecture, pruning prioritizes preserving weight patterns that facilitate parallel computation, allowing the remaining parameters to be packaged into the hardware multiply-accumulate array. This results in structured sparsity after pruning, making it easier for HLS tools to map into parallel computing units. For example, weights can be pruned according to the original convolutional kernel block size, ensuring that the unpruned portion corresponds precisely to the width of the hardware data path.

[0037] Hardware-Aware Quantization: This method employs mixed-precision quantization, allocating bit widths based on the precision requirements of different layers and their resource overhead on the FPGA. Typically, the first and output layers of the network are more sensitive to quantization, thus requiring higher bit widths (e.g., 12-16 bits); while intermediate layers are more tolerant and can use lower bit widths (e.g., 3-8 bits). Hardware resources are also considered: increasing weight bit width significantly increases DSP usage, while increasing activation bit width consumes more BRAM and multiplier logic. Therefore, within the allowable precision range, layers with a significant increase in resource consumption (such as those with the largest activation feature map size or the largest weights) are prioritized for lower bit widths to save on device BRAM and LUTs. The specific steps are as follows: For a given pruned network, multiple quantization experiments are conducted, and a hardware simulation program is used to evaluate the impact of different bit width configurations on FPGA resources and performance. The bit width configuration scheme is then selected based on the precision changes. During the search process, this hardware feedback mechanism can dynamically adjust the bit width of certain layers. For example, if the BRAM of a certain layer is nearly full during hardware evaluation, the bit width of that layer is reduced; conversely, a higher bit width can be tried to improve precision. After implementing these strategies, the resulting compressed model not only reduces the number of parameters and computational cost, but also closely matches the FPGA parallel computing architecture, enabling the compressed network to run with lower latency on actual hardware.

[0038] Multi-level performance profiling and search space guidance: During the optimization process, hierarchical profiling is used to guide the dynamic adjustment of the search space, improving search efficiency and avoiding invalid regions. The methods include: Layer-level sensitivity analysis: In the initial search phase, multiple trials are performed on each layer of the neural network to compare the accuracy degradation under different pruning rates and quantization bit widths. The accuracy change trend of each layer under different compression intensities is recorded as a sensitivity curve. This allows for the differentiation between accuracy-sensitive and tolerance layers, setting soft constraints on pruning rates and minimum bit widths for subsequent searches. For example, if it is found that the accuracy of layer 3 drops sharply when the pruning rate reaches 50%, while layer 7 can remain relatively stable even with 70% pruning, then the pruning upper limit of layer 3 is strictly controlled during optimization, while higher sparsity is allowed for layer 7. Similarly, if the accuracy drops sharply when the bit width of a certain layer decreases to 6 bits, its minimum bit width can be set to 8 bits or higher.

[0039] Hardware Impact Analysis: Simultaneously, the dependency relationship between global hardware performance indicators and design parameters is analyzed. Before or in the early stages of the search, different configurations are randomly sampled to obtain FPGA resource usage and latency data, evaluating the impact of various hardware parameters (such as linewidth, parallelism, FIFO depth, etc.) on resources and latency. For example, the variation patterns of BRAM and DSP usage under different combinations of total bit width and multiplexing factor are statistically analyzed. Based on this, reasonable upper and lower limits can be set for design parameters: if high bit width (>12 bits) significantly increases BRAM consumption, it is penalized or some regions are directly removed from the search space; if excessive parallelism leads to a decrease in clock frequency, the upper limit of parallelism can be limited. This multi-level analysis eliminates redundant or costly regions from the search, allowing Bayesian optimization to focus on exploring parameter combinations that are truly likely to yield optimal solutions, shortening convergence time, and improving the quality of the final solution.

[0040] Multi-task performance prediction model: To avoid performing a complete FPGA synthesis simulation in each iteration, this invention constructs and trains a multi-task regression model capable of simultaneously predicting accuracy, resources, and latency. The training process is as follows: Data Acquisition and Labeling: A certain number of configuration samples are first randomly or strategically sampled within the joint search space. For each configuration, pruning and quantization training are performed to obtain a compressed neural network model, and the accuracy (Acc) is verified. Then, the same configuration is applied to the target FPGA for synthesis and simulation, recording inference latency (Latency) and the usage of various resources (LUT, DSP, BRAM, FF, etc.). These constitute the training set, where the input features of the training samples are configuration parameters (vectors including pruning rate, layer bit width, hardware parallelism, cache depth, etc.), and the output labels are the corresponding performance metric vectors (Acc, Latency, LUT, DSP, BRAM, FF). Model Structure: This invention employs a Mixture-of-Experts (MoE) multi-task network as the surrogate model, consisting of several expert sub-networks and a gating network. Each expert sub-network is a multi-layer fully connected neural network used to independently fit the nonlinear relationship between configuration and performance metrics; the gating network predicts the weight allocation of each expert based on the input configuration vector (using Softmax to output normalized weights). The total output is the weighted sum of all expert predictions. The structure and working principle are illustrated below. Figure 3 As shown, this design allows different experts to specialize in different configuration spaces or performance tasks, improving overall prediction accuracy. For example, one expert network might focus on BRAM prediction in high-parallelism scenarios, while another expert might focus on latency prediction in highly sparse models.

[0041] Training Method: The collected data is divided into a training set and a validation set (e.g., 5:1). Input features and output metrics are normalized to balance magnitude differences. A multi-task loss function is defined as the sum of the mean squared errors of each prediction metric. Different weights can be assigned to different tasks to balance the learning of various metrics (e.g., the weights for accuracy error and resource error are manually set). The backpropagation algorithm is used to train the expert network and gated network parameters simultaneously until the validation error converges. After training, the model's prediction performance on an independent test set is evaluated to ensure that the average prediction error of all metrics is within an acceptable range. If the prediction accuracy is insufficient, the number of experts or network layers can be increased or decreased for retraining. The trained MoE model can provide performance predictions for new candidate configurations in just a few milliseconds during actual optimization loops. Its training speed and prediction accuracy have been experimentally proven to significantly improve optimization efficiency. To maintain the long-term accuracy of the model, this embodiment periodically performs real FPGA synthesis on the best configurations in the Pareto front, and feeds the results back to fine-tune the MoE model online, achieving closed-loop alignment between the prediction model and the actual evaluation.

[0042] Accelerator auto-generation and backend optimization: For the final selected target configuration, this invention provides an automatic conversion process from the compressed model to the FPGA accelerator implementation: like Figure 4As shown, firstly, based on the optimal network structure and quantization scheme, the compressed neural network model (e.g., ONNX format exported from PyTorch) is input into a high-level synthesis tool (e.g., HLS4ML, Vitis AI, or a self-developed converter). The tool automatically parses the network structure, mapping each layer's operations to corresponding C++ / HLS descriptions. In the generated HLS code, each layer's convolutional or fully connected operation uses a specified mixed-precision fixed-point multiply-accumulate module; the reuse factor parameter determines how many sets of parallel arithmetic units are instantiated in each layer; inputs and outputs are connected using an AXI-Stream pipeline to establish a pipelined data flow between layers.

[0043] Then, the HLS code is synthesized and implemented using the FPGA vendor's synthesis tools (such as Xilinx Vivado / Vitis): after HLS is compiled into RTL, placement is performed in Vivado to generate the final bitstream. Relevant data is automatically extracted from the report after synthesis: clock frequency, resource usage, and component delays, etc. The design is then compared to the preset targets to observe whether it meets the requirements.

[0044] For identified bottlenecks, post-optimization is performed: for example, during functional or timing simulations, the maximum parallelism of the input / output FIFOs of each layer is monitored, and the required FIFO depth is calculated. Then, the FIFO size is adjusted appropriately based on peak demand to ensure data throughput without wasting BRAM resources. If timing constraints occur at a certain layer, the reuse factor of that layer can be increased or a new pipeline stage can be added to reduce latency. If resources are abundant, the parallelism can be reduced to save logic. After comprehensive optimization, the final FPGA accelerator module can work collaboratively with the embedded processor: through the AXI-DMA interface, the embedded CPU loads neural network parameters and input data into external memory, then calls the accelerator module to perform inference and receives the output results, forming an end-to-end automated inference process. The entire automated toolchain requires no manual intervention, significantly shortening the time from network design to hardware deployment.

[0045] Example 2 This embodiment presents the specific application steps of the above method on a certain FPGA platform: Joint search space definition and initialization.

[0046] Taking the deployment of a 10-layer convolutional neural network on an FPGA platform as an example, the joint optimization parameter space is first defined: for each convolutional or fully connected layer in the network, a continuously adjustable variable of pruning rate [0, 0.8] is introduced; the overall quantization bit width of the network is taken as an integer of [3, 16] bits; the fixed-point integer bit length is taken as a ratio of 25% to 75%; the hardware parameters include the data bus bit width set in HLS (consistent with the total quantization bit width of the network), the ratio of integer bit width (as above), the convolution parallel multiplexing factor of [1, 32], and the inter-layer FIFO depth of 1, 2, 4, 8, etc.

[0047] Before the search begins, a preliminary analysis of the space is performed: Several configurations (e.g., 200 groups) are randomly sampled within the defined space. For each configuration, the following operations are performed: (a) Pruning and quantization-aware fine-tuning (e.g., 10-20 epochs) are performed in a deep learning framework according to this configuration to obtain a compressed network and evaluate its accuracy on the validation set. (b) Perform HLS synthesis on the FPGA hardware to obtain the resource usage and theoretical latency corresponding to this configuration.

[0048] Sensitivity analysis of each layer was conducted by statistically analyzing all sample data: Layers 2 and 3 were observed to be most sensitive to pruning, with accuracy dropping sharply when the pruning rate exceeded 50%; while layer 5 was relatively insensitive, achieving 80% pruning with minimal accuracy loss. Quantization revealed a sharp increase in accuracy loss when the bit width of a layer fell below 6 bits. Based on these analyses, the search space could be constrained: the minimum bit width of layers 2 and 3 was constrained to 8 bits, and the pruning upper limit for the less accuracy-sensitive layer 5 was set to 0.6. Simultaneously, based on the FPGA resource mapping, the overall maximum parallelism could be limited to the on-chip multiplication and addition resource limit. In this way, the search space range was pre-adjusted through multi-level profiling, saving computational resources for subsequent searches.

[0049] Multi-objective Bayesian optimization iterative search. After spatial initialization, an iterative search is performed using a multi-objective Bayesian optimization framework. In the initial stage, a set of random configurations (e.g., 20 sets) is selected, and the accuracy and hardware performance of each configuration are evaluated according to Example 1 to initialize the surrogate model and Pareto front set. Then, an iterative loop begins: in each round of Bayesian optimization, a batch (e.g., 5 sets) of candidate configurations are selected based on the current surrogate model and the Hypervolume Improvement (EHVI) criterion. For each candidate configuration: (a) Perform pruning and mixed precision quantization training in a neural network framework to obtain a compressed model and measure the validation accuracy Acc(x); (b) Use the trained MoE prediction model to quickly predict its resources (LUT(x), DSP(x), BRAM(x), FF(x)) and latency (x); (c) Determine whether to update the Pareto front based on Acc(x) and the prediction results: If the configuration is no worse than the existing solution in all objectives and is better than the existing solution in at least one metric, then include it in the front and remove the dominated old solution; otherwise, discard the configuration.

[0050] If the surrogate predicts that a promising candidate may become the frontier, it can be synthesized on a real FPGA to obtain accurate data, which is then used to update the surrogate model. After several iterations, the frontier set gradually stabilizes and converges to the optimal boundary between accuracy and efficiency. This ultimately yields several compromise solutions: such as those prioritizing minimum resources, those prioritizing minimum latency, and those balancing these factors, for different application scenarios.

[0051] Training a multi-task performance prediction model: Partial configuration data was obtained in the pre-analysis of the joint search space definition and initialization steps. This step further expands the training set: using a total of 500 configurations and their corresponding performance metrics sampled within the joint space as training samples. A MoE model with 6 expert networks is constructed: each expert is a two-layer fully connected network (the number of input nodes is similar to the feature dimension, the number of hidden layer nodes is 100-200, and the output is equal to the number of tasks (accuracy 1D + resource 4D + latency 1D = 6D)). The gating network is a one-layer fully connected network plus a Softmax layer. After normalizing the input features (pruning rate, bit width, parallelism, etc. of each layer) and output metrics, the model is trained using the multi-task mean squared error as the loss function, and trained for 200 epochs using the Adam optimization algorithm. Training results show that on the independent test set, the model's mean absolute error is less than 5%, and the correlation coefficients of each task metric are all above 0.95, indicating high prediction accuracy. In the subsequent optimization framework, the performance prediction time for new configurations is less than 10 milliseconds, greatly accelerating the search efficiency.

[0052] Automatic generation and implementation of hardware accelerators: by Figure 5For example, a representative Pareto-optimal solution obtained in the aforementioned iterations (e.g., global pruning rate of 37.3%, total quantization width of 11 bits, fixed-point integer bit ratio of 30%, parallel reuse factor of 27, and inter-layer FIFO depth of 4) is selected for detailed implementation. First, the pruned quantization network model is obtained in PyTorch and exported as ONNX format. Then, the ONNX model is converted into C++ / HLS code using the HLS4ML tool: each layer in the HLS code defines a corresponding fixed-point multiply-accumulate function, and 27 sets of multiply-accumulate units are instantiated according to the parallel reuse factor. Each layer is connected through the AXI-Stream interface to implement a C++-level pipelined network. The generated HLS code is imported into Xilinx Vitis for synthesis. First, HLS generates the RTL, then placement and routing are performed in Vivado, finally generating the FPGA bitstream. The synthesis report shows that this design uses approximately 30,000 LUTs, 51 DSPs, and 26 BRAMs, with a clock speed of up to 100MHz. Next, the inter-layer FIFO usage was monitored during functional simulation, and the maximum parallel depth of the data stream at each layer was measured. Based on the peak data, the FIFO depth of each layer was adjusted to the required peak value plus a safety margin. After resynthesis, the BRAM usage was reduced to 11 memory blocks, with slight changes to other resources, and the timing met the requirements. Finally, the generated bitstream was deployed to a Zynq-7020 FPGA development board and worked in conjunction with the ARM embedded processor and DDR external memory. The embedded CPU controlled the accelerator, transmitted input data, and read output data through the AXI interface. Experimental results showed that the inference accuracy of the FPGA accelerator on the verification image set was consistent with the original model; the inference throughput was improved by approximately 12% compared to the unoptimized reference model; resource usage matched the prediction model; and power consumption was consistent with the prediction estimate. These results further demonstrate the effectiveness and accuracy of the method of this invention.

[0053] Extended applications across different networks and platforms: To verify the versatility of this method, it was also tested on other neural network architectures and hardware platforms. In a set of controlled experiments, the optimization process was applied to a residual network (ResNet-18) deployed on a larger FPGA development board. The results show that, while maintaining accuracy (Top-1 accuracy decrease <1%), the method of this invention reduces FPGA resource usage by 30%–60% and inference latency by more than 15%, significantly outperforming schemes that only perform pruning or quantization. Furthermore, on hardware platforms supporting heterogeneous computing (including FPGAs with ARM cores), the multi-objective optimization framework of this method can also take into account the resource allocation of various computing units, improving overall efficiency.

[0054] In summary, this invention provides an automated design method for co-optimization of neural networks and FPGA hardware. Through joint search of algorithms and hardware parameters, multi-objective Bayesian optimization, hardware-aware pruning quantization, performance prediction, and accelerator generation, it effectively achieves the goal of efficient deep learning inference on embedded platforms.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method of neural network multi-objective optimization and FPGA hardware acceleration co-design, comprising neural network compression and FPGA hardware accelerator design, characterized in that, The method comprises the following steps: a joint optimization space is constructed, the joint optimization space taking neural network compression parameters and FPGA hardware design parameters as unified optimization variables, wherein the neural network compression parameters comprise pruning parameters and quantization bit width configurations of each layer, and the FPGA hardware design parameters comprise operation parallelism parameters and resource configuration parameters; based on a multi-objective Bayesian optimization search strategy, an iterative search is performed in the joint optimization space to simultaneously optimize model accuracy, FPGA resource occupation and inference delay to build optimization objectives, and an optimal candidate configuration is obtained; in the iterative search process, a multi-task performance prediction model is used to quickly predict the accuracy, resource occupation and delay of the candidate configuration to replace complete hardware comprehensive evaluation, thereby accelerating search convergence, and finally an optimal configuration is obtained from a Pareto optimal solution set; the iterative search process uses a hardware-aware compression strategy to make the compressed network structure match the parallel computing architecture of the target FPGA; based on the optimal configuration, a hardware accelerator code for the target FPGA is automatically generated, and synthesis and implementation are completed.

2. The method of claim 1, wherein, The neural network compression parameters comprise pruning sparsity and mixed precision quantization bit width of each layer; the mixed precision quantization parameter comprises a total bit width of the network, independent bit width allocation of each layer weight and activation, and integer bit and decimal bit proportion of a fixed-point number; the FPGA hardware design parameters comprise a data bus bit width, an operation parallelism reuse factor, an on-chip storage buffer depth and a pipeline level.

3. The method of claim 1, wherein, The multi-objective Bayesian optimization search strategy uses a surrogate model to predict the performance of the candidate configuration, and selects a subsequent candidate configuration based on an acquisition function; the iterative search process comprises: in each iteration, based on the current surrogate model and existing Pareto front information, a multi-objective acquisition function is used to select a next or a batch of candidate configurations; each selected candidate configuration is subjected to multi-objective evaluation, including model accuracy evaluation, FPGA resource occupation evaluation and inference delay evaluation, and whether the candidate configuration should be added to the Pareto optimal solution set is determined according to the evaluation results; a target function is used to optimize the iterative process, and after the front set converges, the optimal candidate configuration is obtained.

4. The method of claim 3, wherein, The target function is as follows: ， wherein, is a weight coefficient of each performance index, Acc ( x ) represents model accuracy, LUT ( x ), DSP ( x ), BRAM ( x ) represents FPGA resource occupation, Latency ( x ) represents inference delay.

5. The method of claim 3, wherein, The iterative search process further comprises: a hardware-aware model compression strategy is used to guide pruning sparsity and quantization bit width allocation of each layer according to parallel computing and storage access characteristics of the target FPGA, so that the compressed network structure matches the FPGA hardware architecture, specifically comprising: hardware-aware pruning: based on sensitivity analysis of each layer pruning on model accuracy and requirements of the FPGA parallel architecture on data packaging, pruning sparsity of each layer is determined, and a structured pruning mode is preferentially used; hardware-aware quantization: based on sensitivity analysis of each layer quantization on accuracy and influence of bit width of each layer on different resource types of the FPGA, appropriate bit width is allocated to each layer weight and activation.

6. The method of claim 3, wherein, The iterative search process further comprises multi-level performance profiling and search space guidance, specifically comprising: In the initial search, multiple sets of experiments are performed on each layer of the neural network, and the accuracy reduction under different pruning rates and quantization bit widths is compared. The accuracy change trend of each layer under different compression intensity is recorded as the sensitivity curve, and the tolerance of each layer to pruning and quantization is determined to set soft constraints for each layer parameter; At the same time, the dependence of global hardware performance indicators and design parameters is analyzed. Before or at the beginning of the search, different configurations are randomly sampled to obtain FPGA resource usage and delay data, and the influence of each hardware parameter on resource and time delay is evaluated. Reasonable upper and lower limits are set for hardware parameters, or inefficient areas are removed from the search space.

7. The method of claim 1, wherein, The multi-task performance prediction model adopts a hybrid expert network structure, including multiple expert sub-networks and a gating network. The expert sub-network is used to learn the nonlinear mapping relationship between configuration parameters and performance indicators, and the gating network is used to assign weights to each expert sub-network according to the input configuration. The total output of the model is the weighted sum of the outputs of each expert sub-network.

8. The method of claim 1, wherein, The training data of the multi-task performance prediction model is obtained by the following method: Sample multiple configurations in the joint optimization space, perform model compression training for each configuration, obtain the accuracy indicator, and perform FPGA synthesis simulation for the same configuration to obtain the resource occupation and delay indicator, which constitutes the training sample set.

9. The method of claim 1, wherein, The generation process of the hardware accelerator code includes: Export the compressed network model corresponding to the optimal configuration into an intermediate representation format; Parse the intermediate representation through a high-level synthesis tool to automatically generate hardware description code containing a specified parallelism, where each layer operation uses a mixed precision fixed-point operation module corresponding to the quantization scheme; Configure the inter-layer data flow interface to establish a pipeline data transmission channel; Use the FPGA vendor tool chain to complete synthesis, placement and routing, and generate a bitstream file.

10. A neural network multi-objective optimization and FPGA hardware acceleration co-design system, characterized in that, It includes: A joint optimization space construction module for integrating neural network compression parameters and FPGA hardware design parameters into a joint optimization space; A multi-objective optimization search module for iterative search in the joint optimization space based on a multi-objective Bayesian optimization strategy; A multi-task performance prediction module for quickly predicting the accuracy, resource occupation and delay of the candidate configuration; An accelerator generation module for automatically generating hardware accelerator code for the target FPGA according to the optimal configuration.

Images