An Adaptive Reinforcement Learning-Driven Accelerator Multi-Objective Optimization Method

Abstract

This invention discloses an adaptive reinforcement learning-driven multi-objective optimization method for accelerators, belonging to the technical field of accelerator optimization design. First, it analyzes the mechanism of action of key design variables such as convolution loop unrolling factor, quantization accuracy, and cache partitioning, establishes a system-level state modeling method, and designs the DDPG framework suitable for continuous action space optimization as a learner for the configuration strategy. Simultaneously, a lightweight searcher is implemented by running Python scripts on the CPU to search and allocate the block parameters of Loop-3 and Loop-4 in real time, thereby reducing the computational burden on the reinforcement learning agent. Finally, an adaptive configuration method driven by reinforcement learning and a corresponding search process are proposed to achieve joint optimization of unrolling scale, quantization accuracy, and block parameters.

Description

Technical Field

[0001] This invention belongs to the technical field of accelerator optimization design, and relates to an adaptive reinforcement learning-driven multi-objective optimization method for accelerators. Specifically, it is a method for achieving efficient parameter configuration search and hardware mapping of convolution operation accelerators through the collaborative implementation of iterative optimization techniques and an adaptive reinforcement learning-driven multi-objective optimization framework. Background Technology

[0002] In recent years, FPGA (Field-Programmable Gate Array)-based Convolutional Neural Network (CNN) accelerators have demonstrated broad application potential in fields such as image recognition, object detection, and edge computing due to their excellent energy efficiency, flexible reconfigurability, and adaptability to diverse task scenarios. Especially in embedded and low-power devices, FPGAs provide a solution that balances performance and energy efficiency for model deployment.

[0003] Convolution is a core operator in deep learning used for image and feature processing. It mainly extracts spatial features by performing local computations on the input feature map through a sliding convolution kernel. Figure 1 This demonstrates the basic process of convolution operations. Convolution operations involve computation in six dimensions: the number of channels in the input feature map (N...). if ), the number of channels in the output feature map (N) of ), height of the output feature map (N) iy ), the width of the input feature map (N) ix ), height of the convolution kernel (N) ky ), kernel width (N) kx ).

[0004] In practical implementations, convolution operations are typically performed using four nested loops, named Loop-1, Loop-2, Loop-3, and Loop-4. The horizontal dimensions (height and width) of the input image are processed by Loop-3, corresponding to the sliding window operation of the convolution kernel on the input image. The horizontal dimensions (height and width) of the convolution kernel are processed by Loop-1, representing the multiply-accumulate (MAC) operation of the convolution kernel in a single channel. The number of input channels and output channels are handled by two independent nested loops, Loop-2 and Loop-4, respectively, because each output channel requires traversing all input channels. In this way, the original six-dimensional convolution calculation is effectively simplified to four nested loops, improving computational efficiency and optimizing hardware resource utilization.

[0005] Currently, deploying large-scale deep neural networks (such as VGG and ResNet) on FPGA platforms still faces many challenges, mainly including limited computing resources, off-chip communication bandwidth bottlenecks, and a large space for configuring deployment parameters. These factors collectively lead to the practical dilemma of low deployment efficiency, poor hardware adaptability, and a large workload for manual parameter tuning. To address these challenges, researchers widely employ loop optimization techniques to efficiently map the convolution operation process. Commonly used methods include loop unrolling, loop tiling, and loop interchange. Among them, loop unrolling improves computational throughput by increasing parallel processing units, loop tiling divides feature maps and weights into blocks to adapt to on-chip cache and reduce off-chip access, and loop interchange optimizes bandwidth usage and data reuse strategies by adjusting the computation order.

[0006] However, current research still has significant shortcomings in loop optimization. For example, most designs only focus on inner nested loops (such as Loop-1 and Loop-2), ignoring the differences in output space dimensions (Loop-3 and Loop-4) between convolutional layers and the impact of uneven resource usage. This directly limits the adaptability and scalability of accelerators across network layers. Furthermore, on resource-constrained FPGA platforms, fixing the parallelism configuration to the outer loop can easily lead to wasted on-chip resources and runtime bottlenecks, especially when processing 1×1 convolutions or residual structures.

[0007] Current research attempts to optimize performance on specific convolutional layers through data reuse mechanisms and automated structure search tools. For example, some accelerators customized for 5×5 convolutional kernels lead to resource waste when dealing with 1×1 convolutional layers; similarly, designs that concentrate all parallelism on Loop-4 suffer from efficiency degradation when dealing with shallow networks with small spatial receptive fields. Therefore, another key challenge in CNN deployment is the high spatial dimension and strong coupling of parameter configuration. Hardware implementation parameters (such as P...) if P of K b F bComplex interactions exist between CNNs (such as CNNs, etc.), making it difficult for traditional manual experience methods or static search algorithms (such as greedy search and genetic algorithms) to achieve the globally optimal configuration within a reasonable timeframe. To address this issue, reinforcement learning (RL) has been introduced into the accelerator design space exploration (DSE) task. Among them, the deep deterministic policy gradient (DDPG) algorithm, which is a reinforcement learning agent, has become a powerful tool for adapting to the hyperparameter tuning task of CNN accelerators due to its ability to handle continuous action spaces. Summary of the Invention

[0008] This invention proposes an adaptive reinforcement learning-driven multi-objective optimization method for accelerators, combining loop optimization techniques with reinforcement learning strategies to automatically optimize the parameter configuration combination of convolutional accelerators under platform resource constraints and network model structure conditions. This method constructs a state-action-reward interaction mechanism, designs a joint objective function, and combines a lightweight search module to decouple and optimize some independent factors, effectively improving the efficiency and intelligence level of accelerator deployment, especially suitable for resource-constrained embedded SoC (System-on-a-Chip) platforms. Specifically, this invention first analyzes the role mechanism of key design variables such as convolution loop unrolling factor, quantization accuracy, and cache partitioning, establishes a system-level state modeling method, and designs the DDPG framework suitable for continuous action space optimization as a learner for the configuration strategy. Simultaneously, a Python script is run on the CPU to implement a lightweight searcher, performing real-time search and allocation of the block parameters of Loop-3 and Loop-4 in convolution operations, thereby reducing the computational burden on the reinforcement learning agent. Finally, a reinforcement learning-driven adaptive configuration method and a corresponding search process are proposed to achieve joint optimization of unrolling scale, quantization accuracy, and block parameters.

[0009] The technical solution of the present invention:

[0010] An adaptive reinforcement learning-driven multi-objective optimization method for accelerators, comprising the following steps:

[0011] Step 1: Construct a search framework for reinforcement learning (RL) models.

[0012] Step 1.1: Define the state space: Establish a state vector that can describe the accelerator system. The state vector includes the structural parameters of the convolutional neural network (CNN), the available resources of the FPGA at present, and the action selected by the reinforcement learning agent in the current state.

[0013] The state vector is represented as: , ;

[0014] in, and These represent the width and height of the convolution kernel, respectively. and These represent the width and height of the input feature map, respectively. and These represent the number of channels in the input feature map and the output feature map, respectively. and These represent the width and height of the output feature map, respectively. and These represent the stride of the convolution operation in the horizontal and vertical directions, respectively. This indicates the number of convolution kernel groups or the number of clusters for parallel computation, used to distinguish between grouped convolution and parallel scale. Indicates the amount of on-chip cache resources (BRAM / URAM resources). Indicates the number of available DSP computing units. This indicates the first step chosen by the reinforcement learning agent. One action, or .

[0015] Step 1.2: Define the continuous action space: The designed continuous action space includes two types of factors: one is the hardware-related loop unrolling factor, and the other is the quantization precision factor related to the convolutional neural network. To ensure that the generated actions meet the FPGA's resource constraints, the continuous actions, after output, need to be mapped to a preset search interval through a discretization function to obtain valid configuration parameters.

[0016] In each training iteration, the reinforcement learning model generates four actions. These four actions are divided into two groups: Action-1 and Action-2. Action-1 contains three actions: , 1, 2, or 3; corresponding to the cyclic expansion factor, respectively. ,in The loop expansion factor, representing the direction of the input channel, determines the parallelism of the input channel; This represents the loop expansion factor in the horizontal direction of the output feature map, which determines the spatial parallelism. The loop unrolling factor, representing the output channel direction, determines the parallelism of the output channel. Action-2 contains one action. , ; Used to select the quantization precision factor for convolutional neural networks ,in Indicates the quantization bit width of the convolution weights. This represents the quantization bit width of the input feature map. The convolution operation is performed using four nested loops, named Loop-1, Loop-2, Loop-3, and Loop-4. The horizontal dimensions (height and width) of the input image are processed by Loop-3; the horizontal dimensions (height and width) of the convolution kernel are processed by Loop-1; the number of input channels and output channels are handled by two independent nested loops, Loop-2 and Loop-4, respectively. The specific generation process for these four actions is as follows:

[0017] The generation process for the action in Action-1 is as follows: The reinforcement learning agent first outputs a continuous value in the interval [0,1]. By discretizing the function Map this continuous value to the cycle expansion factor The integer value. This mapping process will be based on the search interval of each loop expansion factor. Scaling is applied to ensure the output meets the FPGA's resource constraints. The formula is as follows:

[0018]

[0019] The action generation process in Action-2 is as follows: the reinforcement learning agent outputs a continuous number. And it is mapped to discrete bit width using the following formula:

[0020]

[0021] in, and These represent the minimum and maximum values ​​of the optional quantization bit width, respectively;

[0022] Step 1.3: Construct a joint reward function: Based on the combined constraints of inference latency and accuracy, design a joint reward function to evaluate the merits of each action selection and guide the reinforcement learning model to gradually learn the optimal configuration strategy.

[0023] In reinforcement learning models, in each round of training iterations, a convolutional neural network quantization model is first used to quantize the model according to the selected uniform quantization bit width. The convolutional neural network is quantized, and then a delay model is called to obtain the quantized inference prediction delay. ,Will With target latency limit Compared to. If At that time, the convolutional neural network quantization model is fine-tuned by performing one epoch of training to restore accuracy, and the accuracy index is obtained. .like At this point, fine-tuning training is skipped and a penalty is applied directly. The joint reward function is defined in piecewise form:

[0024]

[0025] in, To quantize the validation set accuracy of the convolutional neural network model, To quantize the baseline accuracy of the convolutional neural network model, The scaling factor is used to explore E episodes in each experimental setting, calculate the reward according to the above formula, and update the policy accordingly until convergence.

[0026] Step 2: Define the parameter optimization process and hardware mapping strategy;

[0027] Step 2.1: Experience Replay and Input Sampling: The experience replay mechanism stores the data generated by the interaction between the reinforcement learning agent and the environment, and randomly samples it during the training process, thereby breaking the temporal correlation and ensuring the diversity and independence of training samples.

[0028] The reinforcement learning model first stores the interaction data of the reinforcement learning agent in the environment in the experience replay area, and then randomly samples a batch of data during the training process, so as to be exposed to more diverse and independent samples during training to improve the stability and convergence speed of training; the interaction data includes the current state, the current action, the current reward and the next state.

[0029] Step 2.2: Reinforcement learning model training: The reinforcement learning model is trained using data obtained from random sampling. The reinforcement learning model predicts the Q-value based on the input state and action, and continuously improves the policy quality through parameter updates to ensure that the learning process gradually converges.

[0030] Randomly sampled data is used to train the reinforcement learning model. The reinforcement learning model predicts a value, the Q-value, based on the current state and the current action. The Q-value represents the long-term overall benefit obtained by taking a certain action in a given state and is a core indicator for evaluating the quality of the action. During training, the reinforcement learning model continuously refers to the Q-value to update its parameters to ensure the stability of policy learning and avoid drastic oscillations.

[0031] Step 2.3: Strategy Update and Action Selection: Adopting The greedy strategy strikes a balance between exploration and exploitation. An action is randomly selected when the exploration conditions are met; otherwise, the action with the highest current Q-value is chosen. The final output action represents the optimal configuration for the accelerator system.

[0032] Once the reinforcement learning model is trained, the accelerator system... A greedy strategy is used to select actions. Specifically, the accelerator system first generates a random number between 0 and 1. and with preset threshold Compare; when At one time, the accelerator system randomly selects an action to ensure the exploratory nature of the strategy; when At this time, the accelerator system selects the action with the highest predicted Q value to ensure that existing experience is utilized. The selected action corresponds to the optimal parameter configuration of the accelerator system, including the cycle expansion factor and the quantization accuracy factor.

[0033] Step 3: Introduce a lightweight search engine to complete the decoupling parameter optimization;

[0034] Each convolutional neural network layer L has an independent block configuration. This refers to the block size in the Loop-3 and Loop-4 directions, which determines the data partitioning scale of each network layer. A lightweight searcher is implemented by running a Python script in real time on the CPU to perform the search for loop block parameters in real time, thereby achieving efficient block optimization without additional time overhead. During the data reading process of each network layer L, the accelerator system will run the search process on the CPU to dynamically find the optimal block configuration. .

[0035] Step 3.1: Preliminary Path Selection Optimization. Determine if each loop has been flattened, filter out paths that meet the criteria and can enter subsequent optimization, and avoid resource waste or bandwidth bottlenecks caused by unreasonable configuration.

[0036] First, check if Loop-1 and Loop-2 are not tiled. If not, it may cause additional transmission overhead due to computation. Then, determine if Loop-3 and Loop-4 are tiled. If they are tiled, there is potential for further optimization, and the process proceeds to subsequent data flow analysis and mapping strategies. If the conditions are not met, it may lead to computational idleness or bandwidth bottlenecks.

[0037] Step 3.2: Data Transmission Optimization and Resource Allocation. Under the premise of meeting resource constraints, various allocation attempts are made for on-chip cache and DSP computing units. The data transmission volume of each configuration scheme is calculated, and the optimal configuration scheme is selected to achieve a balance between storage and computing.

[0038] With Loop-3 and Loop-4 already tiled, first determine if the condition is met. and If the conditions are met, the pixel buffer and weight buffer are allocated sequentially according to different proportions, and the data transmission volumes BW1 and BW2 under different configuration schemes are calculated respectively, thereby selecting the optimal configuration scheme. Here, BW1 and BW2 represent the total data transmission volumes when Loop-3 and Loop-4 are used as inner loops, respectively, to compare the bandwidth consumption under different loop unrolling orders, thus selecting the optimal configuration scheme. Represents the cycle expansion factor , Represents block configuration , Represents the scale of a convolutional neural network . Represents the cycle expansion factor No more than the block configuration Otherwise, if the hardware parallelism exceeds the data size of a single block, it will result in idleness and waste; block configuration It cannot exceed the total scale of the convolutional neural network. Otherwise, the meaning of segmentation is lost. In the condition, This indicates the time required to pass the feature block to the on-chip cache. This represents the data processing time under the current design. The calculation formula is as follows:

[0039]

[0040] in, Indicates the transmission frequency of external input data; Indicates the data transmission width of the external bus; , , These represent the block scale for the width, height, and number of channels of the input feature map, respectively. , , These represent the block size of the convolution kernel width, convolution kernel height, and number of channels in the output feature map, respectively, and are used to describe the size of the data block that can be processed in the on-chip cache at one time.

[0041] Step 3.3: Strategy Selection and Loop Unrolling: Efficiency comparison of candidate schemes is performed, and the optimal loop unrolling order is finally determined. The result is output as the hardware configuration to achieve efficient execution of convolution operations and optimization of data transmission.

[0042] The configuration scheme with the smallest data transfer volume is selected, and BW1 and BW2 are compared: if BW1 is greater than BW2, Loop-3 is selected as the inner loop; otherwise, Loop-4 is selected. When Loop-3 is used as the inner loop, the weighted regroups of the input pixel blocks after each segment are read sequentially to improve the utilization of the on-chip pixel buffer and weight buffer. When Loop-4 is selected as the inner loop, the accelerator system loads pixels into the pixel buffer and weight buffer sequentially to support parallel reading of weight groups. Finally, based on the evaluation results, the optimal loop unrolling order and corresponding cache management strategy are output, achieving efficient execution of convolution operations and optimal data transfer configuration.

[0043] The beneficial effects of this invention are as follows: This invention combines reinforcement learning with lightweight search to achieve intelligent automatic optimization of convolutional accelerator parameters. While meeting latency constraints, it also takes into account accuracy and resource utilization, significantly improving deployment efficiency and adaptability. Attached Figure Description

[0044] Figure 1 This describes the basic process of convolution operations. Among them, and These represent the step size in the horizontal and vertical directions, respectively.

[0045] Figure 2 A search framework for reinforcement learning models.

[0046] Figure 3 This is a process for determining and initially optimizing the path selection for cyclic tiling.

[0047] Figure 4 To evaluate the experimental results of an adaptive reinforcement learning-driven accelerator multi-objective optimization method, where (a) is the consistency between the predicted and observed delays; and (b) is the prediction error. Detailed Implementation

[0048] The present invention will now be described in further detail with reference to the accompanying drawings and technical solutions.

[0049] An adaptive reinforcement learning-driven multi-objective optimization method for accelerators, comprising the following steps:

[0050] Step 1: Construct the search framework for the reinforcement learning (RL) model, the details of which are as follows: Figure 2 As shown.

[0051] Step 1.1: Define the state space: Establish a state vector that can describe the accelerator system. The state vector includes the structural parameters of the convolutional neural network (CNN), the available resources of the current FPGA, and the action selected by the reinforcement learning agent in the current state, which is used to comprehensively characterize the state of the accelerator system.

[0052] The state vector is represented as: , ;

[0053] in, and These represent the width and height of the convolution kernel, respectively. and These represent the width and height of the input feature map, respectively. and These represent the number of channels in the input feature map and the output feature map, respectively. and These represent the width and height of the output feature map, respectively. and These represent the stride of the convolution operation in the horizontal and vertical directions, respectively. This indicates the number of convolution kernel groups or the number of clusters for parallel computation, used to distinguish between grouped convolution and parallel scale. Indicates the amount of on-chip cache resources (BRAM / URAM resources). Indicates the number of available DSP computing units. This indicates the first step chosen by the reinforcement learning agent. One action, or To ensure the comparability of parameters with different dimensions during training, all state variables in the above state vectors are normalized and mapped to the [0,1] interval to improve the stability and convergence speed of reinforcement learning training. Figure 1 This describes the basic process of convolution operations.

[0054] Step 1.2: Define the continuous action space: The designed continuous action space includes two types of factors: one is the hardware-related loop unrolling factor, and the other is the quantization precision factor related to the convolutional neural network. To ensure that the generated actions meet the FPGA's resource constraints, the continuous actions, after output, need to be mapped to a preset search interval through a discretization function to obtain valid configuration parameters.

[0055] In each training iteration, the reinforcement learning model generates four actions. These four actions are divided into two groups: Action-1 and Action-2. Action-1 contains three actions: , =1, 2, or 3; corresponding to the cyclic expansion factor, respectively. ,in The loop expansion factor (Loop-2) represents the direction of the input channel and determines the parallelism of the input channel; The loop unrolling factor (Loop-3) in the horizontal direction of the output feature map determines the spatial parallelism. The loop unrolling factor (Loop-4), representing the output channel direction, determines the parallelism of the output channels. Action-2 contains one action. , ; Used to select the quantization precision factor for convolutional neural networks ,in Indicates the quantization bit width of the convolution weights. This indicates the quantization bit width of the input feature map. In this invention... and Maintaining consistency between the two ensures a uniform data format during convolution calculations and reduces hardware overhead. The convolution operation is performed using four nested loops, named Loop-1, Loop-2, Loop-3, and Loop-4. The horizontal dimensions (height and width) of the input image are processed by Loop-3; the horizontal dimensions (height and width) of the convolution kernel are processed by Loop-1; the number of input and output channels are handled by two independent nested loops, Loop-2 and Loop-4, respectively. The specific generation process for these four actions is as follows:

[0056] The generation process for the action in Action-1 is as follows: The reinforcement learning agent first outputs a continuous value in the interval [0,1]. By discretizing the function Map this continuous value to the cycle expansion factor The integer value. This mapping process will be based on the search interval of each loop expansion factor. Scaling is applied to ensure the output meets the FPGA's resource constraints. The formula is as follows:

[0057]

[0058] The action generation process in Action-2 is as follows: the reinforcement learning agent outputs a continuous number. And it is mapped to discrete bit width using the following formula:

[0059]

[0060] in, and These represent the minimum and maximum values ​​of the optional quantization bit width, respectively. In the example presented here, the optional bit width is {4, 8, 16}. , The span is 16−4+1=13, and the "−0.5" in the formula is used to match... The operation ensures that the output value is correctly rounded to the nearest valid bit width, resulting in the final value. This refers to the integer bit width of the quantization precision factor (e.g., 4, 8, 16). Action-2 only needs to determine one quantization bit width, and only needs to output one action in a single reinforcement learning process.

[0061] Step 1.3: Construct a joint reward function: Based on the combined constraints of inference latency and accuracy, design a joint reward function to evaluate the merits of each action selection and guide the reinforcement learning model to gradually learn the optimal configuration strategy.

[0062] In reinforcement learning models, in each round of training iterations, a convolutional neural network quantization model is first used to quantize the model according to the selected uniform quantization bit width. The convolutional neural network is quantized, and then a delay model is called to obtain the quantized inference prediction delay. ,Will With target latency limit Compared to. If When the inference prediction delay meets the delay constraint, the convolutional neural network quantization model is fine-tuned by one epoch to restore accuracy, thus obtaining the accuracy index. .like When the inference prediction delay exceeds the limit, skip the fine-tuning training and apply a penalty directly. The joint reward function is defined in piecewise form:

[0063]

[0064] in, To quantize the validation set accuracy of the convolutional neural network model, To quantize the baseline accuracy of a convolutional neural network model, such as the accuracy of an unquantized or established reference model on the same validation set; The scaling factor, preferably 0.01, is used to map the accuracy difference to a reasonable range of (-1, 1). This design prioritizes satisfying the latency constraint (exceeding the constraint results in a negative reward, and the magnitude of the negative reward increases with the excess rate), and then pursues higher accuracy after satisfying the latency constraint. Simultaneously, the "latency-first, fine-tuning" process avoids unnecessary training on unsuitable configurations, reducing search overhead. The reinforcement learning model explores E episodes in each experimental setting, calculates the reward according to the above formula, and updates the policy accordingly until convergence.

[0065] Step 2: Define the parameter optimization process and hardware mapping strategy;

[0066] Step 2.1: Experience Replay and Input Sampling: The experience replay mechanism stores the data generated by the interaction between the reinforcement learning agent and the environment, and randomly samples it during the training process, thereby breaking the temporal correlation and ensuring the diversity and independence of training samples.

[0067] The reinforcement learning model first stores the interaction data of the reinforcement learning agent in the environment in the experience replay area, and then randomly samples a batch of data during the training process, so as to be exposed to more diverse and independent samples during training to improve the stability and convergence speed of training; the interaction data includes the current state, the current action, the current reward and the next state.

[0068] Step 2.2: Reinforcement learning model training: The reinforcement learning model is trained using data obtained from random sampling. The reinforcement learning model predicts the Q-value based on the input state and action, and continuously improves the policy quality through parameter updates to ensure that the learning process gradually converges.

[0069] Randomly sampled data is used to train the reinforcement learning model. The reinforcement learning model predicts a value, the Q-value, based on the current state and the current action. The Q-value represents the long-term overall benefit obtained by taking a certain action in a given state and is a core indicator for evaluating the quality of the action. During training, the reinforcement learning model continuously refers to the Q-value to update its parameters to ensure the stability of policy learning and avoid drastic oscillations.

[0070] Step 2.3: Strategy Update and Action Selection: Adopting The greedy strategy strikes a balance between exploration and exploitation. An action is randomly selected when the exploration conditions are met; otherwise, the action with the highest current Q-value is chosen. The final output action represents the optimal configuration for the accelerator system.

[0071] Once the reinforcement learning model is trained, the accelerator system... A greedy strategy is used to select actions. Specifically, the accelerator system first generates a random number between 0 and 1. and with preset threshold Compare; when At one time, the accelerator system randomly selects an action to ensure the exploratory nature of the strategy; when At this time, the accelerator system selects the action with the highest predicted Q value to ensure that existing experience is utilized. It is important to note that... These are randomly generated numbers used to trigger exploration events. The exploration probability threshold, set manually, is typically set to a large value in the early stages of training to increase exploration, and then gradually decays to enhance utilization. The comparison between the two essentially involves a probabilistic trade-off between "exploring new actions" and "utilizing the optimal action," ensuring that the strategy avoids getting trapped in local optima while gradually converging to the optimal solution. The selected action corresponds to the optimal parameter configuration of the accelerator system, including the loop unrolling factor and the quantization precision factor.

[0072] Step 3: Introduce a lightweight search engine to complete the decoupling parameter optimization;

[0073] Each convolutional neural network layer L has an independent block configuration. This refers to the block size in the Loop-3 and Loop-4 directions, which determines the data partitioning scale of each network layer. To reduce the computational burden on the reinforcement learning agent, this invention runs a lightweight searcher on the CPU in real time using a Python script to perform the search for loop block parameters, thereby achieving efficient block optimization without additional time overhead. During the data reading process of each network layer L, the accelerator system will run a script on the CPU such as... Figure 3 The search process shown dynamically finds the optimal block configuration. .

[0074] Step 3.1: Initial Path Selection Optimization. Determine if each loop has been flattened, and filter out paths that meet the criteria and can proceed to subsequent optimization, avoiding resource waste or bandwidth bottlenecks caused by improper configuration. Details are as follows... Figure 3 As shown.

[0075] First, check if Loop-1 and Loop-2 are not tiled. If not, it may cause additional transmission overhead due to computation. Then, determine if Loop-3 and Loop-4 are tiled. If they are tiled, there is potential for further optimization, and the process proceeds to subsequent data flow analysis and mapping strategies. If the conditions are not met, it may lead to computational idleness or bandwidth bottlenecks.

[0076] Step 3.2: Data Transmission Optimization and Resource Allocation. Under the premise of meeting resource constraints, various allocation attempts are made for on-chip cache and DSP computing units. The data transmission volume of each configuration scheme is calculated, and the optimal configuration scheme is selected to achieve a balance between storage and computing.

[0077] With Loop-3 and Loop-4 already tiled, first determine if the condition is met. and If the conditions are met, the pixel buffer and weight buffer are allocated sequentially according to different proportions, and the data transmission volumes BW1 and BW2 under different configuration schemes are calculated respectively, thereby selecting the optimal configuration scheme. Here, BW1 and BW2 represent the total data transmission volumes when Loop-3 and Loop-4 are used as inner loops, respectively, to compare the bandwidth consumption under different loop unrolling orders, thus selecting the optimal configuration scheme. Represents the cycle expansion factor , Represents block configuration , Represents the scale of a convolutional neural network . Represents the cycle expansion factor No more than the block configuration Otherwise, if the hardware parallelism exceeds the data size of a single block, it will result in idleness and waste; block configuration It cannot exceed the total scale of the convolutional neural network. Otherwise, the meaning of segmentation is lost. In the condition, This indicates the time required to pass the feature block to the on-chip cache. This represents the data processing time under the current design. The calculation formula is as follows:

[0078]

[0079] in, Indicates the transmission frequency of external input data; Indicates the data transmission width of the external bus; , , These represent the block scale for the width, height, and number of channels of the input feature map, respectively. , , These represent the block size of the convolution kernel width, convolution kernel height, and number of channels in the output feature map, respectively, and are used to describe the size of the data block that can be processed in the on-chip cache at one time.

[0080] Step 3.3: Strategy Selection and Loop Unrolling: Efficiency comparison of candidate schemes is performed, and the optimal loop unrolling order is finally determined. The result is output as the hardware configuration to achieve efficient execution of convolution operations and optimization of data transmission.

[0081] The configuration scheme with the smallest data transfer volume is selected, and BW1 and BW2 are compared: if BW1 is greater than BW2, Loop-3 is selected as the inner loop; otherwise, Loop-4 is selected. When Loop-3 is used as the inner loop, the weighted regroups of the input pixel blocks after each segment are read sequentially to improve the utilization of the on-chip pixel buffer and weight buffer. When Loop-4 is selected as the inner loop, the accelerator system loads pixels into the pixel buffer and weight buffer sequentially to support parallel reading of weight groups. Finally, based on the evaluation results, the optimal loop unrolling order and corresponding cache management strategy are output, achieving efficient execution of convolution operations and optimal data transfer configuration.

[0082] This method was validated on the Xilinx ZCU102 FPGA platform. The search space created by this platform contains approximately 2430 accelerator configuration combinations. These configurations were tested to evaluate the prediction accuracy of the reinforcement learning model for neural network inference latency. Experimental results are as follows: Figure 4As shown: Figure 4 (a) in the figure shows the distribution of prediction error as a function of actual delay; Figure 4 Figure (b) shows the alignment between the predicted latency and the actual observed latency. Most sample points are distributed near the diagonal, indicating that the model's predictions are highly consistent with the measured values. It can be seen that the error fluctuation is relatively large when the workload is small and the latency is low, while the prediction error gradually decreases and tends to stabilize as the workload and latency increase. Overall, the prediction error of most samples is controlled below 3%, verifying that the constructed reinforcement learning model has high prediction accuracy under different configurations.

Claims

1. An adaptive reinforcement learning-driven multi-objective optimization method for accelerators, characterized in that, The steps are as follows: Step 1: Construct the search framework for the reinforcement learning model; Step 1.1: Define the state space: Establish a state vector that can describe the accelerator system. The state vector includes the structural parameters of the convolutional neural network, the available resources of the FPGA at present, and the action selected by the reinforcement learning agent in the current state. The state vector is represented as: ; in, and These represent the width and height of the convolution kernel, respectively. and These represent the width and height of the input feature map, respectively. and These represent the number of channels in the input feature map and the output feature map, respectively. and These represent the width and height of the output feature map, respectively. and These represent the stride of the convolution operation in the horizontal and vertical directions, respectively. This indicates the number of convolution kernel groups or the number of clusters for parallel computation, used to distinguish between grouped convolution and parallel scale. Indicates the amount of on-chip cache resources (BRAM / URAM resources). Indicates the number of available DSP computing units. This indicates the first step chosen by the reinforcement learning agent. One action, or ; Step 1.2: Define the continuous action space: The designed continuous action space includes two types of factors: one is the hardware-related loop unrolling factor, and the other is the quantization precision factor related to the convolutional neural network. In order to ensure that the generated actions meet the resource constraints of the FPGA, the continuous actions need to be mapped to the preset search interval through the discretization function after output to obtain the legal configuration parameters. In each training iteration, the reinforcement learning model generates four actions; these four actions are divided into two groups: Action-1 and Action-2; Action-1 contains three actions: , 1, 2, or 3; corresponding to the cyclic expansion factor, respectively. ,in The loop expansion factor, representing the direction of the input channel, determines the parallelism of the input channel; This represents the loop expansion factor in the horizontal direction of the output feature map, which determines the spatial parallelism. The loop unrolling factor, representing the output channel direction, determines the parallelism of the output channel; Action-2 contains an action. , ; Used to select the quantization precision factor for convolutional neural networks ,in Indicates the quantization bit width of the convolution weights. This represents the quantization bit width of the input feature map. The convolution operation is performed using four nested loops, named Loop-1, Loop-2, Loop-3, and Loop-4. The horizontal direction of the input image is processed by Loop-3, and the horizontal direction of the convolution kernel is processed by Loop-1. The number of input and output channels are handled by two independent nested loops, Loop-2 and Loop-4, respectively. The specific generation process for these four actions is as follows: The generation process for the action in Action-1 is as follows: The reinforcement learning agent first outputs a continuous value in the interval [0,1]. By discretizing the function Map this continuous value to the cycle expansion factor The integer value; the mapping process will be based on the search interval of each loop expansion factor. Scaling is applied to ensure the output meets the FPGA's resource constraints; the formula is as follows: The action generation process in Action-2 is as follows: the reinforcement learning agent outputs a continuous number. And it is mapped to discrete bit width using the following formula: in, and These represent the minimum and maximum values ​​of the optional quantization bit width, respectively; Step 1.3: Construct a joint reward function: Based on the combined constraints of inference latency and accuracy, design a joint reward function to evaluate the merits of each action selection and guide the reinforcement learning model to gradually learn the optimal configuration strategy; In reinforcement learning models, in each round of training iterations, a convolutional neural network quantization model is first used to quantize the model according to the selected uniform quantization bit width. The convolutional neural network is quantized, and then a delay model is called to obtain the quantized inference prediction delay. ,Will With target latency limit Compared to; if At that time, the convolutional neural network quantization model is fine-tuned by performing one epoch of training to restore accuracy, and the accuracy index is obtained. ;like At this time, skip fine-tuning training and directly apply the penalty; the joint reward function is defined in piecewise form: in, To quantize the validation set accuracy of the convolutional neural network model, To establish the baseline accuracy of the convolutional neural network quantization model, The scaling factor is used to explore E episodes in each experimental setting, calculate the reward according to the above formula, and update the policy accordingly until convergence. Step 2: Define the parameter optimization process and hardware mapping strategy; Step 2.1: Experience replay and input sampling: The data generated by the interaction between the reinforcement learning agent and the environment is stored through the experience replay mechanism, and random sampling is performed during the training process to break the temporal correlation and ensure the diversity and independence of training samples; The reinforcement learning model first stores the interaction data of the reinforcement learning agent in the environment in the experience replay area, and then randomly samples a batch of data during the training process, so as to be exposed to more diverse and independent samples during training to improve the stability and convergence speed of training; the interaction data includes the current state, the current action, the current reward and the next state; Step 2.2: Reinforcement learning model training: The reinforcement learning model is trained using data obtained from random sampling. The reinforcement learning model predicts the Q-value based on the input state and action, and continuously improves the policy quality through parameter updates to ensure that the learning process gradually converges. Randomly sampled data is used to train the reinforcement learning model. The reinforcement learning model predicts a value, namely the Q value, based on the current state and the current action. The Q value represents the long-term comprehensive benefit that can be obtained after taking a certain action in a given state, and is the core indicator for measuring the quality of the action. During the training process, the reinforcement learning model continuously refers to the Q value to update the parameters to ensure the stability of policy learning and avoid violent oscillations. Step 2.3: Strategy Update and Action Selection: Adopting The greedy strategy strikes a balance between exploration and exploitation; it randomly selects an action when the exploration conditions are met, otherwise it selects the action with the highest current Q value; the final output action is the optimal configuration scheme of the accelerator system. Once the reinforcement learning model is trained, the accelerator system... The greedy strategy selects actions; specifically, the accelerator system first generates a random number between 0 and 1. and with preset threshold Compare; when At one time, the accelerator system randomly selects an action to ensure the exploratory nature of the strategy; when At that time, the accelerator system selects the action with the highest current predicted Q value to ensure that existing experience is utilized; the selected action corresponds to the optimal parameter configuration of the accelerator system, including the cycle expansion factor and the quantization precision factor; Step 3: Introduce a lightweight search engine to complete the decoupling parameter optimization; Each convolutional neural network layer L has an independent block configuration. This refers to the block size in the Loop-3 and Loop-4 directions, which determines the data partitioning scale of each network layer. A lightweight searcher is implemented by running a Python script in real time on the CPU to perform real-time searching of the loop block parameters, thereby achieving efficient block optimization without additional time overhead. During the data reading process of each network layer L, the accelerator system will run the search process on the CPU to dynamically find the optimal block configuration. ; Step 3.1: Preliminary optimization of path selection; determine whether each loop has been flattened, filter out paths that meet the conditions and can enter the subsequent optimization, and avoid resource waste or bandwidth bottlenecks caused by unreasonable configuration; First, check if Loop-1 and Loop-2 are not tiled. If they are not tiled, it may cause additional transmission overhead due to computation. Then, determine if Loop-3 and Loop-4 are tiled. If they are tiled, there is a possibility of further optimization, and then proceed to the subsequent data flow analysis and mapping strategy. If the conditions are not met, it may lead to computational idleness or bandwidth bottleneck. Step 3.2: Data transmission optimization and resource allocation; Under the premise of meeting resource constraints, various allocation attempts are made for on-chip cache and DSP computing unit, the data transmission volume of each configuration scheme is calculated, and the optimal configuration scheme is selected to achieve a balance between storage and computing; With Loop-3 and Loop-4 already tiled, first determine if the condition is met. and If the conditions are met, the pixel buffer and weight buffer are allocated in sequence according to different proportions, and the data transmission volume BW1 and BW2 under different configuration schemes are calculated respectively, so as to select the optimal configuration scheme. Among them, BW1 and BW2 represent the total data transmission volume when Loop-3 and Loop-4 are used as inner loops, respectively, to compare the bandwidth consumption under different loop unfolding orders, so as to select the optimal configuration scheme. Represents the cycle expansion factor , Represents block configuration , Represents the scale of a convolutional neural network ; Represents the cycle expansion factor No more than the block configuration Otherwise, if the hardware parallelism exceeds the data size of a single block, it will result in idleness and waste; block configuration It cannot exceed the total scale of the convolutional neural network. Otherwise, the meaning of segmentation is lost; in In the condition, This indicates the time required to pass the feature block to the on-chip cache. This represents the data processing time under the current design. The calculation formula is as follows: in, W represents the transmission frequency of external input data; W represents the data transmission width of the external bus. , , These represent the block scale for the width, height, and number of channels of the input feature map, respectively. , , These represent the block size of the convolution kernel width, convolution kernel height, and number of channels in the output feature map, respectively, and are used to describe the size of the data block that can be processed in the on-chip cache at one time. Step 3.3: Strategy Selection and Loop Unrolling: Efficiency comparison of candidate schemes is performed, the optimal loop unrolling order is determined, and the result is output as hardware configuration to achieve efficient execution of convolution operations and optimization of data transmission; The configuration scheme with the smallest data transfer volume is selected, and BW1 and BW2 are compared: if BW1 is greater than BW2, Loop-3 is selected as the inner loop; otherwise, Loop-4 is selected as the inner loop. When Loop-3 is used as the inner loop, the weighted groups of the input pixel blocks after each segment are read sequentially to improve the utilization of the on-chip pixel buffer and weight buffer. When Loop-4 is selected as the inner loop, the accelerator system loads pixels into the pixel buffer and weight buffer sequentially to support the parallel reading of weight groups. Finally, based on the evaluation results, the optimal loop unrolling order and the corresponding cache management strategy are output to achieve efficient execution of convolution operations and optimal configuration of data transfer.

Images