A neural network structure pruning method and device based on deep reinforcement learning, and a medium

Abstract

The application relates to the technical field of deep reinforcement learning model compression and hardware acceleration, and relates to a neural network structured pruning method and device based on deep reinforcement learning and a medium. The method comprises the following steps: acquiring a neural network model to be pruned, constructing a multi-dimensional state vector for each layer of the neural network; constructing a hierarchical adaptive hybrid pruning model; inputting the multi-dimensional state vector into the hierarchical adaptive hybrid pruning model, outputting a joint action of a continuous pruning proportion and a discrete pruning strategy through a double-branch Actor network; the Critic network quantifies long-term benefits of the action through a hardware perception type composite reward function; iteratively training the hierarchical adaptive hybrid pruning model, performing a pruning action according to the pruning proportion and the pruning strategy of each layer, and selecting an optimal strategy; and deploying the neural network pruning according to the optimal strategy on an FPGA. The application provides an innovative solution for precise and efficient deployment of a neural network model on an end-side device such as an FPGA.

Description

Technical Field

[0001] This application relates to the field of deep reinforcement learning model compression and hardware acceleration technology, and to a method, device, and medium for structured pruning of neural networks based on deep reinforcement learning. Background Technology

[0002] Currently, neural networks, as the core architecture of deep learning, have demonstrated outstanding performance in artificial intelligence tasks such as image recognition and object detection, and have been widely applied in practical scenarios such as intelligent security and autonomous driving. To pursue higher inference accuracy, modern neural network models continuously optimize performance by increasing network depth and introducing multi-branch structures. However, this architectural evolution has led to a surge in the number of model parameters and computational load, posing a dual obstacle to the deployment of high-precision neural network models on embedded platforms such as FPGAs: high-parameter layers such as fully connected layers continuously put pressure on BRAM storage resources, while computationally intensive operators such as depthwise separable convolutions require high DSP parallel computing capabilities. Therefore, model pruning has become a necessary preprocessing step to balance model accuracy and hardware adaptability.

[0003] Existing automated pruning methods typically optimize global accuracy loss and abstract computational overhead (such as FLOPs), employing homogeneous compression strategies for each layer of the neural network. This approach fails to accurately differentiate the varying impacts of parameter size and computational load on hardware, and lacks deep adaptation to storage architecture and computational characteristics. This coarse-grained pruning mechanism exacerbates storage fragmentation in storage-intensive layers, while structured pruning of computationally intensive layers, if not fully considering hardware parallelism and computational unit alignment, struggles to fully translate into actual speedup gains. Consequently, the pruned model, under resource constraints, fails to achieve an optimal balance between accuracy preservation and hardware efficiency.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0006] This disclosure provides a method, apparatus, and medium for structured pruning of neural networks based on deep reinforcement learning. It constructs a hierarchical adaptive hybrid pruning mechanism, which achieves precise coordination between parameter size compression and computational load optimization by independently generating pruning ratios and strategy combinations for each network layer. This solves the problems of low hardware resource utilization and single strategy in traditional pruning algorithms, resulting in accuracy and resource imbalance. It provides an innovative solution for the precise and efficient deployment of neural network models on edge devices such as FPGAs.

[0007] In some embodiments, the method includes:

[0008] Obtain the neural network model to be pruned, and construct a multi-dimensional state vector for each layer of the neural network;

[0009] A hierarchical adaptive hybrid pruning model is constructed, based on a dual-branch Actor-Critic network architecture that integrates deep deterministic policy gradient algorithm and Gumbel-Softmax. The Actor network includes a hierarchical feature extraction module and a dual-branch output module, while the Critic network includes a state feature extraction module, an action feature extraction module, and a dual-channel fusion module.

[0010] The multidimensional state vector is input into the hierarchical adaptive hybrid pruning model, and the joint action of continuous pruning ratio and discrete pruning policy is output through a two-branch Actor network. The continuous branch generates the pruning ratio through the Sigmoid function, and the discrete branch generates the policy probability distribution through Gumbel-Softmax.

[0011] The Critic network quantifies the long-term benefits of joint actions using a hardware-aware composite reward function.

[0012] The iterative training of the hierarchical adaptive hybrid pruning model executes pruning actions according to the pruning ratio and pruning strategy of each layer, and selects the pruning strategy with the highest reward in the training round as the optimal strategy.

[0013] The neural network is pruned according to the optimal strategy, and after exporting and fine-tuning, it is deployed on the FPGA.

[0014] Preferably, the multidimensional state vector includes the static attributes of the current layer, the global resource budget, and the association information of adjacent layers.

[0015] Preferably, the Actor network takes a multi-dimensional state vector as input, outputs the pruning ratio through continuous branches, and outputs the policy probability distribution through discrete branches; including:

[0016] The hierarchical feature extraction module sequentially passes the input multidimensional state vector through a first fully connected layer and a second fully connected layer to capture the correlation features between inter-layer sensitivity differences and the global compression target. Specifically, the first fully connected layer maps high-dimensional state information to the hidden space and then introduces nonlinearity through the ReLU activation function. The output features of the second fully connected layer further extract inter-layer sensitivity difference features and then capture the correlation pattern between the global compression target and local pruning actions through ReLU activation.

[0017] The dual-branch output module takes the output features from the hierarchical feature extraction module as input, processes them through the third fully connected layer, and decouples the output tensor through dimensional dissection. The first half is the dimension of the continuous action space, and the pruning ratio is generated by the Sigmoid function for the continuous action space. The second half is the number of classes in the discrete policy space, which is converted into a policy probability distribution by Gumbel-Softmax reparameterization for discrete policy selection.

[0018] Preferably, the number of neurons in the third fully connected layer is equal to the sum of the continuous action dimension and the number of discrete pruning strategies.

[0019] Preferably, the exploration intensity of Gumbel-Softmax is dynamically adjusted using temperature parameters. A high temperature coefficient is set in the early stage of training to enhance strategy diversity, and the temperature coefficient is reduced to stabilize the strategy as the training progresses.

[0020] During backpropagation of continuous and discrete branches, the gradient signal of the feature extraction layer is shared, enabling coordinated optimization of pruning ratio adjustment and strategy selection.

[0021] Preferably, the Critic network takes a multi-dimensional state vector and action vector as input, outputs an action-value estimate, and evaluates the long-term reward of the current state-action pair. The action vector is a joint action of the concatenated pruning ratio and policy probability distribution; including:

[0022] State feature extraction module: Extracts high-dimensional features from the multidimensional state vector through a fully connected layer, and maps them into a state feature vector through non-linear activation;

[0023] Action feature extraction module: Maps action vectors into action feature vectors through a fully connected layer;

[0024] Dual-channel fusion module: Adds the state feature vector and the action feature vector element by element, and outputs the Q value through deep nonlinear transformation.

[0025] Preferably, the hardware-aware composite reward function is as follows:

[0026] ,

[0027] ,

[0028] ,

[0029] ,

[0030] ,

[0031] in, This represents the reward value based on accuracy. This represents the reward value based on FLOPs. This represents the reward value based on params. Indicates accuracy rate. This indicates that taking the logarithm makes the calculation of the reward function more stable. This represents the number of floating-point operations performed on the model after pruning according to the current pruning ratio and pruning strategy. This indicates the number of floating-point operations performed on the original model.

[0032] Preferably, during the fine-tuning stage, when the detection accuracy of the validation set is lower than a set threshold, a key layer reverse recovery mechanism is implemented.

[0033] In some embodiments, the apparatus includes a processor and a memory storing program instructions, the processor being configured to execute the deep reinforcement learning-based neural network structured pruning method when the program instructions are executed.

[0034] In some embodiments, the storage medium stores a computer program that, when executed by a processor, implements the neural network structured pruning method based on deep reinforcement learning.

[0035] The present disclosure provides a neural network structured pruning method, apparatus, and medium based on deep reinforcement learning, which can achieve the following technical effects:

[0036] Multi-dimensional state space design: global resource awareness and dynamic context modeling

[0037] The state space design in this invention breaks through the limitation of traditional pruning algorithms that only focus on single-layer features, and constructs a composite state representation that includes time dimension (pruning history), spatial dimension (inter-layer dependency), and hardware resource constraints (BRAM and DSP).

[0038] Hybrid motion space design: dual-branch collaborative optimization

[0039] This invention overcomes the limitation of the traditional DDPG algorithm, which only supports continuous action space. By constructing an Actor network architecture with a dual-branch output layer, it achieves joint decision-making on continuous pruning ratio and discrete pruning strategy: the bottom layer of the Actor network shares a feature encoding layer, while the upper layer is divided into continuous and discrete branches. The continuous branch outputs the pruning ratio through the Sigmoid function, and the discrete branch outputs a K-dimensional policy logits vector. The policy probability distribution is generated through Gumbel-Softmax transformation, forming a joint action pair.

[0040] By designing a joint action space and jointly training the pruning ratio and pruning strategy selection, the two are implicitly linked during the learning process. This design overcomes the decision-making bottleneck of a single action dimension, enabling the agent to autonomously generate the optimal combination of action vectors based on layer characteristics.

[0041] DDPG combined with Gumbel-Softmax algorithm: Hybrid action optimization of structured pruning

[0042] While the Deep Deterministic Policy Gradient (DDPG) algorithm effectively optimizes continuous action space problems, its inherent architecture struggles to adapt to the hybrid action space characteristics of pruning tasks, where continuous pruning ratios and discrete policy selections are jointly decided. Therefore, this invention introduces the Gumbel-Softmax technique. Through Gumbel noise reparameterization, discrete policy selection is reconstructed into a differentiable operation, enabling the Actor network to jointly optimize continuous pruning ratios and discrete policy distributions end-to-end. The Critic network's Q-value gradient for joint action pairs simultaneously applies to both the pruning ratio gradient of continuous branches and the policy distribution gradient of discrete branches. A chain rule is used to achieve coordinated updates of the two types of action parameters: the continuous pruning ratio finely adjusts the parameter compression intensity along the gradient direction, while the discrete policy distribution uses the probability density gradient to inversely correct the selection weights of the hardware adaptation mode.

[0043] To overcome the challenge of exploration stability caused by the coupling of continuous and discrete actions, this invention uses a temperature annealing strategy to dynamically control the exploration intensity, achieving structured exploration and precise policy locking while ensuring stable gradient propagation: In the early stage of training, a high temperature coefficient is used to expand the entropy of the policy distribution, enhancing the exploration of the diversity of pruning policy combinations; as the training progresses, the temperature coefficient is gradually reduced to guide the policy distribution to converge toward the pruning mode with optimal hardware efficiency.

[0044] This fusion algorithm retains the advantages of continuous control by DDPG while giving the agent the ability to accurately select discrete strategies. It solves the problems of gradient breakage and inefficient exploration in the optimization of mixed action space, enabling the entire model to handle mixed action spaces and increasing the model's flexibility.

[0045] Composite reward function: multi-objective constraints and hardware-driven optimization

[0046] Since it is necessary to comprehensively consider the memory resources (BRAM) and computing resources (DSP) in the FPGA, Acc, Params and Flops are introduced as penalty terms in the reward function to construct a composite reward function, which drives the policy network to achieve a balance between accuracy, resource utilization and computational efficiency.

[0047] The core value of this invention lies in achieving deep coupling between algorithmic layer pruning strategies and hardware layer resource constraints. Through the above improvements, this invention can autonomously derive the optimal pruning ratio and pruning strategy for each layer, thereby realizing differentiated pruning strategies between neural network layers. This allows model pruning to achieve optimal pruning within a controllable range of accuracy degradation and can be deployed on an FPGA.

[0048] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0049] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein:

[0050] Figure 1 Schematic diagram of the method flow of this invention;

[0051] Figure 2 A schematic diagram of the DDPG integration Gumbel-Softmax technology framework;

[0052] Figure 3 This is a schematic diagram of the improved Actor network architecture;

[0053] Figure 4 This is a schematic diagram of the improved Critic network architecture;

[0054] Figure 5 A schematic diagram of a resource-aware pruning framework driven by a multi-strategy hybrid action space;

[0055] Figure 6 This is a schematic diagram of the device structure of the present invention. Detailed Implementation

[0056] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0057] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0058] Unless otherwise stated, the term "multiple" means two or more.

[0059] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0060] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0061] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0062] Example 1

[0063] like Figure 1 , Figure 2 As shown, a neural network structured pruning method based on deep reinforcement learning is proposed. It performs hierarchical adaptive pruning and comprehensively considers the accuracy, computational load, and parameter count of the model after pruning, taking into account the limitations of FPGA storage resources (BRAM) and computing resources (DSP), in order to achieve the best balance between model performance and resource utilization after pruning.

[0064] Specifically, it includes:

[0065] S1: Obtain the neural network model to be pruned and construct a multi-dimensional state vector for each layer of the neural network.

[0066] As a refinement of the above embodiments, the input is a pre-trained neural network model. During the initialization phase, a multi-dimensional state vector is constructed for each layer of the neural network, including the static layer attributes of the current layer (such as the current layer type, number of input and output channels, kernel size, convolution stride, number of parameters, and flops), the global resource budget (such as the number of parameters and flops removed from the previous layer, and the number of parameters and flops remaining in the subsequent layer), and the relationship information between adjacent layers (the pruning ratio and pruning strategy of the previous layer). The neural network structure, dynamic resource constraints, and inter-layer dependencies are encoded into a state space that the agent can perceive. This provides a decision-making basis for the joint optimization of accuracy and efficiency of the Actor-Critic network.

[0067] S2: Construct a hierarchical adaptive hybrid pruning model based on a dual-branch Actor-Critic network architecture that integrates a deep deterministic policy gradient algorithm with Gumbel-Softmax. The Actor network includes a hierarchical feature extraction module and a dual-branch output module, while the Critic network includes a state feature extraction module, an action feature extraction module, and a dual-channel fusion module.

[0068] As a refinement of the above embodiments, the Actor network adopts a hierarchical feature extraction and dual-branch collaborative output architecture, while the Critic network adopts a dual-channel feature fusion mechanism and a deep value evaluation architecture.

[0069] In structured pruning of neural networks, the traditional DDPG algorithm outputs the pruning ratio of each layer through a continuous action space, treating all network layers as homogeneous structures for coarse-grained compression, ignoring the functional heterogeneity and structural sensitivity differences between neural network layers. More refined pruning can be achieved by employing different pruning strategies for the functional characteristics and hardware sensitivities of different network layers. To meet the need for both continuous intensity control and discretized strategy selection in neural network pruning tasks, this invention introduces DDPG combined with Gumbel-Softmax technology to construct a dual-branch Actor-Critic network architecture.

[0070] Specifically, the Actor network adopts a hierarchical feature extraction and dual-branch collaborative output architecture, responsible for the policy network parameters. Iterative updates, responsible for adjusting based on the current environment state. (The multidimensional state vector of the layer being processed at the current time step, i.e., the current environment in which the Agent is located) Selects a deterministic action. It directly controls the behavioral decisions of intelligent agents.

[0071] The Critic network employs a dual-channel feature fusion mechanism and a deep value evaluation architecture, responsible for the value network parameters. Iterative updates, inputting the current environment state. Concatenation with action vectors Output Action - Value Estimation Evaluate the long-term benefits of the current state-action pair.

[0072] S3: Input the multidimensional state vector into the hierarchical adaptive hybrid pruning model, and output the joint action of continuous pruning ratio and discrete pruning policy through a two-branch Actor network. The continuous branch generates the pruning ratio through the Sigmoid function, and the discrete branch generates the policy probability distribution through Gumbel-Softmax.

[0073] As a refinement of the above embodiments, such as Figure 3 As shown, it specifically includes:

[0074] S301: The Actor network is in the current environment state. As input.

[0075] S302: Feature abstraction is performed through two fully connected layers: the first layer maps high-dimensional state information to the hidden space and introduces nonlinearity through the ReLU activation function; the second layer further extracts the inter-layer sensitivity difference features and then captures the association pattern between the global compression target and the local pruning action through ReLU activation.

[0076] S303: The output features of the second fully connected layer are processed by the ReLU activation function and then input into the third fully connected layer. The third fully connected layer is a bi-branch output structure, and the number of neurons in this layer is equal to the sum of the continuous action dimension and the number of discrete pruning strategies. In this embodiment, the continuous action dimension is set to 1, representing that the pruning ratio of the Actor output (a floating-point number) is only for the current layer. If the continuous action dimension is n, it means that the pruning ratio of the current Actor output (not a floating-point number, but multiple floating-point numbers) is for multiple layers. The discrete pruning strategy is the probability distribution of each currently set discrete strategy being selected, and the sum of the probabilities is 1. During forward propagation, the original tensor output by the third fully connected layer is decoupled into two physically independent subspaces through dimensional anatomical cutting: the first half slice targets the continuous action space, and continuous values ​​are generated through the Sigmoid function. , indicating the pruning ratio of the current layer, This indicates the minimum pruning percentage for each layer. This represents the highest pruning ratio; the latter half of the slice is geared towards discrete policy selection, and the original K-dimensional policy logits vector is transformed into a differentiable policy distribution through Gumbel-Softmax reparameterization. This represents the probability distribution of the K pruning strategies in the current layer. It represents a vector space consisting of K real numbers and supports different pruning strategies such as L1 regularization pruning, L2 regularization pruning, and FPGM pruning.

[0077] This invention utilizes Gumbel-Softmax technology for differentiable sampling of discrete decisions. During the training phase, the exploration intensity is dynamically adjusted using temperature parameters: a smooth probability distribution under initial high-temperature conditions promotes diverse strategy exploration, while as the temperature decays exponentially, it gradually approaches a deterministic choice to stabilize the strategy. These two branches share the gradient signal from the underlying feature extraction layer during backpropagation, enabling synergistic optimization between pruning ratio adjustment and strategy selection. When the Critic network evaluates that a certain strategy combination can significantly improve compression efficiency, the gradient signal simultaneously corrects the logical parameters of strategy selection and the generation weights of pruning rates at each layer. This design retains the advantages of traditional DDPG for fine-grained continuous control while overcoming the limitations of single action types. It allows the agent to dynamically switch pruning strategies based on model structural characteristics, while precisely controlling the compression intensity of each layer, ultimately achieving an optimal balance between model accuracy and computational efficiency.

[0078] S4: The Critic network quantifies the long-term benefits of joint actions through a hardware-aware composite reward function.

[0079] As a refinement of the above embodiments, such as Figure 4 As shown, its core structure models the value function by processing the environment state and agent actions in parallel; specifically, it includes:

[0080] S401: Environmental Status High-dimensional features are extracted via fully connected layers and then nonlinearly activated to form a state feature vector.

[0081] S402: The pruning ratio and policy index are encoded into a unified vector to represent the agent's actions. The pruning ratio tensor directly represents the compression strength of each layer, while the policy selection expresses the algorithm's preference through the relaxed probability vector generated by Gumbel-Softmax. The two are concatenated and mapped to the action feature vector through an independent fully connected layer.

[0082] S403: The state feature vector and action feature vector are added element-wise in the hidden space, which enhances the correlation between state and action through implicit feature alignment. This preserves the ability of state information to encode global constraints and captures the local effects of specific pruning strategies in different layer structures.

[0083] S404: The fused joint features undergo deep nonlinear transformation and finally output a single scalar Q value. This value comprehensively reflects the expected long-term benefits of performing a specific hybrid action in the current state. It considers both the global impact of discrete strategy selection (such as sensitivity pruning may improve parameter utilization but increase computational overhead) and the local effects of continuous pruning ratio adjustment (such as the risk of accuracy collapse caused by excessive pruning in a certain layer).

[0084] This dual-channel fusion mechanism enables the Critic to simultaneously provide gradient signals for continuous action fine-tuning and discrete strategy optimization to the Actor during backpropagation, ensuring that the two evolve synergistically in the trade-off between compression efficiency and model performance. Its value evaluation process essentially constructs a dynamic mapping relationship between the hybrid action space and the compression objective, providing reliable directional guidance for the Actor's exploration-exploitation balance.

[0085] When evaluating the Q-value of action pairs in a Critic network, a reward function needs to be designed to accurately quantify the long-term benefits of the action. This invention addresses the limitation of existing pruning algorithms in their reward function design, which "emphasizes software metrics and neglects hardware mapping," by proposing a hardware-aware composite reward mechanism. The reward function formula is as follows:

[0086] ,

[0087] in, ,

[0088] ,

[0089] ,

[0090] ,

[0091] in, This represents the reward value based on accuracy. This represents the reward value based on FLOPs. This represents the reward value based on params. Indicates accuracy rate. This indicates that taking the logarithm makes the calculation of the reward function more stable. This represents the number of floating-point operations performed on the model after pruning according to the current pruning ratio and pruning strategy. This indicates the number of floating-point operations performed on the original model.

[0092] Traditional methods assume a linear relationship between FLOPs and Params when optimizing models, neglecting the hardware sensitivity differences between them in FPGA platforms, which correspond to DSP computing units and BRAM storage resources respectively. This leads to an imbalance in pruning strategies during hardware deployment, where "computational acceleration gains are artificially inflated while storage optimization potential is underutilized." This function innovatively reconstructs the reward mechanism from three dimensions: hardware resource decoupling, dynamic weight adaptation, and multi-objective collaborative optimization. First, based on the resource bottleneck characteristics of DSP and BRAM in FPGAs, FLOPs and Params are modeled as independent reward terms, where Params correspond to storage resources in the FPGA, and FLOPs correspond to computing resources in the FPGA. By introducing the reduced params and FLOPs, and the remaining params and FLOPs, the reinforcement learning strategy can dynamically adjust the compression ratio of subsequent layers. This avoids both over-pruning in the early layers leading to accuracy degradation and under-pruning in the early layers leading to over-pruning in subsequent layers, effectively coordinating the pruning ratio of all layers. Secondly, the weights are dynamically adjusted and optimized based on the training cycle. In the early stage of training, the focus is on optimizing the hardware adaptability of FLOPs / Params. The weight of accuracy is gradually increased through an exponential decay function, which ensures efficient use of hardware resources in the early stage and avoids accuracy collapse caused by excessive compression in the later stage.

[0093] S5: Iteratively trains a hierarchical adaptive hybrid pruning model, performing pruning actions according to the pruning ratio and pruning strategy of each layer. The model is non-destructively pruned using a mask matrix, and the pruning strategy with the highest reward in the training round is selected as the optimal strategy.

[0094] The overall process is as follows Figure 5 As shown, the agent processes the network in layers: for each layer The agent will receive an environmental state representing the useful features of that layer. It outputs the pruning ratio and policy probability distribution of the current layer, i.e., the action. In the first Layer by layer After pruning, the agent moves to the next level. Once the strategy search is complete, the optimal exploration model will be fine-tuned to achieve the best performance.

[0095] S6: Prune the neural network according to the optimal strategy, and deploy it on the FPGA after exporting and fine-tuning.

[0096] It should be noted that the DDPG algorithm achieves co-optimization of policy and value through the Actor-Critic framework. Its core advantage lies in using the value gradient of the Critic network to guide the deterministic policy improvement of the Actor network, while stabilizing the training process with the soft update mechanism of the target network. In this invention, the dual-branch shared low-level feature encoding network establishes a dynamic matching rule between pruning ratio and policy type through implicit association learning. The agent breaks through the decision-making bottleneck of a single action dimension, dynamically matching the optimal pruning ratio-pruning policy combination based on the characteristics of the layer structure. This ensures both the sensitivity and pruning accuracy of the key layers, and achieves dynamic, sparse, and efficient compression of redundant layers.

[0097] Example 2

[0098] This embodiment describes the above solution in detail using the field of real-time crop identification technology on the agricultural drone end-side.

[0099] Real-time crop identification technology on the agricultural drone side has developed rapidly due to the demand for efficient and low-cost field monitoring in precision agriculture. Drones equipped with high-resolution multispectral cameras collect images of farmland and identify the growth status of crops such as corn, wheat, and rice in real time, guiding variable fertilization and pest and disease control. Existing systems mostly rely on cloud processing, but poor farmland network coverage leads to decision delays of up to 3-5 seconds, missing the optimal time for pesticide application. Mainstream agricultural drones (such as the DJI Agras T40) equipped with the Xilinx KriaKV260 FPGA edge computing module only have 4.5MB RAM, 252 DSP units, and 141KLUT resources, which cannot support conventional CNN models. Therefore, to achieve low-latency, high-precision real-time identification under the constraints of limited computing resources and power consumption of edge computing devices, this invention proposes a neural network structured pruning method based on deep reinforcement learning, as follows:

[0100] Step 1: Pruning strategy search based on DDPG fusion with Gumbel-Softmax

[0101] Using a multispectral crop image dataset collected in the field (containing near-infrared-RGB fused images of maize leaf spot, wheat rust, and rice blast) as input, and combining various search parameters such as target compression ratio, layer-by-layer pruning lower limit, reward function selection, DDPG exploration noise coefficient, and Gumbel-Softmax temperature, based on the storage and computing limitations of the target device and the flight altitude of the UAV, we seek the Pareto optimal solution for the model's accuracy, real-time inference speed, and hardware resource consumption.

[0102] Initialize the state space layer_embedding, experience pool Critic network parameters Actor network parameters Critic-Target network parameters Actor-Target network parameters .

[0103] During the warm-up phase, samples are generated by completely random actions to fill the experience pool, providing diverse initial data for subsequent training and ensuring training stability and the foundation for policy optimization.

[0104] During the training phase, the agent performs actions. Receive environmental awards and new status Transfer samples Store in experience pool The Critic network draws from the experience pool. Historical data on the number of random sampling batches N Evaluate the state action pair Generate an estimated Q value The Actor-Target network outputs the next action. The Critic-Target network evaluates the Q-value of the action. Substituting into the Bellman equation, the target Q value is calculated. :

[0105] ,

[0106] in, This determines how much future rewards decay in relation to the value of the current state. and equivalence.

[0107] Calculate the mean square error between the target Q value and the current Q value. :

[0108] ,

[0109] in, It is the variance between the target Q value and the current Q value when the agent performs action a in an environment state of s.

[0110] The optimal value estimate of the Bellman equation is gradually approximated by minimizing the loss function through gradient descent.

[0111] Actor Network and Environment Status Interactive, output branch 1 outputs the pruning ratio via sigmoid. Output branch 2 is distributed using the Gumbel-Softmax output strategy. Generate action pairs We construct a hybrid action search space for lightweight real-time recognition.

[0112] Critic network evaluates the value of state-action pairs The gradient of the Critic network with respect to action a is obtained through the chain rule. Parameters transmitted to the Actor network The strategy is updated along the direction of Q-value growth, and the Actor network parameters are updated through gradient ascent:

[0113] ,

[0114] To maximize expectations:

[0115] ,

[0116] This makes it more likely that future actions will achieve a high Q value under the same conditions.

[0117] The Actor-Target network is responsible for replaying the next state sampled from the pool based on experience. Choose the optimal next action Network parameters Regularly from Copying, gradually synchronizing parameters through soft updates:

[0118] ,

[0119] in, Used to control the speed at which target network parameters approach online network parameters.

[0120] The Critic-Target network is responsible for output. Thus, the target Q value is calculated:

[0121] ,

[0122] Critic-Target network parameters Regularly from Copying, gradually synchronizing parameters through soft updates:

[0123] .

[0124] In each training round, pruning actions are performed according to the pruning ratio and pruning strategy of each layer. The model is pruned non-destructively using a mask matrix. The flops and params after pruning each layer are calculated according to the pruning ratio to measure the compression rate and reward. The pruning strategy with the highest reward in the training round is selected as the optimal strategy.

[0125] Step 2: Export

[0126] First, the optimal strategy file for agricultural tasks, generated during the search phase, is loaded. This strategy has been explored using deep reinforcement learning to determine the optimal pruning ratio and pruning strategy for each layer under the constraints of UAV-side hardware resources. Based on the pruning strategy, the importance score of each channel is calculated along the input channel dimension. Channels are sorted in descending order of importance score. The top K channels are retained according to the current layer's pruning ratio, generating a Boolean mask. Retained channels are marked as True, and removed channels are marked as False. The output channels of the previous layer are aligned with the new input channels of the current layer. The new weights are solved using the least squares method, and the reconstructed weights are adjusted to match the shape of the pruned structure. The reconstructed weight tensor is loaded into the current layer, and channels marked as False are physically removed based on the mask, completing layer-by-layer physical pruning and deriving the pruned lightweight neural network model.

[0127] Step 3: Online Enhanced Accuracy Recovery and Adaptive Tuning

[0128] In the fine-tuning phase of drone-based agricultural applications, the training process is adaptively adjusted to closely align with the characteristics of field operations. For crop images acquired by multispectral cameras, complex field environments are simulated by applying random occlusion (simulating leaf overlap), spectral shift (simulating light variations), and Gaussian noise (simulating sensor noise) to the training images to improve the model's generalization ability. Taking cotton Verticillium wilt leaves as an example, the area of ​​randomly erased regions is controlled between 10% and 25% to simulate the typical occlusion ratio of overlapping leaf clusters. In the hardware deployment and verification phase, when the measured inference latency exceeds the target threshold, the pruning rate of shallow convolutions is preferentially tightened—because shallow feature extraction layers play a decisive role in crop texture (such as the edges of rice blast spots).

[0129] When the validation set detection accuracy (for small-target diseases such as wheat stripe rust) falls below 85%, a key layer inverse recovery mechanism is activated: the last spatial resolution preservation layer is sorted by channel sensitivity, and the top 20% of pruned channels with a correlation higher than 0.7 with lesion color features (LAB color space a channel response) are reactivated and made to participate in gradient updates through dynamic mask expansion. The spectral consistency constraint introduced in the optimization objective is used to force the variance of the model's feature response in the three agricultural characteristic bands of 450nm (blue light), 680nm (red light), and 720nm (near-infrared) to not exceed 1.3 times that of the original model, ensuring the stability of key functions such as chlorophyll content estimation. The final lightweight model maintains 85.66% disease identification accuracy while meeting the stringent power consumption and memory limitations of the UAV terminal hardware, and can maintain a real-time analysis capability of 15 frames per second under wind force interference of level 6.

[0130] In real-world testing in a wheat field, the drone equipped with this system (flying at an altitude of 10m and a speed of 8m / s) achieved the following performance improvements:

[0131] Identification accuracy: The mAP for leaf spot disease identification decreased from 91.2% before pruning to 85.66%, meeting the needs of field monitoring. A small number of missed detections can be compensated for by repeated inspections using drones.

[0132] Real-time performance: The end-to-end processing latency has been reduced from 3.2s in the cloud solution to 301ms, meeting the positioning accuracy requirement of ±0.5m at a flight speed of 10m / s.

[0133] Hardware efficiency: Model storage usage was reduced from 4.232MB to 1.389MB, and the number of floating-point calculations was reduced from 12.714M to 4.718M, meeting the deployment limitations of drones.

[0134] Example 3

[0135] Combination Figure 6 As shown, this disclosure provides a neural network structured pruning device 300 based on deep reinforcement learning, including a processor 304 and a memory 301. Optionally, the device may further include a communication interface 302 and a bus 303. The processor 304, communication interface 302, and memory 301 can communicate with each other via the bus 303. The communication interface 302 can be used for information transmission. The processor 304 can call logical instructions in the memory 301 to execute the neural network structured pruning method based on deep reinforcement learning described in the above embodiment.

[0136] Furthermore, the logic instructions in the aforementioned memory 301 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0137] The memory 301, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 304 executes functional applications and data processing by running the program instructions / modules stored in the memory 301, that is, it implements the neural network structured pruning method based on deep reinforcement learning in the above embodiments.

[0138] The memory 301 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 301 may include high-speed random access memory and may also include non-volatile memory.

[0139] This disclosure provides a computer-readable storage medium storing computer-executable instructions configured to execute the aforementioned neural network structured pruning method based on deep reinforcement learning.

[0140] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0141] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code. It can also be a transient storage medium.

[0142] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

Claims

1. A structured pruning method for neural networks based on deep reinforcement learning, characterized in that, Includes the following steps: Obtain the neural network model to be pruned, and construct a multi-dimensional state vector for each layer of the neural network; A hierarchical adaptive hybrid pruning model is constructed, based on a dual-branch Actor-Critic network architecture that integrates deep deterministic policy gradient algorithm and Gumbel-Softmax. The Actor network includes a hierarchical feature extraction module and a dual-branch output module, while the Critic network includes a state feature extraction module, an action feature extraction module, and a dual-channel fusion module. The multidimensional state vector is input into the hierarchical adaptive hybrid pruning model, and the joint action of continuous pruning ratio and discrete pruning policy is output through a two-branch Actor network. The continuous branch generates the pruning ratio through the Sigmoid function, and the discrete branch generates the policy probability distribution through Gumbel-Softmax. The Critic network quantifies the long-term benefits of joint actions using a hardware-aware composite reward function. The iterative training of the hierarchical adaptive hybrid pruning model executes pruning actions according to the pruning ratio and pruning strategy of each layer, and selects the pruning strategy with the highest reward in the training round as the optimal strategy. The neural network is pruned according to the optimal strategy, and after being exported and fine-tuned, it is deployed on the FPGA. The hardware-aware composite reward function is as follows: ， ， in, This represents the reward value based on accuracy. This represents the reward value based on FLOPs. This represents the reward value based on params.

2. The neural network structured pruning method based on deep reinforcement learning according to claim 1, characterized in that, The multidimensional state vector contains the static attributes of the current layer, the global resource budget, and the association information between adjacent layers.

3. The neural network structured pruning method based on deep reinforcement learning according to claim 1, characterized in that, The Actor network takes a multidimensional state vector as input, outputs the pruning ratio through continuous branches, and outputs the policy probability distribution through discrete branches; including: The hierarchical feature extraction module: The input multidimensional state vector is sequentially passed through the first fully connected layer and the second fully connected layer to capture the correlation features between inter-layer sensitivity differences and the global compression target; Specifically, the first fully connected layer maps the high-dimensional state information to the hidden space, and then introduces nonlinearity through the ReLU activation function; The output features of the second fully connected layer further extract the inter-layer sensitivity difference features, and then capture the correlation pattern between the global compression target and the local pruning action through ReLU activation; The dual-branch output module takes the output features from the hierarchical feature extraction module as input, processes them through the third fully connected layer, and decouples the output tensor through dimensional dissection. The first half is the dimension of the continuous action space, and the pruning ratio is generated by the Sigmoid function for the continuous action space. The second half is the number of classes in the discrete policy space, which is converted into a policy probability distribution by Gumbel-Softmax reparameterization for discrete policy selection.

4. The neural network structured pruning method based on deep reinforcement learning according to claim 3, characterized in that, The number of neurons in the third fully connected layer is equal to the sum of the continuous action dimension and the number of discrete pruning strategies.

5. The neural network structured pruning method based on deep reinforcement learning according to claim 3, characterized in that, The exploration intensity of Gumbel-Softmax is dynamically adjusted using temperature parameters. A high temperature coefficient is set in the early stage of training to enhance strategy diversity, and the temperature coefficient is reduced to stabilize the strategy as the training progresses. During backpropagation of continuous and discrete branches, the gradient signal of the feature extraction layer is shared, enabling coordinated optimization of pruning ratio adjustment and strategy selection.

6. The neural network structured pruning method based on deep reinforcement learning according to claim 1, characterized in that, The Critic network takes a multidimensional state vector and action vector as input, outputs an action-value estimate, and evaluates the long-term reward of the current state-action pair. The action vector is a joint action of the concatenated pruning ratio and policy probability distribution; including: State feature extraction module: Extracts high-dimensional features from the multidimensional state vector through a fully connected layer, and maps them into a state feature vector through non-linear activation; Action feature extraction module: Maps action vectors into action feature vectors through a fully connected layer; Dual-channel fusion module: Adds the state feature vector and the action feature vector element by element, and outputs the Q value through deep nonlinear transformation.

7. The neural network structured pruning method based on deep reinforcement learning according to claim 1, characterized in that, The parameter values ​​in the hardware-aware composite reward function are determined as follows: ， ， ， in, Indicates accuracy rate. This indicates that taking the logarithm makes the calculation of the reward function more stable. This represents the number of floating-point operations performed on the model after pruning according to the current pruning ratio and pruning strategy. This indicates the number of floating-point operations performed on the original model.

8. The neural network structured pruning method based on deep reinforcement learning according to claim 1, characterized in that, During the fine-tuning phase, when the detection accuracy of the validation set is lower than a set threshold, a key layer reverse recovery mechanism is implemented.

9. A neural network structure pruning device based on deep reinforcement learning, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to execute, when running the program instructions, the neural network structure pruning method based on deep reinforcement learning as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the neural network structured pruning method based on deep reinforcement learning as described in any one of claims 1-8.

Images