Neural network structured pruning method and device based on deep reinforcement learning, and medium

Abstract

The invention relates to the technical field of deep reinforcement learning model compression and hardware acceleration, and relates to a neural network structured pruning method and device based on deep reinforcement learning, and a medium. The method comprises the following steps: acquiring a neural network model to be pruned, and constructing a multi-dimensional state vector for each layer of the neural network; constructing a hierarchical self-adaptive hybrid pruning model; inputting the multi-dimensional state vector into a hierarchical self-adaptive hybrid pruning model, and outputting a combined action of a continuous pruning proportion and a discrete pruning strategy through a double-branch Actor network; the Critic network quantifies the long-term income of the action through a hardware sensing type composite reward function; iteratively training a hierarchical self-adaptive hybrid pruning model, executing pruning actions according to the pruning proportion and pruning strategy of each layer, and selecting an optimal strategy; and pruning and deploying the neural network to the FPGA according to the optimal strategy. According to the method, an innovative solution is provided for accurate and efficient deployment of the neural network model in end-side equipment such as an FPGA (Field Programmable Gate Array).

Description

Technical Field

[0001] The present application relates to the technical field of deep reinforcement learning model compression and hardware acceleration, and relates to a neural network structured pruning method, device, and medium based on deep reinforcement learning. Background Art

[0002] Currently, as the core architecture of deep learning, neural networks have demonstrated excellent performance in artificial intelligence tasks such as image recognition and object detection, and have been widely applied to practical scenarios such as intelligent security and autonomous driving. To pursue higher inference accuracy, modern neural network models continuously optimize their performance by increasing the network depth, introducing multi-branch structures, etc. However, this architectural evolution has led to a sharp increase in the number of model parameters and the amount of computation, making it difficult for high-precision neural network models to be deployed on embedded platforms such as FPGAs. The high-parameter layers such as the fully connected layer continuously pressure the BRAM storage resources, while the computationally intensive operators such as depthwise separable convolution require a high level of DSP parallel computing power. Therefore, model pruning has become a necessary preprocessing step to balance model accuracy and hardware adaptability.

[0003] Existing automated pruning methods usually take the global accuracy loss and abstract computational amount (such as FLOPs) as the optimization objectives, and adopt a homogeneous compression strategy for each layer of the neural network. They neither accurately distinguish the different impacts of parameter scale and computational load on the hardware, nor deeply adapt to the storage architecture and computational characteristics. This coarse-grained pruning mechanism leads to increased fragmentation of the storage space due to parameter pruning in storage-intensive layers. If the structured pruning of computationally intensive layers does not fully consider the hardware parallel characteristics and computational unit alignment, it is still difficult to fully convert into actual acceleration benefits, resulting in difficulty in achieving the optimal balance between accuracy retention and hardware efficiency for the pruned model under resource constraints.

[0004] It should be noted that the information disclosed in the above background art section is only used to enhance the understanding of the background of the present application, and thus may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a comprehensive review, nor is it intended to identify key / important elements or delineate the scope of protection of these embodiments. Instead, it serves as a preamble to the following detailed description.

[0006] The embodiments of the present disclosure provide a neural network structured pruning method, device, and medium based on deep reinforcement learning, which constructs a hierarchical adaptive hybrid pruning mechanism. By independently generating pruning ratios and policy combinations for each network layer, it realizes the precise coordination of parameter scale compression and computing load optimization, solves the problems of low hardware resource utilization and accuracy-resource imbalance caused by single strategies in traditional pruning algorithms, and provides an innovative solution for the precise and efficient deployment of neural network models on edge devices such as FPGAs.

[0007] In some embodiments, the method includes: Obtain the neural network model to be pruned, and construct a multi-dimensional state vector for each layer of the neural network; Construct a hierarchical adaptive hybrid pruning model based on a dual-branch Actor-Critic network architecture that combines the deep deterministic policy gradient algorithm and Gumbel-Softmax. The Actor network includes a hierarchical feature extraction module and a dual-branch output module, and the Critic network includes a state feature extraction module, an action feature extraction module, and a dual-channel fusion module; Input the multi-dimensional state vector into the hierarchical adaptive hybrid pruning model, and output a joint action of a continuous pruning ratio and a discrete pruning strategy through the dual-branch Actor network. Among them, the continuous branch generates the pruning ratio through the Sigmoid function, and the discrete branch generates the policy probability distribution through Gumbel-Softmax; The Critic network quantifies the long-term benefits of the joint action through a hardware-aware composite reward function; Iteratively train the hierarchical adaptive hybrid pruning model, perform pruning actions according to the pruning ratio and pruning strategy of each layer, and select the pruning strategy of the round of training with the highest reward as the optimal strategy; Prune the neural network according to the optimal strategy, and after export and fine-tuning, deploy it on the FPGA.

[0008] Preferably, the multi-dimensional state vector includes the static attributes of the current layer, the global resource budget, and the adjacent layer association information.

[0009] Preferably, the Actor network takes the multi-dimensional state vector as input, outputs the pruning ratio through the continuous branch, and outputs the policy probability distribution through the discrete branch; it includes: Hierarchical feature extraction module: sequentially pass the input multi-dimensional state vector through the first fully connected layer and the second fully connected layer to capture the correlation features between the inter-layer sensitivity differences and the global compression target; specifically, the first fully connected layer maps the high-dimensional state information to the hidden space, and then introduces non-linearity through the ReLU activation function; the output features of the second fully connected layer further extract the inter-layer sensitivity difference features, and then capture the correlation pattern between the global compression target and the local pruning action through ReLU activation.

[0010] Dual-branch output module: It takes the output features of the hierarchical feature extraction module as input, processes them through the third fully connected layer, and decouples the output tensor through dimensional dissection. Among them, the first half is the dimension of the continuous action space. For the continuous action space, the pruning ratio is generated through the Sigmoid function; the second half is the number of categories in the discrete policy space. For discrete policy selection, it is transformed into a policy probability distribution through Gumbel-Softmax reparameterization.

[0011] Preferably, the number of neurons in the third fully connected layer is equal to the sum of the continuous action dimension and the number of discrete pruning strategies.

[0012] Preferably, a temperature parameter is used to dynamically adjust the exploration intensity of Gumbel-Softmax. A high temperature coefficient is set at the beginning of training to enhance policy diversity, and the temperature coefficient is decayed during the training process to stabilize the policy; During the backpropagation process of the continuous branch and the discrete branch, the gradient signals of the shared feature extraction layer are shared, so that the pruning ratio adjustment and policy selection are co-optimized.

[0013] Preferably, the Critic network takes a multi-dimensional state vector and an action vector as input, outputs an action-value estimate, and evaluates the long-term reward of the current state-action pair. The action vector is the joint action of the spliced pruning ratio and the policy probability distribution; it includes: State feature extraction module: It extracts high-dimensional features from the multi-dimensional state vector through a fully connected layer and maps them to a state feature vector through a non-linear activation; Action feature extraction module: It maps the action vector to an action feature vector through a fully connected layer; Dual-channel fusion module: It adds the state feature vector and the action feature vector element by element and outputs the Q value through a deep non-linear transformation.

[0014] Preferably, the hardware-aware composite reward function is as follows: , , , , , Among them, represents the reward value based on accuracy, represents the reward value based on flops, represents the reward value based on params, represents accuracy, represents taking the logarithm to make the calculation of the reward function more stable, Represents the floating-point operation count of the model after pruning according to the current pruning ratio and pruning strategy. Represents the floating-point operation count of the original model.

[0015] Preferably, in the fine-tuning stage, when the detection accuracy of the validation set is lower than the set threshold, the key layer reverse recovery mechanism is adopted.

[0016] In some embodiments, the device includes a processor and a memory storing program instructions. The processor is configured to execute the neural network structured pruning method based on deep reinforcement learning when running the program instructions.

[0017] In some embodiments, the storage medium stores a computer program, which implements the neural network structured pruning method based on deep reinforcement learning when executed by the processor.

[0018] A neural network structured pruning method, device, and medium based on deep reinforcement learning provided by the embodiments of the present disclosure can achieve the following technical effects: Multi-dimensional state space design: Global resource awareness and dynamic context modeling In the present invention, the design of the state space breaks through the limitation of traditional pruning algorithms that only focus on single-layer features, and constructs a composite state representation including a time dimension (pruning history), a space dimension (inter-layer dependence), and hardware resource constraints (BRAM and DSP).

[0019] Hybrid action space design: Dual-branch collaborative optimization The present invention breaks through the limitation of traditional DDPG algorithms that only support continuous action spaces. By constructing an Actor network architecture with a dual-branch output layer, it realizes the joint decision-making of continuous pruning ratio and discrete pruning strategy: The bottom layer of the Actor network shares a feature encoding layer, and the upper layer is divided into a continuous branch and a discrete branch. The continuous branch outputs the pruning ratio through the Sigmoid function, and the discrete branch outputs a K-dimensional policy logits vector, which generates a policy probability distribution through the Gumbel-Softmax transformation to form a joint action pair.

[0020] Through the joint action space design, the joint training of pruning ratio and pruning strategy selection is carried out, making the two implicitly associated during the learning process. This design breaks through the decision-making bottleneck of a single action dimension, enabling the intelligent agent to autonomously generate an optimal action vector combination according to layer characteristics.

[0021] DDPG integrated with Gumbel-Softmax algorithm: Hybrid action optimization for structured pruning Although the Deep Deterministic Policy Gradient (DDPG) algorithm can effectively optimize problems in continuous action spaces, its inherent architecture is difficult to adapt to the characteristics of the hybrid action space with the joint decision-making of continuous pruning ratios and discrete policy selections in pruning tasks. Therefore, the present invention introduces the Gumbel-Softmax technique. Through the Gumbel noise reparameterization technique, the discrete policy selection is reconstructed into a differentiable operation, enabling the Actor network to jointly optimize the continuous pruning ratio and the discrete policy distribution end-to-end. The Q-value gradient of the joint action pair of the Critic network simultaneously acts on the pruning ratio gradient of the continuous branch and the policy distribution gradient of the discrete branch, and the coordinated update of the two types of action parameters is achieved through the chain rule - the continuous pruning ratio finely adjusts the parameter compression intensity along the gradient direction, and the discrete policy distribution reversely corrects the selection weights of the hardware adaptation mode through the probability density gradient.

[0022] To overcome the exploration stability challenges brought about by the continuous-discrete action coupling, the present invention uses a temperature annealing strategy to dynamically regulate the exploration intensity, achieving structured exploration and precise policy locking on the premise of ensuring stable gradient propagation: at the initial stage of training, a high temperature coefficient is used to expand the entropy value of the policy distribution, enhancing the diversity exploration of pruning strategy combinations; as the training progresses, the temperature coefficient is gradually reduced to guide the policy distribution to converge directionally to the pruning mode with the optimal hardware efficiency.

[0023] While retaining the continuous control advantages of DDPG, this fusion algorithm endows the agent with the ability to precisely select discrete policies, solves the problems of gradient breakage and inefficient exploration in hybrid action space optimization, enables the entire model to handle the hybrid action space, and increases the flexibility of the model.

[0024] Composite Reward Function: Multi-Objective Constraints and Hardware-Driven Optimization Since it is necessary to comprehensively consider the storage resources (BRAM) and computing resources (DSP) in the FPGA, Acc, Params, and Flops are introduced as penalty terms in the reward function to construct a composite reward function, driving the policy network to achieve a balance among accuracy, resource utilization, and computing efficiency.

[0025] The core value of the present invention lies in achieving a deep coupling between the pruning strategy at the algorithm layer and the resource constraints at the hardware layer. Through the above improvements, the present invention can autonomously obtain the optimal pruning ratio and pruning strategy for each layer, thereby realizing the inter-layer differential pruning strategy of the neural network, enabling the model pruning to achieve optimal pruning within a controllable range of accuracy degradation and being deployed on the FPGA.

[0026] The above general description and the following description are only exemplary and explanatory, and are not used to limit this application. Brief Description of the Drawings

[0027] One or more embodiments are exemplarily illustrated by corresponding drawings. These exemplary illustrations and the drawings do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements. The drawings do not constitute a scale limitation, and wherein: Figure 1 Schematic diagram of the method flow of the present invention; Figure 2 Schematic diagram of the DDPG integrated with Gumbel-Softmax technology framework; Figure 3 Schematic diagram of the improved Actor network architecture; Figure 4 Schematic diagram of the improved Critic network architecture; Figure 5 Schematic diagram of the resource-aware pruning framework driven by a multi-policy hybrid action space; Figure 6 Schematic diagram of the device structure of the present invention. Detailed implementation manners

[0028] In order to be able to understand the features and technical content of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the drawings. The attached drawings are only for reference and illustration purposes and are not used to limit the embodiments of the present disclosure. In the following technical description, for the sake of explanation, numerous details are provided to give a thorough understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, well-known structures and devices can be shown in a simplified manner.

[0029] In the embodiments of the present disclosure, terms such as "first" and "second" in the specification and claims of the embodiments and the above drawings are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so as to implement the embodiments of the present disclosure described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion.

[0030] Unless otherwise specified, the term "plurality" means two or more.

[0031] In the embodiments of the present disclosure, the character " / " indicates that the objects before and after are in an "or" relationship. For example, A / B means: A or B.

[0032] The term "and / or" is a description of the associated relationship of an object, indicating that there can be three relationships. For example, A and / or B means: A or B, or, A and B these three relationships.

[0033] The term "corresponding" may refer to an association relationship or a binding relationship. That A corresponds to B means there is an association relationship or a binding relationship between A and B.

[0034] Embodiment 1 As Figure 1 , Figure 2 shown, a neural network structured pruning method based on deep reinforcement learning performs hierarchical adaptive pruning. Considering the limitations of the storage resources (BRAM) and computing resources (DSP) of the FPGA, and comprehensively considering the accuracy, computational amount, and number of parameters of the model after pruning, to achieve the best balance between the performance of the pruned model and resource utilization.

[0035] Specifically, it includes: S1: Obtain the neural network model to be pruned, and construct a multi-dimensional state vector for each layer of the neural network.

[0036] As a refinement of the above embodiment, the input is a pre-trained neural network model. In the initialization stage, a multi-dimensional state vector is constructed for each layer of the neural network, including the static layer attributes of the current layer (such as the current layer type, number of input and output channels, convolution kernel size, convolution stride, number of parameters, flops), the global resource budget (such as the number of parameters and flops removed by the previous layer, the number of parameters and flops remaining in the subsequent layer), and the adjacent layer association information (pruning ratio and pruning strategy of the previous layer). The neural network structure, dynamic resource constraints, and inter-layer dependence relationships are encoded into a state space that can be perceived by the agent , providing a decision basis for the Actor-Critic network to jointly optimize accuracy and efficiency.

[0037] S2: Construct a hierarchical adaptive hybrid pruning model, based on a dual-branch Actor-Critic network architecture that fuses the deep deterministic policy gradient algorithm and Gumbel-Softmax. The Actor network includes a hierarchical feature extraction module and a dual-branch output module, and the Critic network includes a state feature extraction module, an action feature extraction module, and a dual-channel fusion module.

[0038] As a refinement of the above embodiment, the Actor network adopts an architecture of hierarchical feature extraction and dual-branch collaborative output, and the Critic network adopts a dual-channel feature fusion mechanism and a deep value evaluation architecture.

[0039] In neural network structured pruning, the traditional DDPG algorithm outputs the pruning ratio of each layer through a continuous action space, treats all network layers as a homogeneous structure for coarse-grained compression, and ignores the functional heterogeneity and structural sensitivity differences between different levels of the neural network. If different pruning strategies are adopted according to the functional characteristics and hardware sensitivity of different network layers, more refined pruning can be achieved. To meet the requirements of "continuous intensity control" and "discrete policy selection" in the neural network pruning task, the present invention introduces the DDPG fusion Gumbel-Softmax technology to construct a double-branch Actor-Critic network architecture.

[0040] Specifically, the Actor network adopts an architecture of hierarchical feature extraction and double-branch collaborative output, and is responsible for the iterative update of the policy network parameters and is responsible for selecting a deterministic action according to the current environmental state (the multi-dimensional state vector of the layer being processed at the current time step, that is, the current environment where the Agent is located), and directly controls the behavior decision of the intelligent agent.

[0041] The Critic network adopts a dual-channel feature fusion mechanism and a deep value evaluation architecture, and is responsible for the iterative update of the value network parameters , inputs the concatenation of the current environmental state and the action vector , and outputs the action-value estimate to evaluate the long-term reward of the current state-action pair.

[0042] S3: Input the multi-dimensional state vector into a hierarchical adaptive hybrid pruning model, and output the joint action of the continuous pruning ratio and the discrete pruning strategy through the double-branch Actor network, where the continuous branch generates the pruning ratio through the Sigmoid function, and the discrete branch generates the policy probability distribution through Gumbel-Softmax.

[0043] As a refinement of the above embodiment, as Figure 3 shown, it specifically includes: S301: The Actor network takes the current environmental state as the input.

[0044] S302: Perform feature abstraction through two fully connected layers: the first layer maps the high-dimensional state information to the hidden space, introduces non-linearity through the ReLU activation function; the second layer further extracts the inter-layer sensitivity difference features, and then captures the association pattern between the global compression target and the local pruning action through ReLU activation.

[0045] S303: The output features of the second fully connected layer are processed by the ReLU activation function and then input into the third fully connected layer. The third fully connected layer has a dual-branch output structure, and the number of neurons in this layer is equal to the sum of the continuous action dimension and the number of discrete pruning strategies. In this embodiment, the continuous action dimension is set to 1, representing the pruning ratio (a floating-point number) output by the Actor, which only targets the current layer. If the continuous action dimension is n, it represents the pruning ratio (not a single floating-point number, but multiple floating-point numbers) output by the current Actor for multiple layers. The discrete pruning strategy is the probability distribution of each discrete strategy currently set, and the sum of the probabilities is 1. During forward propagation, the original tensor output by the third fully connected layer is decoupled into two physically independent subspaces through dimensional dissection slicing: the first half slice targets the continuous action space and generates continuous values through the Sigmoid function , representing the pruning ratio of the current layer, representing the minimum pruning ratio for each layer, representing the maximum pruning ratio; the second half slice is for discrete strategy selection, and through Gumbel-Softmax reparameterization, the original K-dimensional policy logits vector is transformed into a differentiable policy distribution representing the probability distribution of the K pruning strategies of the current layer, representing a vector space composed of K real numbers, supporting differential strategy selections such as L1 regularization pruning, L2 regularization pruning, and FPGM pruning.

[0046] In the present invention, Gumbel-Softmax technology is used to perform differentiable sampling on discrete decisions. During the training phase, the exploration intensity is dynamically adjusted through the temperature parameter: the smooth probability distribution in the initial high-temperature state promotes the exploration of policy diversity. As the temperature decays exponentially, it gradually approaches deterministic selection to stabilize the policy. These two branches share the gradient signals of the underlying feature extraction layer during backpropagation, enabling the pruning ratio adjustment and policy selection to form collaborative optimization - when the Critic network evaluates that a certain policy combination can significantly improve the compression efficiency, the gradient signal simultaneously corrects the logical parameters of the policy selection and the generation weights of the pruning rates for each layer. This design not only retains the advantages of traditional DDPG for refined continuous control but also breaks through the limitation of a single action type, enabling the intelligent agent to dynamically switch pruning strategies according to the characteristics of the model structure, while precisely regulating the compression intensity of each layer, ultimately achieving the optimal balance between model accuracy and computational efficiency.

[0047] S4: The Critic network quantifies the long-term benefits of joint actions through a hardware-aware composite reward function.

[0048] As a refinement of the above embodiment, as Figure 4 shown, its core structure realizes value function modeling by parallel processing the environmental state and the agent's actions; specifically including: S401: Environmental Status Extract high-dimensional features through a fully connected layer, and form a state feature vector after non-linear activation; S402: The pruning ratio and policy index are encoded into a unified vector to represent the agent's action - the pruning ratio tensor directly represents the compression strength of each layer, and the policy selection is expressed by the relaxed probability vector generated by Gumbel-Softmax to represent the algorithm preference. After splicing, they are mapped into an action feature vector through an independent fully connected layer.

[0049] S403: The state feature vector and the action feature vector are added element by element in the hidden space, so that the correlation between the state and the action is enhanced through implicit feature alignment, which not only retains the encoding ability of the state information for global constraints, but also captures the local effects of specific pruning strategies in different layer structures.

[0050] S404: The fused joint features undergo deep non-linear transformation, and finally a single scalar Q value is output. This value comprehensively reflects the long-term return expectation of executing a specific mixed action in the current state, considering both the global impact brought by discrete policy selection (such as choosing sensitivity pruning may improve parameter utilization but increase computational overhead), and evaluating the local effects of continuous pruning ratio adjustment (such as the risk of accuracy collapse caused by excessive pruning of a certain layer).

[0051] This dual-channel fusion mechanism enables the Critic to provide gradient signals for continuous action fine-tuning and discrete policy optimization to the Actor during backpropagation, ensuring their co-evolution in the trade-off between compression efficiency and model performance. Its value evaluation process essentially constructs a dynamic mapping relationship between the mixed action space and the compression target, providing a reliable direction guidance for the exploration-exploitation balance of the Actor.

[0052] When the Critic network evaluates the Q value of the action pair, a reward function needs to be designed to accurately quantify the long-term return of the action. Aiming at the limitation of "emphasizing software metrics and neglecting hardware mapping" in the existing pruning algorithms, the present invention proposes a hardware-aware composite reward mechanism, and the reward function formula is as follows: , where, , , , , where, represents the reward value based on accuracy, represents the reward value based on flops, represents the reward value based on params, Denotes accuracy rate, Denotes taking logarithm to make the calculation of the reward function more stable, Denotes the number of floating-point operations of the model after pruning according to the current pruning ratio and pruning strategy, Denotes the number of floating-point operations of the original model.

[0053] When traditional methods optimize the model, they consider a linear relationship between FLOPs and Params, ignoring the hardware sensitivity differences between them corresponding to the DSP computing unit and BRAM storage resources in the FPGA platform, resulting in an imbalance problem of "overestimated computational acceleration benefits and unfulfilled storage optimization potential" when the pruning strategy is deployed on hardware. This function innovatively reconstructs the reward mechanism from three dimensions: hardware resource decoupling, dynamic weight adaptation, and multi-objective collaborative optimization. First, based on the resource bottleneck characteristics of DSP and BRAM in the FPGA, FLOPs and Params are respectively modeled as independent reward terms, where Params correspond to the storage resources in the FPGA and Flops correspond to the computing resources in the FPGA. By introducing the reduced params and flops, and the remaining params and flops, the reinforcement learning strategy can dynamically adjust the compression ratio of the subsequent layers, avoiding both the accuracy degradation caused by excessive pruning in the previous layer and the excessive pruning in the subsequent layer caused by insufficient pruning in the previous layer, and being able to well coordinate the pruning ratios of all layers. Second, based on the training cycle, the optimization weights are dynamically adjusted, focusing on the hardware adaptability optimization of FLOPs / Params in the initial stage of training, and gradually increasing the weight proportion of accuracy through an exponential decay function, ensuring both the efficient utilization of early hardware resources and avoiding the accuracy collapse caused by excessive compression in the later stage.

[0054] S5: Iteratively train the hierarchical adaptive hybrid pruning model, execute the pruning action according to the pruning ratio and pruning strategy of each layer, perform non-destructive pruning on the model through the mask matrix, and select the pruning strategy of the round with the highest reward as the optimal strategy.

[0055] The overall process is as Figure 5 shown. The agent processes the network layer by layer: for each layer , the agent receives an environmental state representing the useful features of that layer , and outputs the pruning ratio and policy probability distribution of the current layer, that is, the action . After pruning the th layer , the agent enters the next layer . When the policy search is completed, the best exploration model will be fine-tuned to achieve the best performance.

[0056] S6: Prune the neural network according to the optimal strategy, and deploy it on the FPGA after export and fine-tuning.

[0057] It should be noted that: The DDPG algorithm realizes the collaborative optimization of policies and values through the Actor-Critic framework. Its core advantage lies in using the value gradient of the Critic network to guide the improvement of the deterministic policy of the Actor network, and at the same time stabilizing the training process by means of the soft update mechanism of the target network. In the present invention, the dual-branch shared underlying feature encoding network establishes a dynamic matching rule between the pruning ratio and the policy type through implicit association learning. The agent breaks through the decision-making bottleneck of a single action dimension, and dynamically matches the optimal pruning ratio-pruning strategy combination according to the characteristics of the layer structure, which not only ensures the sensitivity and pruning accuracy of the key layer, but also realizes the dynamic sparse and efficient compression of the redundant layer.

[0058] Example 2 This example elaborates on the above solution in the field of real-time crop recognition on the agricultural drone side.

[0059] The real-time crop recognition technology on the agricultural drone side has developed rapidly due to the demand for efficient and low-cost field monitoring in precision agriculture. The drone is equipped with a high-resolution multispectral camera to collect farmland images, and real-time identifies the growth status of crops such as corn, wheat, and rice to guide variable fertilization and pest control. Existing systems mostly rely on cloud processing, but the farmland network coverage is poor, resulting in a decision-making delay of up to 3-5 seconds, missing the best timing for spraying pesticides. The Xilinx Kria KV260 FPGA edge computing module installed on mainstream agricultural drones (such as DJI Agras T40) only has 4.5MB BRAM, 252 DSP units and 141K LUT resources, and cannot carry a conventional CNN model. Therefore, in order to meet the requirements of low-latency and high-precision real-time recognition under the limited computing resources and power consumption constraints of edge computing devices, the present invention proposes a neural network structured pruning method based on deep reinforcement learning, specifically as follows: Step 1: Pruning strategy search based on DDPG integrated with Gumbel-Softmax Taking the multispectral crop image dataset collected in the field (including near-infrared-RGB fusion images of corn leaf spot, wheat rust, and rice blast) as the input, and dynamically adjusting various search parameters according to the storage and computing limitations of the target device and the flight altitude of the drone, such as the target compression rate, the per-layer pruning lower limit, the reward function selection, the DDPG exploration noise coefficient, the Gumbel-Softmax temperature, etc., to seek the Pareto optimal solution of the model's accuracy, real-time inference speed, and hardware resource occupancy.

[0060] Initialize the state space layer_embedding, the experience pool , the Critic network parameters , the Actor network parameters , Critic-Target network parameters , Actor-Target network parameters .

[0061] In the warm-up stage, samples are generated by completely random actions to fill the experience pool, providing diverse initial data for subsequent training to ensure training stability and the basis for policy optimization.

[0062] In the training stage, the agent executes actions to obtain environmental rewards and new states , and transfer the samples to the experience pool . The Critic network randomly samples historical data of the batch size value N from the experience pool , evaluates the state-action pair , and generates an estimated Q value . The Actor-Target network outputs the next action , the Critic-Target network evaluates the Q value of this action , substitutes it into the Bellman equation to calculate the target Q value : , where determines the attenuation degree of future rewards in the current state value and are equivalent.

[0063] Calculate the mean square error between the target Q value and the current Q value : , where is the variance between the target Q value and the current Q value when the environmental state is s and the agent executes action a.

[0064] Gradually approximate the optimal value estimation of the Bellman equation by minimizing the loss function through gradient descent.

[0065] The Actor network interacts with the environmental state , the output branch 1 outputs the pruning ratio through sigmoid , the output branch 2 outputs the policy distribution through Gumbel-Softmax , generates action pairs , and constructs a hybrid action search space for lightweight real-time recognition.

[0066] The Critic network evaluates the value of the state-action pair​ , the gradient of the Critic network with respect to the action a is passed to the Actor network parameters via the chain rule , and the policy is updated in the direction of increasing Q-value. The Actor network parameters are updated by gradient ascent: , to maximize the expectation: , making the actions generated in the future more likely to obtain high Q-values in the same state.

[0067] The Actor-Target network is responsible for selecting the optimal next action according to the next state sampled from the experience replay pool , and the network parameters are periodically copied from and gradually synchronized through soft updates: , where is used to control the speed at which the target network parameters approach the online network parameters.

[0068] The Critic-Target network is responsible for outputting to calculate the target Q-value: , The Critic-Target network parameters are periodically copied from and gradually synchronized through soft updates: .

[0069] In each round of training, pruning actions are performed according to the pruning ratio and pruning strategy for each layer. The model is non-destructively pruned through the mask matrix. The flops and params after pruning for each layer are calculated according to the pruning ratio to measure the compression rate and reward. The pruning strategy of the round of training with the highest reward is selected as the optimal strategy.

[0070] Step 2: Export First, load the optimal strategy file optimized for agricultural tasks generated in the search phase. This strategy has explored the optimal pruning ratio and pruning strategy for each layer under the hardware resource constraints on the drone side through deep reinforcement learning. Calculate the importance scores of each channel along the input channel dimension according to the pruning strategy, sort the channels in descending order of importance scores, retain the top K channels according to the pruning ratio of the current layer, generate a boolean mask, mark the retained channels as True, and mark the removed channels as False. Align the output channels of the previous layer with the new input channels of the current layer, solve the new weights by the least squares method, and adjust the reconstructed weights to match the shape of the pruned structure. Load the reconstructed weight tensor into the current layer, and physically remove the channels marked as False based on the mask to complete the layer-by-layer physical pruning, and export the pruned lightweight neural network model.

[0071] Step 3: Precision recovery and adaptive tuning of online enhancement In the fine-tuning stage of the drone agricultural application, the training process is adaptively adjusted in close accordance with the characteristics of field operations. For the crop images collected by the multispectral camera, simulate the complex field environment, apply random occlusion (simulating leaf overlap), spectral shift (simulating light changes), and Gaussian noise (simulating sensor noise) to the training images to improve the generalization ability of the model. Taking the cotton Verticillium wilt leaves as an example, the area of the randomly erased region is controlled within 10% - 25% to simulate the typical occlusion ratio of leaf clusters. In the hardware-side deployment verification link, when the measured inference latency exceeds the target threshold, first tighten the pruning rate of the shallow convolutional layer - because the shallow feature extraction layer plays a decisive role in crop textures (such as the edges of rice blast spots).

[0072] When the detection accuracy of the validation set (for small target diseases such as wheat stripe rust) is lower than 85%, start the key layer reverse recovery mechanism: sort the last spatial resolution retention layer according to channel sensitivity, reactivate the channels among the top 20% pruned channels with a correlation higher than 0.7 with the disease spot color feature (response of the a channel in the LAB color space), and make them participate in gradient update through dynamic mask expansion. The spectral consistency constraint term introduced in the optimization target forces the variance of the feature responses of the pruned model in the three agricultural characteristic bands of 450nm (blue light), 680nm (red light), and 720nm (near-infrared) not to exceed 1.3 times that of the original model, ensuring the stability of key functions such as chlorophyll content estimation. The finally generated lightweight model maintains a disease recognition accuracy of 85.66% while meeting the stringent power consumption and memory requirements of the drone side hardware, and can maintain a real-time analysis ability of 15 frames per second under 6-level wind interference.

[0073] In the real-scene test of the wheat field, the drone equipped with this system (flight altitude 10m, speed 8m / s) achieved the following performance improvements: Recognition accuracy: The mAP of leaf spot disease recognition decreased from 91.2% before pruning to 85.66%, meeting the requirements of field monitoring. A small number of missed detections can be compensated by repeated inspections using drones.

[0074] Real-time performance: The end-to-end processing delay was reduced from 3.2 s in the cloud solution to 301 ms, meeting the positioning accuracy requirements of ±0.5 m at a flight speed of 10 m / s.

[0075] Hardware efficiency: The model storage occupancy was compressed from the original 4.232 MB to 1.389 MB, and the number of floating-point calculations of the model was compressed from the original 12.714 M to 4.718 M, meeting the deployment limitations on the drone side.

[0076] Embodiment 3 Combined Figure 6 As shown, the present disclosure provides a neural network structured pruning device 300 based on deep reinforcement learning, including a processor 304 and a memory 301. Optionally, the device may further include a communication interface 302 and a bus 303. Among them, the processor 304, the communication interface 302, and the memory 301 can complete mutual communication through the bus 303. The communication interface 302 can be used for information transmission. The processor 304 can call the logical instructions in the memory 301 to execute the neural network structured pruning method based on deep reinforcement learning in the above embodiment.

[0077] In addition, when the logical instructions in the above-mentioned memory 301 are implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium.

[0078] The memory 301, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as the program instructions / modules corresponding to the methods in the embodiments of the present disclosure. The processor 304 executes functional applications and data processing by running the program instructions / modules stored in the memory 301, that is, implements the neural network structured pruning method based on deep reinforcement learning in the above embodiment.

[0079] The memory 301 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 301 may include a high-speed random access memory and may also include a non-volatile memory.

[0080] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, and the computer-executable instructions are configured to execute the above-mentioned neural network structured pruning method based on deep reinforcement learning.

[0081] The above computer-readable storage medium may be a transient computer-readable storage medium or a non-transient computer-readable storage medium.

[0082] The technical solution of the embodiment of the present disclosure may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present disclosure. The foregoing storage medium may be a non-transient storage medium, including: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs, or may also be a transient storage medium.

[0083] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process, and other changes. The embodiments only represent possible variations. Unless explicitly required, the individual components and functions are optional, and the order of operations can vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terms used in this application are only for describing the embodiments and do not limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to also include the plural forms. Similarly, as used in this application, the term "and / or" refers to any and all possible combinations including one or more of the associated listed items. Additionally, when used in this application, the term "comprise" and its variants "comprises" and / or "comprising" etc. mean the presence of the stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups of these. Without further limitation, an element defined by the statement "comprising one..." does not exclude the presence of additional identical elements in the process, method, or apparatus including the element. In this document, each embodiment may focus on the differences from other embodiments, and the same or similar parts among the embodiments can be referred to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method parts disclosed in the embodiments, the relevant parts can refer to the description of the method parts.

Claims

1. A neural network structured pruning method based on deep reinforcement learning, characterized in that: The following steps are involved: Obtain the neural network model to be pruned and construct a multi-dimensional state vector for each layer of the neural network; Construct a hierarchical adaptive hybrid pruning model based on a dual-branch Actor-Critic network architecture that combines a deep deterministic policy gradient algorithm with Gumbel-Softmax. The Actor network includes a hierarchical feature extraction module and a dual-branch output module, and the Critic network includes a state feature extraction module, an action feature extraction module, and a dual-channel fusion module. The multi-dimensional state vector is input into the hierarchical adaptive hybrid pruning model, and the joint action of the continuous pruning ratio and the discrete pruning strategy is output through the dual-branch Actor network. The continuous branch generates the pruning ratio through the Sigmoid function, and the discrete branch generates the strategy probability distribution through Gumbel-Softmax. The critic network quantifies the long-term benefits of joint actions through a hardware-aware composite reward function; Iteratively train the layer-adaptive hybrid pruning model, perform pruning actions according to the pruning ratio and pruning strategy of each layer, and select the pruning strategy with the highest reward in the round of training as the optimal strategy; The neural network is pruned according to the optimal strategy, exported and fine-tuned, and then deployed on the FPGA.

2. The neural network structured pruning method based on deep reinforcement learning according to claim 1, characterized in that: The multidimensional state vector contains the static properties of the current layer, the global resource budget, and the association information of adjacent layers.

3. The neural network structured pruning method based on deep reinforcement learning according to claim 1, characterized in that: The Actor network takes a multi-dimensional state vector as input, outputs the pruning ratio through continuous branches, and outputs the strategy probability distribution through discrete branches; including: Hierarchical feature extraction module: The input multi-dimensional state vector is sequentially passed through the first fully connected layer and the second fully connected layer to capture the correlation features between the sensitivity difference between the layers and the global compression target; specifically, the first fully connected layer maps the high-dimensional state information to the hidden space, and then introduces nonlinearity through the ReLU activation function; the output features of the second fully connected layer further extract the sensitivity difference features between the layers, and then capture the correlation pattern between the global compression target and the local pruning action through ReLU activation; Dual-branch output module: The output features of the hierarchical feature extraction module are taken as input, processed by the third fully connected layer, and the output tensor is decoupled through dimensional dissection cutting. The first half is the dimension of the continuous action space. For the continuous action space, the pruning ratio is generated through the Sigmoid function; the second half is the number of categories in the discrete strategy space. For discrete strategy selection, it is converted into a strategy probability distribution through Gumbel-Softmax reparameterization.

4. The neural network structured pruning method based on deep reinforcement learning according to claim 3, characterized in that: The number of neurons in the third fully connected layer is equal to the sum of the continuous action dimensions and the number of discrete pruning strategies.

5. The neural network structured pruning method based on deep reinforcement learning according to claim 3, characterized in that: The temperature parameter is used to dynamically adjust the exploration intensity of Gumbel-Softmax. A high temperature coefficient is set at the beginning of training to enhance the diversity of strategies, and the temperature coefficient is attenuated as the training progresses to stabilize the strategy. The gradient signals of the feature extraction layer are shared during the back propagation process between the continuous branch and the discrete branch, so that the pruning ratio adjustment and strategy selection are optimized collaboratively.

6. The neural network structured pruning method based on deep reinforcement learning according to claim 1, characterized in that: The Critic network takes a multidimensional state vector and an action vector as input, outputs an action-value estimate, and evaluates the long-term benefit of the current state-action pair. The action vector is a joint action of the spliced ​​pruning ratio and the strategy probability distribution; including: State feature extraction module: extract high-dimensional features from the multi-dimensional state vector through a fully connected layer, and map it to a state feature vector through nonlinear activation; Action feature extraction module: maps the action vector to an action feature vector through a fully connected layer; Dual-channel fusion module: adds the state feature vector and the action feature vector element by element, and outputs the Q value through deep nonlinear transformation.

7. The neural network structured pruning method based on deep reinforcement learning according to claim 1, characterized in that: The hardware-aware composite reward function is as follows: ， ， ， ， ， in, represents the reward value based on accuracy, represents the reward value based on flops, Indicates the reward value based on params, Indicates accuracy, Indicates taking the logarithm to make the calculation of the reward function more stable. Indicates the number of floating-point operations of the model after pruning according to the current pruning ratio and pruning strategy. Indicates the number of floating point operations of the original model.

8. The neural network structured pruning method based on deep reinforcement learning according to claim 1, characterized in that: In the fine-tuning stage, when the detection accuracy of the verification set is lower than the set threshold, the key layer reverse recovery mechanism is used.

9. A neural network structured pruning device based on deep reinforcement learning, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the neural network structured pruning method based on deep reinforcement learning as described in any one of claims 1-8 when running the program instructions.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, which, when executed by a processor, implements a neural network structured pruning method based on deep reinforcement learning as described in any one of claims 1 to 8 above.

Images