An automatic pruning and quantization method and system based on a reinforcement learning framework

Abstract

The present application relates to the technical field of deep learning in general, and proposes an automatic pruning quantization method and system based on a reinforcement learning framework. The method comprises: constructing an environment for the automatic pruning quantization method, the environment being based on static information related to a network structure and dynamic information related to a compression state; constructing an automatic pruning quantization agent; and enabling the automatic pruning quantization agent to interact with the environment to train the automatic pruning quantization agent and output a compression strategy. The present application can simultaneously search for optimal layer-by-layer pruning and layer-by-layer quantization strategies for both light-weight and non-light-weight image recognition neural network models. The present application greatly reduces the search completion time through a new model update algorithm after pruning and optimal strategies. Furthermore, the present application improves the recognition accuracy of the compressed model while reducing the model parameter quantity and computational complexity.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning in general. In particular, the present application relates to an automatic pruning and quantization method and system based on a reinforcement learning framework. BACKGROUND

[0002] Convolutional Neural Network (CNN) is an important algorithm in the field of deep learning, and is widely used in the field of computer vision. However, the convolutional neural network has problems of complex model, redundant parameter quantity and huge calculation amount, which makes it not easy to deploy on mobile terminal devices such as smart phones, wearable smart embedded devices and other resource-limited mobile terminal devices. Therefore, when designing and deploying the convolutional neural network, it is necessary to consider how to reduce the parameter quantity and calculation amount while maintaining high performance.

[0003] To solve the above problems, the optimization scheme proposed by the prior art includes:

[0004] 1) Design efficient network structure, that is, design lightweight model, reduce the parameter quantity of the model through special convolution structure. For example, Google proposed MobileNet model in 2017 (Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv: 1704.04861, 2017.) and its upgraded version MobileNetV2 model (Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. MobileNetV2: Inverted residuals and linear bottlenecks. CVPR, 2018.), MobileNetV3 model (Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, Hartwig Adam. Searching for MobileNetV3. ICCV 2019) and so on.

[0005] 2) Compress the existing network through pruning, quantization and other technologies to reduce the parameter quantity and calculation of the model. For example, Song Han of MIT proposed Deep Compression in 2016 (Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. ICLR, 2016.) and so on, in which the existing model is compressed through pruning and quantization based on Huffman coding.

[0006] The two technical solutions above are orthogonal, that is, model compression on a lightweight model will result in a neural network more suitable for mobile devices. However, as the complexity and parameter amount of the network model structure continue to increase, the performance of the artificially designed pruning and quantization strategy will often degrade when facing new models. Therefore, model compression strategies based on AutoML technology have attracted the attention of researchers. In 2018, the SongHan experimental group proposed a model pruning strategy based on reinforcement learning--AutoML model pruning technology AMC (Y. He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han. AMC: Automl for model compression and acceleration on mobile devices. ECCV, 2018.) and a neural network quantization technology based on reinforcement learning, including ReLeQ (Ahmed T. Elthakeb, Prannoy Pilligundla, Amir Yazdanbakhsh, Sean Kinzer, and Hadi Esmaeilzadeh. ReLeQ: A reinforcement learning approach for deep quantization of neural networks. CoRR, abs / 1811.01704, 2018), HAQ (K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han, Haq: Hardware-aware automated quantization with mixed precision. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.), etc. However, the existing technology still has the problems of long search completion time and low recognition accuracy after model compression. SUMMARY

[0007] To at least partially solve the above problems in the prior art, the present application proposes an automatic pruning and quantization method based on a reinforcement learning framework, comprising the following steps:

[0008] An environment for the automatic pruning and quantization method is constructed, which is based on static information related to the network structure and dynamic information related to the compression state;

[0009] An automatic pruning and quantization agent is constructed; and

[0010] The automatic pruning and quantization agent interacts with the environment to train the automatic pruning and quantization agent and output a compression strategy, the compression strategy including a pruning strategy and a quantization strategy.

[0011] In an embodiment of the present application, the static information includes:

[0012] a layer index including a t-th layer of the current compression network;

[0013] a layer dimension including an input data size in, an output data size out, a convolution kernel size k, and a stride size stride, and a model size including a total parameter quantity n of the network param ; and

[0014] a model size including a total parameter quantity n of the network param ; and / or

[0015] The dynamic information includes:

[0016] pruning information including a total floating point operation number reduced of a pruned layer FLOPs and a total floating point operation number rest of a non-pruned layer FLOPs ;

[0017] quantization information including a quantized bit parameter quantity reduced wszie and a non-quantized bit parameter quantity rest wsize ; and

[0018] strategy information including a pruning strategy of a t-1-th layer of the neural network model a weight quantization strategy and an activation value quantization strategy

[0019] In an embodiment of the present application, the automatic pruning and quantization agent is constructed based on a deep deterministic policy gradient algorithm, wherein the automatic pruning and quantization agent includes:

[0020] a policy network including first to third fully connected layers, first to second rectified linear unit activation layers, and a first sigmoid activation layer; and

[0021] an evaluation network including fourth to seventh fully connected layers and third to fourth rectified linear unit activation layers.

[0022] In an embodiment of the present application, the automatic pruning and quantization agent interacting with the environment includes:

[0023] In the information collection stage, a collection strategy experience is sampled by the automatic pruning and quantization agent and the network is not updated, wherein the collected tuple {rn , s t , s' t , a t , δ t} into an experience replay set, wherein r n represents a reward of the last layer, s t represents an environment before pruning, s' t represents an environment after quantization, a t represents pruning and quantization strategies, δ t is used to indicate whether it is the last layer; and

[0024] In the training update phase, the automatic pruning and quantization agent generates pruning strategies and quantization strategies simultaneously according to environment information through the policy network, and adds noise through normal distribution truncation.

[0025] In an embodiment of the present application, it is specified that the automatic pruning and quantization agent generates pruning strategies and quantization strategies simultaneously according to environment information through the policy network includes the following actions performed by the automatic pruning and quantization agent:

[0026] generating pruning and quantization strategies according to the environment before pruning in the tth layer;

[0027] using the pruning strategy of the tth layer interacting with the environment, and generating weight quantization strategies and activation value quantization strategies according to the environment after pruning and

[0028] merging the strategies to generate compression strategies of the tth layer

[0029] In an embodiment of the present application, it is specified that training the automatic pruning and quantization agent and outputting compression strategies includes the following actions performed by the automatic pruning and quantization agent:

[0030] initializing a current evaluation network C, outputting Q values and a current policy network μ, outputting strategies a t

[0031] copying the above current network to obtain a target evaluation network C', outputting values and a target policy network μ' for updating;

[0032] sampling m samples from the experience replay set;

[0033] using environment information s t before pruning and compression strategies a t calculating Q values through the current evaluation network, represented as Q=C(s t , a t );​

[0034] Based on the quantified environmental information s′ t The reward for the last layer is r n And the next-layer compression policy a predicted by the target policy network. t+1 =μ′(s′ t The target evaluation network C′ is used to calculate... The value is expressed as follows:

[0035]

[0036] Minimize the mean squared error function To optimize the objective, the current evaluation network C is updated using gradient descent and backpropagation;

[0037] Use the updated current evaluation network C, and based on minimizing -C(s) t ,μ(s t )) Calculate the gradient, and update the current policy network μ using gradient descent and backpropagation methods; and

[0038] The updated objective evaluation network C′ and objective policy network μ′ are expressed as follows:

[0039]

[0040] Where τ represents the hyperparameter.

[0041] In one embodiment of the present invention, the automatic pruning quantization agent, after providing the pruning strategy for each layer, performs channel selection based on the L1 norm and reconstructs the feature map using the least squares method; and / or

[0042] After providing the quantization strategy for each layer, the automatic pruning quantization agent uses a linear quantization method to quantize the weights and activation values, as shown in the following formula:

[0043] X = clamp(X, -α, α),

[0044]

[0045]

[0046] Where α represents the truncation value, sf represents the scaling factor, bit represents the quantization bit width, "·" represents rounding, X represents the original parameter value, and q X This represents the quantized parameter values; it includes: performing histogram statistics on the weights and input data to obtain the corresponding frequencies and group endpoints; and calculating the quantized data range n=2 based on the bit width b given by the reinforcement learning algorithm. b-1Furthermore, it provides possible quantization cutoff values ​​and corresponding scaling factors between n+1 and the histogram extrema; it estimates the quantized values ​​for different quantization cutoff values ​​using linear interpolation, based on the noise-to-signal ratio. Determine the optimal cutoff value; and perform quantization using the cutoff value and scaling factor determined in step three; and / or

[0047] The automatic pruning quantization agent, after providing the quantization strategy for each layer, uses the SAT quantization method for weight quantization and the PACT quantization method for activation value quantization.

[0048] The SAT quantization method is expressed as follows:

[0049]

[0050]

[0051]

[0052] Where, q W * It is the weight value after weight quantization, n out This represents the number of outputs for each layer of the model;

[0053] The quantization formula for the PACT method is expressed as follows:

[0054] PACT(X) = clamp(X, -α, α); and / or

[0055] After providing the quantization strategy for each layer, the automatic pruning quantization agent uses a PWLQ quantization network to quantize the weights and activation values, where the parameter quantization formula is expressed as follows:

[0056]

[0057]

[0058]

[0059] q X =q X ×sf

[0060] This includes: finding a breakpoint p in the data during weight quantization, dividing the weight values ​​into three intervals: [-m, -p], [-p, p], and [p, m]; using unsigned uniform quantization for weights distributed in [-m, -p] and [p, m], and signed uniform quantization for weights in [-p, p], using the formula p / m = ln(0.8614m + 0.6079) to determine the breakpoint, where p is the breakpoint and m is the absolute value of the maximum and minimum weight values; and using multiple random training images as a calibration set during activation value quantization, calculating the median of the first training images as the quantization cutoff value, and performing uniform quantization.

[0061] This invention also proposes an automatic pruning quantization system based on a reinforcement learning framework, comprising:

[0062] An environment generation module is configured to construct an environment for an automatic pruning quantization method, the environment being based on static information related to the network structure and dynamic information related to the compression state.

[0063] An agent generation module, configured to construct automatically pruned and quantized agents; and

[0064] An interactive training module is configured to enable the automatic pruning and quantization agent to interact with the environment to train the automatic pruning and quantization agent and output a compression strategy, the compression strategy including a pruning strategy and a quantization strategy.

[0065] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, the computer program performing the steps according to the method when executed by a processor.

[0066] The present invention also proposes a computer system, comprising:

[0067] A processor, configured to execute machine-executable instructions; and

[0068] A memory having machine-executable instructions stored thereon, which, when executed by a processor, perform the steps according to the method.

[0069] The present invention has at least the following beneficial effects: The present invention proposes an automatic pruning method and system based on a reinforcement learning framework. Compared with the prior art, the present invention can simultaneously search for the optimal layer-by-layer pruning and layer-by-layer quantization strategies for both lightweight and non-lightweight image recognition neural network models; and the present invention significantly reduces the search completion time through a new model update algorithm and optimal strategy after pruning, thereby greatly reducing hardware usage and power consumption; in addition, the present invention improves the recognition accuracy of the compressed model while reducing the number of model parameters and computational complexity. Attached Figure Description

[0070] To further illustrate the advantages and other features of the various embodiments of the present invention, a more specific description of the embodiments of the present invention will be presented with reference to the accompanying drawings. It is understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by the same or similar reference numerals for clarity.

[0071] Figure 1 A computer system implementing the system and / or method according to the present invention is shown.

[0072] Figure 2 The diagram illustrates a flowchart of an automatic pruning and quantization method based on a reinforcement learning framework, according to one embodiment of the present invention.

[0073] Figure 3 A schematic diagram of the interaction process between the APQ agent and the environment is shown in one embodiment of the present invention. Detailed Implementation

[0074] It should be noted that the components in the various figures may be shown exaggeratedly for illustrative purposes and are not necessarily to scale. In each figure, the same reference numerals are used for components that are identical or have the same function.

[0075] In this invention, unless otherwise specified, "arranged on," "arranged above," and "arranged on" do not exclude the possibility of an intermediate element between them. Furthermore, "arranged on or above" merely indicates the relative positional relationship between two components, and in certain cases, such as when the product orientation is reversed, it can also be converted to "arranged below or under," and vice versa.

[0076] In this invention, the various embodiments are merely intended to illustrate the solutions of the invention and should not be construed as limiting.

[0077] In this invention, unless otherwise specified, the quantifiers “a” and “one” do not exclude scenarios involving multiple elements.

[0078] It should also be noted that, in the embodiments of the present invention, only a portion of the components or parts may be shown for clarity and simplicity. However, those skilled in the art will understand that, under the teachings of the present invention, necessary components or parts can be added as needed for specific scenarios. Furthermore, unless otherwise stated, features in different embodiments of the present invention can be combined with each other. For example, a feature in the second embodiment can replace a corresponding or functionally identical or similar feature in the first embodiment, and the resulting embodiment will also fall within the scope of disclosure or description of this application.

[0079] It should also be noted that, within the scope of this invention, the terms "same," "equal," and "equal to" do not imply that the two values ​​are absolutely equal, but rather allow for a certain reasonable margin of error. In other words, the terms also encompass "substantially the same," "substantially equal," and "substantially equal to." Similarly, in this invention, the directional terms "perpendicular to," "parallel to," etc., also encompass the meanings of "substantially perpendicular to" and "substantially parallel to."

[0080] Furthermore, the numbering of the steps in the methods of the present invention does not limit the execution order of the method steps. Unless otherwise specified, the method steps may be executed in different orders.

[0081] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0082] Figure 1 A computer system 100 implementing the system and / or method according to the present invention is shown. Unless otherwise specified, the method and / or system according to the present invention can be implemented in... Figure 1 The invention may be implemented in the computer system 100 shown to achieve the objectives of the invention, or the invention may be implemented in a distributed manner in multiple computer systems 100 according to the invention via a network, such as a local area network or the Internet. The computer system 100 of the invention may include various types of computer systems, such as handheld devices, laptop computers, personal digital assistants (PDAs), multiprocessor systems, microprocessor-based or programmable consumer electronic devices, network PCs, minicomputers, mainframes, network servers, tablet computers, etc.

[0083] like Figure 1 As shown, the computer system 100 includes a processor 111, a system bus 101, a system memory 102, a video adapter 105, an audio adapter 107, a hard disk drive interface 109, an optical drive interface 113, a network interface 114, and a Universal Serial Bus (USB) interface 112. The system bus 101 can be any of several bus architecture types, such as a memory bus or memory controller, a peripheral bus, and a local bus using various bus architectures. The system bus 101 is used for communication between the various bus devices. In addition to... Figure 1Other bus devices or interfaces besides those shown are also conceivable. System memory 102 includes read-only memory (ROM) 103 and random access memory (RAM) 104, wherein ROM 103 may, for example, store basic input / output system (BIOS) data for basic routines used to implement information transfer at startup, while RAM 104 provides the system with faster access speed running memory. Computer system 100 also includes hard disk drive 109 for reading and writing to hard disk 110, optical drive interface 113 for reading and writing to optical media such as CD-ROM, etc. Hard disk 110 may, for example, store operating system and application programs. Drives and their associated computer-readable media provide non-volatile storage for computer-readable instructions, data structures, program modules, and other data for computer system 100. Computer system 100 may also include a video adapter 105 for image processing and / or image output, which is used to connect output devices such as display 106. The computer system 100 may also include an audio adapter 107 for audio processing and / or audio output, which is used to connect output devices such as speakers 108. Furthermore, the computer system 100 may also include a network interface 114 for network connectivity, wherein the network interface 114 can be connected to the Internet 116 via a network device such as a router 115, wherein the connection can be wired or wireless. Additionally, the computer system 100 may also include a Universal Serial Bus (USB) interface 112 for connecting peripheral devices, wherein the peripheral devices include, for example, a keyboard 117, a mouse 118, and other peripheral devices such as microphones, cameras, etc.

[0084] When the present invention is Figure 1 When implemented on the computer system 100, the optimal layer-by-layer pruning and quantization strategies can be searched simultaneously for both lightweight and non-lightweight image recognition neural network models. Furthermore, the present invention significantly reduces the search completion time through a new pruned model update algorithm and the optimal strategy. In addition, the present invention improves the recognition accuracy of the compressed model while reducing the number of model parameters and computational complexity.

[0085] Furthermore, the embodiments can be provided as computer program products that may include one or more machine-readable media on which machine-executable instructions are stored, which, when executed by one or more machines such as a computer, computer network, or other electronic equipment, may cause one or more machines to perform operations according to the embodiments of the present invention. Machine-readable media may include, but are not limited to, floppy disks, optical disks, CD-ROMs (compact disc read-only memory) and magneto-optical disks, ROMs (read-only memory), RAMs (random access memory), EPROMs (erasable programmable read-only memory), EEPROMs (electrically erasable programmable read-only memory), magnetic or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions.

[0086] Furthermore, various embodiments can be downloaded as computer program products, wherein the program can be transmitted from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a communication link (e.g., a modem and / or a network connection) using one or more data signals implemented and / or modulated by a carrier wave or other propagation medium. Therefore, the machine-readable medium used herein may include such a carrier wave, but this is not required.

[0087] In this invention, the modules of the system according to the invention can be implemented using software, hardware, firmware, or a combination thereof. When a module is implemented using software, its function can be implemented through computer program flow. For example, the module can be implemented using code segments (such as code segments in languages ​​like C and C++) stored in a storage device (such as a hard disk, memory, etc.), wherein the corresponding function of the module can be implemented when the code segment is executed by a processor. When a module is implemented using hardware, its function can be implemented by setting a corresponding hardware structure. For example, the module's function can be implemented by hardware programming a programmable device such as a field-programmable gate array (FPGA), or by designing an application-specific integrated circuit (ASIC) that includes multiple transistors, resistors, capacitors, and other electronic devices. When a module is implemented using firmware, the module's function can be written into a read-only memory such as an EPROM or EEPROM in the form of program code, and the corresponding function of the module can be implemented when the program code is executed by a processor. In addition, some functions of the module may need to be implemented by separate hardware or by working in cooperation with the hardware. For example, the detection function is implemented by the corresponding sensor (such as a proximity sensor, accelerometer, gyroscope, etc.), the signal transmission function is implemented by the corresponding communication device (such as a Bluetooth device, infrared communication device, baseband communication device, Wi-Fi communication device, etc.), the output function is implemented by the corresponding output device (such as a display, speaker, etc.), and so on.

[0088] Figure 2The diagram illustrates a flowchart of an Automatic Pruning Quantization (APQ) method based on a reinforcement learning framework, according to one embodiment of the present invention. Figure 2 As shown, the method may include the following steps:

[0089] Step 201: Construct an environment for the automatic pruning quantization method, the environment being based on static information related to the network structure and dynamic information related to the compression state.

[0090] Step 202: Construct an automatic pruning quantification agent.

[0091] Step 203: Enable the automatic pruning and quantization agent to interact with the environment to train the automatic pruning and quantization agent and output a compression strategy, the compression strategy including a pruning strategy and a quantization strategy.

[0092] The reinforcement learning method of this invention uses the Deep Deterministic Policy Gradient (DDPG) algorithm.

[0093] During the data collection phase, the APQ agent interacts twice with the current layer (t) of the model. In the first interaction, the policy network uses the information (st) from the model to provide a pruning strategy. During the second interaction, the policy network utilizes the pruned model information (s′). t Provide a weight quantization strategy Activation value quantization strategy and corresponding rewards (r) t Then, the pruning strategy and the quantization strategy are merged into a compression strategy for the current layer. Simultaneously, record (r) in the experience pool n s t ,s′ t a t δ t ), where r n This represents the reward of the last layer of the model.

[0094] During the network update phase, the network's utilization of pre-pruning environmental information is evaluated. t And compression strategy a t The Q-value is calculated, and the policy network utilizes the quantized environmental information s′. t Predict the compression strategy for the next layer a t+1 ,calculate Using Q and Update the evaluation network. Calculate the results using the updated evaluation network based on the pre-pruning environmental information s. tThe corresponding policy μ(s) given by the policy network t To update the policy network.

[0095] In its environmental design, this invention considers both static information related to network structure and dynamic information related to compression state. The static information includes the current layer index t, input size in, output size out, stride, kernel size k, and number of parameters n. param Dynamic information includes reduced floating-point operands. FLOPs The remaining floating-point operands rest FLOPs The number of quantized bit parameters has been reduced. wszie Unquantized bit parameter quantity rest wsize And the compression strategy of the previous layer. and

[0096] In other words

[0097] In one embodiment of the present invention, the compression strategy output by this method can compress the network model. Compared with the prior art, the present invention can simultaneously search for the optimal layer-by-layer pruning and layer-by-layer quantization strategies for both lightweight and non-lightweight image recognition neural network models. Furthermore, the present invention significantly reduces the search completion time through a new pruned model update algorithm and the optimal strategy, thereby greatly reducing hardware usage and power consumption. In addition, the present invention improves the recognition accuracy of the compressed model while reducing the number of model parameters and computational complexity.

[0098] The steps of this method are described in detail below with reference to specific embodiments.

[0099] When training a reinforcement learning model, the Automatic Pruned Quantization (APQ) agent responsible for learning the output policy needs to interact with the environment to obtain environmental information for learning. For the network model compression task, in step 201, this invention provides a dedicated environment to obtain the information required for the APQ agent to learn. The accuracy of the compressed network is affected by the following factors: the interaction between model layers, the weights removed by pruning, and the bandwidth after quantization. The environment takes all of these factors into account. Table 1 shows the static and dynamic information that needs to be observed to construct the environment in one embodiment of this invention.

[0100] Table 1

[0101]

[0102] As shown in Table 1, the static information related to the network structure includes layer metrics, layer dimensions, and model size. Layer metrics include the t-th layer currently being compressed; layer dimensions include the input data size in, the output data size out, the kernel size k, and the stride; and model size includes the total number of network parameters n. param .

[0103] Because the APQ agent employs different pruning quantization strategies during its search space exploration, the pruning dynamics between different layers occur. This method uses floating-point operations (FLOPs) as an indicator of pruning effectiveness. Since the pruning strategy of the current layer is influenced not only by the FLOPs of previous layers but also by the FLOPs of subsequent layers, the dynamic pruning information in this method includes the total number of reduced floating-point operations from pruned layers. FLOPs The total number of floating-point operations of the unpruned layer rest FLOPs Use bit parameter quantity ( Among them, parmeters t This refers to the number of parameters in the current layer, in bits. t This refers to the number of bits in the current layer weights as a quantization metric. The influence of quantized and unquantized layers is also considered. Quantization dynamic information includes the reduced number of quantized bit parameters. wszie Unquantized bit parameter quantity rest wsize The dynamic information also includes the pruning strategy for layer t-1 of the neural network model provided by the APQ agent. Weight quantization strategy and activation value quantization strategy

[0104] In step 202, based on the DDPG algorithm ((Y. He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han. Amc: Automl for model compression and acceleration on mobile devices. ECCV, 2018.), (K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han, Haq: Hardware-aware automated quantization with mixed precision. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.), and (Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In (ICLR, 2016.) To construct the agent of this invention, the DDPG algorithm combines the Deep Q-Network (DQN) algorithm with the Actor-Critic (policy-evaluation) method, including an actor network and a critic network. Table 2 shows the structure of the policy (Actor) network in one embodiment of this invention, and Table 3 shows the structure of the evaluation (Critic) network in one embodiment of this invention.

[0105] Table 2

[0106]

[0107] Table 3

[0108]

[0109] As shown in Table 2, the policy network of the APQ agent can include 3 fully connected layers, 2 ReLU (Rectified Linear Unit) activation layers and 1 Sigmoid activation layer.

[0110] In Table 2, `state_dim` represents the dimension of environmental information, which is 14 in this invention; `action_dim` represents the dimension of the output action, which is 3 in this invention. If we consider the output action of the actor network as a problem of finding the extremum of a function, `action_dim = 3` is equivalent to solving the extremum of f(x, y, z), while `action_dim = 1` and constructing three actor networks is equivalent to treating the function as three unary functions and solving them. Finding the extrema of a multivariate function expands the solution space and allows for a better search for the optimal solution. Therefore, to find a better compression strategy, the pruning and quantization strategies of this invention are updated and output simultaneously. Furthermore, another dimension of the output tensor is determined by the hyperparameter `data_bsize` (which can be set to 20 in this embodiment's experiments), representing the amount of data input in a batch. During training, PyTorch's built-in Adam optimizer can be used with a learning rate `lr_a = 1e-4`.

[0111] As shown in Table 3, the evaluation network of the APQ agent can include four fully connected layers and two ReLU activation layers. The input to the evaluation network includes environmental variables and action variables predicted using the policy network. The first two fully connected layers output a 300×20 tensor, which is then summed and used as the input to the ReLU activation layer for activation. When calculating the Q-value, Where s′ t+1 The quantified environment, a t+1 This represents the prediction strategy, where Y represents the discount coefficient. This indicates whether it is the last layer. During training, you can use PyTorch's built-in Adam optimizer with a learning rate of lr_c = 1e-3.

[0112] Figure 3 A schematic diagram illustrating the interaction process between the APQ agent and the environment in one embodiment of the present invention is shown. Figure 3 As shown, during the information gathering phase, the APQ agent uniformly samples within the action space, enabling it to independently and randomly generate pruning and quantization policies. The quantization policy during this phase is not based on the pruned environment. For the CIFAR-10 dataset, the hyperparameter warmup is set to 100, meaning the agent network is only sampled for policy experience in the first 100 search iterations without updating the network. For the ImageNet dataset, the hyperparameter warmup is set to 40. The collected tuples {rn} ， s t ,s′ t a t δ tStore r in the experience replay set. n s represents the reward for the last level. t s′ represents the environment before pruning. t a represents the quantized environment. t This represents the pruning and quantization strategy, δ t Used to indicate whether it is the last layer.

[0113] In step 203, the interaction process between the APQ agent and the environment is first determined and experience is collected. During the training and update phase, the APQ agent generates pruning and quantization policies simultaneously using the policy network based on the model's current environmental information, and adds noise to improve the model's robustness by truncating the noise using a normal distribution. Normal distribution truncation is performed using scipy.stats.truncnorm.rvs, with the distribution center loc = action and the scaling factor scale = init_delta × delta_decay. (episode-warmup) The hyperparameters are init_delta = 2, delta_decay = 0.95, and episode represents the number of search iterations N, which varies depending on the dataset. For example, episode = 300 for the CIFAR-10 dataset and 600 for the ImageNet dataset. The truncation range is...

[0114] The agent in this method can perform post-pruning quantization (referring to deep compression). Therefore, the agent first generates three policies simultaneously based on the pre-pruning environment, and then only uses the pruned policy. After interacting with the environment, the intelligent system generates three strategies based on the pruned environment, and then quantifies these strategies using weights. and activation value quantization strategy Finally, merge the three strategies. Obtain the compression strategy for layer t.

[0115] Furthermore, the APQ agent is trained and a compression policy is output. During the network update phase, the current evaluation network C and the current policy network μ are initialized, with all weight parameters initialized to init_w = 3 × 10⁻⁶. -3 The target evaluation network C′ and the target policy network μ′ are replicated for updates. m samples are sampled from the experience replay set, where m is the hyperparameter bsize = 64. For the sampled tuple {r n s t ,s′ t a t δ t First, utilize the environmental information s before pruning. t And compression strategy at Calculate Q = C(s) using the current evaluation network. t a t Then, using the quantified environmental information s′ t And the next-layer compression strategy a predicted by the policy target network. t+1 =μ′(s′ t ),calculate Finally, the mean squared error function is used. The parameters of the current evaluation network C are updated via gradient descent direction propagation. Using the updated current evaluation network C, the parameters are calculated according to formula -C(s). t ,μ(s t The parameters of the current policy network μ are updated through gradient descent direction propagation, and finally the parameters of the target evaluation network C′ and the target policy network μ′ are updated. The hyperparameter is set to τ = 0.01.

[0116] Furthermore, we construct efficient pruning and quantization modules and output high-precision compression strategies. The pruning strategy used in this method is channel pruning ((Y. He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han. Amc: Automl for model compression and acceleration on mobile devices. ECCV, 2018.)(He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1389-1397 (2017)). The quantization strategies include linear quantization, SAT quantization (He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1389-1397 (2017)) and PWLQ quantization (Jun Fang, Ali Shafiee, Hamzah Abdel-Aziz, David...). Thorsley, Georgios Georgiadis, Joseph Hassoun. Post-Training Piecewise Linear Quantization for Deep Neural Networks. In ECCV, 2020. The above quantization strategies perform differently in terms of complexity and model readiness. In practical applications, the quantization strategy for APQ search can be determined according to different models and datasets.

[0117] After providing the pruning strategy for each layer (that is, the proportion of weights to be retained in the current layer), the APQ agent selects channels based on the L1 norm. In other words, it calculates the L1 norm of the weights of all channels in the current layer, sorts them in descending order, selects the weights to retain according to the corresponding proportion, and then uses the least squares method to reconstruct the feature map so that the difference between the pruned output and the input before pruning is minimized.

[0118] After the APQ agent provides the quantization strategy for each layer (that is, the weight quantization bit width and activation value quantization bit width for the current layer), different quantization strategies can be used for quantization.

[0119] The quantization formula for linear quantization is expressed as follows:

[0120] X = clamp(X, -α, α),

[0121]

[0122]

[0123] The model's weights and activation values ​​are quantized, where α represents the cutoff value, sf represents the scaling factor, bit represents the quantization bit width, and [·] represents rounding to the nearest integer. The optimal quantization cutoff value α can be determined using histogram optimization. X represents the original parameter values, q... X These are the quantized parameter values. The first step involves performing histogram statistics on the weights and input data to obtain the corresponding frequencies and group endpoints; the second step calculates the quantized data range n=2 based on the bit width b given by the reinforcement learning algorithm. b-1 Then, the possible quantization cutoff values ​​and corresponding scaling factors are given between n+1 and the histogram extrema; the third step uses linear interpolation to estimate the quantized values ​​under different quantization cutoff values, based on the noise-to-signal ratio. The fourth step involves determining the optimal cutoff value; quantization is then performed using the cutoff value and scaling factor determined in the third step.

[0124] The weight quantization formula for the SAT quantization method is expressed as follows:

[0125]

[0126]

[0127]

[0128] q W * It is the weight value after weight quantization, where n out This represents the number of outputs for each layer of the model.

[0129] The activation values ​​of the model can be quantized using the PACT method (Jungwook Cho i, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, Kailash Gopalakrishnan. PACT: Parameterized Clipping Activation for Quantized Neural Networks. arXiv: 1805.06085v2, 2018.), and the quantization formula is expressed as follows:

[0130] PACT(X)=clamp(X,-α,α).

[0131] The cutoff value α is used as a parameter for model training. In this invention, SAT quantization requires training the model after determining the quantization bit width each time. The training parameters are epochs = 2, learning rate Ir = 0.05, gamma = 0.1, and the learning rate decay method is Ir = Ir * gamma. During training, the loss function is loss = MSE(output, target) + 0.0002 × α. 2 .

[0132] The parameter quantization formula for the PWLQ quantization network is expressed as follows:

[0133]

[0134]

[0135]

[0136] q X =q X ×sf.

[0137] The first step in weight quantization is to find a breakpoint p in the data, dividing the weight values ​​into three intervals: [-m, -p], [-p, p], and [p, m]. The second step involves unsigned uniform quantization for weights distributed in [-m, -p] and [p, m], and signed uniform quantization for weights in [-p, p]. This patent uses the formula p / m = ln(0.8614m + 0.6079) to determine the breakpoint, where p is the breakpoint and m is the absolute value of the maximum or minimum weight.

[0138] During activation value quantization, 512 random training images were used as a calibration set, and the median of the Top 10 was calculated as the quantization cutoff value before uniform quantization was performed.

[0139] In one embodiment of the present invention, the model can be further fine-tuned and the performance metrics of the compressed model can be tested.

[0140] The CIFAR-10 and ImageNet datasets can be used for compression, fine-tuning, and testing. When fine-tuning the compressed model, the learning rate Ir is set to 0.05, using an exponential learning rate decay strategy with a decay exponent of decay_rate = 0.96 and a decay rate of decay_steps = 1. Training is performed on a single GPU with 150 epochs, a batch size of 128, and a weight decay of 4 × 10^6. -5 When importing data, num_workers = 8, which means that 8 child processes are used to load the data.

[0141] When testing the fine-tuned model, set the batch size parameter batch_size = 100, and when importing data using ataloader, set num_workers = 32.

[0142] Tables 5 and 6 show the test results using the CIFAR-10 dataset and the ImageNet dataset, respectively. As shown in Tables 5 and 6, comparing the performance of the uncompressed model in the first row and the model after the proposed compression strategy in the last row, it can be seen that the number of bit parameters and the computational cost of the final model obtained by this method are reduced by 94% and 54%, respectively. Compared with the model obtained by using the same AMC compression, the model obtained by the proposed compression method is smaller without a significant change in accuracy.

[0143] Table 5

[0144]

[0145] Table 6

[0146]

[0147] In another embodiment of the present invention, an automatic pruning and quantization system based on a reinforcement learning framework is also proposed, comprising:

[0148] An environment generation module is configured to construct an environment for an automatic pruning quantization method, the environment being based on static information related to the network structure and dynamic information related to the compression state.

[0149] An agent generation module, configured to construct automatically pruned and quantized agents; and

[0150] An interactive training module is configured to enable the automatic pruning and quantization agent to interact with the environment to train the automatic pruning and quantization agent and output a compression strategy, the compression strategy including a pruning strategy and a quantization strategy.

[0151] Although various embodiments of the invention have been described above, it should be understood that they are presented by way of example only and not as limitations. It will be apparent to those skilled in the art that various combinations, modifications, and alterations can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely by the appended claims and their equivalents.

Claims

1. An automatic pruning quantization method based on a reinforcement learning framework, characterized in that, Includes the following steps: An environment is constructed for an automatic pruning quantization method, the environment being based on static information related to the network structure and dynamic information related to the compression state; Constructing an automated pruning quantitative intelligent agent; as well as The automatic pruning and quantization agent is made to interact with the environment to train the automatic pruning and quantization agent and output a compression strategy, the compression strategy including a pruning strategy and a quantization strategy. The static information mentioned above includes: Layer metrics, which include the current layer of the network being compressed. layer; Layer dimension, which includes the size of the input data. Output data size kernel size and stride size ;as well as Model size, which includes the total number of network parameters. ; and\or The dynamic information includes: Pruning information, which includes the total number of floating-point operations reduced by the pruned layers. and the total number of floating-point operations in the unpruned layer ; Quantization information, which includes the amount of quantized bit parameters. and the amount of unquantized bit parameters ;as well as Policy information, including neural network models Layer pruning strategy Weight quantization strategy and activation value quantization strategy ; The automatic pruning and quantization agent is constructed based on a deep deterministic policy gradient algorithm, wherein the automatic pruning and quantization agent includes: The policy network includes first to third fully connected layers, first to second modified linear unit activation layers, and a first sigmoid activation layer; and The evaluation network consists of the fourth to seventh fully connected layers and the third to fourth modified linear unit activation layers; The interaction between the automated pruning quantization agent and the environment includes: During the information collection phase, the automatically pruned quantization agent samples and collects data based on the strategy without updating the network, wherein the collected tuples are... Stored in the experience replay collection, where This indicates the reward for the last level. This describes the environment before pruning. This represents the quantified environment. This indicates pruning and quantification strategies. Used to indicate whether it is the last floor; and During the training and update phase, the automatic pruning and quantization agent generates pruning and quantization policies simultaneously through the policy network based on environmental information, and adds noise by truncating the noise using a normal distribution. The automatic pruning and quantization agent simultaneously generates pruning and quantization policies based on environmental information through the policy network, including the following actions performed by the automatic pruning and quantization agent: In the The layer generates pruning and quantization strategies based on the environment before pruning; Use the Layer pruning strategy It interacts with the environment and generates a weight quantization strategy based on the pruned environment. and activation value quantization strategy ;as well as Merge strategy to generate the first Layer compression strategy ; The training of the automatic pruning quantization agent and the output of the compression strategy include the automatic pruning quantization agent performing the following actions: Initialize the current evaluation network Output Values ​​and the current policy network Output strategy ; The target evaluation network is obtained by replicating the current network described above. Output Value and target policy network In order to update; Sample m samples from the experience replay set; Environmental information before pruning and compression strategies Calculated using the current evaluation network The value is expressed as follows: ; Based on the quantified environmental information The reward of the last level And the next-layer compression strategy predicted by the target policy network. Through target evaluation network calculate The value is expressed as follows: ； Minimize the mean squared error function To optimize the objective, the current evaluation network is updated using gradient descent and backpropagation. ; Use the updated current evaluation network and based on minimization Calculate the gradient and update the current policy network using gradient descent and backpropagation. ;as well as Update target evaluation network and target policy network It can be expressed as the following formula: in, Indicates hyperparameters; The automatic pruning quantization agent, after providing the pruning strategy for each layer, selects channels based on the L1 norm and reconstructs the feature map using the least squares method; and / or After providing the quantization strategy for each layer, the automatic pruning quantization agent uses a linear quantization method to quantize the weights and activation values, as shown in the following formula: in, Indicates the cutoff value, Indicates scaling factor, Indicates quantization bit width, This indicates rounding to the nearest integer; X represents the original parameter value. This represents the quantized parameter values; it includes: histogram statistics of the weights and input data to obtain the corresponding frequencies and group endpoint values; and the bit width given by the reinforcement learning algorithm. Calculate the range of quantized data And in Possible quantization cutoff values ​​and corresponding scaling factors are given between the histogram extrema; the quantized values ​​are estimated using linear interpolation for different quantization cutoff values, based on the noise-to-signal ratio. Determine the optimal cutoff value; and perform quantization using the cutoff value and scaling factor determined in step three; and / or The automatic pruning quantization agent, after providing the quantization strategy for each layer, uses the SAT quantization method for weight quantization and the PACT quantization method for activation value quantization. The SAT quantization method is expressed as follows: in, These are the weight values ​​after weight quantization. This represents the number of outputs for each layer of the model; The quantization formula for the PACT method is expressed as follows: ; and\or After providing the quantization strategy for each layer, the automatic pruning quantization agent uses a PWLQ quantization network to quantize the weights and activation values, where the parameter quantization formula is expressed as follows: This includes finding a breakpoint in the data during weight quantization. Divide the weight values ​​into , , Three intervals; for distribution in and The weights in the code employ unsigned uniform quantization. The weights in the formula are quantized using a signed uniform quantization. Determine the breakpoint, where As a breakpoint, The absolute value of the maximum and minimum weights; and when quantizing activation values, multiple random training images are used as a calibration set, and the median of the first training image is calculated as the cutoff value for quantization, and uniform quantization is performed.

2. An automatic pruning and quantization system based on a reinforcement learning framework, characterized in that, include: An environment generation module is configured to construct an environment for an automatic pruning quantization method, the environment being based on static information related to the network structure and dynamic information related to the compression state. The agent generation module is configured to construct automatically pruned and quantized agents; as well as An interactive training module is configured to enable the automatic pruning and quantization agent to interact with the environment in order to train the automatic pruning and quantization agent and output a compression strategy, the compression strategy including a pruning strategy and a quantization strategy. The static information mentioned above includes: Layer metrics, which include the current layer of the network being compressed. layer; Layer dimension, which includes the size of the input data. Output data size kernel size and stride size ;as well as Model size, which includes the total number of network parameters. ; and\or The dynamic information includes: Pruning information, which includes the total number of floating-point operations reduced by the pruned layers. and the total number of floating-point operations in the unpruned layer ; Quantization information, which includes the amount of quantized bit parameters. and the amount of unquantized bit parameters ;as well as Policy information, including neural network models Layer pruning strategy Weight quantization strategy and activation value quantization strategy ; The automatic pruning and quantization agent is constructed based on a deep deterministic policy gradient algorithm, wherein the automatic pruning and quantization agent includes: The policy network includes first to third fully connected layers, first to second modified linear unit activation layers, and a first sigmoid activation layer; and The evaluation network consists of the fourth to seventh fully connected layers and the third to fourth modified linear unit activation layers; The interaction between the automated pruning quantization agent and the environment includes: During the information collection phase, the automatically pruned quantization agent samples and collects data based on the strategy without updating the network, wherein the collected tuples are... Stored in the experience replay collection, where This indicates the reward for the last level. This describes the environment before pruning. This represents the quantified environment. This indicates pruning and quantification strategies. Used to indicate whether it is the last floor; and During the training and update phase, the automatic pruning and quantization agent generates pruning and quantization policies simultaneously through the policy network based on environmental information, and adds noise by truncating the noise using a normal distribution. The automatic pruning and quantization agent simultaneously generates pruning and quantization policies based on environmental information through the policy network, including the following actions performed by the automatic pruning and quantization agent: In the The layer generates pruning and quantization strategies based on the environment before pruning; Use the Layer pruning strategy It interacts with the environment and generates a weight quantization strategy based on the pruned environment. and activation value quantization strategy ;as well as Merge strategy to generate the first Layer compression strategy ; The training of the automatic pruning quantization agent and the output of the compression strategy include the automatic pruning quantization agent performing the following actions: Initialize the current evaluation network Output Values ​​and the current policy network Output strategy ; The target evaluation network is obtained by replicating the current network described above. Output Value and target policy network In order to update; Sample m samples from the experience replay set; Environmental information before pruning and compression strategies Calculated using the current evaluation network The value is expressed as follows: ; Based on the quantified environmental information The reward of the last level And the next-layer compression strategy predicted by the target policy network. Through target evaluation network calculate The value is expressed as follows: ； Minimize the mean squared error function To optimize the objective, the current evaluation network is updated using gradient descent and backpropagation. ; Use the updated current evaluation network and based on minimization Calculate the gradient and update the current policy network using gradient descent and backpropagation. ;as well as Update target evaluation network and target policy network It can be expressed as the following formula: in, Indicates hyperparameters; The automatic pruning quantization agent, after providing the pruning strategy for each layer, selects channels based on the L1 norm and reconstructs the feature map using the least squares method; and / or After providing the quantization strategy for each layer, the automatic pruning quantization agent uses a linear quantization method to quantize the weights and activation values, as shown in the following formula: in, Indicates the cutoff value, Indicates scaling factor, Indicates quantization bit width, This indicates rounding to the nearest integer; X represents the original parameter value. This represents the quantized parameter values; it includes: histogram statistics of the weights and input data to obtain the corresponding frequencies and group endpoint values; and the bit width given by the reinforcement learning algorithm. Calculate the range of quantized data And in Possible quantization cutoff values ​​and corresponding scaling factors are given between the histogram extrema; the quantized values ​​are estimated using linear interpolation for different quantization cutoff values, based on the noise-to-signal ratio. Determine the optimal cutoff value; and perform quantization using the cutoff value and scaling factor determined in step three; and / or The automatic pruning quantization agent, after providing the quantization strategy for each layer, uses the SAT quantization method for weight quantization and the PACT quantization method for activation value quantization. The SAT quantization method is expressed as follows: in, These are the weight values ​​after weight quantization. This represents the number of outputs for each layer of the model; The quantization formula for the PACT method is expressed as follows: ; and\or After providing the quantization strategy for each layer, the automatic pruning quantization agent uses a PWLQ quantization network to quantize the weights and activation values, where the parameter quantization formula is expressed as follows: This includes finding a breakpoint in the data during weight quantization. Divide the weight values ​​into , , Three intervals; for distribution in and The weights in the code employ unsigned uniform quantization. The weights in the formula are quantized using a signed uniform quantization. Determine the breakpoint, where As a breakpoint, The absolute value of the maximum and minimum weights; and when quantizing activation values, multiple random training images are used as a calibration set, and the median of the first training image is calculated as the cutoff value for quantization, and uniform quantization is performed.

3. A computer-readable storage medium having a computer program stored thereon, the computer program performing the steps of the method according to claim 1 when executed by a processor.

4. A computer system, comprising: A processor, which is configured to execute machine-executable instructions; as well as A memory having stored machine-executable instructions that, when executed by a processor, perform the steps of the method according to claim 1.

Images