A neural network model compression method and system based on optimal combination compression sequences

Abstract

This invention provides a neural network model compression method and system based on optimal combination compression sequences, comprising: Step 1: Training a custom student model using the intermediate output of the original model as a soft target to inherit the inference performance of the original model; Step 2: Judging and deleting specific nodes / node connections and their corresponding model parameters at each layer of the student model using a node importance index constructed by a preset standard; Step 3: Changing the number of bits of the student model parameters, converting each floating-point parameter of the original model into its corresponding integer parameter; Step 4: Dynamically adjusting the number of student model layers required for each inference, with the student model dynamically changing its own results during inference, thus dynamically achieving the compression result. This invention solves the problem of combining multiple model compression methods, achieving convenience and configurability in deploying continuous compression schemes and comprehensively verifying the performance of compressed models.

Description

Technical Field

[0001] This invention relates to the field of neural network model compression technology, and more specifically, to a neural network model compression method and system based on optimal combination compression sequences. Background Technology

[0002] To support the application of neural networks on lightweight computing platforms, various neural network compression techniques have been proposed in recent years. These techniques utilize different aspects, such as network architecture, neurons, and bit precision, to compress deep neural networks with minimal precision.

[0003] Meanwhile, characterized by a fixed computation graph and parameters during the inference phase, dynamic compression demonstrates the ability to adapt network structure or parameters at runtime to accommodate different inputs. When applying multiple compressions, (Qi et al., 2021) combined pruning and quantization to compress CNNs, achieving approximately a 50% reduction in floating-point operations while only decreasing accuracy by 0.15% to 0.37%. (Li et al., 2023) combined early exit with quantization in convolutional neural networks, not only avoiding floating-point computations but also reducing computation by over 50% while decreasing accuracy by 1%–3%.

[0004] (Zhao et al., 2023) proposed a deep hybrid compression network, which uses relaxed hybrid precision quantization, relaxed weight pruning, and knowledge distillation to overcome the limitations of uniform quantization. (Han et al., 2015) utilized pruning, training quantization, and Huffman coding to reduce network parameter storage and energy overhead during neural network execution.

[0005] In recent years, various popular model compression techniques, despite employing different compression methods on neural networks, have not explored the interactions and optimal ranking among three or more classical compression methods. Inspired by the potential to significantly reduce computational and parameter costs, this invention investigates the interactions between compression techniques and establishes an optimal neural network compression sequence. Under the same accuracy loss constraint, this compression sequence paradigm (with appropriate hyperparameters) achieves a compression ratio far exceeding that of compressed models formed by a single compression method. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the purpose of this invention is to provide a neural network model compression method and system based on optimal combination compression sequences.

[0007] The neural network model compression method based on optimal combination compression sequences provided by the present invention includes:

[0008] Step 1: Apply architectural-level technical knowledge distillation, and use the intermediate output of the original model as a soft target to train a custom student model to inherit the inference performance of the original model;

[0009] Step 2: Apply neuron-level pruning techniques to determine and delete specific nodes / node connections and their corresponding model parameters at each layer of the student model using a node importance index based on preset standards.

[0010] Step 3: Apply subneuronal level quantization technology to directly change the number of bits of the student model parameters, converting each floating-point parameter of the original model into the corresponding integer parameter;

[0011] Step 4: Use dynamic architecture-level technology for early exit, which takes effect during inference and dynamically adjusts the number of student model layers required for each inference. The student model dynamically changes its own results while inference, thus dynamically achieving compressed results.

[0012] Preferably, step 1 includes: loading the teacher model as the model to be compressed, initializing the student model as the compressed model defined by hyperparameters, inputting the training set into the teacher model and student model simultaneously in the same order, taking the intermediate output containing all classification result probability values ​​before the SoftMax layer of the teacher model as the soft target, and weighting it with the hard target of the training set as the weight to become the final training target of the student model, and using the cross-entropy loss of the final training target and the output result of the student model as the total loss for normal gradient regression, and the model obtained after training is the compressed model.

[0013] Preferably, step 2 includes: channel pruning by probing the output feature size, convolution kernel size and output feature change magnitude during the model inference process, determining the importance index of each convolution kernel, and removing the corresponding low-importance convolution kernels from the model parameters, thereby compressing the computational complexity and model size.

[0014] Preferably, step 3 includes: 8-bit quantization for the convolutional neural network. The quantization operation is performed during training. This operation converts the 32-bit floating-point model parameters into 8-bit integer parameters through a monotonically increasing function based on the upper and lower limits of the model parameters in the original convolutional kernel or the data distribution. The fine-tuning during training is performed together with the quantization operation. The quantized 8-bit integer parameters are used during forward propagation, and the unquantized 32-bit floating-point parameters are used during gradient regression. The quantized model composed of integers is directly obtained after quantization during training, thereby achieving model compression.

[0015] Preferably, step 4 includes: embedding exit layers at several locations in the middle of the model, and training the model once for all exit layers, so that each exit layer can output the inference result with the highest accuracy that can be obtained by exiting at that location. During inference, each time the feature map of the middle layer reaches an exit layer node, it first attempts to exit from the exit layer. If the exit layer result does not meet the confidence threshold hyperparameter, the feature map continues to propagate forward in the original network. If the exit layer output meets the confidence threshold, the inference ends immediately, and the output of the exit layer is taken as the final inference result.

[0016] The neural network model compression system based on optimal combination compression sequences provided by the present invention includes:

[0017] Module M1: Apply architecture-level technical knowledge distillation, using the intermediate output of the original model as a soft target to train a custom student model to inherit the inference performance of the original model;

[0018] Module M2: Applying neuron-level pruning techniques, specific nodes / node connections and their corresponding model parameters are determined and deleted at each layer of the student model based on a node importance index constructed using preset standards.

[0019] Module M3: Utilizes subneuronal level quantization technology to directly change the number of bits in the student model parameters, converting each floating-point parameter of the original model into its corresponding integer parameter;

[0020] Module M4: Uses dynamic architecture-level technology to exit early, taking effect during inference. It dynamically adjusts the number of student model layers required for each inference, and the student model dynamically changes its own results during inference, thus dynamically achieving compressed results.

[0021] Preferably, module M1 includes: loading the teacher model as the model to be compressed, initializing the student model as the compressed model defined by hyperparameters, inputting the training set into the teacher model and student model simultaneously in the same order, taking the intermediate output containing the probability values ​​of all classification results before the SoftMax layer of the teacher model as the soft target, and weighting it with the hard target of the training set as the weight to become the final training target of the student model, and using the cross-entropy loss of the final training target and the output result of the student model as the total loss for normal gradient regression, and the model obtained after training is the compressed model.

[0022] Preferably, the module M2 includes: channel pruning, which determines the importance index of each convolution kernel by probing the output feature size, convolution kernel size and output feature change magnitude during the model inference process, and removes the corresponding low-importance convolution kernels from the model parameters, thereby compressing the computational complexity and model size.

[0023] Preferably, module M3 includes: 8-bit quantization for convolutional neural networks. The quantization operation is performed during training. This operation converts 32-bit floating-point model parameters into 8-bit integer parameters through a monotonically increasing function based on the upper and lower limits of the model parameters in the original convolutional kernel or the data distribution. The fine-tuning during training is performed together with the quantization operation. The quantized 8-bit integer parameters are used during forward propagation, and the unquantized 32-bit floating-point parameters are used during gradient regression. The quantized model composed of integers is directly obtained after quantization during training, thereby achieving model compression.

[0024] Preferably, the module M4 includes: embedding exit layers at several positions in the middle of the model, and performing model training once for all exit layers, so that each exit layer can output the inference result with the highest accuracy that can be obtained by exiting at that position. During inference, each time the feature map of the middle layer reaches an exit layer node, it first attempts to exit from the exit layer. If the exit layer result does not meet the confidence threshold hyperparameter, the feature map continues to be forward-propagated in the original network. If the exit layer output meets the confidence threshold, the inference ends immediately, and the output of the exit layer is taken as the final inference result.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] (1) By encapsulating and optimizing the compatibility of distillation, pruning, quantization and early exit, the problem of combining multiple model compression methods is solved, and the convenience and configurability of deploying continuous compression schemes and fully verifying the performance of compressed models are realized.

[0027] (2) Through grid search of multiple datasets, multiple models, and multiple hyperparameter combinations, a large-scale lookup table from compressed sequence + hyperparameter combination to final model performance was formed. The relationship and influence between compression methods obtained from the experimental results and the structural understanding of compression methods solved the problem of optimal compressed sequence. The compression criteria from architecture to unit and from static to dynamic were proved, and the optimal compressed sequence paradigm of DPQE was derived. The compressed model generated by this compressed sequence can stably achieve a computational compression rate of 100 to 1000 times on mainstream convolutional and transformer neural networks and common datasets, and the accuracy loss can be reduced to negligible. Attached Figure Description

[0028] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0029] Figure 1 A schematic diagram of a configurable combinatorial sequence compression framework;

[0030] Figure 2This is a schematic diagram illustrating the implementation of channel pruning technology within this framework.

[0031] Figure 3 This is a schematic diagram illustrating the implementation of knowledge distillation technology within this framework.

[0032] Figure 4 This is a schematic diagram illustrating the implementation of quantization techniques within this framework during integer training.

[0033] Figure 5 This is a schematic diagram illustrating the implementation of early exit technology within this framework.

[0034] Figure 6 This is the optimal compressed sequence for the model. Detailed Implementation

[0035] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0036] Example

[0037] This invention constructs a configurable, combinatorial sequence compression framework based on encapsulated implementations and compatibility optimizations of classic model compression techniques (including knowledge distillation, pruning, quantization, and early exit). Figure 1 The optimal compressed sequence paradigm proposed based on the structural characteristics of topological sorting and compression techniques—D (distillation), P (pruning), Q (quantization), E (early exit)—has been proven through experiments using multiple models, multiple datasets, traversing combined sequences, and grid search for hyperparameters within this framework. The compressed model exhibits low accuracy loss and extremely high computational and model compression ratios, far surpassing the performance of individual compression methods.

[0038] This technology includes a neural network model compression sequence paradigm formed by combining four classic model compression techniques and a compatible, configurable, combinatorial compression framework.

[0039] Configurable Combinatorial Compression Framework: To derive the optimal compression sequence paradigm, the software first implements a configurable combinatorial compression framework that supports four classic model compression techniques: Pruning, ... Figure 2 Knowledge distillation (D, such as...) Figure 3 ), Quantization, such as Q (Quantization, such as Q) Figure 4 ) and Early Exit (Early Exit, such as Figure 5Based on the concept of encapsulation, the compression modules of this framework are compatible with each other, while the hyperparameters can be adjusted completely independently. Therefore, this framework allows for the customization of compression sequences and the determination of the inference accuracy, theoretical bit operands, and model size of the final compressed model determined by each compression sequence and parameter combination.

[0040] Model compression sequence paradigm: Based on the understanding of the compression theoretical characteristics of each model, it follows... Figure 6 An optimal model for compressing sequences in the middle order is proposed.

[0041] First, knowledge distillation, an architecture-level technique, is applied. It uses the intermediate outputs of the original model as a soft objective to train a smaller, fully customized student model that largely inherits the inference performance of the original model. This changes the overall class / layer / width (global channel count) of the model, as the student model can be a low-width / layer variant of the original model or a completely different model, resulting in an architectural change. Second, neuron-level pruning is applied. At each layer of the model, a node importance index based on certain criteria is used to determine and delete specific nodes / node connections and their corresponding model parameters. This changes the local node connections within the model, and the model's changes occur at each layer. The model first modifies the nodes and their connections with other nodes. Then, sub-neuronal quantization is applied, which directly changes the number of bits in the model parameters. Each floating-point parameter of the original model is converted into a corresponding integer parameter, and the structure of the model remains unchanged. The changes penetrate deep into the node parameters. Finally, dynamic architecture-level early exit is used. It takes effect during inference and dynamically adjusts the number of model layers required for each inference. Unlike the previous three static methods of changing the model, the model changes in early exit occur after the model training is completed and deployed. The model dynamically changes its own results during inference, dynamically achieving the compressed result. Therefore, early exit is a dynamic model architecture-level compression technique.

[0042] Proof of the compressed sequence paradigm: Using this framework, we can explore the performance of the compressed model for a given compressed sequence and each parameter combination through grid search. From the perspective of topological sorting, we prove that DP, DQ, DE, PQ, PE, and QE have universally better performance than their reverse sequences, thus their topological sorting points to the unique optimal solution DPQE. From the perspective of experimental traversal, we also prove that DPQE has better performance than the other 23 compression methods. The grid search results form a large lookup table from the compressed sequence + hyperparameter combination to the final model performance.

[0043] Figure 1This diagram illustrates a configurable combinatorial sequence compression framework. The compression units within this framework consist of pruning, knowledge distillation, quantization, and early exit compression techniques optimized for compatibility (with other compression methods). The compression modules are highly independent, thus forming the configurable components of the model compression sequence. Users can freely adjust the relative positions of the modules in the compression sequence, freely modify the hyperparameters of each compression module (which determines the configuration of each compression method; different compression methods have different types of hyperparameters), and (furthermore) delete compression modules or repeatedly insert a compression module at different positions in the sequence. The independence and compatibility of the compression modules ensure that the previous compression module will only output the model itself to the next compression module, and the next compression module will continue to transmit the compressed model after compatibility conversion, model compression, and retraining fine-tuning. The coordinate graph represents the compression effect evaluation unit built into the framework. It supports grid search / traversal of hyperparameters and sequential batch generation of compressed models, and automatically verifies the inference results of the compressed models. Therefore, it can automatically verify the inference accuracy, computational compression rate, model compression rate and other custom verification standards of the model, and draw it into an easy-to-understand dot graph.

[0044] Figure 2 This diagram illustrates the implementation of channel pruning within this framework. Taking a convolutional neural network as an example, both the input and output feature maps have different numbers of channels (corresponding to different image features), and each convolutional kernel corresponds to a transformation from one input channel to one output channel (in the form of matrix multiplication). Channel pruning determines the importance index of each convolutional kernel by probing configurable indicators such as the output feature size, convolutional kernel size, and the magnitude of output feature changes during model inference, and removes the corresponding low-importance convolutional kernels from the model parameters. This achieves compression of computational complexity and model size.

[0045] Figure 3 This diagram illustrates the implementation of knowledge distillation within this framework. The teacher model is loaded as the model to be compressed, while the student model is initialized as the compressed model defined by hyperparameters. The training set is input into both models simultaneously in the same order. For each batch of data loaded, the intermediate output containing all classification probability values ​​before the SoftMax layer of the teacher model is taken as the soft objective. This soft objective is weighted with the correct label from the training set (the hard objective) to become the final training objective of the student model. The cross-entropy loss between this objective and the student model's output is used as the total loss for normal gradient regression. The resulting model after training is the compressed model.

[0046] Figure 4This diagram illustrates the implementation of integer-based training-time quantization within this framework. Taking 8-bit quantization of a convolutional neural network as an example, the quantization operation is performed during training. This operation converts the 32-bit floating-point model parameters into 8-bit integer parameters using a monotonically increasing function (usually linear), based on the upper and lower limits of the model parameters in the original convolutional kernel or the data distribution. Training-time fine-tuning is performed concurrently with the quantization operation. The quantized 8-bit integer parameters are used during forward propagation, while the unquantized 32-bit floating-point parameters are used during gradient regression (i.e., backpropagation). Training-time quantization directly yields the quantized model composed of integers. Integer data computation is faster and storage usage is lower, thus achieving model compression.

[0047] Figure 5 This diagram illustrates the implementation of early exit technology within this framework. Depending on the model, exit layers with specific structures are embedded at appropriate locations within the model. An additional model training is performed for all exit layers to ensure that each exit layer outputs the highest accuracy inference result achievable at that exit point. During inference, each time an intermediate layer feature map reaches an exit layer node, it first attempts to exit from that layer. If the exit layer result does not meet the confidence threshold hyperparameter, the feature map continues to propagate forward in the original network, as shown in the first exit layer diagram. If the exit layer output meets the confidence threshold, the inference immediately ends, and the output of that exit layer is taken as the final inference result. Early exit slightly increases the model size due to the addition of exit layers, but effectively reduces the computational cost during inference, representing a dynamic compression technique in the model architecture layer.

[0048] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0049] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A neural network model compression method based on optimal combination compression sequence, characterized in that, include: Step 1: Apply architectural-level technical knowledge distillation, and use the intermediate output of the original model as a soft target to train a custom student model to inherit the inference performance of the original model; Step 2: Apply neuron-level pruning techniques to determine and delete specific nodes / node connections and their corresponding model parameters at each layer of the student model using a node importance index based on preset standards. Step 3: Apply subneuronal level quantization technology to directly change the number of bits of the student model parameters, converting each floating-point parameter of the original model into the corresponding integer parameter; Step 4: Use dynamic architecture-level technology for early exit, which takes effect during inference and dynamically adjusts the number of student model layers required for each inference. The student model dynamically changes its own results while inference, thus dynamically achieving the compressed result. Step 1 includes: loading the teacher model as the model to be compressed, initializing the student model as the compressed model defined by hyperparameters, and inputting the training set into the teacher model and student model simultaneously in the same order. The training set consists of images. For each batch of data loaded, the intermediate output containing the probability values ​​of all classification results before the SoftMax layer of the teacher model is taken as the soft target. This soft target is then weighted with the hard target of the training set to become the final training target of the student model. The cross-entropy loss between the final training target and the output of the student model is used as the total loss for normal gradient regression. The model obtained after training is the compressed model.

2. The neural network model compression method based on optimal combination compression sequence according to claim 1, characterized in that, Step 2 includes: channel pruning. By probing the output feature size, convolution kernel size and output feature change magnitude during the model inference process, the importance index of each convolution kernel is determined, and the corresponding low-importance convolution kernels are removed from the model parameters, thereby compressing the computational complexity and model size.

3. The neural network model compression method based on optimal combination compression sequence according to claim 1, characterized in that, Step 3 includes: 8-bit quantization for the convolutional neural network. The quantization operation is performed during training. This operation converts the 32-bit floating-point model parameters into 8-bit integer parameters through a monotonically increasing function based on the upper and lower limits of the model parameters in the original convolutional kernel or the data distribution. The fine-tuning during training is performed together with the quantization operation. The quantized 8-bit integer parameters are used during forward propagation, and the unquantized 32-bit floating-point parameters are used during gradient regression. The quantized model composed of integers is directly obtained after quantization during training, thereby achieving model compression.

4. The neural network model compression method based on optimal combination compression sequence according to claim 1, characterized in that, Step 4 includes: embedding exit layers at several locations in the middle of the model, and training the model once for all exit layers, so that each exit layer can output the inference result with the highest accuracy that can be obtained by exiting at that location. During inference, each time the feature map of the middle layer reaches an exit layer node, it first attempts to exit from the exit layer. If the exit layer result does not meet the confidence threshold hyperparameter, the feature map continues to propagate forward in the original network. If the exit layer output meets the confidence threshold, the inference ends immediately, and the output of the exit layer is taken as the final inference result. 5.A neural network model compression system based on optimal combination compression sequence, characterized in that, include: Module M1: Apply architecture-level technical knowledge distillation, using the intermediate output of the original model as a soft target to train a custom student model to inherit the inference performance of the original model; Module M2: Applying neuron-level pruning techniques, specific nodes / node connections and their corresponding model parameters are identified and deleted at each layer of the student model based on a node importance index constructed using preset standards. Module M3: Utilizes subneuronal level quantization technology to directly change the number of bits in the student model parameters, converting each floating-point parameter of the original model into its corresponding integer parameter; Module M4: Uses dynamic architecture-level technology for early exit, taking effect during inference, dynamically adjusting the number of student model layers required for each inference, and dynamically changing the student model's own results during inference, thus dynamically achieving compressed results; The module M1 includes: loading the teacher model as the model to be compressed, initializing the student model as the compressed model defined by hyperparameters, and inputting the training set into the teacher model and student model simultaneously in the same order. The training set consists of images. For each batch of data loaded, the intermediate output containing the probability values ​​of all classification results before the SoftMax layer of the teacher model is taken as the soft target. This soft target is then weighted with the hard target of the training set to become the final training target of the student model. The cross-entropy loss between the final training target and the output of the student model is used as the total loss for normal gradient regression. The model obtained after training is the compressed model.

6. The neural network model compression system based on optimal combination compression sequence according to claim 5, wherein, The module M2 includes: Channel pruning determines the importance index of each convolution kernel by probing the output feature size, convolution kernel size and output feature change magnitude during the model inference process, and removes the corresponding low-importance convolution kernels from the model parameters, thereby compressing the computational complexity and model size.

7. The neural network model compression system based on optimal combination compression sequence according to claim 5, characterized in that, The module M3 includes: 8-bit quantization for convolutional neural networks. The quantization operation is performed during training. This operation converts 32-bit floating-point model parameters into 8-bit integer parameters through a monotonically increasing function, based on the upper and lower limits of the model parameters in the original convolutional kernel or the data distribution. The fine-tuning during training is performed together with the quantization operation. The quantized 8-bit integer parameters are used during forward propagation, and the unquantized 32-bit floating-point parameters are used during gradient regression. The quantized model composed of integers is directly obtained after quantization during training, thereby achieving model compression.

8. The neural network model compression system based on optimal combination compression sequence according to claim 5, characterized in that, The module M4 includes: embedding exit layers at several locations in the middle of the model, and performing model training once for all exit layers, so that each exit layer can output the inference result with the highest accuracy that can be obtained by exiting at that location. During inference, each time the feature map of the middle layer reaches an exit layer node, it first attempts to exit from the exit layer. If the exit layer result does not meet the confidence threshold hyperparameter, the feature map continues to propagate forward in the original network; if the exit layer output meets the confidence threshold, the inference ends immediately, and the output of that exit layer is taken as the final inference result.

Images