Extensible neural network pruning method
By using pruning methods such as expanding and weight transfer in neural networks, the problem of insufficient network structure flexibility is solved, and the stability and adaptability of model performance are improved, making it suitable for resource-constrained environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- UNIV OF CHINESE ACAD OF SCI
- Filing Date
- 2026-01-28
- Publication Date
- 2026-05-15
AI Technical Summary
Existing neural network pruning methods lack flexibility in network structure during model compression, have limited model capacity adjustment, are difficult to apply across different network architectures, and are unstable in the optimization process.
After initial pruning using the baseline pruning algorithm, the network structure is expanded and weights are transferred. Combined with lightweight fine-tuning and final fine-tuning, a hierarchical expansion strategy is used to adapt to different neural network types, and secondary pruning is performed to obtain the final model.
It improves the stability and model performance of the pruning search process, enhances adaptability, reduces computation and storage requirements, and is suitable for resource-constrained environments.
Smart Images
Figure CN122047356A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and more specifically, to a scalable neural network pruning method. Background Technology
[0002] In recent years, with the continuous increase in the scale and depth of deep neural network models, deep learning technology has been widely used in fields such as computer vision. However, large-scale neural network models are usually accompanied by high computational overhead and storage requirements, which to some extent limits their deployment and application in resource-constrained environments such as mobile devices and embedded systems. Therefore, how to reduce computational costs while ensuring model performance has become one of the key issues of concern in related technical fields.
[0003] Model compression techniques are considered an important means to alleviate the above problems. Among them, structured pruning methods, which can remove entire channels or convolutional kernels and have good hardware friendliness, have been widely studied and applied. To improve the performance of the pruned model, existing technologies have proposed a class of schemes that expand the neural network structure before pruning. For example, some studies widen the network layers or expand the structure for specific bottleneck layers before pruning, aiming to obtain a sub-network with better performance by increasing the initial model capacity.
[0004] However, existing structured pruning methods are typically based on discrete selection mechanisms that retain or discard channels, placing the pruning search process within a discrete architectural space. This discrete search characteristic makes model performance highly sensitive to changes in network structure, increasing the instability of the optimization process. Furthermore, many existing extended pruning schemes rely on empirical rules for network expansion, and the relationship between their expansion strategies and the pruning search space remains unclear, leaving room for further improvement.
[0005] Therefore, given the aforementioned problems with the stability of structured pruning search and the effectiveness of expansion strategies in existing technologies, it is still necessary to propose an improved neural network pruning method. Summary of the Invention
[0006] Technical issues
[0007] To address the problems of insufficient network structure flexibility, limited model capacity adjustment, and difficulty in generalizing among different network architectures in existing neural network pruning methods during model compression, it is necessary to propose an improved neural network pruning method.
[0008] Technical solution
[0009] To address the aforementioned technical problems, according to one aspect of the present invention, a scalable neural network pruning method is provided, the method comprising:
[0010] (1) The baseline pruning algorithm is used to perform initial pruning on the pre-trained neural network to obtain the initial pruning model;
[0011] (2) The network structure of the initial pruning model is extended, and the weights of the pre-trained neural network are transferred to the extended network structure to obtain a broadened model;
[0012] (3) Perform a light fine-tuning on the widened model to obtain a lightly fine-tuned widened model;
[0013] (4) The baseline pruning algorithm is executed again on the lightly fine-tuned broadened model to obtain a secondary pruning model; and
[0014] (5) Make final fine-tuning of the secondary pruning model to obtain the final model.
[0015] In one implementation, step (2), expanding the network structure of the initial pruning model, includes:
[0016] When the neural network is a convolutional neural network, the initial pruning model is applied according to a preset widening factor. A hierarchical expansion is performed, where the first convolutional layer expands only the number of output channels, the final fully connected layer expands only the number of input channels, and all intermediate layers expand according to the aforementioned widening factor. The number of its input channels and output channels are increased proportionally.
[0017] When the neural network is a Vision Transformer (ViT) model, the widening factor will be... It operates on the core embedding dimension of the model and simultaneously adjusts the parameters related to the embedding dimension in the patch embedding layer, category token, position encoding, multi-head self-attention module, multi-layer perceptron module, and layer normalization module to match the expanded embedding dimension.
[0018] In one implementation, step (2), the weight transfer includes:
[0019] Traversing the source weight tensors of the pre-trained neural network With the target weight tensor of the broadened model Determine the overlapping shape of the two in each dimension. ,in, and Represent tensors respectively and In the The size of each dimension;
[0020] Source weight tensor The sub-tensor corresponding to the overlapping shape is copied to the target weight tensor. The corresponding position; and
[0021] For the target weight tensor Medium size larger than source weight tensor The dimension, that is The weight migration initialization is completed by repeating the existing weight data of that dimension to fill the remaining positions of the target weight tensor.
[0022] In one implementation, the training cycle for the lightweight fine-tuning is set to 1 / 10 of the final fine-tuning training cycle.
[0023] Beneficial effects
[0024] Compared with existing technologies, the scalable neural network pruning method provided by this invention has at least the following beneficial effects:
[0025] (1) This invention expands the neural network structure after initial pruning and combines weight transfer and re-pruning to make the pruning search process take place in the expanded network space, which helps to alleviate the optimization instability caused by discrete channel selection in traditional structured pruning, so as to maintain model performance even with a high pruning ratio.
[0026] (2) When extending the network structure, the present invention designs a hierarchical extension strategy for convolutional neural networks and visual Transformer models respectively, so that the extension operation can match the structural characteristics of different types of neural networks, thereby improving the applicability of the method in various network architectures.
[0027] (3) By introducing weight transfer and weight repetition filling strategies into the extended network structure, the present invention can fully inherit the effective parameter information in the original pre-trained neural network, reduce the impact of the extension and re-pruning process on the model performance, and reduce the risk of model accuracy degradation.
[0028] (4) The present invention combines light fine-tuning and final fine-tuning during the pruning process, so that the model parameters can gradually adapt to the changes in network structure at each stage. While ensuring the model compression effect, it helps to improve the inference accuracy of the pruned model.
[0029] (5) The neural network model obtained by the method of the present invention is effectively compressed in terms of parameter scale and computational overhead, and can better adapt to the resource-constrained computing environment, and has high engineering application value. Attached Figure Description
[0030] Figure 1 This is a flowchart illustrating a scalable neural network pruning method according to an embodiment of the present invention;
[0031] Figure 2 This is a schematic diagram illustrating the process of expanding the network structure of an initial pruning model and transferring the weights of a pre-trained neural network according to an embodiment of the present invention. Detailed Implementation
[0032] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that those skilled in the art can readily understand the present invention.
[0033] This document only describes the parts necessary for understanding the technical content of the present invention, and the description of the remaining parts will be omitted to avoid confusion about the essence of the present invention. This should be noted. Moreover, in this process, for the sake of clarity and convenience of description, the thickness of the lines or the size of the constituent elements shown in the figures may be exaggerated.
[0034] The terminology used herein is for describing embodiments and is not intended to limit or restrict the invention. When describing a component as being "connected," "combined," or "joined" with another component, this includes not only direct connections but also indirect connections where other components are present in between. Furthermore, terms such as "comprising," "including," or "having" indicate the presence of features, numbers, steps, operations, components, or combinations thereof described in the specification, and do not exclude the existence or additional possibility of one or more other features, numbers, steps, operations, components, or combinations thereof. Additionally, terms such as "first" and "second," which may be used herein, are used only to distinguish one component from another and, unless specifically stated otherwise, do not limit the order or importance of the components. Therefore, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.
[0035] The structure and method of the present invention will be described in detail below with reference to the accompanying drawings.
[0036] Figure 1 This is a flowchart illustrating a scalable neural network pruning method according to an embodiment of the present invention.
[0037] Reference Figure 1 The scalable neural network pruning method of the present invention includes the following steps:
[0038] In step S1, based on the pre-trained neural network, the baseline pruning algorithm is executed to perform initial pruning processing on the pre-trained neural network to obtain an initial pruning model.
[0039] The baseline pruning algorithm can be a structured pruning algorithm known to those skilled in the art, such as a channel- or kernel-level structured pruning algorithm, used to remove network structural units that contribute little to model performance. This invention does not limit the specific form of the baseline pruning algorithm.
[0040] Next, step S2 is performed using a scalable neural network pruning method.
[0041] In step S2, the network structure of the initial pruning model obtained in step S1 is expanded and the weights of the pre-trained neural network are transferred.
[0042] Specifically, refer to Figure 2 The network structure of the initial pruning model is expanded and the weights of the pre-trained neural network are transferred.
[0043] Specifically, such as Figure 2 As shown, in step S201, an initial pruning model is obtained, and a preset widening factor is determined. Specify the type of neural network to be extended (convolutional neural network or Vision Transformer model).
[0044] Then, if the type of neural network to be expanded is a convolutional neural network, proceed to step S202 and adjust according to the widening factor. The network structure of the initial pruning model is expanded in layers, as follows:
[0045] Sub-step 1: For the first convolutional layer of the initial pruning model, only expand its output channel number, while keeping the input channel number consistent with the initial pruning model;
[0046] Sub-step 2: For the final fully connected layer of the initial pruning model, only expand its input channel number, while keeping the output channel number unchanged to match the target task requirements;
[0047] Sub-step 3: For all intermediate layers (excluding the first convolutional layer and the final fully connected layer) of the initial pruning model, apply a widening factor. The number of input channels and output channels are expanded synchronously in a proportional manner.
[0048] Then, if the neural network to be expanded is a Vision Transformer (ViT) model, proceed to step S203 and adjust according to the widening factor. The network structure of the initial pruning model is expanded in layers, as follows:
[0049] Sub-step 1: Widening factor Applying to the core embedding dimension of the ViT model, we determine the widened target embedding dimension.
[0050] Sub-step 2: Synchronously adjust the output channel dimension, class token dimension, and positional encoding dimension of the patch embedding layer to match the widened core embedding dimension.
[0051] Sub-step 3: Based on the widened embedding dimension, simultaneously widen the width of the associated linear layers in the multi-head self-attention (MSA) block and the multilayer perceptron (MLP) block, including the query, key, and value projection layers in the MSA block, as well as all linear layers in the MLP block;
[0052] Sub-step 4: Update the feature dimension parameters in the LayerNorm module to ensure that they completely match the widened embedding dimension, thus guaranteeing the effectiveness of the normalization operation.
[0053] Then, proceed to step S204, in which all weight tensors S of the original pre-trained neural network are obtained, and the target weight tensor T of the expanded model is initialized (the dimension matches the expanded structure of S202 or S203).
[0054] Then, proceed to step S205. In step S205, iterate through each dimension of the weight tensors S and T, and calculate the overlap shape (copy_shape) of the two tensors in each dimension. The calculation formula is as follows: ,in and Represent tensors respectively and In the The size of each dimension.
[0055] Then, proceed to step S206, in which the overlapping sub-tensor corresponding to copy_shape in the pre-trained weight tensor S is completely copied to the corresponding position in the widened model target weight tensor T, thus completing the basic weight transfer.
[0056] Then, proceed to step S207, in which the dimensions of the target weight tensor T that are larger than those of the source weight tensor S (i.e., satisfying...) are... The weights of the expanded model are filled by repeating the weights data that have been migrated in the dimension T, thus completing the full initialization of the weights of the expanded model.
[0057] Next, return Figure 1 Step S3 is performed in the scalable neural network pruning method.
[0058] In step S3, a light fine-tuning process is performed based on the broadened model obtained in step S2.
[0059] Specifically, a gradient descent algorithm with a low learning rate is used to fine-tune the broadened model after initial pruning to recover the impact of the pruning operation on model performance, thereby obtaining a lightweight target model. This lightweight fine-tuning process can employ post-pruning fine-tuning strategies known to those skilled in the art.
[0060] In one embodiment, the training cycle for the lightweight fine-tuning is set to 1 / 10 of the final fine-tuning training cycle to quickly adapt the broadened model parameters.
[0061] Next, step S4 is performed in the scalable neural network pruning method.
[0062] In step S4, based on the broadened model obtained in step S3 after slight fine-tuning, the baseline pruning algorithm is executed again for pruning.
[0063] The baseline pruning algorithm is a structured pruning algorithm known to those skilled in the art, and is consistent with the baseline pruning algorithm used in step S1.
[0064] Next, step S5 is performed in the scalable neural network pruning method.
[0065] In step S5, the final fine-tuning process is performed based on the secondary pruning model obtained in step S4.
[0066] The method used for the final fine-tuning is the same as the method used for the light fine-tuning in step S3.
[0067] After completing the final fine-tuning in step S5, the final pruned neural network model is obtained.
[0068] This completes the implementation process of the scalable neural network pruning method described in this embodiment.
[0069] The scope of the claims of this invention is not limited to the specific embodiments described above. Various other embodiments, including modifications or alterations that can be made by those skilled in the art without departing from the spirit and intent of the invention as described in the claims, should also be included within the scope of the claims of this invention.
Claims
1. A scalable neural network pruning method, characterized in that, include: The baseline pruning algorithm is used to perform initial pruning on the pre-trained neural network to obtain the initial pruning model; The network structure of the initial pruning model is expanded, and the weights of the pre-trained neural network are transferred to the expanded network structure to obtain a broadened model. The broadened model was lightly fine-tuned; The baseline pruning algorithm is then applied again to the broadened model after minor adjustments to obtain the secondary pruning model. as well as The secondary pruning model is then fine-tuned.
2. The method according to claim 1, characterized in that, According to the preset widening factor The network structure of the initial pruning model is extended; Wherein, when the neural network is a convolutional neural network, the extension includes: The number of output channels is expanded only for the first convolutional layer of the initial pruning model; Only the number of input channels is expanded for the final fully connected layer; and Apply the widening factor to all intermediate layers. The number of its input channels and output channels are increased proportionally.
3. The method according to claim 1, characterized in that, When the neural network is a visual converter model, according to the widening factor Extending the network structure of the initial pruning model includes: The widening factor The core embedding dimension that operates on the visual converter model; Synchronously adjust the dimensions of the patch embedding layer, category token, and location encoding; Widen the linear layers related to the embedding dimension in the multi-head self-attention module and the multilayer perceptron module; and Update the feature dimension parameters in the layer normalization module to match the widened embedding dimension.
4. The method according to claim 2 or 3, characterized in that, The widening factor The value range is from 1.25 to 2.
0.
5. The method according to claim 1, characterized in that, The transfer of weights from the pre-trained neural network to the broadened model includes: Determine the overlap region in each dimension between the source weight tensor of the pre-trained neural network and the target weight tensor of the broadened model; Copy the weight data corresponding to the overlapping region in the source weight tensor to the corresponding position in the target weight tensor; and For dimensions in the target weight tensor that are larger than the source weight tensor, the remaining positions in the target weight tensor are filled by repeating the migrated weight data in that dimension.
6. The method according to claim 1, characterized in that, The process of performing lightweight fine-tuning on the broadened model includes setting the training period for lightweight fine-tuning to 1 / 10 of the training period for final fine-tuning.
7. The method according to claim 1, characterized in that, The step of expanding the network structure of the initial pruning model and transferring the pre-trained neural network weights is performed after the initial pruning is completed and before the baseline pruning algorithm is executed again.
8. The method according to claim 1, characterized in that, The initial pruning and the subsequent baseline pruning both employ the same type of structured pruning algorithm.
9. A computer-readable storage medium having stored thereon computer-readable instructions, which, when executed by a processor, are used to implement the scalable neural network pruning method of any one of claims 1 to 8.