Structured pruning algorithm based on intelligent pruning

Abstract

The invention discloses a structured pruning algorithm based on intelligent pruning, which is characterized in that model parameters of each layer are deeply evaluated through data analysis, and accurate pruning of a model structure is realized. The layer-by-layer parameter interpretation and design differentiation pruning algorithm ensures intelligent processing of different types of layers and operators, and the model volume is effectively reduced. After pruning is completed, model compatibility is ensured through data preprocessing, then intelligent retraining is carried out, the system spontaneously cuts part of pruning operation according to a built-in strategy, and the model precision is rapidly and effectively recovered. The patent provides a set of comprehensive and intelligent pruning process, an efficient and reliable technical scheme is provided for model optimization, compression and deployment, and the overall process is as shown in the figure 'intelligent pruning process'.

Description

Technical Field

[0001] This application relates to a structured pruning algorithm based on intelligent pruning. By reading the model to identify the parameters of each layer of the model, comprehensive analysis is carried out, and hierarchical intelligent pruning is performed with reference to the pruning algorithm library to compress the model volume. Subsequently, model retraining is carried out to complete accuracy recovery. Background Art

[0002] As a technology widely used in the field of computing, the pruning algorithm has its development rooted in the evolution of algorithm design and the theoretical exploration of computational complexity. The core idea of this algorithm is to reduce the search space of the problem through ingenious strategies to improve computational efficiency. Algorithm designers, by deeply studying the structure of the problem itself, are committed to identifying and eliminating redundant steps in the computational process, making the pruning algorithm perform outstandingly in solving various combinatorial optimization problems, search problems, and tasks such as decision tree construction in the field of artificial intelligence.

[0003] In addition, the development of the pruning algorithm is closely related to the theory of computational complexity. As the problem scale expands, the time complexity of traditional algorithms grows exponentially, making them ineffective in practical applications. The pruning algorithm, through in-depth analysis of problem complexity, selectively conducts searches, thereby theoretically reducing the time complexity of the algorithm and improving the feasibility of problem-solving. In the field of artificial intelligence, the pruning algorithm plays a key role. Whether in the process of decision tree construction or in the training of deep learning models, this algorithm provides an efficient learning path for the model, reduces unnecessary computational overhead, and enables artificial intelligence systems to make more accurate decisions more quickly under limited resources.

[0004] Compared with traditional pruning algorithms, the structured pruning algorithm based on intelligent pruning has the following advantages:

[0005] 1. Precise pruning target: The intelligent pruning algorithm has significant advantages in the selection of pruning targets. Through in-depth analysis and comprehensive parameter identification, this algorithm can accurately identify the parts that have less impact on the model performance and achieve more precise pruning. Compared with traditional heuristic rules, this precision enables the model to more effectively remove redundant information during the optimization process and minimize the complexity of the model structure to the greatest extent.

[0006] 2. Hierarchical processing: Adopting a hierarchical processing strategy is unique to the intelligent pruning algorithm. By processing the model structure in layers, the overall structure information of the model is retained, thereby reducing the adverse impact on the model performance. This hierarchical processing method makes the algorithm more meticulous and conservative, helps to avoid over-pruning, and thus maintains the overall characteristics and learning ability of the model.

[0007] 3. Intelligent Selection Strategy: The intelligent selection strategy of the intelligent pruning algorithm is an important source of its superiority. Based on deep learning and a profound understanding of the model, the algorithm can adopt a more intelligent selection strategy during pruning, which is more targeted and flexible than traditional heuristic rules. This makes the pruning process more guiding and better able to adapt to different types of model structures and task requirements.

[0008] This patent involves a structured pruning algorithm based on intelligent pruning. Its innovation lies in using data analysis methods to deeply analyze the model parameters of each layer and comprehensively consider the characteristics of the entire model. First, based on data analysis, the model parameters are interpreted through layer-by-layer analysis, and they are understood to the smallest granularity. This step aims to deeply understand the parameter distribution of each layer and provide comprehensive background information for subsequent pruning operations.

[0009] According to the analysis results, optimization schemes are designed using different pruning algorithms. In this process, different pruning strategies are adopted for different types of layers and operators to ensure that the pruning process is more intelligent and targeted. Then, the model is pruned according to requirements, deleting unnecessary parameters and layers, thereby significantly reducing the volume of the model and providing a more flexible choice for the deployment of the model in resource-constrained environments.

[0010] After pruning, by re-performing data preprocessing, it is ensured that the new model structure is consistent with the previous input and output. This step prepares for the subsequent retraining of the model. During the model retraining process, combined with the previous pruning process, the system will spontaneously prune some pruning operations according to the built-in strategy to balance the accuracy of the model and the optimization of the structure. This mechanism ensures the efficiency and reliability in restoring the model accuracy.

[0011] Overall, this patent provides a brand-new intelligent pruning algorithm, which realizes the structured pruning of the model through data analysis, hierarchical processing, and intelligent strategies. This innovation provides a more intelligent and efficient solution for model optimization, compression, and deployment.

[0012] The first step, data analysis and layer parameter evaluation. According to the intelligent pruning algorithm, data analysis is performed on the parameters of each layer of the model. By evaluating the distribution of the parameters of each layer, the structure and feature distribution of the model are understood, providing basic information for the pruning process (such as Figure 1 ).

[0013] The second step, layer-by-layer analysis and parameter interpretation. Based on the first step, layer-by-layer analysis is carried out to interpret the model parameters to the smallest granularity. This helps to deeply understand the role, contribution, and correlation of each layer, providing detailed parameter characteristics for precise pruning.

[0014] Step 3: Design of pruning strategies for different operators. Based on the results of layer-by-layer analysis, after considering the characteristics of the parameters in each layer in Step 2, different pruning algorithms are designed for different types of layers and operators. This ensures that the pruning process fully considers the diversity of the model structure and optimizes it more specifically.

[0015] Step 4: Model pruning operation. Prune the model according to the designed pruning algorithm. By deleting unnecessary parameters and layers, the volume of the model is effectively reduced, thereby reducing storage and computational costs. The model obtained after pruning has an improved structure compared to the previous model and is a new model (such as Figure 2 ).

[0016] Step 5: Data preprocessing. After obtaining the new model, the previous dataset needs to be preprocessed again. This step is to adapt to the new model structure, ensure that the input and output of the data are consistent with the model before pruning, and prepare for the next retraining.

[0017] Step 6: Model retraining and accuracy recovery. Retrain the model. In this process, combined with the previous pruning process, ensure that the model can quickly and effectively recover its accuracy under the new structure. When the accuracy recovery cannot be completed, the algorithm will spontaneously prune some pruning operations according to the built-in strategy to balance the optimization of model accuracy and structure.

[0018] This complete process of the intelligent pruning algorithm realizes precise and hierarchical pruning of the model through in-depth understanding of the model structure, parameters, and data, and ensures the accuracy recovery of the model through retraining. This algorithm not only has significant advantages in pruning effect but also provides strong technical support for the optimization and intelligence of the model structure. Description of the Drawings

[0019] Figure 1 To obtain a sample of the parameter distribution pattern of a certain layer of the model

[0020] Figure 2 Model structure changes before and after pruning Detailed Implementation Manner

[0021] To better describe the structured pruning algorithm based on intelligent pruning, the following gives the detailed implementation manner of this application.

[0022] First, conduct a detailed analysis of the parameters of each layer of the model through data analysis methods. In this step, data science techniques are used to evaluate the distribution of the parameters in each layer and explore the internal characteristics of the model. This helps to establish a comprehensive picture of the parameters and provides a basis for subsequent intelligent pruning.

[0023] Secondly, the model parameters are interpreted layer by layer in depth to understand the model structure down to the smallest granularity. Through this process, we can better understand the role and correlation of each layer, providing detailed parameter characteristics for subsequent differential pruning algorithms.

[0024] Next, based on the results of layer-by-layer analysis, pruning algorithms are designed for different types of layers and operators. This involves understanding the differences in the model structure and accurately grasping the pruning requirements. The differential pruning strategy ensures the intelligence of the algorithm, enabling it to more specifically remove redundant parameters and reduce the model size.

[0025] After the pruning operation is completed, data preprocessing needs to be redone to adapt to the new model structure. This step is to ensure that the input and output of the new model are consistent with those of the model before pruning, guaranteeing the compatibility of the model in practical applications.

[0026] Finally, the model is retrained. During this process, combined with the previous pruning process, the system will spontaneously prune some pruning operations according to the built-in strategy. This intelligent self-adaptive strategy ensures that the model can more quickly and efficiently recover its accuracy during retraining while maintaining the optimized structure.

Claims

1. A structured pruning algorithm based on intelligent pruning, characterized in that: Use data analysis methods to analyze the parameters of each layer. Considering the entire model comprehensively, different methods are used to prune different types of layers to reduce the model size. After pruning, the model accuracy will be restored through retraining. The specific steps are as follows: Step 1) Based on the interpretation of layer parameters by data analysis, evaluate the parameter distribution of each layer; Step 2) Call different pruning algorithms to design an optimization plan; Step 3) Prune the model according to requirements; Step 4) Re - perform data pre - processing according to the new model; Step 5) Retrain the model to restore accuracy.

2. The structured pruning algorithm for intelligent pruning according to claim 1, wherein Step 1) Analyze the parameters of the model layer by layer and interpret the model parameters at the smallest granularity.

3. The structured pruning algorithm for intelligent pruning according to claim 1, wherein Step 2) Different pruning algorithms are designed for different operators, and pruning operations are performed while retaining the operator characteristics.

4. The structured pruning algorithm for intelligent pruning according to claim 1, wherein Step 3) Prune the model according to the model to obtain the pruned model. The model has improvements in structure compared to the previous model and is a brand - new model.

5. The structured pruning algorithm for intelligent pruning according to claim 1, characterized in that Step 4) After obtaining the new model, it is necessary to re - pre - process the previous dataset to facilitate retraining.

6. The structured pruning algorithm for intelligent pruning according to claim 1, wherein Step 5) During the retraining process, the previous pruning process will be combined. When the accuracy restoration cannot be completed, some pruning strategies will be spontaneously removed.

Images