Model pruning method, device and computer-readable storage medium

Abstract

The present application provides a model pruning method, device and computer-readable storage medium. During the initial model training process, the connection strength and node importance between the layers of the initial model are dynamically evaluated, and the connections or nodes that contribute less to the model performance can be accurately identified. Redundant target connections and target nodes are selected for pruning based on dynamic indicators, and those connections or nodes that contribute less to the model performance are removed. Finally, based on the model performance of the initial model, the pruned initial model is fine-tuned to obtain a target model that meets the model performance requirements, and the performance of the model that may be lost due to pruning is restored, thereby improving the accuracy of pruning while maintaining or improving the performance of the model.

Description

Technical Field

[0001] The present application relates to the technical field of deep learning model optimization, and in particular to a model pruning method, device, and computer-readable storage medium. Background Art

[0002] With the booming development of artificial intelligence, large language models (LLMs) such as GPT and BERT, leveraging their massive parameter sizes, have demonstrated remarkable intelligence in various fields, including natural language processing (NLP), mathematical reasoning, and image recognition. For example, in natural language processing, large language models enable high-quality text generation, accurate sentiment analysis, and efficient machine translation. In mathematics, they can perform complex theorem proofs and numerical computations. In image recognition, they can achieve high-precision object detection and image classification.

[0003] However, this powerful processing capability comes with a huge demand for computing resources and storage space. In this context, model pruning technology has become an effective method. Model pruning aims to reduce the complexity of the model by removing redundant weights or neurons, thereby improving the model's operational efficiency.

[0004] Traditional model pruning methods rely on weight size to determine pruning targets, or employ random selection strategies. From a model structure perspective, weight size does not fully equate to its importance to the model's functionality. Randomly selecting pruning targets lacks in-depth consideration of the model's structure and functionality, and can inadvertently delete important model structures, significantly degrading model performance and failing to meet the accuracy requirements of practical applications. Summary of the Invention

[0005] To overcome the problems existing in the related art, this specification provides a model pruning method, device and computer-readable storage medium.

[0006] According to a first aspect of an embodiment of this specification, a model pruning method is provided, the method comprising:

[0007] During the initial model training process, dynamically evaluating the connection strength and node importance between the layers of the initial model;

[0008] Select redundant target connections and target nodes for pruning based on dynamic indicators;

[0009] Based on the model performance of the initial model, the pruned initial model is fine-tuned to obtain a target model that meets the model performance requirements.

[0010] According to a model pruning method provided by the present application, during the initial model training process, the connection strength and node importance between the layers of the initial model are dynamically evaluated, including:

[0011] The connection strength between the layers is obtained by normalizing the sum of the absolute values ​​of the weight matrices between the layers during the initial model training process;

[0012] The sum of the connection strengths of all layers directly connected to a node is determined to obtain the node importance of the node.

[0013] According to a model pruning method provided by the present application, during the initial model training process, after dynamically evaluating the connection strength and node importance between the layers of the initial model, the method further includes:

[0014] Periodically record the connection strength of each connection and the node importance of each node in n time steps;

[0015] Taking a weighted average of the connection strength of each connection in n time steps and the node importance of each node in n time steps to obtain a weighted average connection strength and a weighted average node importance;

[0016] The dynamic index is determined according to the distribution characteristics of the weighted average connection strength and the weighted average node importance. The dynamic index includes a connection strength threshold and a node importance threshold.

[0017] According to a model pruning method provided by the present application, taking a weighted average of the connection strength of each connection in n time steps and the node importance of each node in n time steps to obtain a weighted average connection strength and a weighted average node importance, including:

[0018] Determining a weight of the connection strength of each connection in n time steps and the node importance of each node in n time steps, wherein the weight decays exponentially with the advancement of the time step so that the closer the time step, the greater the weight;

[0019] Based on the weights, the weighted average connection strength of each connection in n time steps and the weighted average node importance of each node in n time steps are obtained.

[0020] According to a model pruning method provided by the present application, determining the dynamic index according to the distribution characteristics of the weighted average connection strength and the weighted average node importance includes:

[0021] According to the distribution characteristics of the weighted average connection strength and the weighted average node importance, the connection strength threshold is determined according to the weighted average connection strength corresponding to the set percentile, and the node importance threshold is determined according to the weighted average node importance corresponding to the set percentile.

[0022] According to a model pruning method provided by the present application, the method of selecting redundant target connections and target nodes for pruning based on dynamic indicators includes:

[0023] Determine a connection corresponding to a weighted average connection strength less than a connection strength threshold as a target connection, and determine a node corresponding to a weighted average node importance less than a node importance threshold as a target node;

[0024] A pruning operation is performed to reset the weights of the target connection and the target node to zero.

[0025] According to a model pruning method provided by the present application, the target node includes an isolated node,

[0026] The selecting redundant target connections and target nodes for pruning according to the dynamic indicators further includes:

[0027] Identify nodes without input or output connections as isolated nodes;

[0028] A pruning operation is performed to reset the weights of the isolated nodes to zero.

[0029] According to a model pruning method provided by the present application, the method of selecting redundant target connections and target nodes for pruning based on dynamic indicators further includes:

[0030] Identify the critical path for maintaining the core functionality of the initial model;

[0031] For connections and nodes outside the critical path, redundant target connections and target nodes are selected for pruning based on dynamic indicators.

[0032] According to a model pruning method provided by the present application, after selecting redundant target connections and target nodes for pruning based on dynamic indicators, the method further includes:

[0033] Dynamically monitoring the real-time performance of the initial model after pruning;

[0034] If the degradation of the real-time performance exceeds a preset acceptable performance loss, lowering the dynamic indicator;

[0035] Based on the adjusted dynamic indicator, redundant target connections and target nodes are selected for pruning according to the dynamic indicator.

[0036] According to a model pruning method provided by the present application, the method further includes:

[0037] If the degradation of the real-time performance does not exceed a preset acceptable performance loss, monitoring the parameter quantity of the pruning process model;

[0038] If the parameter amount reaches the expected parameter amount ratio, the pruning operation is terminated.

[0039] The present application also provides a model pruning device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the model pruning method described above is implemented.

[0040] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described model pruning methods.

[0041] The model pruning method, device and computer-readable storage medium in the embodiments of this specification, compared with the current pruning method that relies on the size of weights and random selection, which will delete important model structures and reduce model performance, dynamically evaluates the connection strength and node importance between the layers of the initial model during the initial model training process, and can accurately identify connections or nodes that contribute less to model performance. Redundant target connections and target nodes are selected for pruning based on dynamic indicators, and those connections or nodes that contribute less to model performance are removed. Finally, based on the model performance of the initial model, the pruned initial model is fine-tuned to obtain a target model that meets the model performance requirements, and the performance of the model that may be lost due to pruning is restored, thereby improving the accuracy of pruning while maintaining or improving the performance of the model.

[0042] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0044] Figure 1 is a flowchart of a model pruning method according to an exemplary embodiment of this specification;

[0045] Figure 2 is a flowchart of a rule calculation module according to an exemplary embodiment of this specification;

[0046] Figure 3 is a flow chart of a model pruning module according to an exemplary embodiment of the present specification;

[0047] Figure 4 is a flow chart of a model fine-tuning module according to an exemplary embodiment of this specification;

[0048] Figure 5 is a schematic diagram of a model pruning device according to an exemplary embodiment of this specification;

[0049] Figure 6 This is a schematic block diagram of a model pruning device according to an exemplary embodiment of the present specification. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments (or "implementations") of the present application will be described clearly and completely here with reference to the accompanying drawings. In the following description, when referring to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.

[0051] If there are terms in the embodiments of this application that refer to directional indications or positional relationships (such as up, down, left, right, front, back, inside, outside, top, bottom, center, vertical, horizontal, longitudinal, transverse, length, width, counterclockwise, clockwise, axial, radial, circumferential, etc.), such terms are only used to explain the relative positional relationship and movement of the components in a specific posture (as shown in the accompanying drawings); if the specific posture changes, the directional indication or positional relationship will also change accordingly. In addition, the terms "first" and "second" in the embodiments of this application are used only for descriptive convenience and should not be understood as indicating or implying relative importance.

[0052] This application provides a model pruning method, device, and computer-readable storage medium. This application is described in detail below with reference to the accompanying drawings. The features of the following embodiments and implementations may be combined with each other unless they conflict.

[0053] Traditional model pruning methods rely on the size of weights to determine pruning targets, or adopt a random selection strategy. From the perspective of model structure, the size of weights does not completely equate to their importance to the model's functionality. Some weights, while numerically small, may play a key role in the model's complex semantic understanding and feature representation. Deleting them may disrupt the model's key information transmission paths and affect the model's effective processing of input data. Randomly selecting pruning targets lacks in-depth consideration of model structure and functionality, and is very likely to inadvertently delete important model structures, resulting in a significant decline in model performance and an inability to meet the accuracy requirements of practical applications.

[0054] In order to solve the above technical problems, this specification provides a model pruning method.

[0055] By dynamically analyzing the inter-layer connections and node importance during training, the parts that have the greatest impact on model performance are retained while removing redundant structures. This solves the problem of incorrectly deleting important model structures in existing technologies, improves the accuracy of pruning, and maintains or improves model performance.

[0056] Figure 1 This is a flow chart of a model pruning method provided in an embodiment of this specification, which includes the following steps:

[0057] Step S100, during the initial model training process, dynamically evaluating the connection strength and node importance between the layers of the initial model;

[0058] Step S200, selecting redundant target connections and target nodes for pruning based on dynamic indicators;

[0059] Step S300: Based on the model performance of the initial model, fine-tune the pruned initial model to obtain a target model that meets the model performance requirements.

[0060] As an example, the initial model includes but is not limited to neural network models such as large language models (such as GPT, BERT, etc.), graphic recognition models (such as CNN models, etc.), recommendation system models (such as MLP, DNN-based, etc.), etc.

[0061] In step S100, during the initial model training process, the connection strength between layers and the importance of each node are evaluated during the training process to identify which connections or nodes contribute less to the overall performance of the model, and the pruning threshold is calculated to guide subsequent pruning steps.

[0062] As an example, during the initial model training process, dynamically evaluating the connection strength and node importance between the layers of the initial model includes:

[0063] The connection strength between the layers is obtained by normalizing the sum of the absolute values ​​of the weight matrices between the layers during the initial model training process;

[0064] The sum of the connection strengths of all layers directly connected to a node is determined to obtain the node importance of the node.

[0065] Reference Figure 2 ,First, the model is trained for t steps using the test data.

[0066] During the training process, the connection strength between layers and the importance of each node are periodically calculated.

[0067] The process of calculating the connection strength is to obtain the connection strength between each layer by normalizing the absolute value sum of the weight matrix between each layer during the initial model training process. As shown in the following formula (1):

[0068] At time step Time Layer and Connection strength between layers The definition of

[0069] (1)

[0070] in, Indicates that at time step Time Layer and The weight matrix between layers, yes of Norm, used to normalize the sum of the absolute values ​​of the weights.

[0071] The connection strength of each connection in the initial model at time step t is calculated using the above formula (1), which quantifies the dynamic importance of inter-layer connections and avoids static pruning that only relies on the absolute value of the weight.

[0072] The process of calculating node importance is to determine the sum of the connection strengths of all layers directly connected to the node to obtain the node importance of the node. This is expressed as the following formula (2):

[0073] At time step Node at time Importance is defined as:

[0074] (2)

[0075] in, Representation and Node All layers that are directly connected, From the layer To Node connection strength.

[0076] The node importance of each node in the initial model at time step t is calculated by formula (2) to evaluate the contribution of a single node to the model output and avoid accidental deletion of key nodes.

[0077] The aforementioned connection strength and node importance provide dynamic metrics for pruning, ensuring that pruning strategies are based on the model's actual learning process and avoiding the blindness of static pruning. These dynamic metrics include connection strength thresholds and node importance thresholds, which are explained in detail in the following sections.

[0078] Then, based on the connection strength of each connection and the node importance of each node, the pruning threshold is determined by weighted average and percentile method.

[0079] As an example, during the initial model training process, after dynamically evaluating the connection strength and node importance between the layers of the initial model, the method further includes:

[0080] Periodically record the connection strength of each connection and the node importance of each node in n time steps;

[0081] Taking a weighted average of the connection strength of each connection in n time steps and the node importance of each node in n time steps to obtain a weighted average connection strength and a weighted average node importance;

[0082] The dynamic index is determined according to the distribution characteristics of the weighted average connection strength and the weighted average node importance. The dynamic index includes a connection strength threshold and a node importance threshold.

[0083] Continue the above process and continue to record n ​​time steps through the above formulas (1) and (2) , ,…, The connection strength of each connection in The node importance of each node Information, record n ​​time steps , ,…, The connection strength in and information.

[0084] For the record and Take the weighted average to obtain the weighted average connection strength of each connection in n time steps The weighted average node importance of each node in n time steps .

[0085] It should be noted that in the process of obtaining the weighted average connection strength and the weighted average node importance, the weights of the connection strength of each connection in n time steps and the node importance of each node in n time steps are determined, and based on the weights, the weighted average connection strength of each connection in n time steps and the weighted average node importance of each node in n time steps are obtained.

[0086] As the model continuously updates weights during training, connection strengths or node importances can vary significantly across timesteps. Weighted averaging assigns higher weights to recent timesteps, ensuring that the pruning threshold reflects the model's current optimization trend or state. Therefore, we set the weights to decay exponentially with timesteps, giving more recent training steps a greater influence on the threshold calculation and adapting to the model's dynamic optimization process.

[0087] As an example, the weight setting rule follows the following formula:

[0088] (3)

[0089] in, is the initial weight coefficient (used to control the overall weight range, usually set to 0.8-1.0), is the attenuation coefficient (used to control the rate at which the weight decreases over time, usually set to 0.7-0.9).

[0090] Implement the weight factor for recent time steps (larger t values) The exponential decay characteristics of the time step are significantly higher than those of the early time step (smaller t value).

[0091] In this embodiment, the connection strength or node importance at different time steps (such as periodic evaluation points in the training process) is dynamically integrated through time-decay weights to highlight the impact of recent data and keep the pruning strategy consistent with the latest optimization state of the model.

[0092] Based on this, the weighted average connection strength of each connection in n time steps is obtained The weighted average node importance of each node in n time steps :

[0093] (4)

[0094] (5)

[0095] in, is the weight at time step t, obtained by the above formula (3).

[0096] Subsequently, according to the distribution characteristics of the weighted average connection strength and the weighted average node importance, the connection strength threshold is determined according to the weighted average connection strength corresponding to the set percentile, and the node importance threshold is determined according to the weighted average node importance corresponding to the set percentile.

[0097] As an example, based on the weighted average connection strength distribution and node importance distribution, the set percentile is taken as the connection strength threshold Dynamic importance threshold of nodes .

[0098] It should be noted that the percentiles set are used to determine the pruning threshold, that is, to determine which low-importance weights can be discarded, that is, set to 0, meaning that they do not participate in the calculation. In this way, redundant connections and nodes can be accurately removed while maintaining the basic performance of the model.

[0099] Among them, the percentile is dynamically adjusted as the model performance changes during the pruning process.

[0100] For example, take the first percentile as the connection strength threshold Dynamic importance threshold of nodes The "first percentile" refers to the threshold at which only 1% of the data in the distribution of connection strength or node importance falls below this threshold. This means that pruning removes the bottom 1% of low-strength connections and low-importance nodes. If the performance of the pruned model degrades by more than a preset value, the pruning threshold needs to be adjusted. For example, by lowering the pruning threshold percentile by half, from 1% to 0.5%, only the bottom 0.5% of connections or nodes will be pruned. This reduces model complexity while ensuring conservative and robust pruning through adaptive threshold selection based on distribution characteristics.

[0101] Through the above embodiment, the importance of inter-layer connections and nodes in the learning process is dynamically analyzed to quantify their contribution to model performance. This provides dynamic indicators for the pruning module, ensuring that the pruning strategy is based on the actual learning process of the model, avoiding the blindness of static pruning.

[0102] In step S200, unimportant connections or nodes are selected for pruning based on the connection strength, node importance and pruning threshold provided above.

[0103] As an example, selecting redundant target connections and target nodes for pruning based on dynamic indicators includes:

[0104] Determine a connection corresponding to a weighted average connection strength less than a connection strength threshold as a target connection, and determine a node corresponding to a weighted average node importance less than a node importance threshold as a target node;

[0105] A pruning operation is performed to reset the weights of the target connection and the target node to zero.

[0106] Reference Figure 3 First, according to the weighted average connection strength of each connection calculated above and the weighted average node importance of each node , filter out the pruning targets. That is, filter out < The connection to be pruned is the target connection, and the < The node is the target node to be pruned.

[0107] Next, a pruning operation is performed to reset the weights of the selected target connections and target nodes to zero.

[0108] During the pruning process, it is necessary to ensure the rationality of the model structure and check whether the pruning operation is performed according to the rules.

[0109] As an example, the target node includes an isolated node.

[0110] The selecting redundant target connections and target nodes for pruning according to the dynamic indicators further includes:

[0111] Identify nodes without input or output connections as isolated nodes;

[0112] A pruning operation is performed to reset the weights of the isolated nodes to zero.

[0113] The isolated nodes described in this specification refer to nodes in the model that have no input or output connections and are also targets for pruning. It is understood that if the weights of all input or output connections of a node are set to zero, the weight of the node is also set to zero.

[0114] In this way, network connectivity and functionality are maintained.

[0115] As another example, selecting redundant target connections and target nodes for pruning based on dynamic indicators further includes:

[0116] Identify the critical path for maintaining the core functionality of the initial model;

[0117] For connections and nodes outside the critical path, redundant target connections and target nodes are selected for pruning based on dynamic indicators.

[0118] The critical path described in this specification refers to the structure that is crucial to the model's functionality, such as residual connections (skip connections in ResNet), core weights in the attention mechanism (such as the Query-Key matrix in Transformer), or feature transfer paths related to specific tasks.

[0119] During pruning, it is necessary to ensure that the critical path is not affected to maintain the basic functionality of the model. Therefore, the critical path is excluded from pruning. It can be understood that the critical path can be used to calculate the connection strength and node importance in the above process, but it is not the target of pruning.

[0120] Through the above embodiments, the generation of isolated nodes is monitored and avoided, the critical path is protected, and the correctness of the model structure is ensured, thereby ensuring the accurate operation of the model.

[0121] Continue to refer to Figure 3 ,After each pruning operation, the model performance is evaluated and the pruning process is dynamically adjusted according to the model performance.

[0122] As an example, after selecting redundant target connections and target nodes for pruning according to the dynamic indicator, the method further includes:

[0123] Dynamically monitoring the real-time performance of the initial model after pruning;

[0124] If the degradation of the real-time performance exceeds a preset acceptable performance loss, lowering the dynamic indicator;

[0125] Based on the adjusted dynamic indicator, redundant target connections and target nodes are selected for pruning according to the dynamic indicator.

[0126] The term "real-time performance degradation" for this specification refers to a decrease in the accuracy, recall, and other evaluation metrics of the model on the test data after pruning compared to before pruning. For example, if the accuracy of a model on the test set is 95%, and after pruning it drops to 85%, this is considered performance loss.

[0127] The preset acceptable performance loss described in this specification is a pre-set limit used to control the impact of pruning on model performance.

[0128] If the performance degradation exceeds a preset acceptable performance loss (e.g., 10%), the dynamic index is lowered to restore the weights that were set to zero during the current pruning process. Based on the adjusted dynamic index, redundant target connections and target nodes are selected for pruning.

[0129] As you can understand, when evaluating model performance on the validation set, if the loss is too large, it indicates that the current threshold is too aggressive. This means that the pruning range and amount need to be narrowed to ensure model stability. Therefore, it is necessary to lower the dynamic indicators, that is, lower the connection strength threshold and the node importance threshold.

[0130] Because the dynamic metric is related to the percentile determined based on the distribution characteristics of the weighted average connection strength and weighted average node importance, the dynamic metric is adjusted by adjusting the percentile. As an example, if the performance degradation exceeds a preset acceptable performance loss (e.g., 10%), the pruning threshold percentile is reduced by half. For example, if the accuracy drops by 15% after pruning at the original threshold, the threshold is adjusted from 1% before pruning to only 0.5% of the low-strength connections before pruning.

[0131] In this way, the threshold percentile is actively lowered, reducing the pruning intensity to protect the performance of the model, and the threshold is dynamically adjusted through performance feedback to balance pruning efficiency and model performance.

[0132] As another example, the method further includes:

[0133] If the degradation of the real-time performance does not exceed a preset acceptable performance loss, monitoring the parameter quantity of the pruning process model;

[0134] If the parameter amount reaches the expected parameter amount ratio, the pruning operation is terminated.

[0135] The number of parameters mentioned in this manual refers to the number of parameters such as weights and biases in the model.

[0136] The core goal of pruning is to reduce the number of parameters, with the ultimate goal being a parameter reduction ratio, rather than a strict percentage. If partial pruning can meet the parameter target, early termination or dynamic adjustment of the threshold can be used to eliminate the need to prune the remaining low-importance connections. This avoids meaningless over-pruning, unnecessary computation, and potential performance risks.

[0137] That is, if the performance degradation does not exceed the preset acceptable performance loss, it is determined whether the number of parameters after pruning reaches the expected parameter reduction ratio. If the expected parameter reduction ratio is reached, the pruning operation is terminated early.

[0138] If the expected parameter ratio is not reached, a new round of pruning will be continued based on the current pruned model until the pruned parameter ratio reaches the expected parameter reduction ratio, and the pruning process ends.

[0139] In this way, through multiple pruning iterations, the parameter quantity target and performance protection are dynamically balanced to achieve efficient pruning.

[0140] Through the above-described embodiment, connections or nodes that contribute little to model performance can be accurately identified and removed. This not only reduces unnecessary computational operations but also reduces the time complexity of each forward and backward propagation. During the pruning process, the degree of degradation of model performance is monitored in real time, and the pruning threshold is dynamically adjusted to balance the pruning effect and model performance. This ensures the integrity and stability of the model structure, thereby removing redundant model structures and reducing the number of model parameters.

[0141] In step S300, the pruned model is fine-tuned using the dataset to recover the performance that may have been lost due to pruning.

[0142] Reference Figure 4 First, select some of the original training data or a richer dataset based on the task requirements. For large prediction models, these tasks include but are not limited to text generation, semantic understanding, question-answering interaction, text translation, mathematical reasoning, and more.

[0143] Next, the selected fine-tuning dataset is appropriately preprocessed to ensure the quality of the input data. For large language model datasets, the preprocessing methods include but are not limited to data cleaning, data enhancement, data formatting and structuring, etc.

[0144] Then, initialize the fine-tuning parameters and set a low learning rate to avoid large fluctuations in model performance due to large updates. For example, set the learning rate to 1 / 5 of the original learning rate.

[0145] Fine-tune the pruned initial model. Regularization and early stopping are added during training to prevent overfitting and improve generalization. The learning rate is also adaptively adjusted to accelerate model convergence and prevent overfitting.

[0146] After the above process, the pruned initial model is fine-tuned, and the performance loss caused by pruning is compensated by parameter adjustment, and the pruned model with restored performance is output, which is also the target model.

[0147] Finally, save the fine-tuned target model.

[0148] In this way, the pruned model is further optimized and the performance of the pruned model is restored through an adaptive learning strategy, so that the model maintains good performance while reducing the number of parameters, ensuring a balance between efficiency and accuracy.

[0149] The present application provides a model pruning method, device and computer-readable storage medium. Compared with the current pruning method that relies on the size of weights and random selection, which will delete important model structures and reduce model performance, the present application provides a model pruning method, device and computer-readable storage medium. During the initial model training process, the connection strength and node importance between the layers of the initial model are dynamically evaluated, and the connections or nodes that contribute less to the model performance can be accurately identified. Redundant target connections and target nodes are selected for pruning based on dynamic indicators, and those connections or nodes that contribute less to the model performance are removed. Finally, based on the model performance of the initial model, the pruned initial model is fine-tuned to obtain a target model that meets the performance requirements of the model, and the performance of the model that may be lost due to pruning is restored, thereby improving the accuracy of pruning while maintaining or improving the performance of the model.

[0150] Based on the same application concept as the above method, the embodiment of the present application also proposes a model pruning device, such as Figure 5 shown.

[0151] The device comprises:

[0152] A rule calculation module, used to dynamically evaluate the connection strength and node importance between the layers of the initial model during the initial model training process;

[0153] Model pruning module, used to select redundant target connections and target nodes for pruning based on dynamic indicators;

[0154] The model fine-tuning module is used to fine-tune the pruned initial model based on the model performance of the initial model to obtain a target model that meets the model performance requirements.

[0155] The implementation process of the functions and effects of each module / submodule / unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and the same technical effects can be achieved, so it will not be repeated here.

[0156] Figure 6 The following is an example of a physical structure diagram of a model pruning device, such as Figure 6 As shown, the model pruning device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call logic instructions in the memory 830 to execute the model pruning method.

[0157] In addition, the logical instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0158] On the other hand, the present application also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the model pruning method provided by the above methods.

[0159] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the model pruning method provided by the above methods.

[0160] It should be noted that the technical solutions or technical features described in the above embodiments can be combined or supplemented with each other without conflict. The scope of protection of this application is not limited to the precise structures described in the above embodiments and shown in the accompanying drawings; all modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this application shall be included in the scope of protection of this application.

Claims

1. A model pruning method, characterized in that: The method comprises: During the training of an initial model for image processing, dynamically evaluating the connection strength and node importance between layers of the initial model; wherein the dataset used to train the initial model is an image dataset; Select redundant target connections and target nodes for pruning based on dynamic indicators; Based on the model performance of the initial model, fine-tuning the pruned initial model using the image dataset to obtain a target model that meets the model performance requirements; After dynamically evaluating the connection strength and node importance between layers of the initial model during the initial model training process for image processing, the method further includes: Periodically record the connection strength of each connection and the node importance of each node in n time steps; Taking a weighted average of the connection strength of each connection in n time steps and the node importance of each node in n time steps to obtain a weighted average connection strength and a weighted average node importance; The dynamic indicator is determined according to the distribution characteristics of the weighted average connection strength and the weighted average node importance, and the dynamic indicator includes a connection strength threshold and a node importance threshold; wherein, the dynamic indicator is determined according to the distribution characteristics of the weighted average connection strength and the weighted average node importance, including: determining the connection strength threshold according to the weighted average connection strength corresponding to the set percentile according to the distribution characteristics of the weighted average connection strength and the weighted average node importance, and determining the node importance threshold according to the weighted average node importance corresponding to the set percentile.

2. The model pruning method according to claim 1, wherein: The method of dynamically evaluating the connection strength and node importance between layers of the initial model during the initial model training process for image processing includes: The connection strength between the layers is obtained by normalizing the sum of the absolute values ​​of the weight matrices between the layers during the initial model training process; The sum of the connection strengths of all layers directly connected to a node is determined to obtain the node importance of the node.

3. The model pruning method according to claim 1, wherein: The weighted average of the connection strength of each connection in n time steps and the node importance of each node in n time steps is taken to obtain the weighted average connection strength and the weighted average node importance, including: Determining a weight of the connection strength of each connection in n time steps and the node importance of each node in n time steps, wherein the weight decays exponentially with the advancement of the time step so that the closer the time step, the greater the weight; Based on the weights, the weighted average connection strength of each connection in n time steps and the weighted average node importance of each node in n time steps are obtained.

4. The model pruning method according to claim 1, wherein: The step of selecting redundant target connections and target nodes for pruning according to dynamic indicators includes: Determine a connection corresponding to a weighted average connection strength less than a connection strength threshold as a target connection, and determine a node corresponding to a weighted average node importance less than a node importance threshold as a target node; A pruning operation is performed to reset the weights of the target connection and the target node to zero.

5. The model pruning method according to claim 4, wherein: The target node includes an isolated node, The selecting redundant target connections and target nodes for pruning according to the dynamic indicators further includes: Identify nodes without input or output connections as isolated nodes; A pruning operation is performed to reset the weights of the isolated nodes to zero.

6. The model pruning method according to claim 4, wherein: The selecting redundant target connections and target nodes for pruning according to the dynamic indicators further includes: Identify the critical path for maintaining the core functionality of the initial model; For connections and nodes outside the critical path, redundant target connections and target nodes are selected for pruning based on dynamic indicators.

7. The model pruning method according to claim 1, wherein: After selecting redundant target connections and target nodes for pruning according to the dynamic indicators, the method further includes: Dynamically monitoring the real-time performance of the initial model after pruning; If the degradation of the real-time performance exceeds a preset acceptable performance loss, lowering the dynamic indicator; Based on the adjusted dynamic indicator, redundant target connections and target nodes are selected for pruning according to the dynamic indicator.

8. The model pruning method according to claim 7, characterized in that: The method further comprises: If the degradation of the real-time performance does not exceed a preset acceptable performance loss, monitoring the parameter quantity of the pruning process model; If the parameter amount reaches the expected parameter amount ratio, the pruning operation is terminated.

9. A model pruning device, characterized in that: The method comprises a memory, a processor and a model pruning program stored in the memory and executable on the processor, wherein the processor implements the steps of the model pruning method according to any one of claims 1 to 8 when executing the model pruning program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a model pruning program, which, when executed, implements the steps of the model pruning method according to any one of claims 1 to 8.

Images