An optimization method and system for training a post-sparse model

Abstract

The application discloses an optimization method and system for a post-training sparse model. The optimization method optimizes the sparsification of models for different task types: for classification and detection tasks, a sparsity of 50% to 80% is selected, and the FCPTS method is preferentially used for sparse allocation, combined with block-by-block reconstruction techniques with or without error correction; for generation tasks, the sparsity is controlled below 50%, and FCPTS is preferentially selected, combined with block-by-block reconstruction techniques with or without error correction. If the task includes a generation task, special processing is performed to maintain the sparsity below 50%. The application optimizes the post-training sparse model through an innovative method, significantly improving the efficiency and performance of the model in practical applications.

Description

Technical Field

[0001] This invention relates to an optimization method for a post-trained sparse model, and also to a corresponding optimization system for a post-trained sparse model, belonging to the field of machine learning technology. Background Technology

[0002] Deep learning technology has garnered significant attention due to its wide application across various fields, but its enormous demands on memory and computing power also present challenges. To address this issue, researchers have developed various model compression strategies, including model quantization, sparsification, network distillation, lightweight network design, and low-rank decomposition. Among these methods, post-training sparsification (PTS) has attracted attention due to its relatively low training cost.

[0003] Several representative methods have emerged in the field of post-training sparsity, such as the Post-training Optimization Tool (POT), AdaPrune, and Optimal Brain Surgeon (OBC). These methods can maintain model performance with almost no loss after sparsification. Although these high-performance PTS methods have been implemented, the relationship between models and sparsity has not been fully explored. Current research mostly evaluates models on specific models and tasks, lacking discussion on the sparsification potential of different models. Therefore, it remains unclear which models are more suitable for sparsification for different architectures, sizes, and tasks.

[0004] Existing PTS research often selects commonly used models (such as ResNet and RegNetX) or datasets to validate methods, but lacks a comprehensive evaluation of the relationship between models and sparsity. In practical applications, it is often necessary to sparsify multiple network architectures with similar sizes to achieve efficient inference, but it is currently unclear which network architecture is more sparsity-friendly and achieves better performance. Furthermore, with changes in deployment platforms, different model sizes (e.g., different numbers of layers) may be required. Current research has not clearly demonstrated whether network architectures with different sizes are robust to sparse algorithms. At the same time, existing research has not considered the generalization ability of sparse models to different tasks. Model structures for different tasks typically need to be designed and modified for each task, and the benefits of these task-oriented designs and modifications to sparsity are unclear. Therefore, optimizing models to improve their sparsity capabilities for specific application needs and / or specific models is essential. Summary of the Invention

[0005] The primary technical problem to be solved by this invention is to provide an optimization method for a trained sparse model (hereinafter referred to as the model).

[0006] Another technical problem to be solved by the present invention is to provide an optimization system for a sparse model after training.

[0007] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution:

[0008] According to a first aspect of the present invention, an optimization method for a trained sparse model is provided, comprising the following steps:

[0009] S1: Determine whether the model is used for a single task. If not, proceed to step S5. If it is used for a classification task, proceed to step S2. If it is used for a detection task, proceed to step S3. If it is used for a generation task, proceed to step S4.

[0010] S2: For models used in classification tasks, the sparsity rate must be 50% ≤ sparsity < 80%, and the sparsity allocation method should be selected in the priority order of FCPTS > ERK > L2Norm. The reconstruction technique should employ error correction and block-by-block reconstruction techniques.

[0011] S3: For models used in detection tasks, the sparsity rate must be 50% ≤ sparsity < 80%, and the sparsity allocation method should be selected in the priority order of FCPTS > ERK > L2Norm. The reconstruction technique should be error-free correction and block-by-block reconstruction.

[0012] S4: For the model used for the generation task, the sparsity rate should be ≤50%, and the sparsity allocation method should be selected in the priority order of FCPTS > L2Norm > ERK. The reconstruction technique should be either error-free and block-by-block reconstruction, or error-corrected and block-by-block reconstruction.

[0013] S5: Determine whether the task contains a generated task. If it does, proceed to step S6; otherwise, proceed to step S7.

[0014] S6: If the sparsity is ≤50% and the sparse allocation method is FCPTS, the reconstruction technique adopts error-free correction and block-by-block reconstruction.

[0015] S7: If the sparsity is 50% ≤ sparsity < 80%, and the sparsity allocation method is selected in the priority order of FCPTS > ERK > L2Norm, the reconstruction technique adopts error-free correction and block-by-block reconstruction.

[0016] According to a second aspect of the present invention, another method for optimizing a trained sparse model is provided, comprising the following steps:

[0017] S10: Determine whether it is a model for a classification task. If not, proceed to step S20. If it is, it must satisfy 50% ≤ sparsity rate < 80%, and the sparsity allocation method must be selected in the priority order of FCPTS > ERK > L2Norm. The reconstruction technique adopts the technique of error correction and block-by-block reconstruction.

[0018] S20: Determine whether it is a model used for the detection task. If not, proceed to step S30. If it is, for the model used for the detection task, it satisfies 50% ≤ sparsity rate < 80%, and the sparsity allocation method is selected in the priority order of FCPTS > ERK > L2Norm. The reconstruction technique adopts error-free correction and block-by-block reconstruction techniques.

[0019] S30: For the model used for the generation task, the sparsity is ≤50%, and the sparsity allocation method is selected in the priority order of FCPTS > L2Norm > ERK. The reconstruction technique adopts the technique of error-free correction and block-by-block reconstruction, or the technique of error correction and block-by-block reconstruction.

[0020] According to a third aspect of the present invention, another method for optimizing a trained sparse model is provided, comprising the following steps:

[0021] S01: Determine the model size. If the model size is large, select a Transformer-based model architecture; if the model size is medium, select a CNN-based model architecture.

[0022] S02: Determine the task type. If it is a classification task, proceed to step S03; if it is a detection task, proceed to step S04; if it is a generation task, proceed to step S05.

[0023] S03: For models used in classification tasks, the sparsity rate must be 50% ≤ sparsity < 80%, and the sparsity allocation method should be selected in the priority order of FCPTS > ERK > L2Norm. The reconstruction technique should employ error correction and block-by-block reconstruction techniques.

[0024] S04: For models used in detection tasks, the sparsity rate must be 50% ≤ sparsity < 80%, and the sparsity allocation method must be selected in the priority order of FCPTS > ERK > L2Norm. The reconstruction technique adopts error-free correction and block-by-block reconstruction techniques.

[0025] S05: For the model used for the generation task, the sparsity should be ≤50%, and the sparsity allocation method should be selected in the priority order of FCPTS > L2Norm > ERK. The reconstruction technique should be either error-free and block-by-block reconstruction, or error-corrected and block-by-block reconstruction.

[0026] Preferably, when the sparse allocation method selects FCPTS, the FCPTS includes FCPTS(ERK) and FCPTS(L2Norm), and FCPTS(ERK) is preferred over FCPTS(L2Norm).

[0027] Preferably, in the sparse allocation method, the sparsity of the last layer is ≤50%.

[0028] Preferably, in the sparse allocation method, the sparsity of the downsampling layer is ≤50%.

[0029] Preferably, the Transformer-based model architecture is the ViT model architecture.

[0030] Preferably, the CNN-based model architecture is the MobileNetV3 model architecture.

[0031] According to a fourth aspect of the present invention, an optimization system for a trained sparse model is provided, comprising a processor and a memory, wherein the processor and the memory are coupled together; wherein the memory is used to store a computer program; and the processor is used to run the computer program stored in the memory to execute the above-described optimization method for the trained sparse model.

[0032] Compared to existing technologies, this invention optimizes the trained sparse model through an innovative method, significantly improving the model's efficiency and performance in practical applications. By employing customized optimization strategies for different task types (classification, detection, and generation), this invention maintains model accuracy and effectiveness while reducing model complexity. Furthermore, this invention considers the adaptability of model size and architecture, providing flexible model selection to meet the needs of applications of varying scales. Through this optimization, the model can better adapt to changing deployment environments with less resource consumption, exhibiting stronger robustness and generalization ability. Ultimately, this invention not only improves the model's runtime efficiency but also enhances its applicability and practicality in diverse tasks. Attached Figure Description

[0033] Figure 1A A schematic radar chart illustrating the evaluation indicators;

[0034] Figure 1B Figure 1 shows a schematic diagram illustrating the relationship between the evaluation metrics and the corresponding tracks.

[0035] Figure 2 A schematic diagram of the sparse paradigm used in PTS technology;

[0036] Figure 3 A flowchart of the optimization method for a trained sparse model provided in the first embodiment of the present invention;

[0037] Figure 4 This is a flowchart illustrating the selection of sparse allocation methods according to the priority order of FCPTS(ERK) > FCPTS(L2Norm) in the first embodiment of the present invention.

[0038] Figure 5 A flowchart of the optimization method for a trained sparse model provided in the second embodiment of the present invention;

[0039] Figure 6 A flowchart of the optimization method for a trained sparse model provided in the third embodiment of the present invention;

[0040] Figure 7 This is a schematic diagram of the optimization system for the trained sparse model provided in the fourth embodiment of the present invention. Detailed Implementation

[0041] The technical content of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0042] This invention provides a method and system for optimizing post-trained sparse models, aiming to optimize the original model to improve its sparsity capability according to specific needs, such as specific tasks, model size, or model architecture. Before explaining the optimization method and system for post-trained sparse models provided by this invention, it is necessary to first explain the benchmark evaluation method for post-trained sparse models.

[0043] Benchmark evaluation methods for post-trained sparse models aim to construct a comprehensive evaluation system to better select and optimize models. For example... Figure 1A and Figure 1B As shown, the evaluation system includes five metrics: sparsity allocation method, reconstruction technique, model architecture accuracy, size robustness, and task accuracy for different tasks (including classification, detection, and image generation). To optimize the evaluation results, this method first employs a standardized sparsity paradigm, performing sparsification on the original model based on these five evaluation metrics to generate the model under test. Subsequently, the same five metrics are used to evaluate the sparsification capability of the model under test. It should be noted that the standardized sparsity paradigm is not limited to the method in this embodiment; other sparsity paradigms can also be used, as long as the original models are sparsified under the same evaluation metrics for comparison of sparsification capabilities.

[0044] Specifically, this invention first addresses the needs of actual model production by designing a post-training sparse benchmark evaluation system that includes five tracks and their corresponding metrics, such as... Figure 1A and Figure 1BAs shown, this system provides a comprehensive evaluation framework through five metrics—sparse allocation method, reconstruction technique, model architecture accuracy, size robustness, and task accuracy for different tasks (classification, detection, and image generation)—aiming to overcome the limitations of current research on post-trained sparsity (PTS) and to effectively evaluate various PTS models and their performance. In one embodiment of this invention, we first sparsify the original model using a standardized sparsity paradigm to generate the model under test. This process involves sparsifying the five evaluation metrics of the original model. Then, these five evaluation metrics are used to evaluate the sparsified model under test. Therefore, the following content will first introduce the standardized sparsity paradigm and then explain how to apply the post-trained sparsity benchmark evaluation system composed of these five evaluation metrics to evaluate the model under test.

[0045] In one embodiment of the present invention, the original model is sparsified using a standardized sparse paradigm to obtain the model to be tested, including the following steps:

[0046] S11: Calculate the indicator A for a specific task based on the original model. * Then, sparse models are obtained using different sparse allocation methods, and the performance metrics of the sparse models for specific tasks are calculated. Finally, based on A * Calculate the relative values ​​of the metrics for all architectures and datasets on each task, and use their mean as the overall sparse distribution metric OM. alloc .

[0047] As is well known, PTS (Parasitic Sparse Tree) techniques aim to sparsify pre-trained neural networks while maintaining their accuracy on specific tasks (classification, detection, and image generation, or any one or more). Currently, existing PTS techniques employ a two-step sparsity paradigm involving sparse allocation and reconstruction, such as... Figure 1B As shown. During the sparse allocation process, a specific sparsity rate is assigned to each layer according to a predetermined index, and the weights at the corresponding positions are reset to zero according to the sparsity criteria. The reconstruction process is used to restore the model accuracy that has decreased due to sparsification.

[0048] Currently, sparse allocation methods can be divided into the following three categories:

[0049] Heuristic strategy: In this strategy, the sparsity of each layer is manually predetermined, such as uniform sparsity;

[0050] Metric-based strategy: Sort the weights of all layers according to a specific metric and set the weights with lower scores to zero according to a certain proportion;

[0051] Learning-based strategy: Learn the sparsity rate of each layer by optimizing the loss function to achieve optimal sparsity distribution.

[0052] In one embodiment of the present invention, in order to evaluate the performance of different sparse allocation methods and to study in depth the characteristics and essence of an ideal sparse allocation, any one of the following four sparse allocation methods can be selected:

[0053] Uniform sparse allocation: This is a widely adopted heuristic strategy for sparse allocation that assigns the same sparsity to each layer.

[0054] L2 Norm (L2 Normalization): This is a metric-based sparse allocation method that determines the sparsity rate by calculating the L2 norm of each layer;

[0055] ERK (Effective Rank-based Knowledge): This is also a sparse allocation method based on metrics, but it is more refined than L2Norm. Both L2Norm and ERK methods have high efficiency and high citation rates.

[0056] FCPTS (Fine-Grained Channel Pruning via Reinforcement Learning): This is a typical learning-based method and the only learning-based sparse allocation method.

[0057] It should be noted that although four sparse allocation methods were selected in the embodiments of this invention to sparsify the original model, and three types of methods were used: heuristic, index-based, and learning-based, ERK and L2Norm are two representative index-based methods. However, those skilled in the art can also use other sparse allocation methods.

[0058] Using the accuracy of the original model as a benchmark, calculate the average relative accuracy of all architectures and datasets on each task, including the following sub-steps.

[0059] S111: Calculate the performance index A of the original model on one or more tasks. * .

[0060] In one embodiment of the present invention, the accuracy A of the original model on the classification (CLS) task is calculated respectively. CLS The mAP (mean Average Precision) value A for the detection (DET) task. DET Or the FID (Frechet Inception Distance) value A of the image generation (GEN) task. GEN Here, we use accuracy A. * Representing A CLS A DET and AGEN and collectively.

[0061] Accuracy in classification tasks, mAP in detection tasks, and FID in image generation tasks are commonly used metrics for evaluating models, which will not be elaborated here.

[0062] S112: Based on the selected sparse allocation method, the original model is sparsified to obtain a sparse model.

[0063] As mentioned above, the sparsity allocation methods in this embodiment of the invention include four types (uniform sparsity allocation, L2Norm, ERK, and FCPTS). Therefore, the original model is sparsified using any one of these four methods to obtain a sparse model. It is understood that the sparsity allocation methods are not limited to these four; other sparsity allocation methods, such as gradient-based sparsity rate allocation or density estimation-based sparsity rate allocation (KDE), can also be used, as long as the same sparsity allocation method is applied to each original model to achieve evaluation.

[0064] S113: Calculate the sparsity allocation index of the sparse model in one or more tasks.

[0065] In one embodiment of the present invention, the accuracy of the sparse model on the classification task is calculated respectively. mAP value of the detection task or the FID value of the image generation task Here, sparse allocation index is used. represent and and collective term,

[0066] S114: Calculate the quadratic mean of the relative values ​​of the indicators for each task as the overall sparse distribution indicator OM. alloc .

[0067]

[0068] Among them, A * This represents the metrics calculated by the original model for a specific task (i.e., classification, detection, and image generation). This represents the index results calculated by various sparse models under different sparsity rates for a specific task. This is the mean operator. It's important to note that FID is an indicator where lower values ​​are better, and its range is [1, +∞]. To ensure the overall sparse distribution, the OM indicator... alloc The value is in the interval [0, 1], so we take its reciprocal.

[0069] Through the above steps, the selected original model is sparsified under different sparse allocation algorithms, and the OM corresponding to each sparse allocation algorithm is calculated. alloc As can be seen, this step measures the accuracy of the original model after sparsification without reconstruction. This allows for direct observation of the impact of different sparsity allocation strategies, distinguishing the accuracy effects of different sparsity allocation strategies and different reconstruction methods. This distinction is more conducive to the performance analysis and optimization of the PTS method.

[0070] S12: Based on the sparse model in step S11, calculate the difference in the index results before and after reconstruction for all architectures and datasets on each task, and then calculate the average value to obtain the overall reconstruction index.

[0071] After sparsity ratio allocation, the PTS method applies reconstruction to reconstruct the sparse activations to compensate for the accuracy loss caused by sparsity. Refined reconstruction techniques that can significantly affect the effectiveness of sparsity during the sparsification process include: error correction, reconstruction input, and reconstruction granularity, as detailed below:

[0072] Error correction is widely used in many post-training quantization (PTQ) methods. Its purpose is to align the compressed weight distribution with the original weight distribution. Specifically, the error correction process can be written as follows:

[0073]

[0074] After the error correction operation and It is the corrected weights and biases, W s This represents the weights of the sparse model before correction. d W d and X d These represent the biases, weights, and input activations in the original model. f(W, X) indicates that the network layer performs a convolution operation on the input X with weights W. E and σ are the mean and standard deviation operators, respectively, and ∈ is a small constant. In this way, errors caused by the distributional shift of the weights and biases can be corrected.

[0075] During the reconstruction process, you can choose to use the output of the previous reconstruction unit in the dense model as input (dense input) or the output of the previous sparse unit as input (sparse input). The reconstruction input has a significant impact on the results.

[0076] The reconstruction process can be performed at different granularities. Many post-training quantization methods have demonstrated that the reconstruction granularity has a significant impact on quantization performance.

[0077] In one embodiment of the present invention, three reconstruction granularities were tested:

[0078] Single reconstruction: Weight reconstruction is performed based on each individual layer, which is the smallest reconstruction granularity;

[0079] Layer-wise reconstruction: Weight reconstruction is performed at the layer level. For example, in a convolutional neural network (CNN), reconstruction is performed according to a combination of convolutional layers (CONV), batch normalization layers (BN), and ReLU activation layers;

[0080] Block-wise reconstruction: Weight reconstruction is performed at the block level (e.g., residual blocks). Blocks are core components of neural network architectures, such as residual bottleneck blocks.

[0081] In one embodiment of the present invention, step S12 includes the following sub-steps.

[0082] S121: Based on the sparse model in step S11, calculate the performance difference before and after reconstruction for all architectures and datasets on each task.

[0083] Specifically, the performance of reconstruction processes equipped with different refined reconstruction techniques is compared with that of cases without reconstruction, to obtain the performance difference before and after reconstruction.

[0084]

[0085] in, This represents the sparse accuracy achieved using a specific refined reconstruction technique for a given task. This indicates the sparse accuracy without employing the specific fine-grained reconstruction technique for a given task. yes A general term, These represent the performance differences before and after reconstruction in classification, detection, and generation tasks, respectively. Here, a specific reconstruction technique includes different techniques that vary the use of error, reconstruction granularity, and reconstruction input, with only one of these factors being changed for each computation.

[0086] S122: Based on the difference in index results before and after reconstruction Calculate the average value to obtain the overall reconstruction index OM. recon .

[0087] To standardize the scale, the FID score of the generation task is calculated as an index for the overall reconstruction index, since the scale is different from that of the other two tasks.

[0088] The overall reconstruction index can be calculated using the following formula:

[0089]

[0090] Similarly, the overall reconstruction index also uses the quadratic mean method.

[0091] In addition, to evaluate the sparsity capability of the model, this embodiment of the invention uses three evaluation metrics: "model architecture", "model size robustness" and "application task".

[0092] S13: Calculate the accuracy of the original model and the sparse model under each model architecture, and calculate the average relative accuracy of all architectures and datasets under each model architecture to obtain the overall model metric OM. arch .

[0093] Overall Model Indicator (OM) arch The calculation is as follows:

[0094]

[0095] in, This represents the accuracy of a specific model architecture at the i-th sparsity level. This represents the accuracy of the i-th sparsity ratio of the sparse model obtained after sparsification of this specific model; C represents the number of sparsity ratios. Simply put, OM... arch The metrics evaluate the average performance at different sparsity rates within the same model architecture.

[0096] S14: Calculate the accuracy of the sparse model based on the accuracy of the original model at size j, and calculate the average relative accuracy of all architectures and datasets at each size as a robustness metric.

[0097] In practical applications, the scale of deployed models varies due to different requirements and resources. The post-training sparse benchmark evaluation method provided by this invention demonstrates consistent effectiveness across models with the same architecture but different depths and widths. In other words, if a model exhibits superior sparsity performance only in versions with a specific number of parameters, but shows significant instability at other parameter scales, the post-training sparse benchmark evaluation method provided by this invention can assess this instability and thus not consider it a friendly sparse architecture design.

[0098] Therefore, the post-training sparse benchmark evaluation method provided by the present invention utilizes robustness metrics to test and quantify the robustness of model size.

[0099] In one embodiment of the present invention, step S14 includes the following sub-steps:

[0100] S141: Calculate the relative accuracy G of the quadratic mean of the model architecture of a specific size using the following formula. size:

[0101]

[0102] in, It represents the accuracy of a specific original model on a specific model architecture; This represents the accuracy of the sparse model obtained after sparsifying the specific original model at the j-th sparsity ratio under the same model architecture. C represents the number of sparsity ratios.

[0103] S142: Calculate the standard deviation between different sizes of the same model architecture as a robustness indicator OM. robust ,

[0104] OM robust =std(G size )

[0105] Here, std(·) represents the standard deviation operator.

[0106] It should be noted that, to avoid the reconstruction process being affected by different model sizes, G should be calculated without performing a reconstruction. size .

[0107] S15: Using the accuracy of the original model on task n as a benchmark, calculate the accuracy of the sparse model on task n, and calculate the average relative accuracy of all architectures and datasets across all models, as the overall task metric OM. task .

[0108] In various computer vision tasks, models are often combined and enhanced before deployment. For example, ResNet is used as a backbone in detection tasks, connected to neck and head modules. In image generation tasks, CNNs and Transformers are also used in combination. Therefore, it is necessary to test sparsity capabilities from a task-specific perspective.

[0109] Therefore, to evaluate the sparsity capabilities of various models on different tasks, this embodiment of the invention uses three commonly used typical tasks (classification, detection, and image generation) as examples for evaluation, and calculates the overall task index OM using the formula. task :

[0110]

[0111] in, This represents the accuracy of a specific model on the nth task without sparsity. This represents the accuracy of the nth task after sparse processing; N represents the number of tasks (in this embodiment of the invention, N = 3 is preferred).

[0112] It should be noted that if the evaluation purpose is only one task (e.g., only image generation task, no other tasks), such as in a specific application scenario where there is only one task, then in the aforementioned steps, only the evaluation values ​​of each indicator for that one task should be evaluated, and there is no need to evaluate the other two tasks.

[0113] In addition, the post-training sparse benchmark evaluation method provided by this invention includes three evaluation metrics: the accuracy of the model architecture, the robustness of the size, and the task accuracy for different tasks. However, depending on the application scenario, only one or two of these three evaluation metrics may be used, and other metrics may also be added.

[0114] It should be noted that, among the aforementioned five evaluation indicators, except for OM... robust Furthermore, all indicators are positive indicators. Moreover, the calculation steps for each evaluation indicator can be performed simultaneously or in any order. The above embodiments only use a specific order for ease of description, and this does not constitute a limitation of the present invention.

[0115] Furthermore, to standardize the overall metrics across different tracks, all metrics in the post-training sparse evaluation benchmark are calculated using a quadratic mean. This method mitigates the excessive influence of particularly poor performance on the metrics, thereby providing a more accurate assessment of the overall performance of each track.

[0116] Based on the above-described post-training sparse benchmark evaluation method, we can assess the sparsification performance of different PTS models and the impact of pluggable sparsification algorithms on performance. If the evaluation results fail to meet the requirements of practical applications, the post-training sparse model optimization method provided by this invention can be used to improve these PTS models or sparsification algorithms. It should be noted that even without using the aforementioned benchmark evaluation method, based solely on the actual training results, those skilled in the art can still understand the sparsification capabilities of different PTS models and the impact of their algorithms, and accordingly apply the post-training sparse model optimization method provided by this invention for adjustments. In other words, the above-described post-training sparse benchmark evaluation method is not a necessary step in applying the post-training sparse model optimization method provided by this invention.

[0117] First Embodiment

[0118] During model optimization, in order to achieve a transition from dense to sparse models through continued training to adapt to specific application requirements, such as... Figure 3 As shown, the first embodiment of the present invention provides an optimization method for a trained sparse model, which includes at least the following steps:

[0119] S1: Determine whether the model is used for a single task. If not, proceed to step S5. If it is used for a classification task, proceed to step S2. If it is used for a detection task, proceed to step S3. If it is used for a generation task, proceed to step S4.

[0120] Furthermore, this step can be omitted, that is, proceed directly to steps S2 to S4 (e.g. Figure 3 The dashed line indicates an omitted step.

[0121] S2: For models used in classification tasks, the sparsity rate must be 50% ≤ sparsity < 80%, and the sparsity allocation method should be selected in the order of priority: FCPTS > ERK > L2Norm. The reconstruction technique should employ error correction and block-by-block reconstruction techniques.

[0122] It should be noted that the sparse allocation method in this step is FCPTS. FCPTS includes FCPTS(ERK) and FCPTS(L2Norm), and FCPTS(ERK) is preferred over FCPTS(L2Norm).

[0123] If FCPTS cannot be selected or is not suitable for the actual application, the ERK method can be selected; the L2Norm method can be selected as another option.

[0124] That is, sparsity assignment methods can be selected one by one in the priority order of FCPTS(ERK) > FCPTS(L2Norm) > ERK > L2Norm, and the sparsity capability of the model under different sparsity assignment methods can be evaluated to finally select the appropriate sparsity assignment method. If it is not possible to evaluate the sparsity capability of the model under different sparsity assignment methods, then the sparsity assignment method can be directly selected in the aforementioned priority order.

[0125] Moreover, a more preferred approach is to assign a lower sparsity to the last layer, i.e., satisfying 50% ≤ sparsity < 80%, and the sparsity of the last layer ≤ 50%.

[0126] In this step, since the sparsity allocation method uses FCPTS (including FCPTS(ERK) and FCPTS(L2Norm)), ERK or L2Norm, in conjunction with the above, when using the FCPTS(ERK) and FCPTS(L2Norm) sparsity allocation methods, a lower sparsity rate (e.g. ≤50%) is allocated to the downsampling layer; when using the ERK or L2Norm sparsity allocation methods, no sparsification is performed on the downsampling layer.

[0127] A more preferred technical solution is that this step uses sparse input during reconstruction, using the output of the sparse model as the reconstruction input for the current unit. More specifically, the higher the sparsity ratio, the higher the accuracy of using sparse input (at the same sparsity ratio, the greater the difference in accuracy compared to dense input).

[0128] S3: For models used in detection tasks, the sparsity rate must be 50% ≤ sparsity < 80%, and the sparsity allocation method should be selected in the priority order of FCPTS > ERK > L2Norm. The reconstruction technique should employ error-free correction and block-by-block reconstruction.

[0129] S4: For the model used for the generation task, the sparsity should be ≤50%, and the sparsity allocation method should be selected in the priority order of FCPTS > L2Norm > ERK. The reconstruction technique should be either error-free and block-by-block reconstruction, or error-corrected and block-by-block reconstruction.

[0130] S5: Determine whether the task contains a generated task. If it does, proceed to step S6; otherwise, proceed to step S7.

[0131] S6: If the sparsity is ≤50% and the sparse allocation method is FCPTS, the reconstruction technique adopts error-free correction and block-by-block reconstruction.

[0132] Even better, in this step, the sparse allocation method follows the order FCPTS(ERK) > FCPTS(L2Norm) (e.g. Figure 4 The priority order selection is shown in the figure.

[0133] S7: If the sparsity is 50% ≤ sparsity < 80%, and the sparsity allocation method is selected in the priority order of FCPTS > ERK > L2Norm, the reconstruction technique adopts error-free correction and block-by-block reconstruction.

[0134] More specifically, the sparse allocation method is selected in the priority order of FCPTS(ERK)>FCPTS(L2Norm)>ERK>L2Norm.

[0135] As mentioned earlier, error correction (with correction) is preferred for classification tasks (CLS), while error-free correction (without correction) is preferred for inspection tasks (DET). However, considering that inspection tasks have higher accuracy requirements, an error-free correction method is adopted when both types of tasks are considered.

[0136] As an alternative to embodiments of the present invention, such as Figure 3 As shown by the dashed lines, steps S1 and S5-S7 can be omitted. Because for a single-task model, it's possible to proceed directly to steps S2-S4 without further consideration. That is, the embodiment of this invention includes the following steps:

[0137] S2: Determine whether it is a model for classification tasks. If not, proceed to step S3. If it is, it must satisfy 50% ≤ sparsity rate < 80%, and the sparsity allocation method must be selected in the priority order of FCPTS > ERK > L2Norm. The reconstruction technique adopts the technique of error correction and block-by-block reconstruction.

[0138] S3: Determine whether it is a model used for the detection task. If not, proceed to step S4. If it is, for the model used for the detection task, it satisfies 50% ≤ sparsity rate < 80%, and the sparsity allocation method is selected in the priority order of FCPTS > ERK > L2Norm. The reconstruction technique adopts error-free correction and block-by-block reconstruction techniques.

[0139] S4: For the model used for the generation task, the sparsity should be ≤50%, and the sparsity allocation method should be selected in the priority order of FCPTS>L2Norm>ERK. The reconstruction technique should be either error-free correction and block-by-block reconstruction, or error correction and block-by-block reconstruction.

[0140] Second Embodiment

[0141] like Figure 5 As shown, compared to the first embodiment, this embodiment of the invention adds a step of selecting the model architecture based on the model size.

[0142] Specifically, based on the first embodiment, the following steps are added after steps S2 to S4 or S6 to S7:

[0143] S8: Determine the model size. If the model size is large, select the Transformer-based model architecture; if the model size is medium, select the CNN-based model architecture.

[0144] If it is a lightweight model, then no changes are made.

[0145] It's important to note that while both ViT and DeiT are based on the Transformer architecture, they differ in handling data scale and training efficiency. ViT excels on large-scale datasets, while DeiT focuses on improving training efficiency on small datasets. Therefore, if the model size increases, the ViT model architecture can be chosen. On the other hand, among various CNN models, MobileNetV3 has the greatest sparsity potential and is the preferred model based on the CNN model architecture. Therefore, MobileNetV3 can be selected during the optimization of sparsity strategies based on the CNN model architecture.

[0146] S8: Determine the model size. If the model size is large, select the Transformer-based model architecture; if the model size is medium, select the CNN-based model architecture.

[0147] S9: Determine the task type. If it is a classification task, proceed to step S2; if it is a detection task, proceed to step S3; if it is a generation task, proceed to step S4.

[0148] S2: For models used in classification tasks, the sparsity rate must be 50% ≤ sparsity < 80%, and the sparsity allocation method should be selected in the priority order of FCPTS > ERK > L2Norm. The reconstruction technique should employ error correction and block-by-block reconstruction techniques.

[0149] S3: For models used in detection tasks, the sparsity rate must be 50% ≤ sparsity < 80%, and the sparsity allocation method must be selected in the priority order of FCPTS > ERK > L2Norm. The reconstruction technique adopts error-free correction and block-by-block reconstruction techniques.

[0150] S4: For the model used for the generation task, the sparsity should be ≤50%, and the sparsity allocation method should be selected in the priority order of FCPTS>L2Norm>ERK. The reconstruction technique should be either error-free correction and block-by-block reconstruction, or error correction and block-by-block reconstruction.

[0151] Third Embodiment

[0152] like Figure 6 As shown, compared to the first embodiment, the present invention adds a step S8 to select a suitable model architecture as the model size increases before step S1 or S5.

[0153] Compared to the second embodiment, the present invention only changes the order of S8, and designs the step of selecting the model architecture according to the model size before step S1 or S5.

[0154] Specifically, the optimization method for a post-trained sparse model provided in this embodiment of the invention includes the following steps:

[0155] S8: Determine the model size. If the model size is large, select the Transformer-based model architecture; if the model size is medium, select the CNN-based model architecture.

[0156] S9: Determine the task type. If it is a classification task, proceed to step S2; if it is a detection task, proceed to step S3; if it is a generation task, proceed to step S4.

[0157] S2: For models used in classification tasks, the sparsity rate must be 50% ≤ sparsity < 80%, and the sparsity allocation method should be selected in the priority order of FCPTS > ERK > L2Norm. The reconstruction technique should employ error correction and block-by-block reconstruction techniques.

[0158] S3: For models used in detection tasks, the sparsity rate must be 50% ≤ sparsity < 80%, and the sparsity allocation method should be selected in the priority order of FCPTS > ERK > L2Norm. The reconstruction technique should be error-free correction and block-by-block reconstruction.

[0159] S4: For the model used for the generation task, the sparsity should be ≤50%, and the sparsity allocation method should be selected in the priority order of FCPTS > L2Norm > ERK. The reconstruction technique should be either error-free and block-by-block reconstruction, or error-corrected and block-by-block reconstruction.

[0160] Fourth embodiment

[0161] Based on the above-described optimization method for post-trained sparse models, the fourth embodiment of this invention further provides an optimization system for post-trained sparse models. For example... Figure 7 As shown, the optimization system for the trained sparse model includes one or more processors and a memory. The memory is coupled to the processors and is used to store one or more computer programs. When these programs are executed by the processors, they enable the processors to implement the optimization method for the trained sparse model as described in the above embodiment.

[0162] The processor controls the overall operation of the optimization system for the trained sparse model, completing all or part of the steps of the optimization method described above. This processor module can be a central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), digital signal processing (DSP), etc. The memory stores various types of data to support the operation of the optimization system for the trained sparse model. This data may include, for example, instructions for any application or method used to operate the optimization system for the trained sparse model, as well as application-related data. This memory module can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, etc.

[0163] It should be noted that the above embodiments are merely illustrative examples. The technical solutions of the various embodiments can be combined, the order of the steps can be changed, or some steps can be omitted, and all such changes are within the protection scope of this invention.

[0164] The optimization method and system for the trained sparse model provided by this invention have been described in detail above. Any obvious modifications made by those skilled in the art without departing from the essence of this invention will constitute an infringement of the patent rights of this invention and will incur corresponding legal liability.

Claims

1. An optimization method for a sparse model after training, characterized in that... Includes the following steps: S1: Determine whether the model is used for a single task. If not, proceed to step S5. If it is used for a classification task, proceed to step S2. If it is used for a detection task, proceed to step S3. If it is used for a generation task, proceed to step S4. S2: For models used in classification tasks, the sparsity rate must be 50% ≤ sparsity < 80%, and the sparsity allocation method should be selected in the priority order of FCPTS > ERK > L2Norm. The reconstruction technique should be an error correction and block-by-block reconstruction technique. S3: For models used in detection tasks, the sparsity rate must be 50% ≤ sparsity < 80%, and the sparsity allocation method should be selected in the priority order of FCPTS > ERK > L2Norm. The reconstruction technique should adopt error-free correction and block-by-block reconstruction techniques. S4: For the model used for the generation task, the sparsity should be ≤50%, and the sparsity allocation method should be selected in the priority order of FCPTS>L2Norm>ERK. The reconstruction technique should be either error-free and block-by-block reconstruction, or error-corrected and block-by-block reconstruction. S5: Determine whether the task contains a generated task. If it does, proceed to step S6; otherwise, proceed to step S7. S6: If the sparsity is ≤50% and the sparse allocation method is FCPTS, the reconstruction technique adopts error-free correction and block-by-block reconstruction. S7: If the sparsity is 50%≤sparser rate<80%, and the sparsity allocation method is selected in the priority order of FCPTS>ERK>L2Norm, the reconstruction technique adopts error-free correction and block-by-block reconstruction. Among them, FCPTS is a fine-grained channel pruning method based on reinforcement learning, ERK is a sparse allocation method based on effective rank knowledge, and L2Norm is a sparse allocation method based on L2 norm.

2. An optimization method for a sparse model after training, characterized in that... Includes the following steps: S10: Determine whether it is a model for classification tasks. If not, proceed to step S20. If it is, it must satisfy 50%≤sparse rate<80%, and the sparsity allocation method must be selected in the priority order of FCPTS>ERK>L2Norm. The reconstruction technique adopts the technique of error correction and block-by-block reconstruction. S20: Determine whether it is a model used for the detection task. If not, proceed to step S30. If it is, for the model used for the detection task, it satisfies 50%≤sparse rate<80%, and the sparsity allocation method is selected in the priority order of FCPTS>ERK>L2Norm. The reconstruction technique adopts error-free correction and block-by-block reconstruction techniques. S30: For the model used for the generation task, the sparsity rate is ≤50%, and the sparsity allocation method is selected in the priority order of FCPTS>L2Norm>ERK. The reconstruction technique adopts the technique of no error correction and block-by-block reconstruction, or the technique of error correction and block-by-block reconstruction. Among them, FCPTS is a fine-grained channel pruning method based on reinforcement learning, ERK is a sparse allocation method based on effective rank knowledge, and L2Norm is a sparse allocation method based on L2 norm.

3. An optimization method for a sparse model after training, characterized in that... Includes the following steps: S01: Determine the model size. If the model size is large, select the Transformer-based model architecture. For medium-sized models, choose a CNN-based model architecture; S02: Determine the task type. If it is a classification task, proceed to step S03; if it is a detection task, proceed to step S04; if it is a generation task, proceed to step S05. S03: For models used in classification tasks, the sparsity rate must be 50% ≤ sparsity < 80%, and the sparsity allocation method should be selected in the priority order of FCPTS > ERK > L2Norm. The reconstruction technique should employ error correction and block-by-block reconstruction techniques. S04: For models used in detection tasks, the sparsity rate must be 50% ≤ sparsity < 80%, and the sparsity allocation method should be selected in the priority order of FCPTS > ERK > L2Norm. The reconstruction technique should adopt error-free correction and block-by-block reconstruction techniques. S05: For models used for generation tasks, the sparsity should be ≤50%, and the sparsity allocation method should be selected in the priority order of FCPTS > L2Norm > ERK. The reconstruction technique should be either error-free and block-by-block reconstruction, or error-corrected and block-by-block reconstruction. Among them, FCPTS is a fine-grained channel pruning method based on reinforcement learning, ERK is a sparse allocation method based on effective rank knowledge, and L2Norm is a sparse allocation method based on L2 norm.

4. The optimization method for a post-trained sparse model as described in any one of claims 1 to 3, characterized in that: When the sparse allocation method selects FCPTS, FCPTS includes FCPTS(ERK) and FCPTS(L2Norm), and FCPTS(ERK) takes precedence over FCPTS(L2Norm). Among them, FCPTS is a fine-grained channel pruning method based on reinforcement learning, FCPTS(ERK) is an FCPTS method based on the ERK index, and FCPTS(L2Norm) is an FCPTS method based on the L2Norm index.

5. The optimization method for a post-trained sparse model as described in any one of claims 1 to 3, characterized in that: In the sparse allocation method, the sparsity of the last layer is ≤50%.

6. The optimization method for a post-trained sparse model as described in any one of claims 1 to 3, characterized in that: In the sparse allocation method, the sparsity of the downsampling layer is ≤50%.

7. The optimization method for the trained sparse model as described in claim 3, characterized in that: The Transformer-based model architecture is the ViT model architecture.

8. The optimization method for the trained sparse model as described in claim 3, characterized in that: The CNN-based model architecture is the MobileNetV3 model architecture.

9. An optimization system for a trained sparse model, comprising a processor and a memory, wherein the processor and the memory are coupled; wherein, The memory is used to store computer programs; the processor is used to run the computer programs stored in the memory and execute the optimization method for the trained sparse model according to any one of claims 1 to 8.

Images