A method and apparatus for training a post-sparse benchmark evaluation

Abstract

The application discloses a kind of training after sparse benchmark evaluation method and device.The method constructs a precise evaluation system by five indexes: sparse distribution mode, reconstruction technology, model architecture accuracy, size robustness and different task accuracy. After the original model is sparsified using the standardized sparse paradigm, the sparsification performance of the model is evaluated using these indexes. This method can compare different post-training sparse models horizontally, comprehensively evaluate the performance of the model from both the algorithm and the model, and distinguish the impact of sparsification and reconstruction on performance, as well as the impact of architecture, size or task on sparsification performance. Using the present application, not only the model compression efficiency is improved, but also an important reference is provided for model selection and optimization.

Description

Technical Field

[0001] This invention relates to a method for evaluating sparse benchmarks after training, and also to a corresponding device for evaluating sparse benchmarks after training, belonging to the field of machine learning technology. Background Technology

[0002] Deep learning technology has been widely applied in many fields, but its huge demand for memory and computing power has become a challenge. To address this challenge, researchers have developed various model compression strategies, including model quantization, model sparsification, network distillation, lightweight network design, and low-rank decomposition. Among these strategies, post-trained sparsity (PTS) has attracted attention due to its lower training cost.

[0003] Currently, several representative methods have emerged in the field of post-training sparse optimization, such as Post-training Optimization Tool (POT), AdaPrune, and Optimal Brain Surgeon (OBC). These cutting-edge techniques can maintain model performance after sparsification. Nevertheless, PTS methods still face two main problems:

[0004] First, the fine-grained exploration of post-trained sparse algorithms is still insufficient. Current PTS methods generally follow a fixed process: first, a sparsity rate is assigned to each layer of the model for sparsification, and then performance is restored by reconstructing activations. Although research follows this process, fine-grained exploration of PTS techniques remains inadequate. For example, most PTS methods use a layer-by-layer reconstruction approach during the reconstruction process, while block-by-block reconstruction, which has proven effective in quantization methods, has not yet been explored in PTS. The lack of in-depth analysis of fine-grained techniques limits the further development of PTS methods.

[0005] Secondly, the relationship between models and sparsity has not been fully explored. While current PTS research has evaluated specific models and tasks, it has not discussed the sparsification potential of different models. It remains unclear which types of models are more suitable for sparsification for different architectures, sizes, and tasks. Researchers often choose commonly used models (such as ResNet and RegNetX) or datasets to validate their methods, but a comprehensive evaluation of the relationship between models and sparsity is lacking. In practical applications, it is often necessary to sparsify multiple network architectures of similar sizes to achieve efficient inference, but it is currently unclear which network architecture is more sparsity-friendly. Furthermore, with changes in deployment platforms, different sized models may need to be used, and the robustness of these models in sparse algorithms has not been studied. Simultaneously, current research has not considered the generalization ability of sparse models to different tasks; model structures for different tasks need to be designed and modified for each task, and the impact of these task-oriented designs and modifications on sparsity is unknown. Therefore, a comprehensive evaluation of model architectures from both theoretical and practical perspectives is essential. Summary of the Invention

[0006] The primary technical problem to be solved by this invention is to provide a method for evaluating sparse benchmarks after training.

[0007] Another technical problem to be solved by the present invention is to provide a sparse benchmark evaluation device after training.

[0008] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution:

[0009] According to a first aspect of the present invention, a method for evaluating sparse benchmarks after training is provided, comprising the following steps:

[0010] S11: Calculate the indicator A for a specific task based on the original model. * Then, sparse models are obtained using different sparse allocation methods, and the performance metrics of the sparse models for specific tasks are calculated. Finally based on A * Calculate the relative values ​​of the metrics for all architectures and datasets on each task, and use their mean as the overall sparse distribution metric OM. alloc ;

[0011] S12: Based on the sparse model in step S11, calculate the difference in the index results before and after reconstruction for all architectures and datasets on each task, and then calculate the average value to obtain the overall reconstruction index.

[0012] S13: Calculate the accuracy of the original model and the sparse model under each model architecture, and calculate the average relative accuracy of all architectures and datasets under each model architecture to obtain the overall model metric OM. arch ;

[0013] S14: Using the accuracy of the original model at size j as a baseline, calculate the accuracy of the sparse model, and calculate the average relative accuracy of all architectures and datasets at each size as the robustness metric OM. robust ;

[0014] S15: Using the accuracy of the original model on task n as a benchmark, calculate the accuracy of the sparse model on task n, and calculate the average relative accuracy of all architectures and datasets across all models, as the overall task metric OM. task .

[0015] Preferably, step S11 includes the following sub-steps:

[0016] S111: Calculate the performance index A of the original model on one or more tasks. * ;

[0017] S112: Based on the selected sparse allocation method, the original model is sparsified to obtain a sparse model;

[0018] S113: Calculate the sparsity allocation index of the sparse model in one or more tasks.

[0019] S114: Calculate the quadratic mean of the relative values ​​of the indicators for each task as the overall sparse distribution indicator OM. alloc ;

[0020]

[0021] Among them, A * This represents the index result calculated by the original model under a specific task; This represents the index results calculated by various sparse models under different sparsity rates for a specific task. include A * Including A CLS A DET A GEN ; It is the mean operator.

[0022] Preferably, step S12 includes the following sub-steps.

[0023] S121: Based on the sparse model in step S11, calculate the performance difference before and after reconstruction for all architectures and datasets on each task.

[0024]

[0025] in, This represents the sparse accuracy achieved using a specific reconstruction technique for a specific task. This indicates the sparse accuracy without using the specific reconstruction technique for a particular task. yes A general term, These represent the performance differences before and after reconstruction in classification, detection, and generation tasks, respectively.

[0026] S122: Based on the difference in index results before and after reconstruction Calculate the average value to obtain the overall reconstruction index OM. recon ;

[0027]

[0028] Preferably, the specific reconstruction technique includes different reconstruction techniques that vary the use of error, reconstruction granularity, and reconstruction input, and only one of these is changed each time a calculation is performed.

[0029] Preferably, the overall model index OM arch The calculation is as follows:

[0030]

[0031] in, This represents the accuracy of a specific model architecture at the i-th sparsity level. C represents the accuracy of the i-th sparsity ratio of the sparse model obtained after sparsification of the specific model; C represents the number of sparsity ratios.

[0032] Preferably, step S14 includes the following sub-steps:

[0033] S141: Calculate the relative accuracy G of the quadratic mean of the model architecture of a specific size using the following formula. size ,

[0034]

[0035] in, It represents the accuracy of a specific original model on a specific model architecture; This represents the accuracy of the sparse model obtained after sparsifying the specific original model at the j-th sparsity ratio under the same model architecture; C represents the number of sparsity ratios.

[0036] S142: Calculate the standard deviation between different sizes of the same model architecture as a robustness indicator OM. robust ,

[0037] OM robust =std(G size )

[0038] Here, std(·) represents the standard deviation operator.

[0039] Preferably, the task includes any one or more of classification tasks, detection tasks, and generation tasks.

[0040] According to a second aspect of the present invention, a post-training sparse benchmark evaluation apparatus is provided, comprising a processor and a memory, wherein the processor and the memory are coupled together; wherein the memory is used to store a computer program; and the processor is used to run the computer program stored in the memory to perform the above-described post-training sparse benchmark evaluation method.

[0041] Compared with existing technologies, this invention constructs a precise evaluation system comprising five indicators, enabling a comprehensive assessment of the sparsification capabilities and technical performance of different Post-Trained Sparse (PTS) models. These five indicators include sparsity allocation methods, reconstruction techniques, model architecture accuracy, size robustness, and task accuracy for different tasks (classification, detection, and image generation). The evaluation process first employs a standardized sparsity paradigm to sparsify the original model, generating the model under test. Then, the sparsification capability of the tested model is evaluated using the aforementioned indicators. Using this invention, firstly, it allows for a precise horizontal comparison of the sparsification capabilities of different PTS models and pluggable sparsification algorithms; secondly, it enables a comprehensive evaluation of different PTS models from both algorithmic and model perspectives, including refined post-trained sparsification techniques and sparsified models; thirdly, it distinguishes the impact of sparsification and reconstruction on model performance; and finally, it identifies the influence of architecture, size, or task on model sparsification performance. Attached Figure Description

[0042] Figure 1A This is a schematic radar diagram of the evaluation indicators in the first embodiment of the present invention;

[0043] Figure 1B Figure 1 shows a schematic diagram illustrating the relationship between the evaluation metrics and the corresponding tracks.

[0044] Figure 2 A schematic diagram of the sparse paradigm used in PTS technology;

[0045] Figure 3 This is a sparse rate distribution diagram of each layer in the second embodiment of the present invention, where different sparse allocation methods are used for each model.

[0046] Figure 4 A graph showing the average relative accuracy loss for different model sizes;

[0047] Figure 5 This is a schematic diagram of the post-training sparse benchmark evaluation device provided in the third embodiment of the present invention. Detailed Implementation

[0048] The technical content of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0049] This invention proposes a post-training sparse benchmark evaluation method, aiming to construct a comprehensive evaluation system for better model selection and optimization. For example... Figure 1A and Figure 1B As shown, the evaluation system includes five metrics: sparsity allocation method, reconstruction technique, model architecture accuracy, size robustness, and task accuracy for different tasks (including classification, detection, and image generation). To optimize the evaluation results, this method first employs a standardized sparsity paradigm, performing sparsification on the original model based on these five evaluation metrics to generate the model under test. Subsequently, the same five metrics are used to evaluate the sparsification capability of the model under test. It should be noted that the standardized sparsity paradigm is not limited to the method in this embodiment; other sparsity paradigms can also be used, as long as the original models are sparsified under the same evaluation metrics for comparison of sparsification capabilities.

[0050] First Embodiment

[0051] This invention addresses the needs of practical model production by designing a post-training sparse benchmark evaluation system that includes five tracks and their corresponding metrics, such as... Figure 1A and Figure 1B As shown, this system provides a comprehensive evaluation framework through five metrics—sparse allocation method, reconstruction technique, model architecture accuracy, size robustness, and task accuracy for different tasks (classification, detection, and image generation)—aiming to overcome the limitations of current research on post-trained sparsity (PTS) and to effectively evaluate various PTS models and their performance. In one embodiment of this invention, we first sparsify the original model using a standardized sparsity paradigm to generate the model under test. This process involves sparsifying the five evaluation metrics of the original model. Then, these five evaluation metrics are used to evaluate the sparsified model under test. Therefore, the following content will first introduce the standardized sparsity paradigm and then explain how to apply the post-trained sparsity benchmark evaluation system composed of these five evaluation metrics to evaluate the model under test.

[0052] In one embodiment of the present invention, the original model is sparsified using a standardized sparse paradigm to obtain the model to be tested, including the following steps:

[0053] S11: Calculate the indicator A for a specific task based on the original model. * Then, sparse models are obtained using different sparse allocation methods, and the performance metrics of the sparse models for specific tasks are calculated. Finally based on A * Calculate the relative values ​​of the metrics for all architectures and datasets on each task, and use their mean as the overall sparse distribution metric OM. alloc .

[0054] As is well known, PTS (Parasitic Sparse Tree) techniques aim to sparsify pre-trained neural networks while maintaining their accuracy on specific tasks (classification, detection, and image generation, or any one or more). Currently, existing PTS techniques employ a two-step sparsity paradigm involving sparse allocation and reconstruction, such as... Figure 1B As shown. During the sparse allocation process, a specific sparsity rate is assigned to each layer according to a predetermined index, and the weights at the corresponding positions are reset to zero according to the sparsity criteria. The reconstruction process is used to restore the model accuracy that has decreased due to sparsification.

[0055] Currently, sparse allocation methods can be divided into the following three categories:

[0056] Heuristic strategy: In this strategy, the sparsity of each layer is manually predetermined, such as uniform sparsity;

[0057] Metric-based strategy: Sort the weights of all layers according to a specific metric and set the weights with lower scores to zero according to a certain proportion;

[0058] Learning-based strategy: Learn the sparsity rate of each layer by optimizing the loss function to achieve optimal sparsity distribution.

[0059] In one embodiment of the present invention, in order to evaluate the performance of different sparse allocation methods and to study in depth the characteristics and essence of an ideal sparse allocation, any one of the following four sparse allocation methods can be selected:

[0060] Uniform sparse allocation: This is a widely adopted heuristic strategy for sparse allocation that assigns the same sparsity to each layer.

[0061] L2 Norm (L2 Normalization): This is a metric-based sparse allocation method that determines the sparsity rate by calculating the L2 norm of each layer;

[0062] ERK (Effective Rank-based Knowledge): This is also a sparse allocation method based on metrics, but it is more refined than L2Norm. Both L2Norm and ERK methods have high efficiency and high citation rates.

[0063] FCPTS (Fine-Grained Channel Pruning via Reinforcement Learning): This is a typical learning-based method and the only learning-based sparse allocation method.

[0064] It should be noted that although this embodiment selects four sparse allocation methods to sparsify the original model, and categorizes these methods into three types—heuristic, index-based, and learning-based—ERK and L2Norm are two representative index-based methods. However, those skilled in the art can also use other sparse allocation methods.

[0065] Using the accuracy of the original model as a benchmark, calculate the average relative accuracy of all architectures and datasets on each task, including the following sub-steps.

[0066] S111: Calculate the performance index A of the original model on one or more tasks. * .

[0067] In one embodiment of the present invention, the accuracy A of the original model on the classification (CLS) task is calculated respectively. CLS The mAP (mean Average Precision) value A for the detection (DET) task. DET Or the FID (Frechet Inception Distance) value A of the image generation (GEN) task. GEN Here, we use accuracy A. * Representing A CLS A DET and A GEN and collectively.

[0068] Accuracy in classification tasks, mAP in detection tasks, and FID in image generation tasks are commonly used metrics for evaluating models, which will not be elaborated here.

[0069] S112: Based on the selected sparse allocation method, the original model is sparsified to obtain a sparse model.

[0070] As mentioned above, the sparsity allocation methods in this embodiment include four types (uniform sparsity allocation, L2Norm, ERK, and FCPTS). Therefore, the original model is sparsified using any one of these four methods to obtain a sparse model. It is understood that the sparsity allocation methods are not limited to these four; other sparsity allocation methods, such as gradient-based sparsity allocation or density estimation-based sparsity allocation (KDE), can also be used, as long as the same sparsity allocation method is applied to each original model to achieve evaluation.

[0071] S113: Calculate the sparsity allocation index of the sparse model in one or more tasks.

[0072] In one embodiment of the present invention, the accuracy of the sparse model on the classification task is calculated respectively. mAP value of the detection task or the FID value of the image generation task Here, sparse allocation index is used. represent and and collective term,

[0073] S114: Calculate the quadratic mean of the relative values ​​of the indicators for each task as the overall sparse distribution indicator OM. alloc .

[0074]

[0075] Among them, A * This represents the metrics calculated by the original model for a specific task (i.e., classification, detection, and image generation). This represents the index results calculated by various sparse models under different sparsity rates for a specific task. This is the mean operator. It's important to note that FID is an indicator where lower values ​​are better, and its range is [1, +∞]. To ensure the overall sparse distribution, the OM indicator... alloc The value is in the interval [0, 1], so we take its reciprocal.

[0076] Through the above steps, the selected original model is sparsified under different sparse allocation algorithms, and the OM corresponding to each sparse allocation algorithm is calculated. alloc As can be seen, this step measures the accuracy of the original model after sparsification without reconstruction. This allows for direct observation of the impact of different sparsity allocation strategies, distinguishing the accuracy effects of different sparsity allocation strategies and different reconstruction methods. This distinction is more conducive to the performance analysis and optimization of the PTS method.

[0077] S12: Based on the sparse model in step S11, calculate the difference in the index results before and after reconstruction for all architectures and datasets on each task, and then calculate the average value to obtain the overall reconstruction index.

[0078] After sparsity ratio allocation, the PTS method applies reconstruction to reconstruct the sparse activations to compensate for the accuracy loss caused by sparsity. Refined reconstruction techniques that can significantly affect the effectiveness of sparsity during the sparsification process include: error correction, reconstruction input, and reconstruction granularity, as detailed below:

[0079] Error correction is widely used in many post-training quantization (PTQ) methods. Its purpose is to align the compressed weight distribution with the original weight distribution. Specifically, the error correction process can be written as follows:

[0080]

[0081] After the error correction operation and It is the corrected weights and biases, W s This represents the weights of the sparse model before correction. d W d and X d These represent the biases, weights, and input activations in the original model. f(W, X) indicates that the network layer performs a convolution operation on the input X with weights W. E and σ are the mean and standard deviation operators, respectively, and ∈ is a small constant. In this way, errors caused by the distributional shift of the weights and biases can be corrected.

[0082] During the reconstruction process, you can choose to use the output of the previous reconstruction unit in the dense model as input (dense input) or the output of the previous sparse unit as input (sparse input). The reconstruction input has a significant impact on the results.

[0083] The reconstruction process can be performed at different granularities. Many post-training quantization methods have demonstrated that the reconstruction granularity has a significant impact on quantization performance.

[0084] In one embodiment of the present invention, three reconstruction granularities were tested:

[0085] Single reconstruction: Weight reconstruction is performed based on each individual layer, which is the smallest reconstruction granularity;

[0086] Layer-wise reconstruction: Weight reconstruction is performed at the layer level. For example, in a convolutional neural network (CNN), reconstruction is performed according to a combination of convolutional layers (CONV), batch normalization layers (BN), and ReLU activation layers;

[0087] Block-wise reconstruction: Weight reconstruction is performed at the block level (e.g., residual blocks). Blocks are core components of neural network architectures, such as residual bottleneck blocks.

[0088] In one embodiment of the present invention, step S12 includes the following sub-steps.

[0089] S121: Based on the sparse model in step S11, calculate the performance difference before and after reconstruction for all architectures and datasets on each task.

[0090] Specifically, the performance of reconstruction processes equipped with different refined reconstruction techniques is compared with that of cases without reconstruction, to obtain the performance difference before and after reconstruction.

[0091]

[0092] in, This represents the sparse accuracy achieved using a specific refined reconstruction technique for a given task. This indicates the sparse accuracy without employing the specific fine-grained reconstruction technique for a given task. yes A general term, These represent the performance differences before and after reconstruction in classification, detection, and generation tasks, respectively. Here, a specific reconstruction technique includes different techniques that vary the use of error, reconstruction granularity, and reconstruction input, with only one of these factors being changed for each computation.

[0093] S122: Based on the difference in index results before and after reconstruction Calculate the average value to obtain the overall reconstruction index OM. recon .

[0094] To standardize the scale, the FID score of the generation task is calculated as an index for the overall reconstruction index, since the scale is different from that of the other two tasks.

[0095] The overall reconstruction index can be calculated using the following formula:

[0096]

[0097] Similarly, the overall reconstruction index also uses the quadratic mean method.

[0098] In addition, to evaluate the sparsity capability of the model, this embodiment uses three evaluation metrics: "model architecture", "model size robustness" and "application task".

[0099] S13: Calculate the accuracy of the original model and the sparse model under each model architecture, and calculate the average relative accuracy of all architectures and datasets under each model architecture to obtain the overall model metric OM. arch .

[0100] Overall Model Indicator (OM) arch The calculation is as follows:

[0101]

[0102] in, This represents the accuracy of a specific model architecture at the i-th sparsity level. This represents the accuracy of the i-th sparsity ratio of the sparse model obtained after sparsification of this specific model; C represents the number of sparsity ratios. Simply put, OM... arch The metrics evaluate the average performance at different sparsity rates within the same model architecture.

[0103] S14: Calculate the accuracy of the sparse model based on the accuracy of the original model at size j, and calculate the average relative accuracy of all architectures and datasets at each size as a robustness metric.

[0104] In practical applications, the scale of deployed models varies due to different requirements and resources. The post-training sparse benchmark evaluation method provided in this embodiment of the invention demonstrates consistent effectiveness across models with the same architecture but different depths and widths. In other words, if a model exhibits superior sparsity performance only in versions with a specific number of parameters, but shows significant instability at other parameter scales, the post-training sparse benchmark evaluation method provided in this embodiment of the invention can evaluate this instability and thus will not consider it a friendly sparse architecture design.

[0105] Therefore, the post-training sparse benchmark evaluation method provided in this embodiment of the invention utilizes robustness metrics to test and quantify the robustness of model size.

[0106] In one embodiment of the present invention, step S14 includes the following sub-steps:

[0107] S141: Calculate the relative accuracy G of the quadratic mean of the model architecture of a specific size using the following formula. size :

[0108]

[0109] in, It represents the accuracy of a specific original model on a specific model architecture; This represents the accuracy of the sparse model obtained after sparsifying the specific original model at the j-th sparsity ratio under the same model architecture. C represents the number of sparsity ratios.

[0110] S142: Calculate the standard deviation between different sizes of the same model architecture as a robustness indicator OM. robust ,

[0111] OM robust =std(G size )

[0112] Here, std(·) represents the standard deviation operator.

[0113] It should be noted that, to avoid the reconstruction process being affected by different model sizes, G should be calculated without performing a reconstruction. size .

[0114] S15: Using the accuracy of the original model on task n as a benchmark, calculate the accuracy of the sparse model on task n, and calculate the average relative accuracy of all architectures and datasets across all models, as the overall task metric OM. task .

[0115] In various computer vision tasks, models are often combined and enhanced before deployment. For example, ResNet is used as a backbone in detection tasks, connected to neck and head modules. In image generation tasks, CNNs and Transformers are also used in combination. Therefore, it is necessary to test sparsity capabilities from a task-specific perspective.

[0116] Therefore, to evaluate the sparsity capabilities of various models on different tasks, this embodiment of the invention uses three commonly used typical tasks (classification, detection, and image generation) as examples for evaluation, and calculates the overall task index OM using the formula. task :

[0117]

[0118] in, This represents the accuracy of a specific model on the nth task without sparsity. This represents the accuracy of the nth task after sparse processing; N represents the number of tasks (in this embodiment, N = 3 is preferred).

[0119] It should be noted that if the evaluation purpose is only one task (e.g., only image generation task, no other tasks), such as in a specific application scenario where there is only one task, then in the aforementioned steps, only the evaluation values ​​of each indicator for that one task should be evaluated, and there is no need to evaluate the other two tasks.

[0120] In addition, the post-training sparse benchmark evaluation method provided in this embodiment of the invention includes three evaluation metrics: the accuracy of the model architecture, the robustness of the size, and the task accuracy for different tasks. However, depending on the application scenario, only one or two of these three evaluation metrics may be used, and other metrics may be added.

[0121] It should be noted that, among the aforementioned five evaluation indicators, except for OM... robust Furthermore, all indicators are positive indicators. Moreover, the calculation steps for each evaluation indicator can be performed simultaneously or in any order. The above embodiments only use a specific order for ease of description, and this does not constitute a limitation of the present invention.

[0122] Furthermore, in various embodiments of the present invention, in order to unify the overall indicators of different tracks, each indicator in the sparse evaluation benchmark after training adopts a quadratic mean form. This method can reduce the excessive influence of particularly poor performance results on the indicators, thereby more accurately evaluating the overall performance of each track.

[0123] The post-training sparse benchmark evaluation method provided in this invention can achieve the following technical effects: 1) It constructs an accurate evaluation system for horizontal comparison of the sparsification capabilities and pluggable sparsification algorithms of different PTS models; 2) It comprehensively evaluates different PTS models from both algorithm and model perspectives, and can evaluate the fine-grained sparsification techniques and sparsification models after training; 3) It can distinguish the impact of sparsification and reconstruction on sparsification performance; 4) It can distinguish the impact of architecture, size, or task on sparsification performance.

[0124] Second Embodiment

[0125] The second embodiment of the present invention provides a specific application example of a post-training sparse benchmark evaluation method.

[0126] To verify the post-training sparse benchmark evaluation method provided in this embodiment of the invention, the known post-training sparse model architectures (as shown in Table 1) were tested.

[0127] Table 1: Model Architecture Used in This Embodiment

[0128]

[0129] The post-training sparse benchmark evaluation method provided in this embodiment of the invention is implemented using the PyTorch framework, according to... Figure 2 The process described involves sparsifying the original model: training the original model with the dataset, then performing sparsification and reconstruction to obtain the final sparse model. During reconstruction, this embodiment uses the SGD (Stochastic Gradient Descent) optimizer. The momentum is set to 0.9, and the learning rate is set to 1e-4. 1024 images from the training dataset are randomly selected as the calibration dataset, and calibration is performed for 20,000 epochs. The batch size is set to 64. During implementation, it was observed that when the sparsity ratio is below 50%, the performance degradation of almost all models after sparsification is negligible. On the other hand, when the sparsity ratio is above 80%, the accuracy of almost all model settings collapses. Therefore, this embodiment mainly presents the results for sparsity ratios of {0.5, 0.6, 0.7, 0.8}.

[0130] In this embodiment, during the evaluation of the main steps of the two PTS processes, "sparse rate allocation" and "reconstruction", for the classification task, this embodiment tested ResNet-18 / 50, RegNetX-200M / 400M, MobileNetV2 and the Transformer-based ViT model on the ImageNet-1K dataset; and tested ResNet32 / 56 and VGG-19 on the CIFAR-10 / 100 dataset.

[0131] For the detection task, this embodiment tested RetinaNet-r18 / 50 on the MSCOCO-2017 dataset and MobileNetV1-SSD and MobileNetV2-SSDLite on the PASCAL VOC07 dataset.

[0132] For the image generation task, this embodiment evaluates stable diffusion on the LSUN-Churches / Bedroom dataset.

[0133] ①Sparseness distribution

[0134] Four sparse allocation methods were selected: Uniform, L2Norm, ERK, and FCPTS. Furthermore, combinations of FCPTS and L2Norm (denoted as FCPTS(L2Norm)) and combinations of FCPTS and ERK (denoted as FCPTS(ERK)) were also selected.

[0135] The overall sparse distribution index OM is calculated according to the method in the first embodiment. alloc The results are shown in Table 2 below, and are expressed as follows: Figure 1A The radar chart in the upper left corner is displayed visually.

[0136] Table 2: OM (Obligations and Sparsity Assignments) under Different Tasks with Different Sparse Allocation Methods alloc value

[0137]

[0138] The evaluation results using different sparse allocation strategies are shown in Table 2. For more detailed analysis, Table 2 also provides the OM (Objectives and Metrics) for each task. alloc The root mean square component (denoted as MS) and its specific performance at each sparsity rate.

[0139] As shown in Table 2, the sparse allocation method has a crucial and significant impact on sparsity. Across various metrics, the Uniform strategy consistently performs the worst, while learning-based methods generally perform well. The difference between the two can reach 20%. ERK and L2Norm perform similarly, but the FCPTS initialized with the two strategies differ significantly. This indicates that initialization is critical for learning-based methods, and ERK has better fine-tuning potential than L2Norm (i.e., ERK sparse allocation is closer to the optimal distribution).

[0140] The results also showed that effective sparsity allocation benefited from assigning lower sparsity to more sensitive layers. To better determine the sparsity of each layer, in Figure 3 The image visualizes the sparsity allocation of different sparse allocation methods on the CIFAR-100 dataset. It can be observed that effective methods consistently assign a low sparsity rate to the last layer. This is because the last layer is directly related to the network's output features, making the output highly sensitive to changes in the weights of the last layer. Therefore, the last layer is not suitable for large-scale sparsification.

[0141] Furthermore, it can be observed that in poorly performing methods, the weights of certain layers are preserved as much as possible, while in more effective methods, the weights of these layers are significantly pruned, particularly in ResNet-32 and ResNet-56. Research has found that these layers are often downsampling layers, which are considered unsuitable for sparsity reduction in ERK and L2Norm. However, in FCPTS, downsampling layers have proven robust to sparsity reduction. Therefore, assigning lower sparsity rates to more sensitive layers is beneficial.

[0142] ②Reconstruction

[0143] The refined reconstruction techniques used in this embodiment include: error correction, reconstruction input, and reconstruction granularity, specifically including error correction (w / correction), error-free correction (w / ocorrection), dense input, sparse input, single reconstruction, layer-wise reconstruction, and block-wise reconstruction. The evaluation results are shown in Table 3, and... Figure 1A The radar chart in the upper right corner is displayed visually.

[0144] Table 3 Performance of different fine-grained reconstruction techniques with different sparsity rates under different tasks.

[0145]

[0146] Practical evaluations show that error correction performs differently across various tasks. For classification tasks, error correction consistently demonstrates better performance across different sparsity rates. However, the effects of error correction differ between detection and generation tasks. In detection tasks, applying error correction results in a significant performance drop, while generation tasks appear insensitive to distortions in weight distribution. In object detection tasks, maintaining the integrity of spatial information is crucial, as detection involves not only identifying "what" (object category) but also "where" (object location). Sparsity reduction and subsequent weight distribution adjustments can disrupt the spatial features learned by the model, which is particularly critical for detection tasks. This can weaken the model's ability to accurately locate objects, thus impacting its overall detection performance.

[0147] It can be observed that using the output of the sparse model as the reconstruction input is beneficial. As shown in Table 3, in most settings, using sparse input yields a higher performance improvement than the original input, especially at higher sparsity rates (e.g., 33.86% vs. 23.40% at 80% sparsity for classification tasks). This is likely because using the output of the sparse model allows the PTS algorithm to recognize the reconstruction error of the previous layer, thus avoiding error accumulation throughout the network. Therefore, using the output of the sparse model as the reconstruction input for the current unit is advantageous.

[0148] Furthermore, the inventors observed that block-by-block reconstruction is consistently optimal. Block-by-block reconstruction achieves best results in most configurations, while layer-by-layer reconstruction outperforms standalone reconstruction. For example, when sparsifying a classification model to 80% sparsity, block-by-block reconstruction can outperform layer-by-layer reconstruction by 3% and standalone reconstruction by 16%. At high sparsity, fine-grained reconstruction design is particularly important for classification tasks. For instance, the differences are not significant at 50% sparsity, but become significant at 80% sparsity. However, for detection tasks, this gap is already apparent at low sparsity.

[0149] The main advantage of block-by-block reconstruction is that it may better preserve the interactions between weights. If the sparsity of the model is more concentrated at the block level, then block-based reconstruction may be more effective in recovering this sparsity. Furthermore, block-by-block reconstruction can reduce noise during the reconstruction process and provide more stable results. On the other hand, layer-by-layer reconstruction may lead to the loss of some key connections between weights across layers, which may limit the model's expressive power and make it difficult to recover sparsity.

[0150] Block-by-block reconstruction also has advantages in terms of efficiency and resource consumption. Given the large scale of the feature maps, calculating the loss can require significant time and memory resources. Block-based reconstruction can reduce the number of loss calculations.

[0151] (3) Model Architecture

[0152] Table 4 shows the evaluation results and metrics OM under different sparsity rates for different model architectures. arch and OM robust For ease of understanding, in Figure 1A The radar chart is schematically provided in the lower left corner.

[0153] Table 4

[0154]

[0155] Table 4 shows that different architectures exhibit varying sparsity capabilities, with attention-based models possessing greater sparsity potential. Specifically, Table 4 reveals that ViT, MobileNetV3, and DeiT rank among the top in overall metrics (OM) across all evaluated models (1st, 2nd, and 4th, respectively). This contrasts sharply with models possessing the worst OM. atch MobileNetV2 and MobileNetV3 significantly improved the sparsity potential of these two similar architectures. The main difference between these two architectures lies in the fact that MobileNetV3 introduces a lightweight attention mechanism module. The attention mechanism allows the network to focus more on features that are crucial to the final task and reduce its reliance on unimportant features. In the case of sparsity, this helps to more effectively preserve information that is critical to performance.

[0156] Furthermore, it was observed that training strategies can affect sparsity potential. DeiT and ViT have almost identical model architectures, but DeiT's performance is significantly worse. This is due to significant differences in their training strategies. ViT is pre-trained on extremely large datasets (such as JFT-300M), enabling it to learn more general feature representations. This makes it more robust during sparsification. Even with a reduction in the number of parameters, these learned feature representations may still be effective, helping to maintain performance. DeiT, on the other hand, employs knowledge distillation as one of its core strategies to help train efficient models with less data. However, if the sparsification method interferes with the features learned through knowledge distillation during the sparsification process, this can adversely affect DeiT's performance.

[0157] ④ Robustness of model size

[0158] Dimensional robustness (OM) of each model in actual evaluation robust The results are shown in Table 4, and as... Figure 1A The radar image is shown in the lower center.

[0159] It can be observed that the high sparsity potential of a model of a certain size ≠ the high sparsity potential of models of all sizes. For example, in OM... arch DeiT and ResNet models performed well in OMrobust The model performed poorly on the above metrics, while the ViT and MobileNetV3 models performed consistently on both metrics. This indicates that there is no strict positive correlation between a model's sparsity potential and its size robustness. However, models with high sparsity potential are more likely to have good size robustness.

[0160] Furthermore, a larger model size does not necessarily mean better sparsity. Figure 4 The average relative accuracy loss (G) is shown for different model sizes. size The results will be used to provide a more detailed analysis. Figure 4 The following conclusions can be drawn from this:

[0161] For CNNs, the average relative accuracy loss first increases and then decreases as the model size increases. Therefore, medium-sized networks are the optimal choice for CNN-based network architectures.

[0162] For DeiT and ViT, the average relative accuracy loss decreases as the model size increases, indicating that the Transformer is better suited for sparsification at larger model sizes.

[0163] (5) Different tasks

[0164] Table 5 provides the performance and OM (Operational Name) of different tasks at different sparsity rates. task Value. And, for ease of understanding, in Figure 1A The radar chart is shown schematically in the lower right corner.

[0165] Table 5: Performance and OM of different tasks at different sparsity ratios task

[0166]

[0167] In the detection model, the PTS method outperforms the classification model. Table 5 shows that the OM (Optical Model) in the detection task... task The score can reach 88.97, while the score for the classification task is 84.05. This shows that attaching subsequent structures (such as the neck and head parts in the detection model) to the backbone network does not reduce the sparsity potential; on the contrary, it may make the model more suitable for sparsification due to the introduction of additional parameters.

[0168] PTS methods still require further exploration in the field of image generation. Generation tasks can only maintain accuracy at a sparsity of 50%, with performance deteriorating at higher sparsity rates. Sparsification methods used for other types of models may not be suitable for diffusion models, or at least perform poorly without proper tuning. This is likely because these methods do not consider the unique operational mechanisms and parameter distributions of diffusion models.

[0169] Overall, as shown in the summary results in Table 6, the best-performing algorithm or model is not necessarily the best in all experimental settings, but it performs well in multiple aspects. In the "result" column, the best-performing algorithms are listed to the left, with performance decreasing from left to right.

[0170] Table 6: Summary Table

[0171]

[0172] Third Embodiment

[0173] Based on the above-described post-training sparse benchmark evaluation method, the third embodiment of the present invention further provides a post-training sparse benchmark evaluation device. For example... Figure 5 As shown, the post-training sparse benchmark evaluation device includes one or more processors and a memory. The memory is coupled to the processors and is used to store one or more computer programs. When the one or more computer programs are executed by the one or more processors, the one or more processors implement the post-training sparse benchmark evaluation method as described in the above embodiments.

[0174] The processor controls the overall operation of the post-trained sparse benchmark evaluation device to complete all or part of the steps of the post-trained sparse benchmark evaluation method. This processor module can be a central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), digital signal processing (DSP) chip, etc. The memory stores various types of data to support the operation of the post-trained sparse benchmark evaluation device. This data may include, for example, instructions for any application or method used to operate the post-trained sparse benchmark evaluation device, as well as application-related data. This memory module can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, etc.

[0175] It should be noted that the above embodiments are merely illustrative examples. The technical solutions of the various embodiments can be combined, and each step can also be used as an evaluation method independently without all steps being required. The order of the steps can also be changed, all of which are within the protection scope of this invention.

[0176] The training-based sparse benchmark evaluation method and apparatus provided by this invention have been described in detail above. Any obvious modifications made by those skilled in the art without departing from the essence of this invention will constitute an infringement of the patent rights of this invention and will incur corresponding legal liability.

Claims

1. A post-training sparse benchmark evaluation method for computer vision tasks, characterized in that... Includes the following steps: S11: For images in the training dataset, calculate the metrics for a specific task based on the original model. Then, sparse models are obtained using different sparse allocation methods, and the performance metrics of the sparse models for specific tasks are calculated. Finally based on Calculate the relative values ​​of the metrics for all architectures and datasets on each task, and use their mean as the overall sparse distribution metric. The specific task includes any one or more of classification tasks, detection tasks, and image generation tasks. S12: Based on the sparse model in step S11, calculate the difference in the index results before and after reconstruction for all architectures and datasets on each task, and then calculate the average value to obtain the overall reconstruction index. S13: Calculate the accuracy of the original model and the sparse model under each model architecture, and calculate the average relative accuracy of all architectures and datasets under each model architecture to obtain the overall model metric. ; S14: Using the accuracy of the original model at size j as a baseline, calculate the accuracy of the sparse model, and calculate the average relative accuracy of all architectures and datasets at each size as a robustness metric. ; S15: Using the accuracy of the original model on task n as a benchmark, calculate the accuracy of the sparse model on task n, and calculate the average relative accuracy of all architectures and datasets across all models as an overall task metric. , In step S11, the quadratic mean of the relative values ​​of the indicators for each task is calculated as the overall sparse distribution indicator. , in, This represents the index result calculated by the original model under a specific task; This represents the index results calculated by various sparse models under different sparsity rates for a specific task. include , , ; include , , ; It is the mean operator; Step S12 includes the following sub-steps: S121: Based on the sparse model in step S11, calculate the performance difference before and after reconstruction for all architectures and datasets on each task. ; in, This represents the sparse accuracy achieved using a specific refined reconstruction technique for a given task. This indicates the sparse accuracy under a specific task, without employing that particular refined reconstruction technique. yes , , A general term, , , These represent the performance differences before and after reconstruction in classification, detection, and image generation tasks, respectively. S122: Based on the difference in index results before and after reconstruction Calculate the average value to obtain the overall reconstruction index. ; ； In step S13, the overall model index Calculated using the following formula: in, This represents the accuracy of a specific model architecture at the i-th sparsity level. This represents the accuracy of the i-th sparsity ratio of the sparse model obtained after sparsification of this specific model; This represents the quantity of sparsity; Step S14 includes the following sub-steps: S141: Calculate the relative accuracy of the quadratic mean of a model architecture of a specific size using the following formula. , in, It represents the accuracy of a specific original model on a specific model architecture; This represents the accuracy of the sparse model obtained after sparsifying the specific original model at the j-th sparsity rate under the same model architecture. This represents the quantity of sparsity; S142: The standard deviation between different sizes of the same model architecture is calculated using the following formula as a robustness indicator. , in, This represents the standard deviation operator.

2. The post-training sparse benchmark evaluation method as described in claim 1, characterized in that... Step S11 includes the following sub-steps: S111: Calculate the metrics of the original model on one or more tasks. ; S112: Based on the selected sparse allocation method, the original model is sparsified to obtain a sparse model; S113: Calculate the sparsity allocation index of the sparse model in one or more tasks. .

3. The post-training sparse benchmark evaluation method as described in claim 1, characterized in that: The specific refined reconstruction technique includes different reconstruction techniques that vary the use of error, reconstruction granularity, and reconstruction input, and only one of these is changed each time a calculation is performed.

4. A post-training sparse benchmark evaluation device, characterized in that... It includes a processor and a memory, the processor and the memory being coupled together; wherein the memory is used to store computer programs; The processor is used to run a computer program stored in the memory to perform the post-training sparse benchmark evaluation method according to any one of claims 1 to 3.

Images