Post-training sparse reference evaluation method and device

Abstract

The invention discloses a post-training sparse benchmark evaluation method and device. According to the method, an accurate evaluation system is constructed through five indexes, namely a sparse distribution mode, a reconstruction technology, model architecture accuracy, size robustness and different task accuracy. After a standardized sparse normal form is adopted to carry out sparse processing on an original model, the sparse performance of the model is evaluated by using the indexes. According to the method, different trained sparse models can be transversely compared, the performance of the models in the aspects of algorithms and the models is comprehensively evaluated, and the influence of sparsification and reconstruction on the performance and the influence of architecture, size or tasks on the sparsification performance are distinguished. By means of the method, the model compression efficiency is improved, and important reference is provided for model selection and optimization.

Description

Technical Field

[0001] The present invention relates to a post-training sparse benchmark evaluation method and also relates to a corresponding post-training sparse benchmark evaluation device, belonging to the technical field of machine learning. Background Art

[0002] Deep learning technology has been widely used in many fields, but its huge demand for memory and computing power has become a challenge. To meet this challenge, researchers have developed a variety of model compression strategies, including model quantization, model sparsification, network distillation, lightweight network design, and low-rank decomposition. Among these strategies, post-training sparsity (PTS) has attracted attention due to its low training cost.

[0003] At present, some representative methods have emerged in the field of post-training sparsity, such as Post-training Optimization Tool (POT), AdaPrune, and Optimal Brain Surgeon (OBC). These cutting-edge technologies can maintain model performance after sparsification. However, the PTS method still faces two major problems:

[0004] First, the fine-grained exploration of post-training sparsification algorithms is not sufficient. Current PTS methods generally follow a fixed process: first assign a sparsity rate to each layer of the model for sparsification, and then restore performance by reconstructing activations. Although the research follows this process, the fine-grained exploration of PTS technology is still insufficient. For example, most PTS methods use layer-by-layer reconstruction during the reconstruction process, while block-by-block reconstruction has been proven to be effective in quantization methods but has not yet been explored in PTS. The lack of in-depth analysis of fine-grained techniques limits the further development of PTS methods.

[0005] Second, the relationship between models and sparsity has not been fully explored. Although current PTS studies have been evaluated on specific models and tasks, the sparsification potential of different models has not been discussed. It is still unclear which types of models are more suitable for sparsification for different architectures, sizes, and tasks. Researchers usually choose common models (such as ResNet and RegNetX) or datasets to validate their methods, but lack a comprehensive evaluation of the relationship between models and sparsity. In practical applications, multiple network architectures of similar sizes often need to be sparsified for efficient reasoning, but it is not clear which network architecture is more sparsity-friendly. In addition, as the deployment platform changes, models of different sizes may need to be used, and the robustness of these models in sparse algorithms has not been studied. At the same time, current studies have not considered the generalization ability of sparse models for different tasks. The model structure under different tasks needs to be designed and modified for each task, and the impact of these task-oriented designs and modifications on sparsity is also unknown. Therefore, it is necessary to conduct a comprehensive evaluation of the model architecture from both theoretical and practical perspectives. Summary of the invention

[0006] The primary technical problem to be solved by the present invention is to provide a post-training sparse benchmark evaluation method.

[0007] Another technical problem to be solved by the present invention is to provide a post-training sparse benchmark evaluation device.

[0008] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0009] According to a first aspect of an embodiment of the present invention, a post-training sparse benchmark evaluation method is provided, comprising the following steps:

[0010] S11: Calculate the index A of a specific task based on the original model * , and then use different sparse allocation methods to obtain the sparse model, and then calculate the index of the sparse model in the specific task Finally, based on A * Calculate the relative value of the index for all architectures and datasets on each task, and use its mean as the overall sparse allocation index OM alloc ;

[0011] S12: Based on the sparse model in step S11, the difference in the index results before and after reconstruction of all architectures and data sets on each task is calculated, and then the average value is calculated to obtain the overall reconstruction index;

[0012] S13: Calculate the accuracy of the original model and the sparse model under each model architecture, and calculate the average relative accuracy of all architectures and data sets in each model architecture to obtain the overall model indicator OM arch ;

[0013] S14: Calculate the accuracy of the sparse model based on the accuracy of the original model at the jth size, and calculate the average relative accuracy of all architectures and datasets at each size as the robustness indicator OM robust ;

[0014] S15: Based on the accuracy of the original model in the nth task, calculate the accuracy of the sparse model in the nth task, and calculate the average relative accuracy of all architectures and datasets on each model as the overall task indicator OM task .

[0015] Preferably, the step S11 includes the following sub-steps:

[0016] S111: Calculate the index A of the original model in one or more tasks * ;

[0017] S112: based on the selected sparse allocation method, the original model is sparsely distributed to obtain a sparse model;

[0018] S113: Calculate the sparse allocation index of the sparse model in one or more tasks

[0019] S114: Calculate the quadratic mean of the relative values ​​of the indicators of each task as the overall sparse allocation indicator OM alloc ;

[0020]

[0021] Among them, A * Indicates the indicator results calculated by the original model under a specific task; Indicates the indicator results calculated by each sparse model at different sparsity rates under a specific task; include A * Including A CLS , A DET , A GEN ; is the mean operator.

[0022] Preferably, step S12 includes the following sub-steps.

[0023] S121: Based on the sparse model in step S11, calculate the performance difference before and after reconstruction of all architectures and datasets on each task

[0024]

[0025] in, Indicates the accuracy of sparse reconstruction using a specific reconstruction technique for a specific task; It represents the accuracy of sparse reconstruction without using the specific reconstruction technology under a specific task; yes A general term for Respectively represent the performance difference before and after reconstruction in the classification task, detection task, and generation task;

[0026] S122: Based on the difference in indicator results before and after reconstruction Calculate the average value to get the overall reconstruction index OM recon ;

[0027]

[0028] Preferably, the specific reconstruction technique comprises different reconstruction techniques which respectively change whether to use error, reconstruction granularity and reconstruction input composition, and only one of them is changed each time calculation is performed.

[0029] Preferably, the overall model indicator OM arch The calculation of is as follows:

[0030]

[0031] in, represents the accuracy of a specific model architecture at the i-th sparse rate, represents the accuracy of the i-th sparse rate of the sparse model obtained after the specific model is sparsified; C represents the number of sparse rates.

[0032] Preferably, step S14 includes the following sub-steps:

[0033] S141: The quadratic mean relative accuracy G of a model architecture of a specific size is calculated by the following formula size ,

[0034]

[0035] in, Indicates the accuracy of a specific original model in a specific model architecture; represents the accuracy of the sparse model obtained after the specific original model is sparse at the jth sparse rate under the same model architecture; C represents the number of sparse rates;

[0036] S142: Calculate the standard deviation between different sizes of the same model architecture as the robustness indicator OM robust ,

[0037] OM robust =std(G size )

[0038] Here, std(·) represents the standard deviation operator.

[0039] Preferably, the task includes any one or more of a classification task, a detection task and a generation task.

[0040] According to a second aspect of an embodiment of the present invention, a post-training sparse benchmark evaluation device is provided, comprising a processor and a memory, wherein the processor and the memory are coupled; wherein the memory is used to store a computer program; and the processor is used to run the computer program stored in the memory to execute the above-mentioned post-training sparse benchmark evaluation method.

[0041] Compared with the prior art, the present invention realizes the horizontal comparison of the sparsification capabilities of different post-training sparsity (PTS) models and the comprehensive evaluation of the technical effects by constructing an accurate evaluation system including five indicators. These five indicators include sparse allocation mode, reconstruction technology, accuracy of model architecture, robustness of size, and task accuracy of different tasks (classification, detection and image generation). The evaluation process first uses a standardized sparse paradigm to perform sparsification on the original model to generate a tested model, and then uses the above indicators to evaluate the sparsification capability of the tested model. Using the present invention, firstly, the sparsification capabilities and pluggable sparsification algorithms of different PTS models can be accurately compared horizontally; secondly, different PTS models can be comprehensively evaluated from two aspects of algorithms and models, including the refinement technology and sparsification model of post-training sparsity; thirdly, the influence of sparsification and reconstruction on model performance can be distinguished; finally, the influence of architecture, size or task on the sparsification performance of the model can be identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1A A schematic radar chart of evaluation indicators in the first embodiment of the present invention;

[0043] Figure 1B A schematic diagram showing the relationship between the tracks corresponding to the evaluation indicators in Figure 1;

[0044] Figure 2 Schematic diagram of the sparse paradigm adopted by PTS technology;

[0045] Figure 3 In the second embodiment of the present invention, each model adopts a different sparse allocation method for each layer of the sparse rate allocation diagram;

[0046] Figure 4 is a graph of the average relative accuracy loss for different model sizes;

[0047] Figure 5 A schematic diagram of a post-training sparse benchmark evaluation device provided in the third embodiment of the present invention. DETAILED DESCRIPTION

[0048] The technical content of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] The present invention proposes a set of post-training sparse benchmark evaluation methods, aiming to build a comprehensive evaluation system to better select and optimize models. Figure 1A and Figure 1B As shown, the evaluation system includes five indicators: sparse allocation method, reconstruction technology, accuracy of model architecture, robustness of size, and task accuracy of different tasks (including classification, detection, and image generation). In order to optimize the evaluation results, this method first adopts a standardized sparse paradigm, performs sparse processing according to the five evaluation indicators of the original model, and generates a tested model. Subsequently, the same five indicators are used to evaluate the sparsity capability of the tested model. It should be noted that the standardized sparse paradigm is not limited to the method in the embodiment of the present invention, and other sparse paradigms may also be used, as long as each original model is sparsed under the same evaluation indicators to compare the sparse capabilities.

[0050] First embodiment

[0051] Aiming at the needs of actual model production, this paper designs a post-training sparse benchmark evaluation system that includes five tracks and their corresponding indicators, such as Figure 1A and Figure 1B As shown. This system provides a comprehensive evaluation framework through five indicators - sparse allocation method, reconstruction technology, accuracy of model architecture, robustness of size, and task accuracy of different tasks (classification, detection and image generation), which aims to overcome the limitations of current research on post-training sparsity (PTS) and effectively evaluate a variety of PTS models and their effects. In one embodiment of the present invention, we first use a standardized sparse paradigm to sparsify the original model to generate a tested model. This process involves sparsifying the five evaluation indicators of the original model. Then, the sparse tested model is evaluated using these five evaluation indicators. Therefore, the following content will first introduce the standardized sparse paradigm, and then explain how to use the post-training sparse benchmark evaluation system composed of these five evaluation indicators to evaluate the tested model.

[0052] In one embodiment of the present invention, the original model is sparsely processed using a standardized sparse paradigm to obtain a model to be tested, including the following steps:

[0053] S11: Calculate the index A of a specific task based on the original model * , and then use different sparse allocation methods to obtain the sparse model, and then calculate the index of the sparse model in the specific task Finally, based on A * Calculate the relative value of the index for all architectures and datasets on each task, and use its mean as the overall sparse allocation index OM alloc .

[0054] As is known to all, the purpose of using PTS techniques is to sparse a pre-trained neural network while maintaining its accuracy on a specific task (any one or more of classification, detection, and image generation). Currently, existing PTS techniques all adopt a two-step sparse paradigm including sparse allocation and reconstruction, such as Figure 1B As shown in Figure 2. In the sparse allocation process, a specific sparsity rate is assigned to each layer according to a predetermined indicator, and the weights of the corresponding positions are reset to zero according to the sparsity criterion. The reconstruction process is used to restore the model accuracy that has decreased due to sparsification.

[0055] Currently, sparse allocation methods can be divided into the following three categories:

[0056] Heuristic strategy: In this strategy, the sparsity rate of each layer is manually predetermined, such as uniform sparsity;

[0057] Indicator-based strategy: sort the weights of all layers according to a specific indicator, and set the weights with lower scores to zero in a certain proportion;

[0058] Learning-based strategy: The sparsity rate of each layer is learned by optimizing the loss function to achieve the optimal sparse distribution.

[0059] In one embodiment of the present invention, in order to evaluate the performance of different sparse allocation methods and further study the characteristics and nature of an ideal sparse allocation, any one of the following four sparse allocation methods may be selected:

[0060] Uniform: This is a widely used heuristic strategy for sparse allocation, which assigns the same sparse rate to each layer.

[0061] L2 Norm (L2 Normalization): This is an indicator-based sparse allocation method that determines the sparsity rate by calculating the L2 norm of each layer;

[0062] ERK (Effective Rank-based Knowledge): This is also a sparse allocation method based on indicators, but it is more sophisticated than L2Norm. Both L2Norm and ERK have high effectiveness and high citation rates;

[0063] FCPTS (Fine-Grained Channel Pruning via Reinforcement Learning): This is a typical learning-based method and the only learning-based sparse allocation method.

[0064] It should be noted that although four sparse allocation methods are selected in this embodiment to sparse the original model respectively, three types of methods, heuristic, index and learning, among which ERK and L2Norm are two representative index methods, other sparse allocation methods can also be used by those skilled in the art.

[0065] Taking the accuracy of the original model as the benchmark, calculate the average relative accuracy of all architectures and datasets on each task, including the following sub-steps.

[0066] S111: Calculate the index A of the original model in one or more tasks * .

[0067] In one embodiment of the present invention, the accuracy A of the original model in the classification (CLS) task is calculated respectively. CLS , mAP (mean Average Precision) value of the detection (DET) task A DET Or the FID (Frechet Inception Distance) value A of the image generation (GEN) task GEN Here, we use the accuracy rate A * Representative A CLS , A DET and A GEN and collectively.

[0068] The accuracy of classification tasks, mAP of detection tasks, and FID of image generation tasks are commonly used indicators for evaluating models and will not be discussed here.

[0069] S112: Based on the selected sparse allocation method, the original model is sparsely distributed to obtain a sparse model.

[0070] As mentioned above, the sparse allocation method in this embodiment includes four methods (uniform sparse allocation, L2Norm, ERK and FCPTS), so the original model is sparsely allocated using any one of the four sparse allocation methods to obtain a sparse model. It can be understood that the sparse allocation method is not limited to these four methods, and other sparse allocation methods, such as gradient-based sparse rate allocation or density estimation-based sparse rate allocation (KDE), can also be used. It only needs to ensure that the same sparse allocation method is used for each original model to achieve evaluation.

[0071] S113: Calculate the sparse allocation index of the sparse model in one or more tasks

[0072] In one embodiment of the present invention, the accuracy of the sparse model in the classification task is calculated respectively. mAP value of detection task Or the FID value of the image generation task Here, the sparse allocation index is used represent and and collectively,

[0073] S114: Calculate the quadratic mean of the relative values ​​of the indicators of each task as the overall sparse allocation indicator OM alloc .

[0074]

[0075] Among them, A * Indicates the indicator results calculated by the original model under a specific task (i.e., one of classification, detection, and image generation) (i.e., accuracy of classification tasks, mAP of detection tasks, and FID of image generation tasks); It indicates the index results calculated by each sparse model at different sparsity rates under a specific task. is the mean operator. It should be noted that FID is an indicator that the lower the value, the better, and its value range is [1, +∞]. In order to ensure that the overall sparse allocation indicator OM alloc The value of is in the interval [0, 1], so its reciprocal is taken.

[0076] Through the above steps, the selected original model is sparsely distributed under different sparse allocation algorithms, and the OM corresponding to each sparse allocation algorithm is calculated. alloc It can be seen that in this step, the accuracy of the original model after sparseness is measured without reconstruction, which can directly observe the impact of different sparse allocation strategies and distinguish the accuracy impact brought by different sparse allocation strategies and different reconstruction methods. Such a distinction can be more conducive to the performance analysis and optimization of the PTS method.

[0077] S12: Based on the sparse model in step S11, the difference in the index results before and after reconstruction of all architectures and data sets on each task is calculated, and then the average value is calculated to obtain the overall reconstruction index.

[0078] After the sparsity rate is assigned, the PTS method applies reconstruction to reconstruct the sparse activations to compensate for the accuracy loss caused by sparsity. The refined reconstruction techniques that can significantly affect the effectiveness of sparsity during the sparsification process include: error correction, reconstruction input, and reconstruction granularity, which are described as follows:

[0079] Error correction is widely used in many post-training quantization (PTQ) methods. Its purpose is to align the compressed weight distribution with the original weight distribution. Specifically, the error correction process can be written as follows:

[0080]

[0081] After the error correction operation, and are the corrected weights and biases, W s represents the weight of the sparse model before correction. b d , W d and X d are the bias, weight, and input activation in the original model, respectively. f(W, X) represents the convolution operation of the network layer on the input X with weight W. E and σ are the mean and standard deviation operators, and ∈ is a small constant. In this way, the errors caused by the distribution shift of weights and biases can be corrected.

[0082] During the reconstruction process, you can choose to use the output of the previous reconstruction unit in the dense model as input (dense input), or choose to use the output of the previous sparse unit as input (sparse input). The reconstruction input has a great impact on the result.

[0083] The reconstruction process can be performed at different granularities. Many post-training quantization methods have demonstrated that the reconstruction granularity has a significant impact on the quantization performance.

[0084] In one embodiment of the present invention, three reconstruction granularities are tested:

[0085] Single reconstruction: weight reconstruction is performed based on each individual layer, which is the smallest reconstruction granularity;

[0086] Layer-wise reconstruction: weight reconstruction is performed at the layer level. For example, in a convolutional neural network (CNN), reconstruction is performed in a combination of convolutional layers (CONV), batch normalization layers (BN), and ReLU activation layers;

[0087] Block-wise reconstruction: weight reconstruction is performed based on the block level (e.g. residual block). Blocks are the core components of the neural network architecture, such as the residual bottleneck block.

[0088] In one embodiment of the present invention, step S12 includes the following sub-steps.

[0089] S121: Based on the sparse model in step S11, calculate the performance difference before and after reconstruction of all architectures and datasets on each task

[0090] Specifically, the performance of the reconstruction process equipped with different refined reconstruction techniques is compared with that without reconstruction, and the performance difference before and after reconstruction is obtained.

[0091]

[0092] in, It represents the accuracy of sparse reconstruction using a specific refined reconstruction technique for a specific task. It represents the accuracy of the sparse reconstruction without using the specific refined reconstruction technique for a specific task. yes A general term for Respectively represent the performance difference before and after reconstruction in the classification task, detection task and generation task. Here, the specific reconstruction technique includes different reconstruction techniques including whether to use error, reconstruction granularity and reconstruction input composition, and only one of them is changed in each calculation.

[0093] S122: Based on the difference in indicator results before and after reconstruction Calculate the average value to get the overall reconstruction index OM recon .

[0094] Calculation of the overall reconstruction metric Since the FID score of the generation task is on a different scale from the other two tasks, in order to standardize the scale, the results of the generation task are exponentially calculated.

[0095] The overall reconstruction index can be calculated by the following formula:

[0096]

[0097] Similarly, the overall reconstruction index also uses the quadratic mean method.

[0098] In addition, in order to evaluate the sparsification capability of the model, this embodiment is evaluated through three evaluation indicators: "model architecture", "model size robustness" and "application task".

[0099] S13: Calculate the accuracy of the original model and the sparse model under each model architecture, and calculate the average relative accuracy of all architectures and data sets in each model architecture to obtain the overall model indicator OM arch .

[0100] Overall model indicator OM arch The calculation of is as follows:

[0101]

[0102] in, represents the accuracy of a specific model architecture at the i-th sparse rate, represents the accuracy of the i-th sparse rate of the sparse model obtained after the specific model is sparse; C represents the number of sparse rates. In simple terms, OM arch The metric evaluates the average performance of different sparsity ratios within the same model architecture.

[0103] S14: The accuracy of the sparse model is calculated based on the accuracy of the original model at the jth size, and the average relative accuracy of all architectures and datasets at each size is calculated as a robustness indicator.

[0104] In actual application scenarios, the scale of deployed models will vary due to different requirements and resources. The post-training sparse benchmark evaluation method provided by the embodiment of the present invention can show consistent effectiveness in models with the same model architecture but with different depths and widths. In other words, if a model only shows superior sparsity performance in versions with a certain amount of parameters, but shows greater instability at other parameter scales, the post-training sparse benchmark evaluation method provided by the embodiment of the present invention can evaluate this instability, so that it will not be regarded as a friendly sparse architecture design.

[0105] Therefore, the post-training sparse benchmark evaluation method provided by the embodiment of the present invention utilizes a robustness indicator to test and quantify the robustness of the model size.

[0106] In one embodiment of the present invention, step S14 includes the following sub-steps:

[0107] S141: The quadratic mean relative accuracy G of a model architecture of a specific size is calculated by the following formula size :

[0108]

[0109] in, Indicates the accuracy of a specific original model in a specific model architecture; It represents the accuracy of the sparse model obtained after the sparsification of the specific original model under the jth sparsity rate under the same model architecture. C represents the number of sparsity rates.

[0110] S142: Calculate the standard deviation between different sizes of the same model architecture as the robustness indicator OM robust ,

[0111] OM robust =std(G size )

[0112] Here, std(·) represents the standard deviation operator.

[0113] It should be noted that in order to avoid the reconstruction process being affected by different model sizes, G should be calculated without reconstruction. size .

[0114] S15: Based on the accuracy of the original model in the nth task, calculate the accuracy of the sparse model in the nth task, and calculate the average relative accuracy of all architectures and datasets on each model as the overall task indicator OM task .

[0115] In various computer vision tasks, models are often combined and enhanced before deployment. For example, ResNet is used as the backbone network in detection tasks and connected to the neck and head modules. In image generation tasks, CNN and Transformer are also used in combination. Therefore, it is necessary to test the sparsification capability from the perspective of the task.

[0116] Therefore, in order to evaluate the sparsification ability of each model in different tasks, the embodiment of the present invention takes three commonly used typical tasks (classification, detection and image generation) as examples for evaluation, and uses the formula to calculate the overall task index OM task :

[0117]

[0118] in, represents the accuracy of the nth task of a specific model without sparsity; represents the accuracy of the nth task after sparseness; N represents the number of tasks (in this embodiment, N=3 is preferred).

[0119] It should be noted that if the purpose of the evaluation is only one task (for example, only the image generation task and no other tasks), for example, in a specific application scenario, there is only one task, then during the evaluation, the aforementioned steps only evaluate the evaluation values ​​of the various indicators of this one task, and there is no need to evaluate the remaining two tasks.

[0120] In addition, the post-training sparse benchmark evaluation method provided in an embodiment of the present invention includes three evaluation indicators: the accuracy of the model architecture, the robustness of the size, and the task accuracy of different tasks. However, depending on the application scenario, only one or two of these three evaluation indicators can be used, and other indicators can be added.

[0121] It should be noted that among the above five evaluation indicators, except for OM robust In addition, all indicators are positive indicators. In addition, the calculation steps of each evaluation indicator can be performed simultaneously or in any order. In the above embodiment, only one order is selected for the convenience of description, but it does not constitute a limitation of the present invention.

[0122] In addition, in various embodiments of the present invention, in order to unify the overall indicators of different tracks, each indicator in the post-training sparse evaluation benchmark adopts the quadratic mean form. This method can reduce the excessive impact of particularly poor results on the indicators, thereby more accurately evaluating the comprehensive performance of each track.

[0123] By using the post-training sparse benchmark evaluation method provided by the embodiment of the present invention, the following technical effects can be achieved: 1) an accurate evaluation system is constructed to compare the sparsification capabilities of different PTS models and pluggable sparsification algorithms; 2) different PTS models are comprehensively evaluated from both algorithm and model aspects, and the refinement technology and sparsification model of post-training sparsity can be evaluated; 3) the impact of sparsification and reconstruction on sparsification performance can be distinguished; 4) the impact of architecture, size or task on sparsification performance can be distinguished.

[0124] Second embodiment

[0125] The second embodiment of the present invention provides a specific application example of a post-training sparse benchmark evaluation method.

[0126] In order to verify the post-training sparse benchmark evaluation method provided by the embodiment of the present invention, the currently known post-training sparse model architecture (as shown in Table 1) was tested.

[0127] Table 1: Model architecture used in this example

[0128]

[0129] The post-training sparse benchmark evaluation method provided in the embodiment of the present invention is implemented using the PyTorch framework. Figure 2 The process shown is to sparse the original model: the original model is trained with the data set, and then the sparse distribution and reconstruction are performed to obtain the final sparse model. During the reconstruction process, this embodiment uses the SGD (stochastic gradient descent) optimizer for optimization. Among them, the momentum is set to 0.9 and the learning rate is set to 1e-4. 1024 images in the training data set are randomly selected as the calibration data set, and calibrated for 20,000 epochs. The batch size is set to 64. During the implementation process, it was observed that when the sparsity rate is less than 50%, the performance degradation of almost all models after sparsification is negligible. On the other hand, when the sparsity rate is higher than 80%, the accuracy of almost all model settings collapses. Therefore, this embodiment mainly presents the results under the sparsity rate of {0.5, 0.6, 0.7, 0.8}.

[0130] In the evaluation process of the two main steps of the PTS process of "sparse rate allocation" and "reconstruction" in this embodiment, for classification tasks, this embodiment tested ResNet-18 / 50, RegNetX-200M / 400M, MobileNetV2 and Transformer-based ViT models on the ImageNet-1K dataset; and tested ResNet32 / 56 and VGG-19 on the CIFAR-10 / 100 dataset.

[0131] For the detection task, this embodiment tests RetinaNet-r18 / 50} on the MSCOCO-2017 dataset, and tests MobileNetV1-SSD and MobileNetV2-SSDLite on the PASCAL VOC07 dataset.

[0132] For the image generation task, this embodiment evaluates stable diffusion on the LSUN-Churches / Bedroom dataset.

[0133] ① Sparse rate allocation

[0134] Four sparse allocation methods were selected: uniform sparse allocation (Uniform), L2Norm, ERK and FCPTS. In addition, the combination of FCPTS and L2Norm (denoted as FCPTS(L2Norm)) and the combination of FCPTS and ERK (denoted as FCPTS(ERK)) were also selected.

[0135] According to the method in the first embodiment, the overall sparse allocation index OM is calculated alloc The results are shown in Table 2 below, and the Figure 1A The radar chart in the upper left center provides a visual representation.

[0136] Table 2: OM of different sparse allocation methods under different tasks alloc value

[0137]

[0138] The evaluation results of different sparse allocation strategies are shown in Table 2. In order to facilitate a more detailed analysis, Table 2 also provides the OM of each task. alloc The root mean square component (denoted as MS) of , as well as its specific performance at each sparsity rate.

[0139] As can be seen from Table 2, the effect of the sparse allocation method on sparsification is crucial and significant. On various metrics, the Uniform strategy always performs the worst, while the learning-based methods generally perform well. The gap between the two can reach 20%. ERK and L2Norm perform similarly, while FCPTS initialized with the two strategies has obvious differences, which shows that initialization is very important for learning-based methods and ERK has better fine-tuning potential than L2Norm (i.e., ERK sparse allocation is closer to the optimal distribution).

[0140] According to the results, it is also observed that effective sparsity rate allocation benefits from assigning lower sparsity rates to more sensitive layers. To better determine the sparsity rate of each layer, Figure 3 The sparse allocation of each layer on the CIFAR-100 dataset is visualized in Figure 1. It can be observed that effective methods consistently assign lower sparsity to the last layer. This is because the last layer is directly related to the output features of the network, making the output highly sensitive to changes in the weights of the last layer. Therefore, the last layer is not suitable for large-scale sparsification.

[0141] Furthermore, it can be observed that in the worse performing methods, the weights of certain layers are preserved as much as possible, while in the more effective methods, the weights of these layers are drastically pruned, especially in ResNet-32 and ResNet-56. It is found that these layers tend to be downsampling layers, which are considered as unsuitable layers for sparsification in ERK and L2Norm. However, in FCPTS, downsampling layers are shown to be robust to sparsification. Therefore, it is beneficial to assign lower sparsity rates to more sensitive layers.

[0142] ②Reconstruction

[0143] The refined reconstruction technology used in this embodiment includes: error correction, reconstruction input and reconstruction granularity, including error correction (w / correction), no error correction (w / ocorrection), dense input (dense input), sparse input (sparse input), single reconstruction (single), layer-wise reconstruction (layer-wise), and block-wise reconstruction (block-wise). The evaluation results are shown in Table 3, and Figure 1A The radar chart in the upper right corner of the figure provides a visual display.

[0144] Table 3 Performance of different refined reconstruction techniques at different sparsity rates for different tasks

[0145]

[0146] Through actual evaluation, it can be seen that error correction performs differently in different tasks. For classification tasks, it can be observed that error correction always shows better performance at different sparsity rates. However, for detection and generation tasks, the effects of error correction are different. In the detection task, there is a significant performance degradation after applying error correction, while the generation task seems to be insensitive to the distortion of weight distribution. In object detection tasks, it is crucial to maintain the integrity of spatial information, because detection involves not only "what is" (identifying the object category) but also "where" (locating the object position). Sparsification and subsequent adjustment of weight distribution may destroy the spatial features learned by the model, which is particularly critical for detection tasks. This may weaken the model's ability to accurately locate objects, thereby affecting its overall detection performance.

[0147] It can be observed that using the output of the sparse model as the input for reconstruction is beneficial. As can be seen in Table 3, in most settings, using sparse input can achieve higher performance gains than the original input, especially at higher sparsity rates (e.g., 33.86% vs. 23.40% at 80% sparsity for classification tasks). This may be because using the output of the sparse model can make the PTS algorithm aware of the reconstruction error of the previous layer, thereby avoiding error accumulation in the entire network. Therefore, it is beneficial to use the output of the sparse model as the reconstruction input of the current unit.

[0148] In addition, the inventors also observed that block-by-block reconstruction is always the best. Block-by-block reconstruction achieves the best results in most configurations, while layer-by-layer reconstruction outperforms individual reconstruction. For example, when sparsifying the classification model at an 80% sparsity rate, block-by-block reconstruction can be 3% higher than layer-by-layer reconstruction and 16% higher than individual reconstruction. At high sparsity rates, it is particularly important to design a detailed reconstruction granularity for classification tasks. For example, the differences are not obvious at a 50% sparsity rate, but at an 80% sparsity rate, these differences become significant. However, for detection tasks, this gap is already apparent at low sparsity rates.

[0149] The main advantage of block-by-block reconstruction is that it may better preserve the interactions between weights. If the sparsity of the model is more concentrated at the block level, then block-based reconstruction may be more effective in restoring this sparsity. In addition, block-by-block reconstruction can reduce noise in the reconstruction process and provide more stable results. On the other hand, layer-by-layer reconstruction may cause some key connections between cross-layer weights to be lost, which may limit the expressive power of the model and encounter difficulties in restoring sparsity.

[0150] Block-by-block reconstruction also has advantages in terms of efficiency and resource consumption. Given the large size of feature maps, calculating the loss may require a lot of time and memory resources. Block-based reconstruction can reduce the number of loss calculations.

[0151] (3) Model Architecture

[0152] For different model architectures, the evaluation results and indicators OM under different sparsity rates are shown in Table 4. arch and OM robust For ease of understanding, Figure 1A A radar plot is schematically provided in the lower left of FIG.

[0153] Table 4

[0154]

[0155] It can be observed from Table 4 that different architectures have different sparsity capabilities, and the models based on the attention mechanism have greater sparsity potential. Specifically, it can be found from Table 4 that ViT, MobileNetV3, and DeiT rank among the top overall indicators (OM) among all evaluated models (1st, 2nd, and 4th, respectively). atch MobileNetV2, MobileNetV3 significantly improves its sparsity potential. The main difference between these two similar architectures is that MobileNetV3 introduces a lightweight attention mechanism module. The attention mechanism enables the network to pay more attention to the features that are critical to the final task and reduce its dependence on unimportant features. In the case of sparsification, this helps to more effectively retain information that is critical to performance.

[0156] Moreover, it is also observed that the training strategy can affect its sparsity potential. DeiT has almost the same model architecture as ViT, but its performance is significantly worse. This is due to significant differences in their training strategies. ViT is pre-trained on extremely large datasets such as JFT-300M, allowing it to learn more general feature representations. This makes it more robust during the sparsification process. Even if the number of parameters is reduced, these learned feature representations may still be effective, helping to maintain performance. DeiT adopts knowledge distillation as one of its core strategies to help train efficient models with less data. However, if the sparsification method interferes with the features learned through knowledge distillation during the sparsification process, this may have an adverse effect on the performance of DeiT.

[0157] ④ Robustness of model size

[0158] The robustness of the size of each model actually evaluated robust The results are shown in Table 4, and Figure 1A This is shown in the radar chart in the middle of the lower center.

[0159] It can be observed that high sparse potential for a certain size model ≠ high sparse potential for all sizes of models. For example, in OM arch The well-performing DeiT and ResNet models are in OMrobust The ViT model and the MobileNetV3 model perform poorly on both metrics, while the ViT model and the MobileNetV3 model perform consistently on both metrics. This suggests that there is no strict positive correlation between the sparse potential of a model and its size robustness. However, models with high sparse potential are more likely to have good model size robustness.

[0160] In addition, larger model size does not necessarily mean better sparsity capability. Figure 4 The average relative accuracy loss (G size ) to provide a more detailed analysis. Figure 4 The following conclusions can be drawn:

[0161] For CNNs, the average relative accuracy loss first increases and then decreases as the model size increases. Therefore, a medium-sized network is the best choice for a CNN-based network architecture.

[0162] For DeiT and ViT, the average relative accuracy loss decreases as the model size increases, indicating that the Transformer is more suitable for sparsification at larger model sizes.

[0163] (5) Different tasks

[0164] In Table 5, the performance and OM of different tasks at different sparsity rates are provided. task And, for ease of understanding, in Figure 1A A radar plot is schematically provided in the lower right corner of FIG.

[0165] Table 5: Performance and OM of different tasks at different sparsity rates task

[0166]

[0167] In the detection model, the PTS method performs better than the classification model. As can be seen from Table 5, the OM task It can reach 88.97, while the score of the classification task is 84.05. This shows that attaching subsequent structures (such as the neck and head parts in the detection model) to the backbone network does not reduce the sparse potential; instead, it may make the model more suitable for sparseness due to the introduction of additional parameters.

[0168] The PTS method still needs further exploration in the field of image generation. The generation task can only maintain accuracy at a sparsity rate of 50%, and the performance collapses at higher sparsity rates. Sparsification methods used for other types of models may not be applicable to diffusion models, or at least perform poorly without proper tuning. This may be because these methods do not consider the unique operating mechanism and parameter distribution of diffusion models.

[0169] In general, as shown in the results summarized in Table 6, the best performing algorithms or models do not necessarily perform best in all experimental settings, but perform well in multiple aspects. In the "result" column, the best performance is on the left, and the performance decreases from left to right.

[0170] Table 6: Summary table

[0171]

[0172] Third embodiment

[0173] Based on the above-mentioned post-training sparse benchmark evaluation method, the third embodiment of the present invention further provides a post-training sparse benchmark evaluation device. Figure 5 As shown, the post-training sparse benchmark evaluation device includes one or more processors and a memory. The memory is coupled to the processor and is used to store one or more computer programs. When the one or more computer programs are executed by one or more processors, the one or more processors implement the post-training sparse benchmark evaluation method in the above embodiment.

[0174] Wherein, the processor is used to control the overall operation of the post-training sparse benchmark evaluation device to complete all or part of the steps of the post-training sparse benchmark evaluation method. The processor module can be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a digital signal processing (DSP) chip, etc. The memory is used to store various types of data to support the operation on the post-training sparse benchmark evaluation device, and these data may include instructions for any application or method for the post-training sparse benchmark evaluation device operation, and application-related data. The memory module can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, etc.

[0175] It should be noted that the above embodiments are only examples. The technical solutions of the various embodiments can be combined, and each step can be used as an evaluation method without all steps. The order of the steps can be changed, all within the scope of protection of the present invention.

[0176] The above is a detailed description of the post-training sparse benchmark evaluation method and device provided by the present invention. For those skilled in the art, any obvious changes made to it without departing from the essence of the present invention will constitute an infringement of the patent right of the present invention and will bear corresponding legal responsibilities.

Claims

1. A post-training sparse benchmark evaluation method, characterized in that The steps include: S11: Calculate the index A of a specific task based on the original model * , and then use different sparse allocation methods to obtain the sparse model, and then calculate the index of the sparse model in the specific task Finally, based on A * Calculate the relative value of the index for all architectures and datasets on each task, and use its mean as the overall sparse allocation index OM alloc ; S12: Based on the sparse model in step S11, the difference in the index results before and after reconstruction of all architectures and data sets on each task is calculated, and then the average value is calculated to obtain the overall reconstruction index; S13: Calculate the accuracy of the original model and the sparse model under each model architecture, and calculate the average relative accuracy of all architectures and data sets in each model architecture to obtain the overall model indicator OM arch ; S14: Calculate the accuracy of the sparse model based on the accuracy of the original model at the jth size, and calculate the average relative accuracy of all architectures and datasets at each size as the robustness indicator OM robust ; S15: Based on the accuracy of the original model in the nth task, calculate the accuracy of the sparse model in the nth task, and calculate the average relative accuracy of all architectures and datasets on each model as the overall task indicator OM task .

2. The post-training sparse benchmark evaluation method according to claim 1, characterized in that The step S11 includes the following sub-steps: S111: Calculate the index A of the original model in one or more tasks * ; S112: based on the selected sparse allocation method, the original model is sparsely distributed to obtain a sparse model; S113: Calculate the sparse allocation index of the sparse model in one or more tasks 3. The post-training sparse benchmark evaluation method as claimed in claim 2, characterized in that The step S11 further includes the following sub-steps: S114: Calculate the quadratic mean of the relative values ​​of the indicators of each task as the overall sparse allocation indicator OM alloc , Among them, A * Indicates the indicator results calculated by the original model under a specific task; Indicates the indicator results calculated by each sparse model at different sparsity rates under a specific task; include A * Including A CLS , A DET , A GEN ; is the mean operator.

4. The post-training sparse benchmark evaluation method according to claim 1, characterized in that Step S12 includes the following sub-steps: S121: Based on the sparse model in step S11, calculate the performance difference before and after reconstruction of all architectures and datasets on each task in, Indicates the accuracy of sparse reconstruction using a specific refined reconstruction technique for a specific task; It indicates the accuracy of sparse reconstruction without using the specific refined reconstruction technology under a specific task; yes A general term for Respectively represent the performance difference before and after reconstruction in the classification task, detection task, and generation task; S122: Based on the difference in indicator results before and after reconstruction Calculate the average value to get the overall reconstruction index OM recon ; 5. The post-training sparse benchmark evaluation method according to claim 4, characterized in that: The specific reconstruction techniques include different reconstruction techniques that respectively change whether to use error, reconstruction granularity, and reconstruction input composition, and only one of them is changed each time calculation is performed.

6. The post-training sparse benchmark evaluation method according to claim 1, characterized in that In step S13, the overall model index OM arch Calculated by the following formula: in, represents the accuracy of a specific model architecture at the i-th sparse rate, represents the accuracy of the i-th sparse rate of the sparse model obtained after the specific model is sparsified; C represents the number of sparse rates.

7. The post-training sparse benchmark evaluation method according to claim 1, characterized in that Step S14 includes the following sub-steps: S141: The quadratic mean relative accuracy G of a model architecture of a specific size is calculated by the following formula size , in, Indicates the accuracy of a specific original model in a specific model architecture; represents the accuracy of the sparse model obtained after the specific original model is sparse at the jth sparse rate under the same model architecture; C represents the number of sparse rates; S142: The standard deviation between different sizes of the same model architecture is calculated as the robustness indicator OM by the following formula robust , OM robust =std(G size ) Here, std(·) represents the standard deviation operator.

8. The post-training sparse benchmark evaluation method according to any one of claims 1 to 7, characterized in that: The tasks include any one or more of classification tasks, detection tasks and generation tasks.

9. A post-training sparse benchmark evaluation device, characterized in that The invention comprises a processor and a memory, wherein the processor and the memory are coupled; wherein the memory is used to store a computer program; The processor is used to run the computer program stored in the memory to execute the post-training sparse benchmark evaluation method according to any one of claims 1 to 8.

Images