Optimization method and system for sparse model after training

Abstract

The invention discloses a method and a system for optimizing a sparse model after training. According to the optimization method, sparse optimization is carried out aiming at models of different task types: for classification and detection tasks, a sparse rate of 50%-80% is selected, an FCPTS method is preferentially adopted to carry out sparse distribution, and a block-by-block reconstruction technology with error correction and a block-by-block reconstruction technology without error correction are respectively combined; for a generation task, the sparse rate is controlled below 50%, FCPTS is preferentially selected, and a block-by-block reconstruction technology without error correction or with error correction is combined. And if the tasks include generation tasks, special processing is carried out to keep the sparsity below 50%. According to the method, the trained sparse model is optimized through an innovative method, and the efficiency and performance of the model in practical application are remarkably improved.

Description

Technical Field

[0001] The present invention relates to a method for optimizing a trained sparse model and also relates to a corresponding optimization system for the trained sparse model, belonging to the technical field of machine learning. Background Art

[0002] Deep learning technology has attracted much attention due to its wide application in many fields, but its huge demand for memory and computing power also brings challenges. To address this problem, researchers have developed a variety of model compression strategies, including model quantization, sparsification, network distillation, lightweight network design, and low-rank decomposition. Among these methods, post-training sparsification (PTS) has attracted attention due to its low training cost.

[0003] Some representative methods have emerged in the field of post-training sparsification, such as Post-training Optimization Tool (POT), AdaPrune, and Optimal Brain Surgeon (OBC). These methods can maintain model performance after sparsification with almost no loss. Although these high-performance PTS methods have been implemented, the relationship between models and sparsity has not been fully explored. Current research is mostly evaluated on specific models and tasks, and there is a lack of discussion on the sparsification potential of different models. Therefore, it is not clear which models of different architectures, sizes, and tasks are more suitable for sparsification.

[0004] Existing PTS studies often select commonly used models (such as ResNet and RegNetX) or datasets to verify the methods, but lack a comprehensive evaluation of the relationship between models and sparsity. In practical applications, multiple network architectures of similar size often need to be sparsely processed to achieve efficient reasoning, but it is not clear which network architecture is more sparsity-friendly and can achieve better performance. In addition, as the deployment platform changes, models of different sizes (e.g., different numbers of layers) may need to be used. Current studies have not clearly shown whether network architectures with different sizes are robust in sparse algorithms. At the same time, existing studies have not considered the generalization ability of sparse models for different tasks. The model structure under different tasks usually needs to be designed and modified for each task, and it is not clear whether these task-oriented designs and modifications help sparsity. Therefore, it is very necessary to optimize the model to improve its sparse ability for specific application requirements and / or specific models. Summary of the invention

[0005] The primary technical problem to be solved by the present invention is to provide a method for optimizing a trained sparse model (referred to as model for short).

[0006] Another technical problem to be solved by the present invention is to provide an optimization system for a trained sparse model.

[0007] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0008] According to a first aspect of an embodiment of the present invention, there is provided a method for optimizing a trained sparse model, comprising the following steps:

[0009] S1: Determine whether the model is used for a single task. If not, proceed to step S5; if it is used for a classification task, proceed to step S2; if it is used for a detection task, proceed to step S3; if it is used for a generation task, proceed to step S4;

[0010] S2: For the model used for classification tasks, the sparsity rate satisfies 50% ≤ < 80%, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm. The reconstruction technology uses the technology with error correction and block-by-block reconstruction.

[0011] S3: For the model used for the detection task, the sparsity rate satisfies 50% ≤ < 80%, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm. The reconstruction technology adopts the technology of error-free correction and block-by-block reconstruction.

[0012] S4: For the model used for the generation task, the sparsity rate is set to ≤ 50%, and the sparse allocation method is selected in the order of FCPTS>L2Norm>ERK. The reconstruction technique uses a technique without error correction and block-by-block reconstruction, or a technique with error correction and block-by-block reconstruction.

[0013] S5: Determine whether the task contains the generated task, if so, proceed to step S6; if not, proceed to step S7;

[0014] S6: If the sparseness rate is ≤50%, and the sparse allocation method selects FCPTS, the reconstruction technology adopts the technology of error-free correction and block-by-block reconstruction;

[0015] S7: If 50%≤sparseness rate<80% is satisfied, and the sparse allocation method is selected in the priority order of FCPTS>ERK>L2Norm, the reconstruction technology adopts the technology of error-free correction and block-by-block reconstruction.

[0016] According to a second aspect of an embodiment of the present invention, another method for optimizing a trained sparse model is provided, comprising the following steps:

[0017] S10: Determine whether it is a model for classification tasks, if not, proceed to step S20; if yes, 50%≤sparse rate<80% is satisfied, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm, and the reconstruction technology adopts the technology of error correction and block-by-block reconstruction;

[0018] S20: Determine whether it is a model for the detection task, if not, proceed to step S30; if yes, for the model for the detection task, 50%≤sparse rate<80% is satisfied, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm, and the reconstruction technology adopts the technology of error-free correction and block-by-block reconstruction;

[0019] S30: For the model used for generating the task, the sparsity rate is set to ≤50%, and the sparse allocation method is selected in the priority order of FCPTS>L2Norm>ERK, and the reconstruction technology adopts the technology without error correction and block-by-block reconstruction, or the technology with error correction and block-by-block reconstruction.

[0020] According to a third aspect of an embodiment of the present invention, another method for optimizing a trained sparse model is provided, comprising the following steps:

[0021] S01: Determine the model size. If the model size is large, choose a Transformer-based model architecture; if the model size is medium, choose a CNN-based model architecture;

[0022] S02: Determine the task type. If it is a classification task, proceed to step S03; if it is a detection task, proceed to step S04; if it is a generation task, proceed to step S05;

[0023] S03: For the model used for classification tasks, the sparsity rate satisfies 50% ≤ 80%, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm. The reconstruction technology uses the technology with error correction and block-by-block reconstruction;

[0024] S04: For the model used for the detection task, the sparsity rate satisfies 50%≤50%<80%, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm. The reconstruction technology adopts the technology of error-free correction and block-by-block reconstruction.

[0025] S05: For the model used for generating the task, the sparsity rate is set to ≤50%, and the sparse allocation method is selected in the priority order of FCPTS>L2Norm>ERK, and the reconstruction technology adopts the technology without error correction and block-by-block reconstruction, or the technology with error correction and block-by-block reconstruction.

[0026] Preferably, when the sparse allocation method selects FCPTS, the FCPTS includes FCPTS(ERK) and FCPTS(L2Norm), and FCPTS(ERK) takes precedence over FCPTS(L2Norm).

[0027] Preferably, in the sparse allocation method, the sparse rate of the last layer is ≤50%.

[0028] Preferably, in the sparse allocation method, the sparse rate of the downsampling layer is ≤50%.

[0029] Preferably, the Transformer-based model architecture is a ViT model architecture.

[0030] Preferably, the CNN-based model architecture is a MobileNetV3 model architecture.

[0031] According to a fourth aspect of an embodiment of the present invention, there is provided a system for optimizing a sparse model after training, comprising a processor and a memory, wherein the processor and the memory are coupled; wherein the memory is used to store a computer program; and the processor is used to run the computer program stored in the memory to execute the above-mentioned method for optimizing the sparse model after training.

[0032] Compared with the prior art, the present invention optimizes the trained sparse model through an innovative method, significantly improving the efficiency and performance of the model in practical applications. Through customized optimization strategies for different task types (classification, detection, and generation), the present invention can maintain the accuracy and effect of the model while reducing the complexity of the model. In addition, the present invention also takes into account the adaptability of model size and architecture, and provides flexible model selection to meet the application needs of different scales. Through this optimization, the model can better adapt to changing deployment environments with less resource consumption, and show stronger robustness and generalization capabilities. In the end, the present invention not only improves the operating efficiency of the model, but also enhances the applicability and practicality of the model in diversified tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1A It is a schematic radar chart of the evaluation indicators;

[0034] Figure 1B A schematic diagram showing the relationship between the tracks corresponding to the evaluation indicators in Figure 1;

[0035] Figure 2 Schematic diagram of the sparse paradigm adopted by PTS technology;

[0036] Figure 3 A flow chart of a method for optimizing a sparse model after training provided by the first embodiment of the present invention;

[0037] Figure 4 Flow chart of the sparse allocation method selected according to the priority order of FCPTS(ERK)>FCPTS(L2Norm) in the first embodiment of the present invention

[0038] Figure 5 A flow chart of a method for optimizing a sparse model after training provided in the second embodiment of the present invention;

[0039] Figure 6 A flow chart of a method for optimizing a sparse model after training provided in the third embodiment of the present invention;

[0040] Figure 7 A schematic diagram of a system for optimizing a trained sparse model provided in the fourth embodiment of the present invention. DETAILED DESCRIPTION

[0041] The technical content of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] The present invention provides a method for optimizing a trained sparse model and a corresponding system thereof, aiming to optimize the original model according to specific requirements, such as a specific task, model size or model architecture, so as to improve its sparsification capability. Before explaining in depth the method and system for optimizing a trained sparse model provided by the present invention, it is necessary to first explain the benchmark evaluation method of the trained sparse model.

[0043] The benchmarking method for trained sparse models aims to build a comprehensive evaluation system for better model selection and optimization. Figure 1A and Figure 1B As shown, the evaluation system includes five indicators: sparse allocation method, reconstruction technology, accuracy of model architecture, robustness of size, and task accuracy of different tasks (including classification, detection, and image generation). In order to optimize the evaluation results, this method first adopts a standardized sparse paradigm, performs sparse processing according to the five evaluation indicators of the original model, and generates a tested model. Subsequently, the same five indicators are used to evaluate the sparsity capability of the tested model. It should be noted that the standardized sparse paradigm is not limited to the method in the embodiment of the present invention, and other sparse paradigms may also be used, as long as each original model is sparsed under the same evaluation indicators to compare the sparse capabilities.

[0044] Specifically, the present invention first designs a post-training sparse benchmark evaluation system containing five tracks and their corresponding indicators to meet the needs of actual model production, such as Figure 1A and Figure 1BAs shown. This system provides a comprehensive evaluation framework through five indicators - sparse allocation method, reconstruction technology, accuracy of model architecture, robustness of size, and task accuracy of different tasks (classification, detection and image generation), which aims to overcome the limitations of current research on post-training sparsity (PTS) and effectively evaluate a variety of PTS models and their effects. In one embodiment of the present invention, we first use a standardized sparse paradigm to sparsify the original model to generate a tested model. This process involves sparsifying the five evaluation indicators of the original model. Then, the sparse tested model is evaluated using these five evaluation indicators. Therefore, the following content will first introduce the standardized sparse paradigm, and then explain how to use the post-training sparse benchmark evaluation system composed of these five evaluation indicators to evaluate the tested model.

[0045] In one embodiment of the present invention, the original model is sparsely processed using a standardized sparse paradigm to obtain a model to be tested, including the following steps:

[0046] S11: Calculate the index A of a specific task based on the original model * , and then use different sparse allocation methods to obtain the sparse model, and then calculate the index of the sparse model in the specific task Finally, based on A * Calculate the relative value of the index for all architectures and datasets on each task, and use its mean as the overall sparse allocation index OM alloc .

[0047] As is known to all, the purpose of using PTS techniques is to sparse a pre-trained neural network while maintaining its accuracy on a specific task (any one or more of classification, detection, and image generation). Currently, existing PTS techniques all adopt a two-step sparse paradigm including sparse allocation and reconstruction, such as Figure 1B As shown in Figure 2. In the sparse allocation process, a specific sparsity rate is assigned to each layer according to a predetermined indicator, and the weights of the corresponding positions are reset to zero according to the sparsity criterion. The reconstruction process is used to restore the model accuracy that has decreased due to sparsification.

[0048] Currently, sparse allocation methods can be divided into the following three categories:

[0049] Heuristic strategy: In this strategy, the sparsity rate of each layer is manually predetermined, such as uniform sparsity;

[0050] Indicator-based strategy: sort the weights of all layers according to a specific indicator, and set the weights with lower scores to zero in a certain proportion;

[0051] Learning-based strategy: The sparsity rate of each layer is learned by optimizing the loss function to achieve the optimal sparse distribution.

[0052] In one embodiment of the present invention, in order to evaluate the performance of different sparse allocation methods and further study the characteristics and nature of an ideal sparse allocation, any one of the following four sparse allocation methods may be selected:

[0053] Uniform sparse allocation: This is a widely used heuristic strategy for sparse allocation, which assigns the same sparse rate to each layer;

[0054] L2 Norm (L2 Normalization): This is an indicator-based sparse allocation method that determines the sparsity rate by calculating the L2 norm of each layer;

[0055] ERK (Effective Rank-based Knowledge): This is also a sparse allocation method based on indicators, but it is more sophisticated than L2Norm. Both L2Norm and ERK have high effectiveness and high citation rates;

[0056] FCPTS (Fine-Grained Channel Pruning via Reinforcement Learning): This is a typical learning-based method and the only learning-based sparse allocation method.

[0057] It should be noted that, although four sparse allocation methods are selected in the embodiment of the present invention to sparse the original model respectively, three types of methods, heuristic, index and learning, among which ERK and L2Norm are two representative index methods, other sparse allocation methods may also be used by those skilled in the art.

[0058] Taking the accuracy of the original model as the benchmark, calculate the average relative accuracy of all architectures and datasets on each task, including the following sub-steps.

[0059] S111: Calculate the index A of the original model in one or more tasks * .

[0060] In one embodiment of the present invention, the accuracy A of the original model in the classification (CLS) task is calculated respectively. CLS , mAP (mean Average Precision) value of the detection (DET) task A DET Or the FID (Frechet Inception Distance) value A of the image generation (GEN) task GEN Here, we use the accuracy rate A * Representative A CLS , A DET and AGEN and collectively.

[0061] The accuracy of classification tasks, mAP of detection tasks, and FID of image generation tasks are commonly used indicators for evaluating models and will not be discussed here.

[0062] S112: Based on the selected sparse allocation method, the original model is sparsely distributed to obtain a sparse model.

[0063] As mentioned above, the sparse allocation method in the embodiment of the present invention includes four methods (uniform sparse allocation, L2Norm, ERK and FCPTS), so the original model is sparsely allocated using any one of the four sparse allocation methods to obtain a sparse model. It can be understood that the sparse allocation method is not limited to these four methods, and other sparse allocation methods, such as gradient-based sparse rate allocation or density estimation-based sparse rate allocation (KDE), can also be used. It only needs to ensure that the same sparse allocation method is used for each original model to achieve evaluation.

[0064] S113: Calculate the sparse allocation index of the sparse model in one or more tasks

[0065] In one embodiment of the present invention, the accuracy of the sparse model in the classification task is calculated respectively. mAP value of detection task Or the FID value of the image generation task Here, the sparse allocation index is used represent and and collectively,

[0066] S114: Calculate the quadratic mean of the relative values ​​of the indicators of each task as the overall sparse allocation indicator OM alloc .

[0067]

[0068] Among them, A * Indicates the indicator results calculated by the original model under a specific task (i.e., one of classification, detection, and image generation) (i.e., accuracy of classification tasks, mAP of detection tasks, and FID of image generation tasks); It indicates the index results calculated by each sparse model at different sparsity rates under a specific task. is the mean operator. It should be noted that FID is an indicator that the lower the value, the better, and its value range is [1, +∞]. In order to ensure that the overall sparse allocation indicator OM alloc The value of is in the interval [0, 1], so its reciprocal is taken.

[0069] Through the above steps, the selected original model is sparsely distributed under different sparse allocation algorithms, and the OM corresponding to each sparse allocation algorithm is calculated. alloc It can be seen that in this step, the accuracy of the original model after sparseness is measured without reconstruction, which can directly observe the impact of different sparse allocation strategies and distinguish the accuracy impact brought by different sparse allocation strategies and different reconstruction methods. Such a distinction can be more conducive to the performance analysis and optimization of the PTS method.

[0070] S12: Based on the sparse model in step S11, the difference in the index results before and after reconstruction of all architectures and data sets on each task is calculated, and then the average value is calculated to obtain the overall reconstruction index.

[0071] After the sparsity rate is assigned, the PTS method applies reconstruction to reconstruct the sparse activations to compensate for the accuracy loss caused by sparsity. The refined reconstruction techniques that can significantly affect the effectiveness of sparsity during the sparsification process include: error correction, reconstruction input, and reconstruction granularity, which are described as follows:

[0072] Error correction is widely used in many post-training quantization (PTQ) methods. Its purpose is to align the compressed weight distribution with the original weight distribution. Specifically, the error correction process can be written as follows:

[0073]

[0074] After the error correction operation, and are the corrected weights and biases, W s represents the weight of the sparse model before correction. b d , W d and X d are the bias, weight, and input activation in the original model, respectively. f(W, X) represents the convolution operation of the network layer on the input X with weight W. E and σ are the mean and standard deviation operators, and ∈ is a small constant. In this way, the errors caused by the distribution shift of weights and biases can be corrected.

[0075] During the reconstruction process, you can choose to use the output of the previous reconstruction unit in the dense model as input (dense input), or choose to use the output of the previous sparse unit as input (sparse input). The reconstruction input has a great impact on the result.

[0076] The reconstruction process can be performed at different granularities. Many post-training quantization methods have demonstrated that the reconstruction granularity has a significant impact on the quantization performance.

[0077] In one embodiment of the present invention, three reconstruction granularities are tested:

[0078] Single reconstruction: weight reconstruction is performed based on each individual layer, which is the smallest reconstruction granularity;

[0079] Layer-wise reconstruction: weight reconstruction is performed at the layer level. For example, in a convolutional neural network (CNN), reconstruction is performed in a combination of convolutional layers (CONV), batch normalization layers (BN), and ReLU activation layers;

[0080] Block-wise reconstruction: weight reconstruction is performed based on the block level (e.g. residual block). Blocks are the core components of the neural network architecture, such as the residual bottleneck block.

[0081] In one embodiment of the present invention, step S12 includes the following sub-steps.

[0082] S121: Based on the sparse model in step S11, calculate the performance difference before and after reconstruction of all architectures and datasets on each task

[0083] Specifically, the performance of the reconstruction process equipped with different refined reconstruction techniques is compared with that without reconstruction, and the performance difference before and after reconstruction is obtained.

[0084]

[0085] in, It represents the accuracy of sparse reconstruction using a specific refined reconstruction technique for a specific task. It represents the accuracy of the sparse reconstruction without using the specific refined reconstruction technique for a specific task. yes A general term for Respectively represent the performance difference before and after reconstruction in the classification task, detection task and generation task. Here, the specific reconstruction technique is a different reconstruction technique including whether to use error, reconstruction granularity and reconstruction input composition, and only one of them is changed in each calculation.

[0086] S122: Based on the difference in indicator results before and after reconstruction Calculate the average value to get the overall reconstruction index OM recon .

[0087] Calculation of the overall reconstruction metric Since the FID score of the generation task is on a different scale from the other two tasks, in order to standardize the scale, the results of the generation task are exponentially calculated.

[0088] The overall reconstruction index can be calculated by the following formula:

[0089]

[0090] Similarly, the overall reconstruction index also uses the quadratic mean method.

[0091] In addition, in order to evaluate the sparsification capability of the model, the embodiment of the present invention is evaluated through three evaluation indicators: "model architecture", "model size robustness" and "application task".

[0092] S13: Calculate the accuracy of the original model and the sparse model under each model architecture, and calculate the average relative accuracy of all architectures and data sets in each model architecture to obtain the overall model indicator OM arch .

[0093] Overall model indicator OM arch The calculation of is as follows:

[0094]

[0095] in, represents the accuracy of a specific model architecture at the i-th sparse rate, represents the accuracy of the i-th sparse rate of the sparse model obtained after the specific model is sparse; C represents the number of sparse rates. In simple terms, OM arch The metric evaluates the average performance of different sparsity ratios within the same model architecture.

[0096] S14: The accuracy of the sparse model is calculated based on the accuracy of the original model at the jth size, and the average relative accuracy of all architectures and datasets at each size is calculated as a robustness indicator.

[0097] In actual application scenarios, the scale of deployed models will vary due to different requirements and resources. The post-training sparse benchmark evaluation method provided by the present invention can show consistent effectiveness in models with the same model architecture but different depths and widths. In other words, if a model only shows superior sparsity performance in versions with a certain amount of parameters, but shows greater instability at other parameter scales, the post-training sparse benchmark evaluation method provided by the present invention can evaluate this instability, so that it will not be regarded as a friendly sparse architecture design.

[0098] Therefore, the post-training sparse benchmark evaluation method provided by the present invention utilizes robustness indicators to test and quantify the robustness of the model size.

[0099] In one embodiment of the present invention, step S14 includes the following sub-steps:

[0100] S141: The quadratic mean relative accuracy G of a model architecture of a specific size is calculated by the following formula size:

[0101]

[0102] in, Indicates the accuracy of a specific original model in a specific model architecture; It represents the accuracy of the sparse model obtained after the sparsification of the specific original model under the jth sparsity rate under the same model architecture. C represents the number of sparsity rates.

[0103] S142: Calculate the standard deviation between different sizes of the same model architecture as the robustness indicator OM robust ,

[0104] OM robust =std(G size )

[0105] Here, std(·) represents the standard deviation operator.

[0106] It should be noted that in order to avoid the reconstruction process being affected by different model sizes, G should be calculated without reconstruction. size .

[0107] S15: Based on the accuracy of the original model in the nth task, calculate the accuracy of the sparse model in the nth task, and calculate the average relative accuracy of all architectures and datasets on each model as the overall task indicator OM task .

[0108] In various computer vision tasks, models are often combined and enhanced before deployment. For example, ResNet is used as the backbone network in detection tasks and connected to the neck and head modules. In image generation tasks, CNN and Transformer are also used in combination. Therefore, it is necessary to test the sparsification capability from the perspective of the task.

[0109] Therefore, in order to evaluate the sparsification ability of each model in different tasks, the embodiment of the present invention takes three commonly used typical tasks (classification, detection and image generation) as examples for evaluation, and uses the formula to calculate the overall task index OM task :

[0110]

[0111] in, represents the accuracy of the nth task of a specific model without sparsity; represents the accuracy of the nth task after sparseness; N represents the number of tasks (in the embodiment of the present invention, preferably N=3).

[0112] It should be noted that if the purpose of the evaluation is only one task (for example, only the image generation task and no other tasks), for example, in a specific application scenario, there is only one task, then during the evaluation, the aforementioned steps only evaluate the evaluation values ​​of the various indicators of this one task, and there is no need to evaluate the remaining two tasks.

[0113] In addition, the post-training sparse benchmark evaluation method provided by the present invention includes three evaluation indicators: the accuracy of the model architecture, the robustness of the size, and the task accuracy of different tasks. However, depending on the application scenario, only one or two of these three evaluation indicators can be used, and other indicators can be added.

[0114] It should be noted that among the above five evaluation indicators, except for OM robust In addition, all indicators are positive indicators. In addition, the calculation steps of each evaluation indicator can be performed simultaneously or in any order. In the above embodiment, only one order is selected for the convenience of description, but it does not constitute a limitation of the present invention.

[0115] In addition, in order to unify the overall indicators of different tracks, each indicator in the post-training sparse evaluation benchmark adopts the quadratic mean form. This method can reduce the excessive impact of particularly poor results on the indicators, thereby more accurately evaluating the overall performance of each track.

[0116] Based on the above-mentioned post-training sparse benchmark evaluation method, we can evaluate the sparsification performance of different PTS models and the impact of pluggable sparsification algorithms on performance. If the evaluation results fail to meet the requirements of practical applications, the optimization method of the post-training sparse model provided by the present invention can be used to improve these PTS models or sparsification algorithms. It should be noted that even if the above-mentioned benchmark evaluation method is not used, directly based on the actual training results, the technician can also understand the sparsification capabilities of different PTS models and the impact of their algorithms, and accordingly apply the optimization method of the post-training sparse model provided by the present invention for adjustment. In other words, the above-mentioned post-training sparse benchmark evaluation method is not a necessary step for applying the optimization method of the post-training sparse model provided by the present invention.

[0117] First embodiment

[0118] In the process of model optimization, in order to achieve the transformation of the model from dense to sparse through continued training to adapt to specific application requirements. Figure 3 As shown, the first embodiment of the present invention provides a method for optimizing a trained sparse model, which at least includes the following steps:

[0119] S1: Determine whether the model is used for a single task. If not, proceed to step S5; if it is used for a classification task, proceed to step S2; if it is used for a detection task, proceed to step S3; if it is used for a generation task, proceed to step S4.

[0120] Moreover, this step can be omitted, that is, directly proceed to steps S2 to S4 (such as Figure 3 Dashed lines indicate omitted steps).

[0121] S2: For the model used for classification tasks, the sparsity rate satisfies 50% ≤ 80%, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm. The reconstruction technology uses error correction and block-by-block reconstruction technology.

[0122] It should be noted that the sparse allocation method in this step gives priority to FCPTS. FCPTS includes FCPTS (ERK) and FCPTS (L2Norm), and FCPTS (ERK) takes precedence over FCPTS (L2Norm).

[0123] If FCPTS cannot or is not suitable for practical applications, the ERK method can be selected; then the L2Norm method can be selected.

[0124] That is, the sparse allocation methods can be selected one by one in the priority order of FCPTS(ERK)>FCPTS(L2Norm)>ERK>L2Norm, the sparsity allocation methods can be evaluated under different sparse allocation methods, and finally the appropriate sparse allocation method can be selected. If the sparsity allocation ability of the model under different sparse allocation methods cannot be evaluated, the sparse allocation method can be directly selected according to the above priority order.

[0125] Moreover, a more preferred solution is to allocate a lower sparse rate to the last layer, that is, to satisfy 50%≤sparse rate<80%, and the sparse rate of the last layer≤50%.

[0126] In this step, because the sparse allocation method adopts FCPTS (including FCPTS (ERK) and FCPTS (L2Norm)), ERK or L2Norm, combined with the above content, when the FCPTS (ERK) and FCPTS (L2Norm) sparse allocation methods are adopted, a lower sparse rate (for example, ≤50%) is allocated to the downsampling layer; when the ERK or L2Norm sparse allocation method is adopted, the downsampling layer is not thinned.

[0127] A more preferred technical solution is that this step uses sparse input during reconstruction, and uses the output of the sparse model as the reconstruction input of the current unit. More specifically, the higher the sparsity rate, the higher the accuracy of using sparse input (at the same sparsity rate, the greater the difference with the accuracy of dense input).

[0128] S3: For the model used for the detection task, the sparsity rate satisfies 50% ≤ 80%, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm. The reconstruction technology adopts the technology of error-free correction and block-by-block reconstruction.

[0129] S4: For the model used for generating tasks, the sparsity rate is set to ≤50%, and the sparse allocation method is selected in the order of FCPTS>L2Norm>ERK. The reconstruction technology uses a technology without error correction and block-by-block reconstruction, or a technology with error correction and block-by-block reconstruction.

[0130] S5: Determine whether the task contains the generation task, if so, proceed to step S6; if not, proceed to step S7.

[0131] S6: If the sparseness rate is ≤50%, and the sparse allocation method selects FCPTS, the reconstruction technique adopts the technique of error-free correction and block-by-block reconstruction.

[0132] Preferably, in this step, the sparse allocation method is based on FCPTS(ERK)>FCPTS(L2Norm) (e.g. Figure 4 The priority order is shown).

[0133] S7: If 50%≤sparseness rate<80% is satisfied, and the sparse allocation method is selected in the priority order of FCPTS>ERK>L2Norm, the reconstruction technology adopts the technology of error-free correction and block-by-block reconstruction.

[0134] More specifically, the sparse allocation method is selected in the order of priority of FCPTS(ERK)>FCPTS(L2Norm)>ERK>L2Norm.

[0135] As mentioned above, error correction (w / correction) is the best for classification services (CLS), and no error correction (w / o correction) is the best for detection services (DET). However, because the detection services have higher accuracy requirements when taking these two tasks into account, no error correction is used.

[0136] As an alternative to the embodiments of the present invention, Figure 3 As shown by the dotted line, step S1 and steps S5 to S7 can be omitted. Because for a model of a single task, it is possible to directly enter steps S2 to S4 without judgment. That is, the embodiment of the present invention includes the following steps:

[0137] S2: Determine whether it is a model for classification tasks. If not, proceed to step S3; if yes, 50%≤sparse rate<80% is satisfied, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm. The reconstruction technology adopts the technology of error correction and block-by-block reconstruction.

[0138] S3: Determine whether it is a model for the detection task, if not, proceed to step S4; if yes, for the model for the detection task, 50%≤sparse rate<80% is satisfied, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm, and the reconstruction technology adopts the technology of error-free correction and block-by-block reconstruction;

[0139] S4: For the model used for the generation task, the sparsity rate is set to ≤50%, and the sparse allocation method is selected in the order of FCPTS>L2Norm>ERK. The reconstruction technique uses a technique without error correction and block-by-block reconstruction, or a technique with error correction and block-by-block reconstruction.

[0140] Second embodiment

[0141] like Figure 5 As shown, compared with the first embodiment, the embodiment of the present invention adds a step of selecting a model architecture according to the model size.

[0142] Specifically, on the basis of the first embodiment, the following steps are added after steps S2 to S4 or steps S6 to S7:

[0143] S8: Determine the model size. If the model size is large, choose a Transformer-based model architecture; if the model size is medium, choose a CNN-based model architecture.

[0144] If it is a lightweight model, it will not change.

[0145] It should be noted that both ViT and DeiT are based on the Transformer architecture, but they differ in terms of processing data scale and training efficiency. ViT performs well on large-scale datasets, while DeiT focuses on improving training efficiency on small datasets. Therefore, if the model size increases, the ViT model architecture can be selected. On the other hand, among various CNN models, MobileNetV3 has the greatest sparse potential and is the preferred model based on the CNN model architecture. Therefore, in the process of optimizing the sparse strategy based on the CNN model architecture, MobileNetV3 can be selected.

[0146] S8: Determine the model size. If the model size is large, choose a Transformer-based model architecture; if the model size is medium, choose a CNN-based model architecture;

[0147] S9: Determine the task type. If it is a classification task, proceed to step S2; if it is a detection task, proceed to step S3; if it is a generation task, proceed to step S4;

[0148] S2: For the model used for classification tasks, the sparsity rate satisfies 50% ≤ 80%, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm. The reconstruction technology uses the technology with error correction and block-by-block reconstruction.

[0149] S3: For the model used for detection tasks, the sparsity rate satisfies 50% ≤ 80%, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm. The reconstruction technology uses the technology of error-free correction and block-by-block reconstruction.

[0150] S4: For the model used for the generation task, the sparsity rate is set to ≤50%, and the sparse allocation method is selected in the order of FCPTS>L2Norm>ERK. The reconstruction technique uses a technique without error correction and block-by-block reconstruction, or a technique with error correction and block-by-block reconstruction.

[0151] Third embodiment

[0152] like Figure 6 As shown, compared with the first embodiment, in the embodiment of the present invention, a step S8 of selecting a suitable model architecture as the model size increases is added before step S1 or S5.

[0153] Compared with the second embodiment, the embodiment of the present invention only changes the order of S8, and designs the step of selecting the model architecture according to the model size before step S1 or S5.

[0154] Specifically, the optimization method of the trained sparse model provided by the embodiment of the present invention comprises the following steps:

[0155] S8: Determine the model size. If the model size is large, choose a Transformer-based model architecture; if the model size is medium, choose a CNN-based model architecture;

[0156] S9: Determine the task type. If it is a classification task, proceed to step S2; if it is a detection task, proceed to step S3; if it is a generation task, proceed to step S4;

[0157] S2: For the model used for classification tasks, the sparsity rate satisfies 50% ≤ < 80%, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm. The reconstruction technology uses the technology with error correction and block-by-block reconstruction.

[0158] S3: For the model used for the detection task, the sparsity rate satisfies 50% ≤ < 80%, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm. The reconstruction technology adopts the technology of error-free correction and block-by-block reconstruction.

[0159] S4: For the model used for generating tasks, the sparsity rate is set to ≤50%, and the sparse allocation method is selected in the order of FCPTS>L2Norm>ERK. The reconstruction technology uses a technology without error correction and block-by-block reconstruction, or a technology with error correction and block-by-block reconstruction.

[0160] Fourth embodiment

[0161] Based on the above-mentioned method for optimizing the sparse model after training, the fourth embodiment of the present invention further provides a system for optimizing the sparse model after training. Figure 7 As shown, the optimization system of the trained sparse model includes one or more processors and a memory. The memory is coupled to the processor and is used to store one or more computer programs. When the one or more computer programs are executed by one or more processors, the one or more processors implement the optimization method of the trained sparse model in the above embodiment.

[0162] Wherein, the processor is used to control the overall operation of the optimization system of the trained sparse model to complete all or part of the steps of the optimization method of the trained sparse model. The processor module can be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a digital signal processing (DSP) chip, etc. The memory is used to store various types of data to support operations on the optimization system of the trained sparse model, and these data may include instructions for any application or method for the operation of the optimization system of the trained sparse model, as well as application-related data. The memory module can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, etc.

[0163] It should be noted that the above embodiments are only examples, and the technical solutions of the various embodiments can be combined, the order of the steps can be changed, or some of the steps can be omitted, and these changes are all within the protection scope of the present invention.

[0164] The above is a detailed description of the optimization method and system of the post-training sparse model provided by the present invention. For those skilled in the art, any obvious changes made to it without departing from the essence of the present invention will constitute an infringement of the patent right of the present invention and will bear corresponding legal responsibilities.

Claims

1. A method for optimizing a sparse model after training, characterized in that The following steps are involved: S1: Determine whether the model is used for a single task. If not, proceed to step S5; if it is used for a classification task, proceed to step S2; if it is used for a detection task, proceed to step S3; if it is used for a generation task, proceed to step S4; S2: For the model used for classification tasks, the sparsity rate satisfies 50% ≤ < 80%, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm. The reconstruction technology uses the technology with error correction and block-by-block reconstruction. S3: For the model used for the detection task, the sparsity rate satisfies 50% ≤ < 80%, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm. The reconstruction technology adopts the technology of error-free correction and block-by-block reconstruction. S4: For the model used for the generation task, the sparsity rate is set to ≤ 50%, and the sparse allocation method is selected in the order of FCPTS>L2Norm>ERK. The reconstruction technique uses a technique without error correction and block-by-block reconstruction, or a technique with error correction and block-by-block reconstruction. S5: Determine whether the task contains the generation task, if so, proceed to step S6; if not, proceed to step S7; S6: If the sparseness rate is ≤50%, and the sparse allocation method selects FCPTS, the reconstruction technology adopts the technology of error-free correction and block-by-block reconstruction; S7: If 50%≤sparseness rate<80% is satisfied, and the sparse allocation method is selected in the priority order of FCPTS>ERK>L2Norm, the reconstruction technology adopts the technology of error-free correction and block-by-block reconstruction.

2. A method for optimizing a sparse model after training, characterized in that The following steps are involved: S10: Determine whether it is a model for classification tasks, if not, proceed to step S20; if yes, 50%≤sparse rate<80% is satisfied, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm, and the reconstruction technology adopts the technology of error correction and block-by-block reconstruction; S20: Determine whether it is a model for the detection task, if not, proceed to step S30; if yes, for the model for the detection task, 50%≤sparse rate<80% is satisfied, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm, and the reconstruction technology adopts the technology of error-free correction and block-by-block reconstruction; S30: For the model used for generating the task, the sparsity rate is set to ≤50%, and the sparse allocation method is selected in the priority order of FCPTS>L2Norm>ERK, and the reconstruction technology adopts the technology without error correction and block-by-block reconstruction, or the technology with error correction and block-by-block reconstruction.

3. A method for optimizing a sparse model after training, characterized in that The following steps are involved: S01: Determine the model size. If the model size is large, choose the Transformer-based model architecture. If the model is of medium size, choose a CNN-based model architecture; S02: Determine the task type. If it is a classification task, proceed to step S03; if it is a detection task, proceed to step S04; if it is a generation task, proceed to step S05; S03: For the model used for classification tasks, the sparsity rate satisfies 50% ≤ 80%, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm. The reconstruction technology uses the technology with error correction and block-by-block reconstruction; S04: For the model used for the detection task, the sparsity rate satisfies 50%≤50%<80%, and the sparse allocation method is selected in the order of FCPTS>ERK>L2Norm. The reconstruction technology adopts the technology of error-free correction and block-by-block reconstruction. S05: For the model used for generating the task, the sparsity rate is set to ≤50%, and the sparse allocation method is selected in the priority order of FCPTS>L2Norm>ERK, and the reconstruction technology adopts the technology without error correction and block-by-block reconstruction, or the technology with error correction and block-by-block reconstruction.

4. The method for optimizing a trained sparse model according to any one of claims 1 to 3, characterized in that: When the sparse allocation method selects FCPTS, the FCPTS includes FCPTS(ERK) and FCPTS(L2Norm), and FCPTS(ERK) takes precedence over FCPTS(L2Norm).

5. The method for optimizing a trained sparse model according to any one of claims 1 to 3, characterized in that: In the sparse allocation method, the sparse rate of the last layer is ≤50%.

6. The method for optimizing a trained sparse model according to any one of claims 1 to 3, characterized in that: In the sparse allocation method, the sparse rate of the downsampling layer is ≤50%.

7. The method for optimizing a trained sparse model according to claim 3, wherein: The Transformer-based model architecture is the ViT model architecture.

8. The method for optimizing a trained sparse model according to claim 3, wherein: The CNN-based model architecture is the MobileNetV3 model architecture.

9. A system for optimizing a trained sparse model, comprising a processor and a memory, wherein the processor and the memory are coupled; wherein: The memory is used to store computer programs; the processor is used to run the computer programs stored in the memory to execute the optimization method of the trained sparse model as described in any one of claims 1 to 8.

Images