Information processing method, program and information processing device

Abstract

To provide an information processing method, a program, and an information processing device that can ensure processing performance while compressing a model size of a pre-trained learning model to a prescribed ratio.SOLUTION: A method calculates an evaluation value of pre-trained parameters before a prescribed learning model that has been pre-trained using prescribed learning data to be re-trained in a downstream task. The evaluation value of the parameters is a combined evaluation value obtained by adding together a magnitude-based evaluation value and a gradient-based evaluation value. Then, the method prunes the pre-trained parameters in ascending order of the combined evaluation value until a model size of the learning model reaches a prescribed ratio.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an information processing method, a program, and an information processing device for compressing the model size of a large-scale learning model that requires pre-learning. [Background technology]

[0002] Conventionally, the Transformer model is known as a deep learning model. The Transformer model performs large-scale pre-learning to achieve high performance in various tasks such as image classification, object detection, segmentation, and language processing. One type of Transformer model is the Vision Transformer (hereinafter referred to as "ViT") applied to an image recognition model (for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, International Conference on Learning Representations (2021). Summary of the Invention [Problem to be solved by the invention]

[0004] The Transformer model applied to image classification, etc., calculates the correlation between data using inner products, so the inductive bias is weaker than that of CNN (Convolutional Neural Network), which assumes the localization of information. Therefore, in order for the Transformer model to exceed the performance of CNN, it requires many parameters and a large amount of training data. In addition, when using the Transformer model in the field of computer vision, it is often pre-trained using a large-scale dataset such as ImageNet, and then re-trained for various downstream tasks.

[0005] However, while Transformer models with a large number of parameters have high generalization performance, they are known to contain many redundant parameters that do not contribute to accuracy, consuming more computational resources than necessary. Therefore, in order to deploy Transformer models on devices with limited computational resources or in environments where high inference speed is required, it is important to prune redundant parameters that do not contribute to accuracy.

[0006] In view of the above, the present disclosure aims to provide an information processing method, a program, and an information processing device that are capable of compressing the model size of a learning model that requires large-scale pre-learning with minimal performance degradation before re-learning. [Means for solving the problem]

[0007] According to one aspect of the present disclosure, an information processing method, a program, and an information processing device compress the model size of a predetermined learning model that uses a neural network that has been pre-trained using predetermined training data by pruning some of the parameters after pre-training.

[0008] The information processing method includes performing pruning in a single shot before re-learning in a downstream task, and calculating a combined evaluation value by adding together a magnitude-based evaluation value and a gradient-based evaluation value for the parameters before pruning, in which pruning is performed on parameters in ascending order of combined evaluation value until the combined evaluation value reaches a predetermined percentage of the model size.

[0009] The program includes a pruning process that performs pruning in a single shot before re-learning in a downstream task, and a calculation process that calculates a combined evaluation value by adding together a magnitude-based evaluation value and a gradient-based evaluation value for a parameter before pruning, and in the pruning process, pruning is performed on parameters in ascending order of evaluation value until the evaluation value reaches a predetermined percentage of the model size.

[0010] The information processing device includes a memory (4) that records a program (42) used for compressing the model size of a learned model to a predetermined ratio by pruning a portion of parameters after learning, and a processing unit (5) that reads and executes the program, the program including a pruning process that performs pruning in a single shot before re-learning in a downstream task, and a calculation process that calculates a combined evaluation value by adding up a magnitude-based evaluation value and a gradient-based evaluation value for the parameters before pruning, and in the pruning process, pruning is performed on parameters in ascending order of evaluation value until the model size reaches a predetermined ratio.

[0011] The information processing method, the program, and the information processing device calculate a combined evaluation value by combining a magnitude-based evaluation value and a gradient-based evaluation value for pre-trained parameters before re-learning a downstream task, and perform single-shot pruning in ascending order of combined evaluation value. This makes it possible to compress the model size of a learning model that requires large-scale pre-training before re-learning while minimizing performance degradation.

[0012] The reference symbols in parentheses attached to each component indicate an example of the correspondence between the component and specific components described in the embodiments described below. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing an example of the configuration of an information processing device according to an embodiment; [Diagram 2] FIG. 1 is an explanatory diagram of a learning model using a neural network and its pruning. [Diagram 3] 1 is a flowchart showing a procedure for pruning a large-scale learning model. [Figure 4] FIG. 11 is an explanatory diagram of a pruning algorithm that takes pre-learning into account according to the embodiment. [Diagram 5] This graph shows the change in parameters of the ViT-B / 16 learning model pre-trained on ImageNet-21k before and after re-training with CIFAR-10 as the downstream task. [Figure 6] This graph shows the change in parameters of the Mixer-B / 16 learning model pre-trained on ImageNet-21k before and after re-training with CIFAR-10 as the downstream task. [Figure 7] This graph shows the change in parameters of the ViT-B / 16 learning model, which was pre-trained on ImageNet-1k, before and after re-training with CIFAR-10 as the downstream task. [Figure 8] This graph shows the change in parameters of the Mixer-B / 16 learning model pre-trained on ImageNet-1k before and after re-training with CIFAR-10 as the downstream task. [Figure 9] This is a graph showing the change in parameters before and after training the ViT-B / 16 learning model without pre-training using CIFAR-10 as a downstream task. [Figure 10]This is a graph showing the change in parameters before and after training the Mixer-B / 16 learning model without pre-training using CIFAR-10 as a downstream task. [Figure 11] 11 is a graph showing the relationship between parameters after pre-learning and the amount of change in the parameters due to relearning. [Figure 12] FIG. 13 is a diagram showing a comparison result of classification accuracy when pruning according to the embodiment is performed and when random pruning is performed for various Transformer models. [Figure 13] FIG. 13 is a diagram showing a comparison result of classification accuracy after re-learning with CIFAR-10 for various Transformer models in the case where 98% pruning of the embodiment is performed and the case where 98% pruning of the comparative example is performed. [Figure 14] FIG. 13 is a diagram showing a comparison result of classification accuracy after re-learning on CIFAR-100 for various Transformer models in the case where 98% pruning of the embodiment is performed and the case where 98% pruning of the comparative example is performed. [Figure 15] FIG. 13 is a diagram showing a comparison result of classification accuracy after re-learning with ImageNet-1k for two Transformer models in a case where 98% pruning of the embodiment is performed and a case where 98% pruning of the comparative example is performed. [Figure 16] 13 is a graph showing the relationship between pruning rate and classification accuracy for a pre-trained Transformer model before pruning, after pruning in the embodiment, and after pruning in the comparative example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following embodiments, parts that are the same or equivalent to each other will be denoted by the same reference numerals.

[0015] (Embodiment) An information processing device 1 according to an embodiment will be described.

[0016] As shown in FIG. 1, the information processing device 1 according to the embodiment includes an operation device 2, a display device 3, a memory 4, and a processing unit 5. The information processing device 1 uses a trained learning model 41 recorded in the memory 4, for example, to realize various processes, such as image recognition, language processing, object detection, and segmentation, in accordance with the learning content, that is, downstream tasks. The information processing device 1 can build a learning model 41 with a compressed model size by deleting parameters with low importance in downstream tasks from a large number of parameters used in the pre-trained learning model using a program 42 described later. Details of this will be described later.

[0017] The operation device 2 is, for example, a device that receives an operation by a person and outputs a signal according to the received operation to the processing unit 5. The operation device 2 can be any input device, such as, for example, a mouse, a keyboard, or a touch panel.

[0018] The display device 3 is, for example, a device that displays images. The display device 3 displays, for example, images related to results of various downstream tasks using the learning model 41 or operations therefor.

[0019] The memory 4 is composed of, for example, a RAM which is a rewritable volatile storage medium, a ROM which is a non-rewritable non-volatile storage medium, a flash memory which is a rewritable non-volatile storage medium, etc. The RAM, ROM, and flash memory are non-transient substantial storage media. The RAM and ROM are abbreviations for Random Access Memory and Read Only Memory, respectively. For example, data of a trained learning model 41 and a program 42 for compressing the model size of the learning model 41 are recorded in advance in the memory 4.

[0020] The processing unit 5 executes the learning model 41 and the program 42 stored in the memory 4, and uses the RAM as a working area during the execution, thereby compressing the model size of the learning model 41 and implementing various processes.

[0021] Here, the learning model 41 and the program 42 will be described.

[0022] For example, as shown in Fig. 2, the learning model 41 is a machine learning model using a neural network having a large number of nodes N, in which layers are formed by the nodes N arranged vertically, and the nodes N in different layers are connected to each other by edges E. The learning model 41 is, for example, a Transformer model. Note that in Fig. 2, in order to make the learning model 41 easy to understand, a model that has been deformed to reduce the number of nodes N and edges E is shown, and the pre-trained learning model 41 has a larger number of nodes N and edges E than those shown in the figure.

[0023] For example, when the learning model 41 is an image recognition model such as ViT, the learning model 41 is constructed by pre-training with a large-scale data set such as JFT-300M with approximately 300 million images or ImageNet-21k with more than 14 million images, and re-training with a small-scale data set. Also, when the learning model 41 is a large-scale language model such as GPT-3, the learning model 41 is constructed by pre-training by mixing various text data such as Common Crawl, Web Text, and Wikipedia. Note that GPT is an abbreviation for Generative Pre-trained Transformer. The learning model 41 is, for example, a Transformer model pre-trained using the above-mentioned large-scale data set, and pruned by executing the program 42 to a predetermined model size. Note that the large-scale data set is, for example, but not limited to, one having more than 10 million pieces of learning data.

[0024] Here, a Transformer model that has undergone large-scale pre-training has high generalization performance, but has a very large number of parameters, including many redundant parameters that do not contribute to accuracy, and consumes more computational resources than necessary. For example, if the Transformer model is ViT-H, the number of parameters is 630 million, and the capacity required to hold these parameters is 2.35 GB (gigabytes). Also, if the Transformer model is GPT-3, the number of parameters is 175 billion, and the capacity required to hold these parameters is 558.8 GB. For this reason, a Transformer model that has undergone large-scale pre-training requires compression of the model size in order to deploy it on a terminal with limited computational resources. Note that the capacity required to hold the above parameters is calculated at 4 bytes per parameter, since capacity is also required to hold the structure of the learning model.

[0025] By executing the program 42, the information processing device 1 executes pruning to delete a part of the nodes N and edges E of the pre-trained Transformer model as shown in Fig. 2, for example, and constructs a learning model 41 in which the initial model size is compressed to a predetermined ratio. This enables the information processing device 1 to deploy the learning model 41, which is a Transformer model, on a terminal with limited computational resources or in an environment requiring high inference speed, while minimizing performance degradation. Pruning is also called pruning.

[0026] The program 42 is configured to execute a calculation process for calculating an evaluation value for each parameter of the pre-trained Transformer model by a method described later, and a pruning procedure for pruning based on the calculated evaluation value. In the pruning process, the program 42 is configured to execute pruning in a single shot, in which pre-trained parameters are deleted in ascending order of calculated evaluation value until the model size reaches a predetermined ratio. The predetermined ratio here is a target value (%) of the model size after pruning, with the model size of the pre-trained Transformer model before pruning being 100%, and can be set to any value depending on, for example, the computational resources of a terminal on which the Transformer model is to be deployed.

[0027] The above is the basic configuration of the information processing device 1. Note that the information processing device 1 may be configured as a computer dedicated to pruning that executes the program 42 in the processing unit 5 and mainly performs compression of the model size of the Transformer model recorded in the memory 4 or recorded in an external recording medium.

[0028] [Pruning Algorithm] Next, the learning model pruning algorithm used by the information processing device 1 will be described.

[0029] A learning model 41 optimized for downstream tasks is constructed, for example, through the process shown in Fig. 3. In step S110, for example, an initialized network is created using a known method. Then, in step S120, for example, a large-scale dataset such as JFT-300M or ImageNet-21k is used to pre-train the Transformer model created in step S110.

[0030] Next, in step S130, for example, the information processing device 1 reads and executes the program 42, and calculates an evaluation value (described later) for each parameter of the Transformer model pre-trained in step S120. Then, in step S140, for example, the information processing device 1 performs pruning so that the model size of the Transformer model becomes a predetermined ratio, in ascending order of the evaluation value calculated in step S130 among the pre-trained parameters. Note that the processing of steps S130 and S140 is performed before re-learning in the downstream task.

[0031] Finally, in step S150, the Transformer model pruned in step S140 is optimized for downstream tasks.

[0032] Through the above process, it is possible to obtain a learning model 41 in which the model size is compressed to a predetermined target value while minimizing performance degradation.

[0033] The pruning taking into account the pre-learning in steps S130 to S140 is performed according to the algorithm shown in FIG. 4, for example.

[0034] For example, in the first stage, a process is performed to randomly select data for calculating the evaluation value in the third stage from the set of training data D. In the second stage, for each pre-trained parameter θ, q The process of calculating the value of θ q is the qth (q: integer greater than or equal to 2) parameter included in the network of the learning model.

[0035] In the third stage, the parameter θ q The evaluation value of the change in parameters when perturbation is applied and the parameter θ q In other words, in the third stage, the parameter θ qIn the fourth stage, the process of the third stage is performed for all pre-trained parameters θ.

[0036] In the fifth step, a threshold value corresponding to a pruning rate k (%), which is the desired compression rate, is calculated based on the score calculated in the fourth step. The pruning rate k is the percentage of the model size to be deleted by pruning, with the model size of the learning model before pruning being 100%. For example, when the pruning rate k is set to 90%, the model size of the resulting learning model 41 is about 10% of the model size before pruning.

[0037] In the sixth step, a mask is created to remove parameters that are below the threshold calculated in the fifth step. In the seventh and final step, the mask created in the sixth step is used to remove parameters that are below the threshold calculated in the fifth step, i.e., pruning is performed.

[0038] [Pre-learning and parameter changes] We investigated how parameters change before and after re-training when optimizing a Transformer model for a downstream task, depending on whether or not pre-training was performed, and obtained the results shown in Figures 5 to 11. Specifically, we used ViT-B / 16 and Mixer-B / 16 pre-trained with ImageNet-1k or ImageNet-21k for the training models, and compared the parameters when re-trained with the image recognition task CIFAR-10 for the downstream task.

[0039] In addition, the horizontal axis of the graphs shown in Figs. 5 to 8 is the parameter after pre-learning of the learning model, and the vertical axis is the parameter after re-learning by CIFAR-10. In the graphs shown in Figs. 9 and 10, the horizontal axis is the initial parameter of the learning model, and the vertical axis is the parameter after learning by CIFAR-10. In addition, in Figs. 5 to 10, in order to easily understand the degree of change in the parameter, the dashed line shows a case in which it is assumed that the parameter does not change before and after learning by CIFAR-10. In the graph shown in Fig. 11, the horizontal axis is the parameter after pre-learning of the learning model, and the vertical axis is the amount of change in the parameter due to re-learning by CIFAR-10.

[0040] For ease of explanation, we will refer to ViT-B / 16 pre-trained with ImageNet-1k as "ViT-B / 16 ImageNet-1k", ViT-B / 16 pre-trained with ImageNet-21k as "ViT-B / 16 ImageNet-21k", and ViT-B / 16 without pre-training as "ViT-B / 16 Scratch". Also, we will refer to Mixer-B / 16 pre-trained with ImageNet-1k as "Mixer-B / 16 ImageNet-1k", Mixer-B / 16 pre-trained with ImageNet-21k as "Mixer-B / 16 ImageNet-21k", and Mixer-B / 16 without pre-training as "Mixer-B / 16 Scratch".

[0041] The batch size for training CIFAR-10 was set to 128, the initial learning rate was set to 0.0001, and optimization using Adam was performed for 200 epochs. Cross entropy was used as the error function.

[0042] For example, as shown in Figures 5 and 6, when pre-training was performed using a large dataset such as ViT-B / 16 ImageNet-21k and Mixer-B / 16 ImageNet-21k, the parameter values ​​did not change much before and after re-training. On the other hand, as shown in Figures 7 and 8, in ViT-B / 16 ImageNet-1k and Mixer-B / 16 ImageNet-1k, which have smaller pre-training datasets than ImageNet-21k, the parameter values ​​changed irregularly before and after re-training. Furthermore, as shown in Figures 9 and 10, in ViT-B / 16 Scratch and Mixer-B / 16 Scratch without pre-training, the parameter values ​​changed the most before and after training in the downstream task. In other words, the change in the parameter values ​​of the learning model decreased before and after re-training in proportion to the size of the dataset used for pre-training. These results imply that the knowledge gained by pre-training on ImageNet-21k is fully utilized in downstream tasks such as CIFAR-10.

[0043] In addition, as shown in Fig. 11, the amount of change in a parameter before and after re-learning tends to be larger as the parameter value after pre-learning and before re-learning approaches 0, and smaller as the parameter value increases. Therefore, among parameters, parameters with a large scale that hardly changes before and after re-learning are considered to be useful knowledge obtained by pre-learning. In addition, among parameters, parameters with values ​​near 0 that change more before and after re-learning than others are considered to be new knowledge obtained from downstream tasks.

[0044] In view of the above results, the pruning method according to the embodiment predicts parameters with high importance before relearning to downstream tasks, and deletes parameters in order of least importance, in other words, performs non-structural pruning to compress the model size of a learning model.

[0045] [Evaluation of pruning parameters] Next, parameter evaluation in pruning according to the embodiment will be described, but first, a conventional pruning method will be described.

[0046] As a conventional neural network pruning method, for example, magnitude pruning (hereinafter simply referred to as "magnitude") is known. Magnitude is a pruning method in which post-learning parameters are deleted in order of smallest absolute value. The conventional magnitude-based method, which performs pruning using the magnitude of parameters, compresses the model size of the learning model by repeating learning and pruning.

[0047] Pruning of Transformer models is mainly performed by deleting heads of multi-head attention. For example, a method has been proposed to add gating parameters to the heads included in multi-head attention and regularize them using L0 loss to delete redundant heads. A method has also been proposed to evaluate each head based on attention confidence to achieve pruning. However, these pruning methods are limited to multi-head attention, so they cannot compress the parameters of multi-layer perceptron (MLP). In addition, these pruning methods cannot be applied to Transformer models that do not have multi-head attention, such as MLP-Mixer and PoolFormer.

[0048] As a conventional pruning method that does not require learning, for example, Single-Shot Network Pruning (SNIP) is known. SNIP is a method for quantitatively evaluating the extent to which the error L of the learning model network is affected when a perturbation δ is given to the parameters, as shown in the following equation (1).

[0049]

number

[0050] As an example of a method derived from SNIP, there is Gradient Signal Preservation (GraSP). GraSP is a method that defines the relationship between the parameters of a learning model more strictly by taking into account the Hessian, as shown in the following equation (2).

[0051]

number

[0052] Next, evaluation of parameters in the pruning method according to the embodiment will be described.

[0053] This evaluation is based on the first evaluation value S, which is based on a method for evaluating parameters that are prone to change near 0. a , and a second evaluation value S based on a method for evaluating parameters that are almost invariant over a large scale. b The first evaluation value S is calculated by adding these two values ​​together. a is a gradient-based method and is calculated using the following equation (3).

[0054]

number

[0055] However, when sufficient pre-training is performed using a large data set, or when a network with a large number of dimensions is used, the parameter gradient becomes extremely small. When such parameters are almost unchanged, the gradient-based first evaluation value S a Therefore, the information processing device 1 uses the second evaluation value S b Calculate.

[0056] The second evaluation value S b is a magnitude-based method and is calculated using the following equation (4).

[0057]

number

[0058] Therefore, the information processing device 1 calculates the first evaluation value S a and the second evaluation value S b A combined evaluation value S is calculated based on the above.

[0059]

number

[0060] [Effect of pruning according to the embodiment] Next, the effect of pruning according to the embodiment will be described.

[0061] Evaluation experiments were performed using various Transformer models, and the results shown in Figures 12 to 16 were obtained. In Figure 12, the results of classification accuracy when random pruning was performed are shown in parentheses, and the results of classification accuracy when pruning according to the embodiment was performed are shown outside the parentheses.

[0062] In this evaluation experiment, ViT-B / 16, ViT-L / 16, Mixer-B / 16, Mixer-L / 16, Pool-M36 and Pool-M48 were used as Transformer models, and ImageNet-21k was used for pre-learning. In addition, CIFAR-10, CIFAR-100 and ImageNet-1k were used as downstream tasks, and the pruning rate for each Transformer model was set to 90%, 95% and 98%, and the classification accuracy was compared between the case where the pruning of the embodiment was performed and the case where random pruning was performed. In addition, the batch size when learning CIFAR-10 and CIFAR-100 was set to 128, and the batch size when learning ImageNet-1k was set to 512. In addition, the hyperparameter α used in the pruning method of the embodiment was set to 0.001.

[0063] Furthermore, all token mixers and channel mixers were pruned. However, for the PoolFormer learning model, only the channel mixers were pruned because the token mixers are nonparametric. Pruning was performed unstructured and in a single shot. The pruned network was optimized using Adam with an initial learning rate of 0.0001. The number of training epochs was 200 for CIFAR-10, 250 for CIFAR-100, and 200 for ImageNet-1k. Cross entropy was used as the error function.

[0064] As shown in Fig. 12, the pruned model of the embodiment also showed improved classification accuracy for the evaluation data when re-trained using CIFAR-10 and CIFAR-100, compared to the randomly pruned model, in each of the above-mentioned Transformer models. For ease of explanation, hereinafter, the classification accuracy for the evaluation data when re-trained using CIFAR-10 or CIFAR-100 will be simply referred to as "classification accuracy with CIFAR-10" and "classification accuracy with CIFAR-100".

[0065] For example, ViT-B / 16 randomly pruned at pruning rates of 90%, 95%, and 98% achieved classification accuracies of 85.70%, 83.45%, and 81.95% on CIFAR-10, and 61.24%, 57.22%, and 55.59% on CIFAR-100, respectively.

[0066] On the other hand, the pruned ViT-B / 16 of the embodiment with pruning rates of 90%, 95%, and 98% had classification accuracies of 95.92%, 94.37%, and 90.92% in CIFAR-10, and 81.09%, 77.05%, and 68.35% in CIFAR-100, respectively. Thus, the classification accuracies of the pruned ViT-B / 16 of the embodiment were all improved in CIFAR-10 and CIFAR-100 at all pruning rates compared to the case of random pruning.

[0067] In addition, for all of the models ViT-L / 16, Mixer-B / 16, Mixer-L / 16, Pool-M36, and Pool-M48, the classification accuracy for CIFAR-10 and CIFAR-100 was improved at all pruning rates when the pruning of the embodiment was performed, compared to when the pruning was performed randomly. Furthermore, for all of the Transformer models pruned in the embodiment, the classification accuracy exceeded 90% at pruning rates of 90% and 95%, and even at a pruning rate of 98%, the classification accuracy exceeded 90% except for Mixer-B / 16 and Mixer-L / 16.

[0068] The learning model of PoolFormer tended to have higher classification accuracy than the learning models of ViT and MLP-Mixer. This is thought to be because, while the token mixers of ViT and MLP-Mixer lose most of the features between patches due to pruning, the token mixer of PoolFormer is not affected by pruning and can stably mix information between patches using only the pooling layer.

[0069] Next, the classification accuracy in CIFAR-10 or CIFAR-100 was compared for the model pruned in the embodiment with a pruning rate of 98% and the model pruned in the comparative example, and the results shown in Figures 13 and 14 were obtained.

[0070] The pruning methods in the comparative example were the conventional methods Magnitude, SNIP, and GraSP. The hyperparameter α in the pruning methods in the examples was all set to 0.001.

[0071] As shown in FIG. 13, the classification accuracy of ViT-B / 16 in CIFAR-10 was 83.38% for magnitude, 88.50% for SNIP, and 85.70% for GraSP, while the classification accuracy of ViT-B / 16 in the Example was 90.92%, which was higher than the Comparative Example. ViT-L / 16, Mixer-L / 16, Pool-M36, and Pool-M48 had higher classification accuracy in CIFAR-10 than magnitude, SNIP, and GraSP, as with ViT-B / 16. Mixer-B / 16 had classification accuracy in CIFAR-10 of 84.20% for magnitude, 89.57% for SNIP, and 84.84% for GraSP, while the classification accuracy of Mixer-B / 16 in CIFAR-10 was 87.05% in the Example, which was higher than magnitude and GraSP.

[0072] In addition, as shown in FIG. 14, the classification accuracy of ViT-B / 16 in CIFAR-100 was 57.99% for magnitude, 63.82% for SNIP, and 60.71% for GraSP, while the classification accuracy of ViT-B / 16 in the Example was 68.35%, which was higher than the Comparative Example. ViT-L / 16, Mixer-L / 16, Pool-M36, and Pool-M48 had higher classification accuracy in CIFAR-100 than the magnitude, SNIP, and GraSP, as with ViT-B / 16. Mixer-B / 16 had classification accuracy in CIFAR-10 of 62.44% for magnitude, 67.31% for SNIP, and 61.07% for GraSP, while the classification accuracy of Mixer-B / 16 in CIFAR-10 was 63.26% in the Example, which was higher than the magnitude and GraSP.

[0073] Next, we compared the classification accuracy of the model pruned in the example with a pruning rate of 98% and the model pruned in the comparative example on the evaluation data when re-trained using ImageNet-1k (hereinafter referred to as "classification accuracy with ImageNet-1k").

[0074] As shown in Fig. 15, the classification accuracy of ViT-B / 16 on ImageNet-1k was 78.62% for magnitude, 79.07% for SNIP, and 77.23% for GraSP, while the classification accuracy of the Example was 79.02%, which was higher than that of the Comparative Example. The classification accuracy of ViT-L / 16 on ImageNet-1k was 80.60% for magnitude, 80.61% for SNIP, and 78.53% for GraSP, while the classification accuracy of the Example was 80.79%, which was higher than that of the Comparative Example.

[0075] The results shown in Figures 13 to 15 indicate that the pruning method of the embodiment can obtain higher classification accuracy than the comparative example, and is also effective for datasets of various sizes, from CIFAR-10 to ImageNet-1k.

[0076] In addition, the relationship between the pruning rate and the classification accuracy for each ViT-L / 16 model before pruning, after pruning in the embodiment, and after pruning in the comparative example was compared, and the results shown in Figure 16 were obtained. Figure 16 shows the results obtained when pre-training was performed using ImageNet-21k, the number of parameters before pruning was 310 million, the capacity required to hold this number of parameters was 1.15 GB, and the downstream task was CIFAR-10.

[0077] In the embodiment, the classification accuracy was higher than the comparative examples of Magnitude, SNIP, and GraSP at pruning rates of 90%, 95%, and 98%, and the classification accuracy was 94% or higher. In addition, since the classification accuracy of the learning model before pruning was about 98%, the results shown in Fig. 16 indicate that the pruning method of the embodiment most effectively suppresses the decrease in classification accuracy before and after pruning compared to the conventional pruning method, and can maintain high processing performance while reducing the model size by 90% or more.

[0078] In the above, the pruning of the Transformer model that was pre-trained using a large-scale image data set and used as an image recognition model was described, but even when it is constructed as another model such as a language model, it is expected that both performance assurance and model size compression can be achieved. For example, when the image recognition model ViT-H is compressed by 98% by pruning in the embodiment, the capacity required to hold the parameters is reduced from 2.35 GB to 47 MB ​​(megabytes) while minimizing performance degradation. In addition, when the language model GPT-3 is compressed by 98% by pruning in the embodiment, the capacity required to hold the parameters is reduced from 558.8 GB to 11.2 GB.

[0079] According to the present embodiment, the information processing device 1, the program 42, and the pruning method calculate evaluation values ​​of parameters of a Transformer model pre-trained with a large-scale data set before re-learning in a downstream task, and perform pruning at a predetermined pruning rate k in ascending order of the evaluation values. This makes it possible to build a learning model 41 that is compressed to a model size that can be deployed on a terminal with limited computational resources while suppressing performance degradation.

[0080] In addition, since pruning is performed in a single shot, repeated training and pruning is not required to compress the model size. Furthermore, since pruning is performed before re-learning in downstream tasks, the computational cost of re-learning the pruned learning model is reduced, and the inference time is also accelerated.

[0081] In addition, the system calculates a score based on an evaluation index of parameters that take pre-learning into account, and performs parameter evaluation and pruning, making it possible to apply the system to a variety of models regardless of the network structure of the learning model.

[0082] In the above, the case where the learning model is optimized as an image recognition model or a language model has been described, but the present invention is not limited thereto, and can of course be applied to other uses. In addition, the pruning method according to the embodiment is suitable for application to compression of a learning model pre-trained with a large-scale data set, but even if it is applied to a learning model pre-trained with a medium-sized or smaller data set, the effect of suppressing performance degradation and compressing size can be achieved at the same time.

[0083] (Other embodiments) In the above embodiment, it goes without saying that the elements constituting the embodiment are not necessarily essential, except when they are specifically stated as essential or when they are clearly considered essential in principle. In addition, in the above embodiment, when the numbers, values, amounts, ranges, etc. of the components of the embodiment are mentioned, they are not limited to the specific numbers, except when they are specifically stated as essential or when they are clearly limited to a specific number in principle. In addition, in the above embodiment, when the shapes, positional relationships, etc. of the components are mentioned, they are not limited to the shapes, positional relationships, etc., except when they are specifically stated or when they are limited to a specific shape, positional relationship, etc. in principle.

[0084] Furthermore, the information processing device 1 and the pruning method of the above embodiment have been described based on the examples, but are not limited to the examples and configurations, and may be modified within the possible range. [Explanation of symbols]

[0085] 4. Memory 42 Programs 5 Processing section

Claims

1. 1. An information processing method for compressing a model size of a predetermined learning model using a neural network that has been pre-trained using predetermined learning data by pruning a portion of parameters after pre-training, the method comprising: performing the pruning in a single shot before re-learning on downstream tasks; calculating a combined evaluation value by combining a magnitude-based evaluation value and a gradient-based evaluation value for the parameter before pruning; In the pruning, pruning is performed in ascending order of the combined evaluation value among the parameters until the combined evaluation value reaches the predetermined ratio of the model size.

2. 2. The information processing method according to claim 1, wherein in calculating the combined evaluation value, a magnitude-based evaluation value is calculated for each of the parameters after pre-learning and before re-learning using the magnitude of the parameters, an amount of error of the parameters with respect to a downstream task is calculated for each of the parameters after pre-learning as the gradient-based evaluation value, and a value obtained by multiplying the magnitude-based evaluation value by a hyperparameter is added to the gradient-based evaluation value.

3. A program used to compress a model size of a predetermined learning model using a neural network that has been pre-trained using predetermined learning data by pruning a part of parameters after learning, the program comprising: a pruning process for performing the pruning in a single shot before re-learning in a downstream task; a calculation process of calculating a combined evaluation value by combining a magnitude-based evaluation value and a gradient-based evaluation value for the parameter before performing the pruning, the pruning process pruning the parameters in ascending order of evaluation value until the evaluation value reaches the predetermined ratio of the model size.

4. 4. The program according to claim 3, wherein in calculating the combined evaluation value, a magnitude-based evaluation value is calculated for each of the parameters after pre-learning and before re-learning using the magnitude of the parameters, an amount of error of the parameters with respect to a downstream task is calculated for each of the parameters after pre-learning as the gradient-based evaluation value, and a value obtained by multiplying the magnitude-based evaluation value by a hyperparameter is added to the gradient-based evaluation value.

5. An information processing device, A memory (4) having recorded therein a program (42) used for compressing the model size of a predetermined learning model using a neural network that has been pre-trained using predetermined learning data by pruning a portion of parameters after learning; A processing unit (5) that reads and executes the program, The program includes a pruning process that performs the pruning in a single shot before re-learning in a downstream task, and a calculation process that calculates a combined evaluation value by adding together a magnitude-based evaluation value and a gradient-based evaluation value for the parameters before performing the pruning, and in the pruning process, pruning is performed on the parameters in ascending order of evaluation value until the predetermined percentage of the model size is reached.

6. 6. The information processing device according to claim 5, wherein in calculating the combined evaluation value, a magnitude-based evaluation value is calculated for each of the parameters after pre-learning and before re-learning using a magnitude of the parameter, an amount of error of the parameter with respect to a downstream task is calculated for each of the parameters after pre-learning as the gradient-based evaluation value, and a value obtained by multiplying the magnitude-based evaluation value by a hyperparameter is added to the gradient-based evaluation value.

Images