Machine learning program, machine learning method, and information processing device

Abstract

To realize downsizing of a neural network including multiple layers.SOLUTION: A machine learning program for causing a computer to execute a process including: calculating thresholds of errors in tensors between before and after reduction for each element of a plurality of layers in a trained model of a neural network including the layers; selecting reduction ratio candidates to be applied to each of the layers on the basis of, the thresholds and errors in tensors between before and after reduction in cases where the elements are reduced by each of reduction ratio candidates in each of the layers; and determining reduction ratios to be applied to each of the layers on the basis of, inference accuracy of the trained model and inference accuracy of a reduced model after machine learning, the reduced model being obtained by reducing each element of the layers in the trained model according to the reduction ratio candidates to be applied.SELECTED DRAWING: Figure 16

Description

[Technical field]

[0001] The present invention relates to a machine learning program, a machine learning method, and an information processing device. [Background technology]

[0002] Neural networks (NNs) used in AI (Artificial Intelligence) tasks such as image processing tend to achieve high performance (e.g., high inference accuracy) by making their configurations more complex. On the other hand, the complexity of the NN configuration can increase the number of calculations required to execute the NN on a computer and the memory size used by the computer to execute the NN.

[0003] "Pruning" is known as a method for reducing the number of calculations, in other words, shortening (speeding up) the calculation time, and for reducing the memory size, in other words, making the NN machine learning model more lightweight.

[0004] Pruning is a technique for reducing the data size of a machine learning model and reducing computation time and communication time by reducing (trimming) at least one type of element: edges (weights), nodes, and channels of a NN.

[0005] Excessive pruning can cause a degradation of the inference accuracy of the NN. Therefore, it is important to prune the NN while maintaining the inference accuracy or while keeping the degradation of the inference accuracy at a predetermined level.

[0006] For example, a method is known for selecting layers that do not significantly affect the inference accuracy of a neural network in pruning. This method determines the channels of a convolutional layer to be pruned based on parameters used in a batch normalization (BN) layer following the convolutional layer. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] U.S. Patent Publication No. 2019 / 0057308 [Patent Document 2] U.S. Patent Publication No. 2018 / 0232640 [Patent Document 3] Patent Publication No. 2021-47854 [Patent Document 4] JP 2020-123269 A [Patent Document 5] Patent Publication No. 2021-22050 [Patent Document 6] U.S. Patent Publication No. 2019 / 0080238 [Patent Document 7] U.S. Patent Publication No. 2019 / 0205759 [Patent Document 8] U.S. Patent Publication No. 2020 / 0184333 Summary of the Invention [Problem to be solved by the invention]

[0008] The method of selecting layers that do not significantly affect the inference accuracy of the NN is applied to convolutional layers connected to BN layers, but is not intended to be applied to other layers, such as convolutional layers not connected to BN layers and fully connected layers.

[0009] In one aspect, an object of the present invention is to realize a lightweight neural network including multiple layers. [Means for solving the problem]

[0010] In one aspect, the machine learning program may cause a computer to execute the following process. The process may include a process of calculating a threshold value of a tensor error before and after reduction of elements of each of a plurality of layers in a machine-learned model of a neural network including a plurality of layers. The process may also include a process of selecting a reduction rate candidate to be applied to each of the plurality of layers based on a plurality of the threshold values ​​and a tensor error before and after reduction when the elements are reduced in each of the plurality of layers by each of a plurality of reduction rate candidates. Furthermore, the process may include a process of determining a reduction rate to be applied to each of the plurality of layers based on an inference accuracy by the machine-learned model and an inference accuracy after machine learning of a reduced model obtained by reducing elements of each of the plurality of layers in the machine-learned model according to the reduction rate candidate to be applied. Effect of the Invention

[0011] In one aspect, the present invention can realize a lightweight neural network including multiple layers. [Brief description of the drawings]

[0012] [Figure 1] FIG. 13 is a diagram for explaining an example of a process for determining a channel of a convolutional layer on which pruning is performed. [Diagram 2] FIG. 13 is a diagram illustrating an example of L1 regularization learning. [Diagram 3] FIG. 3 is a diagram showing an example of whether the methods of FIGS. 1 and 2 can be applied in a layer of a NN. [Figure 4] FIG. 2 is a block diagram illustrating an example of a functional configuration of a server according to an embodiment. [Diagram 5] FIG. 13 is a diagram illustrating an example of calculation of a pruning rate with guaranteed accuracy. [Figure 6] FIG. 13 is a diagram illustrating an example of calculation of the accuracy of a model before and after pruning. [Figure 7] FIG. 13 is a diagram illustrating an example of searching for a pruning rate. [Figure 8] FIG. 11 is a diagram illustrating an example of a threshold value derivation method. [Figure 9]FIG. 13 is a diagram illustrating an example of an upper limit of a threshold value and a threshold value. [Figure 10] FIG. 11 is a diagram illustrating an example of a method for determining channels to be pruned. [Figure 11] FIG. 13 is a diagram illustrating an example of calculating a pruning error. [Figure 12] FIG. 11 is a diagram illustrating an example of a method for determining nodes to be pruned. [Figure 13] FIG. 13 is a diagram illustrating an example of calculating a pruning error. [Figure 14] FIG. 11 is a diagram illustrating an example of a method for determining weights for pruning. [Figure 15] FIG. 13 is a diagram illustrating an example of calculating a pruning error. [Figure 16] 11 is a flowchart illustrating an example of an operation of a process by a server according to an embodiment. [Figure 17] This is a diagram showing an example of a model including a convolutional layer not connected to a BN layer and a fully connected layer. [Figure 18] FIG. 13 is a diagram showing a capacity compression rate of model data according to a pruning rate determined by a technique according to an embodiment. [Figure 19] FIG. 19 is a diagram showing an example of the data size of a model according to the number of searches in the results shown in FIG. 18. [Figure 20] FIG. 19 is a diagram showing an example of a pruning rate (overall model) according to the number of searches in the result shown in FIG. [Figure 21] FIG. 19 is a diagram illustrating an example of a pruning rate (for each layer) according to the number of searches in the results illustrated in FIG. 18. [Figure 22] 18 is a diagram illustrating an example of the number of channels or the number of nodes for each layer illustrated in FIG. 17, and a pruning rate for each layer. [Figure 23] FIG. 13 is a diagram illustrating the relationship between the number of searches for the pruning rate and the model size. [Figure 24] FIG. 13 illustrates an example of a pruning error comparison result in response to an update of the trust radius in a method according to an embodiment. [Diagram 25]FIG. 13 is a block diagram showing an example of a functional configuration of a server according to a first modified example. [Figure 26] 11A and 11B are diagrams illustrating an example of a trust radius update process in the case where the trust radius is increased; [Figure 27] 13A and 13B are diagrams illustrating an example of a trust radius update process in the case where a trust radius is decreased; [Figure 28] 13 is a flowchart illustrating an example of the operation of a process by a server according to a first modified example. [Figure 29] FIG. 13 is a diagram illustrating the relationship between the number of searches for the pruning rate and the model size. [Diagram 30] FIG. 13 is a diagram illustrating the relationship between the number of searches for the pruning rate and the model size according to the setting of the initial value of the trust radius. [Diagram 31] FIG. 13 is a block diagram showing an example of a functional configuration of a server according to a second modified example. [Diagram 32] FIG. 13 is a diagram illustrating an example of setting an initial value of a trust radius. [Diagram 33] 13 is a flowchart illustrating an example of the operation of a process by a server according to a second modified example. [Diagram 34] FIG. 13 is a diagram illustrating the relationship between the number of searches for the pruning rate and the model size according to the setting of the initial value of the trust radius. [Diagram 35] FIG. 2 is a block diagram showing an example of the hardware (HW) configuration of a computer 10. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the embodiment described below is merely an example, and is not intended to exclude various modifications or application of techniques not specified below. For example, this embodiment can be modified in various ways without departing from the spirit of the invention. In the drawings used in the following description, parts with the same reference numerals represent the same or similar parts unless otherwise specified.

[0014] [1] One embodiment Fig. 1 is a diagram for explaining an example of a process for determining a channel of a convolutional layer for pruning, and Fig. 2 is a diagram showing an example of L1 regularization learning. Fig. 1 explains a method for selecting a layer that does not significantly affect the inference accuracy of a NN, in which a computer determines a channel of a convolutional layer for pruning using a scaling coefficient γ used in a BN layer 100 following the convolutional layer. The graphs shown for channels 111 to 113 in Fig. 1 represent the distribution of output tensors.

[0015] As shown in FIG. 1, the computer performs normalization processing 101 on each of a plurality of channels 111 (#1 to #n; n is an integer equal to or greater than 2) input from the convolution layer to the BN layer 100. For example, in the normalization processing 101, the computer calculates a mean value μ and a variance σ for each channel 111 according to the following formula (1): 2 By calculating z, multiple channels 112 (#1 to #n) that represent a normalized distribution with a mean of "0" and a variance of "1" are obtained. In the following formula (1), z in and z mid indicate channels 111 and 112, respectively, and μ B and σ B 2 denote the mean and variance in the current mini-batch B, respectively.

number

[0016] The computer also executes scaling 102 on the multiple channels 112 (#1 to #n). For example, in the scaling 102, the computer multiplies each of the multiple channels 112 by a scaling coefficient γ according to the following formula (2) and adds a bias β to the multiplication result, thereby outputting multiple channels 113 (#1 to #n) that represent a distribution scaled by parameters γ and β. In the following formula (2), z out indicates the channel 113. Note that the parameters γ and β may be optimized by machine learning.

number

[0017] Here, when γ is small, the output of channel 113 (channel #n in the example of FIG. 1) resulting from scaling 102 is almost zero. This means that even if the channel is deleted by pruning, it does not have a significant effect on the inference accuracy of the NN. Therefore, the computer determines the channels to be pruned on a channel-by-channel basis by searching for a small γ (for example, 0).

[0018] For example, the computer searches for a small (smaller) γ by applying L1 regularization learning to γ. L1 regularization learning is a machine learning method known for making the parameters of the learning target "sparse" by performing machine learning by adding an L1 regularization term to the loss function calculated by the NN output.

[0019] 2, a computer performs L1 regularization learning on a vector 121 using a loss function 122 to obtain an L1 regularized vector 123. The loss function 122 may be a function L obtained by adding an original loss function (first term) such as cross entropy to an L1 regularization term (second term) that uses an L1 norm (Σg(γ)=Σ|γ|), as shown in the following formula (3).

number

[0020] By the L1 regularization learning, each parameter of the vector 123 becomes a parameter indicating whether each parameter of the vector 121 is zero or non-zero (dichotomized). By using such L1 regularization learning, the computer can specify a channel in which γ is zero (close to zero) as a channel to be pruned.

[0021] The identification of pruning targets using L1 regularization learning shown in Figures 1 and 2 is applied to convolutional layers connected to BN layers, but is not intended to be applied to other layers, such as convolutional layers not connected to BN layers and fully connected layers.

[0022] Fig. 3 is a diagram showing an example of whether the methods of Fig. 1 and Fig. 2 can be applied to layers 131 to 139 of the NN 130. As shown in Fig. 3, the convolutional layers 131 and 133 and the BN layers 132 and 134 are layers to which the L1 regularization learning shown in Fig. 1 and Fig. 2 can be applied, and the convolutional layers 135 to 137 and the fully connected layers 138 and 139 are layers to which the L1 regularization learning shown in Fig. 1 and Fig. 2 cannot be applied.

[0023] Therefore, in one embodiment, a method for achieving a lightweight NN by determining a pruning rate for each layer, regardless of the type of layer, will be described.

[0024] [1-1] Example of functional configuration of server according to one embodiment Fig. 4 is a block diagram showing an example of a functional configuration of a server 1 according to an embodiment. The server 1 is an example of a calculator, computer, or information processing device that outputs a pruning rate. As shown in Fig. 4, the server 1 may exemplarily include a memory unit 11, an acquisition unit 12, a machine learning unit 13, a pruning rate calculation unit (hereinafter simply referred to as a "calculation unit") 14, and an output unit 15. The acquisition unit 12, the machine learning unit 13, the calculation unit 14, and the output unit 15 are examples of a control unit 16.

[0025] The memory unit 11 is an example of a storage area, and stores various data used by the server 1. As illustrated in Fig. 4, the memory unit 11 may be capable of storing, for example, an unlearned model 11a, machine learning data 11b, a machine learning completed model 11c, a pruning rate 11d, and a lightweight model 11e.

[0026] The acquisition unit 12 acquires the unlearned model 11a and the machine learning data 11b, and stores them in the memory unit 11. For example, the acquisition unit 12 may generate one or both of the unlearned model 11a and the machine learning data 11b in the server 1, or may receive them from a computer external to the server 1 via a network (not shown).

[0027] The unlearned model 11a may be a pre-machine learning model of a NN including an unlearned parameter. The NN may include various layers, for example, a DNN (Deep NN). The NN may include, for example, a convolution layer to which a BN layer is not connected, or a fully connected layer, or may include a convolution layer to which a BN layer is connected, and may be, for example, the NN 130 illustrated in FIG. 3.

[0028] The machine learning data 11b may be, for example, a training data set used for machine learning (training) of the unlearned model 11a. As an example, when performing machine learning of a NN for realizing image processing, the machine learning data 11b may include a plurality of pairs of teacher data including training data such as image data and a correct answer label for the training data.

[0029] In the machine learning phase, the machine learning unit 13 executes a machine learning process to machine-learn the unlearned model 11a based on the machine learning data 11b. For example, the machine learning unit 13 may generate the machine-learned model 11c by the machine learning process of the unlearned model 11a. The machine-learned model 11c may be an NN model including machine-learned parameters.

[0030] The machine-learned model 11c may be obtained by updating parameters included in the unlearned model 11a, and may be regarded as a model resulting from a change from the unlearned model 11a to the machine-learned model 11c through machine learning processing, for example. The machine learning processing may be realized by various known methods.

[0031] The calculation unit 14 calculates the pruning rate 11d by executing a pruning rate calculation process for the machine-learned model 11c, and stores the pruning rate 11d in the memory unit 11.

[0032] For example, the calculation unit 14 may include a threshold calculation unit 14a that calculates a threshold for each layer to select one of the pruning rate candidates, and a determination unit 14b that determines the pruning rate 11d to be adopted based on the inference accuracy of the model pruned using the pruning rate candidates.

[0033] The output unit 15 outputs output data based on the pruning ratio 11d generated (acquired) by the calculation unit 14. The output data may include, for example, the pruning ratio 11d itself and / or the lightweight model 11e.

[0034] The lightweight model 11e is data of a model obtained by pruning the machine-learned model 11c based on the pruning rate 11d. For example, the output unit 15 may cooperate with the machine-learning unit 13 to apply the pruning rate 11d to prune and re-learn the machine-learned model 11c, thereby acquiring the lightweight model 11e and storing it in the memory unit 11. Note that the lightweight model 11e may be generated separately from the machine-learned model 11c, for example, or may be data obtained by updating the machine-learned model 11c through pruning and re-learning.

[0035] In outputting the output data, the output unit 15 may, for example, transmit (provide) the output data to another computer (not shown), or may store the output data in the memory unit 11 and manage it so that it can be acquired from the server 1 or another computer. Alternatively, in outputting the output data, the output unit 15 may output information indicating the output data to a screen on an output device such as the server 1, or may output the output data in various other modes.

[0036] [1-2] An example of pruning rate calculation process Next, a description will be given of an example of a pruning rate calculation process performed by the calculation unit 14 of the server 1. In the following description, it is assumed that the pruning rate is calculated for a weighting matrix W, which is an example of a layer parameter.

[0037] The calculation unit 14 determines the pruning rate regardless of the type of layer by utilizing the tensor error for each layer that occurs due to pruning. As an example, the calculation unit 14 may calculate the pruning rate by the following steps (i) to (iii).

[0038] (i) The calculation unit 14 (threshold calculation unit 14a) determines (calculates) a pruning rate for which accuracy can be guaranteed, for each layer.

[0039] Note that "guaranteeing accuracy" means, for example, guaranteeing that the accuracy of inference (inference accuracy) using the lightweight model 11e obtained by pruning the machine-learned model 11c exceeds a predetermined standard.

[0040] Fig. 5 is a diagram showing an example of calculation of a pruning rate that can guarantee accuracy. As illustrated in Fig. 5, in (i), the threshold calculation unit 14a determines a pruning rate to be applied to the weight matrix W of each layer included in the machine-learned model 11c to be pruned, for each weight matrix W of a plurality of layers. Note that, in Fig. 5, the description focuses on layers 131 to 133, but is not limited thereto, and the description of Fig. 5 may be applied to any of layers 131 to 139 illustrated in Fig. 3.

[0041] Here, the pruning rate is an example of the rate at which elements in a layer are reduced (reduction rate), and indicates the rate at which pruning targets in the machine-learned model 11c are made “sparse.” In the example of Figure 2, it means the number of points in vector 123 that are set to “0.”

[0042] As illustrated in Fig. 5, the threshold calculation unit 14a selects one pruning rate from a plurality of pruning rate candidates for each of the weight matrix W1 of the layer 131 (weight matrix W1 connected to the layer 132) and the weight matrix W2 between the layers 132 (weight matrix W2 connected to the layer 133). The pruning rate candidate is an example of a reduction rate candidate, and may be, for example, two or more rates between 0% and 100%, may be common to a plurality of layers, may be different rates for each layer, or may be a combination of these. In the example of Fig. 5, the pruning rate candidates are 0%, 20%, 40%, and 60%.

[0043] The threshold calculation unit 14a calculates the error of the tensor before and after pruning when pruning is performed using each of the pruning rate candidates, and calculates a threshold T W 5, the threshold calculation unit 14a determines the maximum pruning rate candidate among the pruning rate candidates having an error smaller than the threshold T w1 The maximum pruning rate candidate with a smaller error than the threshold T w2 The maximum pruning rate candidate with an error smaller than is determined to be 20% (see arrow 142).

[0044] Threshold T w is a threshold value of the error of the tensor before and after pruning, and is an upper limit of the pruning rate for which the accuracy can be guaranteed. For example, the threshold calculation unit 14a calculates a threshold T w The threshold T w The calculation method will be described in detail later.

[0045] The pruning rate calculated in (i) may be considered as a "provisionally calculated" pruning rate in relation to the processes in (ii) and (iii).

[0046] As described above, the threshold calculation unit 14a calculates the threshold T of the tensor error before and after reduction of elements in each of the multiple layers in the machine-learned model 11c of the NN including multiple layers. Furthermore, the threshold calculation unit 14a selects a reduction ratio candidate to be applied to each of the multiple layers based on the multiple thresholds T and the tensor error before and after reduction when elements are reduced in each of the multiple layers by each of the multiple reduction ratio candidates.

[0047] (ii) The calculation unit 14 (determination unit 14b) determines a pruning rate based on the accuracy of the machine learning model that has been pruned (lightened) using the pruning rate determined in (i) and the accuracy of the machine learning model that has not been pruned.

[0048] For example, the determination unit 14b determines the accuracy Acc of the model pruned at the pruning rate of each layer determined in (i) by taking into account the error due to the approximation formula (first-order Taylor expansion). p and Accuracy Margin Acc m and the accuracy of the unpruned model, Acc wo Accuracy margin Acc m is a margin that allows for a decrease in inference accuracy, and may be set by the designer. Note that the margin may be "0", in which case the determination unit 14b determines the accuracy Acc p and the accuracy of the model without pruning, Acc wo Just compare it with.

[0049] 6 is a diagram illustrating an example of calculation of the accuracy of a model before and after pruning. For example, the determination unit 14b calculates the accuracy Acc of a model (machine-learned model 11c) that is not pruned for all layers (W1, W2, . . . wo (See arrow 143). A model without pruning may be regarded as a model pruned with a pruning rate of 0% for each layer. In addition, the determination unit 14b calculates the accuracy Acc of a model pruned with the pruning rates (W1=40%, W2=20%, ...) calculated in (i). p is calculated (see arrow 144).

[0050] The determination unit 14b determines the sum of the accuracies Acc p +Acc m Accuracy Acc wo If so, the determination unit 14b determines to adopt the pruning rate determined in (i). For example, the determination unit 14b stores the pruning rate determined in (i) in the memory unit 11 as the pruning rate 11d.

[0051] On the other hand, the determination unit 14b determines the sum of the accuracies Acc p +Acc m Accuracy Acc wo If it is less than (i), the determination unit 14b determines to discard the pruning rate determined in (i). For example, the determination unit 14b determines to discard the pruning rate determined in (i) and adopt the pruning rate 11d determined in (ii) immediately before (or the initial pruning rate).

[0052] (iii) The calculation unit 14 (decision unit 14b) searches for the maximum pruning rate for which accuracy can be guaranteed by repeatedly applying (i) and (ii) multiple times.

[0053] Fig. 7 is a diagram showing an example of searching for a pruning rate. In the example of Fig. 7, the calculation unit 14 searches for pruning rates of three layers (131 to 133) three times.

[0054] As shown in FIG. 7, in the first search (see reference numeral 145), the threshold calculation unit 14a calculates a threshold T w Calculate the threshold T w Assume that the pruning rates of the layers 131 to 133 are changed from "0%, 0%, 0%" (initial value) to "40%, 20%, 40%" based on the above. For example, in (ii), the determination unit 14b determines that the Acc p +Acc m <Acc wo If it is determined that this is the case, the pruning rate determined in (i) is discarded, and the previous rate of “0%, 0%, 0%” is adopted.

[0055] In the second search (see reference numeral 146), in (i), the threshold calculation unit 14a calculates the threshold T wCalculate (update) the updated threshold T w It is assumed that the pruning rates of the layers 131 to 133 are changed from "0%, 0%, 0%" to "20%, 20%, 40%" based on the above. For example, in (ii), the determination unit 14b determines that the Acc p +Acc m ≧Acc wo If it is determined that the pruning rate is 20%, 20%, 40% is adopted and stored in the memory unit 11 as the pruning rate 11d.

[0056] In the third search (see reference numeral 147), in (i), the threshold calculation unit 14a calculates the threshold T w Calculate (update) the updated threshold T w Assume that the pruning rates of the layers 131 to 133 are determined from "20%, 20%, 40%" to "20%, 40%, 40%" based on the above. For example, in (ii), the determination unit 14b determines that the Acc p +Acc m ≧Acc wo If it is determined that the pruning rate is 20%, 40%, 40% is adopted and stored (updated) in the memory unit 11 as the pruning rate 11d.

[0057] The decision unit 14b may search for the pruning rate a predetermined number of times, for example, a preset number of times.

[0058] As described above, the determination unit 14b determines the reduction rate to be applied to each of the multiple layers based on the inference accuracy of the machine-learned model 11c and the inference accuracy after machine learning of the reduced model obtained by reducing the elements of each of the multiple layers in the machine-learned model 11c in accordance with the candidate reduction rates to be applied.

[0059] Next, a specific example of the above-mentioned pruning rate calculation process will be described. Fig. 8 is a diagram for explaining an example of a threshold derivation method, and Fig. 9 is a diagram showing an example of an upper limit of a threshold and a threshold.

[0060] The threshold calculation unit 14a calculates the threshold of the pruning rate for which the accuracy can be guaranteed for each layer by performing a first-order Taylor expansion of the loss function when pruning is performed. For example, the error of the tensor for each layer caused by pruning is Δw, the loss function when pruning is performed is L(w+Δw), the loss function of the model to be pruned is L(w), and the loss function when pruning is not performed (L ideal ) to L wo +L m Then, the threshold value of the pruning rate that can guarantee the accuracy is calculated by the following formula (4). wo is the loss function of the model without pruning, and L m is the margin of the loss function set by the designer.

number

[0061] The left side of the above formula (4) (see the dashed box in Figure 8) is a Taylor expansion of the loss function L(w+Δw) when pruning, and includes the weight gradient "∂L(W) / ∂w" for each layer to be pruned. The gradient for each layer may be calculated by backpropagation. In addition, the right side of the above formula (4) (see the dashed-dotted box in Figure 8) is a constraint that the loss function will be smaller than the ideal value (for example, the FP32 loss function) even if pruning is performed.

[0062] In this way, the threshold calculation unit 14a calculates the threshold T based on the value of the loss function of the machine-learned model 11c when reducing the elements of each of the multiple layers and the weight gradient of each of the multiple layers.

[0063] By rearranging the above formula (4), it is possible to derive a condition for the "pruning error" that satisfies the constraint that the loss function after pruning is smaller than the ideal loss function, as shown in the following formula (5). In other words, it is possible to derive an upper limit (threshold) of the error due to pruning that guarantees the accuracy (loss function). The threshold calculation unit 14a sets the right side of the following formula (5) to the threshold T.

number

[0064] As illustrated in Fig. 9, the threshold calculation unit 14a compares a threshold T set for each layer with an error of the L1 norm due to pruning. Then, the threshold calculation unit 14a determines the pruning rate candidate with the maximum value (40% in the example of Fig. 9) among the pruning rate candidates that have an error smaller than the threshold T as the pruning rate resulting from (i).

[0065] As an example, the threshold calculation unit 14a may determine a pruning rate for each layer to be pruned, at which the pruning error (left side) is equal to or smaller than the threshold (right side) according to the following formula (6): In the following formula (6), "||ΔW||1" is the L1 norm of the weights to be pruned, and "n" is the number of weight elements in the layer to be pruned.

number

[0066] As shown in the above formula (6), the threshold T is a parameter derived by approximation. In order to prevent an error in determining the pruning rate due to an approximation error, an upper limit may be set for the threshold T (see FIG. 9). For example, the threshold calculation unit 14a may limit the magnitude of the threshold T by a "trust radius" based on the trust region method. The trust radius is an example of a threshold upper limit. As an example, the threshold calculation unit 14a may scale the threshold T so that the L2 norm of the threshold T for all layers is equal to or less than the trust radius. In the example of FIG. 9, T h indicates the vector by the threshold T of each layer, and “||T h ||2″ denotes the L2 norm of the threshold T of all layers.

[0067] For example, the threshold calculation unit 14a may update the confidence radius (for example, by a constant multiplication) in addition to the pruning rate according to the comparison result of the accuracy in the process (ii) by the determination unit 14b. Note that the initial value of the confidence radius may be set by, for example, a designer.

[0068] As an example, the threshold calculation unit 14a calculates the sum of accuracies Acc p +Acc m is the accuracy Acc wo If it is greater than or equal to the accuracy Acc, the trust radius is multiplied by a constant K (“K>1.0”), and the sum of accuracies Acc p +Acc m is the accuracy Acc wo If it is less than, the trust radius may be multiplied by a constant k (“0<k<1.0”).

[0069] Explanation according to the type of [1-3] pruning target Next, examples of the pruning method and the method for calculating the pruning error according to the type of pruning target will be described. Examples of the type of pruning target include, for example, channel pruning, node pruning, and weight pruning. The calculation unit 14 may determine the pruning target and the pruning error using the weights corresponding to the pruning target according to the type of pruning target.

[0070] Example of [1-3-1] channel pruning FIG. 10 is a diagram for explaining an example of a method for determining a channel to be pruned, and FIG. 11 is a diagram for explaining an example of calculating a pruning error.

[0071] Note that FIGS. 10 and 11 show the processing flow of the convolution operation. Also, H and W with subscripts indicate the sizes of the input data, kernel, and output data, and Ch with subscripts indicates the number of channels of the input data, kernel, and output data. The same applies to the following explanations regarding other types of pruning targets.

[0072] (An example of a method for determining a channel to be pruned) When the type of pruning target is a channel, the calculation unit 14 calculates (computes) the L1 norm in units of kernels corresponding to the channels of the output data. For example, as shown in “before pruning” in FIG. 10, the calculation unit 14 calculates the L1 norm for each of the Ch1 kernels before pruning. As a result, the L1 norms for Ch1 are calculated.

[0073] Next, as illustrated in "after pruning" in FIG. 10, the calculation unit 14 prunes the channels of the corresponding output data in ascending order of the calculated L1 norm according to the set pruning rate.

[0074] (Example of pruning error calculation) 11, the calculation unit 14 calculates the L1 norm of the kernel to be pruned. The L1 norm of the kernel to be pruned is obtained by subtracting the L1 norm of all kernels after pruning from the L1 norm of all kernels before pruning, that is, the difference between the L1 norms before and after pruning.

[0075] The calculation section 14 may obtain the pruning error by dividing the calculated L1 norm by the number of elements in all kernels before pruning.

[0076] [1-3-2] Example of node pruning FIG. 12 is a diagram for explaining an example of a method for determining nodes to be pruned, and FIG. 13 is a diagram for explaining an example of calculating a pruning error.

[0077] (An example of a method for determining which nodes to prune) When the type of pruning target is a node, the calculation unit 14 calculates the L1 norm for each weight connected to the output node. In the example of "before pruning" in Fig. 12, the calculation unit 14 calculates the L1 norm for each of the solid line, the dashed line, and the dashed dotted line.

[0078] Next, the calculation unit 14 prunes the corresponding output nodes in ascending order of the calculated L1 norm according to the set pruning rate, as illustrated in "After pruning" in Fig. 12. For example, the calculation unit 14 determines the output node corresponding to the weight group with the smallest L1 norm as the node to be pruned.

[0079] (Example of pruning error calculation) 13, the calculation unit 14 calculates the L1 norm of the weight group to be pruned. The L1 norm of the weight group to be pruned is obtained by subtracting the L1 norm of all weights after pruning from the L1 norm of all weights before pruning.

[0080] The calculation unit 14 may obtain the pruning error by dividing the calculated L1 norm by the number of elements of all weights before pruning. In the example of "after pruning" in Fig. 13, the calculation unit 14 calculates the L1 norm of the weight group of the two-dot chain lines, and divides the L1 norm by the number of elements of all weights before pruning (= "6"; the number of lines).

[0081] [1-3-3] Example of weight pruning FIG. 14 is a diagram for explaining an example of a method for determining weights for pruning, and FIG. 15 is a diagram for explaining an example of calculating a pruning error.

[0082] (An example of a method for determining the weights to be pruned) When the type of pruning target is weight, the calculation unit 14 calculates the L1 norm for each element of all weights. In the example of "before pruning" in Fig. 14, the number of weight elements is "6", so the calculation unit 14 calculates "6" L1 norms.

[0083] Next, the calculation unit 14 prunes the corresponding weights in ascending order of the calculated L1 norm according to the set pruning rate, as illustrated in "After pruning" in Fig. 14. For example, the calculation unit 14 determines the weight with the smallest L1 norm as the weight to be pruned.

[0084] (Example of pruning error calculation) 15, the calculation unit 14 calculates the L1 norm of the weights to be pruned. The L1 norm of the weights to be pruned is obtained by subtracting the L1 norm of all weights after pruning from the L1 norm of all weights before pruning.

[0085] The calculation unit 14 may obtain the pruning error by dividing the calculated L1 norm by the number of elements of all weights before pruning. In the example of "after pruning" in Fig. 15, the calculation unit 14 calculates the L1 norm of the weights of the dashed lines, and divides the L1 norm by the number of elements of all weights before pruning (= "6"; the number of lines).

[0086] [1-4] Example of operation Next, an operation example of the server 1 according to an embodiment will be described with reference to Fig. 16. Fig. 16 is a flowchart for explaining an operation example of a process by the server 1 according to an embodiment.

[0087] As illustrated in FIG. 16, the machine learning unit 13 executes machine learning of the unlearned model 11a acquired by the acquisition unit 12 without pruning (step S1).

[0088] The calculation unit 14 calculates the inference accuracy (recognition rate) Acc wo is calculated (step S2).

[0089] The threshold calculation unit 14a sets an initial value of the trust radius (step S3).

[0090] The threshold calculation unit 14a calculates a threshold T for each layer and a pruning error for each layer for setting a pruning rate (step S4), and determines whether the L2 norm of the threshold T for all layers is greater than the confidence radius (step S5). If the L2 norm of the threshold T for all layers is equal to or less than the confidence radius (NO in step S5), the process proceeds to step S7.

[0091] If the L2 norm of the threshold T for all layers is greater than the confidence radius (YES in step S5), the threshold calculation unit 14a scales (updates) the threshold so that the L2 norm of the threshold T for all layers = the confidence radius (step S6), and the processing proceeds to step S7.

[0092] In step S7, the threshold calculation unit 14a provisionally calculates a pruning rate for each layer. For example, the threshold calculation unit 14a provisionally sets a pruning rate for each layer from the set pruning rate candidates. Note that steps S4 to S7 are an example of the above process (i).

[0093] The machine learning unit 13 prunes the machine-learned model 11c using the pruning rate provisionally calculated by the threshold calculation unit 14a, and executes re-machine learning of the model after pruning. The calculation unit 14 calculates the inference accuracy Acc of the model after the re-machine learning. p is calculated (step S8).

[0094] The determination unit 14b determines the inference accuracy Acc p +Margin Acc m Acc wo It is determined whether or not it is equal to or greater than this (step S9). By evaluating the inference accuracy (recognition rate), it is possible to compensate for an error in the selection of the pruning rate due to an approximation error.

[0095] Inference accuracy Acc p +Margin Acc m Acc wo If it is equal to or greater than this (YES in step S9), the decision unit 14b decides to prune the machine-learned model 11c at the provisionally calculated pruning rate (step S10), and stores the provisionally calculated pruning rate as the pruning rate 11d in the memory unit 11. In addition, the threshold calculation unit 14a increases the trust radius by a constant factor (step S11), and the process proceeds to step S14.

[0096] On the other hand, the inference accuracy Acc p +Margin Acc m Acc wo If it is less than the pruning rate (NO in step S9), the decision unit 14b discards the provisionally calculated pruning rate (step S12). The threshold calculation unit 14a reduces the trust radius by multiplying it by a constant (step S13), and the process proceeds to step S14. Note that steps S8 to S13 are an example of the process (ii) above.

[0097] In step S14, the determination unit 14b determines whether the search (the process of steps S4 to S13) has been performed a predetermined number of times, in other words, whether the number of times the process of calculating a threshold value, selecting a pruning rate candidate, and determining a pruning rate has been performed satisfies a predetermined condition. If the search has not been performed a predetermined number of times (NO in step S14), the process proceeds to step S4.

[0098] If the search has been performed a predetermined number of times (YES in step S14), the output unit 15 outputs the determined pruning rate 11d (step S15), and the process ends. Note that step S14 is an example of the process (iii) above.

[0099] As described above, the server 1 according to an embodiment calculates the error caused by pruning of the tensor used in the NN by the threshold calculation unit 14a, and generates a threshold from the value of the loss function and the gradient obtained by backpropagation of the NN. The threshold calculation unit 14a also compares the calculated pruning error with the threshold to provisionally calculate the pruning rate. Furthermore, the determination unit 14b compares the inference accuracy of the model after re-learning with the calculated pruning rate with the inference accuracy of the model without pruning, and determines the pruning rate for each layer. At this time, if the threshold calculation unit 14a determines that the inference accuracy with pruning is worse than the inference accuracy without pruning, it resets the upper limit of the threshold so that the threshold is smaller, and searches for the pruning rate again.

[0100] As a result, the server 1 according to an embodiment can determine the pruning rate for each layer regardless of the type of layer. For example, the server 1 can determine the pruning rate to be applied to the machine-learned model 11c including a convolution layer, a fully connected layer, and the like to which a BN layer is not connected.

[0101] FIG. 17 is a diagram showing an example of a model 150 including a convolution layer to which a BN layer is not connected and a fully connected layer. In a method according to an embodiment, different types of pruning targets may be adopted for each layer. In the example of FIG. 17, the pruning target of the convolution layers 151 to 155 to which a BN layer is not connected is channel pruning, and the pruning target of the fully connected layers 156 to 158 is node pruning. Note that, if the output node of the final layer (layer 158) is pruned, class classification becomes impossible, so the final layer is not subject to pruning.

[0102] Fig. 18 is a diagram showing the capacity compression rate of model data according to the pruning rate determined by the method according to an embodiment. Fig. 18 illustrates, as an example, a case in which "CIFAR-10" is used as a data set of machine learning data 11b, and "AlexNet" having the configuration shown in Fig. 17 is used as the unlearned model 11a and the machine learning completed model 11c.

[0103] In the example of Figure 18, the mini-batch size is set to “64”, and the “Optimizer hyperparameters” for the re-machine learning are set to “0.001” for the learning rate, “1e-4” for the weight decay, “0.9” for the momentum, “20%, 10%, 0%” for each search, and “0.05%” for the recognition rate margin. m The figures show the cases where the ratio is "1%, 2%" and the number of searches is "20 times."

[0104] As illustrated in FIG. 18, according to the technique according to one embodiment, the recognition rate Acc wo and the recognition rate Acc when pruned p The difference between the margin and the Acc m With the data size kept below 1% or 2%, a model data compression rate of 94% or more can be achieved.

[0105] 19 to 21 show the results shown in FIG. m22 is a diagram showing an example of the data size of the model 150 according to the number of searches, the pruning rate (whole model), and the pruning rate (for each layer) in the case of "1%". Fig. 22 is a diagram showing an example of the number of channels or the number of nodes (upper part) and the pruning rate for each layer (lower part) for each layer 151 to 157 (convolutional layers 1 to 5, fully connected layers 1 to 2) shown in Fig. 17.

[0106] As illustrated in Fig. 19 to Fig. 22, from the relationship between the number of searches, the data size, and the pruning rate, it can be seen that the pruning rate increases as the search progresses, and the data size is compressed. For example, as illustrated in Fig. 22, the pruning rate is higher in layers closer to the output layer (layer 158 side) than in layers closer to the input layer (layer 151 side). Therefore, in the case of model 150 illustrated in Fig. 17, since layers closer to the input layer have a greater impact on accuracy than layers closer to the output layer, the pruning rate can be determined for each layer to improve the compression effect of the model data capacity.

[0107] As described above, according to the technique of one embodiment, the pruning rate can be determined and output for each layer, thereby realizing weight reduction of NNs including various multiple layers.

[0108] [1-5] Variations Next, a modified example according to an embodiment will be described. In the following description, for the sake of simplicity, the margin of inference accuracy Acc m is "0", in other words, in the comparison of inference accuracy, the inference accuracy Acc p Acc wo Assume that it is determined whether or not the number is equal to or greater than the number.

[0109] [1-5-1] First modified example In a method according to one embodiment, the number of searches for the pruning rate (the number of trials of the process (iii) above) is a hyperparameter that is set manually by, for example, a designer. For this reason, for example, if the number of searches is set to a small number, the machine-learned model 11c may not be sufficiently lightweight, whereas if the number of searches is set to a large number, the machine-learned model 11c may be sufficiently lightweight, but the search time may be long.

[0110] Fig. 23 is a diagram showing the relationship between the number of searches for the pruning rate and the model size. As shown in Fig. 23, when the number of searches is "200", the model size is saturated at "5MB" at the time of the number of searches "50", so it can be seen that searches "51st to 200th" are unnecessary. In addition, when the number of searches is "20", the model size remains at about "11MB", and it can be seen that there is room for further weight reduction.

[0111] FIG. 24 is a diagram illustrating an example of a pruning error comparison result in response to an update of the trust radius in the method according to an embodiment.

[0112] As illustrated in Fig. 24, assume that a pruning rate of "10%" is calculated (determined) in the error comparison result of the mth search (m is an integer equal to or greater than "1"). In this case, the trust radius is updated so as to increase by a constant K times. However, if the updated trust radius is less than the error of the pruning rate candidate that is one greater than the pruning rate candidate determined in the mth search, a pruning rate of "10%" is again calculated in the error comparison result of the (m+1)th search.

[0113] In this way, when the trust radius is set to a constant K or a constant k times, the amount of updating of the threshold is limited by the trust radius, so the same pruning rate candidate may be adopted in multiple searches. A state in which the same combination of pruning rates is searched multiple times leads to an increase in the number of searches for pruning rates without sufficient attempts to prune the model.

[0114] Therefore, in the first modified example, a method for shortening (reducing) the search time (number of searches) for an appropriate pruning rate for reducing the weight of the NN will be described with attention focused on updating the trust radius.

[0115] Fig. 25 is a block diagram showing an example of a functional configuration of a server 1A according to a first modified example. As shown in Fig. 25, the server 1A may include a calculation unit 14A different from the server 1 in Fig. 4. The calculation unit 14A may include a threshold calculation unit 14a' and a determination unit 14b' different from the calculation unit 14 in Fig. 4.

[0116] The calculation unit 14A searches for a different combination of pruning rates for each search. Here, the state in which the combination of pruning rates of all layers of "0%" is selected is a state in which the calculation unit 14A has determined that no more pruning rate searches will be performed. Under such a premise, the calculation unit 14A (the determination unit 14b') terminates the search when the combination of pruning rates of all layers of "0%" is selected.

[0117] The threshold calculation unit 14a' determines, for each layer i (i is an integer equal to or greater than 1), the error of the pruning rate that is one value greater than the searched pruning rate or the absolute value of the difference between the error of the searched pruning rate and the threshold, "E diff,i " is measured.

[0118] For example, the threshold calculation unit 14a′ calculates the inference accuracy Acc p Acc wo If it is equal to or greater than the threshold, the absolute value of the difference between the error of the pruning rate that is one value larger than the pruning rate searched and the threshold, "E diff,i " is measured.

[0119] On the other hand, the threshold calculation unit 14a' calculates the inference accuracy Acc p Acc wo If it is less than 1, the absolute value of the difference between the error of the pruning rate searched and the threshold, “E diff,i " is measured.

[0120] The threshold calculation unit 14a′ calculates the absolute value of the calculated differences for all layers, “E diff,i The smallest value (difference) of "E diff " is obtained. E diff = min(E diff,1 , E diff,2 , ..., E diff,i ) (7)

[0121] The threshold calculation unit 14a' determines a constant multiple of the confidence radius and a difference between the confidence radius and "E diff The trust radius is updated by adopting the sum or difference between the two, whichever has the larger amount of fluctuation.

[0122] For example, the threshold calculation unit 14a′ calculates the inference accuracy Acc p Acc wo If it is equal to or greater than the confidence radius, the constant K times the confidence radius and the difference between the confidence radius and the confidence radius “E diff " and the sum of the two, whichever has the larger amount of variation, is adopted and updated so that the trust radius increases.

[0123] On the other hand, the threshold calculation unit 14a' calculates the inference accuracy Acc p Acc wo If it is less than 0, the constant k times the confidence radius and the difference between the confidence radius and "E diff " is used to update the trust radius so that it decreases.

[0124] In this manner, the threshold calculation unit 14a' updates the confidence radius so that the combinations of pruning rate candidates for each of the multiple layers are different from one another each time the process of selecting pruning rate candidates (in other words, search) is performed.

[0125] FIG. 26 is a diagram for explaining an example of the trust radius update process when the trust radius is increased. As shown in FIG. 26, a case is assumed where the pruning rate searched for the mth time is "(layer 1, layer 2)=(10%, 0%)". The threshold calculation unit 14a' calculates the absolute value "E diff,1 ” and the absolute value of the difference between the error and the trust radius for a pruning rate of “10%” in layer 2, “E diff,2 The threshold calculation unit 14a' calculates the smallest difference "E diff,2 " to "E diff " is obtained.

[0126] Then, the threshold calculation unit 14a′ determines (updates) the (m+1)th (next) trust radius in accordance with the following formula (8). (m+1th trust radius) = max((m-th confidence radius · constant K), (m-th confidence radius + E diff )) (8)

[0127] As a result, the trust radius for the (m+1)th iteration is selected to be at least a value equal to or greater than the sum of the trust radius and the difference, and therefore a pruning rate for the (m+1)th iteration is calculated to have a bit width different from that for the (m)th iteration.

[0128] 26, the confidence radius (upper threshold value) in the (m+1)th search coincides with the error of the pruning rate of "10%" for layer 2. Therefore, in the (m+1)th search, a pruning rate of "(layer 1, layer 2)=(10%, 10%)", which is a combination of pruning rates different from the previous search, is searched for.

[0129] FIG. 27 is a diagram for explaining an example of the trust radius update process when decreasing the trust radius. As shown in FIG. 27, a case is assumed where the pruning rate searched for the mth time is "(layer 1, layer 2)=(10%, 0%)". The threshold calculation unit 14a' calculates the absolute value "E diff,1 ” and the absolute value of the difference between the error and the trust radius for the layer 2 pruning rate of “0%” “Ediff,2 The threshold calculation unit 14a' calculates the smallest difference "E diff,1 " to "E diff " is obtained.

[0130] Then, the threshold calculation unit 14a′ determines (updates) the (m+1)th (next) trust radius in accordance with the following formula (9). (m+1th trust radius) = max((m-th confidence radius · constant), (m-th confidence radius - E diff )) (9)

[0131] As a result, the trust radius for the (m+1)th iteration is selected to be at least a value greater than the difference between the trust radius and the difference, and therefore a pruning rate for the (m+1)th iteration is calculated to be a bit width different from that for the (m)th iteration.

[0132] 27, the confidence radius (upper threshold value) in the (m+1)th search coincides with the error of the pruning rate of "0%" for layer 1. Therefore, in the (m+1)th search, a pruning rate of "(layer 1, layer 2)=(0%, 0%)" is searched for, which is a combination of pruning rates different from the previous search.

[0133] By generalizing the above equations (8) and (9), the next trust radius can be expressed by the following equation (10). Next trust radius = current trust radius * max(constant, Qscale_min) (10)

[0134] Here, in the above formula (10), the constant is K or k, "Qscale_min" is "Qscale" expressed by the following formula (11), and "Qscale" is expressed by the following formula (12). Qscale_min = min(Qscale calculated for all vectors to be quantized) (11) Qscale = 1 + Qdiff / Qth (12)

[0135] In the above formula (12), "Qdiff" is "the difference between the quantization error of the bit width that is one bit narrower than the provisionally calculated bit width (pruning rate) and the threshold value," and "Qth" is the threshold value.

[0136] Next, an operation example of the server 1A according to the first modified example will be described with reference to Fig. 28. Fig. 28 is a flowchart for explaining an operation example of processing by the server 1A according to the first modified example. Fig. 28 is obtained by replacing steps S11, S13, and S14 in the flowchart according to the server 1 shown in Fig. 16 with steps S21, S22, and S23, respectively. Note that, also in the first modified example, the threshold calculation unit 14a' sets an initial value of the trust radius in step S3.

[0137] In step S21, the threshold calculation unit 14a' increases the trust radius by a constant K times or by the "sum of differences", whichever is larger, and the process proceeds to step S23.

[0138] In step S22, the threshold calculation unit 14a' reduces the confidence radius by a constant k times or by the "difference of differences", whichever is larger, and the process proceeds to step S23.

[0139] In step S23, the decision unit 14b' determines whether the pruning rate 11d of all layers is "0%", in other words, whether the pruning rate satisfies a predetermined condition. If the pruning rate 11d of at least one layer is not "0%" (NO in step S23), the process proceeds to step S4.

[0140] If the pruning rate 11d of all layers is "0%" (YES in step S23), the output unit 15 outputs the determined pruning rate 11d (step S15), and the process ends.

[0141] As described above, in the first modified example, the method of updating the trust radius by the threshold calculation unit 14a' and the end condition for determining the end of the search by the determination unit 14b' are different from those in the first embodiment. This allows the server 1A to search for an appropriate pruning rate for sufficiently reducing the weight of the NN in the shortest time (shortest number of times). In addition, the setting (designation) of the number of searches by the designer, etc. can be omitted.

[0142] Fig. 29 is a diagram showing the relationship between the number of searches for the pruning rate and the model size. As illustrated in Fig. 29, according to the first modified example, the search ends when the pruning rate for all layers becomes "0%" near the number of searches "50 times", in other words, when the model size is sufficiently lightened (reaches "5MB" at which it saturates). This makes it possible to prevent unnecessary searches "51st to 200th times" from being performed when the number of searches is specified to "200 times", for example, and to optimize the search time (number of searches).

[0143] [1-5-2] Second modified example In the method according to the embodiment and the first modified example, the initial value of the trust radius is a hyperparameter that is set by a designer or the like.

[0144] Fig. 30 is a diagram showing the relationship between the number of searches for the pruning rate and the model size according to the setting of the initial value of the trust radius. As illustrated in Fig. 30, when the initial value of the trust radius is set large (see dashed line) and when it is set small (see dashed line), the model size is different even with the same number of searches. Also, when the initial value of the trust radius is set large, the number of searches required until the model size is sufficiently reduced is greater than when the initial value of the trust radius is set small.

[0145] In this way, the final model size and the number of searches for the pruning rate vary depending on the initial value of the trust radius; in other words, there is a possibility that the performance of the servers 1 and 1A may vary.

[0146] Therefore, in the second modified example, a method for suppressing the variation in performance between the servers 1 and 1A will be described.

[0147] 31 is a block diagram showing an example of a functional configuration of a server 1B according to a second modified example. As shown in FIG. 31, the server 1B may include a calculation unit 14B different from the server 1 in FIG. 4. The calculation unit 14B may include a threshold calculation unit 14a" and a determination unit 14b" different from the calculation unit 14 in FIG. 4.

[0148] In model pruning, it is known that pruning a model gradually using a small pruning rate can maintain accuracy and compress the model at a higher compression rate than pruning a model all at once using a large pruning rate.

[0149] In addition, as shown in the above formula (5), the threshold T is set according to the inverse of the gradient, so a layer with a large threshold T means that the layer has a small gradient. A layer with a small gradient means that pruning the layer has a small effect on accuracy.

[0150] Therefore, the server 1B (threshold calculation unit 14a"), for example, sets the initial value of the trust radius to a value that minimizes the pruning rate in the first search. To achieve this, the threshold calculation unit 14a" may set the initial value of the trust radius to a value that minimizes the pruning rate in the first search. To achieve this, the threshold calculation unit 14a" may set the initial value of the trust radius to a value that minimizes the pruning rate in the first search.

[0151] By setting the initial value of the trust radius as described above, the server 1B can compress the model size more or maintain accuracy more than if the initial value of the trust radius was manually set to, for example, a larger value.

[0152] Fig. 32 is a diagram for explaining an example of setting the initial value of the trust radius. As shown in the upper part of Fig. 32, when the initial value of the trust radius is not set, the combination of pruning rates searched for is "(layer 1, layer 2) = (10%, 20%)".

[0153] As illustrated in FIG. 32, in the initial search for the pruning rate, the threshold calculation unit 14a'' measures the threshold (max(Th)) of the layer with the largest threshold among all layers and the error (Error) due to the smallest pruning rate (excluding “0%”) of that layer.

[0154] Th denotes a vector of thresholds T1, T2, ... for each layer, and Th = [T1, T2] in the example of Fig. 32. Threshold (max(Th)) is the threshold for the layer with the largest threshold, and is T2 in the example of Fig. 32. Error is the error of the minimum pruning rate for the layer with the largest threshold, and in the example of Fig. 32, the error for a pruning rate of "10%" for layer 2 is measured.

[0155] Next, the threshold calculation unit 14a'' sets an initial value of the confidence radius using the measured threshold and error according to the following formula (13). In the following formula (13), ``||Th||2'' is the L2 norm of the thresholds of all layers.

number

[0156] The threshold calculation unit 14a'' sets the thresholds T1 and T2 so that, based on the initial value of the calculated trust radius, the minimum pruning rate of “10%” is selected as the pruning rate for the layer with the largest threshold (layer 2) and a pruning rate of “0%” is selected for the remaining layer (layer 1).

[0157] As a result, when the initial value of the trust radius is set and the thresholds T1 and T2 are set, the combination of pruning rates to be searched for is "(layer 1, layer 2) = (0%, 10%)" as shown in the lower part of Fig. 32. The layer to be pruned (layer 2) has the largest threshold, in other words, the smallest gradient, so that the effect of pruning on accuracy can be kept small.

[0158] In addition, the functions of the threshold calculation unit 14a" other than the process of setting the initial value of the trust radius may be similar to one or both of the threshold calculation unit 14a according to an embodiment and the threshold calculation unit 14a' according to the first modified example. Furthermore, the determination unit 14b" may be similar to one or both of the determination unit 14b according to an embodiment and the determination unit 14b' according to the first modified example.

[0159] That is, the technique according to the second modified example may be realized in combination with one or both of the embodiment and the first modified example.

[0160] Next, an operation example of the server 1B according to the second modified example will be described with reference to Fig. 33. Fig. 33 is a flowchart for explaining an operation example of processing by the server 1B according to the second modified example. Fig. 33 is a flowchart for the server 1A shown in Fig. 28, in which step S3 is deleted, steps S31 and S32 are added between steps S4 and S5, and steps S21, S22, and S23 are replaced with steps S33, S34, and S35, respectively.

[0161] In step S31, the threshold calculation unit 14a'' determines whether or not this is the first search after calculating the threshold for each layer in step S4. If this is not the first search (NO in step S31), the process proceeds to step S5.

[0162] If it is the first search (YES in step S31), the threshold calculation unit 14a'' sets an initial value of the trust radius based on the threshold of the layer with the largest threshold and the minimum pruning rate error (step S32), and the process proceeds to step S5.

[0163] Steps S33, S34, and S35 may be any of steps S11, S13, and S14 shown in FIG. 16 and steps S21, S22, and S23 shown in FIG. 28, respectively.

[0164] As described above, in the second modified example, the method of setting the initial value of the trust radius by the threshold calculation unit 14a'' is different from that of the embodiment and the first modified example. This enables the server 1B to suppress fluctuations in the number of searches for the final model size and the pruning rate, and suppresses variations in the performance of the servers 1 and 1A.

[0165] Fig. 34 is a diagram showing the relationship between the number of searches for the pruning rate and the model size according to the setting of the initial value of the trust radius. As illustrated in Fig. 34, when the initial value of the trust radius is set large (see the dashed line), the model size at the beginning of the search (for example, the "0th" to "5th" searches), in other words, the pruning rate, becomes large.

[0166] In contrast, by using the method according to the second variant, the initial value of the trust radius is set based on the threshold of the layer with the largest threshold and the minimum pruning rate error, the model size at the beginning of the search, in other words the pruning rate, can be reduced to the same level as when the initial value of the trust radius is set small (see dashed line).

[0167] This allows the server 1B to suppress manual setting of the initial value (hyperparameter) of the trust radius by a designer or the like, and dynamically set the initial value of the trust radius according to the layer of the machine-learned model 11c. Therefore, an appropriate pruning rate can be set for each model, and fluctuations in the number of searches for the final model size and pruning rate can be suppressed regardless of the model, thereby suppressing variations in the performance of the servers 1 and 1A.

[0168] [1-6] Hardware configuration example The servers 1, 1A, and 1B according to the embodiment and the first and second modified examples may each be a virtual machine (VM) or a physical machine. Furthermore, the functions of the servers 1, 1A, and 1B may be realized by one computer or two or more computers. Furthermore, at least a part of the functions of the servers 1, 1A, and 1B may be realized using HW (Hardware) resources and NW (Network) resources provided by a cloud environment.

[0169] Fig. 35 is a block diagram showing an example of a hardware (HW) configuration of the computer 10. The following description will be given taking the computer 10 as an example of hardware (HW) that realizes the functions of the servers 1, 1A, and 1B. Note that when multiple computers are used as HW resources that realize the functions of the servers 1, 1A, and 1B, each computer may have the HW configuration shown in Fig. 35.

[0170] As shown in FIG. 35, the computer 10 may include, as a HW configuration, a processor 10a, a memory 10b, a storage unit 10c, an IF (Interface) unit 10d, an IO (Input / Output) unit 10e, and a reading unit 10f, for example.

[0171] The processor 10a is an example of a processing unit that performs various controls and calculations. The processor 10a may be connected to each block in the computer 10 via a bus 10i so that they can communicate with each other. The processor 10a may be a multiprocessor including a plurality of processors, a multicore processor having a plurality of processor cores, or a configuration having a plurality of multicore processors.

[0172] Examples of the processor 10a include integrated circuits (ICs) such as a CPU, MPU, GPU, APU, DSP, ASIC, and FPGA. CPU is an abbreviation for Central Processing Unit, MPU is an abbreviation for Micro Processing Unit, GPU is an abbreviation for Graphics Processing Unit, and APU is an abbreviation for Accelerated Processing Unit. DSP is an abbreviation for Digital Signal Processor, ASIC is an abbreviation for Application Specific IC, and FPGA is an abbreviation for Field-Programmable Gate Array.

[0173] In addition, a combination of two or more of the above-mentioned integrated circuits may be used as the processor 10a. As an example, the computer 10 may include a first and a second processor 10b. The first processor 10a is an example of a CPU that executes a program 10g (machine learning program) that realizes all or part of various functions of the computer 10. The first processor 10a may realize the functions of the acquisition unit 12, the calculation unit 14, 14A or 14B, and the output unit 15 of the server 1, 1A or 1B (see FIG. 4, FIG. 25 or FIG. 31) based on the program 10g, for example. The second processor 10a is an example of an accelerator that executes arithmetic processing used in NN calculation such as matrix calculation, and may realize the function of the machine learning unit 13 of the server 1, 1A or 1B (see FIG. 4, FIG. 25 or FIG. 31), for example.

[0174] The memory 10b is an example of HW that stores various data, programs, and other information. Examples of the memory 10b include one or both of a volatile memory such as a dynamic random access memory (DRAM) and a non-volatile memory such as a persistent memory (PM).

[0175] The storage unit 10c is an example of HW that stores various data, programs, and other information. Examples of the storage unit 10c include various storage devices such as a magnetic disk device such as a hard disk drive (HDD), a semiconductor drive device such as a solid state drive (SSD), and a nonvolatile memory. Examples of the nonvolatile memory include a flash memory, a storage class memory (SCM), and a read only memory (ROM).

[0176] The storage unit 10c may store a program 10g. For example, the processor 10a of the servers 1, 1A, and 1B can realize the function of the control unit 16 (see FIG. 4, FIG. 25, or FIG. 31) of the servers 1, 1A, and 1B by expanding the program 10g stored in the storage unit 10c into the memory 10b and executing it.

[0177] Moreover, the memory unit 11 illustrated in FIG. 4, FIG. 25, or FIG. 31 may be realized by a storage area included in at least one of the memory 10b and the storage unit 10c.

[0178] The IF unit 10d is an example of a communication IF that controls connection and communication with a network. For example, the IF unit 10d may include an adapter that complies with a LAN (Local Area Network) such as Ethernet (registered trademark) or optical communication such as FC (Fibre Channel). The adapter may support one or both of wireless and wired communication methods. For example, the servers 1, 1A, and 1B may be connected to a computer (not shown) via the IF unit 10d so that they can communicate with each other. One or both of the functions of the acquisition unit 12 and the output unit 15 illustrated in FIG. 4, FIG. 25, or FIG. 31 may be realized by the IF unit 10d. Also, for example, the program 10g may be downloaded from a network to the computer 10 via the communication IF and stored in the storage unit 10c.

[0179] The IO unit 10e may include one or both of an input device and an output device. Examples of the input device include a keyboard, a mouse, and a touch panel. Examples of the output device include a monitor, a projector, and a printer. For example, the output unit 15 illustrated in FIG. 4, FIG. 25, or FIG. 31 may output the pruning rate 11d to the output device of the IO unit 10e and display it.

[0180] The reading unit 10f is an example of a reader that reads out data or program information recorded on the recording medium 10h. The reading unit 10f may include a connection terminal or device to which the recording medium 10h can be connected or inserted. Examples of the reading unit 10f include an adapter that complies with USB (Universal Serial Bus) or the like, a drive device that accesses a recording disk, and a card reader that accesses a flash memory such as an SD card. The recording medium 10h may store a program 10g, and the reading unit 10f may read the program 10g from the recording medium 10h and store it in the storage unit 10c.

[0181] Examples of the recording medium 10h include non-transitory computer-readable recording media such as magnetic / optical disks and flash memories. Examples of the magnetic / optical disks include flexible disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray Discs, and HVDs (Holographic Versatile Discs). Examples of the flash memory include semiconductor memories such as USB memories and SD cards.

[0182] The above-mentioned HW configuration of the computer 10 is an example. Therefore, the HW in the computer 10 may be increased or decreased (for example, adding or deleting any block), divided, integrated in any combination, or added or deleted buses, etc. may be appropriately performed. For example, in the servers 1, 1A, and 1B, at least one of the IO unit 10e and the reading unit 10f may be omitted.

[0183] [2] Other The techniques according to the above-described embodiment and the first and second modifications can be implemented with the following modifications and variations.

[0184] For example, the acquisition unit 12, the machine learning unit 13, the calculation unit 14, 14A or 14B, and the output unit 15 provided in the server 1, 1A or 1B shown in FIG. 4, FIG. 25 or FIG. 31 may be merged or each may be divided.

[0185] 4, 25, or 31 may be configured such that a plurality of devices cooperate with each other via a network to realize each processing function. As an example, in the server 1, 1A, or 1B, the acquisition unit 12 and the output unit 15 may be a Web server and an application server, the machine learning unit 13 and the calculation unit 14, 14A, or 14B may be an application server, and the memory unit 11 may be a DB server, etc. In this case, the Web server, the application server, and the DB server may cooperate with each other via a network to realize the processing function of the server 1, 1A, or 1B.

[0186] [3] Additional Notes Regarding the above embodiment and the first and second modifications, the following supplementary notes are further disclosed.

[0187] (Appendix 1) Calculating a threshold value of a tensor error before and after reduction of elements of each of a plurality of layers in a machine-learned model of a neural network including the plurality of layers; selecting a reduction ratio candidate to be applied to each of the plurality of layers based on the plurality of thresholds and an error of a tensor before and after reduction when the element is reduced by each of the plurality of reduction ratio candidates in each of the plurality of layers; determining a reduction rate to be applied to each of the plurality of layers based on an inference accuracy of the machine-learned model and an inference accuracy after machine learning of a reduced model obtained by reducing elements of each of the plurality of layers in the machine-learned model according to the candidate reduction rate to be applied; A machine learning program that lets a computer carry out processing.

[0188] (Appendix 2) The process of calculating the threshold value includes a process of calculating the threshold value based on a value of a loss function of the machine-learned model when reducing elements of each of the plurality of layers and a weight gradient of each of the plurality of layers. Attachment 1, a machine learning program.

[0189] (Appendix 3) The process of determining the reduction rate includes: discarding the selected reduction rate candidates when a sum of the inference accuracy and a margin after machine learning of the reduced model is less than the inference accuracy of the machine learning model; When a sum of the inference accuracy after machine learning of the reduced model and the margin is equal to or greater than the inference accuracy of the machine learning model, determining the selected multiple reduction rate candidates as reduction rates to be applied to each of the multiple layers. 1. The machine learning program according to claim 1 or 2.

[0190] (Appendix 4) The process of calculating the threshold value includes a process of scaling the threshold value such that an L2 norm of the threshold value of the plurality of layers is equal to or less than an upper threshold limit. Attachment 3, a machine learning program.

[0191] (Appendix 5) The process of calculating the threshold value includes: When a sum of the inference accuracy after machine learning of the reduced model and the margin is less than the inference accuracy of the machine-learned model, decreasing the upper threshold value; When a sum of the inference accuracy after machine learning of the reduced model and the margin is equal to or greater than the inference accuracy of the machine-learned model, the upper threshold is increased. Attachment 4: A machine learning program.

[0192] (Appendix 6) the process of calculating the threshold value includes a process of updating the upper threshold value so that a combination of the reduction rate candidates for each of the plurality of layers is a different combination each time the process of selecting the reduction rate candidates is executed. Attachment 5. The machine learning program described in Appendix 5.

[0193] (Appendix 7) the process of calculating the threshold includes a process of setting an initial value of the upper threshold limit so that an initial value of the upper threshold limit is calculated such that an element of a layer having the maximum threshold among the plurality of layers is reduced and an element of a layer other than the layer having the maximum threshold is not reduced. 6. The machine learning program according to claim 5 or 6.

[0194] (Appendix 8) Repeating the process of calculating the threshold value, the process of selecting the candidate reduction ratio, and the process of determining the reduction ratio until the number of times the process is performed or the reduction ratio satisfies a predetermined condition; outputting the reduction rate determined when the predetermined condition is satisfied; 8. The machine learning program according to any one of claims 1 to 7, which causes the computer to execute a process.

[0195] (Appendix 9) Calculating a threshold value of a tensor error before and after reduction of elements of each of a plurality of layers in a machine-learned model of a neural network including the plurality of layers; selecting a reduction ratio candidate to be applied to each of the plurality of layers based on the plurality of thresholds and an error of a tensor before and after reduction when the element is reduced by each of the plurality of reduction ratio candidates in each of the plurality of layers; determining a reduction rate to be applied to each of the plurality of layers based on an inference accuracy of the machine-learned model and an inference accuracy after machine learning of a reduced model obtained by reducing elements of each of the plurality of layers in the machine-learned model according to the candidate reduction rate to be applied; A machine learning method in which the processing is performed by a computer.

[0196] (Appendix 10) The process of calculating the threshold value includes a process of calculating the threshold value based on a value of a loss function of the machine-learned model when reducing elements of each of the plurality of layers and a weight gradient of each of the plurality of layers. 10. The machine learning method according to claim 9.

[0197] (Appendix 11) The process of determining the reduction rate includes: discarding the selected reduction rate candidates when a sum of the inference accuracy and a margin after machine learning of the reduced model is less than the inference accuracy of the machine learning model; When a sum of the inference accuracy after machine learning of the reduced model and the margin is equal to or greater than the inference accuracy of the machine learning model, determining the selected multiple reduction rate candidates as reduction rates to be applied to each of the multiple layers. 11. The machine learning method according to claim 9 or 10.

[0198] (Appendix 12) The process of calculating the threshold value includes a process of scaling the threshold value such that an L2 norm of the threshold value of the plurality of layers is equal to or less than an upper threshold limit. 12. The machine learning method of claim 11.

[0199] (Appendix 13) The process of calculating the threshold value includes: When a sum of the inference accuracy after machine learning of the reduced model and the margin is less than the inference accuracy of the machine-learned model, decreasing the upper threshold value; When a sum of the inference accuracy after machine learning of the reduced model and the margin is equal to or greater than the inference accuracy of the machine-learned model, the upper threshold is increased. 13. The machine learning method of claim 12.

[0200] (Appendix 14) the process of calculating the threshold value includes a process of updating the upper threshold value so that a combination of the reduction rate candidates for each of the plurality of layers is a different combination each time the process of selecting the reduction rate candidates is executed. 14. The machine learning method of claim 13.

[0201] (Appendix 15) the process of calculating the threshold includes a process of setting an initial value of the upper threshold limit so that an initial value of the upper threshold limit is calculated such that an element of a layer having the maximum threshold among the plurality of layers is reduced and an element of a layer other than the layer having the maximum threshold is not reduced. 15. The machine learning method according to claim 13 or 14.

[0202] (Appendix 16) Repeating the process of calculating the threshold value, the process of selecting the candidate reduction ratio, and the process of determining the reduction ratio until the number of times the process is performed or the reduction ratio satisfies a predetermined condition; outputting the reduction rate determined when the predetermined condition is satisfied; 16. The machine learning method according to any one of claims 9 to 15, wherein processing is executed by the computer.

[0203] (Appendix 17) Calculating a threshold value of a tensor error before and after reduction of elements of each of a plurality of layers in a machine-learned model of a neural network including the plurality of layers; selecting a reduction ratio candidate to be applied to each of the plurality of layers based on the plurality of thresholds and an error of a tensor before and after reduction when the element is reduced by each of the plurality of reduction ratio candidates in each of the plurality of layers; determining a reduction rate to be applied to each of the plurality of layers based on an inference accuracy of the machine-learned model and an inference accuracy after machine learning of a reduced model obtained by reducing elements of each of the plurality of layers in the machine-learned model according to the candidate reduction rate to be applied; An information processing device comprising a control unit.

[0204] (Appendix 18) The control unit calculates the threshold value based on a value of a loss function of the machine-learned model when reducing elements of each of the plurality of layers and a weight gradient of each of the plurality of layers in the process of calculating the threshold value. 18. The information processing device according to claim 17.

[0205] (Appendix 19) The control unit, in the process of determining the reduction rate, discarding the selected reduction rate candidates when a sum of the inference accuracy and a margin after machine learning of the reduced model is less than the inference accuracy of the machine learning model; When a sum of the inference accuracy after machine learning of the reduced model and the margin is equal to or greater than the inference accuracy of the machine learning model, the selected multiple reduction rate candidates are determined as reduction rates to be applied to each of the multiple layers. 19. The information processing device according to claim 17 or 18.

[0206] (Appendix 20) The control unit, in the process of calculating the thresholds, scales the thresholds so that an L2 norm of the thresholds of the plurality of layers is equal to or less than an upper threshold limit. 20. The information processing device according to claim 19. [Explanation of symbols]

[0207] 1, 1A, 1B Server 10. Computers 11 Memory section 11a Untrained model 11b Data for machine learning 11c Machine Learning Models 11d Pruning rate 11e lightweight model 12 Acquisition Department 13 Machine Learning Department 14, 14A, 14B Pruning rate calculation section (calculation section) 14a, 14a', 14a" Threshold calculation unit 14b, 14b', 14b” decision section 15 Output section 16 Control section

Claims

1. Calculating a threshold value of a tensor error before and after reduction of elements of each of a plurality of layers in a machine-learned model of a neural network including the plurality of layers; selecting a reduction ratio candidate to be applied to each of the plurality of layers based on the plurality of thresholds and an error of a tensor before and after reduction when the element is reduced by each of the plurality of reduction ratio candidates in each of the plurality of layers; determining a reduction rate to be applied to each of the plurality of layers based on an inference accuracy of the machine-learned model and an inference accuracy after machine learning of a reduced model obtained by reducing elements of each of the plurality of layers in the machine-learned model according to the candidate reduction rate to be applied; A machine learning program that lets a computer carry out processing.

2. The process of calculating the threshold value includes a process of calculating the threshold value based on a value of a loss function of the machine-learned model when reducing elements of each of the plurality of layers and a weight gradient of each of the plurality of layers. The machine learning program according to claim 1 .

3. The process of determining the reduction rate includes: discarding the selected reduction rate candidates when a sum of the inference accuracy and a margin after machine learning of the reduced model is less than the inference accuracy of the machine learning model; When a sum of the inference accuracy after machine learning of the reduced model and the margin is equal to or greater than the inference accuracy of the machine learning model, determining the selected multiple reduction rate candidates as reduction rates to be applied to each of the multiple layers. The machine learning program according to claim 1 or 2.

4. The process of calculating the threshold value includes a process of scaling the threshold value such that an L2 norm of the threshold value of the plurality of layers is equal to or less than an upper threshold limit. The machine learning program according to claim 3 .

5. The process of calculating the threshold value includes: When a sum of the inference accuracy after machine learning of the reduced model and the margin is less than the inference accuracy of the machine-learned model, decreasing the upper threshold value; When a sum of the inference accuracy after machine learning of the reduced model and the margin is equal to or greater than the inference accuracy of the machine-learned model, the upper threshold is increased. The machine learning program according to claim 4.

6. the process of calculating the threshold value includes a process of updating the upper threshold value so that a combination of the reduction rate candidates for each of the plurality of layers is a different combination each time the process of selecting the reduction rate candidates is executed. The machine learning program according to claim 5 .

7. the process of calculating the threshold includes a process of setting an initial value of the upper threshold limit so that an initial value of the upper threshold limit is calculated such that an element of a layer having the maximum threshold among the plurality of layers is reduced and an element of a layer other than the layer having the maximum threshold is not reduced. The machine learning program according to claim 5 or 6.

8. Repeating the process of calculating the threshold value, the process of selecting the candidate reduction rate, and the process of determining the reduction rate until the number of times the process is performed or the reduction rate satisfies a predetermined condition; outputting the reduction rate determined when the predetermined condition is satisfied; The machine learning program according to any one of claims 1 to 7, which causes the computer to execute a process.

9. Calculating a threshold value of a tensor error before and after reduction of elements of each of a plurality of layers in a machine-learned model of a neural network including the plurality of layers; selecting a reduction ratio candidate to be applied to each of the plurality of layers based on the plurality of thresholds and an error of a tensor before and after reduction when the element is reduced by each of the plurality of reduction ratio candidates in each of the plurality of layers; determining a reduction rate to be applied to each of the plurality of layers based on an inference accuracy of the machine-learned model and an inference accuracy after machine learning of a reduced model obtained by reducing elements of each of the plurality of layers in the machine-learned model according to the candidate reduction rate to be applied; A machine learning method in which the processing is performed by a computer.

10. Calculating a threshold value of a tensor error before and after reduction of elements of each of a plurality of layers in a machine-learned model of a neural network including the plurality of layers; selecting a reduction ratio candidate to be applied to each of the plurality of layers based on the plurality of thresholds and an error of a tensor before and after reduction when the element is reduced by each of the plurality of reduction ratio candidates in each of the plurality of layers; determining a reduction rate to be applied to each of the plurality of layers based on an inference accuracy of the machine-learned model and an inference accuracy after machine learning of a reduced model obtained by reducing elements of each of the plurality of layers in the machine-learned model according to the candidate reduction rate to be applied; An information processing device comprising a control unit.

Images