Computer-readable recording medium, machine learning method, and information processing device

Abstract

A computer-readable recording medium, a machine learning method, and an information processing apparatus are disclosed. A computer-readable recording medium in which a program for causing a computer to execute a process is stored, the process including: calculating, for each element of a plurality of layers in a trained model of a neural network including the plurality of layers, a threshold of an error of a tensor between before reduction of one element and after reduction of the one element; selecting, based on the threshold and the error of the tensor between before reduction and after reduction in a case where the elements in each of the layers are reduced in each of reduction rate candidates, a reduction rate candidate to which one reduction rate candidate is to be applied to each of the layers; and determining a reduction rate to which one reduction rate is to be applied to each of the layers based on an inference accuracy of the trained model and an inference accuracy of a simplified model after machine learning, the simplified model being obtained by reducing each element of the layers in the trained model according to the reduction rate candidate to be applied.

Description

Technical Field

[0001] The implementations discussed herein relate to computer-readable recording media in which machine learning programs are stored, methods for machine learning, and information processing devices. Background Technology

[0002] Neural networks (NNs) used for AI (artificial intelligence) tasks such as image processing often leverage complex configurations to achieve high performance (e.g., high inference accuracy). On the other hand, the complex configuration of NNs can increase the number of computations performed by a calculator when executing NNs, as well as the amount of memory used when executing NNs by a calculator.

[0003] Pruning is known as a method for reducing the number of computations—in other words, shortening the computation time (speeding up)—and for reducing the size of memory—in other words, shrinking the machine learning model of a neural network.

[0004] Pruning is a method used to reduce the data size of a machine learning model and reduce computation and communication duration by reducing (pruning) at least one class of elements in the neural network, such as edges (weights), nodes, and channels.

[0005] Over-pruning can reduce the inference accuracy of a neural network (NN). Therefore, it is important to prune the NN while maintaining inference accuracy or while keeping the reduction in inference accuracy at a predetermined level.

[0006] For example, in pruning, known methods select layers that do not significantly affect the inference accuracy of the neural network. For instance, this method determines the channels of the convolutional layers to be pruned based on the parameters used in the batch normalization (BN) layers following the convolutional layers.

[0007] [List of Citations]

[0008] [Patent Literature]

[0009] [Patent Document 1] U.S. Patent Publication No. 2019 / 0057308.

[0010] [Patent Document 2] U.S. Patent Publication No. 2018 / 0232640.

[0011] [Patent Document 3] Japanese Patent Publication No. 2021-47854.

[0012] [Patent Document 4] Japanese Patent Publication No. 2020-123269.

[0013] [Patent Document 5] Japanese Patent Publication No. 2021-22050.

[0014] [Patent Document 6] U.S. Patent Publication No. 2019 / 0080238.

[0015] [Patent Document 7] U.S. Patent Publication No. 2019 / 0205759.

[0016] [Patent Document 8] U.S. Patent Publication No. 2020 / 0184333. Summary of the Invention

[0017] [The problem this invention aims to solve]

[0018] The method for selecting layers that will not significantly affect the inference accuracy of the neural network is applicable to convolutional layers connected to BN layers, but it is believed that the method is not applicable to other layers, such as convolutional layers or fully connected layers that are not connected to BN layers.

[0019] In one respect, the purpose of this disclosure is to achieve the reduction of neural networks including multiple layers.

[0020] [Problem-solving methods]

[0021] According to one aspect of the implementation, a computer-readable recording medium stores a machine learning program for causing a computer to perform processing, the processing comprising: calculating, for each element of a plurality of layers in a trained model of a neural network comprising multiple layers, a threshold of the error of a tensor between before and after the reduction of an element; selecting a reduction rate candidate to be applied to each of the plurality of layers based on the plurality of thresholds and the error of the tensor between before and after the reduction, in the case that elements in each of the plurality of layers are reduced according to each of a plurality of reduction rate candidates; and determining a reduction rate to be applied to each of the plurality of layers based on the inference accuracy of the trained model and the inference accuracy of a simplified model after machine learning, the simplified model being obtained by reducing each element of the plurality of layers in the trained model according to the reduction rate candidate to be applied.

[0022] [Effects of the Invention]

[0023] In one aspect, this disclosure enables the reduction of neural networks, including those with multiple layers. Attached Figure Description

[0024] Figure 1 This is a diagram illustrating an example of the process of determining the channels of a convolutional layer to be pruned.

[0025] Figure 2 This is a diagram illustrating an example of L1 regularization learning.

[0026] Figure 3 It is shown Figure 1 and Figure 2The diagram shows examples of whether the method applies to or not to layers of a neural network.

[0027] Figure 4 This is a block diagram illustrating an example of the functional configuration of a server according to one implementation.

[0028] Figure 5 This is a diagram illustrating an example of calculating a pruning rate that guarantees accuracy.

[0029] Figure 6 This is a diagram illustrating an example of calculating the accuracy of the model before and after pruning.

[0030] Figure 7 This is a graph showing an example of a search pruning rate.

[0031] Figure 8 This is a diagram illustrating an example of a method used to obtain a threshold.

[0032] Figure 9 This is a diagram showing an example of a threshold and its upper limit.

[0033] Figure 10 This is a diagram illustrating an example of a method used to determine the channels to be trimmed.

[0034] Figure 11 This is a diagram illustrating an example of calculating pruning error.

[0035] Figure 12 This is a diagram illustrating an example of a method for determining which nodes to trim.

[0036] Figure 13 This is a diagram illustrating an example of calculating pruning error.

[0037] Figure 14 This is a diagram illustrating an example of a method used to determine the weights to be pruned.

[0038] Figure 15 This is a diagram illustrating an example of calculating pruning error.

[0039] Figure 16 This is a flowchart illustrating an example of the operation of a server processing according to one implementation.

[0040] Figure 17 This is a diagram illustrating an example of a model that includes convolutional layers and fully connected layers that are not connected to BN layers.

[0041] Figure 18 This is a graph showing the data volume compression rate of the model data based on the pruning rate determined by a method according to one implementation.

[0042] Figure 19 It is shown Figure 18The results shown are an example of the relationship between the model's data size and the number of searches.

[0043] Figure 20 It is shown Figure 18 The graph shows an example of the relationship between the pruning rate (for the entire model) and the number of searches in the results.

[0044] Figure 21 It is shown Figure 18 The results shown are illustrated with a graph illustrating the relationship between the pruning rate (for each layer) and the number of searches.

[0045] Figure 22 It is shown Figure 17 A diagram showing an example of the number of channels or nodes in each layer and the pruning rate for each layer.

[0046] Figure 23 This is a graph showing the relationship between the number of searches for the pruning rate and the model size.

[0047] Figure 24 This is a diagram illustrating an example of the results of a pruning error comparison in response to an update of the trust radius in a method according to one embodiment.

[0048] Figure 25 This is a block diagram illustrating an example of the functional configuration of the server based on the first modification.

[0049] Figure 26 This is a diagram illustrating an example of the trust radius update process when the trust radius is increased.

[0050] Figure 27 This is a diagram illustrating an example of the trust radius update process when the trust radius is reduced.

[0051] Figure 28 This is a flowchart illustrating an example of the operation performed by the server based on the first modification.

[0052] Figure 29 This is a graph showing the relationship between the number of searches for the pruning rate and the model size.

[0053] Figure 30 This is a graph showing the relationship between the number of searches for the pruning rate and the model size, based on the initial value of the trust radius.

[0054] Figure 31 This is a block diagram illustrating an example of the functional configuration of the server according to the second modification.

[0055] Figure 32 This is a diagram illustrating an example of setting the initial value of the trust radius.

[0056] Figure 33 This is a flowchart illustrating an example of the operation performed by the server according to the second modification.

[0057] Figure 34 This is a graph showing the relationship between the number of searches for the pruning rate and the model size, based on the initial value of the trust radius.

[0058] Figure 35 This is a block diagram illustrating an example of a computer's hardware (HW) configuration. Detailed Implementation

[0059] In the following description, embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, the embodiments described below are merely illustrative and are not intended to exclude the application of various modifications and techniques not explicitly described in the embodiments. For example, various modifications and implementations can be made to the present embodiments without departing from their scope. In the drawings used in the following description, unless otherwise stated, the same reference numerals denote the same or similar parts.

[0060] <1> One implementation method

[0061] Figure 1 This is a diagram illustrating an example of the process for determining the channels of a convolutional layer to be pruned, and Figure 2 This is a diagram illustrating an example of L1 regularization learning. Figure 1 The following method is shown as a way to select layers that will not significantly affect the inference accuracy of the NN: In this method, the calculator uses the scaling factor γ used in the BN layer 100 after the convolutional layer to determine the channels of the convolutional layer to be pruned. Figure 1 The graphs shown in channels 111 to 113 represent the distribution of the output tensor.

[0062] like Figure 1 As depicted, the calculator performs normalization 101 for each of the multiple channels 111 (#1 to #n; n is an integer of 2 or greater) from the input of the convolutional layer to the BN layer 100. For example, during normalization 101, the calculator calculates the mean μ and variance σ for each channel 111 according to the following equation (1). 2 To obtain multiple channels 112 (#1 to #n) of a normalized distribution representing a mean "0" and a variance "1". In equation (1) below, z in and z mid These represent channels 111 and 112, respectively, and μ B and σ B 2 These represent the mean and variance of the current mini-batch B, respectively.

[0063] [Formula 1]

[0064]

[0065] The calculator performs scaling 102 for multiple channels 112 (#1 to #n). For example, during scaling 102, according to equation (2), the calculator multiplies each of the multiple channels 112 by a scaling factor γ and adds a bias β to the multiplication result to output multiple channels 113 (#1 to #n) representing the distribution scaled by parameters γ and β. In equation (2), z out This represents channel 113. Parameters γ and β can be optimized using machine learning.

[0066] [Equation 2]

[0067] z out =γz mid +β (2)

[0068] At this step, when γ is small, for channel 113 generated by scaling 102 ( Figure 1 In the example, channels #n) are almost eliminated from the output. This means that even by pruning and removing channels, the inference accuracy of the NN is not significantly affected. Therefore, the calculator identifies channels as pruning targets on a channel-by-channel basis by searching for small (e.g., "0") γ.

[0069] For example, a calculator searches for small (decreasing) γ by applying L1 regularization to γ. L1 regularization is a machine learning technique known to make the parameters to be learned “sparse” by performing machine learning while adding an L1 regularization term to a loss function computed by the neural network at the output.

[0070] like Figure 2 As shown, the calculator performs L1 regularization learning on vector 121 using loss function 122 to obtain vector 123 that has been L1 regularized. Loss function 122 can be a function L as represented by equation (3) below, which is obtained by adding the original loss function (first term) such as cross-entropy and the L1 regularization term (second term) using the L1 norm (Σg(γ)=Σ|γ|).

[0071] [Formula 3]

[0072] L=∑ (x,y) l(f(x,W),y)+λ∑ γ∈Γ g(γ) (3)

[0073] L1 regularization learning makes each parameter of vector 123 indicate (dichotomize) whether each parameter of vector 121 becomes zero or non-zero. By using such L1 regularization learning, the calculator can identify channels where γ becomes zero (close to zero) as channels for pruning.

[0074] use Figure 1 and Figure 2 The L1 regularization learning described in the paper to identify pruning targets is applicable to convolutional layers connected to BN layers, but is considered not applicable to other layers, such as convolutional layers not connected to BN layers and fully connected layers.

[0075] Figure 3 It is shown Figure 1 and Figure 2 The diagram illustrates whether the method applies to or not the examples of layers 131 to 139 of NN 130. For example... Figure 3 The convolutional layers 131 and 133 and the batch normalization (BN) layers 132 and 134 described in the text are... Figure 1 and Figure 2 The layers for which L1 regularization learning is applied, as described in the diagram, are convolutional layers 135 to 137 and fully connected layers 138 and 139. Figure 1 and Figure 2 The layers in which L1 regularization learning is not applicable, as described in the text.

[0076] In view of the above, one implementation describes a method for reducing the neural network by determining the pruning rate of each layer (regardless of the type of layer).

[0077] <1-1> Example of server functional configuration according to one implementation method

[0078] Figure 4 This is a block diagram illustrating an example of the functional configuration of server 1 according to one embodiment. Server 1 is an example of a calculator, computer, or information processing device that outputs a pruning rate. Figure 4 As shown, server 1 may exemplary include a memory unit 11, an acquisition unit 12, a machine learning unit 13, a pruning rate calculation unit (hereinafter referred to as the "computation unit") 14, and an output unit 15. The acquisition unit 12, machine learning unit 13, calculation unit 14, and output unit 15 are examples of control unit 16.

[0079] Memory unit 11 is an example of a storage area, and it stores various data to be used by server 1. For example... Figure 4 As shown, memory unit 11 can illustratively store untrained model 11a, data 11b for machine learning, trained model 11c, pruning rate 11d, and reduced model 11e.

[0080] The acquisition unit 12 acquires the untrained model 11a and the data 11b for machine learning, and stores them in the memory unit 11. For example, the acquisition unit 12 may generate one or both of the untrained model 11a and the data 11b for machine learning in the server 1, or it may receive them from a computer outside the server 1 via a network not shown.

[0081] The untrained model 11a can be a model of a neural network (NN) with untrained parameters included prior to machine learning. The NN can include various layers and can be, for example, a deep neural network (DNN). The NN can include, for example, convolutional layers or fully connected layers not connected to a batch normalization (BN) layer, or it can include convolutional layers connected to a BN layer. And, as an example, the NN can be... Figure 3 The NN 130 shown.

[0082] The data 11b used for machine learning can be, for example, a training dataset used for machine learning (training) an untrained model 11a. For example, when performing machine learning on a neural network used to implement image processing, the data 11b used for machine learning can include, for example, training data with multiple pairs of labels, which includes training data such as image data and real labels for the training data.

[0083] During the machine learning phase, machine learning unit 13 performs machine learning processing on the untrained model 11a based on the machine learning data 11b. For example, machine learning unit 13 can generate a trained model 11c from the machine learning processing of the untrained model 11a. The trained model 11c can be an NN model that includes training parameters.

[0084] The trained model 11c can be obtained by updating the parameters included in the untrained model 11a, and can be viewed, for example, as a model resulting from machine learning processing that transforms the untrained model 11a into the trained model 11c. Machine learning processing can be implemented using various known techniques.

[0085] The calculation unit 14 calculates the pruning rate 11d by performing pruning rate calculation processing on the training model 11c, and stores them in the memory unit 11.

[0086] For example, the calculation unit 14 may include: a threshold calculation unit 14a, which calculates a threshold for selecting one of the pruning rate candidates for each layer; and a determination unit 14b, which determines the pruning rate 11d to be used based on the inference accuracy of the model pruned with the pruning rate candidates.

[0087] Output unit 15 outputs output data based on the pruning rate 11d generated (obtained) by calculation unit 14. The output data may include, for example, the pruning rate 11d itself, the reduction model 11e, or both.

[0088] The reduced model 11e is data from a reduced version of the training model 11c, obtained by pruning the training model 11c based on a pruning rate 11d. For example, in collaboration with the machine learning unit 13, the output unit 15 can obtain the reduced model 11e by performing pruning and relearning on the training model 11c while applying the pruning rate 11d, and can store the obtained model in the memory unit 11. The reduced model 11e can be generated separately from the training model 11c, for example, or it can be updated data of the training model 11c obtained through pruning and relearning.

[0089] When outputting output data, output unit 15 may, for example, transmit (provide) the output data to another computer (not shown), or it may store the output data in memory unit 11 and manage the output data so that it can be retrieved from server 1 or another computer. Alternatively, when outputting output data, output unit 15 may display information indicating the output data on an output device such as server 1, or it may output the output data in various other ways.

[0090] <1-2> Example of pruning rate calculation

[0091] Next, an example of the pruning rate calculation process performed by the computing unit 14 of server 1 will be described. In the following description, it is assumed that the objective of the pruning rate calculation is a weight matrix W, which is an example of the parameters of the layer.

[0092] The calculation unit 14 determines the pruning rate by using the error of the tensor of each layer, regardless of the layer type, which is caused by pruning. As an example, the calculation unit 14 can calculate the pruning rate according to the following procedures (i) to (iii).

[0093] (i) Calculation unit 14 (threshold calculation unit 14a) determines (calculates) a pruning rate that ensures accuracy for each layer.

[0094] The term "guaranteed accuracy" means, for example, guaranteeing that the accuracy of inference using the reduced model 11e (inference accuracy) exceeds a predetermined standard, wherein the reduced model 11e is obtained by pruning the training model 11c.

[0095] Figure 5 This is a graph illustrating an example of calculating a pruning rate that guarantees accuracy. (Example) Figure 5 As shown, in (i), the threshold calculation unit 14a determines the pruning rate to be applied to the weight matrix W of each layer included in the training model 11c with the pruning objective for each weight matrix W of multiple layers. Although Figure 5 Focus on layers 131 to 133, but Figure 5The application of the description is not limited to these layers, and can be Figure 3 Any of the layers 131 to 139 shown.

[0096] Here, the pruning rate is an example of the ratio (reduction rate) used to reduce the number of elements in a layer, and indicates the ratio used to make the pruning objective "sparse" in the trained model 11c. Figure 2 In the example, the pruning rate corresponds to the number of positions in vector 123 that are set to "0".

[0097] like Figure 5 As shown, the threshold calculation unit 14a selects a pruning rate from a plurality of pruning rate candidates for each of the weight matrix W1 of layer 131 (connected to the weight matrix W1 of layer 132) and the weight matrix W2 of layer 132 (connected to the weight matrix W2 of layer 133). The pruning rate candidates are examples of reduction rate candidates and can be, for example, two or more ratios between 0% and 100%, ratios common to multiple layers, ratios different in each layer, or combinations thereof. Figure 5 In the example, we assume that the pruning rate candidates are 0%, 20%, 40%, and 60%.

[0098] For example, when performing pruning for each pruning rate candidate, the threshold calculation unit 14a obtains the error of the tensor between before and after pruning, and determines that the pruning rate candidate with an error less than the threshold T. W Candidates for the maximum pruning rate. Figure 5 In the example, for W1, the threshold calculation unit 14a determines that the error is less than the threshold T. w1 The maximum pruning rate candidate is 40% (see arrow 141). Additionally, for W2, the threshold calculation unit 14a determines the error to be less than the threshold T. w2 The maximum pruning rate candidate is 20% (see arrow 142).

[0099] Threshold T w This is a threshold for the error in the tensor between before and after pruning, and an upper limit for the pruning rate that guarantees accuracy. For example, the threshold calculation unit 14a can calculate the threshold T for each layer by representing the loss function when pruning the pruning target using an approximation such as a first-order Taylor expansion. w The method for calculating the threshold T will be described later. w Details of the method.

[0100] The pruning rate calculated in (i) can be regarded as a “temporarily calculated” pruning rate in relation to the treatments in (ii) and (iii).

[0101] As described above, the threshold calculation unit 14a calculates a threshold T for the tensor error between before and after reducing an element for each element in the training model 11c of the NN comprising multiple layers. In the case where an element in each of the multiple layers is reduced by each of the multiple reduction rate candidates, the threshold calculation unit 14a selects a reduction rate candidate to apply a reduction rate candidate to each of the multiple layers based on the multiple thresholds T and the tensor error between before and after the reduction.

[0102] (ii) Calculation unit 14 (determination unit 14b) determines the pruning rate based on the accuracy of the machine learning model pruned (reduced) by using the pruning rate determined in (i) and the accuracy of the machine learning model that has not undergone pruning.

[0103] For example, determining unit 14b takes into account the error caused by the approximate representation (first-order Taylor expansion), and the accuracy Acc of the model pruned at the pruning rate determined for each layer in (i) is... p and accuracy margin Acc m The sum of the accuracy Acc of the unpruned model wo Compare. Accuracy margin (Acc) m This is a margin that allows for a decrease in inference accuracy, and it can be set by the designer. The margin can be "0", and in this case, determining that unit 14b can reduce accuracy Acc p Accuracy with the unpruned model wo Compare them.

[0104] Figure 6 This is a diagram illustrating an example of calculating the accuracy of the model before and after pruning. For example, determination unit 14b calculates the accuracy Acc of the unpruned model (trained model 11c) for all layers (W1, W2, ...). wo (See arrow 143). The unpruned model can be considered as a model pruned at a pruning rate of 0% for each layer. Determine unit 14b calculates the accuracy Acc of the model pruned at the pruning rate (W1 = 40%, W2 = 20%, ...) calculated for each layer via (i). p (See arrow 144.)

[0105] If the sum of accuracy Acc p +Acc m Acc, equal to or higher than accuracy wo Then, the determining unit 14b determines the pruning rate to be used as determined in (i). For example, the determining unit 14b stores the pruning rate determined in (i) as pruning rate 11d in the memory unit 11.

[0106] On the other hand, if the sum of accuracy Accp +Acc m Below Accuracy wo Then, determining unit 14b determines to discard the pruning rate determined in (i). For example, determining unit 14b discards the pruning rate determined in (i) and determines to adopt the pruning rate 11d (or the initial pruning rate 11d) determined in the most recent (ii).

[0107] (iii) Calculation unit 14 (determination unit 14b) repeats (i) and (ii) multiple times to search for the maximum pruning rate that can guarantee accuracy.

[0108] Figure 7 This is a graph showing an example of a search pruning rate. Figure 7 The example shows the case where the calculation unit 14 uses the pruning rate of three layers (131 to 133) three times.

[0109] like Figure 7 As shown, in the first search (see reference numeral 145), in (i), it is assumed that the threshold calculation unit 14a calculates the threshold T. w And based on the threshold T w The pruning rate of layers 131 to 133 will change from "0%, 0%, 0%" (initial value) to "40%, 20%, 40%". For example, in (ii), if determining unit 14b determines Acc when comparing inference accuracy... p +Acc m <Acc wo Then, the pruning rate determined in (i) is discarded by unit 14b, and “0%, 0%, 0%” is used as the value before determination.

[0110] In the second search (see figure 146), in (i), it is assumed that threshold calculation unit 14a calculates (updates) the threshold T. w And based on the updated threshold T w The pruning rate of layers 131 to 133 will change from "0%, 0%, 0%" to "20%, 20%, 40%". For example, in (ii), if determining unit 14b determines Acc when comparing inference accuracy... p +Acc m ≥Acc wo Then, it is determined that unit 14b adopts "20%, 20%, 40%" and stores them as trimming rate 11d in memory unit 11.

[0111] In the third search (see figure 147), in (i), it is assumed that threshold calculation unit 14a calculates (updates) the threshold T. w And based on the updated threshold T wThe pruning rate for layers 131 to 133 will change from "20%, 20%, 40%" to "20%, 40%, 40%". For example, in (ii), if determining unit 14b determines Acc when comparing inference accuracy... p +Acc m ≥Acc wo Then, it is determined that unit 14b adopts "20%, 40%, 40%", and they are stored (updated) as trimming rate 11d in memory unit 11.

[0112] Unit 14b can search for a predetermined number of pruning rates, such as a preset number.

[0113] As described above, determining unit 14b determines a reduction rate to be applied to each of the multiple layers based on the inference accuracy of the trained model 11c and the inference accuracy of the simplified model after machine learning. The simplified model is obtained by reducing each element of the multiple layers in the trained model 11c according to the reduction rate candidate to be applied.

[0114] Next, a specific example of the above pruning rate calculation process will be described. Figure 8 This is a diagram illustrating an example of a method used to obtain the threshold, and Figure 9 This is a diagram showing an example of a threshold and its upper limit.

[0115] The threshold calculation unit 14a performs a first-order Taylor expansion on the loss function during pruning to calculate the threshold of the pruning rate for each layer that guarantees accuracy. For example, assume: the error of the tensor in each layer is Δw, which is caused by pruning; the loss function during pruning is L(w+Δw); the loss function of the model targeted for pruning is L(w); and the loss function without pruning is (L...). ideal ) for L wo +L m The threshold for the pruning rate that ensures accuracy is calculated by the following formula (4). It should be noted that L... wo It is the loss function of the unpruned model, and L m It is the margin of the loss function set by the designer.

[0116] [Formula 4]

[0117]

[0118] The left side of equation (4) above (see Figure 8 The dashed box in the figure represents the Taylor expansion of the loss function L(w+Δw) in pruning, and includes the weight gradients of each layer of the pruning target. The gradient of each layer can be calculated through backpropagation. The right side of equation (4) above (see...) Figure 8The dashed-dot box in the diagram represents a constraint that the loss function is less than the ideal value (e.g., the loss function of FP32) even when pruning is performed.

[0119] As described above, the threshold calculation unit 14a calculates the threshold T based on the value of the loss function of the model 11c trained when reducing the elements of each of the multiple layers and the weight gradient of each of the multiple layers.

[0120] Rearranging equation (4) above, we can obtain the condition for "pruning error" as expressed by equation (5) below, which satisfies the constraint that the loss function in pruning is less than the ideal loss function. In other words, we can obtain the upper limit (threshold) of the error caused by pruning, which guarantees the accuracy (loss function). The threshold calculation unit 14a sets the right side of equation (5) below as the threshold T.

[0121] [Formula 5]

[0122]

[0123] like Figure 9 As shown, the threshold calculation unit 14a compares the threshold T set for each layer with the error in the form of the L1 norm caused by pruning. Then, the threshold calculation unit 14a determines the maximum value among the pruning rate candidates whose error is less than the threshold T (in...). Figure 9 In the example, 40% is the candidate pruning rate, which is the pruning rate obtained by (i).

[0124] As an example, according to Equation (6), the threshold calculation unit 14a can determine the pruning rate for each layer of the pruning target that makes the pruning error (left side) equal to or less than the threshold (right side). In Equation (6), "||ΔW||1" is the L1 norm of the weights to be considered as the pruning target, and "n" is the number of elements of the weights of the layers in the pruning target.

[0125] [Formula 6]

[0126]

[0127] As shown in equation (6) above, the threshold T is a parameter obtained through approximation. To prevent errors caused by approximation errors when determining the pruning rate, an upper limit can be set for the threshold T (see [reference]). Figure 9 For example, threshold calculation unit 14a can limit the size of the threshold T based on the trust region method using a "trust radius". The trust radius is an example of an upper limit for the threshold. As an example, threshold calculation unit 14a can scale the threshold T such that the L2 norm of the threshold T for all layers becomes equal to or less than the trust radius. Figure 9 In the example, T h This represents a vector based on the threshold T for each layer, and "||T h"||2" represents the L2 norm of the threshold T for all layers.

[0128] For example, based on the accuracy comparison results of the determination unit 14b in the processing of (ii), the threshold calculation unit 14a may update the confidence radius (e.g., by multiplying the confidence radius by a constant factor, etc.) in addition to updating the pruning rate. The initial value of the confidence radius may be set by, for example, the designer, etc.

[0129] As an example, if the sum of accuracy Acc p +Acc m Acc, equal to or higher than accuracy wo Then the threshold calculation unit 14a can multiply the confidence radius by a constant k ("k>1.0"), and if the sum of accuracies Acc p +Acc m Below Accuracy wo Then the threshold calculation unit 14a can multiply the trust radius by a constant k (“0 <k<1.0”)。

[0130] <1-3> Description based on the type of pruning target

[0131] Next, examples of methods for performing pruning and calculating pruning error will be described, depending on the type of pruning target. The type of pruning target can be, for example, channel pruning, node pruning, weight pruning, etc. Depending on the type of pruning target, calculation unit 14 can determine the pruning target and pruning error by utilizing weights corresponding to the pruning target.

[0132] <1-3-1> Example of channel trimming

[0133] Figure 10 This is a diagram illustrating an example of a method for determining the channels to be trimmed, and Figure 11 This is a diagram illustrating an example of calculating pruning error.

[0134] Figure 10 and Figure 11 The processing flow of a convolution operation is illustrated. Subscripts H and W indicate the size of the input, kernel, and output data, and subscript Ch indicates the number of channels in the input, kernel, and output data. This also applies to the description of other types of pruning targets below.

[0135] (An example of a method for determining the channels to be trimmed)

[0136] When the type of the pruning target is channel, the calculation unit 14 calculates the L1 norm on a per-channel basis. For example, Figure 10As shown in "Before pruning", computation unit 14 calculates the corresponding L1 norms of all Ch1 kernels before pruning. Therefore, the Ch1 L1 norms are calculated.

[0137] Next, as Figure 10 As shown in "After trimming", the calculation unit 14 trims the corresponding output data channels according to the set trimming rate in ascending order of the calculated L1 norm.

[0138] (Example of calculating pruning error)

[0139] like Figure 11 As shown, the calculation unit 14 calculates the L1 norm of the kernel of the pruning target. The L1 norm of the kernel of the pruning target is obtained by subtracting the L1 norm of all kernels after pruning from the L1 norm of all kernels before pruning, that is, the difference between the L1 norm before pruning and the L1 norm after pruning.

[0140] The calculation unit 14 can obtain the pruning error by dividing the calculated Ll norm by the number of elements in all kernels before pruning.

[0141] <1-3-2> Example of node pruning

[0142] Figure 12 This is a diagram illustrating an example of a method for determining which nodes to prune, and Figure 13 This is a diagram illustrating an example of calculating pruning error.

[0143] (An example of a method for determining the nodes to be pruned)

[0144] When the pruning target is a node, computation unit 14 calculates the L1 norm in units of weights connected to the output node. Figure 12 In the example “before trimming”, calculation unit 14 calculates the L1 norm in units of solid lines, dashed lines and dotted lines respectively.

[0145] Next, as by Figure 12 As shown in the section "After pruning", the calculation unit 14 prunes the corresponding output nodes according to the set pruning rate, based on the ascending order of the calculated L1 norm. For example, the calculation unit 14 determines that the output node corresponding to the weighted group with the smaller L1 norm is the node to be pruned.

[0146] (Example of calculating pruning error)

[0147] like Figure 13 As shown, the calculation unit 14 calculates the L1 norm of the weight group of the pruning target. The L1 norm of the weight group of the pruning target is obtained by subtracting the L1 norm of all weights after pruning from the L1 norm of all weights before pruning.

[0148] The calculation unit 14 can obtain the pruning error by dividing the calculated L1 norm by the number of elements with all weights before pruning. Figure 13 In the example “After pruning”, the calculation unit 14 calculates the L1 norm of the weight group indicated by the dotted line and divides the L1 norm by the number of elements of all weights before pruning (=“6”; number of lines).

[0149] <1-3-3> Example of weight pruning

[0150] Figure 14 This is a diagram illustrating an example of a method for determining the weights to be pruned, and Figure 15 This is a diagram illustrating an example of calculating pruning error.

[0151] (An example of a method for determining the weights to be pruned)

[0152] When the pruning target type is weighted, calculation unit 14 calculates the L1 norm element-wise for all weights. Figure 14 In the example of "before pruning", since the number of elements of the weight is "6", the calculation unit 14 calculates "6" L1 norms.

[0153] Next, as by Figure 14 As shown in the section "After pruning", the calculation unit 14 prunes the corresponding weights according to the set pruning rate, based on the ascending order of the calculated L1 norm. For example, the calculation unit 14 determines that the weights with smaller L1 norms are the weights to be pruned.

[0154] (Example of calculating pruning error)

[0155] like Figure 15 As shown, the calculation unit 14 calculates the L1 norm of the weights of the pruning target. The L1 norm of the weights of the pruning target is obtained by subtracting the L1 norm of all weights after pruning from the L1 norm of all weights before pruning.

[0156] The calculation unit 14 can obtain the pruning error by dividing the calculated L1 norm by the number of elements with all weights before pruning. Figure 15 In the example “After pruning”, calculation unit 14 calculates the L1 norm of the weights indicated by the dashed lines and divides the L1 norm by the number of elements with all weights before pruning (=“6”; number of lines).

[0157] <1-4> Operation Examples

[0158] Next, we will refer to Figure 16 This describes an operational example of server 1 according to one implementation. Figure 16 This is a flowchart illustrating an operational example of the processing of server 1 according to one embodiment.

[0159] like Figure 16 As shown, in step S1, the machine learning unit 13 performs machine learning on the untrained model 11a obtained by the obtaining unit 12 without pruning.

[0160] Calculation unit 14 calculates the inference accuracy (recognition rate) Acc without pruning. wo (Step S2)

[0161] The threshold calculation unit 14a sets the initial value of the trust radius (step S3).

[0162] Threshold calculation unit 14a calculates the threshold T and the pruning error for each layer to set the pruning rate (step S4) and determines whether the L2 norm of the threshold T for all layers is greater than the confidence radius (step S5). If the L2 norm of the threshold T for all layers is equal to or less than the confidence radius ("No" in step S5), the process proceeds to step S7.

[0163] If the L2 norm of the threshold T of all layers is greater than the confidence radius (yes in step S5), then the threshold calculation unit 14a scales (updates) the threshold so that the L2 norm of the threshold T of all layers becomes equal to the confidence radius (step S6), and the process proceeds to step S7.

[0164] In step S7, the threshold calculation unit 14a temporarily calculates the pruning rate for each layer. For example, the threshold calculation unit 14a temporarily sets the pruning rate for each layer from the set pruning rate candidates. Steps S4 to S7 are examples of the processing described in (i) above.

[0165] Machine learning unit 13 prunes the trained model 11c using a pruning rate temporarily calculated by threshold calculation unit 14a, and then performs machine learning again on the model after pruning. Calculation unit 14 calculates the model's inference accuracy Acc after re-performing machine learning. p (Step S8)

[0166] Determine the accuracy of reasoning in unit 14b (Acc). p + Balance Acc m Is it equal to or higher than the inference accuracy Acc? wo (Step S9). The evaluation of inference accuracy (recognition rate) can compensate for errors caused by approximation errors when selecting the pruning rate.

[0167] If the inference accuracy Acc p + Balance Acc m Equal to or higher than the inference accuracy Acc woIf the condition is "yes" in step S9, then the determining unit 14b determines to prune the training model 11c with a temporarily calculated pruning rate (step S10), and stores the temporarily calculated pruning rate as pruning rate 11d in the memory unit 11. Furthermore, the threshold calculation unit 14a increases the confidence radius by multiplying the confidence radius by a constant factor (step S11), and the process proceeds to step S14.

[0168] On the other hand, if the inference accuracy Acc p + Balance Acc m Below the inference accuracy Acc wo If no result is found in step S9, then the determination unit 14b discards the temporarily calculated pruning rate (step S12). The threshold calculation unit 14a reduces the confidence radius by multiplying the confidence radius by a constant factor (step S13), and the process proceeds to step S14. Steps S8 to S13 are examples of the process described in (ii) above.

[0169] In step S14, the determining unit 14b determines whether the search (processing in steps S4 to S13) has been executed a predetermined number of times; in other words, whether the predetermined conditions are met for the number of executions of the processes including threshold calculation, pruning rate candidate selection, and pruning rate determination. If the search has not been executed a predetermined number of times (no in step S14), the process moves to step S4.

[0170] If the search has been performed a predetermined number of times (yes in step S14), then output unit 15 outputs the determined pruning rate 11d (step S15), and the process ends. Step S14 is an example of the process described in (iii) above.

[0171] As described above, the threshold calculation unit 14a calculates the error of the tensor used for the neural network (the error is generated by pruning) according to one embodiment of the server 1, and generates a threshold based on the value of the loss function and the gradient obtained through backpropagation of the neural network. Furthermore, the threshold calculation unit 14a compares the calculated error in the pruning with the threshold to temporarily calculate the pruning rate. Additionally, the determination unit 14b compares the inference accuracy of the model after relearning using the calculated pruning rate with the inference accuracy of the unpruned model, and determines the pruning rate for each layer. At this point, if it is determined that the inference accuracy with pruning deteriorates compared to the inference accuracy without pruning, the threshold calculation unit 14a resets the upper limit of the threshold, causing the threshold to decrease, and searches for the pruning rate again.

[0172] Therefore, according to one embodiment, server 1 can determine the pruning rate for each layer, regardless of the layer type. For example, server 1 can determine the pruning rate to be applied to a training model 11c, which includes convolutional layers without BN layers, fully connected layers, etc.

[0173] Figure 17 This is a diagram illustrating an example of a model 150 including convolutional layers with unconnected batch normalization (BN) layers and fully connected layers. According to one implementation, different types of pruning targets can be applied to each layer. Figure 17 The example shows the case where the pruning objective for convolutional layers 151 to 155, which are not connected to BN layers, is channel pruning, and the pruning objective for fully connected layers 156 to 158 is node pruning. Because pruning the output nodes of the final layer (layer 158) disables classification, the final layer does not undergo pruning.

[0174] Figure 18 This is a graph showing the data volume compression rate of the model data based on the pruning rate determined by a method according to one implementation. Figure 18 The illustration depicts "CIFAR-10" as a dataset used for machine learning data 11b and has... Figure 17 The "AlexNet" configuration shown is used for both untrained model 11a and trained model 11c.

[0175] Figure 18 The example illustrates the following scenario: the mini-batch size is "64" and the "optimizer hyperparameters" when re-performing machine learning are: learning rate "0.001"; weight decay "1e-4"; momentum "0.9"; pruning rate candidates for each search "20%, 10%, 0%"; and margin Acc for recognition rate. m "1%, 2%"; and the number of searches "20".

[0176] like Figure 18 As shown, the method according to one embodiment can ensure a data compression rate of "94%" or higher for model data, while maintaining the recognition rate Acc in the unpruned case. wo Recognition rate Acc in cases with pruning p The difference between them is kept within a margin of "1% or 2%". m Inside, or kept lower.

[0177] Figures 19 to 21 It shows the basis respectively. Figure 18 The results shown (where the margin Acc) m A graph illustrating the relationship between the data size of model 150 (1%), the pruning rate (for the entire model), the pruning rate (for each layer), and the number of searches. Figure 22 It is shown Figure 17A diagram showing the number of channels or nodes in layers 151 to 157 (convolutional layers 1 to 5 (Conv1 to Conv5), fully connected layers 1 to 2 (fc1 to fc2)) (top row) and examples of the pruning rate for each layer (bottom row).

[0178] like Figures 19 to 22 As shown, the relationship between the number of searches and data size or pruning rate suggests that as the search progresses, the pruning rate increases, and the data size is compressed. For example, as... Figure 22 As shown, the pruning rate of the layers closer to the input layer (layer 151 side) is higher than that of the layers closer to the input layer (layer 151 side). Therefore, in Figure 17 In the case of model 150 shown, since layers closer to the input layer have a greater impact on accuracy than layers closer to the output layer, the effect of compressing the model data volume can be enhanced by determining the pruning rate for each layer.

[0179] As described above, by using a method according to one embodiment, since the pruning rate can be determined for each layer and the pruning rate can be output, the reduction of NNs comprising multiple different layers can be achieved.

[0180] <1-5> Modifications

[0181] Next, modifications according to one implementation will be described. For simplicity, the following description assumes a margin of accuracy Acc for inference. m The value is "0", in other words, when comparing the accuracy of reasoning, the accuracy of reasoning (Acc) is determined. p Is it equal to or higher than the inference accuracy Acc? wo .

[0182] <1-5-1> First Revision

[0183] In one implementation of the method, the number of searches for the pruning rate (the number of attempts in process (iii)) is a hyperparameter set manually, for example, by the designer. As a result, for example, if the number of searches is set small, the trained model 11c may be under-reduced, and if the number of searches is set large, the trained model 11c can be sufficiently reduced, but the search duration may become longer.

[0184] Figure 23 This is a graph showing the relationship between the number of searches for the pruning rate and the model size. For example... Figure 23 As shown, with a search count of "200", the model size saturates at "5MB" on the "50th" search, indicating that searches "51 to 200" are unnecessary. On the other hand, the model size remains around "11MB" on the "20th" search, indicating that there is room for further reduction.

[0185] Figure 24 This is a diagram illustrating an example of the results of a comparison of pruning errors in response to an update of the trust radius in a method according to one embodiment.

[0186] like Figure 24 As shown, in the result of the error comparison of the "m"th search (where m is an integer equal to or greater than "1"), it is assumed that a pruning rate of "10%" is calculated (determined). In this case, the confidence radius is updated to increase it by multiplying by a constant K. However, if the updated confidence radius is less than the error based on a pruning rate candidate that is one size larger than the pruning rate candidate determined in the "m"th search, then the pruning rate of "10%" is calculated again even in the result of the error comparison of the "m+1"th search.

[0187] Thus, when the confidence radius is multiplied by a constant K or k, the amount of threshold update is limited by the confidence radius, making it possible to use the same pruning rate candidate in multiple searches. This state of searching for the same combination of pruning rates multiple times leads to an increase in the number of pruning rate searches, while suppressing the model's pruning efforts from being fully attempted.

[0188] In view of this, the first modification describes a method for shortening (reducing) the search duration (number of searches) for a pruning rate suitable for reducing the NN by focusing on updating the trust radius.

[0189] Figure 25 This is a block diagram illustrating an example of the functional configuration of server 1A according to the first modification. (See diagram for example.) Figure 25 As shown, server 1A may include different Figure 4 The computing unit 14A of server 1. The computing unit 14A may include different from the computing unit 14A. Figure 4 The calculation unit 14 includes a threshold calculation unit 14a' and a determination unit 14b'.

[0190] In each search, calculation unit 14A searches for combinations of different pruning rates. A selected combination having a pruning rate of "0%" for all layers indicates that calculation unit 14A has determined to no longer search for pruning rates. Under this premise, calculation unit 14A (determination unit 14b') terminates the search when a combination with a pruning rate of "0%" for all layers is selected.

[0191] Based on the comparison results of the inference accuracy of determining unit 14b', threshold calculation unit 14a' measures the absolute value of the threshold and the following error, "E", for each layer i (i is an integer equal to or greater than 1). diff,i "The error is either a larger error than the searched pruning rate or the error of the searched pruning rate."

[0192] For example, when the inference accuracy Acc p Equal to or higher than the inference accuracy Acc wo At that time, the threshold calculation unit 14a' measures the absolute value of the difference between the threshold and the error of a pruning rate that is one size larger than the searched pruning rate. diff,i ".

[0193] On the other hand, when the inference accuracy Acc p Below the inference accuracy Acc wo At that time, the threshold calculation unit 14a' measures the absolute value of the different quantities "E" between the error between the threshold and the searched pruning rate. diff,i ".

[0194] As shown by the following equation (7), the threshold calculation unit 14a' calculates the absolute values ​​"E" of different quantities from all layers. diff,i Find the minimum value (distinct quantity) in "E". diff ".

[0195] E diff =min(E diff，1 E diff，2 , ..., E diff，i (7)

[0196] Based on the comparison results of the inference accuracy of the determination unit 14b', the threshold calculation unit 14a' calculates the threshold by multiplying the confidence radius by a constant factor and by using the confidence radius with different quantities "E". diff The confidence radius is updated by using the one with the larger change between the sum or difference of the two terms.

[0197] For example, when the inference accuracy Acc p Equal to or higher than the inference accuracy Acc wo At that time, the threshold calculation unit 14a' uses the trust radius multiplied by a constant K and the trust radius multiplied by different quantities "E". diff The one with the larger change in the sum, and therefore, the trust radius is updated to increase the trust radius.

[0198] On the other hand, when the inference accuracy Acc p Below the inference accuracy Acc wo At that time, the threshold calculation unit 14a' uses the trust radius multiplied by a constant k and the trust radius with different quantities "E". diff The one with the larger change in the difference is the trust radius, and therefore, the trust radius is updated to reduce the trust radius.

[0199] In this way, the threshold calculation unit 14a' updates the trust radius so that the combination of pruning rate candidates from multiple layers is different each time a selection (in other words, search) of pruning rate candidates is performed.

[0200] Figure 26 This is a diagram illustrating an example of the trust radius update process when the trust radius is increased. For example... Figure 26 As shown, assume the pruning rate found in the "m"th search is "(layer 1, layer 2) = (10%, 0%)". Threshold calculation unit 14a' calculates the absolute value "E" of the different amounts between the confidence radius and the error of the pruning rate "20%" for layer 1. diff,1 ", and the absolute value of the different amounts between the confidence radius and the error of the pruning rate "10%" for layer 2, "E". diff,2 According to the above formula (7), the threshold calculation unit 14a' obtains different quantities "E" with smaller values. diff,2 "As "E diff ".

[0201] Then, the threshold calculation unit 14a' determines (updates) the confidence radius for the “m+1”th (next) time according to the following formula (8).

[0202] (Reliance radius of the "m+1"th iteration) = max((Reliance radius of the "m"th iteration · constant K), (Reliance radius of the "m"th iteration + E) diff (8)

[0203] As a result, in the "m+1"th iteration, a value at least equal to or greater than the sum of the confidence radius and the different quantities is selected as the confidence radius, such that in the "m+1"th iteration, the bit width different from that in the "m"th iteration is calculated as the pruning rate.

[0204] exist Figure 26 In the example, the confidence radius (the upper limit of the threshold) of the "m+1"th search is consistent with the error of the pruning rate "10%" for layer 2. Therefore, at the "m+1"th search, the search is for a pruning rate "(layer 1, layer 2) = (10%, 10%)" that constitutes a combination of pruning rates different from the previous one.

[0205] Figure 27 This is a diagram illustrating an example of the trust radius update process when the trust radius is reduced. For example... Figure 27 As shown, assume the pruning rate for the "m"th search is "(layer 1, layer 2) = (10%, 0%)". Threshold calculation unit 14a' calculates the absolute value "E" of the different quantities between the confidence radius and the error of the pruning rate "10%" for layer 1. diff,1 ", and the absolute value of the different amounts between the confidence radius and the error of the pruning rate "0%" for layer 2, "E". diff,2 According to the above formula (7), the threshold calculation unit 14a' obtains different quantities "E" with smaller values. diff,1 "As "E diff ".

[0206] Then, the threshold calculation unit 14a' determines (updates) the confidence radius for the “m+1”th (next) time according to the following formula (9).

[0207] (Reliance radius of the "m+1"th iteration) = max((Reliance radius of the "m"th iteration · constant factor), (Reliance radius of the "m"th iteration - E) diff (9)

[0208] As a result, in the "m+1"th iteration, a value at least equal to or greater than the "difference between the confidence radius and the different quantities" is selected as the confidence radius, such that in the "m+1"th iteration, the bit width different from that in the "m"th iteration is calculated as the pruning rate.

[0209] exist Figure 27 In the example, the confidence radius (the upper limit of the threshold) of the "m+1"th search is consistent with the error of the pruning rate "0%" for layer 1. Therefore, at the "m+1"th search, the search is for a pruning rate "(layer 1, layer 2) = (0%, 0%)" that constitutes a combination of pruning rates different from the previous one.

[0210] When generalizing equations (8) and (9) above, the next confidence radius can be represented by equation (10).

[0211] The next confidence radius = current confidence radius * max(constant factor, Qscale_min) (10)

[0213] In equation (10) above, the constant factor is K or k, "Qscale_min" is "Qscale" as expressed by equation (11) below, and "Qscale" is expressed by equation (12) below.

[0214] Qscale_min = min(Qscale calculated across all quantized target vectors) (11)

[0215] Qscale = 1 + Qdiff / Qth (12)

[0216] In Equation (12) above, “Qdiff” is “the difference between the threshold and the quantization error of a bit width that is one size narrower than the temporarily calculated bit width (pruning rate),” and “Qth” is the threshold.

[0217] Next, refer to Figure 28 This will describe an operational example based on the first modified server 1A. Figure 28 This is a flowchart illustrating an example of the operation performed via server 1A according to the first modification. Figure 28 Corresponding to this flowchart, steps S21, S22, and S23 are respectively replaced with steps S21, S22, and S23 in the flowchart. Figure 16Steps S11, S13, and S14 are shown in the flowchart of server 1. Also in the first modification, in step S3, the threshold calculation unit 14a' sets the initial value of the trust radius.

[0218] In step S21, the threshold calculation unit 14a' increases the confidence radius by using the product with the constant K and the larger of the sum of different quantities, and the process proceeds to step S23.

[0219] In step S22, the threshold calculation unit 14a' reduces the confidence radius by using the larger of the product with the constant k and the difference between the different quantities, and the process proceeds to step S23.

[0220] In step S23, determining unit 14b' determines whether the trimming rate 11d of all layers is "0%", in other words, whether these trimming rates meet a predetermined condition. If the trimming rate 11d of at least one layer is not "0%" (no in step S23), the process moves to step S4.

[0221] If the pruning rate 11d of all layers is "0%" (yes in step S23), then the output unit 15 outputs the determined pruning rate 11d (step S15), and the processing ends.

[0222] As described above, the first modification differs from one implementation in that the method for updating the confidence radius is via the threshold calculation unit 14a' and the termination condition for ending the search is determined via the determination unit 14b'. Therefore, server 1A can search for a pruning rate suitable for sufficiently reducing the NN within the shortest possible duration (fewest attempts). Furthermore, the number of search attempts set (specified) by the designer can be omitted.

[0223] Figure 29 This is a graph showing the relationship between the number of searches for the pruning rate and the model size. For example... Figure 29 As shown, according to the first modification, the search ends near the "50th" search when the pruning rate of all layers reaches "0%", in other words, when the model size is sufficiently reduced (reaching a saturation of "5MB"). Therefore, when the number of searches is specified, for example, as "200", unnecessary searches from "51 to 200" can be prevented, which can optimize the search duration (number of searches).

[0224] <1-5-2> Second Revision

[0225] In one implementation and the first modified method, the initial value of the trust radius is a hyperparameter set by the designer, etc.

[0226] Figure 30This is a graph showing the relationship between the number of searches for the pruning rate and the model size, based on the initial value of the trust radius. (See figure.) Figure 30 As shown, even with the same number of searches, the model size differs between cases where the initial value of the confidence radius is set large (see dashed line) and cases where the initial value of the confidence radius is set small (see dotted line). Furthermore, when the initial value of the confidence radius is set large, the number of searches required to sufficiently reduce the model size increases compared to cases where the initial value of the confidence radius is set small.

[0227] Thus, depending on the initial value of the trust radius, the final model size and the number of searches for the pruning rate may vary; in other words, the performance of servers 1 and 1A may vary.

[0228] Therefore, the second modification describes a method for suppressing performance variations of servers 1 and 1A.

[0229] Figure 31 This is a block diagram illustrating an example of the functional configuration of server 1B according to the second modification. (See diagram for example.) Figure 31 As shown, server 1B may include different Figure 4 The computing unit 14B of server 1. The computing unit 14B may include different from the computing unit 14B. Figure 4 The threshold calculation unit 14a” and the determination unit 14b” of the calculation unit 14”.

[0230] When pruning a model, it is known that, compared to pruning the model all at once using a high pruning rate, gradually pruning the model using a low pruning rate can maintain accuracy and compress the model with a high compression rate.

[0231] As shown in equation (5) above, since the threshold T is set based on the reciprocal of the gradient, a layer with a large threshold T indicates a layer with a small gradient. Layers with small gradients have little impact on accuracy even when pruned.

[0232] Therefore, for example, server 1B (threshold calculation unit 14a”) sets the initial value of the trust radius to a value that minimizes the pruning rate in the first search. To do this, for example, threshold calculation unit 14a” may set the initial value of the trust radius to a value such that the layer with the maximum threshold T in all layers is pruned, and the remaining layers are not pruned (making the pruning rate “0%”).

[0233] By setting the initial value of the confidence radius as described above, Server 1B can further compress the model size or maintain accuracy compared to manually setting the initial value of the confidence radius to, for example, a large value.

[0234] Figure 32 This is a diagram illustrating an example of setting the initial value of the trust radius. (See diagram for example.) Figure 32 As shown in the upper part, without setting an initial value for the trust radius, the combination of pruning rates to be searched is "(layer 1, layer 2) = (10%, 20%)".

[0235] like Figure 32 As shown, in the first search for the pruning rate, the threshold calculation unit 14a” measures the threshold (max(Th)) of the layer with the maximum threshold among all layers and the error (Error) caused by the minimum pruning rate in that layer (except for “0%”).

[0236] Th represents a vector based on thresholds T1, T2, ... for each layer, and... Figure 32 In the example, Th = [T1, T2]. The threshold (max(Th)) is the threshold for the layer where the threshold is the maximum value, and... Figure 32 In the example, it is T2. The error is the error for the minimum pruning rate of the layer with the maximum threshold, and... Figure 32 In the example, the error for measuring the pruning rate "10%" for layer 2 is measured.

[0237] Next, using the measured threshold and error, the threshold calculation unit 14a” sets the initial value of the confidence radius according to the following equation (13). In the following equation (13), “||Th||2” is the L2 norm of the threshold for all layers.

[0238] [Formula 7]

[0239]

[0240] Threshold calculation unit 14a” will calculate the threshold T l T2 is set such that: the minimum pruning rate "10%" is selected as the pruning rate of the layer (layer 2) with the maximum threshold, and the pruning rate (0%) is selected in the remaining layers (layer 1) by the initial value of the calculated trust radius.

[0241] Therefore, as Figure 32 As shown in the lower part, when setting the initial value of the confidence radius and setting the thresholds T1 and T2, the combination of pruning rates to be searched becomes "(layer 1, layer 2) = (0%, 10%)". Since the layer (layer 2) that is the target layer for pruning is the layer with the maximum threshold, in other words, the layer with the minimum gradient, the impact of pruning on accuracy can be minimized.

[0242] Apart from the process of setting the initial value of the confidence radius, the function of the threshold calculation unit 14a” can be similar to that of at least one of the threshold calculation unit 14a according to one embodiment and the threshold calculation unit 14a’ according to the first modification. The determination unit 14b” can be similar to that of at least one of the determination unit 14b according to one embodiment and the determination unit 14b’ according to the first modification.

[0243] In other words, the method according to the second modification can be implemented by a combination of one or both of the first modification and an implementation method.

[0244] Next, refer to Figure 33 This will describe an operational example based on the second modified server 1B. Figure 33 This is a flowchart illustrating an example of operations performed by server 1B according to the second modification. Figure 33 This corresponds to a flowchart like this: In this flowchart, based on Figure 28 In the flowchart of server 1A shown, step S5 is deleted, steps S31 and S32 are added between steps S4 and S7, and steps S21, S22 and S23 are replaced by steps S33, S34 and S35 respectively.

[0245] In step S31, after calculating the threshold for each layer in step S4, the threshold calculation unit 14a" determines whether the search is the first time. If the search is not the first time (no in step S31), the process proceeds to step S7.

[0246] When the search is the first time (yes in step S31), the threshold calculation unit 14a” sets the initial value of the confidence radius based on the threshold of the layer with the maximum threshold and the minimum pruning rate error (step S32), and the process proceeds to step S7.

[0247] Steps S33, S34, and S35 can be respectively Figure 16 Steps S11, S13, and S14 shown or Figure 28 Steps S21, S22, and S23 are shown.

[0248] As described above, the second modification uses a method that sets the initial value of the trust radius through the threshold calculation unit 14a”, which differs from the methods of the first embodiment and the first modification. Therefore, server 1B can suppress changes in the final model size and the number of searches for the pruning rate, and can also suppress performance changes in servers 1 and 1A.

[0249] Figure 34 This is a graph showing the relationship between the number of searches for the pruning rate and the model size, based on the initial value of the trust radius. (See figure.) Figure 34 As shown, when the initial value of the confidence radius is set large (see dashed line), the model size in the initial search phase (e.g., from the "0th" to the "5th") becomes smaller; in other words, the pruning rate becomes higher.

[0250] In contrast, using the method according to the second modification, the initial value of the confidence radius is set based on the minimum pruning rate error in the layer with the maximum value of the threshold, and the model size in the initial stage is searched. In other words, the pruning rate can be reduced to the same extent as if the initial value of the confidence radius were set small (see dashed line).

[0251] Therefore, server 1B can suppress manual setting of the initial value (hyperparameter) of the trust radius by designers, and can dynamically set the initial value of the trust radius based on the layers of the trained model 11c. Thus, an appropriate pruning rate can be set for each model, and regardless of the model, variations in the final model size and the number of searches for the pruning rate can be suppressed, thereby mitigating performance variations of servers 1 and 1A.

[0252] <1-6> Examples of Hardware Configurations

[0253] According to one implementation method and the first and second modifications, servers 1, 1A, and 1B can all be virtual machines (VMs) or physical machines. The functions of servers 1, 1A, and 1B can be implemented by one computer or by two or more computers. At least some of the functions of servers 1, 1A, and 1B can be implemented using HW (hardware) resources and NW (network) resources provided by a cloud environment.

[0254] Figure 35 This is a block diagram illustrating an example of the hardware configuration of computer 10. In the following text, computer 10 is illustrated as the hardware (HW) implementing each function of servers 1, 1A, and 1B. When multiple computers are used as HW resources for implementing each function of servers 1, 1A, and 1B, each computer may include... Figure 35 The HW configuration shown.

[0255] like Figure 35 As shown, in an HW configuration, computer 10 may illustratively include processor 10a, memory 10b, storage unit 10c, IF (interface) unit 10d, IO (input / output) unit 10e, and reader 10f.

[0256] Processor 10a is an example of an arithmetic processing device that performs various control and calculation tasks. Processor 10a can be connected to each block in computer 10 via bus 10i to enable communication with each other. Processor 10a can be a multiprocessor including multiple processors or a multi-core processor with multiple processor cores, or it can be configured to have multiple multi-core processors.

[0257] The processor 10a may be, for example, an integrated circuit (IC), such as a CPU (Central Processing Unit), MPU (Microprocessor Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), DSP (Digital Signal Processor), ASIC (Application-Specific Integrated Circuit), or FPGA (Field Programmable Gate Array).

[0258] As processor 10a, a combination of two or more of the aforementioned integrated circuits can be used. As an example, computer 10 may include a first processor 10a and a second processor 10a. The first processor 10a is an example of a CPU that executes all or part of a program 10g (a machine learning program) that implements various functions of computer 10. For example, based on program 10g, the first processor 10a can implement server 1, 1A, or 1B (see...). Figure 4 , Figure 25 or Figure 31 The second processor 10a is an example of an accelerator that performs arithmetic processing for NN computations, such as matrix computations, and can implement, for example, servers 1, 1A, or 1B (see...). Figure 4 , Figure 25 or Figure 31 The functionality of machine learning unit 13.

[0259] Memory 10b is an example of a HW that stores various data and programs. For example, memory 10b may be at least one of volatile memory such as DRAM (Dynamic Random Access Memory) and non-volatile memory such as PM (Persistent Memory).

[0260] Storage unit 10c is an example of a HW that stores information such as various data and programs. For example, storage unit 10c can be a disk device such as an HDD (hard disk drive), a semiconductor drive device such as an SSD (solid-state drive), or a various storage device such as non-volatile memory. Non-volatile memory can be, for example, flash memory, SCM (storage class memory), ROM (read-only memory), etc.

[0261] Storage unit 10c can store program 10g. For example, the processor 10a of servers 1, 1A, and 1B can function as servers 1, 1A, and 1B by extending the program 10g stored in storage unit 10c to memory 10b and executing program 10g (see [link to documentation]). Figure 4 , Figure 25 or Figure 31 The function of the control unit 16.

[0262] Figure 4 , Figure 25 or Figure 31 The memory cell 11 shown can be implemented by a storage region included in at least one of the memory 10b and the storage cell 10c.

[0263] IF unit 10d is an example of a communication IF that controls connection and communication with a network. For example, IF unit 10d may include an adapter compatible with LAN (Local Area Network) such as Ethernet (registered trademark) or optical communication such as FC (Fibre Channel). The adapter can be adapted to at least one of wireless and wired communication schemes. For example, servers 1, 1A, and 1B can be connected to a computer (not shown) via IF unit 10d to enable communication between them. Figure 4 , Figure 25 or Figure 31 One or both of the functions of the acquisition unit 12 and the output unit 15 shown can be implemented by the IF unit 10d. For example, the program 10g can be downloaded from the network to the computer 10 and stored in the storage unit 10c via the communication IF.

[0264] The I / O unit 10e may include one or both of an input device and an output device. The input device may be, for example, a keyboard, mouse, or touch panel. The output device may be, for example, a monitor, projector, or printer. Figure 4 , Figure 25 or Figure 31 The output unit 15 shown can output the trimming rate 11d to the output device of the IO unit 10e to display the trimming rate 11d.

[0265] Reader 10f is an example of a reader that reads information about data and programs recorded on recording medium 10h. Reader 10f may include connection terminals or devices to which recording medium 10h can be connected or inserted. For example, reader 10f may be an adapter compatible with USB (Universal Serial Bus), a drive device for accessing recording disks, a card reader for accessing flash memory such as an SD card, etc. Recording medium 10h may store program 10g, or reader 10f may read program 10g from recording medium 10h and store it in storage unit 10c.

[0266] The recording medium 10h can be illustratively defined as a non-transitory computer-readable recording medium, such as a disk / optical disc or flash memory. A disk / optical disc can be illustratively defined as a floppy disk, CD (Compact Disc), DVD (Digital Versatile Disc), Blu-ray Disc, HVD (Holographic Versatile Disc), etc. Flash memory can be illustratively defined as a solid-state storage device, such as a USB flash drive or SD card.

[0267] The HW configuration of computer 10 described above is illustrative only. Therefore, the HW of computer 10 can be appropriately increased or decreased (e.g., added or removed from any block), partitioned, integrated in any combination, and the bus can be added or removed. For example, servers 1, 1A, and 1B can omit at least one of the I / O unit 10e and reader 10f.

[0268] <2> Miscellaneous

[0269] According to the implementation method and the first and second modifications, the above-described techniques can be modified and implemented as follows.

[0270] For example, it can be Figure 4 , Figure 25 or Figure 31 The acquisition unit 12, machine learning unit 13, computing unit 14, 14A or 14B and output unit 15 included in the server 1, 1A or 1B shown can be combined or can be divided into separate units.

[0271] For example, Figure 4 , Figure 25 or Figure 31 The servers 1, 1A, or 1B shown can be configured to implement each processing function through multiple devices cooperating with each other via a network. As an example, in servers 1, 1A, or 1B, the acquisition unit 12 and output unit 15 can be a web server and an application server, the machine learning unit 13 and computing units 14, 14A, or 14B can be application servers, and the storage unit 11 can be a database server, etc. In this case, the web server, application server, and database server can cooperate with each other via a network to implement the processing functions of servers 1, 1A, or 1B.

[0272] [List of reference numerals]

[0273] Servers 1, 1A, and 1B

[0274] 10 Computers

[0275] 11 Memory Units

[0276] 11a Untrained Model

[0277] 11b Data for Machine Learning

[0278] 11c training model

[0279] 11d pruning rate

[0280] 11e Reduced Model

[0281] 12 Acquisition Unit

[0282] 13 Machine Learning Units

[0283] 14, 14A, 14B Pruning Rate Calculation Unit

[0284] Threshold calculation unit 14a, 14a', 14a”

[0285] 14b, 14b', 14b” Determine the unit

[0286] 15 Output Units

[0287] 16 Control Unit

Claims

1. A computer-readable recording medium storing a machine learning program for causing a computer to perform processing, said processing comprising: For each of the multiple layers included in the training model of the neural network, for each element of the multiple layers, a threshold of the error of the tensor between before and after the reduction of an element is calculated, the threshold being an upper limit of the reduction rate of the elements of the multiple layers, the threshold being scaled such that the L2 norm of the threshold of the multiple layers becomes equal to or less than the upper limit of the threshold, wherein the neural network is used for image processing; In the case where the elements in each of the plurality of layers are reduced according to each of the plurality of reduction rate candidates, for each of the plurality of layers, a reduction rate candidate to be applied to each of the plurality of layers is selected based on the plurality of thresholds calculated for each of the layers and the error of the tensor for each of the layers between before and after the reduction; and For each of the plurality of layers, a reduction rate to be applied to each of the plurality of layers is determined based on the inference accuracy of the trained model and the inference accuracy of the simplified model after machine learning, the simplified model being obtained by reducing each element of the plurality of layers in the trained model according to the reduction rate candidate to be applied; The calculation of the threshold includes setting an initial value for the upper limit of the threshold to calculate the threshold such that the elements of the layer with the largest threshold among the plurality of layers are reduced, while the elements of layers other than the layer with the largest threshold are not reduced. Determining the reduction rate includes: When the sum of the inference accuracy and margin of the simplified model after machine learning is lower than the inference accuracy of the trained model, several selected reduction rate candidates are discarded; and When the sum of the inference accuracy of the simplified model after machine learning and the margin is equal to or higher than the inference accuracy of the trained model, a plurality of selected reduction rate candidates are determined as the reduction rate to be applied to each of the plurality of layers. The process further includes: Repeatedly perform the calculation of the threshold, the selection of the reduction rate candidate, and the determination of the reduction rate until the reduction rate meets a predetermined condition; and When the predetermined conditions are met, a determined reduction rate is output, and Specifically, the predetermined condition is met when the size of the computer's memory required to store the simplified model reaches the saturation size of the memory.

2. The computer-readable recording medium according to claim 1, wherein, Calculating the threshold involves using the value of the loss function of the trained model as elements of each of the plurality of layers are reduced, and the weight gradient of each of the plurality of layers.

3. The computer-readable recording medium according to claim 1 or 2, wherein, Calculating the threshold includes: When the sum of the inference accuracy of the simplified model after machine learning and the margin is lower than the inference accuracy of the trained model, the upper threshold is reduced; and The upper limit of the threshold is increased when the sum of the inference accuracy of the simplified model after machine learning and the margin is equal to or higher than the inference accuracy of the trained model.

4. The computer-readable recording medium according to claim 3, wherein, Calculating the threshold includes updating the upper limit of the threshold so that the combination of reduction rate candidates for the multiple layers is different each time the reduction rate candidate is selected.

5. A method for machine learning, the method comprising: For each of the multiple layers included in the training model of the neural network, for each element of the multiple layers, a threshold of the error of the tensor between before and after the reduction of an element is calculated, the threshold being an upper limit of the reduction rate of the elements of the multiple layers, the threshold being scaled such that the L2 norm of the threshold of the multiple layers becomes equal to or less than the upper limit of the threshold, wherein the neural network is used for image processing; In the case where the elements in each of the plurality of layers are reduced according to each of the plurality of reduction rate candidates, for each of the plurality of layers, a reduction rate candidate to be applied to each of the plurality of layers is selected based on the plurality of thresholds calculated for each of the layers and the error of the tensor for each of the layers between before and after the reduction; and For each of the plurality of layers, a reduction rate to be applied to each of the plurality of layers is determined based on the inference accuracy of the trained model and the inference accuracy of the simplified model after machine learning, the simplified model being obtained by reducing each element of the plurality of layers in the trained model according to the reduction rate candidate to be applied; The calculation of the threshold includes setting an initial value for the upper limit of the threshold to calculate the threshold such that the elements of the layer with the largest threshold among the plurality of layers are reduced, while the elements of layers other than the layer with the largest threshold are not reduced. Determining the reduction rate includes: When the sum of the inference accuracy and margin of the simplified model after machine learning is lower than the inference accuracy of the trained model, several selected reduction rate candidates are discarded; and When the sum of the inference accuracy of the simplified model after machine learning and the margin is equal to or higher than the inference accuracy of the trained model, a plurality of selected reduction rate candidates are determined as the reduction rate to be applied to each of the plurality of layers. The method further includes: Repeatedly perform the calculation of the threshold, the selection of the reduction rate candidate, and the determination of the reduction rate until the reduction rate meets a predetermined condition; and When the predetermined conditions are met, a determined reduction rate is output, and The predetermined condition is met when the size of the computer's memory required to store the simplified model reaches the saturation size of the memory.

6. The method according to claim 5, wherein, Calculating the threshold involves using the value of the loss function of the trained model as elements of each of the plurality of layers are reduced, and the weight gradient of each of the plurality of layers.

7. The method according to claim 5 or 6, wherein, Calculating the threshold includes: When the sum of the inference accuracy of the simplified model after machine learning and the margin is lower than the inference accuracy of the trained model, the upper threshold is reduced; and The upper limit of the threshold is increased when the sum of the inference accuracy of the simplified model after machine learning and the margin is equal to or higher than the inference accuracy of the trained model.

8. The method according to claim 7, wherein, Calculating the threshold includes updating the upper limit of the threshold so that the combination of reduction rate candidates for the multiple layers is different each time the reduction rate candidate is selected.

9. An information processing apparatus, comprising a controller to perform processing, the processing including: For each of the multiple layers included in the training model of the neural network, for each element of the multiple layers, a threshold of the error of the tensor between before and after the reduction of an element is calculated, the threshold being an upper limit of the reduction rate of the elements of the multiple layers, the threshold being scaled such that the L2 norm of the threshold of the multiple layers becomes equal to or less than the upper limit of the threshold, wherein the neural network is used for image processing; In the case where the elements in each of the plurality of layers are reduced according to each of the plurality of reduction rate candidates, for each of the plurality of layers, a reduction rate candidate to be applied to each of the plurality of layers is selected based on the plurality of thresholds calculated for each of the layers and the error of the tensor for each of the layers between before and after the reduction; and For each of the plurality of layers, a reduction rate to be applied to each of the plurality of layers is determined based on the inference accuracy of the trained model and the inference accuracy of the simplified model after machine learning, the simplified model being obtained by reducing each element of the plurality of layers in the trained model according to the reduction rate candidate to be applied; The calculation of the threshold includes setting an initial value for the upper limit of the threshold to calculate the threshold such that the elements of the layer with the largest threshold among the plurality of layers are reduced, while the elements of layers other than the layer with the largest threshold are not reduced. Determining the reduction rate includes: When the sum of the inference accuracy and margin of the simplified model after machine learning is lower than the inference accuracy of the trained model, several selected reduction rate candidates are discarded; and When the sum of the inference accuracy of the simplified model after machine learning and the margin is equal to or higher than the inference accuracy of the trained model, a plurality of selected reduction rate candidates are determined as the reduction rate to be applied to each of the plurality of layers. The process further includes: Repeatedly perform the calculation of the threshold, the selection of the reduction rate candidate, and the determination of the reduction rate until the reduction rate meets a predetermined condition; and When the predetermined conditions are met, a determined reduction rate is output, and The predetermined condition is met when the size of the computer's memory required to store the simplified model reaches the saturation size of the memory.

Images