Machine learning program, machine learning method, and information processing device

Abstract

To realize a lightweight neural network that contains multiple layers.SOLUTION: A machine learning program makes a computer execute a process for selecting a reduction ratio of an element in each of a plurality of layers in a machine learned model of a neural network including the plurality of layers, and a process for, when the neural network includes a calculation process that outputs a tensor of the predetermined calculation result for a tensor from a first layer and a tensor from one or more second layers preceding the first layer, adjusting the first reduction ratio and the one or more second reduction ratios based on one or more elements to be reduced in the first layer based on the first reduction ratio and one or more elements to be reduced in each of the one or more second layers based on the one or more second reduction ratio.SELECTED DRAWING: Figure 21

Description

[Technical Field]

[0001] The present invention relates to a machine learning program, a machine learning method, and an information processing device. [Background technology]

[0002] Neural networks (NNs) used in AI (Artificial Intelligence) tasks such as image processing tend to achieve higher performance (e.g., higher inference accuracy) by increasing the complexity of their configuration. However, increasing the complexity of the NN configuration can increase the number of calculations required to execute the NN on a computer and the memory size required by the computer to execute the NN.

[0003] "Pruning" is known as a method for reducing the number of calculations, in other words, shortening (speeding up) the calculation time, and reducing the memory size, in other words, making the NN machine learning model lightweight.

[0004] Pruning is a technique for reducing the data size of a machine learning model and reducing computation and communication time by eliminating (cutting out) at least one type of element: edges (weights), nodes, and channels of a neural network.

[0005] Excessive pruning can cause a degradation of the inference accuracy of the NN. Therefore, it is important to prune the NN while maintaining the inference accuracy or while keeping the degradation of the inference accuracy at a predetermined level.

[0006] For example, there is a known method for selecting layers that do not significantly affect the inference accuracy of a neural network during pruning. This method determines the channels of a convolutional layer to be pruned based on the parameters used in the batch normalization (BN) layer that follows the convolutional layer. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2021-124949 Summary of the Invention [Problem to be solved by the invention]

[0008] The method of selecting layers that do not significantly affect the inference accuracy of the NN is applied to convolutional layers connected to BN layers, but is not intended to be applied to other layers, such as convolutional layers not connected to BN layers or fully connected layers.

[0009] Furthermore, a NN that includes these multiple layers may include a concatenate operation unit that performs a concatenate operation to connect inputs from two or more layers. Note that, hereinafter, the concatenate operation may be referred to as the concat operation, and the concatenate operation unit may be referred to as the concat unit.

[0010] The concat unit performs a computation process of shortcut-combining tensors input from a certain layer and one or more layers preceding that layer, and outputting a single tensor. For example, shortcut combinations include operations on the input tensors, such as addition for each dimension and each element.

[0011] For example, consider the case where the method for selecting layers that do not significantly affect the inference accuracy of a NN can be applied to multiple layers as described above, and the NN includes a concat section. In this case, if pruning is performed using this method, the dimensions (matrix sizes) of the tensors input to the concat section may not match between two or more layers, and the concat section may not output a correct calculation result.

[0012] For this reason, it is possible to exclude two or more layers that input tensors to the concat section from the layers to be pruned. However, in this case, the pruning rate of the entire machine learning model decreases, and the effect of pruning on compressing (reducing) the data size of the machine learning model is reduced.

[0013] In one aspect, an object of the present invention is to realize a lightweight neural network including multiple layers. [Means for solving the problem]

[0014] In one aspect, the machine learning program may cause a computer to perform the following process: The process may include: creating a neural network including multiple layers; a neural network including an arithmetic process for outputting a tensor that is a result of a predetermined operation performed on a tensor from a first layer and a tensor from one or more second layers that are located before the first layer; The method may further include a process of selecting a reduction rate for each of the elements of the plurality of layers in the machine-learned model. , th The method may include adjusting the first reduction rate and the one or more second reduction rates based on one or more elements to be reduced in the first layer based on a reduction rate of one and one or more elements to be reduced in each of the one or more second layers based on one or more second reduction rates. [Effects of the Invention]

[0015] In one aspect, the present invention can achieve weight reduction of a neural network including multiple layers. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 10 is a diagram illustrating an example of a process for determining a channel of a convolutional layer on which pruning is performed. [Figure 2] FIG. 10 is a diagram illustrating an example of L1 regularization learning. [Figure 3] FIG. 3 is a diagram showing an example of whether the methods of FIGS. 1 and 2 can be applied to a layer of a neural network. [Figure 4] FIG. 2 is a block diagram illustrating an example of a functional configuration of a server according to an embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of calculation of a pruning rate with guaranteed accuracy. [Figure 6] FIG. 10 is a diagram illustrating an example of calculation of the accuracy of a model before and after pruning. [Figure 7] FIG. 10 is a diagram illustrating an example of searching for a pruning rate. [Figure 8] FIG. 10 is a diagram illustrating an example of a threshold value derivation method. [Figure 9] FIG. 10 is a diagram illustrating an example of an upper limit of a threshold value and a threshold value. [Figure 10] FIG. 10 is a diagram illustrating an example of a method for determining channels to be pruned. [Figure 11] FIG. 10 is a diagram illustrating an example of calculating a pruning error. [Figure 12] FIG. 10 is a diagram illustrating an example of a method for determining nodes to be pruned. [Figure 13] FIG. 10 is a diagram illustrating an example of calculating a pruning error. [Figure 14] FIG. 10 is a diagram illustrating an example of a method for determining weights for pruning. [Figure 15] FIG. 10 is a diagram illustrating an example of calculating a pruning error. [Figure 16] FIG. 10 is a diagram illustrating an example of a NN including a concat section. [Figure 17] FIG. 10 is a diagram illustrating an example of a tensor before and after application of a method according to an embodiment to a NN including a concat section. [Figure 18] FIG. 10 is a diagram for explaining an example of the operation of a calculation unit for a NN including a concat unit. [Figure 19] FIG. 10 is a diagram illustrating an example of a NN including multiple concat parts. [Figure 20] 20 is a diagram showing an example of the accuracy before and after pruning of the NN shown in FIG. 19 and the compression rate of the data size, depending on whether the conditions (I) and (II) are applied. FIG. [Figure 21] 10 is a flowchart illustrating an example of the operation of a process performed by a server according to an embodiment. [Figure 22]FIG. 10 is a diagram illustrating an example of a pruning error comparison result in response to an update of the confidence radius in a method according to an embodiment. [Figure 23] FIG. 10 is a block diagram showing an example of a functional configuration of a server according to a first modified example. [Figure 24] 10A and 10B are diagrams illustrating an example of a trust radius update process when increasing a trust radius. [Figure 25] 10A and 10B are diagrams illustrating an example of a trust radius update process when a trust radius is decreased. [Figure 26] 10 is a flowchart illustrating an example of the operation of a process performed by a server according to a first modified example. [Figure 27] FIG. 10 is a block diagram showing an example of the functional configuration of a server according to a second modified example. [Figure 28] FIG. 10 is a diagram illustrating an example of setting an initial value of a trust radius. [Figure 29] 10 is a flowchart illustrating an example of the operation of a process performed by a server according to a second modified example. [Figure 30] FIG. 1 is a block diagram showing an example of the hardware (HW) configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiments described below are merely examples, and are not intended to exclude various modifications or application of techniques not explicitly described below. For example, the present embodiment can be implemented with various modifications within the scope of the spirit thereof. In the drawings used in the following description, parts with the same reference numerals represent the same or similar parts unless otherwise specified.

[0018] [1] One embodiment Fig. 1 is a diagram illustrating an example of a process for determining the convolutional layer channel for pruning, and Fig. 2 is a diagram illustrating an example of L1 regularization learning. Fig. 1 illustrates a method for selecting a layer that does not significantly affect the inference accuracy of a neural network (NN), in which a computer determines the convolutional layer channel for pruning using a scaling coefficient γ used in the BN layer 100 following the convolutional layer. Note that the graphs shown for channels 111 to 113 in Fig. 1 represent the distribution of the output tensor.

[0019] As shown in FIG. 1, the computer performs normalization processing 101 on each of a plurality of channels 111 (#1 to #n; n is an integer equal to or greater than 2) input from the convolutional layer to the BN layer 100. For example, in the normalization processing 101, the computer calculates a mean value μ and a variance σ for each channel 111 according to the following formula (1): 2 By calculating z, multiple channels 112 (#1 to #n) that represent a normalized distribution with a mean of "0" and a variance of "1" are obtained. in and z mid indicate channels 111 and 112, respectively, and μ B and σ B 2 denote the mean and variance in the current mini-batch B, respectively.

number

[0020] The computer also performs scaling 102 on the multiple channels 112 (#1 to #n). For example, in scaling 102, the computer multiplies each of the multiple channels 112 by a scaling coefficient γ according to the following equation (2), and adds a bias β to the multiplication result, thereby outputting multiple channels 113 (#1 to #n) that represent a distribution scaled by the parameters γ and β. In the following equation (2), z out indicates the channel 113. Note that the parameters γ and β may be optimized by machine learning.

number

[0021] Here, when γ is small, the output of channel 113 (channel #n in the example of FIG. 1) resulting from scaling 102 is almost zero. This means that even if the channel is deleted by pruning, it does not have a significant impact on the inference accuracy of the NN. Therefore, the computer determines the channels to be pruned on a channel-by-channel basis by searching for a small γ (for example, "0").

[0022] For example, a computer searches for a small (smaller) γ by applying L1 regularization learning to γ. L1 regularization learning is a machine learning method known for making the parameters of the learning target "sparse" by performing machine learning by adding an L1 regularization term to the loss function calculated by the NN output.

[0023] 2, a computer performs L1 regularization learning on a certain vector 121 using a loss function 122 to obtain an L1 regularized vector 123. The loss function 122 may be a function L obtained by adding an original loss function (first term) such as cross-entropy to an L1 regularization term (second term) that uses an L1 norm (Σg(γ)=Σ|γ|), as shown in the following formula (3).

number

[0024] By L1 regularization learning, each parameter of vector 123 becomes a parameter indicating whether each parameter of vector 121 is zero or non-zero (is bifurcated). By using such L1 regularization learning, the computer can identify channels where γ is zero (close to zero) as channels to be pruned.

[0025] The identification of pruning targets using L1 regularization learning shown in Figures 1 and 2 is applied to convolutional layers connected to BN layers, but is not intended to be applied to other layers, such as convolutional layers not connected to BN layers and fully connected layers.

[0026] Fig. 3 is a diagram showing an example of whether the techniques of Fig. 1 and Fig. 2 can be applied to layers 131 to 139 of the NN 130. As shown in Fig. 3, the convolutional layers 131 and 133 and the BN layers 132 and 134 are layers to which the L1 regularization learning shown in Fig. 1 and Fig. 2 can be applied, while the convolutional layers 135 to 137 and the fully connected layers 138 and 139 are layers to which the L1 regularization learning shown in Fig. 1 and Fig. 2 cannot be applied.

[0027] Therefore, in one embodiment, a method for realizing a lightweight NN by determining a pruning rate for each layer regardless of the type of layer will be described.

[0028] [1-1] Example of functional configuration of server according to one embodiment 4 is a block diagram showing an example of a functional configuration of server 1 according to an embodiment. Server 1 is an example of a calculator, computer, or information processing device that outputs a pruning rate. As shown in FIG. 4, server 1 may illustratively include a memory unit 11, an acquisition unit 12, a machine learning unit 13, a pruning rate calculation unit (hereinafter simply referred to as a "calculation unit") 14, and an output unit 15. The acquisition unit 12, the machine learning unit 13, the calculation unit 14, and the output unit 15 are examples of a control unit 16.

[0029] The memory unit 11 is an example of a storage area, and stores various data used by the server 1. As shown in Fig. 4, the memory unit 11 may be capable of storing, for example, an unlearned model 11a, machine learning data 11b, a machine-learned model 11c, a pruning rate 11d, and a lightweight model 11e.

[0030] The acquisition unit 12 acquires the unlearned model 11a and the machine learning data 11b and stores them in the memory unit 11. For example, the acquisition unit 12 may generate one or both of the unlearned model 11a and the machine learning data 11b in the server 1, or may receive them from a computer external to the server 1 via a network (not shown).

[0031] The unlearned model 11a may be a pre-machine learning model of a neural network (NN) including unlearned parameters. The NN may include various layers, such as a deep neural network (DNN). The NN may include, for example, a convolutional layer not connected to a BN layer, a fully connected layer, or a convolutional layer connected to a BN layer. For example, the NN may be the NN 130 illustrated in FIG. 3.

[0032] The machine learning data 11b may be, for example, a training data set used for machine learning (training) of the unlearned model 11a. As an example, when performing machine learning of a neural network to realize image processing, the machine learning data 11b may include, for example, a plurality of pairs of training data such as image data and teacher data including a correct label for the training data.

[0033] In the machine learning phase, the machine learning unit 13 executes a machine learning process to train the unlearned model 11a based on the machine learning data 11b. For example, the machine learning unit 13 may generate the machine-learned model 11c by the machine learning process of the unlearned model 11a. The machine-learned model 11c may be an NN model including machine-learned parameters.

[0034] The machine-learned model 11c may be obtained by updating parameters included in the unlearned model 11a, and may be regarded as a model resulting from a change from the unlearned model 11a to the machine-learned model 11c through machine learning processing. The machine learning processing may be realized by various known techniques.

[0035] The calculation unit 14 calculates the pruning rate 11d by executing a pruning rate calculation process on the machine-learned model 11c, and stores the calculated pruning rate 11d in the memory unit 11.

[0036] For example, the calculation unit 14 may include a threshold calculation unit 14a that calculates a threshold for each layer to select one of the pruning rate candidates, and a determination unit 14b that determines the pruning rate 11d to be adopted based on the inference accuracy of the model pruned using the pruning rate candidates.

[0037] The output unit 15 outputs output data based on the pruning rate 11d generated (acquired) by the calculation unit 14. The output data may include, for example, the pruning rate 11d itself and / or the light-weighted model 11e.

[0038] The lightweight model 11e is data of a model obtained by pruning the machine-learned model 11c based on the pruning rate 11d. For example, the output unit 15 may cooperate with the machine learning unit 13 to apply the pruning rate 11d to prune and re-learn the machine-learned model 11c, thereby acquiring the lightweight model 11e and storing it in the memory unit 11. Note that the lightweight model 11e may be generated separately from the machine-learned model 11c, or may be data obtained by updating the machine-learned model 11c through pruning and re-learning, for example.

[0039] When outputting the output data, the output unit 15 may, for example, transmit (provide) the output data to another computer (not shown), or may store the output data in the memory unit 11 and manage it so that it can be acquired from the server 1 or another computer. Alternatively, when outputting the output data, the output unit 15 may output information indicating the output data to the screen of an output device such as the server 1, or may output the output data in various other modes.

[0040] [1-2] Example of pruning rate calculation process Next, a description will be given of an example of a pruning rate calculation process performed by the calculation unit 14 of the server 1. In the following description, it is assumed that the target for calculating the pruning rate is a weight matrix W, which is an example of a layer parameter.

[0041] The calculation unit 14 determines the pruning rate regardless of the type of layer by utilizing the tensor error for each layer that occurs due to pruning. As an example, the calculation unit 14 may calculate the pruning rate by the following steps (i) to (iii).

[0042] (i) The calculation unit 14 (threshold calculation unit 14a) determines (calculates) a pruning rate for which accuracy can be guaranteed for each layer.

[0043] Note that "guaranteeing accuracy" means, for example, guaranteeing that the accuracy of inference (inference accuracy) using the light-weight model 11e obtained by pruning the machine-learned model 11c exceeds a predetermined standard.

[0044] 5 is a diagram showing an example of calculation of a pruning rate for which accuracy can be guaranteed. As illustrated in FIG. 5, in (i), the threshold calculation unit 14a determines a pruning rate to be applied to the weight matrix W of each layer included in the machine-learned model 11c to be pruned, for each weight matrix W of a plurality of layers. Note that, while FIG. 5 focuses on layers 131 to 133 for explanation, this is not limiting, and the explanation of FIG. 5 may be applied to any of layers 131 to 139 illustrated in FIG. 3.

[0045] Here, the pruning rate is an example of the rate at which elements in a layer are reduced (reduction rate), and indicates the rate at which pruning targets in the machine-learned model 11c are made "sparse." In the example of Figure 2, this means the number of points in vector 123 that are set to "0."

[0046] As illustrated in Fig. 5, the threshold calculation unit 14a selects one pruning rate from a plurality of pruning rate candidates for each of the weight matrix W1 of the layer 131 (weight matrix W1 connected to the layer 132) and the weight matrix W2 between the layers 132 (weight matrix W2 connected to the layer 133). The pruning rate candidate is an example of a reduction rate candidate, and may be, for example, two or more rates between 0% and 100%, may be common to multiple layers, may be different for each layer, or may be a combination of these. In the example of Fig. 5, the pruning rate candidates are assumed to be 0%, 20%, 40%, and 60%.

[0047] The threshold calculation unit 14a calculates the error of the tensor before and after pruning when pruning is performed using each of the pruning rate candidates, and calculates a threshold T W 5, the threshold calculation unit 14a determines the maximum pruning rate candidate among the pruning rate candidates having an error smaller than the threshold T w1 The maximum pruning rate candidate with a smaller error than the threshold T w2 The maximum pruning rate candidate with a smaller error than is determined to be 20% (see arrow 142).

[0048] Threshold T w is a threshold value of the error of the tensor before and after pruning, and is the upper limit of the pruning rate at which accuracy can be guaranteed. For example, the threshold calculation unit 14a calculates a threshold T w The threshold T w The calculation method will be described in detail later.

[0049] The pruning rate calculated in (i) may be considered as a "provisionally calculated" pruning rate in relation to the processes in (ii) and (iii).

[0050] As described above, the threshold calculation unit 14a calculates the threshold T for the tensor error before and after reduction of elements in each of the multiple layers in the machine-learned model 11c of the NN including multiple layers. Furthermore, the threshold calculation unit 14a selects a reduction rate candidate to be applied to each of the multiple layers based on the multiple thresholds T and the tensor error before and after reduction when elements are reduced in each of the multiple layers by each of the multiple reduction rate candidates.

[0051] (ii) The calculation unit 14 (determination unit 14b) determines the pruning rate based on the accuracy of the machine learning model that has been pruned (lightened) using the pruning rate determined in (i) and the accuracy of the machine learning model that has not been pruned.

[0052] For example, the determination unit 14b determines the accuracy Acc of the model pruned at the pruning rate for each layer determined in (i) by taking into account the error due to the approximation formula (first-order Taylor expansion). p and Accuracy Margin Acc m and the accuracy of the unpruned model Acc wo Accuracy margin Acc m is a margin that allows for a decrease in inference accuracy, and may be set by the designer. Note that the margin may be "0", in which case the determining unit 14b determines the accuracy Acc p and the accuracy Acc of the model without pruning wo Just compare it with.

[0053] 6 is a diagram illustrating an example of calculation of the accuracy of a model before and after pruning. For example, the determination unit 14b calculates the accuracy Acc of a model (machine-learned model 11c) that is not pruned for all layers (W1, W2, . . . ). wo (See arrow 143). A model without pruning may be regarded as a model pruned with a pruning rate of 0% for each layer. Furthermore, the determination unit 14b calculates the accuracy Acc of a model pruned with a pruning rate of 0% for each layer (W1=40%, W2=20%, ...) calculated in (i). p is calculated (see arrow 144).

[0054] The determination unit 14b determines the sum of the accuracies Acc p +Acc m Accuracy Acc wo If this is the case, the determination unit 14b determines to adopt the pruning rate determined in (i). For example, the determination unit 14b stores the pruning rate determined in (i) in the memory unit 11 as the pruning rate 11d.

[0055] On the other hand, the determination unit 14b determines the sum of the accuracies Acc p +Acc m Accuracy Acc wo If the pruning rate is less than the predetermined value, the determination unit 14b determines to discard the pruning rate determined in (i). For example, the determination unit 14b determines to discard the pruning rate determined in (i) and adopt the pruning rate 11d determined in the immediately preceding (ii) (or the initial) pruning rate 11d.

[0056] (iii) The calculation unit 14 (determination unit 14b) searches for the maximum pruning rate for which accuracy can be guaranteed by repeatedly applying (i) and (ii) multiple times.

[0057] Fig. 7 is a diagram showing an example of searching for a pruning rate. In the example of Fig. 7, the calculation unit 14 searches for pruning rates for three layers (131 to 133) three times.

[0058] As illustrated in FIG. 7, in the first search (see reference numeral 145), in (i), the threshold calculation unit 14a calculates the threshold T w Calculate the threshold T w It is assumed that the pruning rates of the layers 131 to 133 are changed from "0%, 0%, 0%" (initial value) to "40%, 20%, 40%" based on the above. For example, in (ii), the determination unit 14b determines whether the Acc p +Acc m <Acc wo If it is determined that the pruning rate is 0%, the pruning rate determined in (i) is discarded, and the previous rate of 0%, 0%, 0% is adopted.

[0059] In the second search (see reference numeral 146), in (i), the threshold calculation unit 14a calculates the threshold T wCalculate (update) the updated threshold T w It is assumed that the pruning rates of the layers 131 to 133 are determined from "0%, 0%, 0%" to "20%, 20%, 40%" based on the above. For example, in (ii), the determination unit 14b determines whether the Acc p +Acc m ≧Acc wo If it is determined that the pruning rate is 20%, 20%, 40%, it adopts the pruning rate 11d and stores it in the memory unit 11.

[0060] In the third search (see reference numeral 147), in (i), the threshold calculation unit 14a calculates the threshold T w Calculate (update) the updated threshold T w For example, in (ii), the determination unit 14b determines the pruning rates of the layers 131 to 133 from "20%, 20%, 40%" to "20%, 40%, 40%" based on the above. p +Acc m ≧Acc wo If it is determined that "20%, 40%, 40%" is adopted, it is stored (updated) in the memory unit 11 as the pruning rate 11d.

[0061] The decision unit 14b may search for the pruning rate a predetermined number of times, for example, a preset number of times.

[0062] As described above, the determination unit 14b determines the reduction rate to be applied to each of the multiple layers based on the inference accuracy of the machine-learned model 11c and the inference accuracy after machine learning of the reduced model obtained by reducing the elements of each of the multiple layers in the machine-learned model 11c according to the candidate reduction rates to be applied.

[0063] Next, a specific example of the above-mentioned pruning rate calculation process will be described. Fig. 8 is a diagram for explaining an example of a threshold value derivation method, and Fig. 9 is a diagram showing an example of the upper limit of the threshold value and the threshold value.

[0064] The threshold calculation unit 14a calculates a threshold value of the pruning rate for each layer that can guarantee accuracy by performing a first-order Taylor expansion of the loss function when pruning is performed. For example, the error of the tensor for each layer that occurs due to pruning is Δw, the loss function when pruning is performed is L(w+Δw), the loss function of the model to be pruned is L(w), and the loss function when not pruning is L ideal ) to L wo +L m Then, the threshold value of the pruning rate that can guarantee accuracy is calculated by the following formula (4). wo is the loss function of the model without pruning, and L m is the margin of the loss function set by the designer.

number

[0065] The left side of the above equation (4) (see the dashed box in Figure 8) is a Taylor expansion of the loss function L(w + Δw) after pruning, and includes the weight gradient "∂L(W) / ∂w" for each layer to be pruned. The gradient for each layer may be calculated by backpropagation. The right side of the above equation (4) (see the dashed-dotted box in Figure 8) is a constraint that the loss function will be smaller than the ideal value (for example, the FP32 loss function) even after pruning.

[0066] In this way, the threshold calculation unit 14a calculates the threshold T based on the value of the loss function of the machine-learned model 11c when reducing the elements of each of the multiple layers and the weight gradient of each of the multiple layers.

[0067] By rearranging the above formula (4), it is possible to derive a condition for the "pruning error" that satisfies the constraint that the loss function after pruning is smaller than the ideal loss function, as shown in the following formula (5). In other words, it is possible to derive an upper limit (threshold) for the error due to pruning that guarantees accuracy (loss function). The threshold calculation unit 14a sets the right-hand side of the following formula (5) as the threshold T.

number

[0068] 9, the threshold calculation unit 14a compares a threshold T set for each layer with the error of the L1 norm due to pruning. Then, the threshold calculation unit 14a determines the pruning rate candidate with the largest value (40% in the example of FIG. 9) among the pruning rate candidates that result in an error smaller than the threshold T as the pruning rate resulting from (i).

[0069] As an example, the threshold calculation unit 14a may determine a pruning rate for each layer to be pruned, at which the pruning error (left side) is equal to or smaller than the threshold (right side) according to the following formula (6): In the following formula (6), "||ΔW||1" is the L1 norm of the weights to be pruned, and "n" is the number of weight elements in the layer to be pruned.

number

[0070] As shown in the above formula (6), the threshold T is a parameter derived by approximation. To prevent an error in determining the pruning rate due to an approximation error, an upper limit may be set for the threshold T (see FIG. 9). For example, the threshold calculation unit 14a may limit the magnitude of the threshold T by a "trust radius" based on the trust region method. The trust radius is an example of an upper threshold limit. As an example, the threshold calculation unit 14a may scale the threshold T so that the L2 norm of the threshold T for all layers is equal to or less than the trust radius. In the example of FIG. 9, T h indicates the vector of the threshold T for each layer, and "||T h ||2″ denotes the L2 norm of the threshold T of all layers.

[0071] For example, the threshold calculation unit 14a may update the confidence radius (for example, by a constant factor) in addition to the pruning rate depending on the accuracy comparison result in the process (ii) by the determination unit 14b. Note that the initial value of the confidence radius may be set by, for example, a designer.

[0072] As an example, the threshold calculation unit 14a calculates the sum of accuracies Acc p +Acc m is the accuracy Acc wo If it is greater than or equal to the accuracy Acc, the trust radius is multiplied by a constant K ("K>1.0"), and the sum of accuracies Acc p +Acc m is the accuracy Acc wo If it is less than, the trust radius may be multiplied by a constant k ("0<k<1.0").

[0073] 〔1-3〕Explanation according to the type of pruning target Next, examples of pruning methods and pruning error calculation methods according to the type of pruning target will be described. Examples of the type of pruning target include, for example, channel pruning, node pruning, and weight pruning. The calculation unit 14 may determine the pruning target and the pruning error using the weights corresponding to the pruning target according to the type of pruning target.

[0074] 〔1-3-1〕Example of channel pruning FIG. 10 is a diagram for explaining an example of a method for determining a channel to be pruned, and FIG. 11 is a diagram for explaining an example of calculating a pruning error.

[0075] Note that FIGS. 10 and 11 show the processing flow of the convolution operation. Also, H and W with subscripts indicate the sizes of the input data, kernel, and output data, and Ch with subscripts indicates the number of channels of the input data, kernel, and output data. The same applies to the following explanations regarding other types of pruning targets.

[0076] (An example of a method for determining a channel to be pruned) When the type of pruning target is a channel, the calculation unit 14 calculates (computes) the L1 norm in units of kernels corresponding to the channels of the output data. For example, as shown in "before pruning" in FIG. 10, the calculation unit 14 calculates the L1 norm for each of the Ch1 kernels before pruning. As a result, the L1 norms for Ch1 are calculated.

[0077] Next, as illustrated in "after pruning" in FIG. 10, the calculation unit 14 prunes the channels of the corresponding output data in ascending order of the calculated L1 norm according to the set pruning rate.

[0078] (Example of pruning error calculation) 11, the calculation unit 14 calculates the L1 norm of the kernel to be pruned. The L1 norm of the kernel to be pruned is the L1 norm of all kernels before pruning minus the L1 norm of all kernels after pruning, that is, the difference between the L1 norms before and after pruning.

[0079] The calculation unit 14 may obtain the pruning error by dividing the calculated L1 norm by the number of elements in all kernels before pruning.

[0080] [1-3-2] Example of node pruning FIG. 12 is a diagram for explaining an example of a method for determining nodes to be pruned, and FIG. 13 is a diagram for explaining an example of calculating a pruning error.

[0081] (An example of a method for determining which nodes to prune) When the type of pruning target is a node, the calculation unit 14 calculates the L1 norm for each weight connected to the output node. In the example "before pruning" in Fig. 12, the calculation unit 14 calculates the L1 norm for each of the solid line, dashed line, and dashed dotted line.

[0082] Next, the calculation unit 14 prunes the corresponding output nodes in ascending order of the calculated L1 norm according to the set pruning rate, as illustrated in "After pruning" in Fig. 12. For example, the calculation unit 14 determines the output node corresponding to the weight group with the smallest L1 norm as the node to be pruned.

[0083] (Example of pruning error calculation) 13, the calculation unit 14 calculates the L1 norm of the weight group to be pruned. The L1 norm of the weight group to be pruned is obtained by subtracting the L1 norm of all weights after pruning from the L1 norm of all weights before pruning.

[0084] The calculation unit 14 may obtain the pruning error by dividing the calculated L1 norm by the number of elements in all weights before pruning. In the example of "after pruning" in Fig. 13, the calculation unit 14 calculates the L1 norm of the weight group of the two-dot chain lines, and divides the L1 norm by the number of elements in all weights before pruning (="6"; the number of lines).

[0085] [1-3-3] Example of weight pruning FIG. 14 is a diagram for explaining an example of a method for determining weights for pruning, and FIG. 15 is a diagram for explaining an example of calculating a pruning error.

[0086] (An example of a method for determining the weights to be pruned) When the type of pruning target is weight, the calculation unit 14 calculates the L1 norm for each element of all weights. In the example of "before pruning" in Fig. 14, the number of weight elements is "6", so the calculation unit 14 calculates "6" L1 norms.

[0087] Next, the calculation unit 14 prunes the corresponding weights in ascending order of the calculated L1 norm according to the set pruning rate, as illustrated in "After pruning" in Fig. 14. For example, the calculation unit 14 determines the weight with the smallest L1 norm as the weight to be pruned.

[0088] (Example of pruning error calculation) 15, the calculation unit 14 calculates the L1 norm of the weights to be pruned. The L1 norm of the weights to be pruned is obtained by subtracting the L1 norm of all weights after pruning from the L1 norm of all weights before pruning.

[0089] The calculation unit 14 may obtain the pruning error by dividing the calculated L1 norm by the number of elements in all weights before pruning. In the example of "after pruning" in Fig. 15, the calculation unit 14 calculates the L1 norm of the weights of the dashed lines and divides the L1 norm by the number of elements in all weights before pruning (="6"; the number of lines).

[0090] [1-4] Explanation of pruning process for NN including concat part FIG. 16 is a diagram showing an example of an NN 150 including a concat unit 154, and FIG. 17 is a diagram showing an example of a tensor before and after application of the method according to the embodiment described above to the NN 150 including the concat unit 154.

[0091] 16, NN 150 includes layers 151 to 153, 155 (represented as layers #0 to #3, respectively) and concat unit 154. Concat unit 154 receives tensors from layers #0 and #2 as input, adds these tensors, and outputs the result of the addition to layer #3. The concat operation is an example of an arithmetic process that outputs a tensor that is the result of a predetermined operation on a tensor from layer #0 and a tensor from layer #2.

[0092] For simplicity, the following will be an example in which the elements to be pruned are channels, and each of layers #0 and #2 outputs a one-dimensional tensor with four channels (number of elements "4"). Note that elements can include, for example, weights or nodes in addition to channels.

[0093] 16, the concat unit 154 adds the tensor

[1010] output from layer #0 and the tensor

[0110] output from layer #2 for each channel, and outputs the resulting tensor

[1120] to layer #3. Note that each column in the tensor corresponds to an element (e.g., a channel) to be pruned.

[0094] Here, when the technique according to the embodiment described above is applied to the NN150 shown in FIG. 16, the pruning rate for each of layers #0 to #3 is determined individually, and pruning of each of layers #0 to #3 is performed based on the determined pruning rate.

[0095] For example, as shown in Fig. 17, assume that the tensor output from layer #0 becomes

[10] after pruning, and the tensor output from layer #2 becomes

[0010] after pruning. In this case, concat unit 154 cannot perform the concat operation because the number of elements (size) of the tensors input from layers #0 and #2, respectively, is different, being "2" and "3".

[0096] To avoid the concat operation becoming impossible, for example, it is conceivable to uniformly exclude all layers that output tensors that are input to the concat operation from the targets for determining the pruning rate. However, in this case, the pruning rate of the entire machine learning model of the NN decreases as the number of concat units 154 included in the NN increases, and the effect of compressing (reducing) the data size of the machine learning model by pruning decreases.

[0097] Therefore, the calculation unit 14 according to one embodiment matches the number of elements (size), for example, the number of channels, after pruning in all layers (hereinafter, sometimes referred to as "layers immediately before concat") that input tensors to the same concat unit 154. For example, the calculation unit 14 may adjust the pruning rate by applying the following conditions (I) and (II) to the provisionally calculated pruning rate so that the number of channels between the layers immediately before concat is the same.

[0098] (I) Prune elements (e.g., channels) that are commonly pruned in all layers immediately preceding the concat. (II) Elements (e.g., channels) that are not pruned in at least one layer immediately before the concat are not pruned.

[0099] This makes it possible to make the number of elements (size) of tensors input from multiple layers immediately before concat equal for each concat unit 154. This increases the likelihood that the layer immediately before concat will be pruned, and improves the compression rate of the data size of the machine learning model through pruning.

[0100] Fig. 18 is a diagram for explaining an example of the operation of the calculation unit 14 for a NN including the concat unit 154. For example, Fig. 18 shows an example of processing by the calculation unit 14 for layers #0 and #1, which are layers immediately before concat.

[0101] 18 may be executed by selecting a pruning rate candidate when the NN to be pruned includes the concat unit 154, and may be suppressed from being executed when the NN does not include the concat unit 154. For example, the calculation unit 14 may determine whether the NN includes the concat unit 154 by referring to configuration information (not shown) that defines the configuration of the NN, such as the configuration of each layer and the connection relationships between layers. Furthermore, the calculation unit 14 may identify the layer immediately before concat for each concat unit 154 based on the configuration information.

[0102] 18, the subscripts H and W indicate the sizes of the input data, kernel, and output data, and the subscript Ch indicates the number of channels of the input data, kernel, and output data. The same applies to the following explanation.

[0103] 18 illustrates an example in which, in the above-described (i), the calculation unit 14 calculates (calculates) the L1 norm for each kernel corresponding to the channel of the output data, and provisionally calculates the pruning rate by L1 regularization learning (see FIG. 2), etc. For example, if the provisionally calculated pruning rate is "50%" for both of the two layers, the calculation unit 14 prunes two of the four channels for each of the layers #0 and #1.

[0104] In the example of Figure 18, it is assumed that the calculation unit 14 has determined (provisionally set) channels #1 and #3 of layer #0 (see the top row of Figure 18) and channels #0 and #3 of layer #1 (see the bottom row of Figure 18) as the channels to be pruned.

[0105] The calculation unit 14 determines whether the layers #0 and #1 satisfy the above condition (I) or (II) for each element to be pruned, for example, for each channel. An example of a determination method is an OR operation using a mask.

[0106] For example, the calculation unit 14 may generate a mask for each layer immediately before the concat, in which a channel to be pruned is indicated by "0" and a channel not to be pruned is indicated by "1". In the example of Fig. 18, the calculation unit 14 generates a mask

[1010] for layer #0 and a mask

[0110] for layer #1.

[0107] Then, the calculation unit 14 performs an OR operation on the masks of the layers immediately before the concat, and identifies channels for which the OR operation result is "0" and channels for which the OR operation result is "1". Channels for which the OR operation result is "0" are channels that are commonly pruned in all layers immediately before the concat, and are elements that satisfy the above (I). Channels for which the OR operation result is "1" are channels that are not pruned in at least one layer immediately before the concat, and are elements that satisfy the above (II).

[0108] In the example of Fig. 18, the calculation unit 14 performs an OR operation on the mask

[1010] of layer #0 and the mask

[0110] of layer #1 to obtain the OR operation result

[1110] . As a result, the calculation unit 14 determines that channel #3, whose value is "0", should be pruned in layers #0 and #1, and determines that channels #0, #1, and #2, whose value is "1", should not be pruned in layers #0 and #1 (to exclude them from the elements to be pruned). For example, if the output data of channels #0 to #2 of layer #0 are "11", "21", and "31", and the output data of channels #0 to #2 of layer #1 are "12", "22", and "32", the output of the concat operation when channel #3 is not pruned will be as follows: Concat output = [11+12, 21+22, 31+32]

[0109] In this way, when the NN includes the concat unit 154, the calculation unit 14 updates the pruning rate and the elements to be pruned that are provisionally calculated for the layer immediately before the concat, based on a comparison of the pruning targets between the layers immediately before the concat. In other words, the calculation unit 14 adjusts the first and second reduction rates based on one or more elements to be reduced in the first layer of the layer immediately before the concat based on the first reduction rate and one or more elements to be reduced in each of one or more second layers of the layer immediately before the concat based on one or more second reduction rates.

[0110] This makes it possible to match the number of output channels of the layer immediately preceding the concat unit 154, thereby increasing the possibility of pruning at least one channel of the layer immediately preceding the concat unit 154. In other words, it becomes possible to determine an appropriate pruning rate for the layer immediately preceding the concat unit 154, thereby increasing the compression rate of the NN including the concat unit 154.

[0111] The process shown in Fig. 18 may be a part of the process (i) performed by the threshold calculation unit 14a. In other words, the process shown in Fig. 18 may be executed by the threshold calculation unit 14a.

[0112] Furthermore, the processing of the calculation unit 14 after the execution of the processing described with reference to FIG. 18 is similar to the processing of (ii) and (iii).

[0113] Fig. 19 is a diagram showing an example of an NN 160 including multiple concat units 161 to 168. Fig. 19 exemplarily shows a case where "CIFAR-10" is used as a data set of machine-learning data 11b, and "ResNet-18" having the configuration shown in Fig. 19 is used as an unlearned model 11a and a machine-learned model 11c.

[0114] In the example of Figure 19, the arrows indicate the direction of data (tensors) flow in the NN 160. Also, "conv" indicates a convolutional layer, "bn" indicates a BN layer, and "fc" indicates a fully connected layer. Furthermore, the subscripts "l1" to "l4" added to the beginning of each layer indicate the layer group, and the subscripts "1" to "5" added to the end of each layer are layer identifiers.

[0115] When the method for determining the pruning rate 11d according to one embodiment is applied to the NN 160 shown in FIG. 19, the calculation unit 14 matches the number of output channels between the layers immediately before concat for each of the concat units 161 to 168.

[0116] For example, with regard to the concat unit 161, the calculation unit 14 adjusts the pruning rate and the number of channels so that the number of output channels of each layer, "bn1," "l1_bn2," and "l1_conv3," shown by dashed lines in FIG. 19, match. In addition, the output of the concat unit 161 is shortcut-connected to the concat unit 162. Therefore, with regard to the concat unit 162, the calculation unit 14 adjusts the pruning rate and the number of channels so that the number of output channels of each layer, "bn1," "l1_bn2," "l1_bn4," and "l2_conv1," shown by dashed lines in FIG. 19, match.

[0117] Similarly, the calculation unit 14 adjusts the pruning rate and the number of channels for the concat units 163 and 164 so that the number of output channels for each layer, "l2_bn2," "l2_bn3," "l2_conv4," "l2_bn5," and "l3_conv1," indicated by dash-dot lines, match. Furthermore, the calculation unit 14 adjusts the pruning rate and the number of channels for the concat units 165 and 166 so that the number of output channels for each layer, "l3_bn2," "l3_bn3," "l3_conv4," "l3_bn5," and "l4_conv1," indicated by dash-dot lines, match. Furthermore, the calculation unit 14 adjusts the pruning rate and the number of channels for the concat units 167 and 168 so that the number of output channels for each layer, "l4_bn2", "l4_bn3", "l4_conv4", and "l4_bn5", indicated by long dashed lines, matches.

[0118] In the example of FIG. 19, pruning the final layer (FC) may result in a possibility of making it impossible to perform class classification, so the calculation unit 14 may exclude the final layer from the pruning target.

[0119] Fig. 20 is a diagram showing an example of the accuracy before and after pruning of the NN 160 shown in Fig. 19 and the compression rate of the data size according to whether or not the conditions (I) and (II) are applied. In Fig. 20, the case where the conditions (I) and (II) are not applied means that the layers immediately before the concat of the concat units 161 to 168 are not subject to pruning.

[0120] 20, when the conditions (I) and (II) are applied, it becomes possible to prune a plurality of layers that are input to each of the concat units 161 to 168. As a result, compared to when the conditions (I) and (II) are not applied, it is possible to improve the compression rate of the data size of the lightweight model 11e by about 12.3% while suppressing a significant deterioration in accuracy.

[0121] [1-5] Example of operation Next, an example of the operation of the server 1 according to an embodiment will be described with reference to Fig. 21. Fig. 21 is a flowchart for describing an example of the operation of processing by the server 1 according to an embodiment.

[0122] As illustrated in FIG. 21, the machine learning unit 13 performs machine learning on the unlearned model 11a acquired by the acquisition unit 12 without pruning (step S1).

[0123] The calculation unit 14 calculates the inference accuracy (recognition rate) Acc when no pruning is performed. wo is calculated (step S2).

[0124] The threshold calculation unit 14a sets an initial value of the reliability radius (step S3).

[0125] The threshold calculation unit 14a calculates a threshold T for each layer and a pruning error for each layer for setting a pruning rate (step S4), and determines whether the L2 norm of the threshold T for all layers is greater than the confidence radius (step S5). If the L2 norm of the threshold T for all layers is equal to or less than the confidence radius (NO in step S5), the process proceeds to step S7.

[0126] If the L2 norm of the threshold T for all layers is greater than the confidence radius (YES in step S5), the threshold calculation unit 14a scales (updates) the threshold so that the L2 norm of the threshold T for all layers = the confidence radius (step S6), and the processing proceeds to step S7.

[0127] In step S7, the threshold calculation unit 14a provisionally calculates a pruning rate for each layer. For example, the threshold calculation unit 14a provisionally sets a pruning rate for each layer from the set pruning rate candidates. Note that steps S4 to S7 are an example of the above process (i).

[0128] The calculation unit 14 determines whether the layer immediately before the concat is included in the layer for which the pruning rate has been provisionally calculated (step S8). If the layer immediately before the concat is not included in the layer for which the pruning rate has been provisionally calculated (NO in step S8), the process proceeds to step S10.

[0129] If the layer immediately before the concat is included in the layers for which the pruning rate has been provisionally calculated (YES in step S8), the calculation unit 14 executes the process of step S9, and the process proceeds to step S10.

[0130] In step S9, the calculation unit 14 identifies, for each concat unit 154, multiple layers immediately before concat that input tensors to the same concat unit 154, based on configuration information, etc. Then, the calculation unit 14 adjusts the provisionally calculated pruning rate and target elements for pruning between the layers immediately before concat. For example, the calculation unit 14 applies conditions (I) and (II) to target elements (e.g., channels) between the layers immediately before concat, and updates the pruning rate and target elements so that the number of target elements (e.g., the number of channels) between the layers immediately before concat are consistent (see FIG. 18 ).

[0131] The machine learning unit 13 prunes the machine-learned model 11c using the pruning rate provisionally calculated or adjusted by the threshold calculation unit 14a, and performs re-machine learning on the model after pruning. The calculation unit 14 calculates the inference accuracy Acc of the model after re-machine learning. p is calculated (step S10).

[0132] The determination unit 14b determines the inference accuracy Acc p +Margin Acc m is the inference accuracy Acc wo It is determined whether or not the accuracy is equal to or greater than this (step S11). By evaluating the inference accuracy (recognition rate), it is possible to compensate for an error in the selection of the pruning rate due to an approximation error.

[0133] Inference accuracy Acc p +Margin Acc m is the inference accuracy Acc woIf the result is equal to or greater than this (YES in step S11), the decision unit 14b decides to prune the machine-learned model 11c at the provisionally calculated pruning rate (step S12), and stores the provisionally calculated pruning rate as the pruning rate 11d in the memory unit 11. Furthermore, the threshold calculation unit 14a increases the confidence radius by a constant factor (step S13), and the process proceeds to step S16.

[0134] On the other hand, inference accuracy Acc p +Margin Acc m is the inference accuracy Acc wo If it is less than (NO in step S11), the decision unit 14b discards the provisionally calculated pruning rate (step S14). The threshold calculation unit 14a reduces the trust radius by multiplying it by a constant (step S15), and the process proceeds to step S16. Note that steps S10 to S15 are an example of the process (ii) above.

[0135] In step S16, the determination unit 14b determines whether the search (the processing of steps S4 to S15) has been performed a predetermined number of times, in other words, whether the number of times the processing of threshold value calculation, pruning rate candidate selection, and pruning rate determination has been performed satisfies a predetermined condition. If the search has not been performed a predetermined number of times (NO in step S16), the process proceeds to step S4.

[0136] If the search has been performed a predetermined number of times (YES in step S16), the output unit 15 outputs the determined pruning rate 11d (step S17), and the process ends. Note that step S16 is an example of the process (iii) above.

[0137] As described above, in the server 1 according to one embodiment, the threshold calculation unit 14a calculates the error resulting from pruning of the tensors used in the NN, and generates a threshold from the value of the loss function and the gradient obtained by backpropagation of the NN. The threshold calculation unit 14a also compares the calculated pruning error with a threshold to provisionally calculate a pruning rate. The determination unit 14b then compares the inference accuracy of the model after re-learning using the calculated pruning rate with the inference accuracy of the model without pruning, and determines a pruning rate for each layer. If the threshold calculation unit 14a determines that the inference accuracy with pruning is lower than the inference accuracy without pruning, it resets the upper limit of the threshold to a smaller value and searches for a new pruning rate.

[0138] As a result, the server 1 according to an embodiment can determine the pruning rate for each layer regardless of the type of layer. For example, the server 1 can determine the pruning rate for each layer to be applied to the machine-learned model 11c including a convolutional layer not connected to a BN layer, a fully connected layer, and the like.

[0139] Furthermore, according to the server 1, even if the NN 160 includes the concat sections 161 to 168, it is possible to increase the possibility of pruning at least one of the layers immediately before the concat, thereby improving the compression rate of the data size of the light-weight model 11e.

[0140] [1-6] Variations Next, a modified example according to one embodiment will be described. In the following description, for simplicity, the margin of inference accuracy Acc m is "0", in other words, in the comparison of inference accuracy, the inference accuracy Acc p is the inference accuracy Acc wo In the following description, an example is given in which the NN does not include a concat portion, but the processing described with reference to Figures 16 to 16E can be similarly applied to both the first and second modified examples described below.

[0141] [1-6-1] First modified example In a method according to one embodiment, the number of searches for the pruning rate (the number of attempts at the process (iii) above) is a hyperparameter that is set manually by, for example, a designer. For this reason, for example, if the number of searches is set to a small number, the machine-learned model 11c may not be sufficiently lightweight, whereas if the number of searches is set to a large number, the machine-learned model 11c may be sufficiently lightweight, but the search time may be long.

[0142] FIG. 22 is a diagram illustrating an example of a comparison result of pruning errors according to an update of the confidence radius in the method according to an embodiment.

[0143] As illustrated in FIG. 22, assume that a pruning rate of "10%" is calculated (determined) in the error comparison results of the mth search (m is an integer equal to or greater than "1"). In this case, the confidence radius is updated so that it increases by a constant K times. However, if the updated confidence radius is less than the error of the pruning rate candidate that is one step larger than the pruning rate candidate determined in the mth search, a pruning rate of "10%" is again calculated in the error comparison results of the (m+1)th search.

[0144] In this way, when the trust radius is set to a constant K or multiplied by a constant k, the amount of threshold update is limited by the trust radius, so the same pruning rate candidate may be adopted in multiple searches. If the same pruning rate combination is searched multiple times, the number of pruning rate searches will increase without sufficient attempts to prune the model.

[0145] Therefore, in the first modification, attention is focused on updating the trust radius, and a method for shortening (reducing) the search time (number of searches) for an appropriate pruning rate for reducing the weight of the NN will be described.

[0146] Fig. 23 is a block diagram showing an example of the functional configuration of a server 1A according to the first modified example. As shown in Fig. 23, the server 1A may include a calculation unit 14A that is different from the server 1 in Fig. 4. The calculation unit 14A may include a threshold calculation unit 14a' and a determination unit 14b' that are different from the calculation unit 14 in Fig. 4.

[0147] The calculation unit 14A searches for a different combination of pruning rates for each search. Here, a state in which a combination of pruning rates of "0%" for all layers is selected is considered to be a state in which the calculation unit 14A has determined that no further search for pruning rates will be performed. Under this assumption, the calculation unit 14A (determination unit 14b') terminates the search when a combination of pruning rates of "0%" for all layers is selected.

[0148] The threshold calculation unit 14a' determines, for each layer i (i is an integer equal to or greater than 1), the error of the pruning rate that is one value greater than the searched pruning rate or the absolute value of the difference between the error of the searched pruning rate and the threshold, "E diff,i " is measured.

[0149] For example, the threshold calculation unit 14a' calculates the inference accuracy Acc p is the inference accuracy Acc wo If it is equal to or greater than the threshold, the absolute value of the difference between the error of the pruning rate that is one value greater than the searched pruning rate and the threshold, "E diff,i " is measured.

[0150] On the other hand, the threshold calculation unit 14a' calculates the inference accuracy Acc p is the inference accuracy Acc wo If it is less than 1, the absolute value of the difference between the error of the searched pruning rate and the threshold, "E diff,i " is measured.

[0151] The threshold calculation unit 14a' calculates the absolute value of the calculated differences for all layers, "E diff,i The smallest value (difference) of "E diff " to get Ediff = min(E diff,1 , E diff,2 , ..., E diff,i ) (7)

[0152] The threshold calculation unit 14a' determines a constant multiple of the confidence radius and a difference "E" between the confidence radius and the confidence radius according to the comparison result of the inference accuracy by the determination unit 14b'. diff The confidence radius is updated by adopting the sum or difference between the two, whichever has the larger amount of fluctuation.

[0153] For example, the threshold calculation unit 14a' calculates the inference accuracy Acc p is the inference accuracy Acc wo If it is equal to or greater than the confidence radius, the constant K times the confidence radius and the difference "E diff " and the sum of " and ", whichever has the larger amount of variation is adopted and updated so that the trust radius increases.

[0154] On the other hand, the threshold calculation unit 14a' calculates the inference accuracy Acc p is the inference accuracy Acc wo If it is less than the confidence radius, the constant k times the confidence radius and the difference "E diff ", the difference with the larger amount of variation is adopted and updated so that the trust radius decreases.

[0155] In this way, the threshold calculation unit 14a' updates the confidence radius so that the combinations of pruning rate candidates for each of the multiple layers are different from one another each time the process of selecting pruning rate candidates (in other words, search) is performed.

[0156] FIG. 24 is a diagram illustrating an example of the reliability radius update process when increasing the reliability radius. As shown in FIG. 24, it is assumed that the pruning rate found the mth time is "(Layer 1, Layer 2)=(10%, 0%)". The threshold calculation unit 14a' calculates the absolute value "E diff,1 ” and the absolute value of the difference between the error and the trust radius for a pruning rate of 10% for Layer 2, “E diff,2The threshold calculation unit 14a' calculates the smallest difference "E diff,2 " to "E diff " is obtained.

[0157] Then, the threshold calculation unit 14a' determines (updates) the (m+1)th (next) reliability radius according to the following formula (8). (m+1th trust radius) = max((m-th confidence radius · constant K), (m-th confidence radius + E diff )) (8)

[0158] As a result, the trust radius for the (m+1)th iteration is selected to be at least equal to the sum of the trust radius and the difference, and therefore a bit width different from that for the (m+1)th iteration is calculated as the pruning rate.

[0159] 24, the confidence radius (upper limit of the threshold) in the (m+1)th search matches the error of the pruning rate of "10%" for layer 2. Therefore, in the (m+1)th search, a pruning rate combination of "(layer 1, layer 2)=(10%, 10%)" that is different from the previous search is searched for.

[0160] FIG. 25 is a diagram illustrating an example of the trust radius update process when the trust radius is decreased. As shown in FIG. 25, it is assumed that the pruning rate found the mth time is "(Layer 1, Layer 2)=(10%, 0%)". The threshold calculation unit 14a' calculates the absolute value "E diff,1 ” and the absolute value of the difference between the error and the trust radius for a pruning rate of 0% for layer 2, “E diff,2 The threshold calculation unit 14a' calculates the smallest difference "E diff,1 " to "E diff " is obtained.

[0161] Then, the threshold calculation unit 14a' determines (updates) the (m+1)th (next) reliability radius according to the following formula (9). (m+1th trust radius) = max((m-th confidence radius · constant), (m-th confidence radius - E diff )) (9)

[0162] As a result, the trust radius for the (m+1)th time is selected to be at least a value greater than the difference between the trust radius and the difference, and therefore a bit width different from that for the (m+1)th time is calculated as the pruning rate.

[0163] 25, the confidence radius (upper limit of the threshold) in the (m+1)th search matches the error of the pruning rate of "0%" for layer 1. Therefore, in the (m+1)th search, a pruning rate combination of "(layer 1, layer 2)=(0%, 0%)" that is different from the previous search is searched for.

[0164] By generalizing the above equations (8) and (9), the next trust radius can be expressed by the following equation (10). Next confidence radius = Current confidence radius * max(Constant, Qscale_min) (10)

[0165] Here, in the above equation (10), the constant is K or k, "Qscale_min" is "Qscale" expressed by the following equation (11), and "Qscale" is expressed by the following equation (12). Qscale_min = min(Qscale calculated for all target vectors) (11) Qscale = 1 + Qdiff / Qth (12)

[0166] In the above equation (12), "Qdiff" is "the difference between the quantization error of the bit width that is one bit narrower than the provisionally calculated bit width (pruning rate) and the threshold value," and "Qth" is the threshold value.

[0167] Next, an example of operation of the server 1A according to the first modified example will be described with reference to Fig. 26. Fig. 26 is a flowchart for describing an example of operation of processing by the server 1A according to the first modified example. In Fig. 26, steps S13, S15, and S16 in the flowchart according to the server 1 shown in Fig. 21 are replaced with steps S21, S22, and S23, respectively. Note that also in the first modified example, the threshold calculation unit 14a' sets an initial value of the trust radius in step S3.

[0168] In step S21, the threshold calculation unit 14a' increases the confidence radius by a constant K or the "sum of differences," whichever is larger, and the process proceeds to step S23.

[0169] In step S22, the threshold calculation unit 14a' reduces the confidence radius by a constant k times or by the "difference of differences," whichever is larger, and the process proceeds to step S23.

[0170] In step S23, the determination unit 14b' determines whether the pruning rate 11d of all layers is "0%", in other words, whether the pruning rate satisfies a predetermined condition. If the pruning rate 11d of at least one layer is not "0%" (NO in step S23), the process proceeds to step S4.

[0171] If the pruning rate 11d of all layers is "0%" (YES in step S23), the output unit 15 outputs the determined pruning rate 11d (step S17), and the process ends.

[0172] As described above, in the first modified example, the method of updating the trust radius by the threshold calculation unit 14a' and the termination condition for determining the termination of the search by the determination unit 14b' are different from those in the first embodiment. This allows the server 1A to search for an appropriate pruning rate for sufficiently reducing the weight of the NN in the shortest time (the shortest number of times). In addition, it is possible to omit the setting (specification) of the number of searches by a designer or the like.

[0173] [1-6-2] Second Modified Example In the method according to the embodiment and the first modification, the initial value of the trust radius is a hyperparameter set by a designer or the like.

[0174] The model size may differ even if the number of searches is the same when the initial value of the trust radius is set large or small. Also, when the initial value of the trust radius is set large, the number of searches required to sufficiently reduce the model size may be greater than when the initial value of the trust radius is set small.

[0175] In this way, the final model size and the number of searches for the pruning rate may vary depending on the initial value of the trust radius; in other words, the performance of the servers 1 and 1A may vary.

[0176] Therefore, in the second modification, a method for suppressing the variation in performance between the servers 1 and 1A will be described.

[0177] FIG. 27 is a block diagram showing an example of a functional configuration of a server 1B according to a second modified example. As illustrated in FIG. 27, the server 1B may include a calculation unit 14B that is different from the server 1 in FIG. 4. The calculation unit 14B may include a threshold calculation unit 14a" and a determination unit 14b" that are different from the calculation unit 14 in FIG. 4.

[0178] In model pruning, it is known that pruning a model gradually using a small pruning rate can maintain accuracy and compress the model at a higher compression rate than pruning a model all at once using a large pruning rate.

[0179] Furthermore, as shown in the above formula (5), the threshold T is set according to the inverse of the gradient, so a layer with a large threshold T means that the layer has a small gradient. A layer with a small gradient means that pruning has little effect on accuracy.

[0180] Therefore, the server 1B (threshold calculation unit 14a"), for example, sets the initial value of the trust radius to a value that minimizes the pruning rate in the first search. For this purpose, the threshold calculation unit 14a" may set the initial value of the trust radius to a value that prunes the layer with the largest threshold T among all layers and does not prune the remaining layers (the pruning rate is "0%").

[0181] By setting the initial value of the trust radius as described above, the server 1B can compress the model size more or maintain accuracy more than if the initial value of the trust radius were manually set to a larger value, for example.

[0182] Fig. 28 is a diagram illustrating an example of setting the initial value of the trust radius. As shown in the upper part of Fig. 28, when the initial value of the trust radius is not set, the combination of pruning rates searched is "(Layer 1, Layer 2) = (10%, 20%)".

[0183] As illustrated in FIG. 28, in the initial search for the pruning rate, the threshold calculation unit 14a″ measures the threshold (max(Th)) of the layer with the largest threshold among all layers and the error (Error) due to the smallest pruning rate (excluding “0%”) of that layer.

[0184] Th denotes a vector of thresholds T1, T2, ... for each layer, and in the example of FIG. 28, Th = [T1, T2]. The threshold (max(Th)) is the threshold for the layer with the largest threshold, and in the example of FIG. 28, this is T2. The error (Error) is the error of the minimum pruning rate for the layer with the largest threshold, and in the example of FIG. 28, the error for a pruning rate of "10%" for layer 2 is measured.

[0185] Next, the threshold calculation unit 14a" sets the initial value of the confidence radius using the measured threshold and error according to the following equation (13). In the following equation (13), "||Th||2" is the L2 norm of the thresholds of all layers.

number

[0186] The threshold calculation unit 14a'' sets the thresholds T1 and T2 based on the calculated initial value of the trust radius so that the minimum pruning rate of "10%" is selected as the pruning rate for the layer with the largest threshold (layer 2), and a pruning rate of "0%" is selected for the remaining layers (layer 1).

[0187] As a result, when the initial value of the trust radius is set and the thresholds T1 and T2 are set, the combination of pruning rates to be searched for is "(Layer 1, Layer 2) = (0%, 10%)", as shown in the lower part of Fig. 28. The layer to be pruned (Layer 2) has the largest threshold, in other words, the layer with the smallest gradient, so the impact of pruning on accuracy can be kept small.

[0188] Note that the functions of the threshold calculation unit 14a" other than the process of setting the initial value of the reliability radius may be similar to either or both of the threshold calculation unit 14a according to the embodiment and the threshold calculation unit 14a' according to the first modified example. Furthermore, the determination unit 14b" may be similar to either or both of the determination unit 14b according to the embodiment and the determination unit 14b' according to the first modified example.

[0189] That is, the technique according to the second modification may be realized by combining with one or both of the embodiment and the first modification.

[0190] Next, an example of the operation of the server 1B according to the second modified example will be described with reference to Fig. 29. Fig. 29 is a flowchart for explaining an example of the operation of processing by the server 1B according to the second modified example. Fig. 29 is obtained by deleting step S3 from the flowchart for the server 1 shown in Fig. 21, adding steps S31 and S32 between steps S4 and S5, and replacing steps S13, S15, and S16 with steps S33, S34, and S35, respectively.

[0191] In step S31, the threshold calculation unit 14a'' determines whether or not this is the first search after calculating the threshold for each layer in step S4. If this is not the first search (NO in step S31), the process proceeds to step S5.

[0192] If it is the first search (YES in step S31), the threshold calculation unit 14a" sets an initial value of the confidence radius based on the threshold of the layer with the largest threshold and the minimum pruning rate error (step S32), and the process proceeds to step S5.

[0193] Steps S33, S34, and S35 may be any of steps S13, S15, and S16 shown in FIG. 21 and steps S21, S22, and S23 shown in FIG. 26, respectively.

[0194] As described above, in the second modified example, the method of setting the initial value of the trust radius by the threshold calculation unit 14a" is different from that of the embodiment and the first modified example. This enables the server 1B to suppress fluctuations in the number of searches for the final model size and the pruning rate, and suppresses variations in the performance of the servers 1 and 1A.

[0195] Furthermore, the server 1B can prevent a designer or the like from manually setting the initial value (hyperparameter) of the trust radius, and can dynamically set the initial value of the trust radius according to the layer of the machine-learned model 11c. Therefore, an appropriate pruning rate can be set for each model, and fluctuations in the number of searches for the final model size and pruning rate can be suppressed regardless of the model, thereby suppressing variations in the performance of the servers 1 and 1A.

[0196] [1-7] Hardware configuration example The servers 1, 1A, and 1B according to the embodiment and the first and second modifications may each be a virtual machine (VM) or a physical machine. Furthermore, the functions of the servers 1, 1A, and 1B may be implemented by one computer or two or more computers. Furthermore, at least some of the functions of the servers 1, 1A, and 1B may be implemented using hardware (HW) resources and network (NW) resources provided by a cloud environment.

[0197] 30 is a block diagram showing an example of the hardware (HW) configuration of the computer 10. The following description will be given taking the computer 10 as an example of the hardware (HW) that realizes the functions of the servers 1, 1A, and 1B. Note that when multiple computers are used as HW resources that realize the functions of the servers 1, 1A, and 1B, each computer may have the HW configuration shown in FIG.

[0198] As shown in FIG. 30, the computer 10 may include, as a HW configuration, a processor 10a, a memory 10b, a storage unit 10c, an IF (Interface) unit 10d, an IO (Input / Output) unit 10e, and a reading unit 10f, for example.

[0199] The processor 10a is an example of a processing unit that performs various controls and calculations. The processor 10a may be connected to each block in the computer 10 via a bus 10i so that they can communicate with each other. The processor 10a may be a multiprocessor including multiple processors, a multi-core processor having multiple processor cores, or a configuration having multiple multi-core processors.

[0200] Examples of the processor 10a include integrated circuits (ICs) such as a CPU, MPU, GPU, APU, DSP, ASIC, and FPGA. CPU is an abbreviation for Central Processing Unit, MPU is an abbreviation for Micro Processing Unit, GPU is an abbreviation for Graphics Processing Unit, and APU is an abbreviation for Accelerated Processing Unit. DSP is an abbreviation for Digital Signal Processor, ASIC is an abbreviation for Application Specific IC, and FPGA is an abbreviation for Field-Programmable Gate Array.

[0201] Note that a combination of two or more of the above-described integrated circuits may be used as the processor 10a. As an example, the computer 10 may include a first and a second processor 10a. The first processor 10a is an example of a CPU that executes a program 10g (machine learning program) that implements all or part of the various functions of the computer 10. For example, the first processor 10a may implement the functions of the acquisition unit 12, the calculation unit 14, 14A, or 14B, and the output unit 15 of the server 1, 1A, or 1B (see FIG. 4, FIG. 23, or FIG. 27) based on the program 10g. The second processor 10a is an example of an accelerator that executes arithmetic processing used in NN calculations such as matrix operations, and may implement the functions of the machine learning unit 13 of the server 1, 1A, or 1B (see FIG. 4, FIG. 23, or FIG. 27).

[0202] The memory 10b is an example of HW that stores various types of data, programs, and other information. Examples of the memory 10b include one or both of a volatile memory such as a dynamic random access memory (DRAM) and a non-volatile memory such as a persistent memory (PM).

[0203] The storage unit 10c is an example of HW that stores various types of data, programs, and other information. Examples of the storage unit 10c include various storage devices such as a magnetic disk device such as a hard disk drive (HDD), a semiconductor drive device such as a solid state drive (SSD), and a nonvolatile memory. Examples of nonvolatile memory include a flash memory, a storage class memory (SCM), and a read-only memory (ROM).

[0204] The storage unit 10c may store a program 10g. For example, the processor 10a of each of the servers 1, 1A, and 1B can implement the function of the control unit 16 (see FIG. 4, FIG. 23, or FIG. 27) of each of the servers 1, 1A, and 1B by expanding the program 10g stored in the storage unit 10c into the memory 10b and executing it.

[0205] Furthermore, the memory unit 11 illustrated in FIG. 4, FIG. 23, or FIG. 27 may be realized by a storage area included in at least one of the memory 10b and the storage unit 10c.

[0206] The IF unit 10d is an example of a communication IF that controls connection and communication with a network. For example, the IF unit 10d may include an adapter that complies with a LAN (Local Area Network) such as Ethernet (registered trademark) or optical communication such as FC (Fibre Channel). The adapter may support one or both of wireless and wired communication methods. For example, the servers 1, 1A, and 1B may be connected to a computer (not shown) via the IF unit 10d so that they can communicate with each other. One or both of the functions of the acquisition unit 12 and the output unit 15 illustrated in FIG. 4, FIG. 23, or FIG. 27 may be realized by the IF unit 10d. Furthermore, for example, the program 10g may be downloaded from the network to the computer 10 via the communication IF and stored in the storage unit 10c.

[0207] The IO unit 10e may include one or both of an input device and an output device. Examples of the input device include a keyboard, a mouse, and a touch panel. Examples of the output device include a monitor, a projector, and a printer. For example, the output unit 15 illustrated in FIG. 4, 23, or 27 may output the pruning rate 11d to the output device of the IO unit 10e and display it.

[0208] The reading unit 10f is an example of a reader that reads data and program information recorded on the recording medium 10h. The reading unit 10f may include a connection terminal or device to which the recording medium 10h can be connected or inserted. Examples of the reading unit 10f include an adapter that complies with USB (Universal Serial Bus) or the like, a drive device that accesses a recording disk, and a card reader that accesses a flash memory such as an SD card. Note that the recording medium 10h may store the program 10g, and the reading unit 10f may read the program 10g from the recording medium 10h and store it in the memory unit 10c.

[0209] Examples of the recording medium 10h include non-transitory computer-readable recording media such as magnetic / optical disks and flash memories. Examples of magnetic / optical disks include flexible disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray Discs, and HVDs (Holographic Versatile Discs). Examples of flash memories include semiconductor memories such as USB memories and SD cards.

[0210] The above-described hardware configuration of the computer 10 is an example. Therefore, the hardware in the computer 10 may be increased or decreased (for example, adding or deleting any block), divided, integrated in any combination, or buses may be added or deleted as appropriate. For example, in the servers 1, 1A, and 1B, at least one of the IO unit 10e and the reading unit 10f may be omitted.

[0211] [2] Other The techniques according to the above-described embodiment and the first and second modifications can be implemented with the following modifications and variations.

[0212] For example, the acquisition unit 12, machine learning unit 13, calculation unit 14, 14A or 14B, and output unit 15 provided in the server 1, 1A or 1B shown in Figure 4, Figure 23 or Figure 27 may be merged or separated.

[0213] 4, 23, or 27 may be configured such that each processing function is realized by a plurality of devices cooperating with each other via a network. As an example, in server 1, 1A, or 1B, acquisition unit 12 and output unit 15 may be a web server and an application server, machine learning unit 13 and calculation unit 14, 14A, or 14B may be an application server, and memory unit 11 may be a DB server, etc. In this case, the web server, application server, and DB server may be cooperating with each other via a network to realize the processing function of server 1, 1A, or 1B.

[0214] Furthermore, for example, the method of applying (I) and (II) to a NN including a concat part, which has been described with reference to Figures 16 to 20, is not limited to application to the pruning process by the server 1, 1A, or 1B shown in Figure 4, Figure 23, or Figure 27. For example, the method of applying (I) and (II) may be applied to various methods of determining a pruning rate for each layer of a NN.

[0215] [3] Supplementary Note The following additional notes are further disclosed regarding the above embodiment and the first and second modifications.

[0216] (Appendix 1) Selecting a reduction rate for elements in each of a plurality of layers in a machine-learned model of a neural network including the plurality of layers; When the neural network includes an arithmetic process for outputting a tensor that is a result of a predetermined arithmetic operation performed on a tensor from a first layer and a tensor from one or more second layers that are located before the first layer, the first reduction rate and the one or more second reduction rates are adjusted based on one or more elements to be reduced in the first layer based on a first reduction rate and one or more elements to be reduced in each of the one or more second layers based on one or more second reduction rates. A machine learning program that lets a computer perform processing.

[0217] (Appendix 2) the adjusting process includes adjusting the first reduction rate and the one or more second reduction rates so that the number of elements of the tensor from the first layer matches the number of elements of the tensor from each of the one or more second layers. The machine learning program described in Appendix 1.

[0218] (Appendix 3) the adjusting process includes adjusting the first reduction rate and the one or more second reduction rates so that elements that are targets of reduction in the first layer and all of the one or more second layers are targets of reduction in all of the layers, and elements that are not targets of reduction in at least one of the first layer and the one or more second layers are excluded from the targets of reduction in all of the layers; 1. A machine learning program according to claim 1 or 2.

[0219] (Appendix 4) Calculating a threshold value for the error of the tensor before and after the reduction of elements of each of the plurality of layers; causing the computer to execute a process; the selecting process includes a process of selecting, as the reduction rate, a reduction rate candidate to be applied to each of the plurality of layers based on the plurality of thresholds and an error of a tensor before and after reduction when the element is reduced by each of a plurality of reduction rate candidates in each of the plurality of layers; the adjusting process includes a process of adjusting the selected reduction rate candidates for the first tier and the one or more second tiers as the first reduction rate and the one or more second reduction rates, the determining process includes a process of determining a reduction rate to be applied to each of the plurality of layers based on the inference accuracy of the machine-learned model and the inference accuracy after machine learning of a reduced model obtained by reducing elements of each of the plurality of layers in the machine-learned model according to the reduction rate candidate. The machine learning program according to any one of Supplementary Note 1 to Supplementary Note 3.

[0220] (Appendix 5) The element is a channel, a weight, or a node. The machine learning program according to any one of Supplementary Note 1 to Supplementary Note 4.

[0221] (Appendix 6) Selecting a reduction rate for elements in each of a plurality of layers in a machine-learned model of a neural network including the plurality of layers; When the neural network includes an arithmetic process for outputting a tensor that is a result of a predetermined arithmetic operation performed on a tensor from a first layer and a tensor from one or more second layers that are located before the first layer, the first reduction rate and the one or more second reduction rates are adjusted based on one or more elements to be reduced in the first layer based on a first reduction rate and one or more elements to be reduced in each of the one or more second layers based on one or more second reduction rates. A machine learning method in which processing is performed by a computer.

[0222] (Appendix 7) the adjusting process includes adjusting the first reduction rate and the one or more second reduction rates so that the number of elements of the tensor from the first layer matches the number of elements of the tensor from each of the one or more second layers. The machine learning method described in Appendix 6.

[0223] (Appendix 8) the adjusting process includes adjusting the first reduction rate and the one or more second reduction rates so that elements that are targets of reduction in the first layer and all of the one or more second layers are targets of reduction in all of the layers, and elements that are not targets of reduction in at least one of the first layer and the one or more second layers are excluded from the targets of reduction in all of the layers; 8. The machine learning method according to claim 6 or 7.

[0224] (Appendix 9) Calculating a threshold value for the error of the tensor before and after the reduction of elements of each of the plurality of layers; causing the computer to execute a process; the selecting process includes a process of selecting, as the reduction rate, a reduction rate candidate to be applied to each of the plurality of layers based on the plurality of thresholds and an error of a tensor before and after reduction when the element is reduced by each of a plurality of reduction rate candidates in each of the plurality of layers; the adjusting process includes a process of adjusting the selected reduction rate candidates for the first tier and the one or more second tiers as the first reduction rate and the one or more second reduction rates, the determining process includes a process of determining a reduction rate to be applied to each of the plurality of layers based on the inference accuracy of the machine-learned model and the inference accuracy after machine learning of a reduced model obtained by reducing elements of each of the plurality of layers in the machine-learned model according to the reduction rate candidate. The machine learning method according to any one of Supplementary Note 6 to Supplementary Note 8.

[0225] (Appendix 10) The element is a channel, a weight, or a node. The machine learning method according to any one of Supplementary Notes 6 to 9.

[0226] (Appendix 11) Selecting a reduction rate for elements in each of a plurality of layers in a machine-learned model of a neural network including the plurality of layers; When the neural network includes an arithmetic process for outputting a tensor that is a result of a predetermined arithmetic operation performed on a tensor from a first layer and a tensor from one or more second layers that are located before the first layer, the first reduction rate and the one or more second reduction rates are adjusted based on one or more elements to be reduced in the first layer based on a first reduction rate and one or more elements to be reduced in each of the one or more second layers based on one or more second reduction rates. An information processing device comprising a control unit.

[0227] (Appendix 12) In the adjusting process, the control unit adjusts the first reduction rate and the one or more second reduction rates so that the number of elements of the tensor from the first layer matches the number of elements of the tensor from each of the one or more second layers. 12. The information processing device according to claim 11.

[0228] (Appendix 13) In the adjustment process, the control unit adjusts the first reduction rate and the one or more second reduction rates so that elements that are targets of reduction in the first layer and all of the one or more second layers are targets of reduction in all of the layers, and elements that are not targets of reduction in at least one of the first layer and the one or more second layers are excluded from the targets of reduction in all of the layers. 13. The information processing device according to claim 11 or 12.

[0229] (Appendix 14) The control unit Calculating a threshold value for the error of the tensor before and after element reduction for each of the plurality of layers; In the selecting process, a reduction rate candidate to be applied to each of the plurality of layers is selected as the reduction rate based on the plurality of thresholds and an error of a tensor before and after reduction when the element is reduced by each of a plurality of reduction rate candidates in each of the plurality of layers; In the adjustment process, the selected reduction rate candidates for the first layer and the one or more second layers are adjusted as the first reduction rate and the one or more second reduction rates; In the determining process, a reduction rate to be applied to each of the plurality of layers is determined based on the inference accuracy of the machine-learned model and the inference accuracy after machine learning of a reduced model obtained by reducing elements of each of the plurality of layers in the machine-learned model according to the reduction rate candidate. An information processing device according to any one of Supplementary notes 11 to 13.

[0230] (Appendix 15) The element is a channel, a weight, or a node. An information processing device according to any one of Supplementary Note 11 to Supplementary Note 14. [Explanation of symbols]

[0231] 1, 1A, 1B Server 10. Computers 11 Memory section 11a Untrained model 11b Data for machine learning 11c Machine Learning Model 11d pruning rate 11e lightweight model 12 Acquisition Department 13 Machine Learning Department 14, 14A, 14B Pruning rate calculation section (calculation section) 14a, 14a', 14a" Threshold calculation unit 14b, 14b', 14b” decision section 15 Output section 16 Control Unit

Claims

1. a neural network including a plurality of layers, the neural network including an arithmetic process for outputting a tensor that is a result of a predetermined operation on a tensor from a first layer and a tensor from one or more second layers that are located before the first layer; selecting a reduction rate for each of the elements in the plurality of layers in a machine-learned model of the neural network; adjusting the first reduction rate and the one or more second reduction rates based on one or more elements to be reduced in the first layer based on a first reduction rate and one or more elements to be reduced in each of the one or more second layers based on one or more second reduction rates; A machine learning program that lets a computer perform processing.

2. the adjusting process includes adjusting the first reduction rate and the one or more second reduction rates so that the number of elements of the tensor from the first layer matches the number of elements of the tensor from each of the one or more second layers. The machine learning program according to claim 1 .

3. the adjusting process includes adjusting the first reduction rate and the one or more second reduction rates so that elements that are targets of reduction in the first layer and all of the one or more second layers are targets of reduction in all of the layers, and elements that are not targets of reduction in at least one of the first layer and the one or more second layers are excluded from the targets of reduction in all of the layers. The machine learning program according to claim 1 or 2.

4. Calculating a threshold value for the error of the tensor before and after the reduction of elements of each of the plurality of layers; causing the computer to execute a process; the selecting process includes a process of selecting, as the reduction rate, a reduction rate candidate to be applied to each of the plurality of layers based on the plurality of thresholds and an error of a tensor before and after reduction when the element is reduced by each of a plurality of reduction rate candidates in each of the plurality of layers; the adjusting process includes a process of adjusting the selected reduction rate candidates for the first layer and the one or more second layers as the first reduction rate and the one or more second reduction rates, and determining a reduction rate to be applied to each of the plurality of layers based on the inference accuracy of the machine-learned model and the inference accuracy after machine learning of a reduced model obtained by reducing each element of the plurality of layers in the machine-learned model according to the reduction rate candidate. The machine learning program according to any one of claims 1 to 3.

5. The element is a channel, a weight, or a node. The machine learning program according to any one of claims 1 to 4.

6. a neural network including a plurality of layers, the neural network including an arithmetic process for outputting a tensor that is a result of a predetermined operation on a tensor from a first layer and a tensor from one or more second layers that are located before the first layer; selecting a reduction rate for each of the elements in the plurality of layers in a machine-learned model of the neural network; adjusting the first reduction rate and the one or more second reduction rates based on one or more elements to be reduced in the first layer based on a first reduction rate and one or more elements to be reduced in each of the one or more second layers based on one or more second reduction rates; A machine learning method in which processing is performed by a computer.

7. a neural network including a plurality of layers, the neural network including an arithmetic process for outputting a tensor that is a result of a predetermined operation on a tensor from a first layer and a tensor from one or more second layers that are located before the first layer; selecting a reduction rate for each of the elements in the plurality of layers in a machine-learned model of the neural network; adjusting the first reduction rate and the one or more second reduction rates based on one or more elements to be reduced in the first layer based on a first reduction rate and one or more elements to be reduced in each of the one or more second layers based on one or more second reduction rates; An information processing device comprising a control unit.

Images