Machine learning program, machine learning method, and information processing device

Abstract

To reduce the weight of a neural network including a plurality of layers.SOLUTION: A machine learning program causes a computer to execute processing of: acquiring, in a machine learned model of a neural network 170 including a plurality of layers, the reduction ratio of an element of each of the plurality of layers; when the neural network includes arithmetic processing 161 of outputting a tensor of a result of a predetermined arithmetic operation on a tensor from a pre-stage layer 165 and one or more tensors from one or more layers 163 shortcut coupled to the arithmetic processing 161 from before the pre-stage layer 165, inserting second layers 171, 172 performing zero-padding processing between the layers 163, 165 and the arithmetic processing; and performing padding by one or more zero matrices in the second layers associated with the layers 163, 165 so that the number of elements of the tensors input to the arithmetic processing from the layers 163, 165 after the reduction of elements according to the reduction ratio becomes a first number.SELECTED DRAWING: Figure 18

Description

[Technical Field]

[0001] The present invention relates to a machine learning program, a machine learning method, and an information processing device. [Background technology]

[0002] Neural networks (NNs) used in AI (Artificial Intelligence) tasks such as image processing tend to achieve higher performance (e.g., higher inference accuracy) by increasing the complexity of their configuration. On the other hand, increasing the complexity of the NN configuration can increase the number of calculations required to execute the NN on a computer and the memory size required by the computer to execute the NN.

[0003] "Pruning" is known as a method for reducing the number of calculations, in other words, shortening (speeding up) the calculation time, and reducing the memory size, in other words, making the NN machine learning model lightweight.

[0004] Pruning is a technique for reducing the data size of a machine learning model and reducing computation and communication time by eliminating (cutting out) at least one type of element: edges (weights), nodes, and channels of a neural network.

[0005] Excessive pruning can cause a degradation of the inference accuracy of the NN. Therefore, it is important to prune the NN while maintaining the inference accuracy or while keeping the degradation of the inference accuracy at a predetermined level.

[0006] For example, there is a known method for selecting layers that do not significantly affect the inference accuracy of a neural network during pruning. This method determines the channels of a convolutional layer to be pruned based on the parameters used in the batch normalization (BN) layer that follows the convolutional layer. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 2019-49977 Summary of the Invention [Problem to be solved by the invention]

[0008] The method of selecting layers that do not significantly affect the inference accuracy of the NN is applied to convolutional layers connected to BN layers, but is not intended to be applied to other layers, such as convolutional layers not connected to BN layers or fully connected layers.

[0009] Furthermore, a NN that includes these multiple layers may include a concatenate operation unit that performs a concatenate operation to connect inputs from two or more layers. Note that, hereinafter, the concatenate operation may be referred to as the concat operation, and the concatenate operation unit may be referred to as the concat unit.

[0010] The concat unit performs a computation process of shortcut-combining tensors input from a certain layer and one or more layers preceding that layer, and outputting a single tensor. For example, shortcut combinations include operations on the input tensors, such as addition for each dimension and each element.

[0011] For example, consider the case where the method for selecting layers that do not significantly affect the inference accuracy of a NN can be applied to multiple layers as described above, and the NN includes a concat section. In this case, if pruning is performed using this method, the dimensions (matrix sizes) of the tensors input to the concat section may not match between two or more layers, and the concat section may not output a correct calculation result.

[0012] For this reason, it is possible to exclude two or more layers that input tensors to the concat section from the layers to be pruned. However, in this case, the pruning rate of the entire machine learning model decreases, and the effect of pruning on compressing (reducing) the data size of the machine learning model is reduced.

[0013] In one aspect, an object of the present invention is to realize a lightweight neural network including multiple layers. [Means for solving the problem]

[0014] In one aspect, the machine learning program may cause a computer to perform the following process: The process may include: creating a neural network including multiple layers; the neural network includes a calculation process that outputs a tensor that is a result of a predetermined calculation performed on a plurality of input tensors, and the calculation process receives tensors output from each of a plurality of first layers that are in a stage prior to the calculation process. The method may include a process of obtaining a reduction rate for each element of the plurality of layers in the machine-learned model. Here, the plurality of first layers may include a layer preceding the arithmetic processing, and one or more layers preceding the preceding layer that are shortcut-coupled to the arithmetic processing. Moreover, the processing The aforementioned performing a zero padding process between each of the plurality of first layers and the calculation process; a plurality of second layers, each layer corresponding to a respective one of the plurality of first layers; The process may further include inserting a second layer into each of the plurality of first layers. The aforementioned The plurality of first layers are each subjected to the calculation process after the elements are reduced in accordance with a reduction ratio, so that the number of elements of each of the plurality of tensors input from the plurality of first layers to the calculation process becomes a first number. No. This may include padding with one or more zero matrices in each of the two layers. [Effects of the Invention]

[0015] In one aspect, the present invention can achieve weight reduction of a neural network including multiple layers. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 10 is a diagram illustrating an example of a process for determining a channel of a convolutional layer on which pruning is performed. [Figure 2] FIG. 10 is a diagram illustrating an example of L1 regularization learning. [Figure 3] FIG. 3 is a diagram showing an example of whether the methods of FIGS. 1 and 2 can be applied to a layer of a neural network. [Figure 4] FIG. 2 is a block diagram illustrating an example of a functional configuration of a server according to an embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of calculation of a pruning rate with guaranteed accuracy. [Figure 6] FIG. 10 is a diagram illustrating an example of calculation of the accuracy of a model before and after pruning. [Figure 7] FIG. 10 is a diagram illustrating an example of searching for a pruning rate. [Figure 8] FIG. 10 is a diagram illustrating an example of a threshold value derivation method. [Figure 9] FIG. 10 is a diagram illustrating an example of an upper limit of a threshold value and a threshold value. [Figure 10] FIG. 10 is a diagram illustrating an example of a method for determining channels to be pruned. [Figure 11] FIG. 10 is a diagram illustrating an example of calculating a pruning error. [Figure 12] FIG. 10 is a diagram illustrating an example of a method for determining nodes to be pruned. [Figure 13] FIG. 10 is a diagram illustrating an example of calculating a pruning error. [Figure 14] FIG. 10 is a diagram illustrating an example of a method for determining weights for pruning. [Figure 15] FIG. 10 is a diagram illustrating an example of calculating a pruning error. [Figure 16] FIG. 10 is a diagram illustrating an example of a NN including a concat section. [Figure 17] FIG. 10 is a diagram illustrating an example of a tensor before and after application of a method according to an embodiment to a NN including a concat section. [Figure 18] FIG. 10 is a diagram illustrating an example of inserting a zero-padding layer into a model. [Figure 19] FIG. 10 is a diagram illustrating an example of zero padding for a layer immediately before the concat in the concat part. [Figure 20] FIG. 10 is a diagram illustrating an example of zero padding for a layer immediately before the concat in the concat part. [Figure 21] FIG. 10 is a diagram illustrating an example of zero padding for a layer immediately before the concat in the concat part. [Figure 22] FIG. 10 is a diagram illustrating an example of zero padding for a layer immediately before the concat in the concat part. [Figure 23] FIG. 10 is a diagram illustrating an example of zero padding for a layer immediately before the concat in the concat part. [Figure 24] FIG. 10 is a diagram illustrating an example of zero padding for a layer immediately before the concat in the concat part. [Figure 25] FIG. 10 is a diagram illustrating an example of the accuracy before and after pruning of a neural network and the compression rate of data size, depending on whether or not zero padding processing is applied. [Figure 26] FIG. 10 is a diagram illustrating an example of the number of output channels in each layer of a neural network depending on whether or not zero padding processing is applied. [Figure 27] 10 is a flowchart illustrating an example of the operation of a process performed by a server according to an embodiment. [Figure 28] FIG. 10 is a diagram illustrating an example of a pruning error comparison result in response to an update of the confidence radius in a method according to an embodiment. [Figure 29] FIG. 10 is a block diagram showing an example of the functional configuration of a server according to a first modified example. [Figure 30] 10A and 10B are diagrams illustrating an example of a trust radius update process when increasing a trust radius. [Figure 31] 10A and 10B are diagrams illustrating an example of a trust radius update process when a trust radius is decreased. [Figure 32] 10 is a flowchart illustrating an example of the operation of a process performed by a server according to a first modified example. [Figure 33] FIG. 10 is a block diagram showing an example of the functional configuration of a server according to a second modified example. [Figure 34] FIG. 10 is a diagram illustrating an example of setting an initial value of a trust radius. [Figure 35] 10 is a flowchart illustrating an example of the operation of a process performed by a server according to a second modified example. [Figure 36]FIG. 1 is a block diagram showing an example of the hardware (HW) configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiments described below are merely examples, and are not intended to exclude various modifications or application of techniques not explicitly described below. For example, the present embodiment can be implemented with various modifications within the scope of the spirit thereof. In the drawings used in the following description, parts with the same reference numerals represent the same or similar parts unless otherwise specified.

[0018] [1] One embodiment Fig. 1 is a diagram illustrating an example of a process for determining the convolutional layer channel for pruning, and Fig. 2 is a diagram illustrating an example of L1 regularization learning. Fig. 1 illustrates a method for selecting a layer that does not significantly affect the inference accuracy of a neural network (NN), in which a computer determines the convolutional layer channel for pruning using a scaling coefficient γ used in the BN layer 100 following the convolutional layer. Note that the graphs shown for channels 111 to 113 in Fig. 1 represent the distribution of the output tensor.

[0019] As shown in FIG. 1, the computer performs normalization processing 101 on each of a plurality of channels 111 (#1 to #n; n is an integer equal to or greater than 2) input from the convolutional layer to the BN layer 100. For example, in the normalization processing 101, the computer calculates a mean value μ and a variance σ for each channel 111 according to the following formula (1): 2 By calculating z, multiple channels 112 (#1 to #n) that represent a normalized distribution with a mean of "0" and a variance of "1" are obtained. in and z mid indicate channels 111 and 112, respectively, and μ B and σ B 2 denote the mean and variance in the current mini-batch B, respectively.

number

[0020] The computer also performs scaling 102 on the multiple channels 112 (#1 to #n). For example, in scaling 102, the computer multiplies each of the multiple channels 112 by a scaling coefficient γ according to the following equation (2), and adds a bias β to the multiplication result, thereby outputting multiple channels 113 (#1 to #n) that represent a distribution scaled by the parameters γ and β. In the following equation (2), z out indicates the channel 113. Note that the parameters γ and β may be optimized by machine learning.

number

[0021] Here, when γ is small, the output of channel 113 (channel #n in the example of FIG. 1) resulting from scaling 102 is almost zero. This means that even if the channel is deleted by pruning, it does not have a significant impact on the inference accuracy of the NN. Therefore, the computer determines the channels to be pruned on a channel-by-channel basis by searching for a small γ (for example, "0").

[0022] For example, a computer searches for a small (smaller) γ by applying L1 regularization learning to γ. L1 regularization learning is a machine learning method known for making the parameters of the learning target "sparse" by performing machine learning by adding an L1 regularization term to the loss function calculated by the NN output.

[0023] 2, a computer performs L1 regularization learning on a certain vector 121 using a loss function 122 to obtain an L1 regularized vector 123. The loss function 122 may be a function L obtained by adding an original loss function (first term) such as cross-entropy to an L1 regularization term (second term) that uses an L1 norm (Σg(γ)=Σ|γ|), as shown in the following formula (3).

number

[0024] By L1 regularization learning, each parameter of vector 123 becomes a parameter indicating whether each parameter of vector 121 is zero or non-zero (is bifurcated). By using such L1 regularization learning, the computer can identify channels where γ is zero (close to zero) as channels to be pruned.

[0025] The identification of pruning targets using L1 regularization learning shown in Figures 1 and 2 is applied to convolutional layers connected to BN layers, but is not intended to be applied to other layers, such as convolutional layers not connected to BN layers and fully connected layers.

[0026] Fig. 3 is a diagram showing an example of whether the techniques of Fig. 1 and Fig. 2 can be applied to layers 131 to 139 of the NN 130. As shown in Fig. 3, the convolutional layers 131 and 133 and the BN layers 132 and 134 are layers to which the L1 regularization learning shown in Fig. 1 and Fig. 2 can be applied, while the convolutional layers 135 to 137 and the fully connected layers 138 and 139 are layers to which the L1 regularization learning shown in Fig. 1 and Fig. 2 cannot be applied.

[0027] Therefore, in one embodiment, a method for realizing a lightweight NN by determining a pruning rate for each layer regardless of the type of layer will be described.

[0028] [1-1] Example of functional configuration of server according to one embodiment 4 is a block diagram showing an example of a functional configuration of server 1 according to an embodiment. Server 1 is an example of a calculator, computer, or information processing device that outputs a pruning rate. As shown in FIG. 4, server 1 may illustratively include a memory unit 11, an acquisition unit 12, a machine learning unit 13, a pruning rate calculation unit (hereinafter simply referred to as a "calculation unit") 14, and an output unit 15. The acquisition unit 12, the machine learning unit 13, the calculation unit 14, and the output unit 15 are examples of a control unit 16.

[0029] The memory unit 11 is an example of a storage area, and stores various data used by the server 1. As shown in Fig. 4, the memory unit 11 may be capable of storing, for example, an unlearned model 11a, machine learning data 11b, a machine-learned model 11c, a pruning rate 11d, and a lightweight model 11e.

[0030] The acquisition unit 12 acquires the unlearned model 11a and the machine learning data 11b and stores them in the memory unit 11. For example, the acquisition unit 12 may generate one or both of the unlearned model 11a and the machine learning data 11b in the server 1, or may receive them from a computer external to the server 1 via a network (not shown).

[0031] The unlearned model 11a may be a pre-machine learning model of a neural network (NN) including unlearned parameters. The NN may include various layers, such as a deep neural network (DNN). The NN may include, for example, a convolutional layer not connected to a BN layer, a fully connected layer, or a convolutional layer connected to a BN layer. For example, the NN may be the NN 130 illustrated in FIG. 3.

[0032] The machine learning data 11b may be, for example, a training data set used for machine learning (training) of the unlearned model 11a. As an example, when performing machine learning of a neural network to realize image processing, the machine learning data 11b may include, for example, a plurality of pairs of training data such as image data and teacher data including a correct label for the training data.

[0033] In the machine learning phase, the machine learning unit 13 executes a machine learning process to train the unlearned model 11a based on the machine learning data 11b. For example, the machine learning unit 13 may generate the machine-learned model 11c by the machine learning process of the unlearned model 11a. The machine-learned model 11c may be an NN model including machine-learned parameters.

[0034] The machine-learned model 11c may be obtained by updating parameters included in the unlearned model 11a, and may be regarded as a model resulting from a change from the unlearned model 11a to the machine-learned model 11c through machine learning processing. The machine learning processing may be realized by various known techniques.

[0035] The calculation unit 14 calculates the pruning rate 11d by executing a pruning rate calculation process on the machine-learned model 11c, and stores the calculated pruning rate 11d in the memory unit 11.

[0036] For example, the calculation unit 14 may include a threshold calculation unit 14a that calculates a threshold for each layer to select one of the pruning rate candidates, and a determination unit 14b that determines the pruning rate 11d to be adopted based on the inference accuracy of the model pruned using the pruning rate candidates.

[0037] The output unit 15 outputs output data based on the pruning rate 11d generated (acquired) by the calculation unit 14. The output data may include, for example, the pruning rate 11d itself and / or the light-weighted model 11e.

[0038] The lightweight model 11e is data of a model obtained by pruning the machine-learned model 11c based on the pruning rate 11d. For example, the output unit 15 may cooperate with the machine learning unit 13 to apply the pruning rate 11d to prune and re-learn the machine-learned model 11c, thereby acquiring the lightweight model 11e and storing it in the memory unit 11. Note that the lightweight model 11e may be generated separately from the machine-learned model 11c, or may be data obtained by updating the machine-learned model 11c through pruning and re-learning, for example.

[0039] When outputting the output data, the output unit 15 may, for example, transmit (provide) the output data to another computer (not shown), or may store the output data in the memory unit 11 and manage it so that it can be acquired from the server 1 or another computer. Alternatively, when outputting the output data, the output unit 15 may output information indicating the output data to the screen of an output device such as the server 1, or may output the output data in various other modes.

[0040] [1-2] Example of pruning rate calculation process Next, a description will be given of an example of a pruning rate calculation process performed by the calculation unit 14 of the server 1. In the following description, it is assumed that the target for calculating the pruning rate is a weight matrix W, which is an example of a layer parameter.

[0041] The calculation unit 14 determines the pruning rate regardless of the type of layer by utilizing the tensor error for each layer that occurs due to pruning. As an example, the calculation unit 14 may calculate the pruning rate by the following steps (i) to (iii).

[0042] (i) The calculation unit 14 (threshold calculation unit 14a) determines (calculates) a pruning rate for which accuracy can be guaranteed for each layer.

[0043] Note that "guaranteeing accuracy" means, for example, guaranteeing that the accuracy of inference (inference accuracy) using the light-weight model 11e obtained by pruning the machine-learned model 11c exceeds a predetermined standard.

[0044] 5 is a diagram showing an example of calculation of a pruning rate for which accuracy can be guaranteed. As illustrated in FIG. 5, in (i), the threshold calculation unit 14a determines a pruning rate to be applied to the weight matrix W of each layer included in the machine-learned model 11c to be pruned, for each weight matrix W of a plurality of layers. Note that, while FIG. 5 focuses on layers 131 to 133 for explanation, this is not limiting, and the explanation of FIG. 5 may be applied to any of layers 131 to 139 illustrated in FIG. 3.

[0045] Here, the pruning rate is an example of the rate at which elements in a layer are reduced (reduction rate), and indicates the rate at which pruning targets in the machine-learned model 11c are made "sparse." In the example of Figure 2, it means the number of points in vector 123 that are set to "0."

[0046] As illustrated in Fig. 5, the threshold calculation unit 14a selects one pruning rate from a plurality of pruning rate candidates for each of the weight matrix W1 of the layer 131 (weight matrix W1 connected to the layer 132) and the weight matrix W2 between the layers 132 (weight matrix W2 connected to the layer 133). The pruning rate candidate is an example of a reduction rate candidate, and may be, for example, two or more rates between 0% and 100%, may be common to multiple layers, may be different for each layer, or may be a combination of these. In the example of Fig. 5, the pruning rate candidates are assumed to be 0%, 20%, 40%, and 60%.

[0047] The threshold calculation unit 14a calculates the error of the tensor before and after pruning when pruning is performed using each of the pruning rate candidates, and calculates a threshold T W 5, the threshold calculation unit 14a determines the maximum pruning rate candidate among the pruning rate candidates having an error smaller than the threshold T w1The maximum pruning rate candidate with a smaller error than the threshold T w2 The maximum pruning rate candidate with a smaller error than is determined to be 20% (see arrow 142).

[0048] Threshold T w is a threshold value of the error of the tensor before and after pruning, and is the upper limit of the pruning rate at which accuracy can be guaranteed. For example, the threshold calculation unit 14a calculates a threshold T w The threshold T w The calculation method will be described in detail later.

[0049] The pruning rate calculated in (i) may be considered as a "provisionally calculated" pruning rate in relation to the processes in (ii) and (iii).

[0050] As described above, the threshold calculation unit 14a calculates the threshold T for the tensor error before and after reduction of elements in each of the multiple layers in the machine-learned model 11c of the NN including multiple layers. Furthermore, the threshold calculation unit 14a selects a reduction rate candidate to be applied to each of the multiple layers based on the multiple thresholds T and the tensor error before and after reduction when elements are reduced in each of the multiple layers by each of the multiple reduction rate candidates.

[0051] (ii) The calculation unit 14 (determination unit 14b) determines the pruning rate based on the accuracy of the machine learning model that has been pruned (lightened) using the pruning rate determined in (i) and the accuracy of the machine learning model that has not been pruned.

[0052] For example, the determination unit 14b determines the accuracy Acc of the model pruned at the pruning rate for each layer determined in (i) by taking into account the error due to the approximation formula (first-order Taylor expansion). p and Accuracy Margin Acc m and the accuracy of the unpruned model Acc wo Accuracy margin Accm is a margin that allows for a decrease in inference accuracy, and may be set by the designer. Note that the margin may be "0", in which case the determining unit 14b determines the accuracy Acc p and the accuracy Acc of the model without pruning wo Just compare it with.

[0053] 6 is a diagram illustrating an example of calculation of the accuracy of a model before and after pruning. For example, the determination unit 14b calculates the accuracy Acc of a model (machine-learned model 11c) that is not pruned for all layers (W1, W2, . . . ). wo (See arrow 143). A model without pruning may be regarded as a model pruned with a pruning rate of 0% for each layer. Furthermore, the determination unit 14b calculates the accuracy Acc of a model pruned with a pruning rate of 0% for each layer (W1=40%, W2=20%, ...) calculated in (i). p is calculated (see arrow 144).

[0054] The determination unit 14b determines the sum of the accuracies Acc p +Acc m Accuracy Acc wo If this is the case, the determination unit 14b determines to adopt the pruning rate determined in (i). For example, the determination unit 14b stores the pruning rate determined in (i) in the memory unit 11 as the pruning rate 11d.

[0055] On the other hand, the determination unit 14b determines the sum of the accuracies Acc p +Acc m Accuracy Acc wo If the pruning rate is less than the predetermined value, the determination unit 14b determines to discard the pruning rate determined in (i). For example, the determination unit 14b determines to discard the pruning rate determined in (i) and adopt the pruning rate 11d determined in the immediately preceding (ii) (or the initial) pruning rate 11d.

[0056] (iii) The calculation unit 14 (determination unit 14b) searches for the maximum pruning rate for which accuracy can be guaranteed by repeatedly applying (i) and (ii) multiple times.

[0057] Fig. 7 is a diagram showing an example of searching for a pruning rate. In the example of Fig. 7, the calculation unit 14 searches for pruning rates for three layers (131 to 133) three times.

[0058] As illustrated in FIG. 7, in the first search (see reference numeral 145), in (i), the threshold calculation unit 14a calculates the threshold T w Calculate the threshold T w It is assumed that the pruning rates of the layers 131 to 133 are changed from "0%, 0%, 0%" (initial value) to "40%, 20%, 40%" based on the above. For example, in (ii), the determination unit 14b determines whether the Acc p +Acc m <Acc wo If it is determined that the pruning rate is 0%, the pruning rate determined in (i) is discarded, and the previous rate of 0%, 0%, 0% is adopted.

[0059] In the second search (see reference numeral 146), in (i), the threshold calculation unit 14a calculates the threshold T w Calculate (update) the updated threshold T w It is assumed that the pruning rates of the layers 131 to 133 are determined from "0%, 0%, 0%" to "20%, 20%, 40%" based on the above. For example, in (ii), the determination unit 14b determines whether the Acc p +Acc m ≧Acc wo If it is determined that the pruning rate is 20%, 20%, 40%, it adopts the pruning rate 11d and stores it in the memory unit 11.

[0060] In the third search (see reference numeral 147), in (i), the threshold calculation unit 14a calculates the threshold T w Calculate (update) the updated threshold T w For example, in (ii), the determination unit 14b determines the pruning rates of the layers 131 to 133 from "20%, 20%, 40%" to "20%, 40%, 40%" based on the above. p +Acc m ≧Acc woIf it is determined that "20%, 40%, 40%" is adopted, it is stored (updated) in the memory unit 11 as the pruning rate 11d.

[0061] The decision unit 14b may search for the pruning rate a predetermined number of times, for example, a preset number of times.

[0062] As described above, the determination unit 14b determines the reduction rate to be applied to each of the multiple layers based on the inference accuracy of the machine-learned model 11c and the inference accuracy after machine learning of the reduced model obtained by reducing the elements of each of the multiple layers in the machine-learned model 11c according to the candidate reduction rates to be applied.

[0063] Next, a specific example of the above-mentioned pruning rate calculation process will be described. Fig. 8 is a diagram for explaining an example of a threshold value derivation method, and Fig. 9 is a diagram showing an example of the upper limit of the threshold value and the threshold value.

[0064] The threshold calculation unit 14a calculates a threshold value of the pruning rate for each layer that can guarantee accuracy by performing a first-order Taylor expansion of the loss function when pruning is performed. For example, the error of the tensor for each layer that occurs due to pruning is Δw, the loss function when pruning is performed is L(w+Δw), the loss function of the model to be pruned is L(w), and the loss function when not pruning is L ideal ) to L wo +L m Then, the threshold value of the pruning rate that can guarantee accuracy is calculated by the following formula (4). wo is the loss function of the model without pruning, and L m is the margin of the loss function set by the designer.

number

[0065] The left side of the above equation (4) (see the dashed box in Figure 8) is a Taylor expansion of the loss function L(w + Δw) after pruning, and includes the weight gradient "∂L(W) / ∂w" for each layer to be pruned. The gradient for each layer may be calculated by backpropagation. The right side of the above equation (4) (see the dashed-dotted box in Figure 8) is a constraint that the loss function will be smaller than the ideal value (for example, the FP32 loss function) even after pruning.

[0066] In this way, the threshold calculation unit 14a calculates the threshold T based on the value of the loss function of the machine-learned model 11c when reducing the elements of each of the multiple layers and the weight gradient of each of the multiple layers.

[0067] By rearranging the above formula (4), it is possible to derive a condition for the "pruning error" that satisfies the constraint that the loss function after pruning is smaller than the ideal loss function, as shown in the following formula (5). In other words, it is possible to derive an upper limit (threshold) for the error due to pruning that guarantees accuracy (loss function). The threshold calculation unit 14a sets the right-hand side of the following formula (5) as the threshold T.

number

[0068] 9, the threshold calculation unit 14a compares a threshold T set for each layer with the error of the L1 norm due to pruning. Then, the threshold calculation unit 14a determines the pruning rate candidate with the largest value (40% in the example of FIG. 9) among the pruning rate candidates that result in an error smaller than the threshold T as the pruning rate resulting from (i).

[0069] As an example, the threshold calculation unit 14a may determine a pruning rate for each layer to be pruned, at which the pruning error (left side) is equal to or smaller than the threshold (right side) according to the following formula (6): In the following formula (6), "||ΔW||1" is the L1 norm of the weights to be pruned, and "n" is the number of weight elements in the layer to be pruned.

number

[0070] As shown in the above formula (6), the threshold value T is a parameter derived by approximation. In order to prevent an error in determining the pruning rate due to the approximation error, an upper limit may be set for the threshold value T (see FIG. 9). For example, the threshold value calculation unit 14a may limit the magnitude of the threshold value T by the "trust radius" based on the trust region method. The trust radius is an example of the threshold upper limit. As an example, the threshold value calculation unit 14a may scale the threshold value T so that the L2 norm of the threshold value T for all layers is less than or equal to the trust radius. In the example of FIG. 9, T h represents a vector by the threshold value T for each layer, and "||T h ||2" represents the L2 norm of the threshold value T for all layers.

[0071] For example, the threshold value calculation unit 14a may update the trust radius (e.g., by a constant multiple, etc.) in addition to the pruning rate according to the comparison result of the accuracy in the process of (ii) by the determination unit 14b. Note that the initial value of the trust radius may be set by, for example, a designer or the like.

[0072] As an example, when the sum of accuracies Acc p +Acc m is greater than or equal to the accuracy Acc wo , the trust radius is multiplied by a constant K ("K>1.0"), and when the sum of accuracies Acc p +Acc m is less than the accuracy Acc wo , the trust radius may be multiplied by a constant k ("0<k<1.0").

[0073] [1-3] Explanation according to the type of pruning target Next, examples of pruning methods and pruning error calculation methods according to the type of pruning target will be described. Examples of the type of pruning target include, for example, channel pruning, node pruning, and weight pruning. The calculation unit 14 may determine the pruning target and the pruning error using the weight corresponding to the pruning target according to the type of pruning target.

[0074] [1-3-1] Example of channel pruning FIG. 10 is a diagram for explaining an example of a method for determining channels to be pruned, and FIG. 11 is a diagram for explaining an example of calculating a pruning error.

[0075] 10 and 11 show the processing flow of the convolution operation. Also, the subscripts H and W indicate the sizes of the input data, kernel, and output data, and the subscript Ch indicates the number of channels of the input data, kernel, and output data. The same applies to the following explanations of other types of pruning targets.

[0076] (An example of a method for determining which channels to prune) When the type of pruning target is a channel, the calculation unit 14 calculates the L1 norm for each kernel corresponding to the channel of the output data. For example, as shown in "Before pruning" in Fig. 10, the calculation unit 14 calculates the L1 norm for each of all kernels for one channel before pruning. As a result, the L1 norm for one channel is calculated.

[0077] Next, as illustrated in "after pruning" in FIG. 10, the calculation unit 14 prunes the channels of the corresponding output data in ascending order of the calculated L1 norm according to the set pruning rate.

[0078] (Example of pruning error calculation) 11, the calculation unit 14 calculates the L1 norm of the kernel to be pruned. The L1 norm of the kernel to be pruned is the L1 norm of all kernels before pruning minus the L1 norm of all kernels after pruning, that is, the difference between the L1 norms before and after pruning.

[0079] The calculation unit 14 may obtain the pruning error by dividing the calculated L1 norm by the number of elements in all kernels before pruning.

[0080] [1-3-2] Example of node pruning FIG. 12 is a diagram for explaining an example of a method for determining nodes to be pruned, and FIG. 13 is a diagram for explaining an example of calculating a pruning error.

[0081] (An example of a method for determining which nodes to prune) When the type of pruning target is a node, the calculation unit 14 calculates the L1 norm for each weight connected to the output node. In the example "before pruning" in Fig. 12, the calculation unit 14 calculates the L1 norm for each of the solid line, dashed line, and dashed dotted line.

[0082] Next, the calculation unit 14 prunes the corresponding output nodes in ascending order of the calculated L1 norm according to the set pruning rate, as illustrated in "After pruning" in Fig. 12. For example, the calculation unit 14 determines the output node corresponding to the weight group with the smallest L1 norm as the node to be pruned.

[0083] (Example of pruning error calculation) 13, the calculation unit 14 calculates the L1 norm of the weight group to be pruned. The L1 norm of the weight group to be pruned is obtained by subtracting the L1 norm of all weights after pruning from the L1 norm of all weights before pruning.

[0084] The calculation unit 14 may obtain the pruning error by dividing the calculated L1 norm by the number of elements in all weights before pruning. In the example of "after pruning" in Fig. 13, the calculation unit 14 calculates the L1 norm of the weight group of the two-dot chain lines, and divides the L1 norm by the number of elements in all weights before pruning (="6"; the number of lines).

[0085] [1-3-3] Weight pruning example FIG. 14 is a diagram for explaining an example of a method for determining weights for pruning, and FIG. 15 is a diagram for explaining an example of calculating a pruning error.

[0086] (An example of a method for determining the weights to be pruned) When the type of pruning target is weight, the calculation unit 14 calculates the L1 norm for each element of all weights. In the example of "before pruning" in Fig. 14, the number of weight elements is "6", so the calculation unit 14 calculates "6" L1 norms.

[0087] Next, the calculation unit 14 prunes the corresponding weights in ascending order of the calculated L1 norm according to the set pruning rate, as illustrated in "After pruning" in Fig. 14. For example, the calculation unit 14 determines the weight with the smallest L1 norm as the weight to be pruned.

[0088] (Example of pruning error calculation) 15, the calculation unit 14 calculates the L1 norm of the weights to be pruned. The L1 norm of the weights to be pruned is obtained by subtracting the L1 norm of all weights after pruning from the L1 norm of all weights before pruning.

[0089] The calculation unit 14 may obtain the pruning error by dividing the calculated L1 norm by the number of elements in all weights before pruning. In the example of "after pruning" in Fig. 15, the calculation unit 14 calculates the L1 norm of the weights of the dashed lines and divides the L1 norm by the number of elements in all weights before pruning (="6"; the number of lines).

[0090] [1-4] Explanation of pruning process for NN including concat part FIG. 16 is a diagram showing an example of an NN 150 including a concat unit 154, and FIG. 17 is a diagram showing an example of a tensor before and after application of the method according to the embodiment described above to the NN 150 including the concat unit 154.

[0091] As illustrated in FIG. 16 , the NN 150 includes layers 151 to 153 and 155 (represented as layers 1, 3, 2, and 4, respectively) and a concat unit 154. The concat unit 154 receives tensors from layers 1 and 2 as input, adds these tensors, and outputs the result of the addition to layer 4. The concat operation is an example of an arithmetic process that outputs a tensor that is the result of a predetermined operation on the tensors from layer 1 and the tensors from layer 2. Note that the following description will be given as an example of a case where there are two input layers that input tensors to the concat unit 154, but this is not limiting, and there may be three or more input layers. In other words, the concat unit 154 may receive as input a tensor from a previous layer connected before the concat unit 154 and one or more tensors from one or more layers that are previous to the previous layer and are shortcut-coupled to the concat unit 154.

[0092] For simplicity, the following will be an example in which the elements to be pruned are channels, and each of layers 1 and 2 outputs a one-dimensional tensor with three channels (number of elements "3"). Note that elements can include, for example, weights or nodes in addition to channels.

[0093] In the example of FIG. 16, the concat unit 154 adds the tensor

[0101] output from layer 1 and the tensor

[0011] output from layer 2 for each channel, and outputs the resulting tensor

[0112] to layer 4. Note that each column in the tensor corresponds to an element (e.g., a channel) to be pruned. Hereinafter, each column in the tensor may be expressed as [ch1,2]. [ch1,2] means channel 2 of layer 1. The position of a channel within a layer can be identified (specified) by an index.

[0094] Here, when the method according to the embodiment described above is applied to the NN 150 shown in FIG. 16, the pruning rate for each of the layers 1 to 4 is determined individually, and pruning of each of the layers 1 to 4 is performed according to the determined pruning rate.

[0095] For example, as shown in Fig. 17, assume that [ch1,3] of the tensors [Ch1,1], [ch1,2], and [ch1,3] output from layer 1 is pruned, and [ch2,2] and [ch2,3] of the tensors [Ch2,1], [ch2,2], and [ch2,3] output from layer 2 are pruned. In this case, concat unit 154 cannot perform the concat operation because the number of channels input from layer 1 and layer 2, respectively, in other words, the number of elements (size) of the channel dimensions of the tensor, are different, being "2" and "1." In the example of Fig. 17, this is because there is no addition partner for [ch1,2].

[0096] To avoid the concat operation becoming impossible, for example, it is conceivable to uniformly exclude all layers that output tensors that are input to the concat operation from the targets for determining the pruning rate. However, in this case, the pruning rate of the entire machine learning model of the NN decreases as the number of concat units 154 included in the NN increases, and the effect of compressing (reducing) the data size of the machine learning model by pruning decreases.

[0097] Therefore, the calculation unit 14 according to one embodiment inserts a zero-padding layer on the output side of all layers (hereinafter, sometimes referred to as "layers immediately before concat") that are input to the concat unit 154 (that output tensors to the concat unit 154).

[0098] A zero-padding layer is a layer that pads certain elements (e.g., channels) of a tensor with "0" (zero). Padding is an operation that increases the size of a tensor (e.g., the number of channels) by embedding values ​​such as zeros in the tensor. The layer immediately before concat is an example of multiple first layers, and a zero-padding layer is an example of multiple second layers.

[0099] For example, the calculation unit 14 may match the number of elements (sizes) of tensors after pruning in all layers immediately before concat that input tensors to the same concat unit 154, for example, the number of dimensions of the channels of the tensors, by zero padding in the zero padding layer. For example, the calculation unit 14 may identify the number of channels in the layer immediately before concat based on the provisionally calculated pruning rate, and determine the number of channels to perform zero padding on according to the identified number of channels.

[0100] FIG. 18 is a diagram illustrating an example of inserting a zero-padding layer into a model.

[0101] The process of inserting a zero-padding layer may be performed by selecting a pruning rate candidate when the NN to be pruned includes the concat unit 154, and may be suppressed when the NN does not include the concat unit 154. For example, the calculation unit 14 may determine whether the NN includes the concat unit 154 by referring to configuration information (not shown) that defines the configuration of the NN, such as the configuration of each layer and the connection relationship between layers. Furthermore, the calculation unit 14 may identify the layer immediately before the concat for each concat unit 154 based on the configuration information.

[0102] 18 illustrates an example in which, in the above-described (i), the calculation unit 14 calculates (calculates) the L1 norm for each kernel corresponding to the channel of the output data, and provisionally calculates the pruning rate by L1 regularization learning (see FIG. 2), etc. In the example of FIG. 17, if the provisionally calculated pruning rate is "33%" for layer 1 and "66%" for layer 2, the calculation unit 14 prunes one of the three channels of layer 1 and prunes two of the three channels of layer 2.

[0103] 18, model 160 before the insertion of the zero-padding layer includes concat units 161 and 162 and a plurality of layers 163 to 167. Tensors output from layers 163 and 165 are input to concat unit 161, and tensors output from concat unit 161 and layer 167 are input to concat unit 162.

[0104] In model 160, if the number of output channels of layer 163 becomes "10" and the number of output channels of layer 165 becomes "6" as a result of pruning, the numbers of input channels to concat unit 161 will not match. Also, even if a tensor with the number of output channels of "10" is output from concat unit 161, if the number of output channels of layer 167 becomes "14", the numbers of input channels to concat unit 162 will not match.

[0105] Therefore, as shown in model 170, the calculation unit 14 inserts (places) zero-padding layers 171 to 174 on the output side of each of layers 163 and 165, concat unit 162, and layer 167, which are layers immediately before concat. Then, the calculation unit 14 performs zero padding for each concat unit so that the number of channels of the tensors input to each concat unit 161 and 162 matches.

[0106] For example, the calculation unit 14 performs zero padding for "4" channels on the output tensor of layer 165 in the zero padding layer 172 so that the number of output channels of all layers immediately preceding the concat of the concat unit 161 matches the maximum value of "10" on the layer 163 side. This allows the concat unit 161 to use a tensor with "10" input channels to output a tensor that is the result of the concat operation with "10" output channels.

[0107] Furthermore, for example, the calculation unit 14 performs zero padding for "4" channels on the output tensor of the concat unit 161 in the zero padding layer 173 so that the number of output channels of all layers immediately preceding the concat unit 162 matches the maximum value of "14" on the layer 167 side. This allows the concat unit 162 to output a tensor that is the result of the concat operation and has "14" output channels, using a tensor with "14" input channels.

[0108] 19 to 24 are diagrams illustrating examples of zero padding performed by concat section 154 on the layer immediately before concat.

[0109] As illustrated in FIG. 19, the calculation unit 14 compares the number of channels after pruning in layers 1 and 2, which are layers immediately before concat, and obtains the maximum number of channels (layer 1's "2"). Then, the calculation unit 14 performs zero padding for the missing number of channels ("2-1=1") in the zero padding layer corresponding to layer 2, which has a channel number of "1" that is less than the maximum value. This allows the concat unit 154 to use the zero matrix added by padding as the addition partner for [Ch1,2].

[0110] The calculation unit 14 may perform zero padding on the number of channels that is less than the maximum value (missing number). For example, as shown in Fig. 20, if the number of channels after pruning in layers 1 and 2 is "3" and "1", respectively, the calculation unit 14 may add zero matrices to layer 2 for the number of channels, "2".

[0111] In this way, in the layer immediately before concat, elements may be reduced according to the reduction rate of each layer immediately before concat, so the number of elements (size) of tensors differs between the layers immediately before concat. Therefore, calculation unit 14 performs padding with one or more zero matrices in each of zero-padding layers 17 so that the size of each of the multiple tensors input from the layer immediately before concat to concat unit 154 becomes the first number. The first number is the number of elements (size) of tensors between the layers immediately before concat.

[0112] Note that the concat unit 154 can tolerate mismatching of element positions, for example, channel indices, in the concat operation. For this reason, the calculation unit 14 may not consider matching of the indices of the channels input to the concat unit 154, or may match the indices by zero padding. For example, the calculation unit 14 may change the number (first number) by which the number of elements (sizes) of tensors between layers immediately before concat are matched by zero padding, in each of the cases where matching of the indices is not considered and where the indices are matched.

[0113] (without considering index matches) As illustrated in FIG. 21, when [Ch2,2] of layer 2 is pruned, the calculation unit 14 may insert zero matrices so that the index order becomes [Ch2,1], [Ch2,3], and [zero]. In this case, the calculation unit 14 pads layer 1 with zero zero matrices and pads layer 2 with one zero matrix. Note that [zero] indicates a zero matrix. The concat unit 154 performs concat operations of [Ch1,1]+[Ch2,1], [Ch1,2]+[Ch2,3], and [Ch1,3]+[zero].

[0114] As another example, as shown in FIG. 22, when [Ch1,3] of Layer 1 and [Ch2,2] of Layer 2 are pruned, the channel indexes do not match, but the number of channels in both layers is the maximum value of "2." In this case, the calculation unit 14 may suppress zero padding. In other words, the calculation unit 14 performs padding with zero zero matrices in each of Layer 1 and Layer 2. The concat unit 154 performs concat operations of [Ch1,1]+[Ch2,1] and [Ch1,2]+[Ch2,3].

[0115] In this way, when index matching is not taken into consideration, the calculation unit 14 uses the maximum number of elements among the multiple tensors output from the multiple layers immediately before concat after element reduction as the first number. The maximum number of elements is 3 in the example of Fig. 21 and 2 in the example of Fig. 22.

[0116] Furthermore, for one or more first layers in which the number of elements of an output tensor is a first number among the multiple layers immediately before the concat after element reduction, the calculation unit 14 may suppress the execution of zero padding in each of one or more second layers associated with the one or more first layers. In the example of Fig. 22, the number of channels in each of Layer 1 and Layer 2 is both the maximum value (first number) "2", so the calculation unit 14 suppresses zero padding for each of Layer 1 and Layer 2.

[0117] (when matching indexes) As illustrated in Fig. 23, when [Ch2,2] of layer 2 is pruned, the calculation unit 14 may insert a zero matrix at the position of the pruned [Ch2,2], unlike the example of Fig. 21. In this case, the calculation unit 14 pads layer 1 with 0 zero matrices and pads layer 2 with one zero matrix. The concat unit 154 performs concat operations on [Ch1,1]+[Ch2,1], [Ch1,2]+[zero], and [Ch1,3]+[Ch2,3].

[0118] 24, when [Ch1,1] and [Ch1,3] of layer 1 and [Ch2,1] and [Ch2,2] of layer 2 are pruned, the calculation unit 14 may insert zero matrices into the pruned positions of [Ch1,3] of layer 1 and [Ch2,2] of layer 2. In this case, the calculation unit 14 may exclude from the zero padding the elements whose index (1) common to all layers of layer 1 and layer 2 is to be deleted ([Ch1,1] of layer 1 and [Ch2,1] of layer 2).

[0119] 24, it can be said that the calculation unit 14 performs padding with one zero matrix in each of layer 1 and layer 2. The concat unit 154 performs concat operations of [Ch1,2]+[zero] and [zero]+[Ch2,3].

[0120] In this way, when matching the indexes, the calculation unit 14 uses, as the first number, a number obtained by subtracting the number of elements of the first index common to the layers immediately preceding the concat, among the elements to be deleted in the layers immediately preceding the concat, from the number of elements (initial value) when elements are not deleted in multiple layers immediately preceding the concat. For example, the number of elements (initial value) is a number common to the layers immediately preceding the concat, and is 3 in both the examples of FIG. 23 and FIG. 24. The number of elements of the first index common to the layers immediately preceding the concat, among the elements to be deleted in the layers immediately preceding the concat, is 0 in the example of FIG. 23 and 1 (index 1) in the example of FIG. 24. Therefore, the first number is 3 in the example of FIG. 23 and 2 in the example of FIG. 24.

[0121] Then, when the second index is not to be deleted in at least one third layer among the multiple layers immediately preceding the concat, the calculation unit 14 inserts a zero matrix into the second index of a fourth layer other than the third layer among the multiple layers immediately preceding the concat.

[0122] In the example of Figure 23, the combination of the third layer, the second index, and the fourth layer is Layer 1, Index (2), and Layer 2. In the example of Figure 24, the combination of the third layer, the second index, and the fourth layer is Layer 1, Index (2), and Layer 2, and Layer 2, Index (3), and Layer 1.

[0123] This allows the calculation unit 14 to input tensors with aligned indexes to the concat unit 154 while reducing the number of elements in the layer immediately before the concat to the extent possible.

[0124] As described above, the zero padding process can make the number of elements (size) of tensors input from multiple layers immediately before concat unit 154 the same for each concat unit 154. Therefore, it becomes possible to prune the layer immediately before concat using the provisionally calculated pruning rate candidate, and the compression rate of the data size of the machine learning model including concat unit 154 can be improved.

[0125] The processing described with reference to FIGS. 18 to 24 may be part of the processing (i) by the threshold calculation unit 14a, or may be executed by the threshold calculation unit 14a.

[0126] Furthermore, the processing of the calculation unit 14 after the execution of the processing described with reference to FIGS. 18 to 24 is similar to the processing of (ii) and (iii).

[0127] The above-described zero padding process is not limited to being performed when the elements are channels, but may also be performed when the elements are weights and / or when the elements are nodes.

[0128] Fig. 25 is a diagram showing an example of the accuracy before and after pruning of a NN and the compression rate of data size depending on whether or not zero padding processing is applied. In Fig. 25, the case where the input layer to the concat unit 154 is not pruned means the case where zero padding processing is not applied and the layer immediately before concat of the concat unit 154 is not subject to pruning.

[0129] 25, when zero padding processing is applied, the compression rate of the data size of the light-weight model 11e can be improved in any of the models Res32, Res56, and Res110, compared to when the input layer to the concat unit 154 is not pruned. Also, significant deterioration of accuracy can be suppressed.

[0130] Fig. 26 is a diagram showing an example of the number of output channels of each layer of a NN depending on whether or not zero padding is applied. Fig. 26 shows an example in which the model is cifar10 and resnet32. In Fig. 26, the layer surrounded by a dashed line (in bold) is the layer immediately before concat, which is input to the concat unit 154. "After pruning (not applied)" indicates the case in which the input layer to the concat unit 154 is not pruned, and "after pruning (applied)" indicates the case in which zero padding is applied.

[0131] 26, when the zero padding process is applied, the number of output channels from the layer immediately before the concat is reduced compared to when the input layer to the concat unit 154 is not pruned. That is, it can be seen that the layer immediately before the concat is appropriately pruned by applying the zero padding process.

[0132] [1-5] Example of operation Next, an example of the operation of the server 1 according to an embodiment will be described with reference to Fig. 27. Fig. 27 is a flowchart for describing an example of the operation of processing by the server 1 according to an embodiment.

[0133] As illustrated in FIG. 27, the machine learning unit 13 performs machine learning on the unlearned model 11a acquired by the acquisition unit 12 without pruning (step S1).

[0134] The calculation unit 14 calculates the inference accuracy (recognition rate) Acc when no pruning is performed. wo is calculated (step S2).

[0135] The threshold calculation unit 14a sets an initial value of the reliability radius (step S3).

[0136] The threshold calculation unit 14a calculates a threshold T for each layer and a pruning error for each layer for setting a pruning rate (step S4), and determines whether the L2 norm of the threshold T for all layers is greater than the confidence radius (step S5). If the L2 norm of the threshold T for all layers is equal to or less than the confidence radius (NO in step S5), the process proceeds to step S7.

[0137] If the L2 norm of the threshold T for all layers is greater than the confidence radius (YES in step S5), the threshold calculation unit 14a scales (updates) the threshold so that the L2 norm of the threshold T for all layers = the confidence radius (step S6), and the processing proceeds to step S7.

[0138] In step S7, the threshold calculation unit 14a provisionally calculates a pruning rate for each layer. For example, the threshold calculation unit 14a provisionally sets a pruning rate for each layer from the set pruning rate candidates.

[0139] The calculation unit 14 determines whether the layer immediately before the concat is included in the layer for which the pruning rate has been provisionally calculated (step S8). If the layer immediately before the concat is not included in the layer for which the pruning rate has been provisionally calculated (NO in step S8), the process proceeds to step S11.

[0140] If the layer for which the pruning rate is provisionally calculated includes the layer immediately before the concat (YES in step S8), the calculation unit 14 inserts a zero-padding layer into the output of the layer immediately before the concat (step S9), performs the processing of step S10, and the processing proceeds to step S11.

[0141] In step S10, the calculation unit 14 identifies, for each concat unit 154, multiple layers immediately before concat that input tensors to the same concat unit 154, based on configuration information, etc. Then, the calculation unit 14 performs zero padding on the zero-padding layer so that the number of elements (e.g., the number of channels) output from the layers immediately before concat are the same. Note that steps S4 to S10 are an example of the process (i) above.

[0142] The machine learning unit 13 prunes the machine-learned model 11c using the pruning rate provisionally calculated by the threshold calculation unit 14a, and performs re-machine learning on the model after pruning. The calculation unit 14 calculates the inference accuracy Acc of the model after re-machine learning. p is calculated (step S11).

[0143] The determination unit 14b determines the inference accuracy Acc p +Margin Acc m is the inference accuracy Acc wo It is determined whether or not it is equal to or greater than this (step S12). By evaluating the inference accuracy (recognition rate), it is possible to compensate for an error in the selection of the pruning rate due to an approximation error.

[0144] Inference accuracy Acc p +Margin Acc m is the inference accuracy Acc wo If the result is equal to or greater than this (YES in step S12), the decision unit 14b decides to prune the machine-learned model 11c at the provisionally calculated pruning rate (step S13), and stores the provisionally calculated pruning rate as the pruning rate 11d in the memory unit 11. Furthermore, the threshold calculation unit 14a increases the confidence radius by a constant factor (step S14), and the process proceeds to step S17.

[0145] On the other hand, inference accuracy Acc p +Margin Acc m is the inference accuracy Acc wo If it is less than (NO in step S12), the decision unit 14b discards the provisionally calculated pruning rate (step S15). The threshold calculation unit 14a reduces the trust radius by multiplying it by a constant (step S16), and the process proceeds to step S17. Note that steps S10 to S16 are an example of the process (ii) above.

[0146] In step S17, the determination unit 14b determines whether the search (the processes of steps S4 to S16) has been performed a predetermined number of times, in other words, whether the number of times the processes of threshold calculation, pruning rate candidate selection, and pruning rate determination have been performed satisfies a predetermined condition. If the search has not been performed a predetermined number of times (NO in step S17), the process proceeds to step S4.

[0147] If the search has been performed a predetermined number of times (YES in step S17), the output unit 15 outputs the determined pruning rate 11d (step S18), and the process ends. Note that step S17 is an example of the process (iii) above.

[0148] As described above, in the server 1 according to one embodiment, the threshold calculation unit 14a calculates the error resulting from pruning of the tensors used in the NN, and generates a threshold from the value of the loss function and the gradient obtained by backpropagation of the NN. The threshold calculation unit 14a also compares the calculated pruning error with a threshold to provisionally calculate a pruning rate. The determination unit 14b then compares the inference accuracy of the model after re-learning using the calculated pruning rate with the inference accuracy of the model without pruning, and determines a pruning rate for each layer. If the threshold calculation unit 14a determines that the inference accuracy with pruning is lower than the inference accuracy without pruning, it resets the upper limit of the threshold to a smaller value and searches for a new pruning rate.

[0149] As a result, the server 1 according to an embodiment can determine the pruning rate for each layer regardless of the type of layer. For example, the server 1 can determine the pruning rate for each layer to be applied to the machine-learned model 11c including a convolutional layer not connected to a BN layer, a fully connected layer, and the like.

[0150] Furthermore, according to the server 1, even when the NN includes the concat unit 154, the layer immediately before the concat can be appropriately pruned, and the compression rate of the data size of the light-weight model 11e can be improved.

[0151] [1-6] Variations Next, a modified example according to one embodiment will be described. In the following description, for simplicity, the margin of inference accuracy Acc m is "0", in other words, in the comparison of inference accuracy, the inference accuracy Acc p is the inference accuracy Acc wo In the following description, an example is given in which the NN does not include a concat portion, but the processing described with reference to Figures 16 to 26 can be similarly applied to both the first and second modified examples described below.

[0152] [1-6-1] First modified example In a method according to one embodiment, the number of searches for the pruning rate (the number of attempts at the process (iii) above) is a hyperparameter that is set manually by, for example, a designer. For this reason, for example, if the number of searches is set to a small number, the machine-learned model 11c may not be sufficiently lightweight, whereas if the number of searches is set to a large number, the machine-learned model 11c may be sufficiently lightweight, but the search time may be long.

[0153] FIG. 28 is a diagram illustrating an example of a comparison result of pruning errors according to an update of the confidence radius in the method according to an embodiment.

[0154] As illustrated in FIG. 28, assume that a pruning rate of "10%" is calculated (determined) in the error comparison results of the mth search (m is an integer equal to or greater than "1"). In this case, the confidence radius is updated so that it increases by a constant K times. However, if the updated confidence radius is less than the error of the pruning rate candidate that is one step larger than the pruning rate candidate determined in the mth search, a pruning rate of "10%" is again calculated in the error comparison results of the (m+1)th search.

[0155] In this way, when the trust radius is set to a constant K or multiplied by a constant k, the amount of threshold update is limited by the trust radius, so the same pruning rate candidate may be adopted in multiple searches. If the same pruning rate combination is searched multiple times, the number of pruning rate searches will increase without sufficient attempts to prune the model.

[0156] Therefore, in the first modification, attention is focused on updating the trust radius, and a method for shortening (reducing) the search time (number of searches) for an appropriate pruning rate for reducing the weight of the NN will be described.

[0157] Fig. 29 is a block diagram showing an example of a functional configuration of a server 1A according to a first modified example. As shown in Fig. 29, the server 1A may include a calculation unit 14A that is different from the server 1 in Fig. 4. The calculation unit 14A may include a threshold calculation unit 14a' and a determination unit 14b' that are different from the calculation unit 14 in Fig. 4.

[0158] The calculation unit 14A searches for a different combination of pruning rates for each search. Here, a state in which a combination of pruning rates of "0%" for all layers is selected is considered to be a state in which the calculation unit 14A has determined that no further search for pruning rates will be performed. Under this assumption, the calculation unit 14A (determination unit 14b') terminates the search when a combination of pruning rates of "0%" for all layers is selected.

[0159] The threshold calculation unit 14a' determines, for each layer i (i is an integer equal to or greater than 1), the error of the pruning rate that is one value greater than the searched pruning rate or the absolute value of the difference between the error of the searched pruning rate and the threshold, "E diff,i " is measured.

[0160] For example, the threshold calculation unit 14a' calculates the inference accuracy Acc p is the inference accuracy Acc wo If it is equal to or greater than the threshold, the absolute value of the difference between the error of the pruning rate that is one value greater than the searched pruning rate and the threshold, "E diff,i " is measured.

[0161] On the other hand, the threshold calculation unit 14a' calculates the inference accuracy Acc p is the inference accuracy Acc wo If it is less than 1, the absolute value of the difference between the error of the searched pruning rate and the threshold, "E diff,i " is measured.

[0162] The threshold calculation unit 14a' calculates the absolute value of the calculated differences for all layers, "E diff,i The smallest value (difference) of "E diff " to get E diff = min(E diff,1 , E diff,2 , ..., E diff,i ) (7)

[0163] The threshold calculation unit 14a' determines a constant multiple of the confidence radius and a difference "E" between the confidence radius and the confidence radius according to the comparison result of the inference accuracy by the determination unit 14b'. diff The confidence radius is updated by adopting the sum or difference between the two, whichever has the larger amount of fluctuation.

[0164] For example, the threshold calculation unit 14a' calculates the inference accuracy Acc p is the inference accuracy Acc wo If it is equal to or greater than the confidence radius, the constant K times the confidence radius and the difference "E diff" and the sum of " and ", whichever has the larger amount of variation is adopted and updated so that the trust radius increases.

[0165] On the other hand, the threshold calculation unit 14a' calculates the inference accuracy Acc p is the inference accuracy Acc wo If it is less than the confidence radius, the constant k times the confidence radius and the difference "E diff " and the difference with the larger amount of variation is adopted and updated so that the trust radius decreases.

[0166] In this way, the threshold calculation unit 14a' updates the confidence radius so that the combinations of pruning rate candidates for each of the multiple layers are different from one another each time the process of selecting pruning rate candidates (in other words, search) is performed.

[0167] FIG. 30 is a diagram illustrating an example of the reliability radius update process when increasing the reliability radius. As shown in FIG. 30, it is assumed that the pruning rate found the mth time is "(Layer 1, Layer 2)=(10%, 0%)". The threshold calculation unit 14a' calculates the absolute value "E diff,1 ” and the absolute value of the difference between the error and the trust radius for a pruning rate of 10% for Layer 2, “E diff,2 The threshold calculation unit 14a' calculates the smallest difference "E diff,2 " to "E diff " is obtained.

[0168] Then, the threshold calculation unit 14a' determines (updates) the (m+1)th (next) reliability radius according to the following formula (8). (m+1th trust radius) = max((m-th confidence radius · constant K), (m-th confidence radius + E diff )) (8)

[0169] As a result, the trust radius for the (m+1)th iteration is selected to be at least equal to the sum of the trust radius and the difference, and therefore a bit width different from that for the (m+1)th iteration is calculated as the pruning rate.

[0170] In the example of Fig. 30, the confidence radius (upper limit of the threshold) in the (m+1)th search matches the error of the pruning rate of "10%" for layer 2. Therefore, in the (m+1)th search, a pruning rate combination of "(layer 1, layer 2) = (10%, 10%)" is searched for, which is a different pruning rate combination from the previous search.

[0171] FIG. 31 is a diagram illustrating an example of the reliability radius update process when the reliability radius is decreased. As shown in FIG. 31, it is assumed that the pruning rate found the mth time is "(Layer 1, Layer 2) = (10%, 0%)". The threshold calculation unit 14a' calculates the absolute value "E diff,1 ” and the absolute value of the difference between the error and the trust radius for a pruning rate of 0% for layer 2, “E diff,2 The threshold calculation unit 14a' calculates the smallest difference "E diff,1 " to "E diff " is obtained.

[0172] Then, the threshold calculation unit 14a' determines (updates) the (m+1)th (next) reliability radius according to the following formula (9). (m+1th trust radius) = max((m-th confidence radius · constant), (m-th confidence radius - E diff )) (9)

[0173] As a result, the trust radius for the (m+1)th time is selected to be at least a value greater than the difference between the trust radius and the difference, and therefore a bit width different from that for the (m+1)th time is calculated as the pruning rate.

[0174] 31, the confidence radius (upper limit of the threshold) in the (m+1)th search matches the error of the pruning rate of "0%" for layer 1. Therefore, in the (m+1)th search, a pruning rate combination of "(layer 1, layer 2)=(0%, 0%)" that is different from the previous search is searched for.

[0175] By generalizing the above equations (8) and (9), the next trust radius can be expressed by the following equation (10). Next confidence radius = Current confidence radius * max(Constant, Qscale_min) (10)

[0176] Here, in the above equation (10), the constant is K or k, "Qscale_min" is "Qscale" expressed by the following equation (11), and "Qscale" is expressed by the following equation (12). Qscale_min = min(Qscale calculated for all target vectors) (11) Qscale = 1 + Qdiff / Qth (12)

[0177] In the above equation (12), "Qdiff" is "the difference between the quantization error of the bit width that is one bit narrower than the provisionally calculated bit width (pruning rate) and the threshold value," and "Qth" is the threshold value.

[0178] Next, an example of operation of the server 1A according to the first modified example will be described with reference to Fig. 32. Fig. 32 is a flowchart for describing an example of operation of processing by the server 1A according to the first modified example. Fig. 32 is obtained by replacing steps S14, S16, and S17 in the flowchart according to the server 1 shown in Fig. 27 with steps S21, S22, and S23, respectively. Note that also in the first modified example, the threshold calculation unit 14a' sets an initial value of the trust radius in step S3.

[0179] In step S21, the threshold calculation unit 14a' increases the confidence radius by a constant K or the "sum of differences," whichever is larger, and the process proceeds to step S23.

[0180] In step S22, the threshold calculation unit 14a' reduces the confidence radius by a constant k times or by the "difference of differences," whichever is larger, and the process proceeds to step S23.

[0181] In step S23, the determination unit 14b' determines whether the pruning rate 11d of all layers is "0%", in other words, whether the pruning rate satisfies a predetermined condition. If the pruning rate 11d of at least one layer is not "0%" (NO in step S23), the process proceeds to step S4.

[0182] If the pruning rate 11d of all layers is "0%" (YES in step S23), the output unit 15 outputs the determined pruning rate 11d (step S18), and the process ends.

[0183] As described above, in the first modified example, the method of updating the trust radius by the threshold calculation unit 14a' and the termination condition for determining the termination of the search by the determination unit 14b' are different from those in the first embodiment. This allows the server 1A to search for an appropriate pruning rate for sufficiently reducing the weight of the NN in the shortest time (the shortest number of times). In addition, it is possible to omit the setting (specification) of the number of searches by a designer or the like.

[0184] [1-6-2] Second Modified Example In the method according to the embodiment and the first modification, the initial value of the trust radius is a hyperparameter set by a designer or the like.

[0185] The model size may differ even if the number of searches is the same when the initial value of the trust radius is set large or small. Also, when the initial value of the trust radius is set large, the number of searches required to sufficiently reduce the model size may be greater than when the initial value of the trust radius is set small.

[0186] In this way, the final model size and the number of searches for the pruning rate may vary depending on the initial value of the trust radius; in other words, the performance of the servers 1 and 1A may vary.

[0187] Therefore, in the second modification, a method for suppressing the variation in performance between the servers 1 and 1A will be described.

[0188] FIG. 33 is a block diagram showing an example of a functional configuration of a server 1B according to a second modified example. As illustrated in FIG. 33, the server 1B may include a calculation unit 14B that is different from the server 1 in FIG. 4. The calculation unit 14B may include a threshold calculation unit 14a" and a determination unit 14b" that are different from the calculation unit 14 in FIG. 4.

[0189] In model pruning, it is known that pruning a model gradually using a small pruning rate can maintain accuracy and compress the model at a higher compression rate than pruning a model all at once using a large pruning rate.

[0190] Furthermore, as shown in the above formula (5), the threshold T is set according to the inverse of the gradient, so a layer with a large threshold T means that the layer has a small gradient. A layer with a small gradient means that pruning has little effect on accuracy.

[0191] Therefore, the server 1B (threshold calculation unit 14a"), for example, sets the initial value of the trust radius to a value that minimizes the pruning rate in the first search. For this purpose, the threshold calculation unit 14a" may set the initial value of the trust radius to a value that prunes the layer with the largest threshold T among all layers and does not prune the remaining layers (the pruning rate is "0%").

[0192] By setting the initial value of the trust radius as described above, the server 1B can compress the model size more or maintain accuracy more than if the initial value of the trust radius were manually set to a larger value, for example.

[0193] Fig. 34 is a diagram illustrating an example of setting the initial value of the trust radius. As shown in the upper part of Fig. 34, when the initial value of the trust radius is not set, the combination of pruning rates searched is "(Layer 1, Layer 2) = (10%, 20%)".

[0194] As illustrated in FIG. 34, in the initial search for the pruning rate, the threshold calculation unit 14a″ measures the threshold (max(Th)) of the layer with the largest threshold among all layers and the error (Error) due to the smallest pruning rate (excluding “0%”) of that layer.

[0195] Th denotes a vector of thresholds T1, T2, ... for each layer, and in the example of Figure 34, Th = [T1, T2]. The threshold (max(Th)) is the threshold for the layer with the largest threshold, and in the example of Figure 34, this is T2. The error (Error) is the error of the minimum pruning rate for the layer with the largest threshold, and in the example of Figure 34, the error for a pruning rate of "10%" for layer 2 is measured.

[0196] Next, the threshold calculation unit 14a" sets the initial value of the confidence radius using the measured threshold and error according to the following equation (13). In the following equation (13), "||Th||2" is the L2 norm of the thresholds of all layers.

number

[0197] The threshold calculation unit 14a'' sets the thresholds T1 and T2 based on the calculated initial value of the trust radius so that the minimum pruning rate of "10%" is selected as the pruning rate for the layer with the largest threshold (layer 2), and a pruning rate of "0%" is selected for the remaining layers (layer 1).

[0198] As a result, when the initial value of the trust radius is set and the thresholds T1 and T2 are set, the combination of pruning rates to be searched for is "(Layer 1, Layer 2) = (0%, 10%)", as shown in the lower part of Fig. 34. The layer to be pruned (Layer 2) has the largest threshold, in other words, the smallest gradient, so the impact of pruning on accuracy can be kept small.

[0199] Note that the functions of the threshold calculation unit 14a" other than the process of setting the initial value of the reliability radius may be similar to either or both of the threshold calculation unit 14a according to the embodiment and the threshold calculation unit 14a' according to the first modified example. Furthermore, the determination unit 14b" may be similar to either or both of the determination unit 14b according to the embodiment and the determination unit 14b' according to the first modified example.

[0200] That is, the technique according to the second modification may be realized by combining with one or both of the embodiment and the first modification.

[0201] Next, an example of operation of the server 1B according to the second modified example will be described with reference to Fig. 35. Fig. 35 is a flowchart for describing an example of operation of processing by the server 1B according to the second modified example. Fig. 35 is obtained by deleting step S3 from the flowchart for the server 1 shown in Fig. 27, adding steps S31 and S32 between steps S4 and S5, and replacing steps S14, S16, and S17 with steps S33, S34, and S35, respectively.

[0202] In step S31, the threshold calculation unit 14a'' determines whether or not this is the first search after calculating the threshold for each layer in step S4. If this is not the first search (NO in step S31), the process proceeds to step S5.

[0203] If it is the first search (YES in step S31), the threshold calculation unit 14a'' sets an initial value of the confidence radius based on the threshold of the layer with the largest threshold and the minimum pruning rate error (step S32), and the process proceeds to step S5.

[0204] Steps S33, S34, and S35 may be any of steps S14, S16, and S17 shown in FIG. 27 and steps S21, S22, and S23 shown in FIG. 32, respectively.

[0205] As described above, in the second modified example, the method of setting the initial value of the trust radius by the threshold calculation unit 14a" is different from that of the embodiment and the first modified example. This enables the server 1B to suppress fluctuations in the number of searches for the final model size and the pruning rate, and suppresses variations in the performance of the servers 1 and 1A.

[0206] Furthermore, the server 1B can prevent a designer or the like from manually setting the initial value (hyperparameter) of the trust radius, and can dynamically set the initial value of the trust radius according to the layer of the machine-learned model 11c. Therefore, an appropriate pruning rate can be set for each model, and fluctuations in the number of searches for the final model size and pruning rate can be suppressed regardless of the model, thereby suppressing variations in the performance of the servers 1 and 1A.

[0207] [1-7] Hardware configuration example The servers 1, 1A, and 1B according to the embodiment and the first and second modifications may each be a virtual machine (VM) or a physical machine. Furthermore, the functions of the servers 1, 1A, and 1B may be implemented by one computer or two or more computers. Furthermore, at least some of the functions of the servers 1, 1A, and 1B may be implemented using hardware (HW) resources and network (NW) resources provided by a cloud environment.

[0208] 36 is a block diagram showing an example of the hardware (HW) configuration of the computer 10. The following description will be given taking the computer 10 as an example of the hardware (HW) that realizes the functions of the servers 1, 1A, and 1B. Note that when multiple computers are used as HW resources that realize the functions of the servers 1, 1A, and 1B, each computer may have the HW configuration shown in FIG.

[0209] As shown in FIG. 36, the computer 10 may include, as a HW configuration, a processor 10a, a memory 10b, a storage unit 10c, an IF (Interface) unit 10d, an IO (Input / Output) unit 10e, and a reading unit 10f, for example.

[0210] The processor 10a is an example of a processing unit that performs various controls and calculations. The processor 10a may be connected to each block in the computer 10 via a bus 10i so that they can communicate with each other. The processor 10a may be a multiprocessor including multiple processors, a multi-core processor having multiple processor cores, or a configuration having multiple multi-core processors.

[0211] Examples of the processor 10a include integrated circuits (ICs) such as a CPU, MPU, GPU, APU, DSP, ASIC, and FPGA. CPU is an abbreviation for Central Processing Unit, MPU is an abbreviation for Micro Processing Unit, GPU is an abbreviation for Graphics Processing Unit, and APU is an abbreviation for Accelerated Processing Unit. DSP is an abbreviation for Digital Signal Processor, ASIC is an abbreviation for Application Specific IC, and FPGA is an abbreviation for Field-Programmable Gate Array.

[0212] Note that a combination of two or more of the above-described integrated circuits may be used as the processor 10a. As an example, the computer 10 may include a first and a second processor 10a. The first processor 10a is an example of a CPU that executes a program 10g (machine learning program) that implements all or part of the various functions of the computer 10. For example, the first processor 10a may implement the functions of the acquisition unit 12, the calculation unit 14, 14A, or 14B, and the output unit 15 of the server 1, 1A, or 1B (see FIG. 4, FIG. 29, or FIG. 33) based on the program 10g. The second processor 10a is an example of an accelerator that executes arithmetic processing used in NN calculations such as matrix operations, and may implement the functions of the machine learning unit 13 of the server 1, 1A, or 1B (see FIG. 4, FIG. 29, or FIG. 33).

[0213] The memory 10b is an example of HW that stores various types of data, programs, and other information. Examples of the memory 10b include one or both of a volatile memory such as a dynamic random access memory (DRAM) and a non-volatile memory such as a persistent memory (PM).

[0214] The storage unit 10c is an example of HW that stores various types of data, programs, and other information. Examples of the storage unit 10c include various storage devices such as a magnetic disk device such as a hard disk drive (HDD), a semiconductor drive device such as a solid state drive (SSD), and a nonvolatile memory. Examples of nonvolatile memory include a flash memory, a storage class memory (SCM), and a read-only memory (ROM).

[0215] The storage unit 10c may store a program 10g. For example, the processor 10a of each of the servers 1, 1A, and 1B can implement the function of the control unit 16 (see FIG. 4, FIG. 29, or FIG. 33) of each of the servers 1, 1A, and 1B by loading the program 10g stored in the storage unit 10c into the memory 10b and executing it.

[0216] Furthermore, the memory unit 11 illustrated in FIG. 4, FIG. 29, or FIG. 33 may be realized by a storage area included in at least one of the memory 10b and the storage unit 10c.

[0217] The IF unit 10d is an example of a communication IF that controls connection and communication with a network. For example, the IF unit 10d may include an adapter that complies with a LAN (Local Area Network) such as Ethernet (registered trademark) or optical communication such as FC (Fibre Channel). The adapter may support one or both of wireless and wired communication methods. For example, the servers 1, 1A, and 1B may be connected to a computer (not shown) via the IF unit 10d so that they can communicate with each other. One or both of the functions of the acquisition unit 12 and the output unit 15 illustrated in FIG. 4, FIG. 29, or FIG. 33 may be realized by the IF unit 10d. Furthermore, for example, the program 10g may be downloaded from the network to the computer 10 via the communication IF and stored in the storage unit 10c.

[0218] The IO unit 10e may include one or both of an input device and an output device. Examples of the input device include a keyboard, a mouse, and a touch panel. Examples of the output device include a monitor, a projector, and a printer. For example, the output unit 15 illustrated in FIG. 4, 29, or 33 may output the pruning rate 11d to the output device of the IO unit 10e and display it.

[0219] The reading unit 10f is an example of a reader that reads data and program information recorded on the recording medium 10h. The reading unit 10f may include a connection terminal or device to which the recording medium 10h can be connected or inserted. Examples of the reading unit 10f include an adapter that complies with USB (Universal Serial Bus) or the like, a drive device that accesses a recording disk, and a card reader that accesses a flash memory such as an SD card. Note that the recording medium 10h may store the program 10g, and the reading unit 10f may read the program 10g from the recording medium 10h and store it in the memory unit 10c.

[0220] Examples of the recording medium 10h include non-transitory computer-readable recording media such as magnetic / optical disks and flash memories. Examples of magnetic / optical disks include flexible disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray Discs, and HVDs (Holographic Versatile Discs). Examples of flash memories include semiconductor memories such as USB memories and SD cards.

[0221] The above-described hardware configuration of the computer 10 is an example. Therefore, the hardware in the computer 10 may be increased or decreased (for example, adding or deleting any block), divided, integrated in any combination, or buses may be added or deleted as appropriate. For example, in the servers 1, 1A, and 1B, at least one of the IO unit 10e and the reading unit 10f may be omitted.

[0222] [2] Other The techniques according to the above-described embodiment and the first and second modifications can be implemented with the following modifications and variations.

[0223] For example, the acquisition unit 12, machine learning unit 13, calculation unit 14, 14A or 14B, and output unit 15 provided in the server 1, 1A or 1B shown in Figure 4, Figure 29 or Figure 33 may be merged or each may be divided.

[0224] 4, 29, or 33 may be configured such that each processing function is realized by a plurality of devices cooperating with each other via a network. As an example, in server 1, 1A, or 1B, acquisition unit 12 and output unit 15 may be a web server and an application server, machine learning unit 13 and calculation unit 14, 14A, or 14B may be an application server, and memory unit 11 may be a DB server, etc. In this case, the web server, application server, and DB server may be cooperating with each other via a network to realize the processing function of server 1, 1A, or 1B.

[0225] Furthermore, for example, the technique of applying zero padding processing to a NN including the concat unit 154 described with reference to Figures 16 to 26 is not limited to application to pruning processing by the server 1, 1A, or 1B shown in Figure 4, Figure 29, or Figure 33. For example, the technique of applying zero padding processing may be applied to various techniques for determining a pruning rate for each layer of a NN.

[0226] [3] Supplementary Note The following additional notes are further disclosed regarding the above embodiment and the first and second modifications.

[0227] (Appendix 1) Obtaining a reduction rate of elements in each of a plurality of layers in a machine-learned model of a neural network including the plurality of layers; When the neural network includes an arithmetic process that outputs a tensor that is a result of a predetermined operation on a plurality of input tensors, and when a tensor from a layer preceding the arithmetic process and one or more tensors from one or more layers preceding the preceding layer that are shortcut-connected to the arithmetic process are input to the arithmetic process, a second layer that performs zero padding processing is inserted between the arithmetic process and each of the plurality of first layers that are the preceding layer and the one or more layers, padding each of the second layers associated with the first layers with one or more zero matrices so that the number of elements of each of the tensors input to the arithmetic processing from the first layers after element reduction according to a reduction ratio for each of the first layers becomes a first number; A machine learning program that lets a computer perform processing.

[0228] (Appendix 2) The first number is the maximum number of elements among the plurality of tensors output from the plurality of first layers after the element reduction. The machine learning program described in Appendix 1.

[0229] (Appendix 3) for one or more first layers among the plurality of first layers after the element reduction, in which the number of elements of an output tensor is the first number, suppressing execution of the padding process in each of one or more second layers associated with the one or more first layers; 3. The machine learning program of claim 2, which causes the computer to execute a process.

[0230] (Appendix 4) the first number is a number obtained by subtracting the number of elements of a first index that are common among the plurality of first layers among the elements to be deleted in the plurality of first layers from the number of elements in the plurality of first layers when no elements are deleted, The padding process includes inserting a zero matrix into the second index of a fourth layer other than the third layer among the plurality of first layers when an element of the second index is not to be deleted in at least one third layer among the plurality of first layers. The machine learning program described in Appendix 1.

[0231] (Appendix 5) the calculation process is a concatenate calculation, the first layers are layers immediately preceding the concatenate operation, the plurality of second layers are zero-padding layers; The machine learning program according to any one of Supplementary Note 1 to Supplementary Note 4.

[0232] (Appendix 6) The element is a channel, a weight, or a node. 10. The machine learning program according to claim 1, wherein the machine learning program is a program for generating a machine learning program.

[0233] (Appendix 7) Obtaining a reduction rate of elements in each of a plurality of layers in a machine-learned model of a neural network including the plurality of layers; When the neural network includes an arithmetic process that outputs a tensor that is a result of a predetermined operation on a plurality of input tensors, and when a tensor from a layer preceding the arithmetic process and one or more tensors from one or more layers preceding the preceding layer that are shortcut-connected to the arithmetic process are input to the arithmetic process, a second layer that performs zero padding processing is inserted between the arithmetic process and each of the plurality of first layers that are the preceding layer and the one or more layers, padding each of the second layers associated with the first layers with one or more zero matrices so that the number of elements of each of the tensors input to the arithmetic processing from the first layers after element reduction according to a reduction ratio for each of the first layers becomes a first number; A machine learning method in which processing is performed by a computer.

[0234] (Appendix 8) The first number is the maximum number of elements among the plurality of tensors output from the plurality of first layers after the element reduction. 7. The machine learning method described in Appendix 7.

[0235] (Appendix 9) for one or more first layers among the plurality of first layers after the element reduction, in which the number of elements of an output tensor is the first number, suppressing execution of the padding process in each of one or more second layers associated with the one or more first layers; 9. The machine learning method of claim 8, wherein the processing is performed by the computer.

[0236] (Appendix 10) the first number is a number obtained by subtracting the number of elements of a first index that are common among the plurality of first layers among the elements to be deleted in the plurality of first layers from the number of elements in the plurality of first layers when no elements are deleted, The padding process includes inserting a zero matrix into the second index of a fourth layer other than the third layer among the plurality of first layers when an element of the second index is not to be deleted in at least one third layer among the plurality of first layers. 7. The machine learning method described in Appendix 7.

[0237] (Appendix 11) the calculation process is a concatenate calculation, the first layers are layers immediately preceding the concatenate operation, the plurality of second layers are zero-padding layers; The machine learning method according to any one of Supplementary Notes 7 to 10.

[0238] (Appendix 12) The element is a channel, a weight, or a node. The machine learning method according to any one of Supplementary Note 7 to Supplementary Note 11.

[0239] (Appendix 13) Obtaining a reduction rate of elements in each of a plurality of layers in a machine-learned model of a neural network including the plurality of layers; When the neural network includes an arithmetic process that outputs a tensor that is a result of a predetermined operation on a plurality of input tensors, and when a tensor from a layer preceding the arithmetic process and one or more tensors from one or more layers preceding the preceding layer that are shortcut-connected to the arithmetic process are input to the arithmetic process, a second layer that performs zero padding processing is inserted between the arithmetic process and each of the plurality of first layers that are the preceding layer and the one or more layers, padding each of the second layers associated with the first layers with one or more zero matrices so that the number of elements of each of the tensors input to the arithmetic processing from the first layers after element reduction according to a reduction ratio for each of the first layers becomes a first number; An information processing device comprising a control unit.

[0240] (Appendix 14) The first number is the maximum number of elements among the plurality of tensors output from the plurality of first layers after the element reduction. 14. The information processing device according to claim 13.

[0241] (Appendix 15) the control unit, for one or more first layers in which the number of elements of an output tensor is the first number among the plurality of first layers after the element reduction, suppresses execution of the padding process in each of one or more second layers associated with the one or more first layers; 15. The information processing device according to claim 14.

[0242] (Appendix 16) the first number is a number obtained by subtracting the number of elements of a first index that are common among the plurality of first layers among the elements to be deleted in the plurality of first layers from the number of elements in the plurality of first layers when no elements are deleted, In the padding process, when an element of a second index in at least one third layer among the plurality of first layers is not to be deleted, the control unit inserts a zero matrix into the second index of a fourth layer other than the third layer among the plurality of first layers. 14. The information processing device according to claim 13.

[0243] (Appendix 17) the calculation process is a concatenate calculation, the first layers are layers immediately preceding the concatenate operation, the plurality of second layers are zero-padding layers; The information processing device according to any one of Supplementary notes 13 to 16.

[0244] (Appendix 18) The element is a channel, a weight, or a node. An information processing device according to any one of Supplementary notes 13 to 17. [Explanation of symbols]

[0245] 1, 1A, 1B Server 10. Computers 11 Memory section 11a Untrained model 11b Data for machine learning 11c Machine Learning Model 11d pruning rate 11e lightweight model 12 Acquisition Department 13 Machine Learning Department 14, 14A, 14B Pruning rate calculation section (calculation section) 14a, 14a', 14a" Threshold calculation unit 14b, 14b', 14b” decision section 15 Output section 16 Control Unit

Claims

1. a neural network including a plurality of layers, the neural network including an arithmetic process for outputting a tensor resulting from a predetermined arithmetic operation on a plurality of input tensors, the tensors output from a plurality of first layers preceding the arithmetic process being input to the arithmetic process, the reduction rate of elements of each of the plurality of layers in a machine-learned model of the neural network being acquired, the plurality of first layers including a layer preceding the arithmetic process and one or more layers preceding the layer preceding the layer preceding the arithmetic process that are shortcut-coupled to the arithmetic process; inserting a plurality of second layers that perform zero padding processing between each of the plurality of first layers and the arithmetic processing, each of the plurality of second layers corresponding to a respective one of the plurality of first layers; padding each of the second layers with one or more zero matrices so that the number of elements of each of the tensors input to the arithmetic processing from the first layers after element reduction according to the reduction rate of each of the first layers becomes a first number; A machine learning program that lets a computer perform processing.

2. the first number is the maximum number of elements among the plurality of tensors output from the plurality of first layers after the element reduction; The machine learning program according to claim 1 .

3. for one or more first layers among the plurality of first layers after the element reduction, in which the number of elements of an output tensor is the first number, execution of the padding process in each of one or more second layers associated with the one or more first layers is suppressed; The machine learning program according to claim 2 , which causes the computer to execute processing.

4. the first number is a number obtained by subtracting the number of elements of a first index that are common among the plurality of first layers among the elements to be deleted in the plurality of first layers from the number of elements in the plurality of first layers when no elements are deleted, the padding process includes inserting a zero matrix into the second index of a fourth layer other than the third layer among the plurality of first layers when an element of a second index in the third layer of at least one of the plurality of first layers is not to be deleted; The machine learning program according to claim 1 .

5. the calculation process is a concatenate calculation, the first layers are layers immediately preceding the concatenate operation, the plurality of second layers are zero-padding layers; The machine learning program according to any one of claims 1 to 4.

6. a neural network including a plurality of layers, the neural network including an arithmetic process for outputting a tensor resulting from a predetermined arithmetic operation on a plurality of input tensors, the tensors output from a plurality of first layers preceding the arithmetic process being input to the arithmetic process, the reduction rate of elements of each of the plurality of layers in a machine-learned model of the neural network being acquired, the plurality of first layers including a layer preceding the arithmetic process and one or more layers preceding the layer preceding the layer preceding the arithmetic process that are shortcut-coupled to the arithmetic process; inserting a plurality of second layers that perform zero padding processing between each of the plurality of first layers and the arithmetic processing, each of the plurality of second layers corresponding to a respective one of the plurality of first layers; padding each of the second layers with one or more zero matrices so that the number of elements of each of the tensors input to the arithmetic processing from the first layers after element reduction according to the reduction rate of each of the first layers becomes a first number; A machine learning method in which processing is performed by a computer.

7. a neural network including a plurality of layers, the neural network including an arithmetic process for outputting a tensor resulting from a predetermined arithmetic operation on a plurality of input tensors, the tensors output from a plurality of first layers preceding the arithmetic process being input to the arithmetic process, the reduction rate of elements of each of the plurality of layers in a machine-learned model of the neural network being acquired, the plurality of first layers including a layer preceding the arithmetic process and one or more layers preceding the layer preceding the layer preceding the arithmetic process that are shortcut-coupled to the arithmetic process; inserting a plurality of second layers that perform zero padding processing between each of the plurality of first layers and the arithmetic processing, each of the plurality of second layers corresponding to a respective one of the plurality of first layers; padding each of the second layers with one or more zero matrices so that the number of elements of each of the tensors input to the arithmetic processing from the first layers after element reduction according to the reduction rate of each of the first layers becomes a first number; An information processing device comprising a control unit.

Images