Machine learning program, machine learning method, and information processing device

Abstract

To realize weight saving of a neural network including an attention structure.SOLUTION: A machine learning program causes a computer to execute processing of: inserting padding layers 181, 182 which perform padding of one or more elements of a tensor, to a subsequent stage of each of a Q layer 161 which outputs a Query as an arithmetic processing result with respect to an input tensor of an attention structure, in a trained machine learning model of a neural network 180 comprising the attention structure 160, and a K layer 162 which outputs a Key; and performing the padding based on the padding layer associated with each of the Q layer after reduction and the K layer after reduction, so that the number of the elements of each of a tensor QT from the Q layer after reduction of the elements based on a first reduction ratio, and a tensor KT from the K layer after reduction of the elements based on a second reduction ratio, may be equal to one another.SELECTED DRAWING: Figure 19

Description

[Technical Field]

[0001] This invention relates to a machine learning program, a machine learning method, and an information processing device. [Background technology]

[0002] Neural networks (NNs) used in AI (Artificial Intelligence) tasks such as image processing tend to achieve high performance (e.g., high inference accuracy) by increasing their complexity. On the other hand, increasing the complexity of the NN's configuration can increase the number of calculations performed by the computer during NN execution, as well as the amount of memory the computer uses to execute the NN.

[0003] Pruning is a known technique for reducing the number of operations, in other words, shortening (speeding up) the computation time, and for reducing the memory size, in other words, for making the machine learning model of a neural network lighter.

[0004] Pruning is a technique that reduces the data size of a machine learning model and decreases computation time and communication time by reducing (trimming) at least one type of element in a neural network: edges (weights), nodes, and channels.

[0005] Excessive pruning can degrade the inference accuracy of neural networks (NNs). Therefore, it is important to prune NNs while maintaining inference accuracy or keeping the decrease in inference accuracy within a predetermined level.

[0006] For example, in pruning, there is a known method for selecting layers that do not significantly affect the inference accuracy of neural networks. This method determines which convolutional layer channels to prune based on the parameters used in the batch normalization (BN) layer following the convolutional layer.

[0007] Furthermore, neural networks (NNs) equipped with attention structures such as the Multi-Head Attention (MHA) structure are known. The attention structure includes three fully connected layers in its input section. The three fully connected layers are layers that output tensors Q (Query), K (Key), and V (Value), respectively. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] U.S. Patent Application Publication No. 2022 / 0036194 [Overview of the Initiative] [Problems that the invention aims to solve]

[0009] The method of selecting layers that do not significantly affect the inference accuracy of neural networks is applicable to convolutional layers connected to BN layers, but its application to other layers, such as convolutional layers without BN layers and fully connected layers, is not envisioned.

[0010] For example, consider a case where a method for selecting layers that do not significantly affect the inference accuracy of a neural network (NN) is applied to multiple layers as described above, and the NN includes an attention structure. In this case, when pruning is performed using this method, none of the three fully connected layers in the input section of the attention structure are pruned, and the overall pruning rate of the machine learning model decreases, thus reducing the data size compression (lightening) effect of pruning on the machine learning model.

[0011] In one aspect, the present invention aims to achieve lightweight neural networks that incorporate attention structures. [Means for solving the problem]

[0012] In one aspect, a machine learning program may cause a computer to perform the following operations: The operations may include inserting a padding layer that pads one or more elements of a tensor after each of the Q layer and K layer, which output a Query and a Key respectively, in a trained machine learning model of a neural network having an attention structure, as the result of an operation on the input tensor of the attention structure. The operations may also include performing padding by the padding layer associated with the reduced Q layer and the reduced K layer respectively, so that the number of elements in the tensor QT from the Q layer after element reduction based on a first reduction ratio and the tensor KT from the K layer after element reduction based on a second reduction ratio are the same. [Effects of the Invention]

[0013] In one respect, the present invention can achieve lightweight neural networks that incorporate attention structures. [Brief explanation of the drawing]

[0014] [Figure 1] This diagram illustrates an example of the process for determining the channels of a convolutional layer to be pruned. [Figure 2] This figure shows an example of L1 regularization learning. [Figure 3] This figure shows an example of whether the methods in Figures 1 and 2 can be applied to the layers of a neural network. [Figure 4] This block diagram shows an example of the functional configuration of a server according to one embodiment. [Figure 5] This figure shows an example of calculating a pruning rate that can guarantee accuracy. [Figure 6] This figure shows an example of calculating the accuracy of a model before and after pruning. [Figure 7] This figure shows an example of exploring pruning rates. [Figure 8] This diagram illustrates one example of a threshold derivation method. [Figure 9] This figure shows an example of a threshold upper limit and a threshold value. [Figure 10] This diagram illustrates an example of a method for determining which channels to prune. [Figure 11] This diagram illustrates an example of pruning error calculation. [Figure 12] This diagram illustrates one example of a method for determining which nodes to prune. [Figure 13] This diagram illustrates an example of pruning error calculation. [Figure 14] This diagram illustrates an example of a method for determining pruning weights. [Figure 15] This diagram illustrates an example of pruning error calculation. [Figure 16] This figure shows an example of a neural network (NN) that incorporates an attention structure. [Figure 17] This figure shows an example of an attention structure. [Figure 18] This figure shows a detailed example of an attention structure. [Figure 19] This figure illustrates an example of inserting a zero-padding layer into a model. [Figure 20] This diagram illustrates an example of zero-padding in a model. [Figure 21] This figure shows an example of the accuracy of a neural network before and after pruning, as well as the data size compression ratio, depending on whether zero-padding is applied. [Figure 22] This is a flowchart illustrating an example of the operation of a server according to one embodiment. [Figure 23] This figure shows an example of the pruning error comparison results corresponding to the update of the confidence radius in a method according to one embodiment. [Figure 24] This is a block diagram showing an example of the server's functional configuration related to the first modified example. [Figure 25] This diagram illustrates an example of the confidence radius update process when increasing the confidence radius. [Figure 26] This diagram illustrates an example of a confidence radius update process when the confidence radius is reduced. [Figure 27]This is a flowchart illustrating an example of server operation related to the first modified example. [Figure 28] This is a block diagram showing an example of the server's functional configuration related to the second modification. [Figure 29] This diagram illustrates an example of setting the initial value of the confidence radius. [Figure 30] This is a flowchart illustrating an example of server operation related to the second modified example. [Figure 31] This is a block diagram showing examples of computer hardware (HW) configurations. [Modes for carrying out the invention]

[0015] Embodiments of the present invention will now be described with reference to the drawings. However, the embodiments described below are merely illustrative and are not intended to exclude various modifications or applications of techniques not explicitly stated below. For example, these embodiments can be implemented with various modifications without departing from their spirit. In the drawings used in the following description, parts denoted by the same reference numerals represent the same or similar parts unless otherwise specified.

[0016] [1] One Embodiment Figure 1 illustrates an example of the process for determining the channels of the convolutional layer to be pruned, and Figure 2 illustrates an example of L1 regularization learning. In Figure 1, a method is described in which the computer determines the channels of the convolutional layer to be pruned using the scaling coefficient γ used in the BN layer 100 following the convolutional layer, as a method for selecting layers that do not significantly affect the inference accuracy of the neural network. Note that the graphs shown for channels 111 to 113 in Figure 1 represent the distribution of the output tensor.

[0017] As shown in Figure 1, the computer performs a normalization process 101 on each of the multiple channels 111 (#1 to #n; n is an integer greater than or equal to 2) that are input from the convolutional layer to the BN layer 100. For example, in the normalization process 101, the computer calculates the mean μ and variance σ for each channel 111 according to the following equation (1).2 By calculating this, we obtain multiple channels 112 (#1~#n) representing a normalized distribution with mean "0" and variance "1". In equation (1) below, z in and z mid These represent channels 111 and 112, respectively, and μ B and σ B 2 These represent the mean and variance of the current mini-batch B, respectively.

number

[0018] Furthermore, the computer performs scaling 102 on multiple channels 112 (#1 to #n). For example, in scaling 102, the computer multiplies each of the multiple channels 112 by a scaling coefficient γ according to equation (2) below, and adds a bias β to the multiplication result to output multiple channels 113 (#1 to #n) that represent the distribution scaled by parameters γ and β. In equation (2) below, z out This indicates channel 113. Note that parameters γ and β may be optimized using machine learning.

number

[0019] Here, when γ is small, the output of channel 113 (channel #n in the example in Figure 1), which is the result of scaling 102, is almost eliminated. This means that even if the channel in question is removed by pruning, it will not have a significant impact on the inference accuracy of the neural network. Therefore, the computer determines which channels should be pruned on a channel-by-channel basis by searching for small γ values ​​(e.g., "0").

[0020] For example, a computer searches for a small (or increasingly small) γ by applying L1 regularization learning to γ. L1 regularization learning is a machine learning technique known for making the parameters being learned "sparse" by adding an L1 regularization term to the loss function that the neural network calculates in its output.

[0021] As illustrated in Figure 2, the computer obtains a vector 123 that has undergone L1 regularization by performing L1 regularization learning on a certain vector 121 using the loss function 122. The loss function 122 may be a function L obtained by adding the original loss function (first term), such as cross-entropy, and the L1 regularization term (second term), which uses the L1 norm (Σg(γ)=Σ|γ|), as shown in equation (3) below.

number

[0022] Through L1 regularization learning, each parameter of vector 123 becomes a parameter that indicates whether each parameter of vector 121 is either zero or non-zero (it is bifurcated). By utilizing this L1 regularization learning, the computer can identify channels where γ is zero (or close to zero) as channels to be pruned.

[0023] The pruning target identification using L1 regularization learning, as shown in Figures 1 and 2, is applicable to convolutional layers connected to BN layers, but its application to other layers, such as convolutional layers without BN layers and fully connected layers, is not envisioned.

[0024] Figure 3 shows an example of whether the methods in Figures 1 and 2 can be applied to layers 131-139 of NN130. As shown in Figure 3, convolutional layers 131 and 133 and BN layers 132 and 134 are layers to which the L1 regularization learning shown in Figures 1 and 2 can be applied, while convolutional layers 135-137 and fully connected layers 138 and 139 are layers to which the L1 regularization learning shown in Figures 1 and 2 cannot be applied.

[0025] Therefore, in one embodiment, a method for reducing the size of a neural network is described by determining the pruning rate for each layer, regardless of the type of layer.

[0026] [1-1] Example of a server's functional configuration according to one embodiment Figure 4 is a block diagram showing an example of the functional configuration of Server 1 according to one embodiment. Server 1 is an example of a computer or information processing device that outputs a pruning rate. As shown in Figure 4, Server 1 may, as an example, include a memory unit 11, an acquisition unit 12, a machine learning unit 13, a pruning rate calculation unit (hereinafter simply referred to as the "calculation unit") 14, and an output unit 15. The acquisition unit 12, the machine learning unit 13, the calculation unit 14, and the output unit 15 are examples of a control unit 16.

[0027] The memory section 11 is an example of a storage area and stores various data used by the server 1. As shown in Figure 4, the memory section 11 may, as an example, store an untrained model 11a, machine learning data 11b, a trained model 11c, a pruning rate 11d, and a lightweight model 11e.

[0028] The acquisition unit 12 acquires the untrained model 11a and the machine learning data 11b and stores them in the memory unit 11. For example, the acquisition unit 12 may generate one or both of the untrained model 11a and the machine learning data 11b on the server 1, or it may receive them from a computer outside the server 1 via a network (not shown).

[0029] The untrained model 11a may be a pre-machine learning model of a neural network (NN) that includes untrained parameters. The NN may include various layers, for example, a DNN (Deep NN). The NN may include, for example, convolutional layers without connected BN layers, or fully connected layers, or convolutional layers with connected BN layers. As an example, it may be the NN130 illustrated in Figure 3.

[0030] The machine learning data 11b may be, for example, a training dataset used for machine learning (training) an untrained model 11a. For example, when performing machine learning of a neural network to realize image processing, the machine learning data 11b may include multiple pairs of training data, such as image data, and training data, which includes the correct labels for the training data.

[0031] In the machine learning phase, the machine learning unit 13 performs machine learning processing to train an untrained model 11a based on the machine learning data 11b. For example, the machine learning unit 13 may generate a trained model 11c by performing machine learning processing on the untrained model 11a. The trained model 11c may be a neural network model that includes trained parameters.

[0032] The machine learning model 11c may be obtained by updating the parameters included in the untrained model 11a, and can be considered, for example, as the model resulting from the transformation from the untrained model 11a to the machine learning model 11c through machine learning processing. The machine learning processing may be implemented using various known methods.

[0033] The calculation unit 14 calculates the pruning rate 11d by executing a pruning rate calculation process on the machine learning model 11c and stores it in the memory unit 11.

[0034] For example, the calculation unit 14 may include a threshold calculation unit 14a that calculates a threshold for each layer to select one of the pruning rate candidates, and a determination unit 14b that determines the pruning rate 11d to be adopted based on the inference accuracy of the model pruned with the pruning rate candidates.

[0035] The output unit 15 outputs output data based on the pruning rate 11d generated (acquired) by the calculation unit 14. The output data may include, for example, the pruning rate 11d itself and / or the lightweight model 11e.

[0036] The lightweight model 11e is data of a model obtained by performing pruning on the machine learning model 11c based on a pruning rate 11d, resulting in a lightweight version of the machine learning model 11c. For example, the output unit 15 may work in cooperation with the machine learning unit 13 to obtain the lightweight model 11e by applying the pruning rate 11d and performing pruning and retraining of the machine learning model 11c, and then store it in the memory unit 11. Note that the lightweight model 11e may be generated separately from the machine learning model 11c, or it may be data obtained by updating the machine learning model 11c through pruning and retraining.

[0037] The output unit 15 may, for example, transmit (provide) the output data to another computer not shown in the diagram, or it may store the output data in the memory unit 11 and manage it so that it can be retrieved from the server 1 or another computer. Alternatively, the output unit 15 may output information indicating the output data to an output device such as the server 1, or it may output the output data in various other ways.

[0038] [1-2] Example of pruning rate calculation process Next, an example of the pruning rate calculation process performed by the calculation unit 14 of server 1 will be described. In the following description, the target of the pruning rate calculation will be the weight matrix W, which is an example of the layer parameters.

[0039] The calculation unit 14 determines the pruning rate regardless of the type of layer by utilizing the tensor error for each layer that occurs due to pruning. As an example, the calculation unit 14 may calculate the pruning rate by following the steps (i) to (iii) below.

[0040] (i) The calculation unit 14 (threshold calculation unit 14a) determines (calculates) a pruning rate that can guarantee accuracy for each layer.

[0041] Note that "accuracy guarantee" means, for example, guaranteeing that the accuracy of inference (inference accuracy) using the lightweight model 11e obtained by pruning the machine-learned model 11c exceeds a predetermined standard.

[0042] FIG. 5 is a diagram showing an example of calculating a pruning rate capable of accuracy guarantee. As illustrated in FIG. 5, the threshold calculation unit 14a determines, in (i), the pruning rate to be applied to the weight matrix W of each layer included in the machine-learned model 11c to be pruned, for each weight matrix W of a plurality of layers. Note that, in FIG. 5, the description will be made by focusing on layers 131 to 133, but the present invention is not limited thereto, and the description of FIG. 5 may be applied to any of layers 131 to 139 illustrated in FIG. 3.

[0043] Here, the pruning rate is an example of the ratio (reduction ratio) of reducing elements of a layer, and indicates the ratio of making the pruning target in the machine-learned model 11c "sparse". In the example of FIG. 2, it means the number of locations set to "0" in the vector 123.

[0044] As illustrated in FIG. 5, the threshold calculation unit 14a selects one pruning rate from among a plurality of pruning rate candidates for each of the weight matrix W1 of layer 131 (weight matrix W1 connected to layer 132) and the weight matrix W2 between layer 132 (weight matrix W2 connected to layer 133). The pruning rate candidate is an example of a reduction ratio candidate, and may be, for example, two or more ratios between 0% and 100%, may be common to a plurality of layers, may be different for each layer, or may be a combination thereof. In the example of FIG. 5, it is assumed that the pruning rate candidates are 0%, 20%, 40%, and 60%.

[0045] The threshold calculation unit 14a obtains, for example, the error of the tensor before and after pruning when pruning is performed according to each of the pruning rate candidates, and determines the maximum pruning rate candidate among the pruning rate candidates having an error smaller than the threshold T W In the example of FIG. 5, for W1, the threshold calculation unit 14a determines the threshold T w1The largest pruning rate candidate with the smallest error is determined to be 40% (see arrow 141). Furthermore, the threshold calculation unit 14a determines the threshold T for W2. w2 We determine that the largest pruning rate candidate with the smallest error is 20% (see arrow 142).

[0046] Threshold T w This is the threshold of the error in the tensor before and after pruning, and is the upper limit of the pruning rate for which accuracy can be guaranteed. For example, the threshold calculation unit 14a approximates the loss function when the pruning target is pruned, for example, by performing a first-order Taylor expansion, and sets a threshold T for each layer. w You may calculate the threshold T. w Details of the calculation method will be described later.

[0047] Furthermore, the pruning rate calculated in (i) may be considered a "provisionally calculated" pruning rate in relation to the processes in (ii) and (iii).

[0048] As described above, the threshold calculation unit 14a calculates a threshold T for the tensor error before and after reduction of each element in each of the multiple layers of the machine-learned neural network model 11c, which includes multiple layers. The threshold calculation unit 14a also selects a reduction ratio candidate to apply to each of the multiple layers based on the multiple thresholds T and the tensor error before and after reduction when elements are reduced by each of the multiple reduction ratio candidates in each of the multiple layers.

[0049] (ii) The calculation unit 14 (determination unit 14b) determines the pruning rate based on the accuracy of the machine learning model that has been pruned (lightened) using the pruning rate determined in (i) and the accuracy of the machine learning model that has not been pruned.

[0050] For example, the determination unit 14b considers the error due to the approximation formula (first-order Taylor expansion) and calculates the accuracy Acc of the model pruned with the pruning rate of each layer determined in (i). p and precision margin Acc m The sum of the two, and the accuracy of the unpruned model Acc wo Compare the following: Accuracy margin Accm This is a margin that allows for a decrease in inference accuracy and may be set by the designer. The margin may also be "0", in which case the determination unit 14b will determine the accuracy Acc. p And the accuracy of the unpruned model Acc wo You should compare them.

[0051] Figure 6 shows an example of calculating the accuracy of the model before and after pruning. For example, the decision unit 14b calculates the accuracy Acc of the model (machine-learned model 11c) that is not pruned for all layers (W1, W2, ...). wo The Acc of the model is calculated (see arrow 143). A model that is not pruned may be considered a pruned model with the pruning rate of each layer set to 0%. The determination unit 14b calculates the accuracy Acc of the model pruned with the pruning rates (W1=40%, W2=20%, ...) calculated in (i) for each layer. p Calculate (see arrow 144).

[0052] The determination unit 14b calculates the sum of precision Acc. p +Acc m Accuracy wo If the above conditions are met, it is decided to adopt the pruning rate determined in (i). For example, the determination unit 14b stores the pruning rate determined in (i) as pruning rate 11d in the memory unit 11.

[0053] On the other hand, the determination unit 14b calculates the sum of precision Acc. p +Acc m Accuracy wo If it is less than (i), it is decided to discard the pruning rate determined in (i). For example, the decision unit 14b decides to discard the pruning rate determined in (i) and adopt the pruning rate 11d determined in the preceding (ii) (or the initial) pruning rate.

[0054] (iii) The calculation unit 14 (determination unit 14b) searches for the maximum pruning rate that can guarantee accuracy by repeatedly applying (i) and (ii) multiple times.

[0055] Figure 7 shows an example of pruning rate exploration. In the example in Figure 7, the calculation unit 14 performs pruning on three layers (131-133) three times. For example, pruning a certain layer with a pruning rate of 20% means that if the layer has "4" elements (e.g., channels), then 20% of "4" is "1" element that is pruned (reduced).

[0056] As illustrated in Figure 7, in the first search (see reference numeral 145), in (i), the threshold calculation unit 14a calculates the threshold T w Calculate the threshold T w Based on this, we assume that the pruning rate for layers 131-133 is determined from "0%,0%,0%" (initial value) to "40%,20%,40%". For example, in (ii), the decision unit 14b performs Acc in the comparison of inference accuracy. p +Acc m <Acc wo If this is determined, the pruning rate determined in (i) is discarded, and the pre-determination value of "0%,0%,0%" is adopted.

[0057] In the second search (see reference numeral 146), in (i), the threshold calculation unit 14a calculates the threshold T w The updated threshold T is calculated (updated). w Based on this, we assume that the pruning rate for layers 131-133 is determined from "0%,0%,0%" to "20%,20%,40%". For example, in (ii), the decision unit 14b determines Acc in the comparison of inference accuracy. p +Acc m ≥Acc wo If this is determined, then "20%, 20%, 40%" is adopted and stored in the memory unit 11 as the pruning rate 11d.

[0058] In the third search (see reference numeral 147), in (i), the threshold calculation unit 14a calculates the threshold T w The updated threshold T is calculated (updated). wBased on this, we assume that the pruning rate for layers 131-133 is determined to be changed from "20%,20%,40%" to "20%,40%,40%". For example, in (ii), the decision unit 14b determines Acc in the comparison of inference accuracy. p +Acc m ≥Acc wo If this is determined, "20%, 40%, 40%" is adopted and stored (updated) in the memory section 11 as the pruning rate 11d.

[0059] The determination unit 14b may search for the pruning rate over a predetermined number of times, such as a pre-set number of times.

[0060] As described above, the decision unit 14b determines the reduction ratio to apply to each of the multiple layers based on the inference accuracy of the machine learning model 11c and the machine learning-based inference accuracy of the reduced model obtained by reducing each element of the multiple layers in the machine learning model 11c according to the candidate reduction ratio to be applied.

[0061] Next, we will explain a specific example of the pruning rate calculation process described above. Figure 8 is a diagram illustrating an example of a threshold derivation method, and Figure 9 is a diagram showing an example of a threshold upper limit and threshold.

[0062] The threshold calculation unit 14a calculates a threshold pruning rate for each layer that guarantees accuracy by performing a first-order Taylor expansion of the loss function when pruning is performed. For example, Δw is the tensor error for each layer caused by pruning, L(w+Δw) is the loss function when pruning is performed, L(w) is the loss function of the model to be pruned, and (L) is the loss function when not pruning is performed. ideal ) to L wo +L m Therefore, the threshold pruning rate at which accuracy can be guaranteed is calculated by the following formula (4). Note that L wo L is the loss function of the model when pruning is not performed. m This is the margin of the loss function set by the designer.

number

[0063] The left-hand side of equation (4) above (see the dashed box in Figure 8) is the Taylor expansion of the loss function L(w+Δw) after pruning, and includes the weight gradient "∂L(W) / ∂w" for each layer being pruned. The gradient for each layer may be calculated by backpropagation. The right-hand side of equation (4) above (see the dashed box in Figure 8) is a constraint that even after pruning, the loss function will be smaller than the ideal value (for example, the loss function of FP32).

[0064] Thus, the threshold calculation unit 14a calculates a threshold T based on the value of the loss function of the machine learning model 11c when reducing each element of the multiple layers, and the weight gradient of each of the multiple layers.

[0065] By rearranging equation (4) above, we can derive the condition for the "pruning error" that satisfies the constraint that the loss function after pruning is smaller than the ideal loss function, as shown in equation (5) below. In other words, we can derive the upper limit (threshold) of the error due to pruning that guarantees accuracy (loss function). The threshold calculation unit 14a sets the right-hand side of equation (5) below to the threshold T.

number

[0066] As illustrated in Figure 9, the threshold calculation unit 14a compares the threshold T set for each layer with the error in the L1 norm due to pruning. The threshold calculation unit 14a then determines the pruning rate candidate with the largest value among the pruning rate candidates that have an error smaller than the threshold T (40% in the example in Figure 9) as the pruning rate resulting from (i).

[0067] As an example, the threshold calculation unit 14a may determine a pruning rate for each layer to be pruned such that the pruning error (left side) is less than or equal to the threshold (right side), according to the following equation (6). In the following equation (6), "||ΔW||1" is the L1 norm of the weights to be pruned, and "n" is the number of elements in the weights of the layer to be pruned. [Numerical]

[0068] As shown in the above formula (6), the threshold value T is a parameter derived by approximation. In order to prevent an error in determining the pruning rate due to the approximation error, an upper limit may be set for the threshold value T (see Fig. 9). For example, the threshold value calculation unit 14a may limit the magnitude of the threshold value T by a "confidence radius" based on the confidence region method. The confidence radius is an example of the threshold upper limit. As an example, the threshold value calculation unit 14a may scale the threshold value T so that the L2 norm of the threshold value T for all layers is less than or equal to the confidence radius. In the example of Fig. 9, T h represents a vector by the threshold value T for each layer, and "||T h ||2" represents the L2 norm of the threshold value T for all layers.

[0069] For example, the threshold value calculation unit 14a may update the confidence radius (for example, by a constant multiple, etc.) in addition to the pruning rate according to the accuracy comparison result in the process of (ii) by the determination unit 14b. Note that the initial value of the confidence radius may be set by, for example, a designer or the like.

[0070] As an example, the threshold value calculation unit 14a is the sum of accuracies Acc p +Acc m is greater than or equal to the accuracy Acc wo In this case, multiply the confidence radius by a constant K ("K>1.0"), and the sum of accuracies Acc p +Acc m is less than the accuracy Acc wo In this case, the confidence radius may be multiplied by a constant k ("0<k<1.0").

[0071] [1-3] Explanation according to the type of pruning target Next, examples of pruning methods and pruning error calculation methods depending on the type of pruning target will be described. Examples of pruning targets include channel pruning, node pruning, and weight pruning. The calculation unit 14 may determine the pruning target and the pruning error using weights corresponding to the pruning target, depending on the type of pruning target.

[0072] [1-3-1] Example of channel pruning Figure 10 illustrates an example of a method for determining which channels to prune, and Figure 11 illustrates an example of calculating the pruning error.

[0073] Figures 10 and 11 show the processing flow of the convolution operation. The subscripts H and W indicate the sizes of the input data, kernel, and output data, respectively, while the subscript Ch indicates the number of channels in the input data, kernel, and output data. The same applies to the descriptions of other types of data to be pruned.

[0074] (An example of a method for determining which channels to prune) If the type of pruning target is a channel, the calculation unit 14 calculates the L1 norm for each kernel corresponding to the channel of the output data. For example, as shown in "Before pruning" in Figure 10, the calculation unit 14 calculates the L1 norm for all kernels of the channel before pruning. This calculates the L1 norm for one channel.

[0075] Next, the calculation unit 14 prunes the channels of the output data according to the set pruning rate, in ascending order of the calculated L1 norm, as illustrated in the “after pruning” section of Figure 10.

[0076] (Example of pruning error calculation) As illustrated in Figure 11, the calculation unit 14 calculates the L1 norm of the kernel to be pruned. The L1 norm of the kernel to be pruned is obtained by subtracting the L1 norm of all kernels after pruning from the L1 norm of all kernels before pruning, that is, it is the difference between the L1 norms before and after pruning.

[0077] The calculation unit 14 may obtain the pruning error by dividing the calculated L1 norm by the total number of elements in the kernel before pruning.

[0078] [1-3-2] Example of node pruning Figure 12 illustrates an example of a method for determining which nodes to prune, and Figure 13 illustrates an example of calculating the pruning error.

[0079] (An example of a method for determining which nodes to prune) If the type of pruning target is a node, the calculation unit 14 calculates the L1 norm in units of the weights connected to the output node. In the “before pruning” example in Figure 12, the calculation unit 14 calculates the L1 norm in units of the solid line, dashed line, and dotted line.

[0080] Next, the calculation unit 14 prunes the corresponding output nodes in order of increasing L1 norm, according to the set pruning rate, as illustrated in the “after pruning” section of Figure 12. For example, the calculation unit 14 determines the output nodes corresponding to the weight group with the smallest L1 norm to be pruned.

[0081] (Example of pruning error calculation) As illustrated in Figure 13, the calculation unit 14 calculates the L1 norm of the weight group to be pruned. The L1 norm of the weight group to be pruned is obtained by subtracting the L1 norm of all weights after pruning from the L1 norm of all weights before pruning.

[0082] The calculation unit 14 may obtain the pruning error by dividing the calculated L1 norm by the total number of elements in the weights before pruning. In the example of “after pruning” in Figure 13, the calculation unit 14 calculates the L1 norm of the weight group of the dashed line and divides the L1 norm by the total number of elements in the weights before pruning (= “6”; number of lines).

[0083] [1-3-3] Example of weighted pruning Figure 14 illustrates an example of a method for determining pruning weights, and Figure 15 illustrates an example of calculating pruning error.

[0084] (An example of a method for determining pruning weights) If the type of pruning target is weights, the calculation unit 14 calculates the L1 norm for each element of all weights. In the example of “before pruning” in Figure 14, the number of elements in the weights is “6”, so the calculation unit 14 calculates “6” L1 norms.

[0085] Next, the calculation unit 14 prunes the weights according to the set pruning rate, in ascending order of the calculated L1 norm, as illustrated in the “After pruning” section of Figure 14. For example, the calculation unit 14 determines the weights with the smallest L1 norm to be pruned.

[0086] (Example of pruning error calculation) As illustrated in Figure 15, the calculation unit 14 calculates the L1 norm of the weights to be pruned. The L1 norm of the weights to be pruned is obtained by subtracting the L1 norm of all weights after pruning from the L1 norm of all weights before pruning.

[0087] The calculation unit 14 may obtain the pruning error by dividing the calculated L1 norm by the total number of elements in the weights before pruning. In the example of “after pruning” in Figure 15, the calculation unit 14 calculates the L1 norm of the dashed line weights and divides the L1 norm by the total number of elements in the weights before pruning (= “6”; number of lines).

[0088] [1-4] Explanation of pruning process for neural networks with attention structure Figure 16 shows an example of an NN150 equipped with an attention structure 160. Figure 16 shows an example where NN150 is a neural network called a Transformer. Note that NN150 is not limited to a Transformer, but may be various types of neural networks equipped with an attention structure 160.

[0089] The NN150 comprises embedding layers 151a and 151b, positional encoding layers 152a and 152b, an encoder 150a, a decoder 150b, a fully coupled layer (labeled "Linear" in Figure 16) 155, and a Softmax 156.

[0090] Encoder 150a includes Add&Norm 153a and 153b, Feed Forward 154a, and MHA 160a. Decoder 150b includes Add&Norm 153c, 153d, and 153e, Feed Forward 154b, and MMHA (Masked MHA) 160b and MHA 160c. Since the Transformer is a known neural network, the description of each layer in NN150 is omitted.

[0091] In NN150 shown in Figure 16, MHA160a, MMHA160b, and MHA160c are each examples of attention structures 160.

[0092] Figure 17 shows an example of an attention structure 160. The attention structure 160 receives an input tensor having two dimensions: tokens and features. A feature is an example of the number of elements.

[0093] The following explanation will use the case where the attention structure 160 is an MHA structure as an example, but it is not limited to this, and the attention structure 160 may also be an attention structure with one head (single head).

[0094] As illustrated in Figure 17, the attention structure 160 includes fully bonded layers 161-163, 166, an attention layer 164, and a concat portion (labeled "Concat" in Figure 17) 165.

[0095] The fully connected layers 161-163 are an example of the input section of the attention structure 160, and are layers that perform operations on the input tensor and output the tensors Q, K, and V, respectively. In the following description, the fully connected layer 161 that outputs the tensor Q may be referred to as the Q layer, the fully connected layer 162 that outputs the tensor K as the K layer, and the fully connected layer 163 that outputs the tensor V as the V layer.

[0096] The attention layer 164 includes, for example, a layer (structure) called Scaled Dot-Product Attention. In the example shown in Figure 17, the attention layer 164 may include H (an integer greater than or equal to 1) scaled dot-product attentions, which is the number of headers.

[0097] The concat section 165 is an example of a concatenation section, and performs a concat operation that combines multiple tensors input from the attention layer 164 and outputs a tensor resulting from the combination.

[0098] The fully connected layer 166 performs operations on the tensor input from the concat section 165 and outputs the resulting tensor.

[0099] Figure 18 shows a detailed example of the attention structure 160. In Figure 18, the attention structure 160 is an MHA with an input tensor 170 having 1 token and 16 features, and a head count H=4.

[0100] The Q layer takes input tensor 170 as input and outputs tensor 171a of Q. The K layer takes input tensor 170 as input and outputs tensor 171b of K. The V layer takes input tensor 170 as input and outputs tensor 171c of V.

[0101] The attention layer 164 may include Split 164a to 164c, Matmul 164d and 164f, and Softmax 164e.

[0102] Split164a~164c divides tensor 171a~171c into multiple heads H based on the feature dimension, thereby creating a multi-head tensor.

[0103] For example, Split164a takes a tensor 171a containing 16-dimensional features as input and splits tensor 171a into four heads, outputting four 4-dimensional tensors 172a. Split164b takes a tensor 171b containing 16-dimensional features as input and splits tensor 171b into four, outputting four 4-dimensional tensors 172b. Split164c takes a tensor 171c containing 16-dimensional features as input and splits tensor 171c into four, outputting four 4-dimensional tensors 172c.

[0104] Matmul164d takes tensor Q 172a and tensor K 172b as input and calculates the matrix product of Q and K.

[0105] For example, the tensor 172a of Q is Q head Q head The elements of q f Let the tensor 172b of K be K head Toshi, K head k f Let A be the result of matrix multiplication by Matmul164d. head Then, matrix product A head The following is how it is calculated. Note that `head` is the index of each head, and in the example in Figure 18, it is an integer from 0 to 3. `f` is the index of each feature, and in the example in Figure 18, it is an integer from 0 to 15. A0 = Q0 · K0 T = q0·k0 + q1·k1 + q2·k2 + q3·k3 A1 = Q1 · K1 T =q4·k4+q5·k5+q6·k6+q7·k7 A2 = Q2 · K2 T = q8·k8+q9·k9+q 10 ·k 10 +q 11 ·k 11 A3=Q3·K3 T =q 12 ·k 12 +q 13 ·k 13 +q 14 ·k 14 +q 15 ·k 15

[0106] As described above, in the matrix multiplication operation in Matmul164d, the product (inner product) of elements with the same index between Q and K is calculated.

[0107] Therefore, it can be said that the attention structure 160 is subject to the following constraints (1') and (2). (constraint 1')Q head and K head The number of heads must be the same (the same number) as the other unit. (Constraint 2)Q head and K head The number of features between the two heads must be the same (equal).

[0108] Softmax164e outputs Att(Attention Weight)173 by normalizing the result of the matrix multiplication calculated by Matmul164d. For example, Softmax164e may calculate Att173 according to the following formula. Workt=Softmax(A head )

[0109] Alternatively, Softmax164e may calculate Att173 according to the following formula. In the following formula, d x is, A head The number of dimensions is (4 in the example in Figure 18), and Softmax{} is the function that performs normalization. Att=Softmax{A head / √(d x )}

[0110] Matmul164f takes Att173 and the tensor 172c of V as inputs and calculates the matrix product of the weights (Att173) and V. For example, Matmul164f outputs four tensors 174 as the result of the matrix product calculation.

[0111] For example, let Att173 be An head and the tensor 172c of V be V head and let the elements of V head be v f and let the result of the matrix product by Matmul164f be C head Then the matrix product C head is calculated as follows. C0 = An0 · V0 = [An0 · v0, An0 · v1, An0 · v2, An0 · v3] C1 = An1 · V1 = [An1 · v4, An1 · v5, An1 · v6, An1 · v7] C2 = An2 · V2 = [An2 · v8, An2 · v9, An2 · v 10 , An2 · v 11 C3 = An3 · V3 = [An3 · v 12 , An3 · v 13 , An3 · v 14 , An3 · v 15

[0112] As described above, in the matrix product operation of Matmul164f, the product (inner product) of the indices of the same head between the weights (Att173) and V is calculated.

[0113] Therefore, it can be said that the following (Constraint 1”) is imposed on the attention structure 160. (Constraint 1”) The number of heads of the weights (Q head and K head ) and V head is the same.

[0114] Note that (Constraint 1’) and (Constraint 1”) can be integrated and regarded as the following (Constraint 1). (Constraint 1) Qhead and K head and V head The number of heads must be the same (the same number) as the other unit.

[0115] The concat unit 165 combines the elements of multiple (four in the example in Figure 18) tensors 174 (minitensors) to output a single tensor 175.

[0116] For example, if we denote the result of the concatenation by concat unit 165 (tensor 175) as C, then the result C is calculated as follows. C=[C0,C1,C2,C3] =[An0·v0,An0·v1,An0·v2,An0·v3 ,An1·v4,An1·v5,An1·v6,An1·v7 An2·v8,An2·v9,An2·v 10 ,An2·v 11 ,An3·v 12 ,An3·v 13 ,An3·v 14 ,An3·v 15 ]

[0117] As described above, the concatenation operation (concat operation) in concat section 165 assumes that the tensor sizes (number of elements in each dimension) of the tensor 175 (C0, C1, C2, C3) input to concat section 165 are the same.

[0118] Therefore, it can be said that the attention structure 160 is subject to the following constraint (3). (Constraint 3)V head The number of features must be the same (equal) between the heads.

[0119] Thus, in order to input the input tensor 170 into the attention structure 160 and obtain the tensor 175, the above-mentioned constraints (1) to (3) must be satisfied. Note that if the attention structure 160 is a single-head attention structure, the constraints will be replaced by only (2') below, instead of (1) to (3). (constraint 2')Q head and K head The number of features between [the two points] must be the same (equal).

[0120] Let's assume that the pruning rates for the fully connected layers 161-163 (Q layer, K layer, V layer) are selected independently (for example, so that at least one is different) using the pruning method of the pruning rate calculation unit 14, as explained with reference to Figures 5-9, etc.

[0121] In this case, at least one of the tensors 171a to 171c output from the fully connected layers 161 to 163 will have a different size than the others, making it impossible to calculate Att173 or tensor 175. Furthermore, since pruning is performed independently for all layers of the machine learning model, it is difficult to determine before pruning which layer—Q, K, or V—in the attention structure 160 will have the maximum number of output nodes.

[0122] To avoid the inability to calculate Att173 and Tensor175, one could consider uniformly excluding the fully connected layers 161-163 in attention structure 160 from the pruning rate determination. However, in this case, the more attention structures included in the NN increase, the lower the overall pruning rate of the NN machine learning model becomes, and the data size compression (lightening) effect of pruning on the machine learning model decreases.

[0123] Therefore, in one embodiment, the calculation unit 14 inserts a zero-padding layer on the output side (downstream) of at least the fully bonded layers 161 and 162 (or fully bonded layers 161 to 163 in the case of an MHA structure).

[0124] A zero-padding layer is a layer used to pad a tensor with "0" (zero) values ​​for specific elements (e.g., channels). Padding is an operation that increases the size of a tensor (e.g., the number of channels) by embedding values ​​such as zero into the tensor. A zero-padding layer is an example of a padding layer that pads one or more elements of a tensor. Padding layers are not limited to zero-padding layers; layers that embed various values ​​such as values ​​close to "0" into the tensor may also be used.

[0125] Figure 19 illustrates an example of inserting a zero-padding layer into a model. For example, Figure 19 shows model 180 after inserting a zero-padding layer into NN150, which includes the attention structure 160 shown in Figure 18.

[0126] The process shown in Figure 19 may be executed by selecting a pruning rate candidate if the NN150 to be pruned includes an attention structure 160, and the execution of the process may be suppressed if it does not include the attention structure 160. For example, the calculation unit 14 may determine whether or not the NN150 includes an attention structure 160 by referring to configuration information (not shown) that defines the configuration of the NN150, such as the configuration of each layer and the connection relationships between layers. The calculation unit 14 may also identify fully connected layers 161 to 163 for each attention structure 160 based on the configuration information.

[0127] Furthermore, Figure 19 shows an example in (i) above where the calculation unit 14 calculates the L1 norm for each kernel corresponding to the channel of the output data, and the pruning rate is provisionally calculated by L1 regularization learning (see Figure 2), etc.

[0128] As illustrated in Figure 19, the calculation unit 14 inserts (places) zero-padding layers (indicated as "Padding" in Figure 19) 181-183 after each of the fully connected layers 161-163 (Q layer, K layer, V layer), for example, after each of the Split layers 164a-164c. Then, if the attention structure 160 is an MHA structure, the calculation unit 14 performs zero-padding using at least one of the zero-padding layers 181-183 so as to satisfy all of the following conditions (I)-(III). For example, the calculation unit 14 may identify the number of channels in the Q layer, K layer, and V layer based on the provisionally calculated pruning rate, and determine the number of channels to be zero-padding according to the identified number of channels.

[0129] (I) The number of heads in each of the following tensors are the same: tensor 172a from layer Q after element reduction based on the first reduction ratio, tensor 172b from layer K after element reduction based on the second reduction ratio, and tensor 172c from layer V after element reduction based on the third reduction ratio.

[0130] (II) The number of elements between the same heads in tensor 172a and tensor 172b is the same.

[0131] (III) The number of elements is the same between the heads of the tensor 172c.

[0132] Furthermore, if the attention structure 160 is a single-head attention structure, the calculation unit 14 may, instead of (I) to (III) above, perform zero padding using zero padding layers inserted on the output side of each of the Q and K layers so as to satisfy the condition (II') below.

[0133] (II') The number of elements between tensor 172a and tensor 172b is the same.

[0134] Tensor 172a from layer Q is an example of tensor QT, tensor 172b from layer K is an example of tensor KT, and tensor 172c from layer V is an example of tensor VT. In the following explanation, tensors 172a, 172b, and 172c may be simply referred to as "Q," "K," and "V," respectively.

[0135] This allows the number of elements (size) of the Q, K, and V tensors in the attention structure 160 to be the same. Consequently, the fully connected layers 161-163 of the attention structure 160 can be pruned, improving the data size compression ratio of the machine learning model through pruning.

[0136] Figure 20 is a diagram illustrating a zero-padding example for Model 180. In the example in Figure 20, for simplicity, the number of features in the input tensor is assumed to be 12; in other words, the output of each of the Q, K, and V layers (e.g., Split164a~164c) is assumed to have 4 heads H and 3 channels per head.

[0137] In Figure 20, code A indicates an example of the unpruned tensors 172a~172c (Q, K, V) output from layers Q, K, and V, respectively.

[0138] In Figure 20, code B indicates an example of tensors 172a to 172c after pruning (or during pruning) output from layers Q, K, and V, respectively.

[0139] In Figure 20, the symbol C indicates an example of head pruning by the calculation unit 14. For example, the calculation unit 14 prunes a head if all elements with the same head number are 0 in the tensors 172a to 172c of the Q, K, and V layers, respectively. The head number is an example of head identification information and corresponds to the head mentioned above. In the example in Figure 20, the calculation unit 14 prunes head 1 as shown by symbols C1 to C3.

[0140] In Figure 20, labels D, E, and F indicate examples of zero padding performed by the calculation unit 14 on the pruned tensors 172a to 172c, indicated by label C.

[0141] As indicated by symbol D, the calculation unit 14 performs zero padding so that the number of elements in tensors other than the tensor with the largest number of elements among the number of elements in Q and the number of elements in K becomes the maximum number of elements. For example, for each identical head number in Q and K, the calculation unit 14 inserts a zero matrix so that the number of elements in the head of a certain head number included in Q and the number of elements in the head of the same head number included in K become the same.

[0142] In the example shown in Figure 20, the maximum number of elements in Q is 2 (q0, q1) between heads 0 of Q and K, indicated by symbol D1, and the maximum number of elements in K is 1 (k9) between heads 3 of Q and K, indicated by symbol D2. Therefore, as shown in symbol D1, the calculation unit 14 inserts one zero (zero matrix) into head 0 of K (k0), which has 1 element, using the padding layer 182 to match the number of elements in head 0 of Q (2). Also, as shown in symbol D2, the calculation unit 14 inserts one zero (zero matrix) into head 3 of Q, which has 0 element, using the padding layer 181 to match the number of elements in head 3 of K (1).

[0143] This ensures that the number of features is the same (matches) between the Q and K heads, thus satisfying constraint (2) above. In other words, the zero-padding indicated by code D is a process that complies with condition (II) above.

[0144] As indicated by the symbol E, the calculation unit 14 zero-paddings tensors other than the one with the largest number of elements in each head of V so that the number of elements in that tensor becomes the maximum possible. For example, the calculation unit 14 inserts a zero matrix so that the number of elements is the same across all heads of V.

[0145] In the example shown in Figure 20, as indicated by the symbol E1, the calculation unit 14 inserts one zero (zero matrix) into head 2 (number of elements: 2 (v6, v7)) using the padding layer 183 to match the number of elements in head 0: 3 (v0, v1, v2). Also, as indicated by the symbol E2, the calculation unit 14 inserts two zeros (zero matrix) into head 3 (number of elements: 1 (v10)) using the padding layer 183 to match the number of elements in head 0: 3 (v0, v1, v2).

[0146] This ensures that the number of features is consistent across the heads of V, thus satisfying constraint (3) above. In other words, the zero-padding indicated by code E is a process that complies with condition (III) above.

[0147] As indicated by the symbol F, the calculation unit 14 inserts zero matrices so that the number of heads in Q, K, and V matches. For example, if there are heads with the same head number in Q, K, and V that do not have any elements, the calculation unit 14 inserts zero matrices into those heads.

[0148] In the example in Figure 20, the head 2 of V contains elements (v6, v7, zero), while the head 2 of Q and the head 2 of K contain no elements, as indicated by the symbols F1 and F2. Therefore, the calculation unit 14 inserts one zero (zero matrix) into the head 2 of Q, as indicated by the symbol F1, and one zero (zero matrix) into the head 2 of K, as indicated by the symbol F2.

[0149] This ensures that the number of heads is the same across Q, K, and V, thus satisfying constraint (1) above. In other words, the zero padding indicated by the symbol F is a process that complies with condition (I) above.

[0150] In Figure 20, the symbol G represents a matrix multiplication operation using Q and K with Matmul164d. In Matmul164d, zero padding in the symbol D ensures that every element of the existing head has an element to "multiply" with, thus enabling matrix multiplication. Note that in matrix multiplication, as long as the indices (e.g., head numbers) of Q and K match, zero padding inserts 0 (or a value close to 0) into the tensor in Q and K, but this has little to no effect on the sum of the inner product result (elemental product).

[0151] For example, Matmul164d outputs the following result G1 as the result of a matrix multiplication operation. A0 = Q0 · K0 T =q0·k0+q1· 0 A2 = Q2 · K2 T = 0. 0 A3=Q3·K3 T = 0·k9

[0152] The symbol H in Figure 20 represents the normalization operation performed by Softmax164e using result G1. For example, Softmax164e outputs the following result H1 as the result of the normalization operation. Result H1 is an example of Att173 shown in Figure 19. An0 = Softmax(A0) An2 = Softmax(A2) An3=Softmax(A3)

[0153] In Figure 20, the symbol I represents a matrix multiplication operation performed by Matmul164f using the results G1 and V. In Matmul164f, zero padding in symbol F ensures that every element of the head has an element to be multiplied, making matrix multiplication possible.

[0154] The V (see code F3) input to Matmul164f is as follows: V0=[v0,v1,v2] V2=[v6,v7, 0] V3=[v 10 , 0, 0]

[0155] For example, Matmul164f outputs the following result I1 as the result of the matrix multiplication operation between result G1 and V (sign F3). Result I1 is an example of tensor 174 shown in Figure 19. C0=An0·V0=[An0·v0,An0·v1,An0·v2] C2=An2·V2=[An2·v6,An2·v7,An2· 0] C3 = An3 · V3 = [An3 · v 10 An3. 0, An3· 0]

[0156] Thus, the attention structure 160 outputs a matrix product (code G1) obtained by normalizing the matrix product of padded Q and padded K, and a matrix product (code I1) based on padded V (code F3).

[0157] In Figure 20, the symbol J represents the concat operation performed by the concat unit 165 using the result I1. In the concat unit 165, zero padding in symbol E makes the number of elements between heads in the input V the same, and the number of features in the multiple vectors to be concatenated (result I1) becomes the same, thus enabling concatenation.

[0158] For example, the concat unit 165 outputs the following result J1 as the result of the concat operation on result I1. Result J1 is an example of the tensor 175 shown in Figure 19. C=[C0,C1,C2] =[An0·V0,An0·V1,An0·V2, = An2·V6,An2·v7, 0, = An3·V 10 , 0, 0]

[0159] As described above, zero-padding allows the number of tensor elements (size) to be the same for each of the Q, K, and V layers. Therefore, it becomes possible to prune the Q, K, and V layers using the provisionally calculated pruning rate candidates, thereby improving the data size compression ratio of machine learning models including attention structure 160.

[0160] The processes described with reference to Figures 18 to 20 may be part of the process (i) performed by the threshold calculation unit 14a, or may be performed by the threshold calculation unit 14a.

[0161] Furthermore, the processing of the calculation unit 14 after the execution of the process described with reference to Figures 18 to 20 is the same as the processing in (ii) and (iii).

[0162] The zero-padding process described above is not limited to cases where the element is a channel, but may also be performed when the element is a weight, or when the element is a node, or both.

[0163] Figure 21 shows an example of the accuracy of a neural network before and after pruning, and the data size compression ratio, depending on whether zero-padding is applied. In Figure 21, the model is a BERT (Bidirectional Encoder Representations from Transformers) base that has been trained on QQP (Binary Classification Task).

[0164] In Figure 21, "No Zero Padding Layer Insertion" means that the fully connected layers 161-163 of the attention structure 160 (MHA structure) are excluded from pruning without applying zero padding. "Zero Padding Layer Insertion" means that the fully connected layers 161-163 of the attention structure 160 (MHA structure) are pruned after applying zero padding.

[0165] As illustrated in Figure 21, when zero-padding is applied, the data size compression ratio of the lightweight model 11e can be improved while suppressing the degradation of accuracy compared to when zero-padding is not applied.

[0166] [1-5] Example of operation Next, an example of the operation of Server 1 according to one embodiment will be described with reference to Figure 22. Figure 22 is a flowchart illustrating an example of the operation of processing by Server 1 according to one embodiment.

[0167] As illustrated in Figure 22, the machine learning unit 13 performs machine learning on the untrained model 11a acquired by the acquisition unit 12 without pruning (step S1).

[0168] The calculation unit 14 calculates the inference accuracy (recognition rate) Acc when pruning is not performed. wo Calculate (Step S2).

[0169] The threshold calculation unit 14a sets the initial value of the confidence radius (step S3).

[0170] The threshold calculation unit 14a calculates the threshold T for each layer and the pruning error for each layer to set the pruning rate (step S4), and determines whether the L2 norm of the threshold T for all layers is greater than the confidence radius (step S5). If the L2 norm of the threshold T for all layers is less than or equal to the confidence radius (NO in step S5), the process proceeds to step S7.

[0171] If the L2 norm of the threshold T for all layers is greater than the confidence radius (YES in step S5), the threshold calculation unit 14a scales (updates) the thresholds so that the L2 norm of the threshold T for all layers equals the confidence radius (step S6), and the process proceeds to step S7.

[0172] In step S7, the threshold calculation unit 14a provisionally calculates the pruning rate for each layer. For example, the threshold calculation unit 14a provisionally sets the pruning rate for each layer from the set pruning rate candidates.

[0173] The calculation unit 14 determines whether the layers for which the pruning rate has been provisionally calculated include the fully connected layers 161-163 of the attention structure 160 (step S8). If the layers for which the pruning rate has been provisionally calculated do not include the fully connected layers 161-163 (NO in step S8), the process proceeds to step S11.

[0174] If the layers for which the pruning rate has been provisionally calculated include the fully connected layers 161-163 of the attention structure 160 (YES in step S8), the calculation unit 14 inserts zero-padding layers 181-183 into the output of each of the fully connected layers 161-163 (step S9), executes the process in step S10, and the process proceeds to step S11.

[0175] In step S10, the calculation unit 14 zero-padding the zero-padding layers 181 to 183 so that the number of heads and elements (number of channels) of each output (Q, K, V) of the fully connected layers 161 to 163 satisfy the above-mentioned conditions (I) to (III). Steps S4 to S10 are an example of the process described in (i) above.

[0176] The machine learning unit 13 prunes the machine learning model 11c using the pruning rate provisionally calculated by the threshold calculation unit 14a, and then performs re-machine learning on the pruned model. The calculation unit 14 calculates the inference accuracy Acc of the re-machine learning model. p Calculate (Step S11).

[0177] The decision unit 14b calculates the inference accuracy Acc. p +MarginAcc m Acc wo It is determined whether the result is above or below (step S12). By evaluating the inference accuracy (recognition rate), errors in pruning rate selection due to approximation errors can be compensated for.

[0178] Inference accuracy Acc p +MarginAcc m Acc woIf the result is as described above (YES in step S12), the decision unit 14b decides to prune the machine learning model 11c with the provisionally calculated pruning rate (step S13), and stores the provisionally calculated pruning rate as the pruning rate 11d in the memory unit 11. The threshold calculation unit 14a then increases the confidence radius by a constant multiplier (step S14), and the process proceeds to step S17.

[0179] On the other hand, inference accuracy Acc p +MarginAcc m Acc wo If the value is less than (NO in step S12), the determination unit 14b discards the provisionally calculated pruning rate (step S15). The threshold calculation unit 14a reduces the confidence radius by a constant multiplier (step S16), and the process proceeds to step S17. Steps S10 to S16 are an example of the process described in (ii) above.

[0180] In step S17, the decision unit 14b determines whether the search (processing in steps S4 to S16) has been performed a predetermined number of times, in other words, whether the number of times the processes of threshold calculation, pruning rate candidate selection, and pruning rate determination have been performed satisfies the predetermined conditions. If the search has not been performed a predetermined number of times (NO in step S17), the process proceeds to step S4.

[0181] If the search is performed a predetermined number of times (YES in step S17), the output unit 15 outputs the determined pruning rate 11d (step S18), and the process ends. Step S17 is an example of the process described in (iii) above.

[0182] As described above, the server 1 according to one embodiment calculates, by the threshold calculation unit 14a, an error generated by pruning of a tensor used in the NN, and generates a threshold from the value of the loss function and the gradient obtained by backpropagation of the NN. Further, the threshold calculation unit 14a compares the calculated pruning error with the threshold and temporarily calculates the pruning rate. Furthermore, the determination unit 14b compares the inference accuracy of the model after re-learning at the calculated pruning rate with the inference accuracy of the model when not pruning, and determines the pruning rate for each layer. At this time, when it is determined that the inference accuracy when pruning is deteriorated compared to the inference accuracy when not pruning, the threshold calculation unit 14a re-sets the upper limit of the threshold so that the threshold becomes smaller, and searches for the pruning rate again.

[0183] According to this, according to the server 1 according to one embodiment, it is possible to determine the pruning rate for each layer regardless of the type of layer. For example, the server 1 can determine the pruning rate for each layer to be applied to the learned machine learning model 11c including a convolutional layer, a fully connected layer, etc. to which the BN layer is not connected.

[0184] Further, according to the server 1, even when the NN includes the attention structure 160, the fully connected layers 161 to 163 of the attention structure 160 can be appropriately pruned, and the compression rate of the data size of the lightweight model 11e can be improved.

[0185] 〔1-6〕Variation Next, a variation according to one embodiment will be described. In the following description, for simplicity, the margin Acc of the inference accuracy m is assumed to be "0". In other words, in the comparison of the inference accuracy, it is assumed that the case where it is determined whether the inference accuracy Acc p is greater than or equal to the inference accuracy Acc wo is considered. Further, in the following description, the case where the NN does not include the attention structure 160 will be taken as an example, but the processing described with reference to FIGS. 16 to 21 is similarly applicable to both of the following first and second variations.

[0186] 〔1-6-1〕First Variation In the method according to one embodiment, the number of search times of the pruning rate (the number of trials of the process in (iii) above) is a hyperparameter manually set by, for example, a designer. Therefore, for example, when the number of search times is set to be small, the machine-learned model 11c may not be sufficiently lightweight, and when the number of search times is set to be large, although the machine-learned model 11c is sufficiently lightweight, the search time may become long.

[0187] FIG. 23 is a diagram showing an example of a comparison result of pruning errors according to an update of a trust radius in a method according to one embodiment.

[0188] As illustrated in FIG. 23, assume a case where a pruning rate of “10%” is calculated (determined) in the error comparison result of the m-th (m is an integer of “1” or more) search. In this case, the trust radius is updated so as to increase by a constant K times. However, if the updated trust radius is less than the error by a pruning rate candidate that is one larger than the pruning rate candidate determined at the m-th time, the pruning rate of “10%” is calculated again in the error comparison result of the (m + 1)-th search.

[0189] Thus, when the trust radius is multiplied by a constant K or a constant k, the update amount of the threshold is limited by the trust radius, so the same pruning rate candidate may be adopted in a plurality of searches. The state where the combination of the same pruning rate is searched for a plurality of times leads to an increase in the number of search times of the pruning rate without sufficiently trying the pruning of the model.

[0190] Therefore, in the first modification, a method for shortening (decreasing) the search time (number of search times) of an appropriate pruning rate for lightweighting the NN will be described by focusing on the update of the trust radius.

[0191] Figure 24 is a block diagram showing an example of the functional configuration of server 1A according to the first modified example. As illustrated in Figure 24, server 1A may be equipped with a calculation unit 14A that is different from server 1 in Figure 4. The calculation unit 14A may be equipped with a threshold calculation unit 14a' and a determination unit 14b' that are different from the calculation unit 14 in Figure 4.

[0192] The calculation unit 14A searches for different combinations of pruning rates for each search. Here, the state in which a combination of pruning rates of "0%" for all layers is selected is assumed to be a state in which the calculation unit 14A has decided not to search for pruning rates any further. Under these conditions, the calculation unit 14A (decision unit 14b') terminates the search when it selects a combination of pruning rates of "0%" for all layers.

[0193] The threshold calculation unit 14a' calculates, according to the comparison result of the inference accuracy by the determination unit 14b', for each layer i (where i is an integer of 1 or more), either the error of the pruning rate that is one value greater than the explored pruning rate, or the absolute value of the difference between the error of the explored pruning rate and the threshold "E diff,i "Measure."

[0194] For example, the threshold calculation unit 14a' calculates the inference accuracy Acc. p Acc wo If the above is true, the absolute value of the difference between the error of a pruning rate one value larger than the explored pruning rate and the threshold is "E". diff,i "Measure."

[0195] On the other hand, the threshold calculation unit 14a' calculates the inference accuracy Acc. p Acc wo If less than the threshold, the absolute value of the difference between the error of the explored pruning rate and the threshold is "E diff,i "Measure."

[0196] The threshold calculation unit 14a' calculates the absolute value of the difference between all layers, “E” as illustrated in equation (7) below. diff,i The smallest value (difference) among them "E diff Obtain ". Ediff = min(E diff,1 , E diff,2 , ..., E diff,i ) (7)

[0197] The threshold calculation unit 14a' calculates a constant multiple of the confidence radius, as well as the difference "E" from the confidence radius, according to the comparison result of the inference accuracy by the determination unit 14b'. diff The confidence radius is updated by adopting the larger of the sum or difference between the two.

[0198] For example, the threshold calculation unit 14a' calculates the inference accuracy Acc. p Acc wo If the above is true, the confidence radius is multiplied by a constant K, and the confidence radius is multiplied by the difference "E". diff The larger of the two values ​​(the sum of the two) is adopted and updated to increase the confidence radius.

[0199] On the other hand, the threshold calculation unit 14a' calculates the inference accuracy Acc. p Acc wo If it is less than the confidence radius, the constant k times the confidence radius, and the difference "E" from the confidence radius. diff The larger of the two differences is adopted, and the confidence radius is updated to decrease.

[0200] In this way, the threshold calculation unit 14a' updates the confidence radius so that the combinations of pruning rate candidates for each of the multiple layers are different from each other each time the process of selecting pruning rate candidates (in other words, the search) is executed.

[0201] Figure 25 illustrates an example of the confidence radius update process when increasing the confidence radius. As shown in Figure 25, it is assumed that the pruning rate explored on the mth time is "(Layer 1, Layer 2) = (10%, 0%)". The threshold calculation unit 14a' calculates the absolute value of the difference between the error of Layer 1's pruning rate "20%" and the confidence radius "E diff,1 ", and the absolute value of the difference between the error and the confidence radius of the pruning rate "10%" for Layer 2 "E diff,2The threshold calculation unit 14a' calculates the smallest difference “E” according to the above formula (7). diff,2 " to "E diff It will be obtained as ".

[0202] Then, the threshold calculation unit 14a' determines (updates) the confidence radius for the m+1th time (next time) according to the following equation (8). (m+1th confidence radius) = max((mth confidence radius · constant K), (mth confidence radius + E diff )) (8)

[0203] As a result, the m+1th confidence radius will be a value greater than or equal to the sum of the confidence radius and the difference, so a different bit width will be calculated as the pruning rate in the m+1th iteration compared to the mth iteration.

[0204] In the example in Figure 25, the confidence radius (upper threshold) in the m+1th search coincides with the error of a pruning rate of "10%" for Layer 2. Therefore, in the m+1th search, a different combination of pruning rates from the previous search, pruning rate "(Layer 1, Layer 2) = (10%, 10%)", is searched.

[0205] Figure 26 illustrates an example of a confidence radius update process when the confidence radius is reduced. As shown in Figure 26, it is assumed that the pruning rate explored on the mth time is "(Layer 1, Layer 2) = (10%, 0%)". The threshold calculation unit 14a' calculates the absolute value of the difference between the error of the pruning rate of Layer 1 at "10%" and the confidence radius "E diff,1 ", and the absolute value of the difference between the error and the confidence radius for a pruning rate of Layer 2 of "0%" "E diff,2 The threshold calculation unit 14a' calculates the smallest difference “E” according to the above formula (7). diff,1 " to "E diff It will be obtained as ".

[0206] Then, the threshold calculation unit 14a' determines (updates) the confidence radius for the m+1th time (next time) according to the following formula (9). (m+1th confidence radius) = max((the trust radius of the m-th iteration · constant), (the trust radius of the m-th iteration - E diff )) (9)

[0207] As a result, for the trust radius of the (m + 1)-th iteration, a value of at least "the difference between the trust radius and the difference" is selected. Therefore, in the (m + 1)-th iteration, a different bit width from the m-th iteration is calculated as the pruning rate.

[0208] In the example of Fig. 26, the trust radius (the upper limit of the threshold) in the search of the (m + 1)-th iteration coincides with the error of the pruning rate "0%" of layer 1. Therefore, in the search of the (m + 1)-th iteration, a combination of pruning rates different from the previous time, i.e., the pruning rate "(layer 1, layer 2)=(0%, 0%)" is searched.

[0209] Generalizing the above formulas (8) and (9), the trust radius of the next iteration can be expressed by the following formula (10). Trust radius of the next iteration = Trust radius of the current iteration * max(constant, Qscale_min) (10)

[0210] Here, in the above formula (10), the constant is K or k, "Qscale_min" is "Qscale" represented by the following formula (11), and "Qscale" is represented by the following formula (12). Qscale_min = min(Qscale calculated for all quantization target vectors) (11) Qscale = 1 + Qdiff / Qth (12)

[0211] In the above formula (12), "Qdiff" is "the difference between the quantization error of a bit width one narrower than the temporarily calculated bit width (pruning rate) and the threshold value", and "Qth" is the threshold value.

[0212] Next, an example of the operation of server 1A according to the first modified example will be explained with reference to Figure 27. Figure 27 is a flowchart for explaining an example of the operation of processing by server 1A according to the first modified example. In Figure 27, steps S14, S16, and S17 in the flowchart for server 1 shown in Figure 22 are replaced with steps S21, S22, and S23, respectively. In the first modified example as well, the threshold calculation unit 14a' sets the initial value of the confidence radius in step S3.

[0213] In step S21, the threshold calculation unit 14a' increases the confidence radius by a constant K or the larger of the "sum of differences", and the process proceeds to step S23.

[0214] In step S22, the threshold calculation unit 14a' reduces the confidence radius by a constant k or by the larger of the "difference in differences", and the process proceeds to step S23.

[0215] In step S23, the determination unit 14b' determines whether the pruning rate 11d of all layers is "0%", in other words, whether the pruning rate satisfies a predetermined condition. If the pruning rate 11d of at least one layer is not "0%" (NO in step S23), the process proceeds to step S4.

[0216] If the pruning rate 11d for all layers is "0%" (YES in step S23), the output unit 15 outputs the determined pruning rate 11d (step S18), and the process ends.

[0217] As described above, in the first modified example, the method for updating the confidence radius by the threshold calculation unit 14a' and the termination conditions for determining the end of the search by the determination unit 14b' are different from those in one embodiment. As a result, the server 1A can search for an appropriate pruning rate to sufficiently lighten the NN in the shortest time (shortest number of attempts). In addition, the setting (specification) of the number of attempts by the designer, etc., can be omitted.

[0218] [1-6-2] Second variation In one embodiment and the first modified example, the initial value of the confidence radius is a hyperparameter set by the designer or the like.

[0219] The model size may differ even with the same number of searches depending on whether the initial value of the confidence radius is set high or low. Furthermore, when the initial value of the confidence radius is set high, the number of searches required to sufficiently reduce the model size may be greater compared to when the initial value is set low.

[0220] Thus, depending on the initial value of the confidence radius, the final model size and the number of iterations required for pruning will vary; in other words, there may be variability in the performance of servers 1 and 1A.

[0221] Therefore, in the second modification, we will explain a method for suppressing performance variations between servers 1 and 1A.

[0222] Figure 28 is a block diagram showing an example of the functional configuration of server 1B according to the second modified example. As illustrated in Figure 28, server 1B may be equipped with a calculation unit 14B that is different from server 1 in Figure 4. The calculation unit 14B may be equipped with a threshold calculation unit 14a'' and a determination unit 14b'' that are different from the calculation unit 14 in Figure 4.

[0223] In model pruning, it is known that gradually pruning the model using a small pruning rate maintains accuracy and compresses the model more effectively than pruning it all at once with a large pruning rate.

[0224] Furthermore, as shown in equation (5) above, the threshold T is set according to the reciprocal of the gradient, so a layer with a large threshold T means a layer with a small gradient. A layer with a small gradient means that pruning will have little impact on the accuracy.

[0225] Therefore, the server 1B (threshold calculation unit 14a") sets the initial value of the confidence radius to a value that minimizes the pruning rate in the first search. To this end, the threshold calculation unit 14a" may set the initial value of the confidence radius to a value such that the layer with the largest threshold T among all layers is pruned, and the remaining layers are not pruned (pruning rate of "0%").

[0226] By setting the initial value of the confidence radius as described above, Server 1B can compress the model size more or maintain accuracy compared to manually setting a larger initial value for the confidence radius.

[0227] Figure 29 illustrates an example of setting the initial value of the confidence radius. As shown in the upper part of Figure 29, if the initial value of the confidence radius is not set, the combination of pruning rates explored is "(Layer 1, Layer 2) = (10%, 20%)".

[0228] As illustrated in Figure 29, the threshold calculation unit 14a'' measures the threshold (max(Th)) of the layer with the largest threshold among all layers and the error (Error) of the smallest pruning rate (excluding "0%)" for that layer during the initial search for the pruning rate.

[0229] Th represents a vector based on the thresholds T1, T2, ... of each layer, where Th = [T1, T2] in the example in Figure 29. The threshold (max(Th)) is the threshold of the layer with the largest threshold, which is T2 in the example in Figure 29. The error is the error of the minimum pruning rate of the layer with the largest threshold, which is the error of the pruning rate of "10%" for layer 2 in the example in Figure 29.

[0230] Next, the threshold calculation unit 14a'' sets the initial value of the confidence radius according to the following equation (13) using the measured threshold and error. In the following equation (13), "||Th||2" is the L2 norm of the thresholds for all layers.

number

[0231] The threshold calculation unit 14a" sets thresholds T1 and T2 such that, based on the calculated initial value of the confidence radius, the minimum pruning rate of "10%" is selected as the pruning rate for the layer with the largest threshold (Layer 2), and the remaining layer (Layer 1) is selected as a pruning rate of "0%".

[0232] As a result, as shown in the lower part of Figure 29, once the initial value of the confidence radius is set and thresholds T1 and T2 are set, the combination of pruning rates to be explored will be "(Layer 1, Layer 2) = (0%, 10%)". The layer to be pruned (Layer 2) is the layer with the largest threshold, in other words, the layer with the smallest gradient, so the impact of pruning on accuracy can be kept to a minimum.

[0233] The functions of the threshold calculation unit 14a” other than the process of setting the initial value of the confidence radius may be the same as either or both of the threshold calculation unit 14a and the threshold calculation unit 14a' according to the first modified example. Also, the determination unit 14b” may be the same as either or both of the determination unit 14b and the determination unit 14b' according to the first modified example.

[0234] In other words, the method relating to the second modification may be realized by combining one or both of the first embodiment and the first modification.

[0235] Next, with reference to Figure 30, an example of the operation of Server 1B according to the second modified example will be described. Figure 30 is a flowchart for explaining an example of the operation of processing by Server 1B according to the second modified example. In Figure 30, step S3 is deleted from the flowchart for Server 1 shown in Figure 22, steps S31 and S32 are added between steps S4 and S5, and steps S14, S16, and S17 are replaced with steps S33, S34, and S35, respectively.

[0236] In step S31, the threshold calculation unit 14a'' determines whether it is the first search or not, after calculating the threshold for each layer in step S4. If it is not the first search (NO in step S31), the process proceeds to step S5.

[0237] If it is the first search (YES in step S31), the threshold calculation unit 14a'' sets the initial value of the confidence radius based on the threshold and minimum pruning rate error of the layer with the highest threshold (step S32), and the process proceeds to step S5.

[0238] Steps S33, S34, and S35 may be any of the steps S14, S16, and S17 shown in Figure 22, or steps S21, S22, and S23 shown in Figure 27, respectively.

[0239] As described above, in the second modified example, the method for setting the initial value of the confidence radius by the threshold calculation unit 14a'' is different from that of the first embodiment and the first modified example. As a result, server 1B can suppress fluctuations in the number of searches for the final model size and pruning rate, and can reduce variations in the performance of servers 1 and 1A.

[0240] Furthermore, Server 1B prevents manual setting of initial confidence radius values ​​(hyperparameters) by designers, etc., and can dynamically set initial confidence radius values ​​according to the layers of the machine learning model 11c. Therefore, an appropriate pruning rate can be set for each model, and fluctuations in the number of searches for the final model size and pruning rate can be suppressed regardless of the model, thereby reducing performance variability between Servers 1 and 1A.

[0241] [1-7] Hardware Configuration Examples Servers 1, 1A, and 1B in one embodiment and in the first and second modifications may be virtual machines (VMs) or physical machines, respectively. Furthermore, the functions of each of servers 1, 1A, and 1B may be implemented by one computer or by two or more computers. In addition, at least a portion of the functions of each of servers 1, 1A, and 1B may be implemented using hardware and network resources provided by a cloud environment.

[0242] Figure 31 is a block diagram showing an example of the hardware (HW) configuration of computer 10. Hereinafter, computer 10 will be used as an example to explain the hardware (HW) that implements the respective functions of servers 1, 1A, and 1B. Note that if multiple computers are used as HW resources to implement the respective functions of servers 1, 1A, and 1B, each computer may have the HW configuration illustrated in Figure 31.

[0243] As shown in Figure 31, the computer 10 may, as an example of its hardware configuration, include a processor 10a, a graphics processing unit 10b, a memory 10c, a storage unit 10d, an IF (Interface) unit 10e, an IO (Input / Output) unit 10f, and a read unit 10g.

[0244] Processor 10a is an example of an arithmetic processing unit that performs various control and calculations. Processor 10a may be connected to each block in the computer 10 via bus 10j so as to be able to communicate with each other. Processor 10a may be a multiprocessor containing multiple processors, a multicore processor having multiple processor cores, or a configuration having multiple multicore processors.

[0245] Examples of processor 10a include integrated circuits (ICs) such as CPUs, MPUs, APUs, DSPs, ASICs, and FPGAs. Note that two or more combinations of these integrated circuits may be used as processor 10a. CPU stands for Central Processing Unit, MPU for Micro Processing Unit, APU for Accelerated Processing Unit, DSP for Digital Signal Processor, ASIC for Application Specific IC, and FPGA for Field-Programmable Gate Array.

[0246] The graphics processing unit 10b controls the screen display for output devices such as monitors, which are part of the I / O unit 10f. The graphics processing unit 10b may also be configured as an accelerator that performs machine learning processing and inference processing using machine learning models. Examples of graphics processing units 10b include various arithmetic processing units, such as integrated circuits (ICs) like GPUs (Graphics Processing Units), APUs, DSPs, ASICs, or FPGAs.

[0247] For example, processor 10a may execute program 10h (machine learning program) that implements all or part of the various functions of computer 10. For example, processor 10a may implement the functions of acquisition unit 12, calculation unit 14, 14A or 14B, and output unit 15 of server 1, 1A or 1B (see Figure 4, Figure 24 or Figure 28) based on program 10h. Also, for example, graphics processing unit 10b may execute arithmetic processing used for NN calculations such as matrix operations, and may implement the functions of machine learning unit 13 of server 1, 1A or 1B (see Figure 4, Figure 24 or Figure 28).

[0248] Memory 10c is an example of hardware that stores various data and program information. Examples of memory 10c include volatile memory such as DRAM (Dynamic Random Access Memory) and non-volatile memory such as PM (Persistent Memory), or both.

[0249] The storage unit 10d is an example of hardware that stores various data and program information. Examples of storage units 10d include magnetic disk devices such as HDDs (Hard Disk Drives), semiconductor drive devices such as SSDs (Solid State Drives), and various storage devices such as non-volatile memory. Examples of non-volatile memory include flash memory, SCM (Storage Class Memory), and ROM (Read Only Memory).

[0250] Furthermore, the storage unit 10d may store the program 10h. For example, the processors 10a of servers 1, 1A, and 1B can perform the functions of the control units 16 of servers 1, 1A, and 1B (see Figures 4, 24, or 28) by loading the program 10h stored in the storage unit 10d into memory 10c and executing it.

[0251] Furthermore, the memory unit 11 illustrated in Figure 4, Figure 24, or Figure 28 may be realized by a storage area having at least one of the memory 10c and the storage unit 10d.

[0252] The IF unit 10e is an example of a communication interface that controls connections and communication with a network. For example, the IF unit 10e may include an adapter compliant with LAN (Local Area Network) such as Ethernet®, or optical communication such as FC (Fibre Channel). The adapter may support wireless, wired, or both communication methods. For example, servers 1, 1A, and 1B may be connected to a computer (not shown) via the IF unit 10e so that they can communicate with each other. The functions of one or both of the acquisition unit 12 and output unit 15 illustrated in Figures 4, 24, or 28 may be implemented by the IF unit 10e. Also, for example, the program 10h may be downloaded from the network to the computer 10 via the communication interface and stored in the storage unit 10d.

[0253] The I / O unit 10f may include one or both of an input device and / or an output device. Examples of input devices include a keyboard, mouse, and touch panel. Examples of output devices include a monitor, projector, and printer. The I / O unit 10f may also include a touch panel or the like that integrates the input and output devices. The output device may be connected to the graphics processing unit 10b. For example, the output unit 15 illustrated in Figures 4, 24, or 28 may output and display the pruning rate 11d to the output device of the I / O unit 10f.

[0254] The reading unit 10g is an example of a reader that reads data and program information recorded on the recording medium 10i. The reading unit 10g may include a connection terminal or device to which the recording medium 10i can be connected or inserted. Examples of the reading unit 10g include an adapter compliant with USB (Universal Serial Bus), a drive device for accessing a recording disk, and a card reader for accessing flash memory such as an SD card. The recording medium 10i may store a program 10h, and the reading unit 10g may read the program 10h from the recording medium 10i and store it in the storage unit 10d.

[0255] Examples of recording media 10i include non-temporary computer-readable recording media such as magnetic / optical discs and flash memory. Examples of magnetic / optical discs include flexible discs, CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray discs, and HVDs (Holographic Versatile Discs). Examples of flash memory include semiconductor memory such as USB memory and SD cards.

[0256] The hardware configuration of computer 10 described above is illustrative. Therefore, the addition or deletion of hardware within computer 10 (for example, adding or deleting arbitrary blocks), division, integration in any combination, or addition or deletion of buses may be performed as appropriate. For example, in servers 1, 1A, and 1B, at least one of the I / O unit 10f and the read unit 10g may be omitted.

[0257] [2] Others The embodiments described above, as well as the technologies relating to the first and second modifications, can be implemented with the following modifications and changes.

[0258] For example, the acquisition unit 12, machine learning unit 13, calculation unit 14, 14A, or 14B, and output unit 15 of the server 1, 1A, or 1B shown in Figure 4, Figure 24, or Figure 28 may be merged or each may be separated.

[0259] Furthermore, for example, the servers 1, 1A, or 1B shown in Figures 4, 24, or 28 may be configured such that multiple devices cooperate with each other via a network to realize their respective processing functions. For example, in servers 1, 1A, or 1B, the acquisition unit 12 and output unit 15 may be a web server and an application server, the machine learning unit 13 and calculation units 14, 14A, or 14B may be an application server, the memory unit 11 may be a DB server, etc. In this case, the web server, application server, and DB server may cooperate with each other via a network to realize the processing functions of servers 1, 1A, or 1B.

[0260] Furthermore, the method of applying zero-padding to a NN including an attention structure 160, as explained with reference to Figures 16 to 21, is not limited to its application to pruning by servers 1, 1A, or 1B shown in Figures 4, 24, or 28. For example, the method of applying zero-padding may be applied to various methods for determining the pruning rate for each layer of the NN.

[0261] [3] Addendum The following additional information is disclosed regarding the above embodiments and the first and second modifications.

[0262] (Note 1) In a trained machine learning model of a neural network equipped with an attention structure, a padding layer is inserted after each of the Q layer, which outputs a Query as a result of processing the input tensor of the attention structure, and the K layer, which outputs a Key, to perform padding of one or more elements of the tensor. Padding is performed by the padding layers associated with the reduced Q layer and the reduced K layer, respectively, so that the number of elements in the tensor QT from the Q layer after element reduction based on the first reduction ratio and the tensor KT from the K layer after element reduction based on the second reduction ratio are the same. A machine learning program that instructs a computer to perform a task.

[0263] (Note 2) The process of performing the aforementioned padding is as follows: Padding is performed such that the number of elements in all tensors except the one with the largest number of elements among the number of elements in tensor QT and the number of elements in tensor KT becomes the largest number of elements. The process includes suppressing padding to the tensor having the largest number of elements. The machine learning program described in Appendix 1.

[0264] (Note 3) If the attention structure is a multi-head attention structure, and each of the Q layer, the K layer, and the V layer which outputs a Value as a result of arithmetic processing on the input tensor in the attention structure outputs the tensor of each of the multiple heads, then the padding layer is inserted after the V layer. The computer is made to perform the process, The padding process includes performing padding by the padding layer associated with the reduced Q layer, the reduced K layer, and the reduced V layer, such that the number of heads of the tensor QT, the tensor KT, and the tensor VT from the V layer after element reduction based on the third reduction ratio match, the number of elements between identical heads in the tensor QT and the tensor KT are the same, and the number of elements between heads in the tensor VT is the same. The machine learning program described in Appendix 1 or Appendix 2.

[0265] (Note 4) The padding process includes a process of performing padding such that, for each identical head number in tensor QT and tensor KT, the number of elements in the head of The machine learning program described in Appendix 3.

[0266] (Note 5) The attention structure outputs a matrix product based on the matrix product obtained by normalizing the matrix product of the padded tensor QT and the padded tensor KT, and the padded tensor VT. The machine learning program described in Appendix 3 or Appendix 4.

[0267] (Note 6) The neural network includes a coupling unit that outputs the result of combining the elements of the matrix product output from the attention structure. The machine learning program described in Appendix 5.

[0268] (Note 7) The aforementioned padding layer is a layer that performs zero padding by inserting a zero matrix into the input tensor. A machine learning program described in any one of the following items from Appendix 1 to Appendix 6.

[0269] (Note 8) In a trained machine learning model of a neural network equipped with an attention structure, a padding layer is inserted after each of the Q layer, which outputs a Query as a result of processing the input tensor of the attention structure, and the K layer, which outputs a Key, to perform padding of one or more elements of the tensor. Padding is performed by the padding layers associated with the reduced Q layer and the reduced K layer, respectively, so that the number of elements in the tensor QT from the Q layer after element reduction based on the first reduction ratio and the tensor KT from the K layer after element reduction based on the second reduction ratio are the same. A machine learning method in which a computer performs a process.

[0270] (Note 9) The process of performing the aforementioned padding is as follows: Padding is performed such that the number of elements in all tensors except the one with the largest number of elements among the number of elements in tensor QT and the number of elements in tensor KT becomes the largest number of elements. The process includes suppressing padding to the tensor having the largest number of elements. The machine learning method described in Appendix 8.

[0271] (Note 10) If the attention structure is a multi-head attention structure, and each of the Q layer, the K layer, and the V layer which outputs a Value as a result of arithmetic processing on the input tensor in the attention structure outputs the tensor of each of the multiple heads, then the padding layer is inserted after the V layer. The computer performs the processing, The padding process includes performing padding by the padding layer associated with the reduced Q layer, the reduced K layer, and the reduced V layer, such that the number of heads of the tensor QT, the tensor KT, and the tensor VT from the V layer after element reduction based on the third reduction ratio match, the number of elements between identical heads in the tensor QT and the tensor KT are the same, and the number of elements between heads in the tensor VT is the same. The machine learning method described in Appendix 8 or Appendix 9.

[0272] (Note 11) The padding process includes a process of performing padding such that, for each identical head number in tensor QT and tensor KT, the number of elements in the head of The machine learning method described in Appendix 10.

[0273] (Note 12) The attention structure outputs a matrix product based on the matrix product obtained by normalizing the matrix product of the padded tensor QT and the padded tensor KT, and the padded tensor VT. The machine learning method described in Appendix 10 or Appendix 11.

[0274] (Note 13) The neural network includes a coupling unit that outputs the result of combining the elements of the matrix product output from the attention structure. The machine learning method described in Appendix 12.

[0275] (Note 14) The aforementioned padding layer is a layer that performs zero padding by inserting a zero matrix into the input tensor. A machine learning method described in any one of the items in Appendix 8 to Appendix 13.

[0276] (Note 15) In a trained machine learning model of a neural network equipped with an attention structure, a padding layer is inserted after each of the Q layer, which outputs a Query as a result of processing the input tensor of the attention structure, and the K layer, which outputs a Key, to perform padding of one or more elements of the tensor. Padding is performed by the padding layers associated with the reduced Q layer and the reduced K layer, respectively, so that the number of elements in the tensor QT from the Q layer after element reduction based on the first reduction ratio and the tensor KT from the K layer after element reduction based on the second reduction ratio are the same. An information processing device equipped with a control unit.

[0277] (Note 16) The control unit, in the process of performing the padding, Padding is performed such that the number of elements in all tensors except the one with the largest number of elements among the number of elements in tensor QT and the number of elements in tensor KT becomes the largest number of elements. To suppress padding to the tensor having the largest number of elements, The information processing device described in Appendix 15.

[0278] (Note 17) The control unit, If the attention structure is a multi-head attention structure, and each of the Q layer, the K layer, and the V layer which outputs a Value as a result of arithmetic processing on the input tensor in the attention structure outputs the respective tensors of multiple heads, then the padding layer is inserted after the V layer. In the process of performing the aforementioned padding, the padding is performed by the padding layer associated with the reduced Q layer, the reduced K layer, and the reduced V layer, such that the number of heads of the tensor QT, the tensor KT, and the tensor VT from the V layer after the reduction of elements based on the third reduction ratio are the same, the number of elements between the same heads in the tensor QT and the tensor KT are the same, and the number of elements between the heads of the tensor VT are the same. The information processing device described in Appendix 15 or Appendix 16.

[0279] (Note 18) The control unit, in the process of performing the padding, performs padding such that for each identical head number in tensor QT and tensor KT, the number of elements of the head of the head of the head of the head of the head of the head of the head of the head of the head of the tensor KT is the same as the number of elements of the head of the head of the head of the head of the head of the tensor KT. The information processing device described in Appendix 17.

[0280] (Note 19) The attention structure outputs a matrix product based on the matrix product obtained by normalizing the matrix product of the padded tensor QT and the padded tensor KT, and the padded tensor VT. The information processing device described in Appendix 17 or Appendix 18.

[0281] (Note 20) The neural network includes a coupling unit that outputs the result of combining the elements of the matrix product output from the attention structure. The information processing device described in Appendix 19. [Explanation of symbols]

[0282] Servers 1, 1A, and 1B 10 Computers 11 Memory section 11a Untrained Model 11b Data for machine learning 11c Machine Learning Pre-Models 11d pruning rate 11e Lightweight Model 12 Acquisition Department 13 Machine Learning Department 14, 14A, 14B Pruning rate calculation unit (calculation unit) 14a, 14a', 14a'' Threshold calculation unit 14b, 14b', 14b” decision section 15 Output section 16 Control Unit

Claims

1. In a trained machine learning model of a neural network having an attention structure, a padding layer is inserted after each of the Q layer and K layer, which output Query and Key respectively as the result of processing the input tensor of the attention structure, to perform padding of one or more elements of the tensor. Padding is performed by the padding layers associated with the reduced Q layer and the reduced K layer, respectively, so that the number of elements in the tensor QT from the Q layer after element reduction based on the first reduction ratio and the tensor KT from the K layer after element reduction based on the second reduction ratio are the same. A machine learning program that instructs a computer to perform a task.

2. The process of performing the aforementioned padding is as follows: Padding is performed such that the number of elements in all tensors except the one with the largest number of elements among the number of elements in tensor QT and the number of elements in tensor KT becomes the maximum number of elements. The process includes suppressing padding to the tensor having the largest number of elements. The machine learning program according to claim 1.

3. In the case where the attention structure is a multi-head attention structure, and each of the Q layer, the K layer, and the V layer which outputs a Value as a result of arithmetic processing on the input tensor in the attention structure outputs the respective tensors of multiple heads, the padding layer is inserted after the V layer. The computer is made to perform the process, The padding process includes performing padding by the padding layer associated with the reduced Q layer, the reduced K layer, and the reduced V layer, such that the number of heads of the tensor QT, the tensor KT, and the tensor VT from the V layer after the reduction of elements based on the third reduction ratio match, the number of elements between the same heads in the tensor QT and the tensor KT are the same, and the number of elements between the heads of the tensor VT is the same. The machine learning program according to claim 1.

4. The attention structure outputs a matrix product based on the matrix product obtained by normalizing the matrix product of the padded tensor QT and the padded tensor KT, and the padded tensor VT. The machine learning program according to claim 3.

5. The neural network includes a coupling unit that outputs the result of combining the elements of the matrix product output from the attention structure. The machine learning program according to claim 4.

6. The aforementioned padding layer is a layer that performs zero padding by inserting a zero matrix into the input tensor. A machine learning program according to any one of claims 1 to 5.

7. In a trained machine learning model of a neural network having an attention structure, a padding layer is inserted after each of the Q layer and K layer, which output Query and Key respectively as the result of processing the input tensor of the attention structure, to perform padding of one or more elements of the tensor. Padding is performed by the padding layers associated with the reduced Q layer and the reduced K layer, respectively, so that the number of elements in the tensor QT from the Q layer after element reduction based on the first reduction ratio and the tensor KT from the K layer after element reduction based on the second reduction ratio are the same. A machine learning method in which a computer performs a process.

8. In a trained machine learning model of a neural network having an attention structure, a padding layer is inserted after each of the Q layer and K layer, which output Query and Key respectively as the result of processing the input tensor of the attention structure, to perform padding of one or more elements of the tensor. Padding is performed by the padding layers associated with the reduced Q layer and the reduced K layer, respectively, so that the number of elements in the tensor QT from the Q layer after element reduction based on the first reduction ratio and the tensor KT from the K layer after element reduction based on the second reduction ratio are the same. An information processing device equipped with a control unit.

Images