Speed-up program, speed-up method, and information processing device

Abstract

To realize reduction of time for completion of learning and increase of the accuracy at the same time.SOLUTION: The information processor calculates dispersion of weight information about a weight updated by a mechanical learning for each layer of a mechanical learning model at predetermined intervals when the mechanical learning model does the mechanical learning. The information processor determines a layer as a suppression target for which the mechanical learning is suppressed, on the basis of the peak value of dispersion calculated at predetermined intervals and the dispersion of the weight information calculated at predetermined intervals.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a speed-up program, a speed-up method, and an information processing device. [Background technology]

[0002] To speed up the machine learning of a machine learning model, it is effective to use a GPU (Graphics Processing Unit), and it is also effective to distribute the processing among multiple GPUs. Up until now, speed-up has been achieved by implementing multiple GPUs in a node, which is an information processing device, and executing machine learning processing in parallel within the node, but it took time to aggregate and reflect gradient information between GPUs.

[0003] In machine learning for such machine learning models, the new learning portion needs to be updated frequently each time it learns, so the learning rate (LR), which determines the update frequency, needs to be set high. On the other hand, the learning rate of the existing learning portion, where machine learning has already been completed, is lower the closer it is to the input side, and in extreme cases the learning rate is often 0. Although the portion with a learning rate of 0 does not need to undergo machine learning processing, it still undergoes the same frequency of gradient information aggregation and reflection processing and weight calculation processing as the new learning portion, resulting in a lot of unnecessary processing.

[0004] For this reason, in recent years, Gradient Skip technology has come into use, which identifies layers that do not require machine learning and skips them without calculating gradient information (Δw) or performing aggregation processing (Allreduce processing). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special Publication No. 2018-520404 [Patent Document 2] Japanese Patent Application Publication No. 10-198645 [Patent Document 3] U.S. Patent No. 6,119,112 Summary of the Invention [Problem to be solved by the invention]

[0006] However, while the above technology can speed up the process by skipping machine learning, depending on the layer and timing of the skip, the accuracy of the machine learning may deteriorate, and the machine learning may terminate without reaching the target accuracy.

[0007] In one aspect, an object of the present invention is to provide a speed-up program, a speed-up method, and an information processing device that can achieve both a reduction in the time required for learning to converge and an improvement in accuracy. [Means for solving the problem]

[0008] In a first proposal, the speed-up program is characterized in that it causes a computer to execute a process of calculating, at a predetermined interval during machine learning of a machine learning model, a variance of weight information regarding weights updated by machine learning for each layer of the machine learning model, and determining a layer to be suppressed for suppressing the machine learning based on the peak value of the variance calculated at the predetermined interval and the variance of the weight information calculated at the predetermined interval. Effect of the Invention

[0009] According to one embodiment, it is possible to achieve both a reduction in the time required for learning to converge and an improvement in accuracy. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating distributed learning of the information processing device according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a reference technique for machine learning. [Diagram 3] FIG. 3 is a diagram illustrating an example of a process of skipping learning in machine learning. [Figure 4]FIG. 4 is a diagram for explaining the learning skip using the braking distance of the learning rate. [Diagram 5] FIG. 5 is a diagram for explaining calculation of the judgment index. [Figure 6] FIG. 6 is a diagram for explaining skip determination for each layer. [Figure 7] FIG. 7 is a diagram illustrating the process of the information processing device according to the first embodiment. [Figure 8] FIG. 8 is a functional block diagram of the information processing device according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating the details of the distributed processing unit. [Figure 10] FIG. 10 is a diagram for explaining the setting of the threshold value. [Figure 11] FIG. 11 is a flowchart of the process according to the first embodiment. [Figure 12] FIG. 12 is a flowchart showing the flow of the error backpropagation process. [Figure 13] FIG. 13 is a diagram illustrating the effect on layers following the stopped layer. [Figure 14] FIG. 14 is a diagram for explaining the influence on subsequent layers when multiple learning skips are performed. [Figure 15] FIG. 15 is a diagram for explaining improvements in the threshold determination timing. [Figure 16] FIG. 16 is a diagram for explaining an improvement in the threshold determination timing. [Figure 17] FIG. 17 is a diagram for explaining the implementation of the distribution calculation in accordance with the BD period. [Figure 18] FIG. 18 is a diagram for explaining the implementation of learning skip for multiple layers. [Figure 19] FIG. 19 is a diagram for explaining the implementation of BD control for multiple layers. [Figure 20] FIG. 20 is a diagram illustrating an example of a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments of the speed-up program, the speed-up method, and the information processing device disclosed in the present application will be described in detail with reference to the drawings. Note that the present invention is not limited to these embodiments. In addition, each embodiment can be appropriately combined within a range that does not cause inconsistency. EXAMPLES

[0012] [Overall configuration] The information processing device 10 according to the first embodiment executes the generation of a machine learning model by distributed processing using a plurality of GPUs in order to realize high-speed machine learning processing. FIG. 1 is a diagram for explaining distributed learning of the information processing device 10 according to the first embodiment. As shown in FIG. 1, the information processing device 10 is an example of a computer having a CPU (Central Processing Unit) and a plurality of GPUs. The CPU executes control over the entire information processing device 10 and integrally manages the generation of the machine learning model. Each GPU executes machine learning of the machine learning model in response to an instruction from the CPU. A known method can be adopted as the distributed processing method.

[0013] In recent years, the Gradient Skip technique has come into use, which uses the learning rate of each layer to identify layers that do not require machine learning, and suppresses (skips) learning without calculating gradient information or performing aggregation processing (Allreduce processing).

[0014] Here, we will explain a reference technology for learning skip (hereinafter, sometimes simply referred to as "skip"). Figure 2 is a diagram explaining a reference technology for machine learning. The reference technology shown in Figure 2 is a technology for generating a machine learning model by deep learning (DL) or the like, and shortens the learning time by acquiring the learning status from the error gradient of DL and skipping the calculation of the error gradient.

[0015] Specifically, the reference technology detects layers with a reduced learning rate, which indicates the progress of learning, and omits learning for the layers, thereby shortening the learning time. For example, for each layer in which the difference between the error gradient at the latest iteration and the error gradient at the immediately preceding iteration is equal to or greater than a threshold, learning is performed as normal in the next iteration, and for each layer in which the difference is less than the threshold, learning is skipped in the next iteration. In other words, for layers in which the learning rate has decreased, subsequent machine learning processes such as calculation of error gradients are suppressed.

[0016] However, in the reference technology, there are some unknowns about the impact of accuracy degradation when machine learning is completely skipped. In other words, in machine learning models such as DNN (Deep Neural Network), accuracy is likely to decrease if the error backpropagation process of multiple layers is judged based on the learning rate (LR) or other criteria and stopped all at once. In addition, if the timing of skipping learning (number of epochs, number of iterations) is poor, accuracy may decrease and the final accuracy may not reach the target.

[0017] Therefore, in recent years, a method has become known in which, rather than suddenly skipping learning for a layer determined to be a learning skip target that suppresses the calculation of error gradients and the various processes of error backpropagation, the learning rate of the layer for which learning is to be stopped is gradually reduced and a certain amount of learning processing is performed before skipping learning.

[0018] Here, an example of learning skip will be described. FIG. 3 is a diagram for explaining an example of the process of learning skip in machine learning. As shown in FIG. 3, in deep learning of a machine learning model, machine learning (calculation process) by forward propagation and update process of weights, etc. by error backpropagation are executed. Here, when updating by error backpropagation, the update of weight information is stopped from an iteration where learning has progressed to a certain extent, and at that time, the layers to be stopped are stopped from the input side. This is because, when the output side is stopped, the learning accuracy may not reach the target accuracy, but the input side has little effect on the accuracy.

[0019] The final accuracy of machine learning using the above-mentioned learning skip tends to decrease less if the learning rate of the layer where learning is stopped is stopped after it has become relatively small. For this reason, rather than suddenly stopping learning of the target layer, the introduction of a braking distance (BD: BRAKING_DISTANCE) for the learning rate is being promoted, which lowers the learning rate when a command to stop (skip) learning is given. In other words, machine learning is being used that reduces the layers that are stopped, starting from the layers closest to the bottom, to a familiar local solution.

[0020] Here, we will explain an example in which a learning rate braking distance (BD: BRAKING_DISTANCE), which is a hyperparameter, is introduced to gradually suppress learning for each layer belonging to a block of skip candidates. Note that a block is a group of multiple layers.

[0021] Fig. 4 is a diagram explaining learning skip using the braking distance of the learning rate. As shown in Fig. 4, the information processing device performing machine learning divides each layer into a first block, a second block, a third block, a fourth block, and a fifth block in order from the input side. Then, after the warm-up is completed, the information processing device controls the learning rate.

[0022] Then, when the first block is determined to be a learning skip target, the information processing device executes machine learning with a learning rate significantly lowered for each iteration than normal learning. After that, when the second block is determined to be a learning skip target, the information processing device executes machine learning with a learning rate significantly lowered for each iteration than normal learning. In this way, the information processing device executes learning skip in order from the block closest to the input layer.

[0023] In deciding whether to skip learning, the last updated value of the weights W for each layer, "W l+1 -W l " is often used. l is the weight in the lth learning. Figure 5 is a diagram explaining the calculation of the judgment index. The final updated value "Wl+1 -W l " is used as the threshold, the scalar value of the last updated value is calculated using the norm calculation formula shown in FIG. 5, and the judgment index is calculated. After that, by setting the epoch (corresponding to iteration, time) on the horizontal axis and the above scalar value on the vertical axis, it can be expressed as shown in FIG. 6. FIG. 6 is a diagram explaining the skip judgment of each layer. In FIG. 6, L6 indicates the 6th layer from the top, L36 indicates the 36th layer, L75 indicates the 75th layer, and L132 indicates the 132nd layer. Here, in the learning skip using the BD described above, a threshold is set for each layer from the input side to the output side according to the tendency of the judgment index shown in FIG. 6, and the start of the learning skip is judged.

[0024] However, there are the following problems. First, it is necessary to set a threshold for each layer. Second, the threshold is set from the weight (W) and weight gradient (ΔW) themselves, but since the tendency of the judgment index varies greatly depending on the settings of the learning rate (LR) and weight decay (Weight-Decay), the learning skip start threshold is optimized after optimizing the hyperparameters LR and Weight-Decay. In other words, it takes a lot of time and is difficult to automate.

[0025] For these reasons, there is a strong need for a method for setting the learning skip start threshold that does not depend on the weight (W) or weight gradient (ΔW) themselves.

[0026] Therefore, the information processing device 10 according to the first embodiment calculates the variance of the weights or weight gradients updated by machine learning for each layer of the machine learning model at a predetermined interval during machine learning of the machine learning model. Then, the information processing device 10 determines a layer to be suppressed for suppressing the machine learning based on the peak value of the variance calculated at the predetermined interval and the variance calculated at the predetermined interval.

[0027] For example, the information processing device 10 calculates the variance of the weight (W) and weight gradient (ΔW) for each layer, and determines that a layer in which the variance has dropped by more than a threshold value from the time-direction (epoch or iteration) peak is eligible for learning skipping, and executes learning skipping or learning skipping using BD.

[0028] Fig. 7 is a diagram for explaining the processing of the information processing device 10 according to the first embodiment. Fig. 7 shows the relationship between the learning progress and the change in the histogram of the weight, which is tensor data having a plurality of elements (hereinafter, may be simply referred to as "tensor"), for the weight of layer 33 (L33). As shown in Fig. 7, before the start of learning (epoch 0), each element of the tensor of weight (W) is a constant value, in the middle of learning (epoch 40), the variance of each element of the tensor of weight (W) is large, and in the final stage of learning (epoch 90), the variance of each element of the tensor of weight (W) is small and the distribution is close to the median.

[0029] In other words, when displaying a histogram of each element of the weight tensor for a certain layer, the variance of the weight (W) approaches the median as the iterations proceed. Therefore, if the variance becomes small to a certain extent, it can be determined that learning is complete. On the other hand, skipping learning during the warm-up process (for example, from 0 to 3850 iterations) performed in the early stages of machine learning can slow down the convergence of accuracy, and skipping learning during warm-up can have a more adverse effect on accuracy than the length of the skip.

[0030] Taking these factors into consideration, the information processing device 10 determines the timing of skipping learning by setting a threshold for each layer of the machine learning model as a rate of decline from the peak of the variance of each element of the weight tensor after the warm-up process has elapsed. In this way, the information processing device 10 can realize a method of setting a threshold for starting learning skip that does not depend on the weight (W) or weight gradient (ΔW) itself, and can achieve both a reduction in the time until learning convergence and an improvement in accuracy.

[0031] [Function configuration] 8 is a functional block diagram showing a functional configuration of the information processing device 10 according to the embodiment 1. As shown in FIG. 8, the information processing device 10 includes a communication unit 11, a storage unit 12, an integrated processing unit 20, and a plurality of distributed processing units 30.

[0032] The communication unit 11 is a processing unit that controls communication with other devices, and is realized by, for example, a communication interface, etc. For example, the communication unit 11 transmits and receives various data and various instructions to and from an administrator terminal.

[0033] The storage unit 12 is a processing unit that stores various data and various programs, and is realized by, for example, a memory, a hard disk, etc. The storage unit 12 stores a training data DB 13 and a machine learning model 14.

[0034] The training data DB 13 is an example of a data set that stores training data used for machine learning of the machine learning model 14. For example, each training data stored in the training data DB 13 is composed of image data and teacher labels. Note that the data set of training data can be divided into subsets (batch sizes) in any unit.

[0035] The machine learning model 14 is a model generated by machine learning such as DL, and is an example of a model using a multi-layer neural network composed of multiple layers. For example, when image data is input, the machine learning model 14 classifies the animals in the image. Note that the machine learning model 14 may employ a DNN or a CNN (Convolutional Neural Network), etc.

[0036] The integrated processing unit 20 is a processing unit that controls the entire information processing device 10, and is realized by, for example, a CPU. For example, the integrated processing unit 20 instructs each distributed processing unit 30 on distributed processing of machine learning, starting and ending machine learning, and the like, and executes overall control related to machine learning.

[0037] Each distributed processing unit 30 is a processing unit that executes distributed processing of machine learning, and is realized by, for example, a GPU. For example, each distributed processing unit 30 executes generation of a machine learning model 14 by machine learning using each training data stored in the training data DB 13 in response to an instruction from the integrated processing unit 20.

[0038] [Details of the distributed processing section] Next, details of each distributed processing unit 30 will be described. Each distributed processing unit 30 has the same configuration. Fig. 9 is a diagram for explaining details of the distributed processing unit 30. As shown in Fig. 9, the distributed processing unit 30 has a forward propagation processing unit 31, an error back propagation processing unit 32, an information management unit 35, and a variance calculation unit 36.

[0039] The forward propagation processing unit 31 is a processing unit that executes forward propagation processing for each layer of the machine learning model 14. Specifically, the forward propagation processing unit 31 executes so-called forward propagation, and detailed description thereof will be omitted. Briefly, for example, the forward propagation processing unit 31 inputs image data, which is training data, to the first layer (input layer) of the machine learning model 14, and obtains a prediction result (classification result), which is a result of continuous numerical calculations performed from the input layer to the last layer (output layer) of the machine learning model 14, from the output layer. Then, the forward propagation processing unit 31 calculates the error between the prediction result and the teacher level using squared error or the like, calculates an error function, and outputs the error function to the error backpropagation processing unit 32.

[0040] The backpropagation processor 32 includes an error gradient calculator 33 and a communication controller 34, and is a processor that calculates the error of each parameter of the machine learning model 14 by the backpropagation method using the error function input from the forward propagation processor 31, and executes parameter update. That is, the backpropagation processor 32 executes so-called backward propagation.

[0041] For example, the error backpropagation processor 32 calculates the error gradient of the edge weight between each node of the neural network in the order (reverse order) from the output layer to the input layer of the machine learning model 14. When the error is regarded as a function of the weight, the error gradient corresponds to a value obtained by partially differentiating the error with respect to the weight, and represents the amount of change in error when the edge weight is changed by a small amount. Then, the error backpropagation processor 32 uses the error gradient to update each parameter such as the weight of each edge so as to reduce the error.

[0042] The error gradient calculation unit 33 is a processing unit that calculates, for each of a plurality of layers included in the machine learning model 14, an error gradient indicating the gradient of the error with respect to each parameter of the machine learning model 14. For example, the error gradient calculation unit 33 calculates the error gradient (g), weight (W), and momentum (m) for each layer of the machine learning model 14 for each iteration, and outputs this information to the information management unit 35.

[0043] Here, when calculating the error gradient, the error gradient calculation unit 33 suppresses the calculation of the error gradient for a layer (learning skip layer) for which it has been determined that learning is to be suppressed. The error gradient calculation unit 33 can also arbitrarily set the error gradient calculation target, such as setting only a specific layer as the error gradient calculation target or setting the convolution layer as the error gradient calculation target. Note that various known methods can be adopted as the calculation method for the error gradient, etc.

[0044] The communication control unit 34 is a processing unit that executes AllReduce communication between GPUs. For example, the communication control unit 34 transmits and receives error gradients between the GPUs, summing up the error gradients calculated by the multiple GPUs for each edge weight, and the error gradients are aggregated between the multiple GPUs. Using information about the error gradients thus aggregated, the error backpropagation processing unit 32 updates various parameters of the machine learning model 14.

[0045] The communication control unit 34 also stops communication to a layer to be skipped in accordance with a control instruction from the variance calculation unit 36, which will be described later. The communication control unit 34 also identifies, from among the layers of the machine learning model 14, a layer in which learning is to be continued without stopping error gradient calculation and communication (Allreduce) and a layer in which learning is to be stopped, and controls communication.

[0046] The information management unit 35 is a processing unit that acquires and manages the error gradient (g), weight (W), and momentum (m) from the error gradient calculation unit 33. For example, the information management unit 35 outputs the weight (W) of each layer acquired from the error gradient calculation unit 33 to the variance calculation unit 36.

[0047] The variance calculation unit 36 ​​is a processing unit that calculates the variance of weights updated by machine learning for each layer of the machine learning model 14 at predetermined intervals during machine learning of the machine learning model 14, and determines the layer to be suppressed for suppressing the machine learning based on the peak value of the variance calculated at the predetermined interval and the variance calculated at the predetermined interval.

[0048] Specifically, the variance calculation unit 36 ​​sets the layer that is closest to the input layer among the layers in which the learning skip has not been executed, in the order from the input layer to the output layer, as the target layer that is the current target of the learning skip judgment. After that, the variance calculation unit 36 ​​executes the calculation of the weight variance and the learning skip judgment for each layer.

[0049] For example, after the form-up process is completed, the variance calculation unit 36 ​​calculates the variance of each element of the tensor of the weight (W) notified by the information management unit 35 for each iteration. Next, the variance calculation unit 36 ​​identifies a peak value using the variance of the weights that are continually calculated. After identifying the peak value, the variance calculation unit 36 ​​calculates the difference between the peak value and the variance of the weights that are continually calculated, and when the difference is equal to or greater than a threshold, determines that the current target layer is to be skipped in learning. Then, the variance calculation unit 36 ​​notifies the communication control unit 34 of the determination result to execute the learning skip, and also determines the next target layer and repeats the determination.

[0050] Here, the setting of the threshold will be described. Specifically, the variance calculation unit 36 ​​sets the threshold according to the change in the variance of the weight. For example, there are three threshold setting patterns: the threshold for the first layer (L0), the threshold when the waveform of the variance change is mountain-shaped, and the threshold when the waveform of the variance change is downhill. The variance calculation unit 36 ​​sets the threshold for each pattern based on the degree of decline from the peak.

[0051] Fig. 10 is a diagram for explaining the setting of the threshold. For the first layer of the machine learning model 14 shown in Fig. 10(a), the variance calculation unit 36 ​​sets the threshold to, for example, a value 6% lower than the peak. For example, when the peak of the variance is "1.0", the variance calculation unit 36 ​​sets the threshold to "0.94".

[0052] 10(b), the variance calculation unit 36 ​​sets the threshold value to, for example, a value 4% below the peak. For example, when the peak of the variance is "1.0", the variance calculation unit 36 ​​sets the threshold value to "0.96".

[0053] 10(b), the variance calculation unit 36 ​​sets the threshold value to, for example, a value 2% below the peak. For example, when the peak of the variance is "1.0", the variance calculation unit 36 ​​sets the threshold value to the variance of "0.98".

[0054] This is because the degree of decline from the peak differs, and the variance calculation unit 36 ​​sets the degree of decline so that, for example, downhill < mountain-shaped < first layer. In addition, the variance calculation unit 36 ​​can determine a mountain-shaped pattern when the continuously calculated variance changes by more than a predetermined value between predetermined iterations, and determine other cases as a downhill pattern.

[0055] Furthermore, for a layer determined to be a learning skip target, the variance calculation unit 36 ​​can not only simply skip learning, but also gradually stop learning using the braking distance (BD) of the learning rate.

[0056] For example, the variance calculation unit 36 ​​does not suddenly stop learning of the layer that is a skip candidate, but when a command to stop learning is given, the variance calculation unit 36 ​​reduces the learning rate using a BD that depends on the iteration and then suppresses it. More specifically, when the LR scheduler used in learning the machine learning model 14 is POW2, the variance calculation unit 36 ​​uses formula (1) to reduce BD in the same way as POW2.

[0057]

number

[0058] The variance calculation unit 36 ​​multiplies BD, which is the braking distance of LR, by the attenuation rate shown in formula (1) to gradually reduce it for each iteration up to the set number of iterations, 7700, thereby decreasing the learning rate. Note that BD in formula (1) is a preset value, and iteration is the number of iterations at the time of calculation.

[0059] Here, an example of executing learning skip using the above decay rate for a layer determined to be a learning skip candidate will be described. When warm-up (3850 iterations) is completed, the variance calculation unit 36 ​​executes machine learning with LR=5. Then, when the iteration is determined to be a learning skip candidate when iteration is 7980, the variance calculation unit 36 ​​calculates the LR at that iteration using formula (2) and executes machine learning using the calculated LR. In this way, the variance calculation unit 36 ​​calculates the LR for each iteration and executes machine learning using the calculated LR.

[0060]

number

[0061] In addition, "LR" in formula (2) is the learning rate to be calculated and used for learning. "End LR" is the LR when it is determined that learning should be skipped, and the learning rate decay (decrease) is repeated until it reaches this "End LR". "LR at the start of BD" is the LR at the time of initial setting. "Iter." is the number of iterations at the time of calculation. After it is determined to be a skip candidate, "LR" is calculated for each iteration, so this is the number of iterations. "Iter." at the start of BD" is the number of iterations when the decay of the learning rate starts. For example, BD = 7700 iterations, warm-up = 3850 iterations, the initial value (Base LR) corresponding to "LR at the start of BD" = 5, the final LR (End LR) = 0.0001, and "iter." at the start of BD" = 7980 iterations.

[0062] As described above, the variance calculation unit 36 ​​does not suddenly stop learning of the learning skip candidate layer, but gradually lowers the learning rate using BD that depends on the iteration, and skips learning after the target learning rate is reached. At this time, the variance calculation unit 36 ​​can improve the learning accuracy and ultimately reduce the number of epochs by skipping learning in order from the layer closest to the input layer.

[0063] [Processing flow] Fig. 11 is a flowchart showing a process flow according to the embodiment 1. As shown in Fig. 11, when an instruction to start machine learning is received (S101: Yes), the forward propagation processing unit 31 reads training data (S102) and executes a forward propagation process (S103).

[0064] Next, the error backpropagation processing unit 32, the information management unit 35, and the variance calculation unit 36 ​​execute the error backpropagation process (S104). Thereafter, if the machine learning is to be continued (S105: No), S102 and subsequent steps are executed, and if the machine learning is to be ended (S105: Yes), the information processing device 10 ends the machine learning process.

[0065] (Flow of backpropagation process) Fig. 12 is a flowchart showing the flow of the error backpropagation process. As shown in Fig. 12, the variance calculation unit 36 ​​determines whether the target layer n, which is the target of the learning skip determination for stopping the error calculation and the error backpropagation, is not set (S201), and if it is not set (S201: Yes), sets the target layer n, which is the target of the learning skip determination (S202).

[0066] If it has already been set (S201: No) or if the target layer n has been set (S202), the variance calculation unit 36 ​​determines whether it is time to determine the learning progress (S203). For example, the variance calculation unit 36 ​​determines that it is time to determine the learning progress when the warm-up process has been completed and the iterations have been completed.

[0067] Here, if the variance calculation unit 36 ​​determines that it is not time to determine the learning progress (S203: No), the error backpropagation processing unit 32 executes normal error backpropagation processing (S204).

[0068] On the other hand, when it is determined that it is time to determine the learning progress (S203: Yes), the variance calculation unit 36 ​​calculates the variance V(n) of the weight (W) of the target layer n (S205). 2 is the expected value of the squared difference between X and the population mean, E((X-μ) 2 ) is calculated as the variance of X.

[0069] Next, the variance calculation unit 36 ​​obtains the peak of the variance V(n) (S206). For example, the variance calculation unit 36 ​​calculates the weight variance value after a certain iteration (here, 5000) and calculates V i Let (n) be the variance value of the nth layer of the learning progress timing i, and V max (n). Here, the variance calculation unit 36 ​​calculates "V i (n)-, V i-1 (n)" is less than 0 ("V i (n)-, V i-1 If the state where "V(n)"<0) continues for a predetermined number of times, the "downward slope threshold" is set. i(n)-, V i-1 (n)" is greater than 0 (V i (n)-, V i-1 (n)" > 0) to less than 0 ("V i (n)-, V i-1 If "(n)" changes to "<0", set the "mountain threshold".

[0070] Thereafter, the variance calculation unit 36 ​​judges whether the variance V(n) has fallen from its peak to below the threshold value T% (S207). Here, if the variance V(n) has fallen from its peak to below the threshold value T% (S207: Yes), the variance calculation unit 36 ​​determines that the nth layer has reached the threshold and therefore to skip learning (S208). On the other hand, if the variance V(n) has not fallen from its peak to below the threshold value T% (S207: No), the variance calculation unit 36 ​​executes normal backpropagation processing since the nth layer has not reached the threshold (S204). For example, the variance calculation unit 36 ​​may determine that "(V i (n) / V max (n)<threshold value, it is determined that learning should be skipped.

[0071] [effect] As described above, the information processing device 10 focuses on the fact that the variation in the weight W decreases as learning progresses, and uses the weight (W) and weight slope (ΔW) to determine the learning progress, and determines whether or not to skip learning based on the rate of decrease in the variance value from the peak. As a result, the hyperparameters are also only variance values, making it possible to identify thresholds in a short time and automating the process easy. For example, even in a case where parameters are usually determined by 60 trials, experimental results have been obtained that show that the parameters can be determined by about 5 trials by using the method of the first embodiment. EXAMPLES

[0072] Now, when learning is stopped for a certain layer by skipping learning, we will consider the effect of skipping learning on the next layer. Figure 13 is a diagram explaining the effect on the layer after the stopped layer. Figure 13 shows a histogram of the weight distribution of the 33rd layer to consider the effect on the 33rd layer immediately after skipping learning up to the 30th layer in the Conv (Convolution) layer.

[0073] As shown in (a) of Figure 13, when L30 is not skipped in learning, there is no particular impact on L33 because of the normal backpropagation process. On the other hand, as shown in (b) of Figure 13, when L30 is skipped in learning, the variance of the weight (W) of L33 rises in a mountain shape. In other words, as shown by the arrow on the heat map of the weight histogram, it can be confirmed that the variance begins to spread from the point where learning was skipped.

[0074] Fig. 14 is a diagram for explaining the influence on subsequent layers when multiple learning skips are performed. Fig. 14 shows a histogram of the weight distribution of the 72nd layer to examine the influence of the 72nd layer immediately after learning skipping up to the 69th layer in the Conv layer.

[0075] As shown in (a) of Figure 14, when L69 is not skipped in learning, there is no particular impact on L72 due to normal error backpropagation processing. Also, as shown in (b) of Figure 14, when L69 is skipped in learning once, it can be confirmed that the spread is seen from around the time of the learning skip, as indicated by the arrows. Also, as shown in (c) of Figure 14, when L69 is skipped in learning twice, it can be confirmed that the spread is seen from around the time of the learning skip, both the first and second times, as indicated by the arrows. In other words, although the weight variance is hardly affected, the spread from around the time of the learning skip can be confirmed in the heat map of the weight histogram.

[0076] As described above, the variance spreads once in the layer immediately following the layer where the learning skip was executed, so that the method of the embodiment 1 may cause a misidentification of a peak or a learning skip in a state where learning is insufficient. For this reason, the timing of threshold judgment is improved in order not to reduce the final learning accuracy after the learning skip.

[0077] Fig. 15 is a diagram for explaining improvements in threshold determination timing. Fig. 15 explains an example of learning skip control using the BD period, but the same applies to normal learning skip. In addition, the sampling of the weight distribution calculation is set to 200 iterations, and the BD period is set to 1000 to 3000 iterations.

[0078] As shown in Fig. 15, the variance calculation unit 36 ​​starts calculating the variance of the weight W in L6, and when the variance of the weight W reaches a threshold, sets a BD period and gradually stops learning of L6. On the other hand, the variance calculation unit 36 ​​starts calculating the variance of the weight L9 at the timing when the variance of the weight L6 reaches the threshold. Furthermore, when the variance of the weight L9 reaches the threshold during the BD period of L6, the variance calculation unit 36 ​​sets a BD period for L9 and gradually stops learning of L9. That is, a period occurs in which the BD period of L6 and the BD period of L9 overlap.

[0079] However, when the BD period of L6 ends during the BD period of L9 and the learning of L6 ends, the variance of the weights of L9 widens even during the BD period of L9, as explained in FIG. 13 and FIG. 14. In other words, since the BD period is greater than the variance sampling interval, the variance value of L9 may become mountain-shaped after reaching the threshold due to the influence of the stopping of learning of the previous layer (L6). As a result, when the variance of the weights of L9 has not decreased sufficiently, in other words, while it is still learning, the learning of L9 may be stopped, which may cause a decrease in the accuracy of the final machine learning model 14.

[0080] Therefore, in order to improve on the improvement in FIG. 15, the information processing device 10 according to the second embodiment controls each layer so as not to execute threshold determination and variance calculation for the subsequent layer during the BD period.

[0081] Fig. 16 is a diagram for explaining the improvement of the threshold judgment timing. In Fig. 16, as in Fig. 15, an example of learning skip control using the BD period is explained, but the same applies to normal learning skip. In addition, the sampling of the weight distribution calculation is set to 200 iterations, and the BD period is set to 1000 to 3000 iterations.

[0082] As shown in Fig. 16, the variance calculation unit 36 ​​starts calculating the variance of the weight W in L6, and when the threshold is reached, sets a BD period and gradually stops learning of L6. At this time, even if the weight of L6 reaches the threshold, the variance calculation unit 36 ​​suppresses the calculation of the variance of the weight of L9 because it is in the BD period. In other words, control is performed so that there is no overlapping period between the BD period of L6 and the BD period of L9.

[0083] Then, the variance calculation unit 36 ​​starts the variance calculation of the weight W in L9 at the timing when the BD period of L6 ends, and when the threshold is reached, sets the BD period of L9 and gradually stops the learning of L9.

[0084] In this way, the variance calculation unit 36 ​​suppresses the weight variance calculation for the subsequent layers during the BD period, and can skip learning of L9 after detecting a decrease in the weight variance including the influence of L6 in the layer (L9) next to L6. As a result, the information processing device 10 can improve the accuracy of the final machine learning model 14. EXAMPLES

[0085] Next, variations in timing control of learning skip and threshold determination executed by the information processing device 10 will be described.

[0086] [Distribution calculation according to BD cycle] Fig. 17 is a diagram for explaining the implementation of distributed calculation in accordance with the BD cycle. In Fig. 17, the target layer for distributed calculation is L0 and L3 layers and onwards, and after the target layer for distribution reaches a threshold, the next layer is made the target for distributed calculation.

[0087] As shown in FIG. 17, the distributed processing unit 30 starts the distributed calculation of the weight (W) for L0, which is the first layer (Conv layer) of the machine learning model, at iteration "n-3000" when the warm-up period ends after starting machine learning, and executes the distributed calculation of the weight (W) for each iteration of iteration "n-2000" and iteration "n-1000". Then, when the variance of the weight (W) calculated at iteration "n" reaches a threshold, the distributed processing unit 30 starts skipping learning by BD control. After that, the BD period of L0 ends at iteration "n+1000", and the distributed processing unit 30 stops learning of L0.

[0088] At this time, since the learning of L0 has stopped at iteration "n+1000", the distributed processing unit 30 starts the weight distribution calculation for L3, which is the next layer after L0, from the same timing iteration "n+1000". Then, the distributed processing unit 30 also executes the weight distribution of L3 at iteration "n+2000", and when the distribution of the weight (W) calculated at iteration "n+3000" reaches a threshold, starts the learning skip by BD control for L3.

[0089] Thereafter, the BD period of L3 ends at iteration "n+4000", and the distributed processing unit 30 stops learning L3. At this time, since the learning of L3 has stopped at iteration "n+4000", the distributed processing unit 30 starts the distributed calculation of weights for L6, which is the layer next to L3, from the same timing, iteration "n+4000".

[0090] Then, the distribution processing unit 30 distributes the weight of L6 in iteration "n+5000", and when the variance of the weight (W) calculated in iteration "n+6000" reaches a threshold, starts learning skip by BD control for L6.

[0091] After that, the BD period of L6 ends at iteration "n+7000", and the distributed processing unit 30 stops learning L6. At this time, since the learning of L6 has stopped at iteration "n+7000", the distributed processing unit 30 starts the distributed calculation of weights for L9, which is the next layer after L6, from the same timing, iteration "n+7000".

[0092] In this way, by adjusting the width of the Braking Distance (BD) to the distributed calculation period, the distributed processing unit 30 can handle the impact on L3 by the distributed calculation of L3, even when the BD period of L0 ends and the impact of stopping learning of the layer is actually reflected in L3. As a result, the distributed processing unit 30 reliably stops learning for each layer, thereby suppressing deterioration in accuracy.

[0093] [Example 1 of handling multiple layers] Fig. 18 is a diagram for explaining the implementation of learning skip for multiple layers. As shown in Fig. 18, the distributed processing unit 30 groups L0, L3, L6, L9, L12, and L15 into block 1, and groups L18, L21, L24, L27, L30, and L33 into block 2. The distributed processing unit 30 then controls the learning skip for each block.

[0094] Specifically, in iteration "n" when the warm-up period ends after the start of machine learning, the distributed processing unit 30 starts the distribution calculation of the weights (W) of each layer of block 1. Then, when the distributed processing unit 30 detects that the distribution of the weights of L9, L12, and L15 reaches a threshold value in the distribution of the weights of each layer calculated in iteration "n+20", the distributed processing unit 30 stops the subsequent distribution calculation of the weights of L9, L12, and L15.

[0095] Thereafter, when the distribution processing unit 30 detects that the weight variance of L3 and L6 in the weight variance of each layer (L0, L3, L6) calculated in iteration "n+40" has reached a threshold, it stops further weight variance calculation for L3 and L6.

[0096] Thereafter, when the distributed processing unit 30 detects that the variance of the weights of L0 calculated in iteration "n+60" has reached the threshold, it stops the subsequent weight variance calculation for L0. As a result, since the weights of all layers in block 1 have reached the threshold, the distributed processing unit 30 executes learning skip for each layer of block 1 from the next iteration "n+80" and starts the variance calculation of the weights of each layer in block 2.

[0097] In this way, the distributed processing unit 30 can stop the calculation of the weight variance when the threshold is reached, and can skip learning of the target layer when all of the layers targeted for calculation are within the threshold range. The distributed processing unit 30 can also increase the number of layers to be skipped in learning at once by varying the number of target layers to be included in a block. Therefore, the distributed processing unit 30 can simultaneously perform the calculation of the weight variance for multiple layers from the input side of the machine learning model 14 (neural network) and increase the number of layers to be skipped in learning, thereby realizing high-speed machine learning.

[0098] [Example 2 of handling multiple layers] Fig. 19 is a diagram for explaining the implementation of BD control for multiple layers. As shown in Fig. 19, the distributed processing unit 30 groups L0, L3, L6, L9, L12, and L15 into block 1, and groups L18, L21, L24, L27, L30, and L33 into block 2. The distributed processing unit 30 then controls learning skip around the blocks.

[0099] Specifically, in iteration "n" when the warm-up period ends after the start of machine learning, the distributed processing unit 30 starts the distribution calculation of the weights (W) of each layer of block 1. Then, when the distributed processing unit 30 detects that the distribution of the weights of L9, L12, and L15 reaches a threshold value in the distribution of the weights of each layer calculated in iteration "n+20", the distributed processing unit 30 stops the subsequent distribution calculation of the weights of L9, L12, and L15.

[0100] Thereafter, when the distribution processing unit 30 detects that the weight variance of L3 and L6 in the weight variance of each layer (L0, L3, L6) calculated in iteration "n+40" has reached a threshold, it stops further weight variance calculation for L3 and L6.

[0101] Thereafter, when the distributed processing unit 30 detects that the variance of the weights of L0 calculated in iteration "n+60" has reached the threshold, it stops the subsequent weight variance calculation for L0. As a result, since the weights of all layers in block 1 have reached the threshold, the distributed processing unit 30 executes learning skip for each layer of block 1 by BD control from the next iteration "n+80".

[0102] Then, the distributed processing unit 30 executes learning skipping that gradually decreases the LR from iteration "n+80" to "n+160", which is the BD period, and the BD period ends at iteration "n+180", completing the learning skipping of each layer in block 1. Here, since the distributed processing unit 30 has completed the learning skipping of each layer in block 1 at iteration "n+180", it starts the distributed calculation of the weights of each layer in block 2 at iteration "n+180".

[0103] In this way, the distributed processing unit 30 stops the calculation of the weight variance when the threshold is reached, and when all of the layers targeted for the calculation are within the threshold range, the distributed processing unit 30 can set a BD period for the target layer and skip learning. Therefore, the distributed processing unit 30 can achieve both high speed machine learning and suppression of accuracy degradation. EXAMPLES

[0104] Although the embodiments of the present invention have been described above, the present invention may be embodied in various different forms other than the above-described embodiments.

[0105] [Numbers, etc.] The number of blocks, number of layers, various thresholds, numerical values, number of GPUs, etc. used in the above embodiment are merely examples and can be changed as desired. The learning rate can be increased as well as decreased. The decision to skip learning can be made not only for each iteration but also for each epoch. It is preferable to use the same scheduler for the LR scheduler and the scheduler that controls the learning rate.

[0106] In addition, the timing of calculating the weight variance is not limited to every iteration, but can be arbitrarily set and changed to every predetermined number of training data, every epoch, etc. In addition, although an example in which the weight variance is used as the weight information has been described, the weight gradient variance can also be used.

[0107] [Block control, etc.] For example, in the above example, a layer may be determined to be skipped depending on whether the error gradient of the last layer among the layers belonging to the block or the average value of the variance of the weights of each layer belonging to the block is less than a threshold value.

[0108] [BD setting example] In the above embodiment, the same BD period can be set for each layer or each block, or different BD periods can be set. For example, for blocks close to the output layer where machine learning is stopped when machine learning has progressed, a shorter BD period can be set than for blocks close to the input layer where machine learning is stopped at a relatively early stage.

[0109] [system] The information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

[0110] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure. In other words, all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0111] Furthermore, each processing function performed by each device may be realized, in whole or in part, by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware using wired logic.

[0112] [Hardware] Next, a hardware configuration example of the information processing device 10 described in the above embodiment will be described. Fig. 20 is a diagram for explaining a hardware configuration example. As shown in Fig. 20, the information processing device 10 includes a communication device 10a, a HDD (Hard Disk Drive) 10b, a memory 10c, a CPU 10d, and multiple GPUs 10e. The components shown in Fig. 20 are connected to each other via a bus or the like.

[0113] The communication device 10a is a network interface card or the like, and communicates with other servers. The HDD 10b stores programs and DBs that operate the functions shown in Figs.

[0114] The CPU 10d executes control of the entire information processing device 10, and for example, reads a program related to machine learning from the HDD 10b or the like and loads it in the memory 10c to operate each process of machine learning. Each GPU 10e executes a process for executing each function described in FIG. 8, FIG. 9, or the like by reading a program that executes the same processing as each processing unit shown in FIG. 8, FIG. 9, or the like from the HDD 10b or the like and loading it in the memory 10c. For example, each GPU 10e reads a program having the same functions as the forward propagation processing unit 31, the backpropagation processing unit 32, the information management unit 35, the variance calculation unit 36, or the like from the HDD 10b or the like. Then, each GPU 10e executes a process for executing the same processing as the forward propagation processing unit 31, the backpropagation processing unit 32, the information management unit 35, the variance calculation unit 36, or the like.

[0115] In this way, the information processing device 10 operates as an information processing device that executes various processing methods by reading and executing a program. The information processing device 10 can also realize functions similar to those of the above-mentioned embodiment by reading the program from a recording medium using a medium reading device and executing the read program. Note that the program in this other embodiment is not limited to being executed by the information processing device 10. For example, the present invention can be similarly applied to cases where another computer or server executes a program, or where these execute a program in cooperation with each other.

[0116] This program can be distributed via a network such as the Internet. This program can also be recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), or a digital versatile disk (DVD), and can be executed by being read out from the recording medium by a computer. [Explanation of symbols]

[0117] 10. Information processing device 11 Communications Department 12 Storage section 13 Training Data DB 14 Machine Learning Models 20 Integrated Processing Unit 30 Distributed Processing Unit 31 Forward propagation processing section 32 Error backpropagation processing section 33 Error gradient calculation unit 34 Communication control section 35 Information Management Department 36 Variance calculation section

Claims

1. On the computer, During machine learning of the machine learning model, a variance of weight information regarding weights updated by machine learning is calculated for each layer of the machine learning model at a predetermined interval; determining a suppression target layer for suppressing the machine learning based on the peak value of the variance calculated at the predetermined interval and the variance of the weight information calculated at the predetermined interval; A high-speed program for executing a process.

2. The calculation process includes: Calculating a variance of the weight information for each layer for each iteration as the predetermined interval; The process of determining includes: Identifying a waveform pattern of the distribution of the weight information for each layer based on a tendency of the distribution of the weight information calculated for each iteration; setting a threshold value for each layer according to the waveform pattern; determining a layer for which a difference between the peak value of the variance and the variance of the weight information calculated at the predetermined interval is equal to or greater than the threshold value as the layer to be suppressed; 2. The high-speed program according to claim 1.

3. The process of determining includes: Select a target layer for determining the suppression target in order from the input layer of the machine learning model; determining whether to suppress the machine learning for the target layer at each predetermined interval based on the peak value of the variance and the variance of the weight information; 3. The high-speed program according to claim 1 or 2.

4. The calculation process includes: After the machine learning of a first layer determined as the layer to be suppressed among the multiple layers included in the machine learning model is stopped, calculation of the variance of the weight information for a second layer located next to the first layer is started.

4. The high-speed program according to claim 3.

5. The calculation process includes: Dividing a plurality of layers of the machine learning model into a plurality of blocks in order from an input layer; Calculating the variance of the weight information for each layer belonging to each of the plurality of blocks; The process of determining includes: Selecting a target block for determining the suppression target in order of proximity to an input of the machine learning model; suppressing the machine learning of each layer belonging to the target block when a difference between a peak value of the variance of each layer belonging to the target block and a variance of the weight information becomes equal to or greater than a threshold value; 3. The high-speed program according to claim 1 or 2.

6. The calculation process includes: After the machine learning of a first block determined to be a layer to be suppressed among the plurality of block layers is stopped, calculation of the variance of the weight information is started for a second block located next to the first block.

6. The high-speed program according to claim 5.

7. The process of determining includes: For a layer to be suppressed in which the machine learning is to be suppressed, the machine learning is executed by reducing a learning rate for each iteration for a certain period until the machine learning is suppressed.

2. The high-speed program according to claim 1.

8. The computer During machine learning of the machine learning model, a variance of weight information regarding weights updated by machine learning is calculated for each layer of the machine learning model at a predetermined interval; determining a suppression target layer for suppressing the machine learning based on the peak value of the variance calculated at the predetermined interval and the variance of the weight information calculated at the predetermined interval; 2. A method for increasing speed, comprising:

9. During machine learning of the machine learning model, a variance of weight information regarding weights updated by machine learning is calculated for each layer of the machine learning model at a predetermined interval; determining a suppression target layer for suppressing the machine learning based on the peak value of the variance calculated at the predetermined interval and the variance of the weight information calculated at the predetermined interval; An information processing device comprising a control unit.

Images