Learning device, learning system, and learning method

Abstract

To efficiently learn.SOLUTION: A learning device according to an embodiment includes a generation unit, a calculation unit, and an updating unit. The generation unit generates a plurality of pieces of partial data from a mini-batch of learning data used for a plurality of times of learning processes for learning a parameter of a neural network by using an objective function. The calculation unit calculates, for each of the plurality of pieces of partial data, a partial gradient being a gradient regarding the parameter of the objective function. The updating unit updates the parameter on the basis of an average value of the plurality of partial gradients corresponding to the plurality of pieces of partial data and a variance for the plurality of partial gradients.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] FIELD An embodiment of the present invention relates to a learning device, a learning system, and a learning method. [Background technology]

[0002] Conventionally, in deep learning, which is one type of machine learning, a method of parallel learning using multiple processors, multi-core processors, or multiple devices is known. Using parallel learning has the advantage that the learning process can be accelerated by the number of parallel processes. On the other hand, there is a problem that the learning effect decreases when the data size (batch size) used for one update becomes large. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-119151 A Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a learning device, a learning system, and a learning method that enable efficient learning. [Means for solving the problem]

[0005] A learning device according to one embodiment includes a generating unit, a calculating unit, and an updating unit. The generating unit generates a plurality of partial data from mini-batches of training data used in a plurality of training processes for training a parameter of a neural network using an objective function. The calculating unit calculates a partial gradient, which is a gradient of the objective function with respect to the parameter, for each of the plurality of partial data. The updating unit updates the parameter based on an average value of a plurality of partial gradients corresponding to the plurality of partial data and a variance of the plurality of partial gradients. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram showing the configuration of an information processing system including a learning device according to a first embodiment. [Diagram 2] 5 is a flowchart of a parameter update process according to the first embodiment. [Diagram 3] 5 is a flowchart showing a specific example of a parameter update process in the first embodiment. [Figure 4] 13 is a flowchart of a parameter update process according to the second embodiment. [Diagram 5] 5 is a flowchart of a partial gradient calculation process in the flowchart of FIG. 4 . [Figure 6] 10 is a flowchart showing a specific example of a parameter update process in the second embodiment. [Figure 7] FIG. 13 is a block diagram showing an example of the configuration of an information processing system including a management device and a plurality of learning devices according to a third embodiment. [Figure 8] 13 is a flowchart showing a specific example of a parameter update process in the third embodiment. [Figure 9] FIG. 13 is a diagram for explaining a communication pattern of ring-type All-Reduce in the third embodiment. [Figure 10] FIG. 4 is a diagram for explaining a specific example in which each node shares the average value of multiple partial gradients in the ring-type All-Reduce in FIG. 3. [Figure 11] FIG. 4 is a diagram for explaining a specific example in which each node shares variance for multiple partial gradients in the ring-type All-Reduce of FIG. 3. [Figure 12] FIG. 4 is a diagram for explaining a specific example in which each node simultaneously shares the mean value and the variance in the ring-type All-Reduce of FIG. 3. [Figure 13] 13 shows evaluation results including learning curves using the parameter update processes according to the first to third embodiments. [Figure 14] 13 is a flowchart showing a specific example of a parameter update process in the fourth embodiment. [Figure 15]13 shows evaluation results including a learning curve using the parameter update process of the fourth embodiment. [Figure 16] FIG. 1 is a block diagram illustrating a hardware configuration of a computer according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] First, an overview of machine learning used in each embodiment will be described. In deep learning, which is one type of machine learning, optimization by Stochastic Gradient Descent (SGD) using mini-batches is generally used. In addition, when updating parameters of a neural network using this SGD, optimization by Momentum SGD that takes momentum into account is also known. In each of the following embodiments, optimization (i.e., parameter update) using Momentum SGD will be described. However, other optimization methods such as SGD and Nesterov acceleration method may be used.

[0008] First, consider updating parameters using a neural network with objective function f and parameters w as optimization by SGD. In this case, the training data x used for the t-th parameter update is t For parameter w t The gradient of g t is expressed by the following equation (1).

[0009]

number

[0010] This gradient g t and the learning coefficient η, the parameter w t+1 is expressed by the following equation (2).

[0011]

number

[0012] Furthermore, in the Momentum SGD optimization, the momentum term v t is used.

[0013]

number

[0014] In equation (3), β represents a momentum coefficient. In the following embodiments, β is set to 0.9. When t=0, the momentum term v t =0.

[0015] This momentum term v t The t+1th time parameter w t+1 is expressed by the following equation (4).

[0016]

number

[0017] The learning coefficient η in the above formula (3) may change during learning. For example, in the conditions of the evaluation results described below, in the case of a batch size of 8192, the learning coefficient η is gradually increased from 0 to 6.4 in the first 10 epochs, attenuated to 0.64 after reaching 80 epochs, and further attenuated to 0.064 after reaching 120 epochs.

[0018] Next, partial gradients and overall gradients, which are gradients related to parameters of an objective function, will be described. In the following embodiments, all data used in learning will be called learning data, data used in one parameter update will be called mini-batch, and data processed by one processor or one device at a time will be called partial data. In addition, the partial gradient is a gradient calculated for the partial data, and the overall gradient is a gradient calculated for the entire mini-batch.

[0019] For example, the mini-batch x used in the tth update t The partial data x t n (n>2) t n The partial gradient g calculated using t n is expressed by the following equation (5).

[0020]

number

[0021] In addition, the overall gradient g t is the partial gradient g t n The average value of these is expressed by the following equation (6).

[0022]

number

[0023] Hereinafter, embodiments of a learning device and a learning system will be described in detail with reference to the drawings.

[0024] (First embodiment) FIG. 1 is a block diagram showing a configuration of an information processing system 100 including a learning device 110 according to a first embodiment. The information processing system 100 in FIG. 1 includes a learning device 110 and an information processing device 120. The learning device 110 and the information processing device 120 are connected via a network NW, for example. The network NW may be in any form, such as a wired network, a wireless network, or the Internet. At least one of the learning device 110 and the information processing device 120 may be realized by a server device, which is a computer having a processor such as a CPU (Central Processing Unit), for example. The server device may also be a cloud server that executes processing on the cloud.

[0025] The learning device 110 is a device that learns a neural network. The information processing device 120 is a device that executes processing (for example, recognition processing and classification processing) using the neural network learned by the learning device 110 or the like.

[0026] The learning device 110 and the information processing device 120 do not have to be separate devices. For example, the learning device 110 may have the functions of the information processing device 120.

[0027] The learning device 110 includes a generation unit 111 , a calculation unit 112 , an update unit 113 , a storage unit 114 , and a communication unit 115 .

[0028] The generating unit 111 generates individual training data used for training the neural network. Specifically, the generating unit 111 generates multiple mini-batches by sampling a portion of the training data. The generating unit 111 also selects a mini-batch to be a target of training processing from the multiple mini-batches, and generates multiple partial data from the selected mini-batch. Note that when the amount of training data is small, it is not necessary to generate multiple mini-batches, and therefore the generating unit 111 may generate multiple partial data from the training data.

[0029] The calculation unit 112 calculates various information used when learning the parameters of the neural network. Specifically, the calculation unit 112 calculates a gradient related to the parameters of the objective function for each of the multiple partial data. As described above, hereinafter, the gradient calculated for the partial data is referred to as a partial gradient. Furthermore, the calculation unit 112 calculates a gradient for the entire mini-batch using the multiple partial gradients. As described above, hereinafter, the gradient calculated for the entire mini-batch is referred to as an overall gradient.

[0030] More specifically, the calculation unit 112 calculates an average value of the plurality of partial gradients and a variance of the plurality of partial gradients. 2 (g t ) is the partial gradient g of the above equation (6). t nUsing the average value of these, it is expressed by the following equation (7).

[0031]

number

[0032] In the above formula (7), the square of the partial gradient is calculated on the right side, but this is not limited to this. For example, the variance σ 2 (g t ), the deviation between the partial gradient and the average value of the partial gradients may be calculated.

[0033] Furthermore, the calculation unit 112 calculates the overall gradient using the average value and the variance. t is the conventional overall gradient g t is expressed by the following equation (8).

[0034]

number

[0035] In other words, the overall gradient is calculated by multiplying the mean value by the inverse of the square root of the sum of the square of the mean value and the variance.

[0036] The calculation unit 112 may be configured with a multi-core processor or may be configured with a plurality of processors, that is, the calculation unit 112 is configured to perform parallel processing in the above calculation.

[0037] The update unit 113 updates the parameters (weights) using the calculation results by the calculation unit 112. The updated parameters may be any information. For example, in the case of learning a neural network, the weights and biases of the neural network are the parameters. In the following, an example will be described in which weights are mainly used as parameters. Furthermore, the weights of a neural network may be rephrased as coefficients (model coefficients) of a network model (also simply called a model).

[0038] Specifically, the update unit 113 updates the weights using the average value and the variance. More specifically, the update unit 113 updates the weights using the overall gradient calculated using the average value and the variance. The update unit 113 may also determine whether to end the learning.

[0039] The storage unit 114 stores various information used in various processes by the learning device 110. Specifically, the storage unit 114 stores parameters of the neural network to be trained, learning data used for training the neural network, and the like.

[0040] The communication unit 115 communicates with the information processing device 120 regarding various information from the learning device 110. Specifically, the communication unit 115 receives learning data from the information processing device 120. In addition, the communication unit 115 transmits (outputs) parameters of the trained neural network to the information processing device 120.

[0041] The information processing device 120 includes a receiving unit 121 , an information processing unit 122 , an output control unit 123 , a storage unit 124 , and a communication unit 125 .

[0042] The receiving unit 121 receives input of various information used in various processes by the information processing device 120. Specifically, the receiving unit 121 receives parameters of the neural network output from the learning device 110.

[0043] The information processing unit 122 executes information processing using a neural network. The information processing is, for example, image recognition processing and image classification processing using a neural network. The information processing is not limited to these, and may be any processing using a neural network. For example, the information processing may be recognition processing and classification processing for data other than images (for example, text and voice).

[0044] The output control unit 123 controls the output of various information by the information processing device 120. Specifically, the output control unit 123 controls the output of learning data to the learning device 110 and the output of information instructing the learning device 110 to start learning, information instructing the learning device 110 to end learning, and the like.

[0045] The storage unit 124 stores various information used in various processes by the information processing device 120. Specifically, the storage unit 124 stores the parameters of the neural network output from the learning device 110.

[0046] The communication unit 125 communicates various information from the information processing device 120 with the learning device 110. Specifically, the communication unit 125 transmits learning data to the learning device 110. In addition, the communication unit 125 receives parameters of the trained neural network from the learning device 110.

[0047] The configuration of the learning device 110 and the like according to the first embodiment has been described above. Next, the operation of the learning device 110 will be described with reference to the flowchart in FIG.

[0048] Fig. 2 is a flowchart of a parameter update process in the first embodiment. The flowchart in Fig. 2 shows, for example, a series of steps in which the learning device 110 learns a neural network from learning data. The flowchart in Fig. 2 starts when the learning device 110 receives learning data and a learning start instruction from the information processing device 120.

[0049] (Step ST210) The generation unit 111 generates a plurality of mini-batches from the training data.

[0050] (Step ST220) After generating a plurality of mini-batches, the generation unit 111 selects a mini-batch to be subjected to a learning process from among the plurality of mini-batches.

[0051] (Step ST230) After selecting the target mini-batch, the generation unit 111 generates a plurality of partial data from the mini-batch.

[0052] (Step ST240) After generating the plurality of partial data, the calculation unit 112 calculates a partial gradient for each of the plurality of partial data.

[0053] (Step ST250) After calculating the multiple partial gradients, the calculation unit 112 calculates the average value of the multiple partial gradients and the variance of the multiple partial gradients.

[0054] (Step ST260) After calculating the mean and variance, the update unit 113 updates the weights using the mean and variance.

[0055] (Step ST270) After updating the weights, the update unit 113 determines whether to end the learning. For example, the update unit 113 determines to end the learning when all of the multiple mini-batches have been processed. If it is determined that the learning is not to be ended, the process returns to step ST220. If it is determined that the learning is to be ended, the process ends.

[0056] The above describes a series of steps for learning a neural network in the first embodiment. Next, a specific description will be given of the processing performed in one parameter update in the first embodiment.

[0057] 3 is a flowchart showing a specific example of a parameter update process in the first embodiment. The flowchart in FIG. 3 relates to the process from step ST230 to step ST260 in the flowchart in FIG. 2. In the following, t The process will now be described in detail.

[0058] (Step ST310) The generation unit 111 generates a mini-batch x tGenerate multiple partial data from mini-batches x t In the case where the data is divided into n pieces of partial data, the generation unit 111 divides the partial data x t 1 From partial data x t n Generate up to.

[0059] (Step ST320) The calculation unit 112 calculates a partial gradient for each of the plurality of partial data. For example, the calculation unit 112 calculates a partial gradient for each of the plurality of partial data x t 1 From partial data x t n The partial gradient g corresponding to t 1 from the partial gradient g t n Calculate the partial gradient g t 1 and the partial gradient g t n are expressed by the following equations (9) and (10), respectively.

[0060]

number

[0061]

number

[0062] (Step ST330) The calculation unit 112 calculates an average value g t (In Fig. 3, g t (a horizontal bar is added above the mean value) and the variance σ 2 (g t ) is calculated. Hereafter, the bars indicating the average value will be omitted in the specification. Average value g t and variance σ 2 (g t ) are expressed by the following equations (11) and (12), respectively.

[0063]

number

[0064]

number

[0065] (Step ST340) The calculation unit 112 calculates the average value g t and variance σ 2 (g t ) to find the global gradient g t Calculate the overall gradient g t is expressed by the following equation (13).

[0066]

number

[0067] (Step ST350) The update unit 113 updates the overall gradient g t For example, in the case of optimization using Momentum SGD, the update unit 113 updates the weights by applying the above-mentioned formulas (3) and (4).

[0068] As described above, the learning device of the first embodiment generates multiple partial data from mini-batches of learning data used in multiple learning processes for learning parameters of a neural network using an objective function, calculates a partial gradient, which is the gradient with respect to the parameters of the objective function, for each of the multiple partial data, and updates the parameters based on the average value of multiple partial gradients corresponding to the multiple partial data and the variance of the multiple partial gradients.

[0069] Therefore, the learning device according to the first embodiment can perform learning efficiently by taking into account the variance components contained in the parameters.

[0070] Furthermore, the learning device calculates an overall gradient, which is the gradient of the objective function for the mini-batch, by using the mean and variance, and updates the parameters based on the overall gradient. Furthermore, the learning device calculates the overall gradient by multiplying the mean by the inverse of the square root of the sum of the square of the mean and the variance.

[0071] For example, parameters include "components with large variance" and "components with small variance." Since the gradient of the "components with large variance" is large, learning of this component ends early. Since the gradient of the "components with small variance" is small, learning of this component is likely to be insufficient even towards the end. For these reasons, it is thought that learning efficiency will decrease if "components with large variance" and "components with small variance" are trained in the same update step (parameter update).

[0072] In order to suppress the above-mentioned decrease in learning efficiency, the learning device according to the first embodiment uses an overall gradient including the inverse of the variance when updating parameters, thereby adjusting the learning speed of the "component with large variance" and the "component with small variance", thereby improving the learning efficiency.

[0073] Second Embodiment In the first embodiment, the weights are updated using the mean and variance, whereas in the second embodiment, the partial gradients are calculated with parameters that are further increased with noise.

[0074] The calculation unit 112 according to the second embodiment further calculates noise to be added to the parameters. t n The partial gradient g calculated using t n In the process of calculating the parameter w t Noise θ t n Add the partial gradient g t n For example, the parameter w t Noise θ t nThe partial gradient g calculated by adding t n is expressed by the following equation (14).

[0075]

number

[0076] In addition, the calculation unit 112 calculates the noise θ t n Based on the previous (e.g., immediately preceding) partial gradient, the calculation unit 112 calculates the immediately preceding overall gradient g t-1 and the previous partial gradient g t-1 n Using the difference between t n Calculate the noise θ t n is calculated, for example, by the following formula (15).

[0077]

number

[0078] Fig. 4 is a flowchart of a parameter update process in the second embodiment. Each step of the flowchart in Fig. 4 is the same as the steps of the flowchart in Fig. 2, except for some steps. The flowchart in Fig. 4 differs from step ST240 in the flowchart in Fig. 2 in that the process of step ST440 is a subroutine. Therefore, only step ST440 will be described.

[0079] (Step ST440) After generating a plurality of partial data, the calculation unit 112 calculates a partial gradient for each of the plurality of partial data. In the second embodiment, the calculation unit 112 calculates a partial gradient for each of the partial data using parameters to which noise is added or parameters to which noise is not added. Hereinafter, the process of step ST440 is referred to as a "partial gradient calculation process." A specific example of the partial gradient calculation process will be described with reference to the flowchart of FIG. 5.

[0080] Fig. 5 is a flowchart of the partial gradient calculation process in the flowchart of Fig. 4. The flow chart of Fig. 5 transitions from step ST430.

[0081] (Step ST510) After generating a plurality of partial data, the calculation unit 112 judges whether or not to calculate noise in the current learning process. For example, the calculation unit 112 judges not to calculate noise when the learning process corresponds to a number of times that noise is not calculated, such as the first learning process. Note that, when it is determined that noise is to be added in the m-th and subsequent learning processes, the calculation unit 112 may judge not to calculate noise in the first to (m-1)th learning processes. If it is determined that noise is to be calculated, the process proceeds to step ST520. If it is determined that noise is not to be calculated, the process proceeds to step ST540.

[0082] (Step ST520) After determining to calculate the noise, the calculation unit 112 calculates the noise using the calculated partial gradient. The calculated partial gradient is, for example, a partial gradient calculated in the immediately previous (for example, one time before) learning process. The calculation unit 112 calculates the noise by applying, for example, the above formula (15).

[0083] (Step ST530) After calculating the noise, the calculation unit 112 calculates a partial gradient for the partial data using the parameters to which the noise has been added. For example, the calculation unit 112 calculates the partial gradient for the parameters to which the noise has been added by applying the above formula (14). After step ST530, the process proceeds to step ST450.

[0084] (Step ST540) After determining not to calculate noise, the calculation unit 112 calculates a partial gradient for the partial data with parameters to which noise is not added. The process of step ST540 is similar to the process of step ST240 in Fig. 2. After step ST540, the process proceeds to step ST450.

[0085] The above describes a series of steps for learning a neural network in the second embodiment. Next, a specific description will be given of processing performed in one parameter update in the second embodiment when noise is added to the model coefficients.

[0086] Fig. 6 is a flowchart showing a specific example of a parameter update process in the second embodiment. The flowchart in Fig. 6 relates to the processes of steps ST430, ST520, ST530, ST450, and ST460 in the flowchart in Fig. 4 and the flowchart in Fig. 5. In the following, t The process will now be described in detail.

[0087] (Step ST610) The generation unit 111 generates a mini-batch x t Generate multiple partial data from mini-batches x t In the case where the data is divided into n pieces of partial data, the generation unit 111 divides the partial data x t 1 From partial data x t n Generate up to.

[0088] (Step ST620) The calculation unit 112 calculates the overall model coefficient w t For example, the calculation unit 112 calculates noise based on each partial gradient calculated in the immediately preceding learning process, and calculates the partial data x t 1 From partial data xt n The partial gradient g corresponding to t 1 from the partial gradient g t n Calculate up to partial data x t 1 Noise θ t 1 and the partial gradient g t 1 are expressed by the following formulas (16) and (17), respectively. t n Noise θ t n and the partial gradient g t n are expressed by the following equations (18) and (19), respectively.

[0089]

number

[0090]

number

[0091]

number

[0092]

number

[0093] (Step ST630) The calculation unit 112 calculates an average value g t and the variance σ for multiple partial gradients 2 (g t ) is calculated. The average value g t and variance σ 2 (g t ) are expressed by the above-mentioned equations (11) and (12), respectively.

[0094] (Step ST640) The calculation unit 112 calculates the average value g t and variance σ 2 (g t ) to find the global gradient g t Calculate the overall gradient g t is expressed by the above-mentioned equation (13).

[0095] (Step ST650) The update unit 113 updates the overall gradient g t For example, in the case of optimization using Momentum SGD, the update unit 113 updates the weights by applying the above-mentioned formulas (3) and (4).

[0096] As described above, the learning device according to the second embodiment generates a plurality of partial data from mini-batches of learning data used in a plurality of learning processes for learning parameters of a neural network using an objective function, calculates a partial gradient, which is a gradient of the parameter of the objective function, for each of the plurality of partial data, and updates the parameter based on an average value of the plurality of partial gradients corresponding to the plurality of partial data and a variance of the plurality of partial gradients. Furthermore, the learning device calculates noise to be added to the parameter for each of the plurality of partial data, and calculates a partial gradient for the parameter to which the noise has been added.

[0097] Therefore, in addition to the effects of the first embodiment, the learning device of the second embodiment can add noise in an appropriate direction to the shape of the objective function, thereby enabling efficient smoothing that matches the objective function.

[0098] (Third embodiment) The parameter update processes in the first and second embodiments have both been described as being executed by one learning device, whereas the parameter update process in the third embodiment will be described as being executed by multiple learning devices.

[0099] FIG. 7 is a block diagram showing a configuration example of an information processing system 700 including a management device 710 and a plurality of learning devices 720-1 to 720-N according to the third embodiment. The information processing system 700 of FIG. 7 includes a management device 710, a plurality of learning devices 720-1 to 720-N, and an information processing device 730. These devices are connected via a network NW, for example. Any one of the management device 710, the plurality of learning devices 720-1 to 720-N, and the information processing device 730 may be realized by, for example, a server device that is a computer having a processor such as a CPU. Each of the plurality of learning devices 720-1 to 720-N may be called a node. The information processing device 730 is substantially the same as the information processing device 120 of FIG. 1, and therefore a description thereof will be omitted.

[0100] The management device 710 is a device that collectively manages the multiple learning devices 720-1 to N. The multiple learning devices 720-1 to N are devices that learn neural networks, respectively. The information processing device 730 is a device that executes processing using the neural networks learned by the multiple learning devices 720-1 to N.

[0101] Note that the management device 710 and the multiple learning devices 720-1 to 720-N do not have to be separate devices. For example, any one of the multiple learning devices 720-1 to 720-N may have the functions of the management device 710. Also, any one of the multiple learning devices 720-1 to 720-N and the information processing device 730 do not have to be separate devices. For example, any one of the multiple learning devices 720-1 to 720-N may have the functions of the information processing device 730. Also, the multiple learning devices 720-1 to 720-N, or a combination of the management device 710 and the multiple learning devices 720-1 to 720-N, may be called a learning system.

[0102] The management device 710 includes a generation unit 711, an allocation unit 712, a storage unit 713, and a communication unit 714.

[0103] The generating unit 711 generates individual training data to be used for training the neural network. Specifically, the generating unit 711 generates multiple mini-batches by sampling a portion of the training data. The generating unit 711 also selects a mini-batch to be used in training processing from the multiple mini-batches, and generates multiple partial data from the selected mini-batch.

[0104] The allocation unit 712 allocates each of the plurality of partial data to each of the plurality of learning devices 720-1 to 720-N.

[0105] The storage unit 713 stores various information used in various processes by the management device 710. Specifically, the storage unit 713 stores parameters of the neural network to be trained, learning data used for training the neural network, and the like.

[0106] The communication unit 714 communicates with the multiple learning devices 720-1 to 720-N and the information processing device 730 regarding various information from the management device 710. Specifically, the communication unit 714 receives learning data from the information processing device 730. The communication unit 714 also transmits (outputs) each of the multiple partial data to each of the multiple learning devices 720-1 to 720-N. The communication unit 714 also outputs parameters of the trained neural network to the information processing device 730.

[0107] The multiple learning devices 720-1 to 720-N execute processes such as calculation of partial gradients for each of multiple partial data generated from one mini-batch in a distributed manner. Since the multiple learning devices 720-1 to 720-N each have the same configuration, they will be simply referred to as learning devices 720 when there is no need to distinguish between them. The number of learning devices 720 is, for example, two or more.

[0108] The learning device 720 includes a calculation unit 721, an update unit 722, a storage unit 723, and a communication unit 724.

[0109] The calculation unit 721 calculates various information used when learning the parameters of the neural network. Specifically, the calculation unit 721 calculates a gradient (partial gradient) of the parameters of the objective function for each of the plurality of partial data. The calculation unit 721 may also calculate noise to be added to the parameters. The calculation unit 721 may also calculate an average value of the plurality of partial gradients, a variance of the plurality of partial gradients, and an overall gradient. Note that the average value of the plurality of partial gradients, the variance of the plurality of partial gradients, and the overall gradient may be calculated only by a specific learning device.

[0110] The update unit 722 updates the parameters (weights) using the calculation results by the calculation unit 721. Specifically, the update unit 722 updates the weights using the average value and the variance. More specifically, the update unit 722 updates the weights using the overall gradient calculated using the average value and the variance.

[0111] The storage unit 723 stores various information used in various processes by the learning device 720. Specifically, the storage unit 723 stores parameters of the neural network to be trained, learning data used for training the neural network, and the like.

[0112] The communication unit 724 communicates with the management device 710, the information processing device 730, and other learning devices regarding various information from the learning device 720. Specifically, the communication unit 724 receives partial data from the management device 710. The communication unit 724 also transmits (outputs) gradient information related to parameter updates to other learning devices. The gradient information related to parameter updates is, for example, information on partial gradients, information on the square of partial gradients, information on the average value of multiple partial gradients, and information on variance of partial gradients. The communication unit 724 also outputs parameters of the trained neural network to the information processing device 730.

[0113] In summary, the management device 710 and the learning device 720 have the same functions as the learning device 110 according to the first embodiment. That is, the management device 710 generates partial data based on the learning data, and the learning device 720 calculates partial gradients based on the partial data. Also, a specific learning device calculates an overall gradient based on a plurality of partial gradients.

[0114] The above has described the management device 710 and the learning device 720 according to the third embodiment. Next, the operations of the management device 710 and the learning device 720 will be described with reference to FIGS.

[0115] 2, management device 710 executes steps ST210 to ST230 and step ST270, and learning device 720 executes steps ST240 to ST260. In step ST270, management device 710 determines whether or not to end learning.

[0116] Between steps ST230 and ST240, a process is included in which the learning device 720 receives partial data from the management device 710. Between steps ST250 and ST260, a process is included in which the learning device 720 shares information on partial gradients and information on the squares of partial gradients with other learning devices. For example, a ring-type All-Reduce is used for this process. The ring-type All-Reduce will be described later.

[0117] 4, the management device 710 executes steps ST410 to ST430 and step ST470, and the learning device 720 executes steps ST440 to ST460. In step ST470, the management device 710 determines whether or not to end the learning.

[0118] Note that between step ST430 and step ST440, there is a process in which the learning device 720 receives partial data from the management device 710. In addition, between step ST450 and step ST460, there is a process in which the learning device 720 shares information on the partial gradient and information on the square of the partial gradient with other learning devices.

[0119] The above describes a series of steps for learning a neural network in the third embodiment. Next, a specific description will be given of a process performed in one parameter update in the third embodiment when noise is added to the model coefficient of each node.

[0120] Fig. 8 is a flowchart showing a specific example of a parameter update process in the third embodiment. The flowchart in Fig. 8 relates to the processes of steps ST430, ST520, ST530, ST450, and ST460 in the flowchart in Fig. 4 and the flowchart in Fig. 5.

[0121] (Step ST810) The generation unit 711 of the management device 710 generates a mini-batch x t Generate multiple partial data from mini-batches x t In the case where the data is divided into n pieces of partial data, the generating unit 711 divides the partial data x t 1 to partial data x t n Generate up to.

[0122] After step ST810, the management device 710 assigns each of the plurality of partial data to the plurality of learning devices. In the next step ST820, the operation of the learning device 720-1 (first node) and the learning device 720-n (nth node) among the plurality of learning devices will be described. Note that the same operation is performed in the other learning devices.

[0123] (Step ST820) The learning device 720-1 uses pre-assigned model coefficients w t 1For example, the learning device 720-1 calculates noise based on the partial gradient calculated in the immediately preceding learning process, and calculates the partial data x t 1 The partial gradient g corresponding to t 1 Calculate the partial data x t 1 Noise θ t 1 and the partial gradient g t 1 are expressed by the following equations (20) and (21), respectively.

[0124]

number

[0125]

number

[0126] The learning device 720-n uses pre-assigned model coefficients w t n For example, the learning device 720-n calculates noise based on the partial gradient calculated in the immediately preceding learning process, and calculates the partial data x t n The partial gradient g corresponding to t n Calculate the partial data x t n Noise θ t n and the partial gradient g t n are expressed by the following equations (22) and (23), respectively.

[0127]

number

[0128]

number

[0129] After step ST820, the multiple learning devices 720-1 to 720-n share information on the partial gradients calculated by each of them. In the next step ST830, the operation of a specific learning device (here, learning device 720-1) will be described.

[0130] (Step ST830) The learning device 720-1 calculates an average value g of the multiple partial gradients based on the multiple partial gradients of the other learning devices. t and the variance σ for multiple partial gradients 2 (g t ) is calculated. The average value g t and variance σ 2 (g t ) are expressed by the above-mentioned equations (11) and (12), respectively.

[0131] After step ST830, the information on the mean and variance (gradient information) calculated by the learning device 720-1 is shared with the other learning devices. In the following steps ST840 and ST850, the operations of the learning device 720 will be described without distinguishing between the learning devices.

[0132] (Step ST840) The learning device 720 calculates the average value g t and variance σ 2 (g t ) to find the global gradient g t Calculate the overall gradient g t is expressed by the above-mentioned equation (13).

[0133] (Step ST850) The learning device 720 calculates the overall gradient g t For example, in the case of optimization using Momentum SGD, the learning device 720 updates the weights by applying the above-mentioned formulas (3) and (4).

[0134] Fig. 9 is a diagram for explaining a communication pattern of ring-type All-Reduce in the third embodiment. A communication pattern 900 in Fig. 9 shows a communication relationship between a first node 910 (node ​​A), a second node 920 (node ​​B), and a third node 930 (node ​​C). Specifically, node A transmits data to node B, node B transmits data to node C, and node C transmits data to node A. In the following description, it is assumed that node A calculates an average value and a variance based on a plurality of partial gradients collected. Note that node A, which calculates the average value and the variance, may be called a specific learning device.

[0135] The following two methods of information sharing between nodes are explained below. The first method is to send one piece of data in one data transmission. One data transmission means sending data from one node to another node. The second method is to send multiple pieces of data in one data transmission. The first method is explained using Figures 10 and 11, and the second method is explained using Figure 12.

[0136] Fig. 10 is a diagram illustrating a specific example in which each node shares the average value of multiple partial gradients in the ring-type All-Reduce of Fig. 3. When the partial gradients are calculated in each node, each node holds the respective partial gradients. For example, node A holds partial gradient g1 for data a, node B holds partial gradient g2 for data b, and node C holds partial gradient g3 for data c.

[0137] (Step ST1010) Node A transmits data a (=g1) to node B. Node B, which receives data a, sums up g1 stored in data a. Specifically, node B adds g1 to data b and stores the sum.

[0138] (Step ST1020) Node B transmits data b (= g1 + g2) to node C. Node C, which receives data b, aggregates g1 + g2 stored in data b. Specifically, node C adds g1 + g2 to data c and stores the result.

[0139] (Step ST1030) Node C transmits data c (= g1 + g2 + g3) to node A. Node A, which receives data c, calculates and stores the average value g of g1 + g2 + g3 stored in data c. Specifically, node A calculates the average value g of multiple partial gradients based on the aggregated partial gradient g1 + g2 + g3 and the total number of nodes, which is 3, and stores it in data a.

[0140] (Step ST1040) Node A transmits data a (=average value g) to node B. Node B, which receives data a, stores average value g in data b.

[0141] (Step ST1050) Node B transmits data b (=average value g) to node C. Node C, which receives data b, stores the average value g in data c. After the process of step ST1050 is performed, each node is in a state of sharing the average value g of multiple partial gradients.

[0142] FIG. 11 is a diagram for explaining a specific example in which each node shares the variance of multiple partial gradients in the ring-type All-Reduce of FIG. 3. After each node shares the average value g of multiple partial gradients, each node holds the square of each partial gradient (hereinafter referred to as squared data). For example, node A holds the squared data g1 2 is stored in data a, and node B stores squared data g2 2 is stored in data b, and node C stores squared data g3 2 is stored in data c.

[0143] (Step ST1110) Node A receives data a(=g1 2) to node B. Node B, which receives data a, transmits g1 2 Specifically, node B aggregates data b with g1 2 Add and hold.

[0144] (Step ST1120) Node B receives data b(=g1 2 +g2 2 ) to node C. Node C, which receives data b, sends g1 2 +g2 2 Specifically, node C aggregates data c with g1 2 +g2 2 Add and hold.

[0145] (Step ST1130) Node C receives data c(=g1 2 +g2 2 +g3 2 ) to node A. Node A, which receives data c, sends g1 stored in data c to node A. 2 +g2 2 +g3 2 Using the already shared mean g, we can calculate the variance σ 2 Calculate (g) and store it in data a. Variance σ 2 (g) is expressed, for example, by the following equation (24).

[0146]

number

[0147] (Step ST1140) Node A receives data a (= variance σ 2 (g)) to node B. Node B, which receives data a, assigns variance σ 2 (g) is held.

[0148] (Step ST1150) Node B uses data b (= variance σ 2(g)) to node C. Node C, which received data b, assigns variance σ 2 After the process of step ST1150, each node holds the variance σ 2 (g) is now shared.

[0149] To summarize the above, in the first method shown in Fig. 10 and Fig. 11, each node (multiple learning devices) shares partial gradient information and squared partial gradient information at different times. While the first method increases communication costs compared to conventional methods, it can keep memory usage the same as conventional methods.

[0150] FIG. 12 is a diagram for explaining a specific example in which each node simultaneously shares the mean value and variance in the ring-type All-Reduce of FIG. 3. When each node calculates a partial gradient, each node holds the partial gradient and the square of the partial gradient (squared data). For example, node A holds (g1, g1 2 ) in data a, and node B stores (g2, g2 2 ) in data b, and node C stores (g3, g3 2 ) is stored in data c.

[0151] (Step ST1210) Node A stores data a(=(g1,g1 2 )) to node B. Node B, which receives data a, transmits (g1, g1) stored in data a. 2 ) for each element. Specifically, node B adds (g1,g1 2 ) and keep them together.

[0152] (Step ST1220) Node B receives data b(=(g1+g2,g1 2 +g2 2 )) to node C. Node C, which received data b, sends the (g1+g2,g1) stored in data b to node C. 2 +g2 2) for each element. Specifically, node C adds (g1+g2,g1 2 +g2 2 ) and keep them together.

[0153] (Step ST1230) Node C stores data c(=(g1+g2+g3,g1 2 +g2 2 +g3 2 )) to node A. Node A, which receives data c, calculates the average value g of g1 + g2 + g3 stored in data c, and then 2 +g2 2 +g3 2 Using the already shared mean g, we can calculate the variance σ 2 Calculate (g) and store it in data a. Variance σ 2 (g) is expressed, for example, by the above-mentioned equation (24).

[0154] (Step ST1240) Node A receives data a (=(mean g, variance σ 2 (g))) to node B. Node B, which received data a, transmits data b with (mean g, variance σ 2 (g)) is held.

[0155] (Step ST1250) Node B receives data b (=(mean g, variance σ 2 (g))) to node C. Node C, which received data b, assigns data c (mean g, variance σ 2 After the process of step ST1250, each node holds the average value g of the multiple partial gradients and the variance σ 2 (g) is now shared.

[0156] To summarize the above, in the second method shown in Fig. 12, each node (multiple learning devices) shares the partial gradient information and the squared partial gradient information at the same time. While the second method requires more memory usage than the conventional method, it can keep the communication cost the same as the conventional method.

[0157] Fig. 13 shows the evaluation results including the learning curves using the parameter update processes from the first to third embodiments. The evaluation result 1300 in Fig. 13 was obtained by constructing a 32-layer ResNet (Residual Networks) and verifying it using the CIFAR-10 dataset as a benchmark. The evaluation result 1300 also includes five learning curves 1310 to 1350 verified under different conditions of batch size and optimization method.

[0158] The learning curve 1310 is the result of optimization using Momentum SGD with the traditional global gradient (see equation (6)) with a batch size of 128. The prediction accuracy in the learning curve 1310 is 94.7%.

[0159] The learning curve 1320 is the result of optimization using Momentum SGD with a batch size of 8192 (8k) and the traditional global gradient. The prediction accuracy in the learning curve 1320 is 10.3%.

[0160] The learning curve 1330 is the result of optimization using Mmentum SGD with a batch size of 8k and a conventional global gradient with partial gradients calculated by adding noise to the parameters. The prediction accuracy in the learning curve 1330 is 87.8%.

[0161] The learning curve 1340 is a result of optimization by Momentum SGD using the global gradient (see formula (8)) of the first embodiment with a batch size of 8k. The prediction accuracy in the learning curve 1340 is 93.7%.

[0162] The learning curve 1350 is a result of optimization by Memntum SGD using the global gradient of the second embodiment (or the third embodiment) (i.e., the global gradient of the first embodiment using the partial gradient calculated by adding noise to the parameters) with a batch size of 8k. The prediction accuracy in the learning curve 1350 is 94.1%.

[0163] Therefore, it can be seen that the prediction accuracy for the learning curve 1340 and the learning curve 1350 is significantly higher than the prediction accuracy for the learning curve 1320. Therefore, by using any of the methods of the above embodiments, it is possible to significantly improve the prediction accuracy compared to the conventional method of simply increasing the batch size.

[0164] As described above, the learning system of the third embodiment includes a plurality of learning devices that learn parameters of a neural network using an objective function, and a management device that manages the plurality of learning devices. The management device generates a plurality of partial data from mini-batches of learning data used in a plurality of learning processes for learning the parameters, and assigns each of the plurality of partial data to each of the plurality of learning devices. Each of the plurality of learning devices calculates a partial gradient, which is a gradient with respect to the parameters of the objective function, for the assigned partial data, and updates the parameters based on an average value of a plurality of partial gradients corresponding to the plurality of partial data and a variance for the plurality of partial gradients.

[0165] Therefore, the learning system according to the third embodiment can enable efficient learning, similar to the first embodiment.

[0166] (Fourth embodiment) In the first to third embodiments, the parameter update process has been described as updating weights using the mean and variance. On the other hand, in the fourth embodiment, a description will be given of updating weights using a moving average of the mean and variance. Hereinafter, the description will be given assuming that the method is applied to the information processing system 700 in FIG. 7, but is not limited thereto.

[0167] The calculation unit 721 of the learning device 720 according to the fourth embodiment may further calculate a moving average of the variance. Note that the moving average of the variance may be calculated only by a specific learning device.

[0168] Specifically, the calculation unit 721 calculates a moving average of the variance by using the history of the variances of the multiple partial gradients at the time of the immediately preceding parameter update. t is expressed by the following equation (25).

[0169]

number

[0170] In equation (25), β2 represents a coefficient that controls the moving average. The coefficient that controls the moving average is, for example, a value between zero and 1. The closer the value is to 1, the smaller the variance of the moving average u t is past history t-1 The closer the value is to zero, the greater the variance of the moving average u t is past history t-1 In the following embodiments, β2 is set to 0.9. When t=0, the moving average u t =0.

[0171] Furthermore, the calculation unit 721 calculates the overall gradient using the moving average of the mean and the variance. The overall gradient gt in the fourth embodiment using the moving average of the mean and the variance is expressed by the following formula (26).

[0172]

number

[0173] In other words, the overall gradient is calculated by multiplying the mean value by the inverse of the square root of the sum of the square of the mean value and the moving average of the variance.

[0174] A series of steps for learning a neural network is almost the same as the flowchart in Fig. 4, and therefore a description thereof will be omitted. In the fourth embodiment, a difference is that an update unit updates weights using a moving average of the mean and variance in step ST460 of the flowchart in Fig. 4. In the following, an example will be described in which a partial gradient is calculated using parameters to which no noise is added, using multiple learning devices.

[0175] Fig. 14 is a flowchart showing a specific example of a parameter update process in the fourth embodiment. The flowchart in Fig. 14 relates to the processes of steps ST430, ST520, ST530, ST450, and ST460 in the flowcharts in Fig. 4 and Fig. 5. However, the process of step ST460 has the above-mentioned differences.

[0176] (Step ST1410) The generation unit 711 of the management device 710 generates a mini-batch x t Generate multiple partial data from mini-batches x t In the case where the data is divided into n pieces of partial data, the generating unit 711 divides the partial data x t 1 to partial data x t n Generate up to.

[0177] After step ST1410, the management device 710 assigns each of the plurality of partial data to the plurality of learning devices. In the next step ST1420, the operation of the learning device 720-1 (first node) and the learning device 720-n (nth node) among the plurality of learning devices will be described. Note that the same operation is performed in the other learning devices.

[0178] (Step ST1420) The learning device 720-1 uses pre-assigned model coefficients w t 1 For example, the learning device 720-1 calculates a partial gradient for the partial data x t 1 The partial gradient g corresponding to t1 Calculate the partial gradient g t 1 is expressed by the following equation (27).

[0179]

number

[0180] The learning device 720-n uses pre-assigned model coefficients w t n For example, the learning device 720-n calculates a partial gradient with respect to the partial data x t n The partial gradient g corresponding to t n Calculate the partial gradient g t n is expressed by the following equation (28).

[0181]

number

[0182] After step ST1420, the multiple learning devices 720-1 to 720-n share information on the partial gradients calculated by each of them. In the next step ST1430, the operation of a specific learning device (here, learning device 720-1) will be described.

[0183] (Step ST1430) The learning device 720-1 calculates an average value g of the multiple partial gradients based on the multiple partial gradients of the other learning devices. t and the variance σ for multiple partial gradients 2 (g t ) is calculated. The average value g t and variance σ 2 (g t ) are expressed by the above-mentioned equations (11) and (12), respectively.

[0184] After step ST1430, the information on the mean and variance (gradient information) calculated by learning device 720-1 is shared with the other learning devices. In the following steps ST1440 and ST1450, the operations of learning device 720 will be described without distinguishing between the learning devices.

[0185] (Step ST1440) The learning device 720 calculates the average value g t and variance σ 2 (g t ) moving average u t The global gradient g used in the t-th update is t Calculate the moving average u t and the overall gradient g t is expressed by the above-mentioned equations (25) and (26).

[0186] (Step ST1450) The learning device 720 calculates the overall gradient g t For example, in the case of optimization using Momentum SGD, the learning device 720 updates the weights by applying the above-mentioned formulas (3) and (4).

[0187] FIG. 15 shows an evaluation result including a learning curve using the parameter update process of the fourth embodiment. The evaluation result 1500 in FIG. 15 was obtained by constructing a 50-layer ResNet and verifying it using the ImageNet-1k dataset as a benchmark. The evaluation result 1500 also includes three learning curves 1510 to 1530 verified under different conditions of the optimization method. In addition, the batch size was set to 131k under all conditions.

[0188] The learning curve 1510 is the result of optimization using Momentum SGD with the conventional global gradient (see equation (6)). The prediction accuracy in the learning curve 1510 is 67.7%.

[0189] The learning curve 1520 is a result of optimization by Momentum SGD using the global gradient (see formula (8)) of the first embodiment. The prediction accuracy of the learning curve 1520 is approximately the same as that of the learning curve 1510.

[0190] The learning curve 1530 is a result of optimization by Momentum SGD using the global gradient (see Equation (26)) of the fourth embodiment. The prediction accuracy in the learning curve 1530 is 69.4%.

[0191] Therefore, it is seen that the prediction accuracy regarding the learning curve 1530 is higher than the prediction accuracy of the learning curve 1510. Therefore, by using the method of the fourth embodiment, it is possible to improve the prediction accuracy compared to the conventional methods.

[0192] As described above, the learning system according to the fourth embodiment generates a plurality of partial data from mini-batches of learning data used in a plurality of learning processes for learning parameters of a neural network using an objective function, calculates a partial gradient, which is a gradient for the parameters of the objective function, for each of the plurality of partial data, and updates the parameters based on an average value of the plurality of partial gradients corresponding to the plurality of partial data and a variance for the plurality of partial gradients. Furthermore, the learning device calculates an overall gradient, which is a gradient of the objective function for the mini-batch, by using the average value and the variance, and updates the parameters based on the overall gradient. Furthermore, the learning device calculates the overall gradient by the product of the average value and the reciprocal of the square root of the sum of the square of the average value and the moving average of the variance.

[0193] Therefore, the learning system according to the fourth embodiment can enable efficient learning, similar to the first embodiment.

[0194] (Hardware configuration) 16 is a block diagram illustrating a hardware configuration of a computer according to an embodiment. The computer 1600 includes, as hardware, a central processing unit (CPU) 1610, a random access memory (RAM) 1620, a program memory 1630, an auxiliary storage device 1640, and an input / output interface 1650. The CPU 1610 communicates with the RAM 1620, the program memory 1630, the auxiliary storage device 1640, and the input / output interface 1650 via a bus 1660.

[0195] The CPU 1610 is an example of a general-purpose processor. The RAM 1620 is used by the CPU 1610 as a working memory. The RAM 1620 includes a volatile memory such as a Synchronous Dynamic Random Access Memory (SDRAM). The program memory 1630 stores various programs including a parameter update processing program. As the program memory 1630, for example, a Read-Only Memory (ROM), a part of the auxiliary storage device 1640, or a combination thereof is used. The auxiliary storage device 1640 stores data non-temporarily. The auxiliary storage device 1640 includes a non-volatile memory such as an HDD or SSD.

[0196] The input / output interface 1650 is an interface for connecting to other devices. The input / output interface 1650 is used, for example, to connect to a sound collection device and an output device.

[0197] Each program stored in the program memory 1630 includes computer-executable instructions. When executed by the CPU 1610, the program (computer-executable instructions) causes the CPU 1610 to execute a predetermined process. For example, when executed by the CPU 1610, a parameter update process program causes the CPU 1610 to execute the series of processes described with respect to each part in FIG. 1 and FIG. 7.

[0198] The program may be provided to the computer 1600 in a state where it is stored in a computer-readable storage medium. In this case, for example, the computer 1600 further includes a drive (not shown) for reading data from the storage medium, and acquires the program from the storage medium. Examples of the storage medium include a magnetic disk, an optical disk (CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), a magneto-optical disk (MO, etc.), and a semiconductor memory. Also, the program may be stored in a server on a communication network, and the computer 1600 may download the program from the server using the input / output interface 1650.

[0199] The processing described in the embodiment is not limited to being performed by a general-purpose hardware processor such as the CPU 1610 executing a program, but may be performed by a dedicated hardware processor such as an ASIC (Application Specific Integrated Circuit). The term processing circuit (processing unit) includes at least one general-purpose hardware processor, at least one dedicated hardware processor, or a combination of at least one general-purpose hardware processor and at least one dedicated hardware processor. In the example shown in FIG. 16, the CPU 1610, the RAM 1620, and the program memory 1630 correspond to the processing circuit.

[0200] Therefore, according to each of the above embodiments, learning can be performed efficiently.

[0201] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0202] 100...information processing system, 110...learning device, 111...generation unit, 112...calculation unit, 113...update unit, 114...storage unit, 115...communication unit, 120...information processing device, 121...reception unit, 122...information processing unit, 123...output control unit, 124...storage unit, 125...communication unit, 700...information processing system, 710...management device, 711...generation unit, 712...allocation unit, 713...storage unit, 714...communication unit, 720-1 to N...learning device, 721...calculation unit, 722...update unit, 723...storage unit, 724...communication unit , 730...information processing device, 900...communication pattern, 910...first node, 920...second node, 930...third node, 1300...evaluation results, 1310,1320,1330,1340,1350...learning curve, 1500...evaluation results, 1510,1520,1530...learning curve, 1600...computer, 1610...CPU, 1620...RAM, 1630...program memory, 1640...auxiliary storage device, 1650...input / output interface, 1660...bus, NW...network.

Claims

1. A generation unit that generates a plurality of partial data from mini-batches of training data used in a plurality of training processes for training parameters of a neural network using an objective function; A calculation unit that calculates a partial gradient, which is a gradient with respect to the parameter of the objective function, for each of the plurality of partial data; an update unit that updates the parameters based on an average value of a plurality of partial gradients corresponding to the plurality of partial data and a variance of the plurality of partial gradients; A learning device comprising:

2. The calculation unit calculates a global gradient, which is a gradient of the objective function for the mini-batch, by using the mean and the variance; The update unit updates the parameters based on the overall gradient. The learning device according to claim 1 .

3. The calculation unit calculates the overall gradient by multiplying the average value by the inverse of the square root of the sum of the square of the average value and the variance. The learning device according to claim 2 .

4. The update unit calculates the overall gradient by multiplying the average value by the inverse of the square root of the sum of the square of the average value and the moving average of the variance. The learning device according to claim 2 .

5. the calculation unit calculates noise to be added to the parameters for each of the plurality of partial data, and calculates the partial gradient for the parameters to which the noise has been added; A learning device according to any one of claims 1 to 4.

6. The calculation unit calculates the noise by using a difference between a previous overall gradient calculated by a previous parameter update and a previous partial gradient of each of the previous plurality of partial data used in the previous parameter update. The learning device according to claim 5 .

7. A plurality of learning devices that learn parameters of a neural network using an objective function; and a management device that manages the plurality of learning devices. Equipped with The management device includes: generating a plurality of partial data from mini-batches of training data used in a plurality of training processes for training the parameters; assigning each of the plurality of partial data to each of the plurality of learning devices; Each of the plurality of learning devices Calculating a partial gradient, which is a gradient of the objective function with respect to the parameter, for the allocated partial data; A learning system that updates the parameters based on an average value of a plurality of partial gradients corresponding to the plurality of partial data and a variance of the plurality of partial gradients.

8. the plurality of learning devices communicate with each other to share gradient information regarding the update of the parameters; The learning system according to claim 7.

9. The gradient information includes information of the partial gradients. The learning system according to claim 8.

10. Each of the plurality of learning devices calculates the square of the partial gradient; The gradient information further includes information of the squares of the partial gradients. The learning system according to claim 9.

11. the plurality of learning devices share information on the partial gradients and information on the squares of the partial gradients at different times; The learning system according to claim 10.

12. the plurality of learning devices share information on the partial gradients and information on the squared partial gradients at the same time. The learning system according to claim 10.

13. a particular learning device among the plurality of learning devices calculates a global gradient, which is a gradient of the objective function for the mini-batch, by using the mean value and the variance; Each of the plurality of learning devices updates the parameters based on the global gradient. A learning system according to any one of claims 7 to 12.

14. the specific learning device calculates the overall gradient by multiplying the average value by the inverse of the square root of the sum of the square of the average value and the variance; The learning system according to claim 13.

15. the specific learning device calculates the overall gradient by multiplying the average value by the inverse of the square root of the sum of the square of the average value and the moving average of the variance; The learning system according to claim 13.

16. each of the plurality of learning devices calculates noise to be added to the parameters for the assigned partial data, and calculates the partial gradient for the parameters to which the noise has been added; A learning system according to any one of claims 7 to 15.

17. Each of the plurality of learning devices calculates the noise using a difference between a previous overall gradient calculated by a previous parameter update and a previous partial gradient of the previous partial data used in the previous parameter update.

17. The learning system of claim 16.

18. generating a plurality of partial data from mini-batches of training data to be used in a plurality of training processes for training parameters of a neural network using an objective function; Calculating a partial gradient, which is a gradient of the parameter of the objective function, for each of the plurality of partial data; updating the parameters based on an average value of a plurality of partial gradients corresponding to the plurality of partial data and a variance of the plurality of partial gradients; 23. A computer implemented learning method comprising:

Images