Information processing device, learning method, and program

Abstract

To provide a technique capable of leading to consensus formation at high speed, in distributed learning.SOLUTION: An information processor which is used as a certain node in a plurality of nodes in a distributed learning system for learning a model by information exchange between the plurality of nodes includes: a reception part for receiving a variable updated according to an update rule obtained by applying operator division from a steady point condition indicated by using a positive definite matrix from an adjacent node; and an update part for updating a model parameter of the model by using the variable.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] This invention relates to distributed learning. [Background technology]

[0002] In recent years, neural network models have attracted significant attention in various fields, including image processing and natural language processing. Training these models typically requires the collection of large amounts of training data.

[0003] However, due to privacy concerns and other issues, it is difficult to collect training data containing personal information on a single server (node). Considering such cases, it is being considered to distribute and store data across multiple nodes that can be arbitrarily connected to a network, and to train a model (for example, a neural network model) that can identify all the data with a high recognition rate (see, for example, Non-Patent Document 1). The above type of training is called decentralized learning.

[0004] In distributed learning, the research focuses on developing algorithms for how to achieve consensus among multiple nodes during the learning process. A distributed learning algorithm that minimizes the number of communications and the amount of information exchanged between nodes, resulting in a fast consensus-building process, is considered to have good performance.

[0005] Non-patent document 1 discloses a distributed learning technique called edge-consensus learning (ECL).

[0006] The advantage of ECL is that it can learn robustly against statistical bias (heterogeneity) in the data subsets accumulated at each node. This is because it solves the cost minimization problem under the consensus constraint that the models across nodes are identical. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Niwa, K., Harada, N., Zhang, G., and Kleijn, WB Edge-consensus learning: Deep learning on P2P networks with nonhomogeneous data. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 668-678, 2020. [Non-Patent Document 2] Martens, J. and Grosse, R. Optimizing neural networks with kronecker-factored approximate curvature. In In-ternational conference on machine learning, pp. 2408-2417. PMLR, 2015 [Overview of the project] [Problems that the invention aims to solve]

[0008] The ECL algorithm disclosed in Non-Patent Document 1 is a first-order convergence algorithm. Therefore, the number of communications between nodes is large before consensus is reached and model training is completed. In addition, with ECL, each communication requires the exchange of information (specifically, dual variables) that is roughly the same size as the model, resulting in a large amount of communication. Communication costs account for the majority of the computation and execution time in distributed learning, and it is necessary to reduce these communication costs in order to speed up distributed learning. It should be noted that this issue is not limited to ECL, but can also occur in other distributed learning methods.

[0009] This invention has been made in view of the above points, and aims to provide a technology that enables high-speed consensus building in distributed learning. [Means for solving the problem]

[0010] According to the disclosed technology, in a distributed learning system that trains a model by exchanging information between nodes in multiple nodes, an information processing device used as one of the multiple nodes, A receiving unit receives variables updated according to an update rule obtained by applying operator partitioning to the stationary point conditions expressed using a positive definite matrix, from adjacent nodes. An update unit that updates the model parameters of the model using the aforementioned variables. An information processing device equipped with [this feature] is provided. [Effects of the Invention]

[0011] According to the disclosed technology, distributed learning that can reach consensus quickly can be achieved. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows an example of the overall configuration of a distributed learning system. [Figure 2] This figure shows an example of a network topology. [Figure 3] This diagram shows the update procedure at node i. [Figure 4] This figure shows the experimental results. [Figure 5] This figure shows the experimental results. [Figure 6] This figure shows the experimental results. [Figure 7] This figure shows the experimental results. [Figure 8] This figure shows the experimental results. [Figure 9] This is a diagram illustrating the configuration of a node (information processing unit). [Figure 10] This figure shows an example of the hardware configuration of an information processing device. [Modes for carrying out the invention]

[0013] Hereinafter, embodiments of the present invention (this embodiment) will be described with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the embodiments described below.

[0014] Below, we will describe the Natural Gradient Primal-Dual method (NGPD), a quadratic optimized distributed learning algorithm for speeding up distributed learning. NGPD is a new distributed learning technique based on ECL disclosed in Non-Patent Document 1, and is equivalent to a quadratic optimized version of ECL.

[0015] In the following embodiment, an NGPD based on ECL will be described, but the technology according to the present invention is applicable not only to distributed learning based on NGPD but also to other distributed learning methods.

[0016] (Example of overall system configuration) Figure 1 shows an example of the overall configuration of the distributed learning system in this embodiment. As shown in Figure 1, this system has a configuration in which multiple nodes 100 are connected to a network 200. Communication between nodes is possible via the network 200. The network 200 can be any network, but for example, it is an IP network such as the Internet.

[0017] Each node 100 is a computer (which may also be called an information processing device) with communication capabilities. This computer may be a physical machine or a virtual machine built on the cloud. Furthermore, one or all of the multiple nodes 100 may be edge devices in edge computing. Also, one or all of the multiple nodes 100 may be IoT devices.

[0018] Each node 100 transmits and receives (exchanges) information (e.g., dual variables) with other nodes 100, thereby performing model training. There is no limitation on the type of the model, which may be a neural network or a model other than a neural network. However, the present embodiment assumes that a neural network model is used.

[0019] The plurality of nodes 100 may be connected in any topology. The topology may be, for example, any one of (a) chain, (b) ring, (c) multiple rings, and (d) fully connected graph as shown in FIG. 2. Furthermore, the topology may be in a form other than these.

[0020] Hereinafter, the processing (algorithm) executed by each node 100 for model learning will be described.

[0021] For convenience of description, in the text of the present specification (text that is not an image), characters for vectors and matrices that are generally indicated in bold are shown in normal font. Even in normal font, it is clear from the context that the character represents a vector or a matrix. In addition, for characters representing sets, normal font is also used in the text of the specification. Furthermore, in the text of the specification, symbols intended to be placed above characters are described at the upper left of characters (e.g., - θ). A summary of the symbols used in the present embodiment (Table 1) is shown in FIG. 3.

[0022] (A: Outline of the Embodiment) An outline of the description of the aforementioned Natural Gradient Primal-Dual (NGPD) method will be described.

[0023] <A-1: Formulation and Theoretical Convergence Analysis of NGPD> As will be described in "C: Proposed Method" described later, in the present embodiment, starting with reformulating the primal-dual distributed learning algorithm (ECL), the constrained sum-of-losses minimization problem is solved by adding a function space proximity (FSP) term according to the natural gradient formulation (Grosse, R. Second-order optimization. https: / / www.cs.toronto.edu / rgrosse / courses / csc2541 2021 / .). This FSP term serves to ensure that the DNN output values do not change abruptly due to parameter updates.

[0024] In addition, after deriving the NGPD update rule for solving this constrained sum-of-losses minimization problem and describing the behavior of the averaged parameters of NGPD (the average of all local node parameters), the theoretical convergence analysis thereof will be described. Under several preconditions, the convergence of the averaged parameters to the global minimum for classification or regression tasks is shown. Note that the update rule may also be referred to as an "update law".

[0025] <A-2: Implementation of Practical Algorithm> As will be described in "D: Practical Implementation Using K-FAC" described later, the NGPD method proposed in the present embodiment includes an update of the preconditioning matrix (Gauss-Newton matrix). In order to make the proposed method practical, it is necessary to significantly reduce the computational complexity. In the present embodiment, two algorithm forms (NGPD Form 1 and NGPD Form 2) are presented using K-FAC (Kronecker Factored Approximate Curvature) (Non-Patent Document 2) that either uses or does not use a further approximation of the Gauss-Newton matrix. In particular, it has been experimentally confirmed that NGPD Form 2, which includes a further approximation of the Gauss-Newton matrix, is an algorithm that can achieve the effect of second-order optimization without increasing computational complexity like existing first-order distributed learning algorithms.

[0026] <A-3: Experimental Verification> As described in "E: Numerical Experiments" below, in the present embodiment, experimental verification is performed using an image classification benchmark test. Under the loss function using DNNs with non-uniform data allocation (which probably has adverse curvature), the NGPD method achieves the best test accuracy. In particular, NGPD Form 2 is a practical implementation. In our numerical experiments, no significant increase in training time was observed compared with existing first-order distributed learning algorithms.

[0027] <A-4: Problem Setting, etc.> Hereinafter, the problem setting, notation, and symbols used in the description of the present embodiment will be described.

[0028] Synchronous distributed network: A set V of V local nodes is connected to form an arbitrary graph G(V,E). Here, the set E is a set of E unidirectional edges. The index set of adjacent nodes connected to the i-th node is E i ={j∈V|(i,j)∈E}.

[0029] The cardinality of a set represents the number of elements. That is, V=|V|, E=|E|, E i =|E i |, and Σ V i=1 E i =2E holds. Assuming that the computational performance of the V local nodes is similar, K(≧1) local updates are performed in each communication round {1,..,R}∈r. For transmission of local parameters and their auxiliary variables, synchronous message passing is only allowed between connected nodes. As an example, the algorithm in the present embodiment assumes a single local update (K=1), but multiple local updates (K>1) may also be performed.

[0030] Non-uniform data subsets: In the present embodiment, assuming that a supervised learning task (for example, an image classification task) is to be solved, the data subset consisting of input-label pairs at the i-th local node is {x i ,ti}, which is expressed as follows. The data is statistically non-uniform among local nodes, and Pr(x i ,t i )≠Pr(x j ,t j ) (i≠j). In "E: Numerical Experiments", non-uniform data subsets are prepared by synthetically allocating "D=Σ V i=1 D i " data samples in accordance with the specified concentration parameter that follows the Dirichlet distribution.

[0031] Non-convex loss function: In the present embodiment, DNNs f i are used to solve supervised learning tasks. The neural network architecture is common across all V local nodes, and the loss function is differentiable at any point in the parameter space. As summarized in FIG. 3, {θ1,...,θ V} represents the set of local parameters of the neural network. When the output of the local DNN is expressed as y i =f i (θ i ,x i ), the non-convex loss function is defined by the following formula (1).

[0032]

Mathematical formula

[0033] (B: Regarding the primal-dual method) As preparation for formulating the NGPD update rule described in "C: Proposed Method," we will outline the existing prim-dual method, ECL (Non-Patent Literature 1). For supervised distributed learning, linearly constrained loss sum minimization is formulated as follows:

[0034]

number

[0035] In ECL, the non-convex loss function L i The current parameter θ (cur) i It can be approximated by a convex function as follows:

[0036]

number

[0037]

number

[0038]

number

[0039] To solve the dual problem (3), two operators are defined. One of them is the following operator, which is the derivative of the first term.

[0040]

number

[0041]

number

[0042]

number

[0043]

number

[0044]

number

[0045] As can be seen from Equation (6), the update rule for θ i is first-order optimization. Convergence requires a large number of communication rounds, which is a challenge for distributed learning.

[0046] Hereinafter, the proposed method in the present embodiment will be described. In the proposed method, the natural gradient form (second-order optimization) is incorporated into ECL to smooth the optimization trajectory of local parameters, and achieve faster convergence to a stationary point with a smaller number of communication rounds.

[0047] (C: Proposed Method) As the proposed method, the NGPD method for distributed learning will be described. In "C-1", the formulation of natural gradient versions of the primal problem (Equation (2)) and the dual problem (Equation (3)) by introducing a Function Space Proximity (FSP) term is described. The FSP term constrains the regularization of the output of the neural network, so that the output value does not change sharply due to parameter update. In "C-2", the NGPD update rule is derived by modifying the metric of the parameter space using the Gauss-Newton matrix and solving the natural gradient version of the dual problem. In "C-3", the averaging sequence represented by the following formula will be described.

[0048] [Mathematics] <C-1: Natural Gradient Versions of Primal / Dual Problems> To convert the primal problem (Equation (2)) and its dual problem (Equation (3)) into the natural gradient form, an FSP is first added. This suppresses abrupt changes in function output when local model parameters are updated. Current local parameter θ (cur) i Let the current function output obtained by using be y (cur) i =f i (θ (cur) i ,x i ), and the function output after the local parameter is updated is denoted as y i =f i (θ i ,x i ). The FSP term is defined by the squared L2 norm of the difference between them. The natural gradient version of the primal problem is given by the following Equation (7).

[0049]

Formula

[0050]

Formula

[0051]

Formula

[0052] [Mathematical] This affects the simplification of the update rule. The design of -Gi will be described in "C-2".

[0053] The main problem of the NGPD method in the present embodiment is formulated as follows.

[0054] [Mathematical] The corresponding dual problem is given as follows.

[0055] [Mathematical] Here, Q * i represents the convex conjugate function of Q i . The only difference from the ECL main problem (formula (3)) is the FSP term in Q * i . In the following, the derivation of the update rule for solving formula (10) according to the natural gradient form is described.

[0056] <C-2: Derivation of NGPD Update Rule> Here, the derivation of the NGPD update rule is described. For this purpose, the derivative of the first term of formula (10) is defined as follows.

[0057] [Mathematical] The subdifferential of the second term is as follows.

[0058] [Mathematical] Steady point ν * The following conditions must be met.

[0059]

number

[0060]

number

[0061]

number

[0062] In this embodiment, each node in the communication system updates model parameters, each variable, and performs message passing according to equation (12) above. Here, the i-th node is denoted as node i, and the j-th node adjacent to node i is denoted as node j.

[0063] Equation (12) shows the processing steps that node i will execute. In equation (12), ~ u j|i , and Φ j This represents the variable that node i receives from node j, ~ u i|j , and Φ i This represents the variable that node i sends to node j.

[0064] In equation (12), node i is θ as shown in equation (12). i , ~ u i|j , Φ i Update the information received from node j. ~ u j|i , Φ j Using ~ ν i|j , G i|j Update.

[0065] The differences between NGPD formula (12) and ECL formula (5) can be summarized as (Difference 1)-(Difference 3) below.

[0066] (Difference 1) Local parameter update using Gauss-Newton matrices: θ in equation (12) i The update rule is the generalized quadratic function Q in equation (8). i (θ i ) and dual variable ~ ν i|j It is designed to minimize the sum of normalization terms using . To simplify the update rule, as shown in equation (13) below, Q i (θi ) including - G i Use this.

[0067]

number

[0068]

number

[0069] (Difference 2) Message passing of Gauss-Newton matrices: Dual variables ~ u i|j In addition, the Gauss-Newton matrix Φ i Alternatively, its components are exchanged between connected nodes.

[0070] (Difference 3) The Gauss-Newton matrix is updated to satisfy the constraint: As shown in formula (12), Φ i and Φ j is used to average G i|j as long as G i|j = G j|i (i∈V,j∈E i ) is guaranteed.

[0071] The above is the description of Differences 1 to 3.

[0072] From formula (6), it can be seen that the θ i update rule of NGPD follows the second-order natural gradient method. Therefore, robustness against ill-conditioned curvature can be expected. However, if the computational, memory, and communication costs increase significantly per communication round, the effect of gradient preprocessing becomes meaningless. Accordingly, a practical implementation for the calculation of G i|j is described in "D: Practical Implementation Using K-FAC". Instead of the full gradient ∇L i ( i ), an implementation using the stochastic gradient ∇Li(θ i ,ξ i ,τ i ) and K local updates are also described in "D: Practical Implementation Using K-FAC". The sequence of averaged parameters shown below is described.

[0073]

Mathematical Expression

[0074]

Mathematical Expression

[0075] (D: A practical implementation using K-FAC) A common problem with quadratic optimization methods is the large overhead involved in calculating the preprocessed Gauss-Newton matrix and its inverse. To mitigate this problem, this embodiment uses K-FAC to approximate the local Gauss-Newton matrix. This implementation (pseudocode) is shown as Algorithm1 in Figure 4. Algorithm1 shows two forms (NGPD forms 1 and 2) together. Algorithm1 corresponds to the pseudocode of the program executed on node i.

[0076] Generally, the local Gauss-Newton matrix Φ i is a dense matrix, and calculating its inverse is impractical. In K-FAC, Φ i This can be approximated by layer-by-layer block diagonalization for an N-layer DNN, as shown below.

[0077]

number

[0078]

number

[0079]

number

[0080]

number

[0081]

number

[0082] (E: Numerical experiment) Numerical experiments were conducted to compare NGPD (forms 1 and 2) in the present embodiment with other comparison methods under non-uniform data distribution and DNNs with a loss function (probably with ill-conditioned curvature).

[0083] <E-1:Experimental Setup> Network / Communication: A ring network topology ((V,E)=(8,8)) was used to connect local nodes. R=2,500 rounds of synchronous communication were performed between nodes. The number of local updates was varied at K={32, 64, 128}, and the robustness against this was experimentally investigated.

[0084] Data Distribution: As a supervised learning task, image recognition using CIFAR-10 was selected. A 50,000-sample training dataset was non-uniformly allocated to V=8 nodes. The data subset held by each local node was set to follow a Dirichlet distribution with concentration hyperparameter α. As α approaches zero, the non-uniformity of data increases. To investigate robustness against non-uniform data distribution, three non-uniformity patterns with α={0.5, 1.0, ∞} were used. In the evaluation phase, all local nodes were allowed to access the 10,000-sample test dataset.

[0085] Non-convex Model: Here, two non-convex DNN models, ResNet-18 and VGG-11, are used, and the loss function l i cross-entropy using the softmax function was used as. Note that since the local data subset is statistically biased, the batch normalization layer of ResNet-18 was replaced with a group normalization layer. Additionally, the local Gauss-Newton matrix of the group normalization layer is not computed.

[0086] Comparison Methods: Here, eight methods consisting of four pure primal methods and four primal-dual methods were tested.

[0087] The four pure primal methods are as follows: (1) DSGD, (2) DSGD + Momentum (Gao, H. and Huang, H. Periodic stochastic gradient descent with momentum for decentralized training. arXiv preprint arXiv:2008.10435, 2020), (3) DSGD + Natural Gradient Descent, (4) Decentralized SCAFFOLD (Liu, Y. Variance reduction in decentralized training over heterogeneous data. 2021).

[0088] The four primal-dual methods are as follows: (5) ECL (Douglas-Rachford splitting), (6) NGPD Form 1, (7) NGPD Form 2, (8) as the reference method, - G i NGPD with diagonal matrix approximation of .

[0089] For each method, hyperparameter tuning for μ, η i|j , and batch size was performed in advance. For (5)-(8), η i|j =1 / (μK(2E) / V) was used, which satisfies η i|j =η j|i . To fairly compare seven methods with different computational costs, the training time including local computation and message passing was recorded.

[0090] Computational environment: A server equipped with 8 GPUs (NVIDIA A100 for NVLink 40GiB HBM2) was used, and Gloo was used for message passing.

[0091] <E-2: Experimental Results> Averaging parameter -The evaluation results for θ are shown in Figures 5 to 8. ResNet-18 was used in Figures 5 and 6, and VGG-11 was used in Figures 7 and 8. In Figures 5(a1) and 7(b1), the learning curves for test accuracy are shown with heterogeneous data distribution (α=1.0) and multiple local updates (K=32) fixed. The learning curves for NGPD forms 1 and 2 are smooth, and the best test accuracy was achieved with NGPD form 1 of ResNet-18 or form 2 of VGG-11.

[0092] In Figures 5(a2) and 7(b2), the horizontal axis was changed from communication rounds to learning time including local computation and message passing in order to fairly compare the computationally intensive first-order method with the second-order NGPD according to this embodiment.

[0093] While the training time for NGPD format 1 was significantly longer than other methods, the training time for NGPD format 2, which includes an additional approximation of the Gauss-Newton matrix, was not significantly longer compared to other first-order optimization methods. Therefore, from the standpoint of computational efficiency, NGPD format 2 is more suitable than format 1.

[0094] In Figures 6(a3) and 8(b3), to investigate robustness to data heterogeneity, the best test accuracy was compared for each α with K=32 fixed and communication rounds of R=2,500. The best test accuracy of the primal-dual methods, including NGPD forms 1 and 2, was not significantly affected by data heterogeneity α. This is because the parameter constraints of the primal-dual methods (2)(7) effectively provide robustness to data heterogeneity.

[0095] Following NGPD formats 1 and 2, ECL and, - G iNGPD using diagonal matrix approximation also achieved relatively high test accuracy under data heterogeneity (α={0.5,1.0}). However, it required more communication rounds to obtain a certain level of test accuracy compared to NGPD. This difference clearly demonstrates the importance of applying quadratic natural gradient methods to distributed learning. On the other hand, pure primal methods, namely distributed SCAFFOLD and DSGD (with natural gradient or momentum), did not perform well when data heterogeneity was large. This is because their theoretical convergence proofs require the assumption of data heterogeneity, and in this experiment, we tested them under conditions of large data heterogeneity (α={0.5,1.0}).

[0096] <Eのまとめ> As explained above, the update rules of the NGPD method allow for matching local parameters without abruptly changing the output values ​​of the neural network when parameters are updated. Furthermore, the learning curves of NGPD forms 1 and 2 are smooth, and the best test accuracy for multiple tasks was obtained using NGPD forms 1 and 2, even under DNNs with data heterogeneity. In addition, NGPD form 2 was found to be more computationally efficient than NGPD form 1.

[0097] (Example of device configuration) Figure 9 shows an example configuration of an information processing device 100 used as a node in a distributed learning system that executes the NGPD method.

[0098] As shown in Figure 9, the information processing device 100 includes a transmission unit 110, a reception unit 120, an update unit 130, and a storage unit 140.

[0099] The information processing device 100 is an information processing device used as one of the nodes in a distributed learning system that learns a model through information exchange between nodes in a plurality of nodes. This distributed learning system performs distributed learning using the NGPD method.

[0100] The receiving unit 120 receives variables from neighboring nodes that have been updated according to an update rule obtained by applying an operator partition to the steady-state conditions represented using a positive definite matrix. The updating unit 130 updates the model parameters of the model using the variables. The transmitting unit 110 transmits the variables updated by the updating unit 130 according to the update rule to the neighboring nodes.

[0101] The storage unit 140 stores received data, data calculated by the update unit 30, data to be transmitted, and the like.

[0102] (Example hardware configuration) The information processing device 100 described in this embodiment can be realized, for example, by having a computer execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0103] In other words, the device can be realized by using hardware resources such as the CPU and memory built into a computer to execute a program corresponding to the processing performed by the device. The program can be recorded on a computer-readable recording medium (such as portable memory), saved, and distributed. It can also be provided via a network, such as the Internet or email.

[0104] Figure 10 shows an example of the hardware configuration of the computer described above. The computer in Figure 10 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., all of which are interconnected by bus B. The computer may also be equipped with a GPU.

[0105] The program that enables processing on the computer is provided, for example, on a recording medium 1001 such as a CD-ROM or memory card. When the recording medium 1001 containing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001; it may also be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files and data.

[0106] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when a program startup command is received. The CPU 1004 implements the functions related to the information processing device 100 according to the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) etc., generated by a program. The input device 1007 consists of a keyboard and mouse, buttons, or a touch panel, etc., and is used to input various operation commands. The output device 1008 outputs the calculation results.

[0107] (Summary of the embodiments) A summary of this embodiment is shown in (1) to (5) below.

[0108] (1) In this embodiment, each node constituting the communication system updates variables / parameters according to an update rule (including approximations) that satisfies the steady-state condition of equation (11).

[0109] As mentioned above, the steady-state condition for the conventional method (ECL) is given by equation (4). That is, equation (11) is obtained by modifying equation (4) using a positive definite matrix G. As mentioned above, G is a submatrix (positive definite matrix) G i|j It is a block diagonalization matrix composed of G i|j =G j|iThis satisfies the following constraint.

[0110] (2) More specifically, each node constituting the communication system updates variables / parameters using equation (12) (or an update rule derived from equation (12)), which is the basic form of the update rule that satisfies the steady-state condition of equation (11). As mentioned above, equation (12) is obtained by decomposing the steady-state condition of equation (11) using Douglas-Rachford Splitting. Equation (12) allows G i|j =G j|i Variable updates are possible while adhering to this constraint.

[0111] The algorithm (processing procedure) involves alternating between updating parameters / variables once at the local node and communicating (message passing) once between connected nodes.

[0112] (3) θ in equation (12) i The update rule (the first line of equation (12)) may be replaced with equation (14). Note that the values ​​included in equation (14) may be calculated using equation (13).

[0113] (4) Φ in equation (12) i This can also be approximated and calculated using equation (16) with K-FAC.

[0114] (5) The algorithm in equation (12) may be modified and implemented as in Algorithm1. Note that there are three changes in Algorithm1 from equation (12). The first change is that after updating K times at the local node, communication (message passing) is performed between connected nodes. The second change is θ i The update rule (the first line of equation (12)) has been replaced with equation (14).

[0115] The third change is that, in calculating the stochastic gradient g, as described in the function gCalc, two calculation methods (NGPD format 1 and format 2) have been introduced, and both can be used.

[0116] Both methods use K-FAC to introduce quadratic optimization into distributed learning in a lightweight manner. NGPD format 1 satisfies the stationary point condition of equation (11), but the computational and memory requirements are somewhat high. On the other hand, NGPD format 2 is an approximation method that does not strictly satisfy the stationary point condition of equation (11), but the computational and memory requirements are significantly lower than NGPD format 2, as is reflected in the experimental results.

[0117] (Effects of the embodiment) By incorporating quadratic optimization (natural gradient method) into distributed learning using the technology of this embodiment, it becomes possible to reach a consensus quickly in distributed learning.

[0118] The following additional information is disclosed regarding the embodiments described above.

[0119] <Note> (Additional note 1) In a distributed learning system that trains a model through information exchange between nodes in multiple nodes, an information processing device used as one of the multiple nodes, Memory and At least one processor connected to the memory, Includes, The aforementioned processor, Variables updated according to the update rules obtained by applying operator partitioning to the stationary point conditions expressed using positive definite matrices are received from neighboring nodes. The model parameters of the model are updated using the aforementioned variables. Information processing device. (Additional note 2) Let the aforementioned node be the i-th node, the adjacent node be the j-th node, and the positive definite matrix at the aforementioned node be G i|j This is expressed as G, and the positive definite matrix in the adjacent node is G j|i When expressed as, G i|j =G j|i satisfies The information processing device described in Appendix 1. (Additional note 3) The processor sends the updated variable according to the update rule to the neighboring node. The information processing device described in Appendix 1. (Additional note 4) The update rule for updating the aforementioned model parameters is a rule that updates the model parameters with an expression that includes a term obtained by multiplying the gradient by the inverse of the averaged positive definite matrix. The information processing device described in Appendix 1. (Additional note 5) The processor updates the positive definite matrix using a Gauss-Newton matrix approximated by K-FAC (Kronecker Factored Approximate Curvature). The information processing device described in Appendix 1. (Additional note 6) The processor performs the calculation of the stochastic gradient of the model for each layer of the model, or updates the positive definite matrix for each layer of the model. The information processing device described in Appendix 5. (Additional note 7) In a distributed learning system that learns a model through information exchange between nodes in multiple nodes, a learning method is performed by an information processing device used as one of the multiple nodes, A receiving step in which variables updated according to an update rule obtained by applying operator partitioning to stationary point conditions expressed using a positive definite matrix are received from an adjacent node, An update step to update the model parameters of the model using the aforementioned variables. A learning method that includes [the following features]. (Additional note 8) A non-temporary storage medium storing a program for causing a computer to function as a component of an information processing device described in any one of the appendices 1 through 6.

[0120] Although this embodiment has been described above, the present invention is not limited to this specific embodiment, and various modifications and changes are possible within the scope of the gist of the invention as described in the claims. [Explanation of Symbols]

[0121] 100 Information Processing Devices 110 Transmitter 120 Receiver 130 Update Department 140 Storage section 200 Networks 1000 drive unit 1001 Recording media 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. In a distributed learning system that trains a model through information exchange between nodes in multiple nodes, an information processing device used as one of the multiple nodes, A receiving unit receives variables updated according to an update rule obtained by applying operator partitioning to the stationary point conditions expressed using a positive definite matrix, from adjacent nodes. An update unit that updates the model parameters of the model using the aforementioned variables. An information processing device equipped with the following features.

2. Let the aforementioned node be the i-th node, the adjacent node be the j-th node, and let the positive definite matrix at the node be G ｉ｜ｊ This is expressed as G, and the positive definite matrix in the adjacent node is G ｊ｜ｉ When expressed as, G ｉ｜ｊ = G ｊ｜ｉ satisfies The information processing apparatus according to claim 1.

3. A transmission unit transmits the variable updated by the update unit according to the update rule to the adjacent node. The information processing apparatus according to claim 1, further comprising:

4. The update rule for updating the aforementioned model parameters is a rule that updates the model parameters with an expression that includes a term obtained by multiplying the gradient by the inverse of the averaged positive-definite matrix. The information processing apparatus according to claim 1.

5. The update unit updates the positive definite matrix using a Gauss-Newton matrix approximated by K-FAC (Kronecker Factored Approximate Curvature). The information processing apparatus according to claim 1.

6. The update unit performs the calculation of the stochastic gradient of the model for each layer of the model, or updates the positive definite matrix for each layer of the model. The information processing apparatus according to claim 5.

7. In a distributed learning system that learns a model through information exchange between nodes in multiple nodes, a learning method is performed by an information processing device used as one of the multiple nodes, A receiving step in which variables updated according to an update rule obtained by applying operator partitioning to stationary point conditions expressed using a positive definite matrix are received from an adjacent node, An update step to update the model parameters of the model using the aforementioned variables. A learning method that includes [the following features].

8. A program for causing a computer to function as a component of an information processing device described in any one of claims 1 to 6.

Images