Information processor, learning method, and program

Abstract

To provide a technique capable of leading to consensus formation at high speed, in distributed learning.SOLUTION: An information processor which is used as a certain node in a plurality of nodes in a distributed learning system for learning a model by information exchange between the plurality of nodes includes: a reception part for receiving a variable updated according to an update rule obtained by applying operator division from a steady point condition indicated by using a positive definite matrix from an adjacent node; and an update part for updating a model parameter of the model by using the variable.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to distributed learning. [Background technology]

[0002] In recent years, models such as neural networks have attracted much attention in various fields such as image processing and natural language processing. In order to train a model, it is usually necessary to collect a large amount of training data.

[0003] However, due to privacy issues, etc., it is difficult to collect learning data that includes personal information on a single server (node). In consideration of such cases, studies have been conducted to realize learning of a model (for example, a neural network model) that can classify all data with a high recognition rate by distributing and storing data on multiple nodes that can be arbitrarily connected to a network (for example, Non-Patent Document 1). The above learning is called decentralized learning.

[0004] In distributed learning, the main research topic is to develop an algorithm for determining how to reach a consensus among multiple nodes while learning. It can be said to perform well as a distributed learning algorithm that can reach a consensus quickly, with fewer communications between nodes and less amount of information being communicated.

[0005] Non-Patent Document 1 discloses a distributed learning technique called edge-consensus learning (ECL).

[0006] The advantage of ECL is that it can learn robustly against the statistical bias (heterogeneity) of the data subsets accumulated in the nodes. This is because it solves the cost minimization problem under the consensus-based constraint that the models between nodes are consistent. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Niwa, K., Harada, N., Zhang, G., and Kleijn, WB Edge-consensus learning: Deep learning on P2P networks with nonhomogeneous data. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 668-678, 2020. [Non-Patent Document 2] Martens, J. and Grosse, R. Optimizing neural networks with kronecker-factored approximate curvature. In In-ternational conference on machine learning, pp. 2408-2417. PMLR, 2015 Summary of the Invention [Problem to be solved by the invention]

[0008] The ECL disclosed in Non-Patent Document 1 is a linear convergence algorithm. Therefore, the number of communications between nodes increases until consensus is reached and model learning is completed. Furthermore, in ECL, information (specifically, dual variables) of a size roughly equivalent to the size of the model needs to be exchanged between nodes in each communication, resulting in a large amount of communication. The communication cost in distributed learning accounts for the majority of the computational volume and execution time, and in order to speed up distributed learning, it is necessary to reduce this communication cost. Note that this issue is not limited to ECL, but can also occur in other distributed learning.

[0009] The present invention has been made in consideration of the above points, and has an object to provide a technique that enables rapid consensus building in distributed learning. [Means for solving the problem]

[0010] According to the disclosed technology, there is provided an information processing device used as a node among a plurality of nodes in a distributed learning system that learns a model by information exchange between the plurality of nodes, the information processing device comprising: a receiving unit that receives, from an adjacent node, a variable updated according to an update rule obtained by applying operator splitting from a stationary point condition expressed using a positive definite matrix; an update unit that updates model parameters of the model using the variables; An information processing device comprising: Effect of the Invention

[0011] According to the disclosed technology, distributed learning that quickly reaches consensus can be realized. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an example of the overall configuration of a distributed learning system. [Diagram 2] FIG. 1 illustrates an example of a network topology. [Diagram 3] FIG. 13 illustrates an update procedure at node i. [Figure 4] FIG. 13 is a diagram showing experimental results. [Diagram 5] FIG. 13 is a diagram showing experimental results. [Figure 6] FIG. 13 is a diagram showing experimental results. [Figure 7] FIG. 13 is a diagram showing experimental results. [Figure 8] FIG. 13 is a diagram showing experimental results. [Figure 9] FIG. 2 is a configuration diagram of a node (information processing device). [Figure 10] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0014] In the following, we will explain the Natural Gradient Primal-Dual method (NGPD), which is a quadratically optimized distributed learning algorithm for speeding up distributed learning. NGPD is a new distributed learning technology based on ECL disclosed in Non-Patent Document 1, and corresponds to a quadratically optimized version of ECL.

[0015] In the following embodiment, an NGPD based on ECL will be described, but the technology according to the present invention is not limited to distributed learning based on NGPD and can be applied to other types of distributed learning.

[0016] (Example of overall system configuration) An example of the overall configuration of a distributed learning system according to this embodiment is shown in Fig. 1. As shown in Fig. 1, this system has a configuration in which a plurality of nodes 100 are connected to a network 200. Communication between the nodes is possible via the network 200. The network 200 may be any network, and is, for example, an IP network such as the Internet.

[0017] Each node 100 is a computer (which may be called an information processing device) having a communication function. The computer may be a physical machine or a virtual machine constructed on a cloud. Any or all of the multiple nodes 100 may be edge devices in edge computing. Any or all of the multiple nodes 100 may be IoT devices.

[0018] Learning of the model is performed by each node 100 transmitting and receiving (exchanging) information (for example, dual variables) with other nodes 100. The type of the model is not limited, and it may be a neural network or a model other than a neural network. However, in the present embodiment, it is assumed that a neural network model is used.

[0019] The plurality of nodes 100 may be connected in any topology. The topology may be, for example, any of (a) chain, (b) ring, (c) multi-ring, and (d) fully connected graph as shown in FIG. 2. Also, the topology may be in other forms than these.

[0020] Hereinafter, the process (algorithm) executed by each node 100 for model learning will be described.

[0021] For convenience of description, in the text of this specification (text that is not an image), vectors and matrix characters that are generally shown in bold are shown in normal font. Even in normal font, it is clear from the context that it is a character indicating a vector or a matrix. In addition, for characters representing sets, normal font is used in the text of the specification. Also, in the text of the specification, symbols intended to be placed above the characters are described at the upper left of the characters (e.g., - θ). A summary of the symbols used in the present embodiment (Table 1) is shown in FIG. 3.

[0022] (A: Outline of the embodiment) An outline of the above-described explanation of the natural gradient principal dual (NGPD) method will be described.

[0023] <A-1: Formulation of NGPD and theoretical convergence analysis> As will be described in the "C: Proposed Method" described later, in this embodiment, starting from reformulating the main dual distributed learning algorithm (ECL), a function space proximity (FSP) term that follows the natural gradient form (Grosse, R. Second-order optimization. https: / / www.cs.toronto.edu / rgrosse / courses / csc2541 2021 / .) is added to solve the constrained loss sum minimization problem. This FSP term serves to ensure that the DNN output values do not change abruptly with parameter updates.

[0024] Also, after deriving the NGPD update rule for solving this constrained loss sum minimization problem and explaining the behavior of the averaging parameter of NGPD (the average of all local node parameters), the theoretical convergence analysis thereof will be described. Under some preconditions, the convergence of the averaging parameter to the global minimum for classification or regression tasks is shown. Note that the update rule may also be referred to as the "update law".

[0025] <A-2: Implementation of Practical Algorithm> As will be described in the "D: Practical Implementation Using K-FAC" described later, in the NGPD method proposed in this embodiment, the update of the preprocessing matrix (Gauss-Newton matrix) is included. However, in order to make the proposed method practical, it is necessary to significantly reduce the computational complexity. In this embodiment, two algorithmic forms (NGPD Form 1 and 2) are shown using K-FAC (Kronecker Factored Approximate Curvature) (Non-Patent Document 2) with (or without) further approximation of the Gauss-Newton matrix. In particular, it has been experimentally confirmed that NGPD Form 2, which includes further approximation of the Gauss-Newton matrix, is an algorithm that can achieve the effect of second-order optimization without increasing the computational complexity like existing first-order distributed learning algorithms.

[0026] <A-3: Experimental Verification> As will be described in the "E: Numerical Experiment" described later, in this embodiment, experimental verification was performed using an image classification benchmark test. Under the loss function using DNNs with non-uniform data assignment (presumably with poor-condition curvature), the highest test accuracy was obtained by the NGPD method. In particular, NGPD form 2 is a practical implementation. In our numerical experiment, no significant increase in learning time was observed compared with existing first-order distributed learning algorithms.

[0027] <A-4: Problem setting, etc.> Hereinafter, the problem setting, notation, and symbols used in the description of this embodiment will be described.

[0028] Synchronous distributed network: A set V of V local nodes is connected to form an arbitrary graph G(V, E). Here, the set E is a set of E unidirectional edges. The index set of adjacent nodes connected to the i-th node is E i ={j ∈ V|(i, j) ∈ E}.

[0029] The density of a set represents the number of elements. That is, V = |V|, E = |E|, E i = |E i |, and Σ V i=1 E i = 2E. Assuming that the computational performance of the V local nodes is similar, K (≥ 1) local updates are performed in each communication round {1,.., R} ∈ r. For the transmission of local parameters and their auxiliary variables, synchronous message passing is possible only between connected nodes. In the algorithm of this embodiment, as an example, a single local update (K = 1) is assumed, but multiple local updates (K > 1) may be performed.

[0030] Non-uniform data subset: In this embodiment, assuming that a supervised learning task (for example, an image classification task) is solved, the data subset consisting of the pair of input and label at the i-th local node is {x i , ti}. The data is statistically heterogeneous among local nodes, and Pr(x i ,t i ) ≠ Pr(x j ,t j ) (i ≠ j). In "E: Numerical Experiments", the non-uniform data subset is distributed according to the Dirichlet distribution D = Σ V i=1 D i It is prepared by synthetically allocating " data samples.

[0031] Non-convex loss functions: In this embodiment, we use DNNs f i The neural network architecture is common across V local nodes, and the loss function is differentiable at any point in the parameter space. As summarized in Figure 3, V} represents the set of local parameters of the neural network. If the output of the local DNN is y i =f i (θ i ,x i ), the non-convex loss function is defined by the following equation (1):

[0032]

number

[0033] (B: Primal-dual method) As a preparation for formulating the NGPD update rule described in "C: Proposed Method", we will provide an overview of the existing primal-dual method, ECL (Non-Patent Document 1). For supervised distributed learning, linearly constrained sum loss minimization is formulated as follows:

[0034]

number

[0035] In ECL, a non-convex loss function L i is the current parameter θ (cur) i It is approximated by a convex function as follows:

[0036]

number

[0037]

number

[0038]

number

[0039] To solve the dual problem (3), two operators are defined. One of them is the following operator, which is the derivative of the first term:

[0040]

number

[0041]

number

[0042]

number

[0043]

number

[0044]

number

[0045] As can be seen from Equation (6), the update rule for θ i is first-order optimization. To converge, many communication rounds are required, which is a challenge for distributed learning.

[0046] Hereinafter, the proposed method in this embodiment will be described. In the proposed method, the natural gradient form (second-order optimization) is incorporated into ECL to smooth the optimization trajectory of the local parameters and achieve faster convergence to a stationary point with a smaller number of communication rounds.

[0047] (C: Proposed Method) This is a proposed method, and the NGPD method for distributed learning will be described. In "C-1", the formulation of the natural gradient version of the primal problem (Equation (2)) and the dual problem (Equation (3)) by introducing the Function Space Proximity (FSP) term will be described. The FSP term restricts the regularization of the output of the neural network so that the output value does not change abruptly due to parameter updates. In "C-2", the NGPD update rule is derived by modifying the metric in the parameter space using the Gauss-Newton matrix and solving the natural gradient version of the dual problem. In "C-3", the averaging sequence shown in the following equation will be described.

[0048] [Number] <C-1: Natural Gradient Version of the Primal / Dual Problem> To convert the primal problem (equation (2)) and its dual problem (equation (3)) into a natural gradient form, we first add FSP, which suppresses abrupt changes in the function output when the local model parameters are updated. The current local parameters θ (cur) i The current function output using y (cur) i =f i (θ (cur) i ,x i ) and the function output after the local parameters are updated is denoted as y i =f i (θ i ,x i ) The FSP terms are defined by the squared L2 norm of their difference. The natural gradient version of the primal problem is given by the following equation (7).

[0049]

number

[0050]

number

[0051]

number

[0052]

Equation

[0053] The main problem of the NGPD method in this embodiment is formulated as follows.

[0054]

Equation

[0055]

Equation

[0056] <C-2: Derivation of the NGPD Update Rule> Here, the derivation of the NGPD update rule will be explained. For this purpose, the differential of the first term of Equation (10) is defined as follows.

[0057]

Equation

[0058]

Equation

[0059]

number

[0060]

number

[0061]

number

[0062] Each node in the communication system in this embodiment updates model parameters, updates each variable, and passes messages according to the above formula (12). Here, the i-th node is node i, and the j-th node adjacent to node i is node j.

[0063] Equation (12) shows the procedure executed by node i. In equation (12), ~ u j|i , and Φ j represents the variables that node i receives from node j, ~ u i|j , and Φ i represents the variable that node i sends to node j.

[0064] In equation (12), node i is determined by θ i , ~ u i|j , Φ i and update the received ~ u j|i , Φ j Using ~ ν i|j , G i|j Update.

[0065] The differences between NGPD equation (12) and ECL equation (5) can be summarized as (Difference 1)-(Difference 3) below.

[0066] (Difference 1) Local parameter update using Gauss-Newton matrices: θ in Equation (12) i The update rule for is the generalized quadratic function Q in Eq. (8). i (θ i ) and the dual variable ~ ν i|j In order to simplify the update rule, we use the following formula (13): i (θi ) - G i is used.

[0067]

number

[0068]

number

[0069] (Difference 2) Message passing for Gauss-Newton matrices: Dual Variables ~ u i|j as well as the Gauss-Newton matrix Φ i Or its components are exchanged between connected nodes.

[0070] (Difference point 3) The Gauss-Newton matrix is updated to satisfy the constraints: As shown in Equation (12), as long as G is updated with the average of Φ i and Φ j , G i|j = G i|j = G j|i (i ∈ V, j ∈ E i ) is guaranteed.

[0071] The above is the explanation of difference points 1 to 3.

[0072] From Equation (6), it can be seen that the θ update rule of NGPD follows the second-order natural gradient method. Therefore, robustness against ill-conditioned curvature can be expected. However, if the computational, memory, and communication costs increase significantly for each communication round, the effect of gradient preprocessing becomes meaningless. Therefore, the practical implementation for the calculation of G i will be explained in "D: Practical implementation using K-FAC". The implementation using the stochastic gradient ∇Li(θ i|j ), ξ i ( i ) instead of the full gradient ∇L i , ξ i , τ i ) and the K local updates will also be explained in "D: Practical implementation using K-FAC". The sequence of averaged parameters shown below will be explained.

[0073]

Number

[0074]

Number

[0075] (D: Practical implementation using K-FAC) A common problem with quadratic optimization methods is the large overhead involved in computing the pre-processing Gauss-Newton matrix and its inverse. To alleviate this problem, in this embodiment, we use K-FAC to approximate the local Gauss-Newton matrix. This implementation (pseudo code) is shown in Figure 4 as Algorithm 1. Two forms (NGPD forms 1 and 2) are shown together in Algorithm 1. Algorithm 1 corresponds to the pseudo code of the program executed on node i.

[0076] In general, the local Gauss-Newton matrix Φ i is a dense matrix, and it is impractical to compute its inverse. In K-FAC, Φ i is approximated by layer-wise block diagonalization for an N-layer DNN as follows:

[0077]

number

[0078]

number

[0079]

number

[0080]

number

[0081]

number

[0082] (E: Numerical Experiments) A numerical experiment was conducted to compare NGPD (Formats 1 and 2) in this embodiment with other comparative methods under non-uniform data distribution and under DNNs with a loss function (presumably with a curvature under adverse conditions).

[0083] <E-1: Experimental Setup> Network / Communication: A ring network topology ((V, E) = (8, 8)) was used to connect the local nodes. R = 2,500 synchronous communication rounds were executed among the nodes. The number of local updates was changed with K = {32, 64, 128}, and its robustness was investigated experimentally.

[0084] Data Distribution: For the supervised learning task, image recognition using CIFAR-10 was selected. A training dataset of 50,000 was unevenly allocated to V = 8 nodes. The data subsets held by each local node were made to follow a Dirichlet distribution with the concentration hyperparameter α. As α approaches zero, the non-uniformity of the data increases. To investigate the robustness against non-uniform data distribution, three non-uniformity patterns of α = {0.5, 1.0, ∞} were used. In the evaluation phase, all local nodes were permitted to access a test dataset of 10,000.

[0085] Non-Convex Model: Here, two non-convex DNN models, ResNet-18 and VGG-11, were used, and cross-entropy using the softmax function was used as the loss function l i Since the local data subsets are statistically biased, the batch normalization layer of ResNet-18 was replaced with a group normalization layer. Also, the local Gaussian-Newton matrix of the group normalization layer was not calculated.

[0086] Comparison Methods: Here, eight methods consisting of four pure primal methods and four primal-dual methods were tested.

[0087] The four pure main methods are as follows: (1) DSGD, (2) DSGD + momentum (Gao, H. and Huang, H. Periodic stochastic gradient descent with momentum for decentralized training. arXiv preprint arXiv:2008.10435, 2020), (3) DSGD + natural gradient descent, (4) decentralized SCAFFOLD (Liu, Y. Variance reduction in decentralized training over heterogeneous data. 2021).

[0088] The four main dual methods are as follows: (5) ECL (Douglas - Rachford splitting), (6) NGPD form 1, (7) NGPD form 2, (8) NGPD with diagonal matrix approximation of - G i as a reference method.

[0089] For each method, hyperparameter tuning for μ, η i|j , and the batch size was performed in advance. For (5) - (8), η i|j = 1 / (μK(2E) / V) was used. This satisfies η i|j = η j|i . To fairly compare the seven methods with different computational costs, the learning time including local computation and message passing was recorded.

[0090] Computing environment: A server equipped with eight GPUs (NVIDIA A100 for NVLink 40GiB HBM2) was used, and Gloo was used for message passing.

[0091] <E - 2: Experimental results> Averaged parameter -The evaluation results of θ are shown in Figures 5 to 8. ResNet-18 was used in Figures 5 and 6, and VGG-11 was used in Figures 7 and 8. Figures 5(a1) and 7(b1) show the learning curves of test accuracy with a fixed non-uniform data distribution (α = 1.0) and multiple local updates (K = 32). The learning curves of NGPD forms 1 and 2 are smooth, and the highest test accuracy was achieved by NGPD form 1 with ResNet-18 or form 2 with VGG-11.

[0092] In Fig. 5(a2) and Fig. 7(b2), in order to fairly compare the first-order method, which requires a light computational load, with the second-order NGPD according to the present embodiment, the horizontal axis is changed from communication rounds to the learning time including local computation and message passing.

[0093] As for NGPD form 1, its training time was significantly longer than other methods, but the training time of NGPD form 2, which includes an additional approximation to the Gauss-Newton matrix, was not significantly longer than other first-order optimization methods. Therefore, from the viewpoint of computational efficiency, NGPD form 2 is more suitable than form 1.

[0094] In Fig. 6(a3) and Fig. 8(b3), we compare the best test accuracy for each α with K=32 fixed and R=2,500 communication rounds to investigate the robustness to data heterogeneity. The best test accuracy of the primal-dual methods including NGPD forms 1 and 2 was not significantly affected by the data heterogeneity α. This is because the parameter constraints of the primal-dual methods (2)(7) effectively provide robustness to data heterogeneity.

[0095] Following NGPD formats 1 and 2, ECL and - G iNGPD, which uses a diagonal matrix approximation of , also achieved relatively high test accuracy under data heterogeneity (α = {0.5, 1.0}). However, to achieve a certain test accuracy, more communication rounds were required compared to NGPD. This difference clearly shows the importance of applying second-order natural gradient methods to distributed learning. On the other hand, the pure primal methods, i.e., distributed SCAFFOLD and DSGD (with natural gradient or momentum), did not perform well when data heterogeneity was large. This is because their theoretical convergence proofs require the assumption of data heterogeneity, and in this experiment, we tested in a situation where data heterogeneity was large (α = {0.5, 1.0}).

[0096] <Eのまとめ> As described above, the update rule of the NGPD method allows the local parameters to match without abrupt changes in the output value of the neural network due to parameter updates. In addition, the learning curves of NGPD forms 1 and 2 are smooth, and the best test accuracy for multiple tasks is obtained by NGPD forms 1 and 2 even under DNNs with data heterogeneity. In addition, it is found that NGPD form 2 is more computationally efficient than NGPD form 1.

[0097] (Device configuration example) FIG. 9 shows an example of the configuration of an information processing device 100 used as a node in a distributed learning system that executes the NGPD method.

[0098] As shown in FIG. 9, the information processing device 100 includes a transmitting unit 110, a receiving unit 120, an updating unit 130, and a storage unit 140.

[0099] The information processing device 100 is an information processing device used as one of a plurality of nodes in a distributed learning system that performs model learning by information exchange between the plurality of nodes. This distributed learning system performs distributed learning by the NGPD method.

[0100] The receiving unit 120 receives variables updated according to an update rule obtained by applying operator splitting from a stationary point condition expressed using a positive definite matrix from an adjacent node. The updating unit 130 updates model parameters of the model using the variables. The transmitting unit 110 transmits the variables updated by the updating unit 130 according to the update rule to the adjacent node.

[0101] The storage unit 140 stores received data, data calculated by the update unit 30, data to be transmitted, and the like.

[0102] (Hardware configuration example) The information processing device 100 described in this embodiment can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0103] That is, the device can be realized by executing a program corresponding to the processing performed by the device, using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0104] Fig. 10 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 10 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are connected to each other via a bus B. The computer may further include a GPU.

[0105] The program for realizing the processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0106] The memory device 1003 reads out and stores the program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the information processing device 100 in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network or the like. The display device 1006 displays a GUI (Graphical User Interface) or the like according to a program. The input device 1007 is composed of a keyboard and mouse, buttons, a touch panel, or the like, and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0107] (Summary of the embodiment) The present embodiment is summarized in the following (1) to (5).

[0108] (1) In this embodiment, each node constituting the communication system updates variables / parameters according to an update rule (including approximation) that satisfies the stationary point condition of equation (11).

[0109] As mentioned above, the stationary point condition of the conventional method (ECL) is Equation (4). In other words, Equation (11) is modified from Equation (4) using the positive definite matrix G. As mentioned above, G is a small matrix (positive definite matrix) G i|j is the block diagonalization matrix composed of G i|j =G j|iThe following constraint is satisfied.

[0110] (2) More specifically, each node constituting the communication system updates variables / parameters using equation (12), which is the basic form of the update rule that satisfies the stationary point condition of equation (11) (or an update rule derived from equation (12)). As mentioned above, equation (12) is obtained by decomposing the stationary point condition of equation (11) using Douglas-Rachford Splitting. From equation (12), G i|j =G j|i It is possible to update variables while observing these constraints.

[0111] The algorithm (processing procedure) involves alternating between updating parameters / variables once at the local node and communicating (message passing) once between connected nodes.

[0112] (3) θ in Eq. (12) i The update rule for (the first line of equation (12)) may be replaced by equation (14). Note that included in equation (14) may be calculated using equation (13).

[0113] (4) Φ in Eq. (12) i may be calculated approximately using K-FAC according to equation (16).

[0114] (5) The algorithm of equation (12) may be modified and implemented as Algorithm 1. Note that there are three changes from equation (12) in Algorithm 1. The first change is that communication (message passing) is performed between connected nodes after updating K times in the local node. The second change is that θ i (the first line of equation (12)) is replaced by equation (14).

[0115] The third change is that, as described in the function gCalc, two calculation methods (NGPD format 1 and format 2) have been provided for the calculation of the stochastic gradient g, and either one can be used.

[0116] Both methods use K-FAC to introduce quadratic optimization into distributed learning in a lightweight manner. NGPD format 1 uses a method that satisfies the stationary point condition in equation (11), but the amount of calculations and memory required is somewhat large. On the other hand, NGPD format 2 is an approximation method that does not strictly satisfy the stationary point condition in equation (11), but the amount of calculations and memory required is much smaller than that of NGPD format 2, as is reflected in the experimental results.

[0117] (Effects of the embodiment) The technology according to the present embodiment incorporates quadratic optimization (natural gradient method) into distributed learning, making it possible to reach a consensus quickly in distributed learning.

[0118] The following supplementary notes are further disclosed regarding the above embodiment.

[0119] <Additional Notes> (Additional note 1) In a distributed learning system in which model learning is performed by information exchange between a plurality of nodes, an information processing device is used as a node among the plurality of nodes, Memory, at least one processor coupled to the memory; Including, The processor, receiving, from adjacent nodes, variables updated according to an update rule obtained by applying operator splitting from a stationary point condition expressed using a positive definite matrix; The variables are used to update model parameters of the model. Information processing device. (Additional note 2) The node is the i-th node, the adjacent node is the j-th node, and the positive definite matrix at the node is G i|j The positive definite matrix at the adjacent node is represented as G j|i If we express it as G i|j =G j|i Fulfill Item 1. An information processing device according to claim 1. (Additional note 3) The processor transmits the updated variables to the neighboring nodes according to the update rule. Item 1. An information processing device according to claim 1. (Additional note 4) The update rule for updating the model parameters is a rule for updating the model parameters by an equation including a term obtained by multiplying the gradient by the inverse matrix of the averaged positive definite matrix. Item 1. An information processing device according to claim 1. (Additional note 5) The processor updates the positive definite matrix using a Gauss-Newton matrix approximated by K-FAC (Kronecker Factored Approximate Curvature). Item 1. An information processing device according to claim 1. (Additional note 6) The processor calculates a stochastic gradient of the model for each layer of the model, or updates the positive definite matrix for each layer of the model. 6. An information processing device according to claim 5. (Additional note 7) A learning method executed by an information processing device used as a node among a plurality of nodes in a distributed learning system that learns a model by information exchange among the plurality of nodes, comprising: a receiving step of receiving, from an adjacent node, variables updated according to an update rule obtained by applying operator splitting from a stationary point condition expressed using a positive definite matrix; updating model parameters of the model using the variables; A learning method that provides: (Additional note 8) A non-transitory storage medium storing a program for causing a computer to function as each unit in the information processing device according to any one of claims 1 to 6.

[0120] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0121] 100 Information processing device 110 Transmitter 120 Receiving unit 130 Update Department 140 Storage section 200 Network 1000 Drive Device 1001 Recording media 1002 Auxiliary storage 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output device

Claims

1. In a distributed learning system in which model learning is performed by information exchange between a plurality of nodes, an information processing device is used as a node among the plurality of nodes, a receiving unit that receives, from an adjacent node, a variable updated according to an update rule obtained by applying operator splitting from a stationary point condition expressed using a positive definite matrix; an update unit that updates model parameters of the model using the variables; An information processing device comprising:

2. The node is the i-th node, the adjacent node is the j-th node, and a positive definite matrix at the node is G ｉ｜ｊ The positive definite matrix at the adjacent node is represented as G ｊ｜ｉ When expressed as: G ｉ｜ｊ = G ｊ｜ｉ Fulfill The information processing device according to claim 1 .

3. a transmission unit that transmits the variables updated by the update unit according to the update rule to the adjacent nodes; The information processing apparatus according to claim 1 , further comprising:

4. The update rule for updating the model parameters is a rule for updating the model parameters by an equation including a term obtained by multiplying the gradient by the inverse matrix of the averaged positive definite matrix. The information processing device according to claim 1 .

5. The update unit updates the positive definite matrix using a Gauss-Newton matrix approximated by K-FAC (Kronecker Factored Approximate Curvature). The information processing device according to claim 1 .

6. The update unit calculates a stochastic gradient of the model for each layer of the model, or updates the positive definite matrix for each layer of the model. The information processing device according to claim 5 .

7. A learning method executed by an information processing device used as a node among a plurality of nodes in a distributed learning system that learns a model by information exchange among the plurality of nodes, comprising: a receiving step of receiving, from an adjacent node, variables updated according to an update rule obtained by applying operator splitting from a stationary point condition expressed using a positive definite matrix; updating model parameters of the model using the variables; A learning method that provides:

8. A program for causing a computer to function as each unit in the information processing device according to any one of claims 1 to 6.

Images