Information processing device, distributed learning system, learning method, and program

C-ECL addresses the high communication costs in ECL by compressing dual variables, ensuring efficient and robust model learning with reduced information exchange, maintaining performance across diverse data distributions and network configurations.

JP2024160818APending Publication Date: 2024-11-15NIPPON TELEGRAPH & TELEPHONE CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023076220
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-02
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing distributed learning techniques, such as Edge-Consensus Learning (ECL), require excessive communication between nodes due to the exchange of dual variables of similar size to the model, leading to high communication costs and prolonged execution times, which hinder practical application.

Method used

Implementing Communication Compressed Edge-Consensus Learning (C-ECL) that uses a compression operator to efficiently compress dual variables exchanged between nodes, reducing the amount of information transferred while maintaining robustness to heterogeneous data.

Benefits of technology

C-ECL achieves reduced communication costs and comparable accuracy to ECL, with improved robustness to non-uniform data distributions, enabling more efficient model learning across various network topologies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024160818000001_ABST
    Figure 2024160818000001_ABST
Patent Text Reader

Abstract

To provide a technology for making it possible to decrease a communication cost in distributed learning.SOLUTION: In a distributed learning system which learns a model by information exchange between nodes among a plurality of nodes, an information processing device is used as a node among the plurality of nodes. The information processing device includes: a receiving unit which receives a compressed dual variable being a dual variable obtained by being compressed while using a compression operator from an adjacent node; and an updating unit which updates a model parameter of the model by using the compressed dual variable.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical Field]

[0001] The present invention relates to distributed learning. [Background technology]

[0002] In recent years, neural network models have attracted much attention in various fields, such as image processing and natural language processing. In order to train a model, it is usually necessary to collect a large amount of training data.

[0003] However, due to privacy issues and other factors, it is difficult to collect training data that includes personal information on a single server (node). In consideration of such cases, studies have been conducted to achieve training of a model (e.g., a neural network model) that can classify all data with a high recognition rate by distributing and storing data on multiple nodes that can be arbitrarily connected to a network (e.g., Non-Patent Document 1). The above-mentioned learning is called decentralized learning.

[0004] In distributed learning, the main research topic is building algorithms that determine how to achieve consensus among multiple nodes while learning. The fewer the number of communications between nodes and the amount of information communicated before reaching consensus, the better the distributed learning algorithm will perform.

[0005] Non-Patent Document 1 discloses a distributed learning technique called edge-consensus learning (ECL).

[0006] The advantage of ECL is that it can learn robustly against the statistical heterogeneity of the data subsets accumulated in the nodes. This is because it solves a cost minimization problem under the consensus-based constraint that the models between nodes must match. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Niwa, K., Harada, N., Zhang, G., and Kleijn, WB Edge-consensus learning: Deep learning on P2P networks with nonhomogeneous data. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 668-678, 2020. Summary of the Invention [Problem to be solved by the invention]

[0008] The ECL disclosed in Non-Patent Document 1 is a linear convergence algorithm. Therefore, the number of communications between nodes increases until consensus is reached and model learning is completed. Furthermore, in ECL, information (specifically, dual variables) of a size roughly equivalent to the size of the model must be exchanged between nodes in each communication, resulting in a large amount of communication. The communication cost in distributed learning accounts for the majority of the computational load and execution time, and for practical distributed learning, this communication cost must be reduced. Note that this issue is not limited to ECL, but can also occur in other types of distributed learning.

[0009] The present invention has been made in view of the above points, and has an object to provide a technique that makes it possible to reduce communication costs in distributed learning. [Means for solving the problem]

[0010] According to the disclosed technology, in a distributed learning system in which model learning is performed by information exchange between a plurality of nodes, there is provided an information processing device used as a node among the plurality of nodes, the information processing device comprising: a receiving unit that receives compressed dual variables from an adjacent node, the compressed dual variables being dual variables compressed using a compression operator; an update unit that updates model parameters of the model using the compressed dual variables; An information processing device comprising: [Effects of the Invention]

[0011] According to the disclosed technology, it is possible to reduce communication costs in distributed learning. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an example of the overall configuration of a distributed learning system. [Figure 2] FIG. 1 illustrates an example of a network topology. [Figure 3] FIG. 10 illustrates the update procedure at node i. [Figure 4] FIG. 10 is a diagram showing experimental results. [Figure 5] FIG. 10 is a diagram showing experimental results. [Figure 6] FIG. 10 is a diagram showing experimental results. [Figure 7] FIG. 10 is a diagram showing experimental results. [Figure 8] FIG. 10 is a diagram showing experimental results. [Figure 9] FIG. 10 is a diagram showing experimental results. [Figure 10] FIG. 10 is a diagram showing experimental results. [Figure 11] FIG. 1 is a configuration diagram of a node (information processing device). [Figure 12] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0014] In the following, we will explain C-ECL (Communication Compressed ECL), a new distributed learning technology based on ECL disclosed in Non-Patent Document 1, for compressing communication volume in distributed learning. In C-ECL, dual variables exchanged between nodes in ECL are efficiently compressed using a compression operator (for example, random projection).

[0015] In the following embodiment, we will explain C-ECL based on ECL, but the technology related to the present invention is not limited to distributed learning based on ECL and can be applied to other types of distributed learning as well.

[0016] (Example of overall system configuration) Fig. 1 shows an example of the overall configuration of a distributed learning system according to this embodiment. As shown in Fig. 1, this system has a configuration in which multiple nodes 100 are connected to a network 200. Communication between the nodes is possible via the network 200. The network 200 may be any network, but is, for example, an IP network such as the Internet.

[0017] Each node 100 is a computer (which may be called an information processing device) with a communication function. The computer may be a physical machine or a virtual machine built on a cloud. In addition, any or all of the multiple nodes 100 may be edge devices in edge computing. In addition, any or all of the multiple nodes 100 may be IoT devices.

[0018] Each node 100 transmits and receives (exchanges) information (e.g., compressed dual variables) with other nodes 100, thereby learning a model. There is no limitation on the type of model, and it may be a neural network or a model other than a neural network. However, in this embodiment, it is assumed that a neural network model is used.

[0019] The plurality of nodes 100 may be connected in any topology. The topology may be in any form, for example, (a) chain, (b) ring, (c) multi-ring, (d) complete connected graph as shown in FIG. 2. Also, the topology may be in other forms than these.

[0020] Hereinafter, the processes (algorithms) executed by each node 100 for model learning will be described. Hereinafter, first, related technologies will be described, and then, the technology according to the present embodiment (that is, C-ECL) will be described.

[0021] Hereinafter, ||·|| represents the L2 norm, 0 represents a vector of all zeros, and I represents an identity matrix.

[0022] For the sake of convenience of description, in the text of this specification (text that is not an image), generally, the characters of vectors and matrices shown in bold are shown in normal font. Even in normal font, it is clear from the context that it is a character indicating a vector or a matrix. In addition, for characters representing sets (for example, a set of real numbers, a set of nodes, etc.), normal font is used in the text of the specification.

[0023] (1. Related Technologies) Here, the problem setting of distributed learning will be shown, and then, the outline of the Gossip-based algorithm and the outline of ECL will be described. Note that the Gossip-based algorithm is one of the algorithms widely used as an algorithm for distributed learning.

[0024] <A. Distributed Learning> Let G = (V, E) be an undirected connected graph representing a network topology. Here, V represents a set of nodes, and E represents a set of edges. For simplicity, V is represented as a set of integers {1, 2,..., |V|}.

[0025] The set of adjacent nodes of node i is N i= {j∈V|(i,j)∈E}. The goal of distributed learning can be formulated as shown in equation (1) below.

[0026]

number

[0027] Equation (1) means that the loss at each node is added up across all nodes, and the parameter w that minimizes this value is found.

[0028] <B.Gossipベースのアルゴリズム> Gossip-based algorithms (e.g., D-PSGD) are widely used approaches for distributed learning. In Gossip-based algorithms, each node i has a gradient ∇f i , and exchange parameters with neighboring nodes, then take their average.,Although Gossip-based algorithms are simple and effective approaches,,they suffer from two major problems: high communication costs and,sensitivity to data heterogeneity.

[0029] In Gossip-based algorithms, nodes must receive model parameters from neighboring nodes. Because neural networks have a large number of model parameters, exchanging model parameters incurs a huge communication cost. This is the first problem. To reduce the communication cost of Gossip-based algorithms, many methods have been proposed to compress the amount of parameters exchanged, using sparsification, quantization, low-rank approximation, etc. It is known that these compression methods can achieve almost the same accuracy as uncompressed Gossip-based algorithms with a smaller amount of parameter exchange.

[0030] The second problem is that Gossip-based algorithms are sensitive to uneven data distribution. When the data distribution of each node is statistically uneven, the loss function Σ i f i The optimal solution and the loss function f i Since the optimal solutions for are significantly different, Gossip-based algorithms do not work well. Various methods have been proposed to address this second problem in Gossip-based algorithms.

[0031] <C.ECL> Next, we will explain ECL disclosed in Non-Patent Document 1. Since ECL itself is a publicly known technology, we will only provide an overview here. By reformulating equation (1), the primal problem can be defined as shown in the following equation (2).

[0032]

number

[0033]

number

[0034]

number

[0035]

number

[0036] The above equations (3) to (5) are the basic update rules of ECL, and equations (3) and (4) are calculated at each node. i are model parameters (model variables), and y i|j represents the dual variable sent from the i-th node to the j-th node. The dimensions of the model parameters and the dual variables are the same. In equation (5), the i-th and j-th nodes send the dual variable y i|j and y j|i (synchronously swap) and use the received information to calculate the dual variable z i|j As described later, the y exchanged through communication is updated.i|j The challenge is how to compress this efficiently (without hindering learning), and C-ECL solves this problem.

[0037] In addition, f i If is non-convex (e.g., the loss function of a neural network), equation (3) cannot generally be solved. Therefore, ECL (and C-ECL) solves equation (3) approximately as follows:

[0038]

number

[0039] Below, we will explain C-ECL, a proposed method that solves the above problems. C-ECL can train a model with fewer parameter exchanges than ECL and is robust to heterogeneous data. C-ECL is based on ECL, and below we will mainly explain the differences between C-ECL and ECL.

[0040] (2.C-ECL) To explain C-ECL, we first explain the compression operator used in C-ECL.

[0041] Assumption 1 (Compression Operator): For some τ∈(0,1], the compression operator comp:R d →R d is for all x,y∈R d and ω, the following holds:

[0042]

number

[0043]

number

[0044]

number

[0045] Example 1. For some k∈(0,100), the operator rand k% :R d →R d is defined as follows: The following operation is called random projection mapping.

[0046]

number

[0047] where the parameter ω of the compression operator corresponds to the randomly sampled vector s. k%satisfies Assumption 1.

[0048] <Details of B.C-ECL> The following details C-ECL, which is a method for compressing dual variables exchanged between nodes using the above-described compression operator.

[0049] In the ECL disclosed in Non-Patent Document 1, for updating z in Equation (5), i|j node i needs to receive y from node j. j|i Since the number of elements of y is the same as the number of elements of the model parameter w, j|i this exchange incurs a large communication cost. As a direct approach to reducing this communication cost, it is conceivable to compress y in Equation (5) using comp as follows. j j|i j|i However, it has been experimentally found that even if y is compressed as described above, it does not solve the problem.

[0050]

Number

[0051]

[0052]

[0053]

Number

[0054] Then, when the model parameters approach the optimal solution, (y j|i -z i|j ) approaches 0. Therefore, in this embodiment, y j|i Instead of compressing, use (y j|i -z i|j ) is to be compressed.

[0055]

number

[0056] Algorithm 1, which shows the processing operation of C-ECL at node i, is shown in Figure 3. Each node constituting the distributed learning system executes this processing.

[0057] Algorithm 1 corresponds to the pseudocode of the program executed by node i. Hereinafter, the line numbers in Figure 3 are referred to as "steps."

[0058] Steps 1 to 11 in FIG. 3 are repeatedly executed starting from r=0 and incrementing r by 1 until r=R.

[0059] In step 2, node i selects w from each parameter / variable at the current time. (r+1) i Node i executes steps 3 to 10 for each j (node ​​j adjacent to node i).

[0060] In step 4, node i is assigned to (r) i|j -2αA i|j w (r+1) i ' and convert it to y (r+1) i|j In step 5, node i receives ω from its neighbor node j. (r+1) i|j In step 6, node i receives ω (r+1) j|i Send.

[0061] In step 7, node i receives comp(y (r+1) j|i ;ω (r+1) i|j In step 8, node i receives comp(y (r+1) i|j ;ω (r+1) j|i ) to send.

[0062] In step 9, node i is assigned a (r) i|j +θcomp(y (r+1) j|i -z (r) i|j ;ω (r+1) i|j ) and convert this to z (r+1) i|j Let's say.

[0063] In Algorithm 1 shown in Fig. 3, for simplicity, in steps 5 and 6, node i and node j use ω i|j and ω j|i However, this process is just an example. Before starting learning, node i and node j share the same seed value and exchange ω i|j and ω j|i By generating ω, node i and node j can achieve ω without exchanging ω. i|j and ω j|i can be obtained.

[0064] Furthermore, when using rand as the compression operator k% node i can obtain ω j|i from the received value comp(y i|j ; ω i|j ). This is because comp(y j|i ; ω i|j ) is stored in sparse matrix form. Therefore, the exchange of these ω i|j and ω j|i can actually be omitted. Thus, in C-ECL, each node i only needs to receive / transmit the compressed value of y j|i / y i|j . Therefore, in C-ECL, model learning can be performed with less information exchange than in ECL.

[0065] Note that in Algorithm 1 shown in Figure 3, the model variable is updated once and communication is performed once alternately for each learning round, but the model variable can be updated multiple times for each learning round. Also, communication can be performed asynchronously for each node.

[0066] (3. Convergence Analysis) Next, it will be explained how the compression in C-ECL affects the convergence rate of ECL. The convergence analysis is based on the analysis in Douglas-Rachford splitting (Giselsson, P. and Boyd, S. P. (2017). Linear convergence and metric selection for Douglas-Rachford splitting and ADMM. In IEEE Transactions on Automatic Control.).

[0067] <A. Prerequisites> Here, additional notations and assumptions are introduced for the convergence analysis. N := |V|, N min := min i {|N i |}, N max := max i {|N i|}, where N i is the set of neighbors of node i, N i (j) is N i Let w∈R be the index of the jth smallest node in dN ,z i ∈R d|N_i| , and z∈R 2d|E| is defined as follows:

[0068]

number

[0069]

number

[0070]

number

[0071] Assumption 2. We assume that f is proper, closed, and convex.

[0072] Assumption 3. Assume that f is L-smooth and μ-strongly convex, with L>0 and μ>0.

[0073] Assumption 4. Assume that there are no isolated nodes in graph G (i.e., N min >0).

[0074] Assumptions 2 and 3 are standard assumptions used in the convergence analysis of the operator splitting method. Assumption 3 is weaker than the assumptions of smoothness and strong convexity of f for all i ∈ V, which are commonly used in distributed learning. In distributed learning, since the graph G is assumed to be connected, Assumption 4 generally holds. Furthermore, define δ ∈ R as follows. i Assuming that Assumptions 2, 3, and 4 hold and α ∈ (0, ∞) holds, since L ≥ μ > 0 and N max ≧N min > 0, then δ ∈ [0, 1) holds.

[0076] <B. Convergence Rates> Theorem 1. - Let z ∈ R2 d|E| be a fixed point of the Douglas - Rachford splitting. Also assume that Assumptions 1, 2, 3, and 4 hold. Note that the " - z" is intended to have a " - " placed above z.

[0077] The following holds.

[0078]

数

[0079]

数

[0080]

数

[0081]

number

[0082] Comparing the domain of θ for convergence in ECL and C-ECL, as τ decreases, the domain of (17) becomes smaller. Also, to ensure that the domain of (17) is not empty, τ must be (1-(1-δ) 2 / (1+δ) 2 ) or more.

[0083] Next, comparing the convergence speeds of ECL and C-ECL, the compression in C-ECL reduces the convergence speed in ECL by the following term:

[0084]

number

[0085] Corollary 2. If assumptions 1, 2, 3, and 4 hold and the following holds, the optimal convergence rate of equation (18) in C-ECL is achieved when θ = 1.

[0086]

number

[0087] Note that θ is generally set as θ ∈ (0, 1]. However, Theorem 1 and System 1 show that for a certain δ, ECL and C-ECL converge even when θ is greater than 1.

[0088] (4. Effects) Next, we use experimental results to explain the effect that C-ECL can achieve almost the same performance as ECL with a smaller amount of information exchange. Furthermore, we explain that C-ECL is more robust to heterogeneous data than gossip-based algorithms.

[0089] <A. Experimental Setup> First, we explain the experimental setup.

[0090] Regarding the dataset, we evaluated C-ECL using FashionMNIST and CIFAR 10, which are datasets for 10-class image classification tasks. As models for both datasets, we used a 5-layer convolutional neural network with group normalization applied.

[0091] Similar to the ECL disclosed in Non-Patent Document 1, we distributed data to nodes in two settings: a homogeneous setting and a heterogeneous setting. In the homogeneous setting, each node has data for all 10 classes, and the data was distributed so that the number of data for each class was approximately the same. In the heterogeneous setting, the data was distributed so that each node has data for 8 randomly selected classes. In both settings, the data was distributed so that each node has the same number of data.

[0092] Regarding network topology, in "B. Experimental Results" described later, all methods were evaluated on a ring network consisting of eight nodes. In addition, in "C. Experimental Results on Various Network Topologies" described later, all methods were evaluated on four network topologies consisting of eight nodes (chain, ring, multiple ring, and fully connected graph shown in Figure 2). Each node was set to exchange parameters / variables with neighboring nodes every five local updates.

[0093] The methods used for evaluation (C-ECL and comparison with C-ECL) are as follows:

[0094] (1) D-PSGD: Uncompressed Gossip-based algorithm.

[0095] (2) PowerGossip: A Gossip-based algorithm that compresses parameters using low-rank approximation. PowerGossip is known to achieve nearly the same performance as other existing compression methods without additional hyperparameter tuning, so we used PowerGossip as the method for our compressed Gossip-based algorithm.

[0096] (3) ECL: A primal-dual algorithm disclosed in Non-Patent Document 1. It is known that ECL converges faster when θ=1 than when θ=0.5, so θ=1 was used.

[0097] (4) C-ECL: Proposed method. Random is used as the compression operator. k% was used. According to Corollary 2, θ = 1. z i|j and y i|j was initialized to 0. However, rand k% Using z i|j Compressing the updated value of z i|j It was found that the convergence was slow because rand remained sparse in the initial training stage. k% We set k% of the parameter to 100% for the first epoch.

[0098] Also, for reference, the results of stochastic gradient descent (SGD) for training the model with a single node having all the training data are also shown. In the experiment, the same learning rate, number of epochs, and batch size were set for all the methods.

[0099] <B. Experimental Results> Explain the evaluation results of accuracy and communication cost when the network topology is set to a ring.

[0100] Uniform setting: First, explain the results in the uniform setting. Fig. 4 shows the accuracy and communication cost in the uniform setting. In Fig. 4, for C-ECL, the numbers in parentheses indicate rand k% 's k. For PowerGossip, the numbers in parentheses indicate the number of power iteration steps. As the communication cost, the average amount of transmitted parameters (data volume) per epoch is shown.

[0101] As shown in Fig. 4, for D-PSGD and ECL, almost the same accuracy is achieved for any dataset. C-ECL and PowerGossip are equivalent, and even when setting rand k% 's k% to 1% and the number of power iteration steps to 1, almost the same accuracy as ECL and D-PSGD is achieved. That is, C-ECL can achieve equivalent accuracy with approximately 50 times less parameter exchange than ECL and D-PSGD in the uniform setting.

[0102] Non-uniform setting: Next, explain the results in the non-uniform setting. Fig. 5 shows the accuracy and communication cost in the non-uniform setting. Also in Fig. 5, for C-ECL, the numbers in parentheses indicate the k of randk%. For PowerGossip, the numbers in parentheses indicate the number of power iteration steps. As the communication cost, the average amount of transmitted parameters (data volume) per epoch is shown.

[0103] As shown in Figure 5, in D-PSGD, the accuracy in the non-uniform setting decreases by about 3% compared to the uniform setting. In PowerGossip, even if the number of power iteration steps is increased, it does not approach the accuracy of D-PSGD and ECL. On the other hand, the accuracy of ECL is almost the same in both the uniform and non-uniform settings, indicating that ECL is more robust to non-uniform data than D-PSGD. In C-ECL, when k% of rand k% is set to 1%, the accuracy in the non-uniform setting decreases by about 10% compared to the accuracy of ECL.

[0104] However, when k% of rand k% is increased, C-ECL approaches ECL and can achieve better performance than D-PSGD and PowerGossip. Specifically, when k% is set to 10% in FashionMNIST, C-ECL is equivalent to ECL and has better performance than D-PSGD and PowerGossip. When k% is set to 20% in CIFAR 10, C-ECL is equivalent to ECL and has better performance than D-PSGD and PowerGossip.

[0105] That is, in the uniform setting, C-ECL can achieve almost the same accuracy as ECL and D-PSGD with about 50 times fewer parameter exchanges. In the non-uniform setting, C-ECL can achieve almost the same accuracy as ECL with about 4 times fewer parameter exchanges and can achieve performance superior to PowerGossip. Furthermore, the results indicate that C-ECL is superior to D-PSGD, a non-compressed Gossip-based algorithm, in terms of both accuracy and communication cost.

[0106] <C. Experimental Results in Various Network Topologies> Here, the accuracy and communication cost when the network topology is changed are described. Figures 6 to 10 show the communication cost and accuracy in FashionMNIST when the network topology is changed to a chain, a ring, a multi-ring, and a fully connected graph.

[0107] Figures 6 and 7 show the uniform setting, while Figures 8 and 9 show the non-uniform setting. Figure 10 shows the average amount of parameters sent per epoch in the uniform and non-uniform settings.

[0108] As shown in Figures 6 and 7, in the uniform setting, the accuracy of all methods is almost the same, and they reach the accuracy of SGD in all network topologies.

[0109] As shown in Figures 8 and 9, the accuracy of D-PSGD and PowerGossip decreases in the non-uniform setting compared to the uniform setting. On the other hand, the accuracy of ECL is almost the same as that in the uniform setting in all network topologies. Furthermore, in all network topologies, C-ECL achieves almost the same accuracy as ECL with fewer parameter exchanges and consistently outperforms PowerGossip. Furthermore, in the non-uniform setting, C-ECL outperforms D-PSGD, an uncompressed Gossip-based algorithm, in both accuracy and communication cost in all network topologies.

[0110] (Device configuration example) FIG. 11 shows an example of the configuration of an information processing device 100 used as a node in a distributed learning system that executes C-ECL.

[0111] As shown in FIG. 11, the information processing device 100 includes a transmitting unit 110, a receiving unit 120, an updating unit 130, and a storage unit 140.

[0112] The information processing device 100 is an information processing device used as one of a plurality of nodes in a distributed learning system that performs model learning by information exchange between the plurality of nodes. This distributed learning system performs distributed learning using C-ECL.

[0113] The receiving unit 120 receives compressed dual variables, which are dual variables compressed using a compression operator, from adjacent nodes. The updating unit 130 updates the model parameters using the received compressed dual variables.

[0114] Furthermore, the transmitting unit 110 transmits to the adjacent node the dual variables compressed using the compression operator by the updating unit 130. The storage unit 140 stores the received data, the data calculated by the updating unit 30, and the data to be transmitted.

[0115] (Example of hardware configuration) The information processing device 100 described in this embodiment can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0116] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0117] Fig. 12 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 12 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B. The computer may further include a GPU.

[0118] A program for realizing processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0119] The memory device 1003 reads and stores the program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the information processing device 100 in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0120] The following additional notes are provided regarding the above-described embodiments.

[0121] <Additional Notes> (Additional note 1) 1. An information processing device used as a node among a plurality of nodes in a distributed learning system that performs model learning by information exchange among the plurality of nodes, Memory and at least one processor coupled to said memory; Including, The processor: receiving compressed dual variables from the adjacent node, the compressed dual variables being dual variables compressed using a compression operator; The compressed dual variables are used to update the model parameters of the model. Information processing device. (Additional note 2) The processor compresses the dual variables to be sent to the adjacent node using the compression operator, and transmits the compressed dual variables to the adjacent node. Item 1. An information processing device according to item 1. (Additional note 3) The compression operator is an operator that takes the Hadamard product of a vector representing the dual variable to be compressed and a sparse vector in which each element has a certain probability of being 1. 3. The information processing device according to claim 1 or 2. (Additional note 4) When the compression operator is comp, x and y are vectors, and τ and ω are parameters, comp is expressed as follows: Eω||comp(x;ω)-x||2≦(1-τ)||x||2, comp(x+y;ω)=comp(x;ω)+comp(y;ω), and comp(-x;ω)=-comp(x;ω) fulfill 4. The information processing device according to any one of claims 1 to 3. (Additional note 5) A distributed learning system including a plurality of information processing devices according to any one of appended claims 1 to 4. (Additional note 6) A learning method executed by an information processing device used as a node among a plurality of nodes in a distributed learning system that learns a model by information exchange among the plurality of nodes, comprising: a receiving step of receiving, from an adjacent node, compressed dual variables that are dual variables compressed using a compression operator; an updating step of updating model parameters of the model using the compressed dual variables; A learning method that includes: (Additional note 7) A non-transitory storage medium storing a program for causing a computer to function as each unit in the information processing device according to any one of claims 1 to 4.

[0122] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0123] 100 Information processing device 110 Transmitter 120 Receiver 130 Update Department 140 Storage section 200 Network 1000 Drive Device 1001 Recording media 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output Device

Claims

1. In a distributed learning system in which model learning is performed by information exchange between a plurality of nodes, an information processing device is used as a node among the plurality of nodes, a receiving unit that receives compressed dual variables from an adjacent node, the compressed dual variables being dual variables compressed using a compression operator; an update unit that updates model parameters of the model using the compressed dual variables; An information processing device comprising:

2. The update unit compresses dual variables to be sent to the adjacent node using the compression operator; A transmission unit that transmits the dual variables compressed by the update unit to the adjacent node. The information processing apparatus according to claim 1 , further comprising:

3. The compression operator is an operator that takes the Hadamard product of a vector representing a dual variable to be compressed and a sparse vector in which each element has a certain probability of being 1. The information processing device according to claim 1 .

4. When the compression operator is defined as comp, x and y are vectors, and τ and ω are parameters, comp is expressed as follows: E ω ||comp(x;ω)-x|| 2 ≦(1-τ)||x|| 2 、 comp(x+y;ω)=comp(x;ω)+comp(y;ω), and comp(-x;ω)=-comp(x;ω) Fulfill The information processing device according to claim 1 .

5. 5. A distributed learning system comprising a plurality of information processing devices according to claim 1.

6. A learning method executed by an information processing device used as a node among a plurality of nodes in a distributed learning system that learns a model by information exchange among the plurality of nodes, comprising: a receiving step of receiving, from an adjacent node, compressed dual variables, which are dual variables compressed using a compression operator; updating model parameters of the model using the compressed dual variables; A learning method that provides:

7. A program for causing a computer to function as each unit in the information processing device according to any one of claims 1 to 4.