Information processing device, information processing method, computer program, and distributed learning system

Abstract

Provided is an information processing device which performs a process for distributed training.　The information processing device comprises: a management unit which manages workers that can be used for the distributed training of a model; and a parameter server unit which distributes tasks to the workers managed by the management unit, gathers gradient data that the workers have been trained about and have obtained, and updates the model. The management unit manages a worker list including available resources and addresses of the workers. In addition, the parameter server unit acquires the worker list from the management unit for each iteration of the distributed training, and distributes the tasks to the workers on the basis of the worker list.

Description

[Technical field]

[0001] The technology disclosed in this specification (hereinafter referred to as "the present disclosure") relates to an information processing device, an information processing method, a computer program, and a distributed learning system that perform processing for distributed learning of a neural model. [Background technology]

[0002] Recently, research and development of deep neural network learning (DNN learning) is progressing rapidly, as it is expected to be applied to various fields, such as recognition processing such as image recognition, voice recognition, and character recognition, as well as the control of autonomous devices such as voice agents, robots, and drones. Since DNN learning requires an enormous number of parameters (gradient data of weight coefficients) to be handled and requires a great deal of time, distributed learning, which distributes the data across multiple computing nodes and processes them in parallel, is often adopted.

[0003] In distributed learning, gradient synchronization of the parameters of the DNN model is required between the computation nodes, and the DNN model is updated after waiting for the completion of gradient calculations on all computation nodes. The method of gradient synchronization differs depending on the mechanism of distributed learning. There are two types of gradient synchronization methods: distributed learning with a parameter server (learning with PS, Centralized DNN Training) (see, for example, Patent Document 1), in which a parameter server collects gradient information from each computation node and updates the DNN model, and learning without a parameter server (learning without PS, Decentralized Distrubuted DNN Training) (see, for example, Non-Patent Document 1), in which gradient information is directly communicated between computation nodes without the intervention of a parameter server to update the DNN model. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-206016 A [Non-patent literature]

[0005] [Non-Patent Document 1] P. Goyal et al.,"Accurate,Large Minibatch SGD:Training ImageNet in 1 Hour,"arXiv:1706.02677 [cs],Apr.2018. Summary of the Invention [Problem to be solved by the invention]

[0006] An object of the present disclosure is to provide an information processing device, an information processing method, a computer program, and a distributed learning system that perform processing for distributed learning using a parameter server. [Means for solving the problem]

[0007] A first aspect of the present disclosure is A management unit that manages workers available for distributed model training; a parameter server unit that distributes tasks to workers managed by the management unit, collects gradient data obtained by the workers through learning, and updates the model; The information processing device includes:

[0008] The management unit manages a worker list including resources and addresses of available workers, and the parameter server unit obtains the worker list from the management unit for each iteration of the distributed learning and distributes tasks to the workers based on the worker list.

[0009] The parameter server unit creates and distributes a task specialized for each worker for each iteration based on information on available resources of each worker.

[0010] The management unit adds information about the added worker to the worker list at any timing within the iteration. The management unit also receives a request to stop a worker at any timing within the iteration, stops the worker requested to be stopped in the next iteration, and provides the parameter server unit with a worker list excluding information about the stopped worker.

[0011] The second aspect of the present disclosure is An administration step to manage the workers available for distributed training of the model; Distributing tasks to workers managed by the managing step; Collecting gradient data obtained by the workers' training; updating the model based on collected gradient data; The information processing method includes the steps of:

[0012] Moreover, a third aspect of the present disclosure is A management unit that manages the workers available for distributed training of the model; a parameter server unit that distributes tasks to workers managed by the management unit, collects gradient data obtained by the workers through learning, and updates the model; It is a computer program written in a computer-readable form to cause a computer to function as a

[0013] The computer program according to the third aspect of the present disclosure defines a computer program written in a computer-readable format to realize a predetermined process on a computer. In other words, by installing the computer program according to the third aspect of the present disclosure on a computer, a cooperative action is exerted on the computer, and the same action and effect as the information processing device according to the first aspect of the present disclosure can be obtained.

[0014] Moreover, a fourth aspect of the present disclosure is The workers available for distributed training of the model, A connection manager that manages the workers; a parameter server that distributes tasks to the workers based on a worker list obtained from a connection manager, collects gradient data obtained by the workers through learning, and updates the model; The present invention relates to a distributed learning system.

[0015] However, the term "system" used here refers to a logical collection of multiple devices (or functional modules that realize a specific function), and it does not matter whether each device or functional module is contained within a single housing. Effect of the Invention

[0016] According to the present disclosure, it is possible to provide an information processing device, an information processing method, a computer program, and a distributed learning system that utilizes a parameter server and performs processing for distributed learning while changing the number of workers.

[0017] Note that the effects described in this specification are merely examples, and the effects brought about by the present disclosure are not limited thereto. Furthermore, the present disclosure may have additional effects in addition to the effects described above.

[0018] Further objects, features and advantages of the present disclosure will become apparent from the following detailed description based on the embodiments and the accompanying drawings. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a distributed learning system 100 that performs learning with PS. [Diagram 2] FIG. 2 is a diagram showing an operation chart of DNN learning in the distributed learning system 100 shown in FIG. [Diagram 3] FIG. 3 is a diagram showing a parameter distribution phase from the parameter server to each worker in the distributed learning system 100 shown in FIG. [Figure 4]FIG. 4 is a diagram showing a model training (train) phase by each worker in the distributed learning system 100 shown in FIG. [Diagram 5] FIG. 5 is a diagram showing the phase in which the parameter server gathers parameters from each worker in the distributed learning system 100 shown in FIG. [Figure 6] FIG. 6 is a diagram showing the update phase of the DNN model by the parameter server in the distributed learning system 100 shown in FIG. [Figure 7] FIG. 7 is a diagram showing an example of the configuration of a distributed learning system 700 that performs learning without PS. [Figure 8] FIG. 8 is a diagram showing an operation chart of DNN learning in the distributed learning system 700 shown in FIG. [Figure 9] FIG. 9 is a diagram showing the training and update phases of a DNN model in the distributed learning system 700 shown in FIG. [Figure 10] FIG. 10 illustrates the gradient synchronization (sync) phase in the distributed learning system 700 shown in FIG. [Figure 11] FIG. 11 is a diagram for explaining a procedure in which a scheduler uses a priority queue to manage tasks to be executed by a cluster computer. [Figure 12] FIG. 12 is a diagram for explaining a procedure in which a scheduler uses a priority queue to manage tasks to be executed by a cluster computer. [Figure 13] FIG. 13 is a diagram illustrating an example of load fluctuations in a cluster computer during the execution of an important task. [Figure 14] Figure 14 shows the load required for DNN learning tasks, along with periods of low load during the execution of important tasks. [Figure 15] Figure 15 shows the fluctuating load while a cluster computer is executing an important task, and the load of a DNN learning task superimposed during periods of low load. [Figure 16]FIG. 16 is a diagram showing a state where learning with PS is performed in a heterogeneous environment. [Figure 17] FIG. 17 is a diagram showing how learning without PS is performed in a heterogeneous environment. [Figure 18] FIG. 18 is a diagram showing an example of the configuration of a distributed learning system 1800 that performs learning with PS in a heterogeneous environment. [Figure 19] FIG. 19 is a diagram showing an operation chart of DNN learning in the distributed learning system 1800 shown in FIG. [Figure 20] FIG. 20 is a diagram showing an example of an operation sequence (including worker registration) in which the distributed learning system performs learning with PS by using the connection manager. [Figure 21] FIG. 21 is a diagram showing an example of an operation sequence (including the addition of a worker) in which a distributed learning system performs learning with PS by using a connection manager. [Figure 22] FIG. 22 is a diagram showing an example of an operation sequence (including removal of a worker by Direct Termination) in which a distributed learning system performs learning with PS using a connection manager. [Figure 23] FIG. 23 is a diagram showing an example of an operation sequence (including removal of a worker by Connection-manager-initiated Termination) in which a distributed learning system performs learning with PS using a connection manager. [Figure 24] FIG. 24 is a diagram showing a functional configuration for updating the amount of rainfall calculated by each worker in the parameter server. [Diagram 25] FIG. 25 is a diagram showing an example of operation when an important task enters the task queue while a DNN learning task is being executed on a cluster computer. [Figure 26] FIG. 26 is a diagram showing an example of operation when an important task enters the task queue while a DNN learning task is being executed on a cluster computer. [Figure 27]FIG. 27 is a diagram showing an example of operation when an important task enters the task queue while a DNN learning task is being executed on a cluster computer. [Figure 28] FIG. 28 is a diagram illustrating an example of load fluctuations in a cluster computer executing an important task. [Figure 29] FIG. 29 is a diagram illustrating an example of load fluctuations in which free resources are allocated to DNN distributed learning in response to load fluctuations in a cluster computer. [Diagram 30] Figure 30 illustrates the load fluctuations across a cluster computer when important tasks and DNN distributed learning are executed simultaneously in parallel. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] The technology according to the present disclosure will be described below with reference to the drawings in the following order. A. Distributed Learning System A-1. Learning with PS A-2. Learning without PS B. Task management in cluster computers C. Learning with PS in a heterogeneous environment C-1. Learning with PS using the Connection Manager C-2. Adding workers C-3. Removal of workers D. Distributed learning that enables automatic rebalancing E. Effects

[0021] A. Distributed Learning System DNN learning is achieved through an iterative process. A training dataset and an initial DNN model are input, and through an iterative process, the parameters of the trained DNN model (gradient data of weight coefficients) are output. One learning iteration is divided into a learning step (Train) and a parameter update step (Update).

[0022] In distributed learning, a data-parallel method is adopted in which the learning dataset is divided into mini-batches of appropriate size and assigned to each worker, and the same DNN model is assigned to each worker. In this case, since the learning dataset assigned to each worker is different, each worker will have different parameters (gradient data) in one learning iteration. For this reason, in addition to the learning step and update step, gradient synchronization of the DNN model parameters is required between the computation nodes for each learning iteration. As already mentioned, there are two gradient synchronization methods: learning with PS and learning without PS.

[0023] A-1. Learning with PS Learning with PS is a distributed learning mechanism that uses a parameter server. One of the multiple computation nodes is a parameter server that performs parameter update processing, and the rest are workers that calculate the amount of parameter updates.

[0024] Fig. 1 shows a configuration example of a distributed learning system 100 that performs learning with PS. For simplicity, the figure shows the distributed learning system 100 consisting of only four computation nodes, but in reality, it is assumed that the distributed learning system is configured with a large number of computation nodes, five or more. The computation nodes 101 to 104 are interconnected via a network (N / W). As the network connecting the computation nodes, a broadband network such as InfiniBand is assumed.

[0025] Each of the computation nodes 101 to 104 is configured with one CPU (Central Processing Unit) or a combination of a CPU and one or more GPUs (Graphic Processing Units). Among these, the computation node 101 operates as a parameter server, and the remaining computation nodes 102 to 104 operate as workers.

[0026] In training with PS, before the training step, the parameter server distributes parameters to each worker. Then, when training of all workers is completed, the parameter server gathers parameters from each worker and updates the model.

[0027] Fig. 2 shows an operation chart of DNN learning in the distributed learning system 100 shown in Fig. 1. Note that the horizontal axis is the time axis.

[0028] When the parameter server (computation node 101) receives a learning data set and an initial state DNN model, it divides the learning data set into mini-batches of an appropriate size, and repeats an iteration consisting of distributing learning data to each worker, gathering learned parameters from each worker, and updating the DNN model for each mini-batch, and outputs the parameters of the learned DNN model. In addition, each of the computation nodes 102 to 104 operating as a worker repeats an iteration consisting of distributing parameters from the parameter server, training, and gathering learned parameters by the parameter server. Figures 3 to 6 illustrate each phase of the distributed learning system 100 shown in Figure 1, including distributing learning data (mini-batches) from the parameter server to each worker, training the DNN model using the distributed learning data by each worker, gathering learned parameters (gradient data) from each worker by the parameter server, and updating the DNN model by the parameter server.

[0029] A-2. Learning without PS On the other hand, in PS-free learning, no parameter server is required, and gradient synchronization is performed between computation nodes operating as workers. FIG. 7 shows a configuration example of a distributed learning system 700 that performs PS-free learning. For simplicity, only four computation nodes are shown in the figure, but it is assumed that the distributed learning system is actually configured with a large number of computation nodes, five or more. Each of the computation nodes 701 to 704 is interconnected via a network (N / W). Each of the computation nodes 701 to 704 is configured with a combination of one CPU and one or more GPUs (Graphic Processing Units), and all of these operate as workers.

[0030] Fig. 8 shows an operation chart of DNN learning in the distributed learning system 700 shown in Fig. 7. The horizontal axis is the time axis. When a learning data set and an initial state DNN model are input, each of the computation nodes 701 to 704 operating as a worker repeats an iteration consisting of learning (train), gradient synchronization (sync), and updating (update) of the DNN model.

[0031] FIG. 9 illustrates the phases of training and updating the DNN model in the distributed learning system 700 shown in FIG. 7. Training and updating the DNN model are performed in mini-batches by dividing the training data set into appropriate mini-batches. FIG. 10 illustrates the phase of gradient synchronization in the distributed learning system 700 shown in FIG. 7. Gradient synchronization is a process of exchanging gradient data between all workers and adding them up element by element.

[0032] B. Task management in cluster computers Distributed DNN learning uses a cluster computer that groups multiple computers (nodes) together. In the cluster computer, a scheduler manages the deployment of tasks to each node, i.e., the schedule of tasks for each worker.

[0033] The scheduler manages tasks in a task queue. The task queue has a data structure called a priority queue, in which tasks are read out in descending order of priority. Figures 11 and 12 show an example of the scheduler using the task queue to manage tasks in a cluster computer.

[0034] As shown in Fig. 11, it is assumed that distributed DNN learning is performed on a cluster computer 1100. In the example shown in Fig. 11, the cluster computer 1100 is composed of six computation nodes #1 to #6, and DNN learning is performed using four of these computation nodes, #1 to #4. Each of the computation nodes #1 to #6 is composed of a combination of one CPU and one or more GPUs.

[0035] Here, an "important task" with a higher priority occurs in the task queue 1101 and is detected by the scheduler 1102 (SEQ1101). The scheduler 1102 must recover resources of the cluster computer from DNN learning in order to execute the important task (SEQ1102). In both learning with PS and learning without PS, the number of nodes used for the task cannot be changed during the learning session. The only way to recover resources is to suspend DNN learning in the computation nodes #1 to #4 of the cluster computer 1100 and put it back into the task queue 1101 (SEQ1103). When the resources are recovered by suspending the DNN learning, the scheduler 1102 takes out the important task with the highest priority from the task queue 1101 and launches it on the cluster computer 1100 (SEQ1104). FIG. 12 shows how important tasks are executed on the cluster computer 1100. In the illustrated example, the important task is executed using all six computing nodes #1 to #6 constituting the cluster computer 1100. The DNN learning task that was interrupted to execute the important task with priority is resumed after waiting until the resources of the cluster computer 1100 are sufficient.

[0036] FIG. 13 illustrates load fluctuations in the cluster computer 1100 while an important task is being executed. However, the horizontal axis of the figure is the time axis, and the vertical axis is the load. In the example shown in FIG. 13, the load of the cluster computer 1100 while an important task is being executed is not uniform, but fluctuates over time, and there are periods of high load when there is no room to execute other tasks, and periods of low load where the DNN learning task can be executed. For reference, FIG. 14 shows the load required for the DNN learning task together with the low load period during the execution of the important task. Also, FIG. 15 shows the load fluctuating while the cluster computer 1100 is executing an important task (see FIG. 13) and the load of the DNN learning task during the low load period (see FIG. 14) superimposed.

[0037] As can be seen from Figure 15, even if the cluster computer is executing an important task, the unused resources when the load is reduced are sufficient to execute the DNN learning task. However, whether learning with PS or learning without PS, the number of nodes used for the task cannot be changed during the session, so DNN learning cannot be advanced using the unused resources while the important task is being executed, as shown in Figure 15. Therefore, there are many unused resources in the cluster computer.

[0038] In addition, neither learning with PS nor learning without PS can handle resource changes during a session. For example, even if a node is added to the cloud computer and the available resources increase, they cannot be used in the middle of a session.

[0039] In a heterogeneous environment, cloud computers do not have uniform resources for each node. In the case of a data-parallel method that divides training data and assigns it to each worker, all workers have no choice but to train in accordance with the worker with the fewest resources and the number of devices used (such as the number of GPUs).

[0040] Fig. 16 shows a state where learning with PS is performed in a heterogeneous environment. In the cloud computer 1600 shown in Fig. 16, a worker 1602 having two GPUs and workers 1603 and 1604 having three GPUs are mixed, but since all the workers 1602 to 1604 perform distributed learning with the number of GPUs set to two, there are unused GPUs among the workers 1603 and 1604 having three GPUs. In Fig. 16, CPUs and GPUs in use are shown as boxes filled with gray, and unused GPUs are shown as white boxes.

[0041] Fig. 17 shows a state where learning without PS is performed in a heterogeneous environment. In the cloud computer 1700 shown in Fig. 17, a worker 1702 having two GPUs and workers 1701, 1703, and 1704 having three GPUs are mixed, but since all the workers 1701 to 1704 perform distributed learning with the number of GPUs set to two, there are unused GPUs among the workers 1701, 1703, and 1704 having three GPUs. In Fig. 17, CPUs and GPUs in use are shown as boxes filled with gray, and unused GPUs are shown as white boxes.

[0042] In short, when distributed learning is performed in a heterogeneous environment, unused resources are likely to occur in both learning with PS and learning without PS, resulting in a lot of waste.

[0043] In addition, in order to simplify the explanation, in Figures 16 and 17, it is assumed that the resources of each GPU are uniform. In reality, however, there are differences between GPUs in terms of the number of cores, calculation speed, and memory capacity, and resources are not necessarily uniform across all GPUs.

[0044] C. Learning with PS in a heterogeneous environment In this section, we explain how to achieve distributed learning of DNN models in a heterogeneous environment by using a parameter server.

[0045] C-1. Learning with PS using the Connection Manager FIG. 18 shows a configuration example of a distributed learning system 1800 that performs learning with PS in a heterogeneous environment. For simplicity, the figure shows the distributed learning system 1800 consisting of only four computation nodes 1801-1804, but it is assumed that the distributed learning system is actually composed of a large number of computation nodes, five or more. The computation nodes 1801-1804 are interconnected via a network (N / W). Of these, the computation node 1801 operates as a parameter server (and connection manager), and the remaining computation nodes 1802-1804 operate as workers. In FIG. 18, CPUs and GPUs in use are shown as boxes filled with gray, but there are no unused GPUs.

[0046] FIG. 19 shows an operation chart of DNN learning in the distributed learning system 1800 shown in FIG. 18. However, the horizontal axis is the time axis. The parameter server (computation node 1801) inputs the learning data set and the initial state DNN model, and repeats iterations consisting of distributing the divided learning data to each worker of the parameters, gathering the learned parameters from each worker, and updating the DNN model, and outputs the parameters of the learned DNN model. In addition, each of the computation nodes 1802 to 1804 operating as a worker repeats iterations consisting of distributing the parameters from the parameter server, learning, and gathering the learned parameters by the parameter server. However, as described later, it is possible to add and remove workers for each iteration during learning, and it is assumed that the number of workers operating in each iteration is different. In the example shown in FIG. 19, only the computation node 1802 operates as a worker in the first iteration.

[0047] Each of the computation nodes 1801 to 1804 is configured with one CPU or a combination of one CPU and one or more GPUs. The computation nodes 1802 to 1804 operating as workers are not uniform in GPU model, and the number of devices (GPUs) and the computation speed are different for each node. In other words, the distributed learning system 1800 is a heterogeneous environment. In the distributed learning system 1800 in which the GPU model differs for each worker, a mechanism is introduced in which a connection manager manages each worker in order to use the GPUs of all the workers. In the system configuration example shown in FIG. 18, the connection manager also operates on the computation node 1801 operating as a parameter server. Of course, the connection manager may operate on a computation node different from the parameter server.

[0048] First, we will explain the operation of DNN learning in a distributed learning system using a connection manager.

[0049] Fig. 20 shows an example of an operation sequence in which the distributed learning system performs learning with PS using the connection manager. The example of the operation sequence shown in Fig. 20 includes a sequence for registering a new worker. In Fig. 20, the horizontal axis is the time axis.

[0050] The parameter server spawns a connection manager, and the connection manager manages the workers that can use the parameter server. When a worker is launched, it first registers with the connection manager (SEQ2001). When a worker is launched, it registers with the connection manager by sending information about its available resources (for example, the number of GPUs installed in the worker) along with its IP address (Worker IP).

[0051] When the connection manager receives a registration request from a worker including information on the IP address and available resources, it updates the pooled worker information (hereinafter also referred to as the "worker list") (Update pool). The connection manager also returns the DNN model definition (DNN Model Definition) to the worker (SEQ2002). The worker initializes the DNN model (Init model) based on the DNN model definition passed from the connection manager, and then waits for a task to arrive from the parameter server.

[0052] First, the parameter server initializes the input DNN model (Init model) (SEQ2003). Then, the parameter server obtains the registered worker list from the connection manager for each iteration (Get Workers) (SEQ2004). The parameter server uses the obtained worker list for only that one iteration. In other words, the parameter server can change the number of workers during learning (for each iteration) by checking the worker list for each iteration.

[0053] For each iteration, the parameter server creates (Create Task) a task specific for each worker based on information about the available resources of each worker listed in the worker list (SEQ2005), and distributes (Distribute) the task to the IP address of each worker (SEQ2006).

[0054] Each worker trains the DNN model based on the tasks distributed by the parameter server for each iteration (SEQ2007).Then, the parameter server collects the gradient data of the DNN model learned by each worker (SEQ2008), updates the DNN model using the gradient data of the previous worker (SEQ2009), and ends the iteration.

[0055] This iteration is then repeated until the DNN training is complete (e.g., until training has been completed on the entire training dataset).

[0056] By using the connection manager, you can add and remove workers to distribute tasks independently of the iteration. In other words, the parameter server can add and remove workers to distribute tasks for each iteration by getting the worker list from the connection manager for each iteration. The operations for adding and removing workers are explained below.

[0057] C-2. Adding workers Next, we will explain the operation of adding a worker during DNN learning in a distributed learning system that uses a connection manager.

[0058] Fig. 21 shows another example of an operation sequence in which the distributed learning system performs learning with PS using a connection manager. The example of the operation sequence shown in Fig. 21 includes a sequence for adding a worker during distributed learning. Note that in Fig. 21, the horizontal axis is the time axis.

[0059] For each iteration, the parameter server gets the list of workers registered from the connection manager (Get Workers), creates a task specific for each worker (Create Task), and distributes the task to the IP address of each worker (Distribute). In the example shown in FIG. 21, two workers, Worker 1 and Worker 2, are preregistered in the connection manager, and the parameter server distributes Task 1 with a batch size of 32 to Worker 1 and Task 2 with a batch size of 32 to Worker 2 according to the worker list, i.e., the resources of each worker. Worker 1 and Worker 2 each perform training (Train) of the DNN model based on Task 1 and Task 2 distributed (Distribute) from the parameter server.

[0060] Meanwhile, the scheduler spawns (SEQ2101) Worker 3 independently of this iteration. Then, Worker 3 registers (Registers) to the connection manager (SEQ2102). In this registration process, Worker 3 sends its IP address (Worker IP) and its available resource information to the connection manager, and the connection manager returns the DNN model definition (DNN Model Definition) to Worker 3.

[0061] After that, the parameter server collects the gradient data obtained by worker 1 and worker 2 through learning, updates the DNN model, and ends this iteration.

[0062] In the next iteration, the parameter server can get the worker list with the information of worker 3 added from the connection manager (Get workers) and pick up a new worker 3. Then, the parameter server creates a task for each worker (Task specific for each worker) including a task for worker 3 (Create Task) and distributes the task to the IP address of each worker (Distribute) (SEQ2103). In this iteration, the parameter server distributes task 1 with a batch size of 21 to worker 1, task 2 with a batch size of 21 to worker 2, and task 3 with a batch size of 22 to worker 3 according to the resources of each worker. Workers 1 to 3 each train the DNN model based on the tasks distributed from the parameter server. After that, the parameter server collects gradient data obtained by workers 1 to 3 through their learning, updates the DNN model, and ends this iteration.

[0063] C-3. Removal of workers Next, we will explain how to remove a worker during DNN training in a distributed learning system that uses a connection manager. There are two ways to remove a worker: Direct Termination and Connection-manager-initiated Termination.

[0064] Figure 22 shows yet another example of an operation sequence in which the distributed learning system performs learning with PS using the connection manager. The example of the operation sequence shown in Figure 22 includes a sequence in which a worker is removed during distributed learning by Direct Termination. Direct Termination means that the scheduler terminates the operation of a worker at any time immediately, independent of the iteration. Note that in Figure 22, the horizontal axis is the time axis.

[0065] For each iteration, the parameter server gets the list of registered workers from the connection manager (Get Workers), creates a task specific for each worker (Create Task), and distributes the task to the IP address of each worker (Distribute). In the example shown in FIG. 22, workers 1 to 3 are preregistered in the connection manager, and the parameter server attempts to distribute task 1 with a batch size of 21 to worker 1, task 2 with a batch size of 21 to worker 2, and task 3 with a batch size of 22 to worker 3 according to the worker list, i.e., the resources of each worker (SEQ2202).

[0066] Meanwhile, the scheduler terminates worker 3 before task 3 is distributed from the parameter server, independent of this iteration. The scheduler further terminates worker 2 during training after task 2 is distributed from the parameter server (SEQ2201). As a result, the parameter server fails to distribute task 3 to worker 3 in SEQ2202. The parameter server also attempts to gather gradient data obtained by each worker through training, but fails to gather gradient data from terminated worker 2 (SEQ2203). If the parameter server fails to distribute tasks or gather gradient data, it cancels the iteration without updating the DNN model (Cancel iteration / no update) and re-runs training on the same training data in the next iteration.

[0067] In the next iteration, the parameter server gets the worker list with the information of worker 2 and worker 3 deleted (Get workers), creates a task for worker 1 (Task specific for each worker) (Create Task), and distributes task 1 with a batch size of 64 to the IP address of worker 1 (Distribute). Worker 1 trains the DNN model based on the task distributed from the parameter server. After that, the parameter server collects the gradient data obtained by worker 1 through learning (Gather), updates the DNN model (Update), and ends this iteration.

[0068] In the method of removing workers by direct termination, if the number of workers participating in distributed learning is unstable, there is a risk of repeated cancellation of iterations, i.e., frequent cancellation of iterations, which prevents distributed learning from progressing.

[0069] Figure 23 shows yet another example of an operation sequence in which the distributed learning system performs learning with PS using a connection manager. The example of the operation sequence shown in Figure 23 includes a sequence in which a worker is removed during distributed learning by a connection manager-initiated termination. Connection manager-initiated termination means that the operation of a worker is terminated independently of an iteration through the intervention of a connection manager. Note that in Figure 23, the horizontal axis is the time axis.

[0070] For each iteration, the parameter server gets the list of registered workers from the connection manager (Get Workers), creates a task specific for each worker (Create Task), and distributes the task to the IP address of each worker (Distribute). In the example shown in Fig. 23, workers 1 to 3 are preregistered in the connection manager, and the parameter server distributes task 1 with a batch size of 21 to worker 1, task 2 with a batch size of 21 to worker 2, and task 3 with a batch size of 22 to worker 3 according to the worker list, i.e., the resources of each worker.

[0071] Meanwhile, the scheduler sends a Termination Request or Remove Request for Worker 2 and Worker 3 to the connection manager to terminate Worker 2 and Worker 3 independently of this iteration (SEQ2301). Even if the connection manager receives the above request from the scheduler, it does not do anything within this iteration. Therefore, Workers 1 to 3 each train the DNN model based on Tasks 1 to 3 distributed by the parameter server. After that, the parameter server collects the gradient data obtained by Workers 1 to 3 through their learning, updates the DNN model, and ends this iteration.

[0072] In the next iteration, the parameter server obtains from the connection manager a worker list from which information about worker 2 and worker 3 requested to be stopped or removed by the scheduler has been deleted (Get workers). The connection manager also sends a termination signal to each of worker 2 and worker 3 requested to be stopped or removed by the scheduler (SEQ2302). Then, worker 2 and worker 3 receive the termination signal from the connection manager and are stopped within this iteration.

[0073] The parameter server gets the worker list from the connection manager (Get workers) with the information of worker 2 and worker 3 deleted in this iteration, creates a task (Create Task) for worker 1 (Task specific for each worker), and distributes task 1 with a batch size of 64 to the IP address of worker 1 (Distribute). Worker 1 trains the DNN model based on the task distributed from the parameter server. After that, the parameter server collects the gradient data obtained by worker 1 through learning, updates the DNN model, and ends this iteration.

[0074] According to the Connection-manager-initiated Termination shown in FIG. 23, unlike the Direct Termination shown in FIG. 22, the iterations are not canceled, so there is less risk of repeated cancellation of the iterations.

[0075] To summarize the learning with PS using the connection manager explained in Section C, it is possible to add and remove workers to be used for learning for each iteration. The parameter server creates tasks according to the resources of each worker using the worker list passed from the connection manager, so the free resources of each worker can be used effectively.

[0076] D. Distributed learning that enables automatic rebalancing The parameter server divides tasks according to the number of workers used in an iteration and the computation capacity of each worker, and distributes them to each worker. In other words, in distributed DNN learning, the per-worker batch size is adjusted according to the processing performance of each worker (where the sum of all per-worker batch sizes is the minibatch-size). The computation capacity of each worker is automatically updated sequentially, for example, for each iteration, based on the results of monitoring the computation time of previous tasks.

[0077] Fig. 24 shows a schematic functional configuration for updating the computational capabilities of each worker in a parameter server. The illustrated parameter server 2400 includes the following functional modules: a trainer 2401, a task splitter 2402, a dispatcher 2403, and a task monitor 2404. Within the parameter server 240, the following processes are performed for each iteration.

[0078] (Process 1) The trainer 2401 fetches the learning data and obtains the worker list from the connection manager.

[0079] (Process 2) The trainer 2402 sends the learning data and the worker list to the task splitter 2402.

[0080] (Process 3) The task splitter 2402 obtains the [average iteration time / per-worker batch size] of each worker from the task monitor 2404, and divides the mini-batch size of the learning data into the per-worker batch size to create per-worker tasks. That is, based on the result of monitoring the calculation time of the task of each worker, the size of the learning data distributed to each worker (per-worker batch size) is updated for each iteration.

[0081] (Process 4) The task splitter 2402 sends the tasks for all workers to the dispatcher 2403.

[0082] (Process 5) The dispatcher 2403 logs the start time of each worker's iteration (Start Time) in association with each per-worker batch size.

[0083] (Process 6) The dispatcher 2403 distributes tasks to each worker. After the tasks are distributed, each worker executes the learning process. Then, each worker returns the calculated gradient data to the dispatcher 2403.

[0084] (Process 7) The dispatcher 2403 logs the end time of the iteration of each worker in association with the per-worker batch size.

[0085] The task monitor 2404 monitors the computational capacity of each worker by dividing the average iteration time of the worker (the average of the difference between the end time of the iteration and the start time of the iteration) by the batch size of the tasks distributed to that worker. The task splitter 2402 determines the batch size of the tasks to be distributed to each worker based on the monitoring result of the computational capacity of each worker in the above (Process 3), and distributes the learning data.

[0086] (Process 8) The dispatcher 2403 sends the gradient data collected from each worker to the trainer 2401.

[0087] (Process 9) The trainer 2401 combines the gradient data collected from each worker and updates the DNN model.

[0088] E. Effects In this section, the effects of applying the present disclosure to a cluster computer will be described.

[0089] According to the distributed learning system with PS to which the present disclosure is applied, it is possible to add and remove workers to be used for learning for each iteration. In addition, the parameter server creates tasks according to the resources of each worker using the worker list passed from the connection manager, so that the free resources of each worker can be used effectively.

[0090] 25 to 27 show an example of an operation when an important task enters a task queue in the middle of executing a DNN learning task in a cluster computer. According to the present disclosure, it is not necessary to stop the DNN learning task in order to start the important task.

[0091] In the example shown in Fig. 25, a cluster computer 2500 is composed of eight computing nodes #1 to #8, and all computing nodes #1 to #8 are used as workers for the DNN distributed learning task. Assume that an important task enters the task queue 2501 while this DNN distributed learning task is being executed (SEQ2501). The task queue 2501 is a priority queue having a data structure in which tasks are read out in descending order of priority.

[0092] At this time, the scheduler 2502 removes the workers of the DNN distributed learning task until the important task can be launched (SEQ2502). The specific processing procedure for removing the workers of the DNN distributed learning task has already been described in C-3 above, so the details will be omitted here. For example, as shown in FIG. 26, six workers out of eight are removed, and the important task is launched on the six computing nodes, while the remaining two workers continue to execute the DNN distributed learning task (SEQ2503). According to the present disclosure, since it is possible to add and delete workers used in the DNN distributed learning for each iteration, unlike the cases shown in FIG. 11 to FIG. 12, the important task can be started without completely stopping the DNN distributed learning in the cluster computer, and the important task and the DNN distributed learning can be executed simultaneously in parallel.

[0093] After that, when the execution of the important task is terminated, the scheduler 2502 releases the resources used for the important task, i.e., the six computation nodes #3 to #8, and adds them as workers for the DNN distributed learning task. The specific processing procedure for adding workers for the DNN distributed learning task has already been explained in C-3 above, so the details will be omitted here. As a result, as shown in FIG. 27, the state returns to one in which all eight computation nodes #1 to #8 constituting the cluster computer 2500 are used as workers for the DNN distributed learning task.

[0094] FIG. 28 illustrates an example of load fluctuation in the cluster computer 2500 while an important task is being executed. However, the horizontal axis of the figure represents time, and the vertical axis represents load. In the example shown in FIG. 28, the load of the cluster computer 2500 while executing an important task is not uniform, but fluctuates over time, and there are periods of high load when there is no room to execute other tasks, and periods of low load when the DNN learning task can be executed.

[0095] According to the distributed learning system with PS to which the present disclosure is applied, it is possible to add and delete workers to be used in the DNN distributed learning for each iteration. In addition, the parameter server can create tasks according to the resources of each worker using the worker list passed from the connection manager, and can effectively use the free resources of each worker.

[0096] FIG. 29 illustrates the load fluctuation of the cluster computer 2500 consumed for DNN distributed learning when DNN distributed learning is carried out using free resources in the cluster computer 2500 according to the load fluctuation of the cluster computer 2500 executing an important task. FIG. 30 illustrates the load fluctuation of the entire cluster computer 2500 when an important task and DNN distributed learning are simultaneously executed in parallel. As can be seen from FIG. 30, by applying the present disclosure, the cluster computer 2500 can simultaneously execute an important task and DNN distributed learning in parallel, and can fully use the resources of each worker.

[0097] In a heterogeneous environment in which the resources of each worker are not uniform, generally, all workers have no choice but to learn in conjunction with the worker with the fewest resources and the number of devices (such as the number of GPUs) to be used. In contrast, in a cluster computer to which the present disclosure is applied, the connection server manages the workers that can use the parameter server. Therefore, the parameter server can distribute tasks according to the computing capabilities of each worker based on the worker list obtained from the connection manager. In this case, as shown in FIG. 18, all worker devices (such as GPUs) can be effectively utilized without surplus. In FIG. 18, the CPUs and GPUs in use are shown as boxes filled in gray, but there are no unused GPUs. [Industrial Applicability]

[0098] Although the present disclosure has been described in detail above with reference to specific embodiments, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure.

[0099] In this specification, we have mainly described an embodiment in which the present disclosure is applied to distributed learning of DNNs, but the gist of the present disclosure is not limited to this and can be applied to distributed learning of various types of machine learning models.

[0100] In short, the present disclosure has been described in the form of examples, and the contents of the specification should not be interpreted as being limited. In order to judge the gist of the present disclosure, the claims should be taken into consideration.

[0101] The present disclosure can also be configured as follows.

[0102] (1) An administration unit that manages workers available for distributed model training; a parameter server unit that distributes tasks to workers managed by the management unit, collects gradient data obtained by the workers through learning, and updates the model; An information processing device comprising:

[0103] (2) the management unit manages a worker list including resources and addresses of available workers; The parameter server unit distributes tasks to workers based on the worker list acquired from the management unit. The information processing device according to (1) above.

[0104] (3) the parameter server unit acquires the worker list from the management unit for each iteration of the distributed learning; The information processing device according to (2) above.

[0105] (4) The parameter server unit creates and distributes a task specialized for each worker based on information of available resources of each worker for each iteration. An information processing device according to any one of (1) to (3) above.

[0106] (5) The management unit provides definition information of the model to the worker that has acquired the resource and address. An information processing device according to any one of (1) to (4) above.

[0107] (6) The management unit adds information about the added worker to the worker list at any timing within the iteration. An information processing device according to any one of (2) and (3) above.

[0108] (7) The management unit receives a request to stop a worker at any timing within an iteration, stops the worker requested to be stopped in the next iteration, and provides the parameter server unit with a worker list excluding information about the stopped worker. An information processing device according to any one of (2), (3), or (6) above.

[0109] (8) The parameter server unit determines a batch size of tasks to be distributed to the workers based on a monitoring result of the computing power of the workers for each iteration. An information processing device according to any one of (1) to (7) above.

[0110] (9) a management step for managing the workers available for distributed training of the model; and Distributing tasks to workers managed by the managing step; Collecting gradient data obtained by the workers' training; updating the model based on collected gradient data; An information processing method comprising the steps of:

[0111] (10) an administration unit that manages workers available for distributed training of the model; a parameter server unit that distributes tasks to workers managed by the management unit, collects gradient data obtained by the workers through learning, and updates the model; A computer program written in a computer-readable form to cause a computer to function as a

[0112] (11) The workers available for distributed training of the model; A connection manager that manages the workers; a parameter server that distributes tasks to the workers based on a worker list obtained from a connection manager, collects gradient data obtained by the workers through learning, and updates the model; A distributed learning system comprising: [Explanation of symbols]

[0113] 100... distributed learning system, 101... computation node (parameter server) 102~104…Compute nodes (workers) 700: Distributed learning system, 701-704: Computation nodes (workers) 1100: Cluster computer 1101...Task queue (priority queue) 1102...Scheduler 1600…Cloud Computer 1601: Computation node (parameter server) 1602~1604...Compute nodes (workers) 1700…Distributed Learning System 1701~1704…Compute nodes (workers) 1800…Cloud Computer 1801: Computation node (parameter server) 1802~1804...Compute nodes (workers) 2400: Parameter server, 2401: Trainer 2402…Task splitter, 2403…Dispatcher 2404…Task monitor 2500…Cluster computer 2501...Task queue (priority queue) 2502…Scheduler

Claims

1. A management unit that manages workers available for distributed model training; a parameter server unit that distributes tasks to workers managed by the management unit, collects gradient data obtained by the workers through learning, and updates the model; Equipped with The management unit manages a worker list including resources and addresses of available workers; The parameter server unit distributes tasks to workers based on the worker list acquired from the management unit. Information processing device.

2. The parameter server unit obtains the worker list from the management unit for each iteration of the distributed learning. The information processing device according to claim 1 .

3. The parameter server unit creates and distributes specialized tasks for each worker based on information about available resources of each worker for each iteration. The information processing device according to claim 1 .

4. The management unit provides definition information of the model to the worker that has acquired the resource and the address. The information processing device according to claim 1 .

5. The management unit adds information about the added worker to a worker list at any timing within the iteration. The information processing device according to claim 1 .

6. The management unit receives a request to stop a worker at any timing within an iteration, stops the worker requested to be stopped in the next iteration, and provides the parameter server unit with a worker list excluding information about the stopped worker. The information processing device according to claim 1 .

7. The parameter server unit determines a batch size of tasks to be distributed to workers based on a monitoring result of the computing power of the workers for each iteration. The information processing device according to claim 1 .

8. A management step of managing workers available for distributed training of a model; Distributing tasks to workers managed by the managing step; Collecting gradient data obtained by the workers' training; updating the model based on collected gradient data; having The management step includes managing a worker list including resources and addresses of available workers; In the task distribution step, tasks are distributed to workers based on the worker list acquired in the management step. Information processing methods.

9. A management unit that manages workers available for distributed learning of a model; a parameter server unit that distributes tasks to workers managed by the management unit, collects gradient data obtained by the workers through learning, and updates the model; The computer functions as The management unit manages a worker list including resources and addresses of available workers; The parameter server unit distributes tasks to workers based on the worker list acquired from the management unit. A computer program written in a computer-readable form.

10. A worker available for distributed training of a model; a connection manager that manages a worker list including resources and addresses of available workers; a parameter server that distributes tasks to workers based on the worker list obtained from the connection manager, collects gradient data obtained by the workers through learning, and updates the model; A distributed learning system comprising:

Images