Programs, computers and learning methods

Abstract

To decrease standby time until the start of sharing of processing results between nodes in parallel computation.SOLUTION: In learning by a plurality of nodes in deep learning, performance of each node in single item is measured, the number of batches of an entirety of the plurality of nodes is allocated to each node in accordance with the performance, and the allocated batches are processed in each of the nodes.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] The present invention relates to a program, a computer, and a learning method. [Background technology]

[0002] Programs that require huge amounts of calculations, such as High Performance Computing (HPC) and Deep Learning (DL), cannot handle the amount of calculations that a single processor can handle, so parallel calculations are sometimes performed using multiple processors or multiple computer nodes. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent Publication No. 10679145 [Patent Document 2] JP 2018-018422 A [Patent Document 3] JP 2019-067084 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, even if multiple nodes are prepared by a system configured with the same hardware and software, the processing speed between the nodes may differ by about several percent due to the influence of performance variations in processing chips, temperature conditions, etc. As a result, when a first process is executed in each node, and then a second process is executed after waiting for the completion of the first process in all nodes, there is a risk of a decrease in processing performance due to the occurrence of waiting time.

[0005] One aspect of the present invention aims to reduce the waiting time until sharing of processing results between nodes during parallel computing is started. [Means for solving the problem]

[0006] In one aspect, the program measures the performance of each node in deep learning by using multiple nodes, and calculates the performance by calculating the number of batches of the entire multiple nodes. Ratio of Distribute to each node according to After that, the batch set in the node with the lowest performance is reduced by a predetermined size, and the batch set in the node with the highest performance is increased by the predetermined size, thereby performing allocation. At each node, the assigned batches are processed, and the processing is executed by a computer. Effect of the Invention

[0007] According to one aspect, it is possible to reduce the waiting time until sharing of processing results between nodes during parallel computing can begin. [Brief description of the drawings]

[0008] [Figure 1] FIG. 13 is a diagram for explaining a Reduce process in a related example. [Diagram 2] FIG. 13 is a diagram for explaining Allreduce processing in a related example. [Diagram 3] FIG. 11 is a diagram illustrating a first detailed example of the Reduce process in the related example. [Figure 4] FIG. 11 is a diagram illustrating a second detailed example of the Reduce process in the related example. [Diagram 5] FIG. 11 is a diagram illustrating a third detailed example of the Reduce process in the related example. [Figure 6] FIG. 11 is a diagram illustrating a fourth detailed example of the Reduce process in the related example. [Figure 7] FIG. 13 is a diagram for explaining timing wait in Allreduce processing in a related example. [Figure 8] 11A and 11B are diagrams illustrating a process of optimizing the number of batches according to an embodiment; [Figure 9] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer according to an embodiment. [Figure 10] 10 is a block diagram illustrating an example of the software configuration of the computer illustrated in FIG. 9. [Figure 11]10 is a diagram for explaining a measurement process of the throughput in the computer shown in FIG. [Figure 12] 10 is a diagram for explaining a process of setting the number of batches in the computer shown in FIG. 9. FIG. [Figure 13] FIG. 10 is a diagram for explaining the actual operation of batch processing in the computer shown in FIG. [Figure 14] 11 is a flowchart illustrating batch processing according to an embodiment. [Figure 15] FIG. 11 is a diagram illustrating a process of optimizing the number of batches as a first modified example. [Figure 16] 13 is a flowchart illustrating a process of optimizing the number of batches according to a first modified example. [Figure 17] FIG. 11 is a diagram illustrating a process of optimizing the number of batches as a second modified example. [Figure 18] 13 is a flowchart illustrating a process of optimizing the number of batches as a second modified example. [Figure 19] 11A to 11C are diagrams illustrating the effects of the optimization process for the number of batches as an example of an embodiment and each modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [A] Related Examples FIG. 1 is a diagram for explaining the Reduce process in the related example.

[0010] Message Passing Interface (MPI) is a standard for using parallel computing. In the implementation of MPI, the Reduce process is a process that combines the values ​​of multiple nodes 6 into one value using a certain function, as shown in Figure 1.

[0011] As shown in Fig. 1, four nodes 6 have values ​​y0, y1, y2, and y3. When the Reduce process is executed, the values ​​are summed up as shown by symbols A1 to A3, and one value is calculated by a function f(y0, y1, y2, y3) such as addition, multiplication, max, min, etc. as shown by symbol A4.

[0012] FIG. 2 is a diagram for explaining the Allreduce process in the related example.

[0013] The Allreduce process is a process in which the results of the Reduce process are shared among all nodes 6 .

[0014] In the example shown in FIG. 2, the function f(y0, y1, y2, y3) is shared by all the nodes 6, as indicated by the reference characters B1 to B4.

[0015] FIG. 3 is a diagram for explaining a first detailed example of the Reduce process in the related example.

[0016] When DL is performed in node 6, as shown by reference symbol C1, data and a label (in other words, a correct answer) are input to the input layer from the storage 61.

[0017] In FIG. 3, dotted arrows indicate forward processing in the learning process cycle, dashed arrows indicate backward processing in the learning process cycle, and dashed arrows indicate update processing in the learning process cycle.

[0018] As shown by symbol C2, in neuron layer #1, weight parameter w is applied to data from the input layer as forward processing. As shown by symbol C3, in neuron layer #2, weight parameter w is applied to data from neuron layer #1 as forward processing. As shown by symbol C4, in neuron layer #3, weight parameter w is applied to data from neuron layer #2 as forward processing.

[0019] Then, as indicated by the reference symbol C5, in forward processing, the output of the neuron layer #3 is acquired as the recognition result.

[0020] As indicated by the reference symbols C6 to C8, in each of the neuron layers #1 to #3, the difference between the label and the recognition result is input as backward processing.

[0021] As indicated by the reference symbols C9 to C11, in the update process, the differences input in each of the neuron layers #1 to #3 are applied to the weight parameter w as gradient information ∇E.

[0022] The above cycle is repeated many times to obtain appropriate weight parameters.

[0023] FIG. 4 is a diagram for explaining a second detailed example of the Reduce process in the related example.

[0024] In FIG. 4, the diagonal arrows indicate forward processing, the black arrows indicate backward processing, the shaded blocks indicate allreduce processing, and the white arrows indicate update processing.

[0025] When handwritten numbers "6", "7", "3", and "9" are input to nodes #1 to #4, respectively, forward processing is performed in the first to third layers, respectively, as indicated by symbols D1 to D3.

[0026] In the example shown in Fig. 4, in nodes #1 to #4, "5", "1", "8", and "4" are output from the third layer as recognition results, respectively. Then, the difference between the recognition result and the correct label is calculated, and backward processing is performed in each of the third layer to the first layer, as shown by symbols D4 to D6.

[0027] As indicated by the reference symbol D7, in each layer, the differences are aggregated by the Allreduce process, update data is calculated, and the Update process is performed.

[0028] When the process indicated by the reference character D7 is completed, the processes indicated by the reference characters D1 and onwards are started for the next input.

[0029] FIG. 5 is a diagram illustrating a third detailed example of the Reduce process in the related example.

[0030] In distributed DL processing, the gradient information of all nodes 6 is added together, and the added value is distributed to each node 6. After that, the value is divided by the number of nodes, the average is taken, and then the weights are updated at each node.

[0031] In the example shown in FIG. 5, as indicated by the symbol E1, ∇E=∇E1+∇E2+∇E3+∇E4 is calculated in each node by the Allreduce process.

[0032] As shown by symbol E2, ∇E is averaged by dividing ∇E by 4 at each node through averaging processing.

[0033] As indicated by the symbol E3, the weight parameter w is updated in each node by using ∇E / 4 through an update process.

[0034] For example, when the learning rate is η, the weight parameter after update for the weight parameter w1 before the update is calculated as w1'=w1-η(∇E / 4).

[0035] FIG. 6 is a diagram illustrating a fourth detailed example of the Reduce process in the related example.

[0036] In distributed processing of DL, a mini-batch method is sometimes used. In the mini-batch method, each node 6 processes multiple mini-batches at a time, rather than processing each batch one by one.

[0037] In the example shown in FIG. 5, the mini-batch in each node 6 is set to 10, and the global batch size in the four nodes 6 is set to 40.

[0038] As indicated by the reference symbol F1, ∇E=∇E1+∇E2+∇E3+∇E4 is calculated at each node by the Allreduce process.

[0039] As indicated by symbol F2, ∇E is averaged by dividing ∇E by 4*10=40 at each node through averaging processing.

[0040] As indicated by the reference symbol F3, the weight parameter w is updated in each node by using ∇E / 40 through an update process.

[0041] For example, when the learning rate is η, the updated weight parameter for the weight parameter w1 before the update is calculated as w1'=w1-η(∇E / 40).

[0042] FIG. 7 is a diagram for explaining timing wait in the Allreduce process in the related example.

[0043] Even if a system is prepared in which multiple nodes are configured with the same hardware and software, there may be a difference of several percent in processing speed between the nodes due to performance variations in processing chips, temperature conditions, and the like.

[0044] For example, the processing speed of a central processing unit (CPU) or a graphic processing unit (GPU) may vary due to automatic adjustment of the operating clock depending on the temperature.

[0045] In the Allreduce process, communication is basically performed after all processes have completed a single gradient calculation, so processing performance is reduced by the waiting time.

[0046] In the example shown in Fig. 7, 100 batches are allocated to each of processes #1 to #4, and the global batch is 400. In addition, in consideration of processing on multiple CPUs or multiple GPUs using multiple processes, the processing unit is hereinafter referred to as a process.

[0047] As indicated by symbols G1 and G2, the time required for the Forward processing + Backward processing of processes #1 to #3 is shorter than the time required for the Forward processing + Backward processing of process #4. As a result, in processes #1 to #3, a Wait time occurs until the start of the Allreduce processing + Update processing (in other words, until the completion of the Forward processing + Backward processing in process #4).

[0048] [B] Embodiment An embodiment will be described below with reference to the drawings. However, the embodiment described below is merely an example, and there is no intention to exclude the application of various modified examples and techniques not specified in the embodiment. In other words, the present embodiment can be implemented with various modifications within the scope of the gist of the embodiment. In addition, each figure does not intend to include only the components shown in the figure, but can include other functions, etc.

[0049] In the following drawings, the same reference numerals denote similar parts, and therefore the description thereof will be omitted.

[0050] [B-1] Configuration example FIG. 8 is a diagram illustrating the process of optimizing the number of batches according to an embodiment.

[0051] In one embodiment, the number of batches is allocated according to the performance of each process, thereby smoothing the processing time. Note that since the global number of batches is not changed, even if the learning rate is not changed, there is no effect on the learning.

[0052] In the example shown in FIG. 8, 110 batches are distributed to processes #1 to #3, and 70 batches are distributed to process #4, so that no wait time occurs in each process between the end of Forward processing + Backward processing and the start of Allreduce processing + Update processing.

[0053] FIG. 9 is a block diagram illustrating an example of a hardware configuration of a computer 1 according to an embodiment.

[0054] As shown in Fig. 9, computer 1 has a server function, and includes a CPU 11, a memory unit 12, a display control unit 13, a storage device 14, an input interface (IF) 15, an external recording medium processing unit 16, and a communication IF 17. A plurality of computers 1 may be provided as nodes 6 as shown in Fig. 1 and Fig. 2, and parallel calculations may be performed in each computer 1. Also, computer 1 may include a plurality of CPUs 11, and parallel calculations may be performed in each CPU 11.

[0055] The memory unit 12 is an example of a storage unit, and is illustratively a Read Only Memory (ROM) and a Random Access Memory (RAM). Programs such as a Basic Input / Output System (BIOS) may be written in the ROM of the memory unit 12. The software programs in the memory unit 12 may be appropriately read and executed by the CPU 11. The RAM of the memory unit 12 may be used as a temporary recording memory or a working memory.

[0056] The display control unit 13 is connected to the display device 130 and controls the display device 130. The display device 130 is a liquid crystal display, an Organic Light-Emitting Diode (OLED) display, a Cathode Ray Tube (CRT), an electronic paper display, or the like, and displays various information for an operator, etc. The display device 130 may be combined with an input device, for example, a touch panel.

[0057] The storage device 14 is a storage device with high IO performance, and may be, for example, a dynamic random access memory (DRAM), an SSD, a storage class memory (SCM), or an HDD.

[0058] The input IF 15 may be connected to input devices such as a mouse 151 and a keyboard 152, and may control the input devices such as the mouse 151 and the keyboard 152. The mouse 151 and the keyboard 152 are examples of input devices, and an operator performs various input operations via these input devices.

[0059] The external recording medium processing unit 16 is configured so that the recording medium 160 can be attached. The external recording medium processing unit 16 is configured so that, when the recording medium 160 is attached, the information recorded on the recording medium 160 can be read. In this example, the recording medium 160 is portable. For example, the recording medium 160 is a flexible disk, an optical disk, a magnetic disk, a magneto-optical disk, a semiconductor memory, or the like.

[0060] The communication IF 17 is an interface for enabling communication with an external device.

[0061] The CPU 11 is an example of a processor, and is a processing device that performs various controls and calculations. The CPU 11 executes an Operating System (OS) and programs loaded into the memory unit 12, thereby implementing various functions.

[0062] The device for controlling the overall operation of the computer 1 is not limited to the CPU 11, and may be, for example, any one of an MPU, a DSP, an ASIC, a PLD, and an FPGA. The device for controlling the overall operation of the computer 1 may be a combination of two or more of the CPU, MPU, DSP, ASIC, PLD, and FPGA. Note that MPU is an abbreviation for Micro Processing Unit, DSP is an abbreviation for Digital Signal Processor, and ASIC is an abbreviation for Application Specific Integrated Circuit. Also, PLD is an abbreviation for Programmable Logic Device, and FPGA is an abbreviation for Field Programmable Gate Array.

[0063] FIG. 10 is a block diagram illustrating an example of the software configuration of the computer 1 shown in FIG.

[0064] The computer 1 functions as a measurement unit 111, a distribution processing unit 112, and a batch processing unit 113.

[0065] The measurement unit 111 measures the performance (in other words, the throughput) of each individual node. Details of the process performed by the measurement unit 111 will be described later with reference to FIG.

[0066] The distribution processor 112 distributes the number of batches for the entire node to each node according to its performance. Details of the process performed by the distribution processor 112 will be described later with reference to FIG.

[0067] The batch processing unit 113 processes the batches assigned to each node. The details of the processing by the batch processing unit 113 will be described later with reference to FIG.

[0068] FIG. 11 is a diagram for explaining the measurement process of the throughput in the computer 1 shown in FIG.

[0069] The measurement unit 111 measures the individual performance of each process. For example, the learning to be performed may be run, and the number of sheets that can be processed in a certain period T (which may be an appropriate time, number of iterations, or number of epochs) may be calculated as the throughput.

[0070] In the measurement process, communication between nodes and updating of weight parameters are not required, and the number of batches can be relatively large to maximize the speed.

[0071] 11, the number of batches in each of processes #1 to #4 is set to 100. However, as indicated by reference symbol H1, during the throughput measurement period, the processing time of the forward processing + backward processing in processes #1 to #3 is calculated to be shorter than the processing time of the forward processing + backward processing in process #4.

[0072] FIG. 12 is a diagram for explaining the process of setting the batch number in the computer 1 shown in FIG.

[0073] Next, the allocation processor 112 allocates mini-batches to each process according to the throughput of each process, with the global batch number as a set value. For example, if the global batch number is 400 and there are four processes with a throughput ratio of 11:11:11:7, the mini-batch numbers for each process are 110:110:110:70. If the mini-batch number cannot be allocated to each process by an integer, for example, the mini-batch allocation ratio may be limited to a divisor of the global batch number so that the global batch number is not changed.

[0074] In the example shown in FIG. 12, the batch number is allocated during a batch number setting period.

[0075] FIG. 13 is a diagram for explaining the actual operation of the batch processing in the computer 1 shown in FIG.

[0076] The batch processor 113 runs each process according to the batch size set by the allocator 112, and starts parallel learning. As a result, during the actual operation period, the waiting period for the AllReduce process disappears, and learning can be performed efficiently.

[0077] In the example shown in FIG. 13, as indicated by the reference symbol H3, in each of the processes #1 to #4, during the actual operation period, Allreduce processing+Update processing can be executed after Forward processing+Backward processing without any Wait time.

[0078] [B-2] Example of operation The batch processing according to an example of the embodiment will be described with reference to the flowchart (steps S1 to S3) shown in FIG.

[0079] The measurement unit 111 learns each process and calculates the throughput (step S1).

[0080] The allocator 112 sets the number of mini-batches for each process from the number of global batches according to the throughput ratio (step S2).

[0081] The batch processing unit 113 executes normal parallel learning in each process with the set number of mini-batches (step S3), and the batch processing ends.

[0082] [C] First modified example FIG. 15 is a diagram illustrating the optimization process for the number of batches according to the first modified example.

[0083] Depending on the temperature of the CPU 11, the usage status of the computer 1, and the like, the throughput of each process may change over time.

[0084] 15, as shown by reference symbol I1, in processes #1 to #3, a wait time occurs between the completion of forward processing + backward processing and the start of communication processing / update processing, so the processing time is smoothed by setting the number of batches. However, as shown by reference symbol I2, as time passes, a wait time occurs in processes #3 and #4 between the completion of forward processing + backward processing and the start of communication processing / update processing.

[0085] Therefore, in the first modified example, the actual throughput of each process may be measured after one or more learning iterations, or at predetermined time intervals, and the number of batches to be allocated to each processor may be changed.

[0086] The process of optimizing the number of batches as the first modified example will be described with reference to the flowchart (steps S11 to S16) shown in FIG.

[0087] The batch processing unit 113 performs initial setting of the number of learning processes, target accuracy, etc. (step S11).

[0088] The allocation processor 112 sets the batch change timing (step S12). The batch change timing may be every 100 learning iterations, every 30 minutes of learning, or when the Wait time exceeds a threshold.

[0089] The batch processing unit 113 performs learning in each process (step S13).

[0090] The batch processing unit 113 determines whether learning has been completed a specified number of times or whether a target accuracy has been reached (step S14).

[0091] When the learning is completed a specified number of times or the target accuracy is reached (see YES route in step S14), the optimization process of the batch number as the first modified example ends.

[0092] On the other hand, if the learning has not been completed the specified number of times and the target accuracy has not been reached (see NO route from step S14), the allocation processing unit 112 determines whether the batch change timing has arrived (step S15).

[0093] If the batch change timing has not yet arrived (see the NO route from step S15), the process returns to step S13.

[0094] On the other hand, if the batch change timing is reached (see YES route in step S15), the allocator 112 stops learning all the processes, calculates and changes the batch size of each process from the throughput (step S16), and then the process returns to step S13.

[0095] [D] Second Modification FIG. 17 is a diagram illustrating the process of optimizing the number of batches according to the second modified example.

[0096] In the second modification, the batch size of the process with the longest wait time (in other words, the fastest processing) is increased by an arbitrary size k, and the batch size of the process with no wait time (in other words, the slowest processing) is decreased by k. By repeatedly performing the optimization process, the number of batches in each process converges to an optimal value.

[0097] In the example shown in Figure 17, as shown by symbol J1, the wait time in process #3 is the longest, while there is no wait time in process #4. Therefore, the batch size in process #3 is increased by k, and the batch size in process #4 is decreased by k. This allows the learning time in each process to be reduced.

[0098] The optimization process for the batch number in the second modified example may be combined with the optimization process for the batch number in the example of the embodiment and / or the optimization process for the batch number in the first modified example. For example, after the batch number of each process is roughly allocated in the optimization process for the batch number in the example of the embodiment, the optimization process for the batch number in the second modified example may aim to optimize the batch number.

[0099] The process of optimizing the number of batches as the second modified example will be described with reference to the flowchart (steps S21 to S26) shown in FIG.

[0100] The batch processing unit 113 performs initial setting of the number of learning processes, target accuracy, etc. (step S21).

[0101] The allocation processor 112 sets the batch change timing and the transfer batch size (step S22). The batch change timing may be every 100 learning iterations, every 30 minutes of learning, or when the wait time exceeds a threshold. The transfer batch size is the batch size that is moved from a process with no wait time (in other words, slow processing) to a process with the longest wait time (in other words, fast processing).

[0102] The batch processing unit 113 performs learning in each process (step S23).

[0103] The batch processing unit 113 determines whether learning has been completed a specified number of times or whether a target accuracy has been reached (step S24).

[0104] When the learning is completed a specified number of times or the target accuracy is reached (see YES route in step S24), the optimization process of the batch number as the second modified example ends.

[0105] On the other hand, if the learning has not been completed the specified number of times and the target accuracy has not been reached (see NO route from step S24), the allocation processing unit 112 determines whether the batch change timing has arrived (step S25).

[0106] If the batch change timing has not yet arrived (see the NO route from step S25), the process returns to step S23.

[0107] On the other hand, if the batch change timing is reached (see the YES route in step S25), the allocation processor 112 stops learning all the processes and transfers the designated number of batches from the process with the slowest processing speed to the process with the fastest processing speed (step S26). Then, the process returns to step S23.

[0108] [E] Effect FIG. 19 is a diagram illustrating the effect of the batch number optimization process according to the example of the embodiment and each of the modified examples.

[0109] When training using multiple processes, adjusting the number of mini-batches per process according to the throughput can reduce communication waiting time and, as a result, reduce the overall training time.

[0110] 19, as shown in K1, 100 batches are assigned to each process, and there is wait time in processes #1 to #3, but there is no wait time in process #4. Therefore, as shown in K2, the batch numbers of processes #1 to #3, in which wait time is occurring, are increased to 110, and the batch number of process #4, which has no wait time, is decreased to 70, thereby reducing the processing time of the entire processes.

[0111] According to the program, the calculator 1, and the learning method in the example of the embodiment and each of the modifications described above, for example, the following advantageous effects can be achieved.

[0112] The measurement unit 111 measures the performance of each individual node in deep learning by multiple nodes. The allocation processing unit 112 measures the performance of each individual node in deep learning by multiple nodes. The batch processing unit 113 processes the batches allocated to each node.

[0113] This makes it possible to reduce the waiting time until the sharing of processing results between nodes during parallel computation begins.

[0114] The allocation processor 112 allocates the batches according to the ratio of the performances, thereby enabling efficient allocation of the number of batches.

[0115] The allocation processor 112 measures the performance and allocates the batches every predetermined number of iterations of learning. The allocation processor 112 measures the performance and allocates the batches every predetermined time. This makes it possible to accurately measure the performance and allocate the batches even if the temperature of the CPU 11 or the usage pattern of the computer 1 changes over time.

[0116] The allocation processor 112 allocates the batches by decreasing the batch size set for the node with the lowest performance by a predetermined size and increasing the batch size set for the node with the highest performance by a predetermined size. This allows fine-grained allocation of the number of batches.

[0117] [F] Other The disclosed technology is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present embodiment. Each configuration and each process of the present embodiment can be selected as necessary, or can be combined as appropriate.

[0118] In the above-described example of the embodiment and each of the modified examples, the number of batches is allocated according to the measurement result of the throughput, but this is not limited to this. For example, the clock of the CPU 111 or the GPU may be monitored, and the number of batches may be allocated according to the monitoring result of the clock.

[0119] [G] Notes The following supplementary notes are further disclosed regarding the above embodiment.

[0120] (Appendix 1) In deep learning, when multiple nodes are used for learning, the performance of each node is measured. Allocating the number of batches across the multiple nodes to each node according to the performance. Each node processes the assigned batches. A program that causes a computer to carry out processing.

[0121] (Appendix 2) The allocation is performed according to the performance ratio. 2. The program of claim 1, which causes the computer to execute a process.

[0122] (Appendix 3) measuring the performance and making the allocation after each predetermined number of iterations of the learning; 3. The program according to claim 1 or 2, which causes the computer to execute the process.

[0123] (Appendix 4) measuring the performance at predetermined time intervals and performing the allocation; 3. The program according to claim 1 or 2, which causes the computer to execute the process.

[0124] (Appendix 5) performing the allocation by reducing the batch set in the node with the lowest performance by a predetermined size and increasing the batch set in the node with the highest performance by the predetermined size; 5. The program according to any one of claims 1 to 4, which causes the computer to execute a process.

[0125] (Appendix 6) A measurement unit that measures the performance of each individual node in deep learning using multiple nodes; an allocation processing unit that allocates the number of batches across the plurality of nodes to each node in accordance with the performance; a batch processing unit for processing the assigned batches in each node; A computer comprising:

[0126] (Appendix 7) The allocation processing unit performs the allocation according to the ratio of the performances. 6. A computer as described in appended claim 6.

[0127] (Appendix 8) The allocation processing unit measures the performance and performs the allocation every predetermined number of iterations of the learning. 8. A computer according to claim 6 or 7.

[0128] (Appendix 9) The distribution processing unit measures the performance at predetermined time intervals and performs the distribution. 8. A computer according to claim 6 or 7.

[0129] (Appendix 10) the allocation processing unit performs the allocation by reducing a batch set in the node with the lowest performance by a predetermined size and increasing a batch set in the node with the highest performance by the predetermined size; 10. The computer according to any one of claims 6 to 9.

[0130] (Appendix 11) In deep learning, when multiple nodes are used for learning, the performance of each node is measured. Allocating the number of batches across the multiple nodes to each node according to the performance. Each node processes the assigned batches. A learning method in which processing is carried out by a computer.

[0131] (Appendix 12) The allocation is performed according to the performance ratio. 12. The learning method of claim 11, wherein the processing is performed by a computer.

[0132] (Appendix 13) measuring the performance and making the allocation after each predetermined number of iterations of the learning; 13. The learning method according to claim 11 or 12, wherein the processing is executed by the computer.

[0133] (Appendix 14) measuring the performance at predetermined time intervals and performing the allocation; 13. The learning method according to claim 11 or 12, wherein the processing is executed by the computer.

[0134] (Appendix 15) performing the allocation by reducing the batch set in the node with the lowest performance by a predetermined size and increasing the batch set in the node with the highest performance by the predetermined size; 15. The learning method according to any one of claims 11 to 14, wherein processing is executed by the computer. [Explanation of symbols]

[0135] 1: Calculator 11: CPU 111: Measuring section 112: Distribution processing unit 113: Batch processing unit 12: Memory section 13: Display control section 130:Display device 14:Storage device 15: Input IF 151: Mouse 152: Keyboard 16: External recording medium processing unit 160: Recording media 17: Communication Interface 6 : Node 61: Storage

Claims

1. In deep learning, when multiple nodes are used for learning, the performance of each node is measured. After allocating the number of batches of all the nodes to each node according to the ratio of the performances, the allocation is performed by decreasing the batch set to the node with the lowest performance by a predetermined size and increasing the batch set to the node with the highest performance by the predetermined size; Each node processes the assigned batches. A program that causes a computer to carry out processing.

2. measuring the performance and making the allocation after each predetermined number of iterations of the learning; The program according to claim 1 , which causes the computer to execute a process.

3. measuring the performance at predetermined time intervals and performing the allocation; The program according to claim 1 , which causes the computer to execute a process.

4. A measurement unit that measures the performance of each individual node in deep learning using multiple nodes; an allocation processing unit that allocates the number of batches of all the nodes to each node according to the ratio of the performances, and then reduces the batch set in the node with the lowest performance by a predetermined size and increases the batch set in the node with the highest performance by the predetermined size, thereby performing the allocation; a batch processing unit for processing the assigned batches in each node; A computer comprising:

5. In deep learning, when multiple nodes are used for learning, the performance of each node is measured. After allocating the number of batches of all the nodes to each node according to the ratio of the performances, the allocation is performed by decreasing the batch set to the node with the lowest performance by a predetermined size and increasing the batch set to the node with the highest performance by the predetermined size; Each node processes the assigned batches. A learning method in which processing is carried out by a computer.

Images