Programs, computers and learning methods

Abstract

To reduce the time of queuing until sharing of a processing result is started between nodes in parallel calculation.SOLUTION: In learning according to a plurality of nodes in deep learning, a batch number corresponding to the respective performance of the plurality of nodes is assigned to the plurality of nodes, respectively, or the learning is determined to terminate at a prescribed timing, in order to adjust a learning rate to be used for the learning, in accordance with a ratio of a preset setting batch number for the plurality of nodes, to an execution batch number executed according to the assignment to the plurality of nodes or executed before the prescribed timing.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] The present invention relates to a program, a computer, and a learning method. [Background technology]

[0002] Programs that require huge amounts of calculations, such as High Performance Computing (HPC) and Deep Learning (DL), often require parallel calculations using multiple processors or multiple computer nodes, because a single processor is insufficient to handle the amount of calculations required. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2018-518744 [Patent Document 2] U.S. Patent Publication No. 2019 / 0258964 Summary of the Invention [Problem to be solved by the invention]

[0004] However, even if multiple nodes are prepared by a system configured with the same hardware and software, the processing speed between the nodes may differ by about several percent due to the influence of performance variations in processing chips, temperature conditions, etc. As a result, when a first process is executed in each node, and then a second process is executed after waiting for the completion of the first process in all nodes, there is a risk of a decrease in processing performance due to the occurrence of waiting time.

[0005] One aspect of the present invention aims to reduce the waiting time until sharing of processing results between nodes during parallel computing is started. [Means for solving the problem]

[0006] In one aspect, the program is a deep learning model for learning by multiple nodes. , place A predetermined number of batches for the plurality of nodes is determined to be a predetermined number of batches for the plurality of nodes, and the number of batches for the plurality of nodes is determined to be a predetermined number of batches for the plurality of nodes. Before The learning rate used in the learning is adjusted according to the ratio to the number of execution batches executed before the predetermined timing. Effect of the Invention

[0007] According to one aspect, it is possible to reduce the waiting time until sharing of processing results between nodes during parallel computing can begin. [Brief description of the drawings]

[0008] [Figure 1] FIG. 13 is a diagram for explaining a Reduce process in a related example. [Diagram 2] FIG. 13 is a diagram for explaining Allreduce processing in a related example. [Diagram 3] FIG. 11 is a diagram illustrating a first detailed example of the Reduce process in the related example. [Figure 4] FIG. 11 is a diagram illustrating a second detailed example of the Reduce process in the related example. [Diagram 5] FIG. 11 is a diagram illustrating a third detailed example of the Reduce process in the related example. [Figure 6] FIG. 11 is a diagram illustrating a fourth detailed example of the Reduce process in the related example. [Figure 7] FIG. 13 is a diagram for explaining timing wait in Allreduce processing in a related example. [Figure 8] 11A and 11B are diagrams illustrating a process of optimizing the number of batches according to an embodiment; [Figure 9] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer according to an embodiment. [Figure 10] 10 is a block diagram illustrating an example of the software configuration of the computer illustrated in FIG. 9. [Figure 11]10 is a diagram for explaining a measurement process of the throughput in the computer shown in FIG. [Figure 12] 10 is a diagram for explaining a process of setting the number of batches in the computer shown in FIG. 9. FIG. [Figure 13] FIG. 10 is a diagram for explaining the actual operation of batch processing in the computer shown in FIG. [Figure 14] 11 is a flowchart illustrating batch processing according to an embodiment. [Figure 15] FIG. 11 is a diagram illustrating a process of optimizing the number of batches as a first modified example. [Figure 16] 13 is a flowchart illustrating a process of optimizing the number of batches according to a first modified example. [Figure 17] FIG. 11 is a diagram illustrating preconditions in the optimization process for the number of batches as a second modified example. [Figure 18] FIG. 13 is a diagram illustrating a process of optimizing the number of batches when a specified process has completed a target number of batches, according to a second modified example. [Figure 19] FIG. 11 is a diagram illustrating a process of optimizing the number of batches when a specified time has elapsed as a second modified example. [Figure 20] FIG. 13 is a diagram illustrating a batch number optimization process according to a second modified example in which all processes have completed a target number of batches. [Figure 21] 13 is a flowchart illustrating a process of optimizing the number of batches as a second modified example. [Figure 22] FIG. 13 is a diagram illustrating a process of optimizing the number of batches as a third modified example. [Diagram 23] 13 is a flowchart illustrating a process of optimizing the number of batches according to a third modified example. [Figure 24] 11A to 11C are diagrams illustrating the effects of the optimization process for the number of batches as an example of an embodiment and each modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [A] Related Examples FIG. 1 is a diagram for explaining the Reduce process in the related example.

[0010] Message Passing Interface (MPI) is a standard for using parallel computing. In the implementation of MPI, the Reduce process is a process that combines the values ​​of multiple nodes 6 into one value using a certain function, as shown in Figure 1.

[0011] As shown in Fig. 1, four nodes 6 have values ​​y0, y1, y2, and y3. When the Reduce process is executed, the values ​​are summed up as shown by symbols A1 to A3, and one value is calculated by a function f(y0, y1, y2, y3) such as addition, multiplication, max, min, etc. as shown by symbol A4.

[0012] FIG. 2 is a diagram for explaining the Allreduce process in the related example.

[0013] The Allreduce process is a process in which the results of the Reduce process are shared among all nodes 6 .

[0014] In the example shown in FIG. 2, the function f(y0, y1, y2, y3) is shared by all the nodes 6, as indicated by the reference characters B1 to B4.

[0015] FIG. 3 is a diagram for explaining a first detailed example of the Reduce process in the related example.

[0016] When DL is performed in node 6, as indicated by reference symbol C1, data and a label (in other words, a correct answer) are input to the input layer from the storage 61.

[0017] In FIG. 3, dotted arrows indicate forward processing in the learning process cycle, dashed arrows indicate backward processing in the learning process cycle, and dashed arrows indicate update processing in the learning process cycle.

[0018] As shown by symbol C2, in neuron layer #1, weight parameter w is applied to data from the input layer as forward processing. As shown by symbol C3, in neuron layer #2, weight parameter w is applied to data from neuron layer #1 as forward processing. As shown by symbol C4, in neuron layer #3, weight parameter w is applied to data from neuron layer #2 as forward processing.

[0019] Then, as indicated by the reference symbol C5, in forward processing, the output of the neuron layer #3 is acquired as the recognition result.

[0020] As indicated by the reference symbols C6 to C8, in each of the neuron layers #1 to #3, the difference between the label and the recognition result is input as backward processing.

[0021] As indicated by the reference symbols C9 to C11, in the update process, the differences input in each of the neuron layers #1 to #3 are applied to the weight parameter w as gradient information ∇E.

[0022] The above cycle is repeated many times to obtain appropriate weight parameters.

[0023] FIG. 4 is a diagram for explaining a second detailed example of the Reduce process in the related example.

[0024] In FIG. 4, the diagonal arrows indicate forward processing, the black arrows indicate backward processing, the shaded blocks indicate allreduce processing, and the white arrows indicate update processing.

[0025] When handwritten numbers "6", "7", "3", and "9" are input to nodes #1 to #4, respectively, forward processing is performed in the first to third layers, respectively, as indicated by symbols D1 to D3.

[0026] In the example shown in Fig. 4, in nodes #1 to #4, "5", "1", "8", and "4" are output from the third layer as recognition results, respectively. Then, the difference between the recognition result and the correct label is calculated, and backward processing is performed in each of the third layer to the first layer, as shown by symbols D4 to D6.

[0027] As indicated by the reference symbol D7, in each layer, the differences are aggregated by the Allreduce process, update data is calculated, and the Update process is performed.

[0028] When the process indicated by the reference character D7 is completed, the processes indicated by the reference characters D1 and onwards are started for the next input.

[0029] FIG. 5 is a diagram illustrating a third detailed example of the Reduce process in the related example.

[0030] In distributed DL processing, the gradient information of all nodes 6 is added together, and the added value is distributed to each node 6. After that, the value is divided by the number of nodes, the average is taken, and then the weights are updated at each node.

[0031] In the example shown in FIG. 5, as indicated by the reference symbol F1, ∇E=∇E1+∇E2+∇E3+∇E4 is calculated at each node by the Allreduce process.

[0032] As shown by symbol F2, ∇E is averaged by dividing ∇E by 4 at each node through averaging processing.

[0033] As indicated by the reference symbol F3, the weight parameter w is updated in each node by using ∇E / 4 through an update process.

[0034] For example, when the learning rate is η, the weight parameter after update for the weight parameter w1 before the update is calculated as w1'=w1-η(∇E / 4).

[0035] FIG. 6 is a diagram illustrating a fourth detailed example of the Reduce process in the related example.

[0036] In distributed processing of DL, a mini-batch method is sometimes used. In the mini-batch method, each node 6 processes multiple mini-batches at a time, rather than processing each batch one by one.

[0037] In the example shown in FIG. 6, the mini-batch in each node 6 is set to 10, and the global batch number in the four nodes 6 is set to 40.

[0038] As indicated by the symbol F4, ∇E=∇E1+∇E2+∇E3+∇E4 is calculated at each node by the Allreduce process.

[0039] As indicated by symbol F5, ∇E is averaged by dividing ∇E by 4*10=40 at each node through averaging processing.

[0040] As indicated by the reference symbol F6, the weight parameter w is updated in each node by using ∇E / 40 through an update process.

[0041] For example, when the learning rate is η, the updated weight parameter for the weight parameter w1 before the update is calculated as w1'=w1-η(∇E / 40).

[0042] FIG. 7 is a diagram for explaining timing wait in the Allreduce process in the related example.

[0043] Even if a system is prepared in which multiple nodes are configured with the same hardware and software, the processing speeds of the nodes may differ by a few percent due to performance variations in processing chips, temperature conditions, and other factors.

[0044] For example, the processing speed of a central processing unit (CPU) or a graphic processing unit (GPU) may vary due to automatic adjustment of the operating clock depending on the temperature.

[0045] In the Allreduce process, communication basically occurs after all processes have completed a single gradient calculation, so processing performance decreases by the waiting time.

[0046] In the example shown in Fig. 7, 100 batches are allocated to each of processes #1 to #4, and the global batch is 400. In addition, in consideration of processing on multiple CPUs or multiple GPUs using multiple processes, the processing unit is hereinafter referred to as a process.

[0047] As indicated by symbols G1 and G2, the time required for the Forward processing + Backward processing of processes #1 to #3 is shorter than the time required for the Forward processing + Backward processing of process #4. As a result, in processes #1 to #3, a Wait time occurs until the start of the Allreduce processing + Update processing (in other words, until the completion of the Forward processing + Backward processing in process #4).

[0048] [B] Embodiment An embodiment will be described below with reference to the drawings. However, the embodiment described below is merely an example, and there is no intention to exclude the application of various modified examples and techniques not specified in the embodiment. In other words, the present embodiment can be implemented with various modifications within the scope of the gist of the embodiment. In addition, each figure does not intend to include only the components shown in the figure, but can include other functions, etc.

[0049] In the following drawings, the same reference numerals denote similar parts, and therefore the description thereof will be omitted.

[0050] [B-1] Configuration example FIG. 8 is a diagram illustrating the process of optimizing the number of batches according to an embodiment.

[0051] In one embodiment, the number of batches (in other words, the "mini-batch number" or "batch size") is changed according to the performance of each process, thereby reducing the wait time and shortening the final learning time. Although the total number of completed batches changes from the global batch depending on the throughput and timing, learning with the same accuracy can be achieved by adjusting the learning rate in proportion to the change.

[0052] As shown in the following formula (1), if the number of samples is sufficiently large compared to the number of mini-batches, the learning rate is proportional to the number of batches when the noise scale is constant. In other words, even if the actual total number of training batches changes from the set global batch number, learning with the same accuracy is possible by adjusting the learning rate in proportion to the change.

[0053]

number

[0054] In the above formula (1), batch and batch' respectively indicate the batch numbers before and after the update, and η and η' respectively indicate the learning rates before and after the update.

[0055] 8, the original global batch number is 400, whereas 100 batch numbers are set for processes #1 to #3, and 80 batch numbers are allocated to process #4, making the total batch number 380. As a result, no wait time occurs in each process between the end of Forward processing + Backward processing and the start of Allreduce processing + Update processing.

[0056] FIG. 9 is a block diagram illustrating an example of a hardware configuration of a computer 1 according to an embodiment.

[0057] As shown in Fig. 9, computer 1 has a server function, and includes a CPU 11, a memory unit 12, a display control unit 13, a storage device 14, an input interface (IF) 15, an external recording medium processing unit 16, and a communication IF 17. A plurality of computers 1 may be provided as nodes 6 as shown in Fig. 1 and Fig. 2, and parallel calculations may be performed in each computer 1. Also, computer 1 may include a plurality of CPUs 11, and parallel calculations may be performed in each CPU 11.

[0058] The memory unit 12 is an example of a storage unit, and is illustratively a Read Only Memory (ROM) and a Random Access Memory (RAM). Programs such as a Basic Input / Output System (BIOS) may be written in the ROM of the memory unit 12. The software programs in the memory unit 12 may be appropriately read and executed by the CPU 11. The RAM of the memory unit 12 may be used as a temporary recording memory or a working memory.

[0059] The display control unit 13 is connected to the display device 130 and controls the display device 130. The display device 130 is a liquid crystal display, an Organic Light-Emitting Diode (OLED) display, a Cathode Ray Tube (CRT), an electronic paper display, or the like, and displays various information for an operator, etc. The display device 130 may be combined with an input device, for example, a touch panel.

[0060] The storage device 14 is a storage device with high IO performance, and may be, for example, a dynamic random access memory (DRAM), an SSD, a storage class memory (SCM), or an HDD.

[0061] The input IF 15 may be connected to input devices such as a mouse 151 and a keyboard 152, and may control the input devices such as the mouse 151 and the keyboard 152. The mouse 151 and the keyboard 152 are examples of input devices, and an operator performs various input operations via these input devices.

[0062] The external recording medium processing unit 16 is configured so that the recording medium 160 can be attached. The external recording medium processing unit 16 is configured so that, when the recording medium 160 is attached, the information recorded on the recording medium 160 can be read. In this example, the recording medium 160 is portable. For example, the recording medium 160 is a flexible disk, an optical disk, a magnetic disk, a magneto-optical disk, a semiconductor memory, or the like.

[0063] The communication IF 17 is an interface for enabling communication with an external device.

[0064] The CPU 11 is an example of a processor, and is a processing device that performs various controls and calculations. The CPU 11 executes an Operating System (OS) and programs loaded into the memory unit 12, thereby implementing various functions.

[0065] The device for controlling the overall operation of the computer 1 is not limited to the CPU 11, and may be, for example, any one of an MPU, a DSP, an ASIC, a PLD, and an FPGA. The device for controlling the overall operation of the computer 1 may be a combination of two or more of the CPU, MPU, DSP, ASIC, PLD, and FPGA. Note that MPU is an abbreviation for Micro Processing Unit, DSP is an abbreviation for Digital Signal Processor, and ASIC is an abbreviation for Application Specific Integrated Circuit. Also, PLD is an abbreviation for Programmable Logic Device, and FPGA is an abbreviation for Field Programmable Gate Array.

[0066] FIG. 10 is a block diagram illustrating an example of the software configuration of the computer 1 shown in FIG.

[0067] The calculator 1 functions as a measurement unit 111 , a batch number setting unit 112 , and a batch processing unit 113 .

[0068] The measurement unit 111 measures the performance (in other words, the throughput) of each individual node. Details of the process performed by the measurement unit 111 will be described later with reference to FIG.

[0069] The batch number setting unit 112 allocates the batch number to each node according to the performance of each node. Details of the process by the batch number setting unit 112 will be described later with reference to FIG.

[0070] The batch processing unit 113 processes the batches set in each node. The details of the processing by the batch processing unit 113 will be described later with reference to FIG.

[0071] FIG. 11 is a diagram for explaining the measurement process of the throughput in the computer 1 shown in FIG.

[0072] The measurement unit 111 measures the individual performance of each process. For example, the learning to be performed may be run, and the number of sheets that can be processed in a certain period T (which may be an appropriate time, number of iterations, or number of epochs) may be calculated as the throughput.

[0073] In the measurement process, communication between nodes and updating of weight parameters are not required, and the number of batches can be relatively large to maximize the speed.

[0074] 11, the number of batches in each of processes #1 to #4 is set to 100. However, as indicated by reference symbol H1, during the throughput measurement period, the processing time of the forward processing + backward processing in processes #1 to #3 is calculated to be shorter than the processing time of the forward processing + backward processing in process #4.

[0075] FIG. 12 is a diagram for explaining the process of setting the batch number in the computer 1 shown in FIG.

[0076] Next, the batch number setting unit 112 allocates mini-batches to each process according to the throughput ratio of each process, with the global batch number as a setting value. For example, if the global batch number is 400 and there are four processes with a throughput ratio of 10:10:10:9, the mini-batch numbers of each process are 100:100:100:90, and the total number of batches is 390.

[0077] 12, the batch number is allocated in a batch number setting period indicated by reference symbol H2. Also, the learning rate is changed from the original learning rate η to a new learning rate η'=η*390 / 400.

[0078] That is, in learning by a plurality of nodes in deep learning, the batch number setting unit 112 assigns a batch number according to the performance of each of the plurality of nodes to each of the plurality of nodes. Also, the batch number setting unit 112 adjusts a learning rate used in learning according to a ratio between a preset set batch number for the plurality of nodes and an execution batch number executed by the allocation in the plurality of nodes.

[0079] FIG. 13 is a diagram for explaining the actual operation of the batch processing in the computer 1 shown in FIG.

[0080] The batch processing unit 113 runs each process with the batch number and new learning rate η' set by the batch number setting unit 112, and starts parallel learning. As a result, the waiting period for the Allreduce process disappears during the actual operation period, and learning can be performed efficiently. Note that the number of images to be learned in one epoch does not change.

[0081] In the example shown in FIG. 13, as indicated by the reference symbol H3, in each of the processes #1 to #4, during the actual operation period, Allreduce processing+Update processing can be executed after Forward processing+Backward processing without any Wait time.

[0082] [B-2] Example of operation The batch processing according to an example of the embodiment will be described with reference to the flowchart (steps S1 to S3) shown in FIG.

[0083] The measurement unit 111 learns each process and calculates the throughput (step S1).

[0084] The batch number setting unit 112 sets the number of mini-batches for each process from the global batch number according to the throughput ratio (step S2).

[0085] The batch processing unit 113 executes normal parallel learning in each process with the set number of mini-batches (step S3), and the batch processing ends.

[0086] [C] First modified example FIG. 15 is a diagram illustrating the optimization process for the number of batches according to the first modified example.

[0087] Depending on the temperature of the CPU 11, the usage status of the computer 1, and the like, the throughput of each process may change over time.

[0088] 15, as shown by reference symbol I1, a wait time occurs in processes #1 to #3 between the completion of forward processing + backward processing and the start of communication processing / update processing, so the processing time is smoothed by setting the batch number and learning rate η'. However, as shown by reference symbol I2, as time passes, a wait time occurs in processes #3 and #4 between the completion of forward processing + backward processing and the start of communication processing / update processing.

[0089] Therefore, in the first modified example, the actual throughput of each process may be measured after one or more learning iterations, or at predetermined time intervals, and the number of batches allocated to each processor may be changed while changing the learning rate η' to a learning rate η''.

[0090] That is, the batch number setting unit 112 may measure the performance and allocate the batch number every predetermined number of iterations of learning. The batch number setting unit 112 may measure the performance and allocate the batch number every predetermined time.

[0091] The process of optimizing the number of batches as the first modified example will be described with reference to the flowchart (steps S11 to S16) shown in FIG.

[0092] The batch processing unit 113 performs initial settings such as the number of learning processes or the target accuracy (step S11).

[0093] The batch number setting unit 112 sets the batch change timing (step S12). The batch change timing may be every 100 learning iterations, every 30 minutes of learning, or when the Wait time exceeds a threshold.

[0094] The batch processing unit 113 performs learning in each process (step S13).

[0095] The batch processing unit 113 determines whether learning has been completed a specified number of times or whether a target accuracy has been reached (step S14).

[0096] When the learning is completed a specified number of times or the target accuracy is reached (see YES route in step S14), the optimization process of the batch number as the first modified example ends.

[0097] On the other hand, if learning has not been completed the specified number of times and the target accuracy has not been reached (see NO route from step S14), the batch number setting unit 112 determines whether the batch change timing has arrived (step S15).

[0098] If the batch change timing has not yet arrived (see the NO route from step S15), the process returns to step S13.

[0099] On the other hand, if the batch change timing is reached (see YES route in step S15), the batch number setting unit 112 stops learning of all processes, calculates and changes the batch number and new learning rate of each process from the throughput (step S16), and the process returns to step S13.

[0100] [D] Second Modification FIG. 17 is a diagram for explaining preconditions in the process of optimizing the number of batches as the second modified example.

[0101] Usually, in the mini-batch method, the procedure is to calculate the number of batches in order from the previous layer before moving to the next layer. In the example shown by the symbol J1, the mini-batch subsets of Layer 1 are processed in groups of two from "1" to "8" in order, then the mini-batch subsets of Layer 2 are processed in groups of two from "1" to "8" in order, the mini-batch subsets of Layer 3 are processed in groups of two from 1 to 8 in order, and finally the mini-batch subsets of Layer 4 are processed in groups of two from 1 to 8 in order.

[0102] On the other hand, in the optimization process of the batch number in the second embodiment, after two mini-batch subsets are processed in all layers, the next two mini-batch subsets are processed in all layers. In the example shown by the symbol J2, after "1" and "2" of the mini-batch subsets are processed in Layers 1 to 4, "3" and "4" of the mini-batch subsets are processed in Layers 1 to 4, "5" and "6" of the mini-batch subsets are processed in Layers 1 to 4, and finally "7" and "8" of the mini-batch subsets are processed in Layers 1 to 4.

[0103] In the second modified example, the batch processing unit 113 forcibly executes the Allreduce process at a specific timing. The specific timing may be, for example, the timing when a specified process completes a target number of batches, the timing when a specified time has elapsed, or the timing when all processes complete a target number of batches.

[0104] For processes with fast processing speeds, if the process is allowed to continue learning without stopping even after a specified number of batches have been processed, the wait time for each process can be reduced, thereby speeding up learning.

[0105] That is, the batch number setting unit 112 determines to terminate learning by a plurality of nodes in deep learning at a predetermined timing. Also, the batch number setting unit 112 adjusts the learning rate used in learning according to the ratio between the preset set batch number for a plurality of nodes and the number of executed batches executed before the predetermined timing.

[0106] FIG. 18 is a diagram illustrating a process of optimizing the number of batches when a specified process has completed a target number of batches, according to the second modified example.

[0107] In the example shown in FIG. 18, the batch numbers of processes #1 to #4 are each 100, the set global batch number is 400, and process #1 is specified and the target batch number is set to 100.

[0108] As indicated by symbol K1, a forced Allreduce process is executed when the specified process #1 reaches the target batch number of 100, even if the other processes #2 to #4 have not completed processing. In this forced Allreduce process, process #1 has completed processing of 100 batches, processes #2 and #3 have each completed processing of 90 batches, and process #4 has completed processing of 80 batches, so the total number of completed batches is 360.

[0109] The updated learning rate η is calculated by the following formula (1) and the Update process is executed. Note that batch indicates the set global batch, and batch' indicates the total number of completed batches.

[0110]

number

[0111] In the forced Allreduce process indicated by the reference symbol K1, the updated learning rate η′=η×(360 / 400) is calculated.

[0112] Furthermore, as indicated by K2, when the target batch number of 100 is reached in the specified process #1, the Allreduce process is executed even if the processing in the other processes #2 to #4 has not been completed. At this time, process #1 has completed processing of 100 batches, process #2 has completed processing of 90 batches, process #3 has completed processing of 80 batches, and process #4 has completed processing of 70 batches, so the total number of completed batches is 340. In the forced Allreduce process indicated by K2, the updated learning rate η''= η × (340 / 400) is calculated.

[0113] When two or more processes are specified, the completed processes can be stopped, or the learning number can be increased beyond the target batch number by continuing the learning process.

[0114] In other words, the predetermined timing for terminating learning may be the timing when the number of batches executed by a first node among the multiple nodes reaches a predetermined number after learning has started.

[0115] FIG. 19 is a diagram illustrating a process of optimizing the number of batches when a specified time has elapsed according to the second modified example.

[0116] In the example shown in FIG. 19, the number of batches in each of processes #1 to #4 is 100, the set global batch is 400, and an arbitrary designated time for executing the forced Allreduce process is set.

[0117] As indicated by reference symbol L1, even if the target batch number of 100 has not been reached in any of processes #1 to #4, a forced Allreduce process is executed when the specified time has elapsed. In this forced Allreduce process, processes #1 to #3 have each completed processing 90 batches, and process #4 has completed processing 80 batches, so the total number of completed batches is 350. The learning rate after the update is calculated as η' = η × (350 / 400), similar to the case shown in FIG.

[0118] As indicated by the symbol L2, a forced Allreduce process is executed when the second specified time has elapsed. In this forced Allreduce process, process #1 completes processing of 120 batches, which is more than the target batch number of 100, processes #2 and #3 complete processing of 100 batches, which is the same as the target batch number of 100, and process #4 completes processing of 90 batches, so the total number of completed batches is 410. In addition, the learning rate after the update is calculated as η''= η×(410 / 400), as in the case shown in FIG. 18.

[0119] In the forced Allreduce process indicated by the reference symbol L2, the process #1 may stop processing when the target number of batches is reached, and a Wait time may be generated.

[0120] In other words, the predetermined timing for terminating learning may be a predetermined time that has elapsed since the start of learning.

[0121] FIG. 20 is a diagram illustrating the optimization process of the batch number in the case where all processes have completed the target batch number according to the second modified example.

[0122] In the example shown in FIG. 20, the number of batches in each of processes #1 to #4 is 100, and the set global batch is 400.

[0123] As indicated by symbol M1, a forced Allreduce process is executed when the target batch number of 100 is reached in all processes #1 to #4 and a processing completion flag is output in all processes. In this forced Allreduce process, process #1 completes processing of batch number 120, processes #2 and #3 complete processing of batch number 110, and process #4 completes processing of batch number 100, so the total number of completed batches is 440. The updated learning rate is calculated as η'=η×(440 / 400), similar to the case shown in FIG.

[0124] As indicated by the symbol M2, a forced Allreduce process is executed when the target batch number of 100 is reached in all processes #1 to #4 and a processing completion flag is output in all processes. In this forced Allreduce process, process #1 completes processing of batch number 130, process #2 completes processing of batch number 120, process #3 completes processing of batch number 100, and process #4 completes processing of batch number 110, so the total number of completed batches is 460. In addition, the learning rate after the update is calculated as η''= η×(460 / 400), similar to the case shown in FIG. 18.

[0125] In other words, the predetermined timing for terminating learning may be the timing when the number of batches executed by all of the multiple nodes reaches a predetermined number after learning has started.

[0126] The process of optimizing the number of batches as the second modified example will be described with reference to the flowchart (steps S21 to S26) shown in FIG.

[0127] The batch processing unit 113 performs initial settings such as the number of learning processes or the target accuracy (step S21).

[0128] The batch number setting unit 112 sets a global batch number and a start condition for the Allreduce process (step S22). The start condition for the Allreduce process may be, for example, the timing when a specified process completes a target batch number, the timing when a specified time has elapsed, or the timing when all processes complete a target batch number.

[0129] The batch processing unit 113 performs learning in each process (step S23).

[0130] The batch processing unit 113 determines whether the start condition for the Allreduce process is satisfied (step S24).

[0131] If the start condition for the Allreduce process is not satisfied (see the NO route from step S24), the process returns to step S23.

[0132] On the other hand, if the start condition for the Allreduce process is met (see the YES route in step S24), the batch number setting unit 112 stops learning of all processes, updates the learning rate η according to the completion size, and executes the Allreduce process and the Update process (step S25).

[0133] The batch number setting unit 112 determines whether learning has been performed a specified number of times or whether a target accuracy has been reached (step S26).

[0134] If the learning has not been completed the specified number of times and the target accuracy has not been reached (see NO route from step S26), the process returns to step S23.

[0135] On the other hand, when the learning has been completed the specified number of times or the target accuracy has been reached (see the YES route in step S26), the optimization process of the batch number as the second modified example ends.

[0136] [E] Third Modification In the second modified example, the order-changed neural network processing is performed to constantly change the learning rate, but in reality, the execution speed may not change in such a short period of time. Therefore, in the third modified example, learning may be performed using the order-changed neural network processing once every predetermined number of iterations or every predetermined period, and thereafter, processing may be performed using normal order neural network processing. In other words, the order-changed neural network processing may be used only when determining whether to optimize the batch number, and the batch number and the learning rate may be fixed and normal order neural network processing may be used for the predetermined number of iterations or for the predetermined period.

[0137] FIG. 22 is a diagram illustrating the optimization process for the number of batches according to the third modified example.

[0138] In the example shown in FIG. 22, the number of batches in each of processes #1 to #4 is 100, and the set global batch is 400.

[0139] As indicated by symbol N1, the optimization process for the batch number is performed, and process #1 completes processing of 100 batches, processes #2 and #3 each complete processing of 90 batches, and process #4 completes processing of 80 batches, so the total number of completed batches is 360. The learning rate after the update is calculated as η' = η × (360 / 400), similar to the case shown in FIG.

[0140] As indicated by reference symbol N2, the process is repeated for a while using the total batch number 360 and the learning rate η′ calculated in reference symbol N1.

[0141] Then, after a predetermined number of iterations or a predetermined period of time has elapsed, the batch number optimization process is executed again. As indicated by symbol N3, process #1 has completed processing of 100 batches, and processes #2 to #4 have each completed processing of 80 batches, so the total number of completed batches is 340. The updated learning rate is calculated as η''=η×(340 / 400), similar to the case shown in FIG. 18.

[0142] As indicated by reference symbol N4, the process is repeated for a while using the total batch number 340 and the learning rate η″ calculated in reference symbol N3.

[0143] That is, the batch number setting unit 112 may terminate the learning after a predetermined number of iterations of the learning.The batch number setting unit 112 may terminate the learning after a predetermined time.

[0144] The process of optimizing the number of batches as the third modified example will be described with reference to the flowchart (steps S31 to S37) shown in FIG.

[0145] The batch processing unit 113 performs initial settings such as the number of learning processes or the target accuracy (step S31).

[0146] The batch number setting unit 112 sets a global batch number, a start condition for the Allreduce process, and a prescribed learning number (step S32). The prescribed learning number is the number of times learning is performed using the batch number and the learning rate after these values ​​are updated, and may be determined according to a predetermined number of iterations or a predetermined period.

[0147] The batch processing unit 113 performs learning in each process (step S33).

[0148] The batch processing unit 113 determines whether the start condition for the Allreduce process is satisfied (step S34).

[0149] If the start condition for the Allreduce process is not satisfied (see the NO route from step S34), the process returns to step S33.

[0150] On the other hand, if the start condition for the Allreduce process is met (see the YES route in step S34), the batch number setting unit 112 stops learning of all processes, updates the learning rate η according to the completion size, and executes the Allreduce process and the Update process (step S35).

[0151] The batch processing unit 113 performs learning a prescribed number of times with the updated batch number and learning rate (step S36).

[0152] The batch number setting unit 112 determines whether learning has been performed a specified number of times or whether a target accuracy has been reached (step S37).

[0153] If the learning has not been completed the specified number of times and the target accuracy has not been reached (see the NO route from step S37), the process returns to step S33.

[0154] On the other hand, when the learning has been completed the specified number of times or the target accuracy has been reached (see the YES route in step S37), the optimization process of the batch number as the third modified example ends.

[0155] [F] Effect FIG. 24 is a diagram illustrating the effect of the batch number optimization process according to the example of the embodiment and each of the modified examples.

[0156] When learning using multiple processes, by adding preconditions for the Allreduce process and setting the learning rate accordingly, the communication waiting time can be reduced, and as a result, the overall learning time can be reduced without sacrificing accuracy.

[0157] 24, as shown in the example of FIG, 100 batch numbers are assigned to each process, and there is a wait time in processes #1 to #3, but there is no wait time in process #4, as shown in the code P2. Therefore, for example, by decreasing the batch numbers in processes #2 to #4 and updating the learning rate, the processing time of the entire process can be reduced.

[0158] According to the program, the calculator 1, and the learning method in the example of the embodiment and each of the modifications described above, for example, the following advantageous effects can be achieved.

[0159] In learning by a plurality of nodes in deep learning, the batch number setting unit 112 assigns a batch number according to the performance of each of the plurality of nodes to each of the plurality of nodes, or determines to terminate the learning at a predetermined timing. In addition, the batch number setting unit 112 adjusts a learning rate used in learning according to a ratio between a preset set batch number for the plurality of nodes and an execution batch number executed by the allocation or executed before the predetermined timing in the plurality of nodes.

[0160] This reduces the waiting time until the sharing of processing results between nodes during parallel calculations begins. In other words, by completing processing on all nodes at the same time, the wait time until the Allreduce process (in other words, the process of sharing results between nodes during parallel calculations) begins can be reduced.

[0161] The batch number setting unit 112 measures performance and allocates the number of batches every predetermined number of iterations of learning, or terminates learning. The batch number setting unit 112 measures performance and allocates the number of batches every predetermined time, or terminates learning. This makes it possible to accurately measure performance and allocate batches even if the temperature of the CPU 11 or the usage trend of the computer 1 changes over time.

[0162] The specified timing for terminating learning is when the number of batches executed by a first node among the multiple nodes reaches a specified number after learning starts. The specified timing for terminating learning is when a specified time has elapsed after learning starts. The specified timing for terminating learning is when the number of batches executed by all of the multiple nodes reaches a specified number after learning starts. This makes it possible to reduce the wait time of each process and speed up learning.

[0163] [G] Other The disclosed technology is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present embodiment. Each configuration and each process of the present embodiment can be selected as necessary, or can be combined as appropriate.

[0164] In the above-described example of the embodiment and the first modified example, the batch number is allocated according to the measurement result of the throughput, but the present invention is not limited to this. For example, the clocks of the CPU 11 and the GPU may be monitored, and the batch number may be allocated according to the monitoring result of the clocks.

[0165] [H] Notes The following supplementary notes are further disclosed regarding the above embodiment.

[0166] (Appendix 1) In learning by a plurality of nodes in deep learning, a batch number according to the performance of each of the plurality of nodes is assigned to each of the plurality of nodes, or the learning is terminated at a predetermined timing; adjusting a learning rate used for the learning in accordance with a ratio between a preset set batch number for the plurality of nodes and a number of execution batches executed in the plurality of nodes by the allocation or executed before the predetermined timing; A program that causes a computer to carry out processing.

[0167] (Appendix 2) performing said performance measurement and said assignment or censoring said training after each predetermined number of iterations of said training; 2. The program of claim 1, which causes the computer to execute a process.

[0168] (Appendix 3) performing the performance measurement and the assignment or terminating the learning at predetermined time intervals; 2. The program of claim 1, which causes the computer to execute a process.

[0169] (Appendix 4) the predetermined timing is a timing when the number of batches executed by a first node among the plurality of nodes reaches a predetermined number after the start of the learning. The program according to any one of appendix 1 to 3.

[0170] (Appendix 5) The predetermined timing is a timing when a predetermined time has elapsed since the start of the learning. The program according to any one of appendix 1 to 3.

[0171] (Appendix 6) the predetermined timing is a timing when the number of batches executed by all of the plurality of nodes reaches a predetermined number after the start of the learning. The program according to any one of appendix 1 to 3.

[0172] (Appendix 7) In learning by a plurality of nodes in deep learning, a batch number according to the performance of each of the plurality of nodes is assigned to each of the plurality of nodes, or the learning is terminated at a predetermined timing; adjusting a learning rate used for the learning in accordance with a ratio between a preset set batch number for the plurality of nodes and a number of execution batches executed in the plurality of nodes by the allocation or executed before the predetermined timing; A computer having a processor.

[0173] (Appendix 8) the processor measures the performance and assigns the performance after a predetermined number of iterations of the learning, or terminates the learning. 7. A computer as defined in claim 7.

[0174] (Appendix 9) The processor measures the performance and assigns the performance, or terminates the learning, at each predetermined time. 7. A computer as defined in claim 7.

[0175] (Appendix 10) the predetermined timing is a timing when the number of batches executed by a first node among the plurality of nodes reaches a predetermined number after the start of the learning. 10. The computer according to any one of appendix 7 to 9.

[0176] (Appendix 11) The predetermined timing is a timing when a predetermined time has elapsed since the start of the learning. 10. The computer according to any one of appendix 7 to 9.

[0177] (Appendix 12) the predetermined timing is a timing when the number of batches executed by all of the plurality of nodes reaches a predetermined number after the start of the learning. 10. The computer according to any one of appendix 7 to 9.

[0178] (Appendix 13) In learning by a plurality of nodes in deep learning, a batch number according to the performance of each of the plurality of nodes is assigned to each of the plurality of nodes, or the learning is terminated at a predetermined timing; adjusting a learning rate used for the learning in accordance with a ratio between a preset set batch number for the plurality of nodes and a number of execution batches executed in the plurality of nodes by the allocation or executed before the predetermined timing; A learning method in which processing is carried out by a computer.

[0179] (Appendix 14) the processor measures the performance and assigns the performance after a predetermined number of iterations of the learning, or terminates the learning. 14. The learning method of claim 13, wherein the processing is performed by a computer.

[0180] (Appendix 15) The processor measures the performance and assigns the performance, or terminates the learning, at each predetermined time. 14. The learning method of claim 13, wherein the processing is performed by a computer.

[0181] (Appendix 16) the predetermined timing is a timing when the number of batches executed by a first node among the plurality of nodes reaches a predetermined number after the start of the learning. 16. A learning method according to any one of appendix 13 to 15.

[0182] (Appendix 17) The predetermined timing is a timing when a predetermined time has elapsed since the start of the learning. 16. A learning method according to any one of appendix 13 to 15.

[0183] (Appendix 18) the predetermined timing is a timing when the number of batches executed by all of the plurality of nodes reaches a predetermined number after the start of the learning. 16. A learning method according to any one of appendix 13 to 15. [Explanation of symbols]

[0184] 1: Calculator 11: CPU 111: Measuring section 112: Batch number setting section 113: Batch processing unit 12: Memory section 13: Display control section 130:Display device 14:Storage device 15: Input IF 151: Mouse 152: Keyboard 16: External recording medium processing unit 160: Recording media 17: Communication Interface 6 : Node 61: Storage

Claims

1. In deep learning by a plurality of nodes, determining to terminate the learning at a predetermined timing; adjusting a learning rate used in the learning in accordance with a ratio between a preset set batch number for the plurality of nodes and a number of execution batches executed in the plurality of nodes before the predetermined timing; A program that causes a computer to carry out processing.

2. truncating the learning every predetermined number of iterations of the learning; The program according to claim 1 , which causes the computer to execute a process.

3. The learning is terminated at predetermined time intervals. The program according to claim 1 , which causes the computer to execute a process.

4. the predetermined timing is a timing when a number of batches executed by a first node among the plurality of nodes reaches a predetermined number after the start of the learning. The program according to any one of claims 1 to 3.

5. The predetermined timing is a timing when a predetermined time has elapsed since the start of the learning. The program according to any one of claims 1 to 3.

6. the predetermined timing is a timing when the number of batches executed by all of the plurality of nodes reaches a predetermined number after the start of the learning. The program according to any one of claims 1 to 3.

7. In deep learning by a plurality of nodes, determining to terminate the learning at a predetermined timing; adjusting a learning rate used in the learning in accordance with a ratio between a preset set batch number for the plurality of nodes and a number of execution batches executed in the plurality of nodes before the predetermined timing; A computer having a processor.

8. In deep learning by a plurality of nodes, determining to terminate the learning at a predetermined timing; adjusting a learning rate used in the learning in accordance with a ratio between a preset set batch number for the plurality of nodes and a number of execution batches executed in the plurality of nodes before the predetermined timing; A learning method in which processing is carried out by a computer.

Images