Information processing program, information processing device, and information processing method

Abstract

To enable delamination of an optimal parallel number for realizing distributed learning.SOLUTION: An information processing program causes a processor to: derive wait time till resources to be used for distributed learning are secured and execution time required for the distributed learning for each execution environment with a different number of nodes; derive a score for each execution environment based on the wait time and execution time derived for each execution environment; and determine the number of nodes to be used for the distributed learning based on a plurality of scores.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing program, an information processing device, and an information processing method. [Background technology]

[0002] In recent years, MNNs (Multiple Neural Networks) that have multiple NNs (Neural Networks) have become known. For example, HDNNPs (High-Dimensional Neural Network Potentials) are known that use MNNs to calculate the potential energy of an entire particle system.

[0003] In addition, as a training method for such MNNs, distributed learning is sometimes used, in which machine learning (training) is performed in parallel on multiple NNs that make up the MNN. In distributed learning of MNNs, a NN is assigned to each of multiple workers, and the calculation results (output) of each worker are distributed to each other by Allreduce communication. After that, the sum of the outputs calculated by each worker is calculated, and the average gradient, etc. are calculated.

[0004] A worker is a computing device such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) provided in a computation node (hereinafter simply referred to as a node). If the worker is a GPU, 1 worker may be 1 GPU. If the worker is a CPU, 1 worker may be 1 CPU, or 1 worker may be 1 core, and can be implemented with appropriate modifications. In the following, in training an MNN, a node that executes the processing related to the training may be referred to as an execution node. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2013-140490 A [Patent Document 2] JP 2015-118434 A [Patent Document 3] US Patent Application Publication No. 2015 / 0169380 [Patent Document 4] US Patent Application Publication No. 2014 / 0075446 [Patent Document 5] US Patent Application Publication No. 2020 / 0236060 Summary of the Invention [Problem to be solved by the invention]

[0006] In distributed learning, by increasing the number of parallel processes, i.e. the number of workers used, the calculation execution time in each worker can be reduced. However, in distributed learning, communication between workers occurs due to Allreduce communication. Generally, communication is a costly process, and the cost of communication between execution nodes is particularly high. Note that communication between execution nodes can be called cross-execution-node communication.

[0007] In distributed learning, it is desirable to allocate workers in a way that minimizes cross-node execution.

[0008] Furthermore, for example, in the case of a cloud computing system or a supercomputer, where the usage fee varies depending on the number of execution nodes, the usage fee increases as the number of execution nodes increases.

[0009] Furthermore, in a multi-tenant environment where a system is shared by multiple users, as the number of execution nodes increases, the time required to secure resources increases, and the waiting time until a job is executed increases.

[0010] Therefore, when performing distributed learning, it is necessary to determine the optimal number of parallel processes.

[0011] In one aspect, an object of the present invention is to make it possible to determine an optimal number of parallel processes for achieving distributed learning. [Means for solving the problem]

[0012] For this reason, this information processing program causes a processor of an information processing device that manages distributed learning using multiple nodes to calculate the waiting time until resources to be used for the distributed learning are secured and the execution time required for the distributed learning for each execution environment with a different number of nodes, calculate a score for each execution environment based on the waiting time and execution time calculated for each execution environment, and execute a process to determine the number of nodes to be used for the distributed learning based on the multiple scores. Effect of the Invention

[0013] According to one embodiment, an optimal number of parallel processes for achieving distributed learning can be determined. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an information processing system according to an embodiment. [Diagram 2] FIG. 1 shows an example of the numbers of amino acid residues constituting the protein TrpCage. [Diagram 3] 1 is a block diagram showing an example of the hardware (HW) configuration of a computer that realizes the functions of a system management device of an information processing system according to an embodiment. [Figure 4] FIG. 11 is a diagram showing an example of a pattern of allocating residues to a group of execution nodes in an information processing system according to an embodiment. [Diagram 5] 11 is a diagram for explaining predictive calculation information in an information processing system according to an embodiment. FIG. [Figure 6] An example of executing a performance prediction job using backfilling is given below. [Figure 7] 1 is a diagram for explaining a method for determining a maximum calculation time in an information processing system according to an embodiment; [Figure 8] 11A and 11B are diagrams illustrating an example of execution of a queue status display command in an information processing system according to an embodiment. [Figure 9] 11 is a diagram illustrating an example of reference information referred to by a parallel number determination unit of an information processing system according to an embodiment. FIG. [Figure 10] 10 is a flowchart illustrating a process of a system management device of an information processing system according to an embodiment. [Figure 11] 11 is a flowchart for explaining details of the process of step S2 in the flowchart shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present information processing program, information processing device, and information processing method will be described with reference to the drawings. However, the embodiments shown below are merely examples, and there is no intention to exclude the application of various modified examples and techniques not explicitly stated in the embodiments. In other words, this embodiment can be implemented with various modifications within the scope of its purpose. Furthermore, each figure is not intended to include only the components shown in the figure, but may include other functions, etc.

[0016] (A) Configuration FIG. 1 is a diagram illustrating a schematic configuration of an information processing system 1 according to an embodiment.

[0017] The information processing system 1 realizes data-parallel distributed learning using multiple execution nodes 5. The information processing system 1 illustrated in Fig. 1 includes a distributed learning unit 7 and multiple client terminals 3. The distributed learning unit 7 and the multiple client terminals 3 are connected to each other via a network 2 so as to be able to communicate with each other.

[0018] The information processing system 1 corresponds to a multi-tenant environment shared by a plurality of users, and includes a plurality of client terminals 3.

[0019] The client terminal 3 is a terminal device used by a user of the information processing system 1, and, for example, a job is input via this client terminal 3. In addition, training data (learning data set) used for distributed learning may be input via this client terminal 3.

[0020] The network 2 may be, for example, a LAN (Local Area Network) such as Ethernet, or optical communication such as FC (Fibre Channel).

[0021] The distributed learning unit 7 includes a system management device 6 and an execution node group 4 .

[0022] The execution node group 4 has a plurality of execution nodes 5. The execution node 5 is an information processing device (computer) having one or more processors, memories, storage devices, etc. (not shown), and realizes the function of a training execution unit 51 by executing an OS (Operating System) and programs stored in the storage device. The processor provided in the execution node 5 is an example of a worker.

[0023] In this embodiment, an example is shown in which there is no or almost no difference in performance among the execution nodes 5 included in the execution node group 4. However, this is not limited to this, and there may be a performance difference among the execution nodes 5, and the embodiment can be modified as appropriate.

[0024] The training execution unit 51 trains the NN. The NNs of the multiple execution nodes 5 included in the execution node group 4 constitute an MNN.

[0025] In this information processing system 1, we show an example of applying HDNNP to the protein TrpCage, which is coarse-grained in units of amino acid residues. That is, the residues that make up the protein are treated as a particle system, and the potential energy of the entire particle system is calculated using a machine learning model. TrpCage is a protein consisting of 20 amino acid residues.

[0026] FIG. 2 is a diagram illustrating the numbers of amino acid residues constituting the protein TrpCage.

[0027] FIG. 2 shows 11 types of combinations of 20 residues that constitute TrpCage.

[0028] In distributed learning using a plurality of execution nodes 5 in the execution node group 4, a NN is prepared for each residue, and each NN corresponds to one of the 11 types of residues in TrpCage.

[0029] Hereinafter, NN may be specified by adding a subscript representing the name of a protein residue to NN. For example, the NN corresponding to proline (PRO) is NN PRO and NN corresponding to tyrosine (TYR) is NN TYR This is expressed as:

[0030] In the execution node 5, a training execution unit 51 can execute training of a NN corresponding to any of the 11 types of residues in TrpCage.

[0031] The training execution unit 51, for example, inputs input data to the input layer of the NM, and sequentially executes predetermined calculations in hidden layers composed of a convolution layer, a pooling layer, etc., thereby executing forward direction processing (forward propagation processing) that sequentially transmits information obtained by the calculation from the input side to the output side. After executing the forward direction processing, it executes backward direction processing (backpropagation processing) that determines parameters to be used in the forward direction processing in order to reduce the value of the error function obtained from the output data output from the output layer and the correct answer data.

[0032] Then, an update process is executed to update variables such as weights based on the result of the backpropagation process. For example, a gradient descent method may be used as an algorithm for determining the update width of the weights used in the calculation of the backpropagation process.

[0033] 3 is a block diagram showing an example of a hardware (HW) configuration of a computer 10 that realizes the functions of the system management device 6 of the information processing system 1 according to an embodiment. When multiple computers are used as HW resources that realize the functions of the system management device 6, each computer may have the HW configuration shown in FIG.

[0034] As shown in FIG. 3, the computer 10 may, as a HW configuration, illustratively include a processor 10a, a graphics processing unit 10b, a memory 10c, a storage unit 10d, an IF (Interface) unit 10e, an IO (Input / Output) unit 10f, and a reading unit 10g.

[0035] The processor 10a is an example of a processing unit that performs various controls and calculations, and is an example of a control unit. The processor 10a may be connected to each block in the computer 10 via a bus 10j so that they can communicate with each other. The processor 10a may be a multiprocessor including multiple processors, a multicore processor having multiple processor cores, or a configuration having multiple multicore processors.

[0036] Examples of the processor 10a include integrated circuits (ICs) such as a CPU, MPU, APU, DSP, ASIC, and FPGA. Note that a combination of two or more of these integrated circuits may be used as the processor 10a. CPU is an abbreviation for Central Processing Unit, MPU is an abbreviation for Micro Processing Unit, APU is an abbreviation for Accelerated Processing Unit, DSP is an abbreviation for Digital Signal Processor, ASIC is an abbreviation for Application Specific IC, and FPGA is an abbreviation for Field-Programmable Gate Array.

[0037] The graphics processing device 10b performs screen display control for an output device such as a monitor in the IO unit 10f. The graphics processing device 10b may be any of a variety of arithmetic processing devices, such as a GPU (Graphics Processing Unit), an APU, a DSP, an ASIC, an FPGA, or other integrated circuits (ICs).

[0038] The memory 10c is an example of HW that stores various data, programs, and other information. The memory 10c may be, for example, one or both of a volatile memory such as a dynamic random access memory (DRAM) and a non-volatile memory such as a persistent memory (PM).

[0039] The storage unit 10d is an example of HW that stores various data, programs, and other information. Examples of the storage unit 10d include various storage devices such as a magnetic disk device such as a hard disk drive (HDD), a semiconductor drive device such as a solid state drive (SSD), and a nonvolatile memory. Examples of the nonvolatile memory include a flash memory, a storage class memory (SCM), and a read only memory (ROM).

[0040] The storage unit 10d may store a program 10h (information processing program) that realizes all or part of the various functions of the computer 10.

[0041] For example, the processor 10a of the system management device 6 (computer 10) can implement a distributed learning control function, which will be described later, by loading a program 10h stored in the storage unit 10d into the memory 10c and executing it.

[0042] The IF unit 10e is an example of a communication IF that controls the connection and communication between the computer 10 and other computers. For example, the IF unit 10e may include an adapter that complies with a LAN such as Ethernet (registered trademark) or optical communication such as FC. The adapter may support one or both of wireless and wired communication methods.

[0043] For example, the system management device 6 may be connected to each of the client terminals 3 and the execution node group 4 illustrated in Fig. 1 so as to be able to communicate with each other via the IF unit 10e and the network 2. The program 10h may be downloaded to the computer 10 from the network 2 via the communication IF and stored in the storage unit 10d.

[0044] The IO unit 10f may include one or both of an input device and an output device. Examples of the input device include a keyboard, a mouse, a touch panel, etc. Examples of the output device include a monitor, a projector, a printer, etc. The IO unit 10f may also include a touch panel in which the input device and the display device are integrated. The output device may be connected to the graphics processing device 10b.

[0045] The reading unit 10g is an example of a reader that reads out data or program information recorded on the recording medium 10i. The reading unit 10g may include a connection terminal or device to which the recording medium 10i can be connected or inserted. Examples of the reading unit 10g include an adapter that complies with USB (Universal Serial Bus) or the like, a drive device that accesses a recording disk, and a card reader that accesses a flash memory such as an SD card. The recording medium 10i may store a program 10h, and the reading unit 10g may read the program 10h from the recording medium 10i and store it in the storage unit 10d.

[0046] Examples of the recording medium 10i include non-transitory computer-readable recording media such as magnetic / optical disks and flash memories. Examples of the magnetic / optical disks include flexible disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray Discs, and HVDs (Holographic Versatile Discs). Examples of the flash memory include semiconductor memories such as USB memories and SD cards.

[0047] The above-described HW configuration of the computer 10 is an example. Therefore, the HW in the computer 10 may be increased or decreased (for example, adding or deleting any block), divided, or integrated in any combination, or buses may be added or deleted, etc. as appropriate.

[0048] (B) Example of functional configuration of system management device 6 As shown in FIG. 1, the system management device 6 has the functions of a job deployment unit 61, a job management unit 62, a prediction unit 63, a parallel number determination unit 64, and a queue 65.

[0049] The queue 65 stores jobs received from the client terminal 3 via the network 2. The jobs input to the queue 65 may be output in a FIFO (First In First Out) order, for example.

[0050] The job management unit 62 manages the execution status of jobs in each execution node 5. In this way, the job management unit 62 manages (monitors) the execution status of jobs in each execution node 5, thereby grasping jobs that are being executed and jobs that have been completed in each execution node 5. This allows the job management unit 62 to grasp the availability and waiting status of each execution node 5.

[0051] The prediction unit 63 predicts execution performance (parallel performance) in a plurality of execution environments in which the number of execution nodes 5 used (number of execution nodes) is different when distributed learning is performed using a plurality of execution nodes 5.

[0052] The prediction unit 63 generates a job for predicting execution performance and places it in a queue 65 . The job for performance prediction taken out from the queue 65 is assigned to an execution node 5 by the job deployment unit 61, which will be described later. Since the upper limit of the execution time of the job for performance prediction is set to a small value, it is expected that the waiting time until execution will be shortened by the backfilling in FIG.

[0053] The prediction unit 63 receives the results of the performance prediction job from the execution node group 4 and predicts the execution performance. The prediction unit 63 assigns residues constituting the protein as jobs to the execution nodes 5. Here, the number of execution nodes 5 used for job assignment may be referred to as the number of execution nodes.

[0054] The prediction unit 63 creates a plurality of allocation patterns while changing the number of execution nodes. A plurality of allocation patterns with different numbers of execution nodes is an example of an execution environment with different numbers of nodes.

[0055] FIG. 4 is a diagram showing an example of a pattern of allocating residues to the execution node group 4 in the information processing system 1 according to an embodiment.

[0056] In the example shown in FIG. 4, 20 residues of 11 types constituting the TrpCage shown in FIG. 2 are assigned in four allocation patterns while changing the number of execution nodes.

[0057] In Figure 4, symbol A indicates an allocation pattern when the number of execution nodes is 2 (two units), symbol B indicates an allocation pattern when the number of execution nodes is 4 (four units), symbol C indicates an allocation pattern when the number of execution nodes is 5 (five units), and symbol D indicates an allocation pattern when the number of execution nodes is 10 (ten units).

[0058] Each allocation pattern is associated with a specific number of execution nodes. A plurality of allocation patterns may be called a set of node numbers.

[0059] When the group of node numbers is represented as a set N, four allocation patterns are shown in the example shown in FIG. 4, and the set can be represented as N={2, 4, 5, 10}.

[0060] 4, each column indicates an execution node 5, and a numerical value (for example, 0 to 9 in the example shown by symbol D) assigned to each column functions as identification information for identifying an execution node 5. Hereinafter, the identification information for identifying an execution node 5 may be referred to as an execution node ID. The execution node ID may be an integer.

[0061] Hereinafter, the execution node 5 may be identified using the symbol # and the execution node ID. For example, the execution node 5 with the execution node ID of 0 may be represented as execution node #0.

[0062] For example, in the allocation pattern shown by symbol C in FIG. 4, four prolines (PRO0 to PRO3) are allocated to the execution node #0. That is, this execution node #0 is an NN PRO This indicates that training is carried out as a job.

[0063] When allocating jobs (residues) to execution nodes 5 of each allocation pattern, the prediction unit 63 may determine the allocation destinations by solving a bin packing problem. At this time, the number of execution nodes is set to the number of bins.

[0064] As a first bin packing strategy, the prediction unit 63 allocates residues so that the number of residues is equal among the execution nodes 5 from the viewpoint of load balancing.

[0065] Here, load equalization between the execution nodes 5 can be achieved by setting the number of execution nodes (number of pins) to a divisor (for example, 2, 4, 5, or 10) of the total number of residues (20 in this embodiment).

[0066] As a second strategy of bin packing, the prediction unit 63 assigns jobs so that the same residue type is contained within the same execution node 5 as much as possible in order to minimize communication across execution nodes 5. This makes it possible to reduce the occurrence of communication across execution nodes 5 due to Allreduce communication.

[0067] Furthermore, when determining an allocation pattern for each number of execution nodes, the prediction unit 63 may allocate residues to a plurality of execution nodes 5 based on a WFD (Worst-Fit Decreasing) algorithm.

[0068] When assigning residues to execution nodes 5 in the WFD algorithm, residue types with a large number of residues are assigned first. In addition, among those with full capacity, the job is deployed to the execution node 5 with the largest amount of free space. This makes it possible to reduce the number of times that residue types with a large number of residues cross execution nodes 5.

[0069] In addition, the prediction unit 63 predicts the calculation time (predicted calculation time) according to the number of residues based on the measurement result of the cumulative calculation time required for forward propagation per epoch, which is obtained as a result of actually training the execution node 5 with 11 types of residues of TrpCage.

[0070] FIG. 5 is a diagram for explaining the predicted calculation time information 11 in the information processing system 1 according to an embodiment.

[0071] In FIG. 5, reference symbol A exemplifies the actually measured cumulative calculation time information 13, and reference symbol B exemplifies the predicted calculation time information 11.

[0072] The actually measured accumulated calculation time information 13 indicates the number of residues and the accumulated calculation time for each of the multiple residues constituting TrpCage.

[0073] Here, the cumulative calculation time is a measurement result (measured value) required for forward propagation per epoch, obtained as a result of experimentally training one execution node 5 included in the execution node group 4 for each of the 11 types of residues that make up TrpCage as a performance prediction job. The cumulative calculation time may be called the actually measured cumulative calculation time. The cumulative calculation time is an example of a measurement result obtained by having one execution node 5 out of multiple execution nodes 5 execute processing related to distributed learning.

[0074] For example, the cumulative calculation time for tyrosine (TYR), which has one residue, is 0.679 s, while the cumulative calculation time for proline (PRO), which has four residues, is 1.51 s. This shows that the calculation time for forward propagation increases in proportion to the number of input data (number of residues).

[0075] Here, NN training (machine learning) has the characteristic of repeatedly executing processing until a termination condition (for example, a predetermined number of epochs is reached, the loss is below a threshold, etc.) is met, so performance prediction with high accuracy is possible even with only a few iterations. Therefore, performance prediction with high accuracy can be performed using the measurement result of the accumulated calculation time obtained by executing only one epoch on the execution node 5 as a performance prediction job.

[0076] In order to minimize the calculation time required for the performance prediction by the prediction unit 63, backfilling may be utilized to shorten the waiting time for execution of a job related to the performance prediction.

[0077] FIG. 6 shows an example of executing a performance prediction job using backfill.

[0078] In FIG. 6, symbol A indicates the state before backfilling is applied, and symbol B indicates the state after backfilling is applied.

[0079] In the process of reserving computing resources (workers) in order of priority, if it is possible to execute a performance prediction job (Job C) using free resources that are not being used by any job, then this performance prediction job (Job C) is executed before the high-priority job (Job B).

[0080] The prediction unit 63 may minimize the calculation time required for performance prediction by utilizing a technique such as backfilling.

[0081] The prediction unit 63 creates the actually measured accumulated calculation time information 13 using the accumulated calculation time measured in this manner.

[0082] Furthermore, the prediction unit 63 predicts the predicted calculation time for each number of residues based on the actually measured cumulative calculation time in the actually measured cumulative calculation time information 13. In the example shown in Fig. 5, the prediction unit 63 calculates (predicts) the predicted calculation time for each of the numbers of residues 1 to 4.

[0083] In addition, when the same number of residues is present in multiple residue types (residue names), such as the example shown by symbol A in Figure 5 where the number of residues is 2 or 4, the prediction unit 63 may calculate the average cumulative calculation time for these numbers of residues as the predicted calculation time.

[0084] 5, the two residue names ALA and ASP have a residue number of 2. The prediction unit 63 calculates (predicts) the average value of the cumulative calculation time of 1.05 s for the residue name ALA and the cumulative calculation time of 1.00 s for the residue name ASP, 1.03 s, as the predicted calculation time for the residue number of 2.

[0085] The prediction unit 63 associates the calculated predicted calculation time with the number of residues to generate predicted calculation time information 11. The predicted calculation time information 11 may be stored in, for example, a predetermined storage area of ​​the memory 10c or the storage unit 10d.

[0086] Furthermore, for each of the created allocation patterns (see FIG. 4), the prediction unit 63 calculates (predicts) the calculation time (predicted cumulative calculation time) for each execution node 5 used in the allocation pattern, and identifies, for each allocation pattern, the execution node 5 with the longest predicted cumulative calculation time. Among the execution nodes 5 included in an allocation pattern, the longest predicted cumulative calculation time may be called the maximum calculation time.

[0087] FIG. 7 is a diagram for explaining a method for determining the maximum calculation time in the information processing system 1 according to an embodiment.

[0088] In Fig. 7, symbol A indicates an example of predicted cumulative calculation time information 12 indicating the predicted cumulative calculation time of each execution node 5 in the allocation pattern in which the number of execution nodes is 10 as shown in Fig. 4. The predicted cumulative calculation time information 12 illustrated in Fig. 7 is configured by associating the assigned residues and the predicted cumulative calculation time with the execution node.

[0089] The execution node is a value that identifies the execution node 5, and in the example shown in FIG. 7, the execution node 5 is identified using the execution node ID.

[0090] The assigned residues are residues (assigned residues) assigned to each execution node 5. The predicted cumulative calculation time is the time (cumulative time) required for processing (training the NN) per epoch for the assigned residues in each execution node 5.

[0091] Among the assigned residues shown in the predicted cumulative computation time information 12, residues of the same type that are distributed and assigned to multiple execution nodes 5 are underlined. That is, the underlined residues in the predicted cumulative computation time information 12 shown in Fig. 7 are node-spanning residues that are assigned to span execution nodes 5. In Fig. 7, the predicted cumulative computation times corresponding to the node-spanning residues are also underlined.

[0092] The prediction unit 63 creates predicted cumulative calculation time information 12 based on the predicted calculation time information 11 .

[0093] For cross-node residues, the prediction unit 63 extracts a predicted calculation time corresponding to the number of residues to which the same residue type is assigned from the predicted calculation time information 11 , and uses this to calculate the predicted cumulative calculation time in the predicted cumulative calculation time information 12 .

[0094] 7, two cross-node residues PRO0 and PRO1 are assigned to execution node #0. The prediction unit 63 obtains a predicted calculation time of 1.03 s corresponding to the number of residues 2 from the predicted calculation time information 11, and sets this as the predicted accumulated calculation time corresponding to execution node #0 in the predicted accumulated calculation time information 12.

[0095] Furthermore, for residues that are not node-spanning residues, the prediction unit 63 extracts the accumulated calculation time corresponding to the residue from the actually measured accumulated calculation time information 13 and uses it to calculate the predicted accumulated calculation time of the predicted accumulated calculation time information 12 .

[0096] 7, two residues, ALA0 and ALA1, are assigned to execution node #6. The prediction unit 63 obtains the actual accumulated calculation time of 1.05 s corresponding to the residue ALA from the actual accumulated calculation time information 13, and sets it as the predicted accumulated calculation time corresponding to execution node #6 in the predicted accumulated calculation time information 12.

[0097] 7, for example, two residues LEU and LYS are assigned to execution node #8. The prediction unit 63 acquires the measured accumulated calculation time of 0.680 s corresponding to the residue LEU and the measured accumulated calculation time of 0.687 s corresponding to the residue LYS from the measured accumulated calculation time information 13. Then, the prediction unit 63 sets the predicted accumulated calculation time corresponding to execution node #8 in the predicted accumulated calculation time information 12 to the sum of these measured accumulated calculation times, 1.37 s (= 0.680 + 0.687).

[0098] In addition, there are execution nodes 5 to which both node-spanning residues and non-node-spanning residues are assigned. For such execution nodes 5, the prediction unit 63 uses, as the predicted cumulative calculation time, the sum of the predicted calculation time corresponding to the number of residues to which the same residue type is assigned, extracted from the predicted calculation time information 11, and the cumulative calculation time corresponding to the relevant residue, extracted from the actual cumulative calculation time information 13.

[0099] 7, one node-crossing residue GLY2 and a non-node-crossing residue ARG are assigned to execution node #3. The prediction unit 63 obtains a predicted calculation time of 0.683 s corresponding to the residue number 1 from the predicted calculation time information 11, and obtains an actual accumulated calculation time of 0.687 s corresponding to the residue ARG from the actual accumulated calculation time information 13. The prediction unit 63 sets the sum of the predicted calculation time 0.683 s and the actual accumulated calculation time 0.687 s, 1.37 s, as the predicted accumulated calculation time corresponding to execution node #3 in the predicted accumulated calculation time information 12.

[0100] In this manner, the prediction unit 63 calculates the predicted cumulative calculation times for all execution nodes 5 in the predicted cumulative calculation time information 12.

[0101] Then, the prediction unit 63 creates predicted accumulated calculation time information 12 for each of the multiple types of allocation patterns.

[0102] The prediction unit 63 obtains the maximum value of the predicted cumulative calculation time in the created predicted cumulative calculation time information 12. In the predicted cumulative calculation time information 12 illustrated by reference symbol A in Fig. 7, the maximum value of the predicted cumulative calculation time is 1.37 seconds.

[0103] The prediction unit 63 obtains the maximum value of the predicted cumulative calculation time (maximum calculation time) from each piece of predicted cumulative calculation time information 12 for all the allocation patterns.

[0104] The prediction unit 63 creates the predicted performance improvement rate information 15 by using the maximum value of the predicted cumulative calculation time (maximum calculation time) of each of the acquired allocation patterns.

[0105] In FIG. 7, reference symbol B indicates an example of the predicted performance improvement rate information 15.

[0106] The predicted performance improvement rate information 15 illustrated in FIG. 7 associates the maximum calculation time and the predicted performance improvement rate with the number of execution nodes.

[0107] The number of execution nodes is the number of execution nodes 5, which is the number of execution nodes 5 used as the allocation destination of residues (jobs). In the example shown by reference character B in Fig. 7, the numbers of execution nodes are 1, 2, 4, 5, 10, and 20. Of these numbers of execution nodes, if the number (2, 4, 5, and 10 in the example shown in Fig. 4) is the same as the number of execution nodes 5 used in the allocation pattern (see Fig. 4), it can be said that the number of execution nodes specifies the allocation pattern.

[0108] The maximum calculation time is the maximum value of the predicted cumulative calculation time for each number of execution nodes (allocation pattern). The maximum calculation time can be said to be the maximum predicted value of the cumulative calculation time for each number of execution nodes (allocation pattern). In the example shown in Figure 7, the more the number of execution nodes, the smaller the value of the maximum calculation time, and the fewer the number of execution nodes, the larger the value of the maximum calculation time.

[0109] The predicted performance improvement rate is a value calculated by calculating the predicted performance improvement rate of the maximum calculation time corresponding to each number of execution nodes, based on the maximum calculation time when the number of execution nodes is 1. The predicted performance improvement rate may be calculated, for example, by dividing the maximum calculation time when the number of execution nodes is 1 by the maximum calculation time for each number of execution nodes.

[0110] This predicted performance improvement rate can be said to be a relative predicted performance improvement rate with respect to the maximum calculation time when the number of execution nodes is one.

[0111] The prediction unit 63 calculates a predicted performance improvement rate for each of a plurality of allocation patterns (execution environments) based on a measurement result (cumulative calculation time) obtained by causing one of the plurality of execution nodes 5 to execute processing related to distributed learning.

[0112] The prediction unit 63 calculates an execution time T l Calculate (predict) (n). Execution time T l (n) represents the performance (parallel performance) when performing distributed learning using n execution nodes 5.

[0113] For example, the prediction unit 63 divides the reference value of the execution time by the predicted performance improvement rate for each number of execution nodes (n) to obtain the execution time T l Calculate (predict) (n).

[0114] Here, the reference value of the execution time may be, for example, an upper limit value of the execution time described in the job, which is determined in advance by the user. Although it is difficult to predict the learning time of the NN (the execution time of the program) in advance, the execution time T l The upper limit of execution time may be set by the user, or may be specified in advance in the specifications, for example. Furthermore, the prediction unit 63 predicts the waiting time required to secure an execution node 5 to be used when distributed learning is performed using a plurality of execution nodes 5.

[0115] For example, the prediction unit 63 uses a machine learning model (prediction model W) for predicting waiting time to predict a waiting time T required to secure n execution nodes 5 as resources. W It is reasonable to predict.

[0116] The prediction model W may be a model that predicts a waiting time when the execution status and queue state of a job in the execution node 5 are input.

[0117] As input data to the prediction model W, for example, information obtained as a result of executing a known command that displays the execution status of a job or information (status) of the queue 65 may be used.

[0118] FIG. 8 is a diagram showing an example of execution of a queue status display command in the information processing system 1 according to an embodiment.

[0119] In FIG. 8, symbol A indicates an example of execution of a command (qstat -f) for displaying the resource status of each execution node 5.

[0120] The execution result of this command displays the resource status of each execution node 5. Here, resv indicates the amount of reserved resources, and used indicates the amount of resources in use. In the execution result of this command, an execution node 5 with resv / used of 0 / 0 indicates an available node. Therefore, the number of available nodes can be found by searching for execution nodes 5 with resv / used of 0 / 0. If there are enough available nodes for the resources you want to reserve, it is believed that no waiting time will occur.

[0121] Also, in FIG. 8, symbol B indicates an example of executing a command (qstat -qc) for displaying the resource status of the entire cluster.

[0122] The execution result of this command displays the resource status of the entire cluster. Here, AVAIL indicates the amount of available resources, and TOTAL indicates the total amount of resources. The smaller the AVAIL / TOTAL ratio, the more congested the cluster is. If there are few free resources, the wait time is likely to be long.

[0123] The machine learning model that predicts waiting time (prediction model W) may input the ratio of resv to used (resv / used ratio) or the ratio of AVAIL to TOTAL (AVAIL / TOTAL ratio) obtained from the result of the command that displays the resource status described above.

[0124] For example, the AVAIL / TOTAL ratio when the number of free execution nodes 5 is 0 and the actual measurement value of the waiting time are used as training data to calculate the waiting time T W A predictive model W may be trained (machine learning) to predict

[0125] That is, the prediction unit 63 uses the prediction model W to perform data-parallel distributed learning in the execution node group 4, and the waiting time T w Predict (n). T w (n) is the waiting time required to secure n execution nodes 5 as resources.

[0126] Based on the values ​​calculated by the prediction unit 63, the parallel number determination unit 64 determines the number of execution nodes 5 (number of execution nodes, number of parallel connections) to be used when performing distributed learning.

[0127] The parallel number determination unit 64 solves the formulated mathematical optimization problem based on the following equation (1) to find the optimal parallel number n.

[0128]

number

[0129] When minimizing the execution time is important, α>β is set. When minimizing the usage fee is important, α<β is set.

[0130] FIG. 9 is a diagram illustrating the reference information 14 referred to by the parallel number determination unit 64 of the information processing system 1 according to an embodiment.

[0131] The parallel number determination unit 64 determines the number of execution nodes by referring to the reference information 14 illustrated in FIG.

[0132] The reference information 14 illustrated in FIG. 9 is a predicted performance improvement rate, T l (n),T w (n),{[T l (n)+T w (n)] / [T l (1)+T w (1)]}=a,n T l (n) / T l (1) = b and a+b correspond to each other.

[0133] In the reference information 14 illustrated in FIG. 9, the upper execution time limit is 6 hours, and α=β=1.

[0134] The number of nodes and the predicted performance improvement rate are the same as the number of execution nodes and the predicted performance improvement rate of the predicted performance improvement rate information 15 indicated by the symbol B in FIG.

[0135] T l (n) is a predicted value of the execution time for each number of execution nodes, calculated by the prediction unit 63 based on the predicted performance improvement rate of the predicted performance improvement rate information 15.

[0136] T w (n) is the waiting time predicted by the prediction unit 63 using the prediction model W, required to secure n execution nodes 5 as resources.

[0137] T l (1) is a predicted value of the execution time when the number of execution nodes is 1 (n=1). In the reference information 14 illustrated in FIG. l (1)=6h. T w (1) is the waiting time when the number of execution nodes is 1 (n=1), and in the reference information 14 illustrated in FIG. w (1)=0.

[0138] {[T l (n)+T w (n)] / [T l (1)+T w (1)]} is the above-mentioned Tl (n),T w (n),T l (1) and T w (1) is used to calculate the value. l (n)+T w (n)] / [T l (1)+T w (1)]}=a. This value of a is an example of the first score related to execution time.

[0139] n T l (n) / T l (1) is the above-mentioned T l (n),T w (n),T l (1) and T w (1) is used to calculate the value. Note that n T l (n) / T l Let (1)=b. This value of b is an example of the second score related to cost.

[0140] a+b is {[T l (n)+T w (n)] / [T l (1)+T w (1)]} and n T l (n) / T l This value of a+b is the sum of the calculated value of (1). w and execution time T l 13 is an example of a score for each execution environment calculated based on the above.

[0141] The parallel number determination unit 64 refers to this reference information 14 and determines the number of execution nodes n with the smallest value of a+b as the number of execution nodes 5 to be used in data-distributed parallel learning (distributed learning). In the example shown in Fig. 9, the parallel number determination unit 64 adopts the number of execution nodes 5 (n=5) with the smallest value of a+b, 1.52, as the number of execution nodes.

[0142] The parallel number determination unit 64 determines the number of execution nodes corresponding to the allocation pattern (execution environment) associated with the smallest score among the multiple scores (a+b) as the number of execution nodes to be used for distributed learning.

[0143] The parallel number determination unit 64 functions as a solver that finds the optimal parallel number n from the formulated mathematical optimization problem based on the above equation (1).

[0144] The job deployment unit 61 submits the distributed learning unit jobs to the execution nodes 5 having the parallel number (number of execution nodes) determined by the parallel number determination unit 64. That is, the job deployment unit 61 assigns the jobs to the execution nodes 5.

[0145] The job deployment unit 61 may, for example, preferentially allocate a job to an execution node 5 with a low load, and may determine an execution node 5 to which a job is allocated from among a plurality of execution nodes 5 using various known methods.

[0146] (B) Operation The processing of the system management device 6 of the information processing system 1 according to the embodiment configured as described above will be described with reference to the flowchart (steps S1 to S6) shown in FIG.

[0147] In step S1, the system management device 6 receives a request to execute distributed learning, which is input from the client terminal 3 or the like.

[0148] In step S2, the prediction unit 63 determines a destination to which the job is to be assigned. Details of this process will be described later with reference to FIG.

[0149] In step S3, the prediction unit 63 calculates the execution time (parallel performance) T l Predict (n).

[0150] In step S4, the prediction unit 63 calculates the waiting time T required to secure n execution nodes 5 as resources. W Predict (n).

[0151] In step S5, the parallel number determination unit 64 determines the number of execution nodes 5 (number of execution nodes, number of parallel numbers) to be used when performing distributed learning.

[0152] In step S6, the job deployment unit 61 submits a distributed learning job to the execution nodes 5 with the parallel number (number of execution nodes) determined by the parallel number determination unit 64. After that, when the job management unit 62 confirms that the execution of the submitted job is completed, the process ends.

[0153] Next, the details of the process of step S2 in the flowchart shown in FIG. 10 will be described with reference to the flowchart (steps S11 to S15) shown in FIG.

[0154] In step S11, the prediction unit 63 predicts a set of node numbers N={n1, n2, . . . , n |N|-1} (multiple allocation patterns).

[0155] In step S12, the prediction unit 63 sorts the residue types constituting the protein in descending order based on the number of residues.

[0156] In step S13, a loop process is started in which the process of step S14 is repeatedly executed for all elements (allocation patterns) i included in the set N of the node number group.

[0157] In step S14, the number of nodes n is determined based on the WFD algorithm. i Assign residues to

[0158] In step S15, a loop end process corresponding to step S13 is performed. When the processes for all allocation patterns included in the node number group are completed, this flow ends.

[0159] (C) Effects In this way, according to the information processing system 1 according to an embodiment, the prediction unit 63 determines the waiting time T w and the execution time T l is calculated for each allocation pattern with a different number of execution nodes.

[0160] Moreover, the prediction unit 63 calculates a score (a+b) for each allocation pattern based on the waiting time Tw and execution time Tl obtained for each of these allocation patterns.

[0161] Then, the parallel number determination unit 64 solves the formulated mathematical optimization problem based on the above formula (1) to find the optimal parallel number n. Specifically, the parallel number determination unit 64 determines the number of execution nodes corresponding to the allocation pattern with the minimum score (a+b) as the number of nodes to be used for distributed learning.

[0162] This makes it possible to set the optimal number of execution nodes (parallelism) in terms of both time and cost, and also minimize the time (including waiting time) from job submission to completion.

[0163] In addition, by appropriately changing the values ​​of α and β in the above formula (1), it is possible to determine the number of execution nodes taking into consideration the balance between cost and execution time.

[0164] The potential energy of a protein can be calculated by having a plurality of execution nodes 5 process NNs provided for each type of residue that constitutes the protein.

[0165] The prediction unit 63 calculates a predicted performance improvement rate for each of the multiple allocation patterns based on the cumulative calculation time obtained by having one of the multiple execution nodes 5 execute the process related to the distributed learning on a trial basis. Then, the execution time T l This allows us to calculate the execution time T l can be calculated.

[0166] (D) Other The disclosed technology is not limited to the above-described embodiment, and can be implemented in various modifications without departing from the spirit of the embodiment.

[0167] For example, in the above embodiment, the prediction unit 63 predicts the waiting time T W However, the present invention is not limited to this. For example, if the system has a function of predicting a waiting time and a function of presenting a predicted execution start time for an input job, the presented information on the predicted execution start time may be used.

[0168] Furthermore, the prediction unit 63 may use the second prediction model L to predict the execution time Tl(n) when performing data-parallel distributed learning. For example, if prediction values ​​are obtained from past execution results in the same way as prediction model W, the second prediction model L may be a machine learning model that inputs the residue allocation pattern to each execution node as shown in Figure 4 and the actual execution time for the allocation pattern as training data, and focuses on the time equivalent to the maximum calculation time in Figure 7. Then, for example, the execution time Tl(n) may be obtained by calculating a predicted performance improvement rate using the reference information 14 illustrated in FIG.

[0169] In the above-described embodiment, an example of calculating the potential energy of a protein is shown, but the present invention is not limited thereto. The information processing system 1 may be used to calculate the potential energy of a particle system other than a protein, and may be implemented in various modifications.

[0170] Furthermore, the above disclosure enables a person skilled in the art to implement and manufacture the present embodiment.

[0171] (E) Notes Regarding the above embodiment, the following supplementary notes are further disclosed.

[0172] (Appendix 1) A processor of an information processing device that manages distributed learning using a plurality of nodes calculating a waiting time until resources used in the distributed learning are secured and an execution time required for the distributed learning for each execution environment having a different number of nodes; calculating a score for each of the execution environments based on the waiting time and the execution time calculated for each of the execution environments; The number of nodes to be used in the distributed learning is determined based on the plurality of scores. Execute the process An information processing program comprising: (Appendix 2) The determined number of nodes to be used in the distributed learning is the number of nodes corresponding to the execution environment with the smallest score among the multiple scores. 2. An information processing program according to claim 1,

[0173] (Appendix 3) The scores include a first score related to execution time and a second score related to cost. 2. An information processing program according to claim 1,

[0174] (Appendix 4) The potential energy of a protein is calculated by processing a NN provided for each residue type constituting the protein in the plurality of nodes. 4. The information processing program according to any one of claims 1 to 3.

[0175] (Appendix 5) The process of calculating the execution time includes: calculating a predicted performance improvement rate for each of the plurality of execution environments based on a measurement result obtained by causing one of the plurality of nodes to execute a process related to the distributed learning; The execution time is calculated by reflecting the predicted performance improvement rate in the upper limit of the execution time. 4. The information processing program according to claim 1, further comprising:

[0176] (Appendix 6) The process of calculating the waiting time includes: The waiting time is obtained by inputting processing state information at the node into a machine learning model. 4. The information processing program according to claim 1, further comprising:

[0177] (Appendix 7) An information processing device for managing distributed learning using a plurality of nodes, calculating a waiting time until resources used in the distributed learning are secured and an execution time required for the distributed learning for each execution environment having a different number of nodes; calculating a score for each of the execution environments based on the waiting time and the execution time calculated for each of the execution environments; The number of nodes to be used in the distributed learning is determined based on the plurality of scores. An information processing device comprising a control unit that executes processing.

[0178] (Appendix 8) The determined number of nodes to be used in the distributed learning is the number of nodes corresponding to the execution environment with the smallest score among the multiple scores. 8. An information processing device according to claim 7. (Appendix 9) The scores include a first score related to execution time and a second score related to cost. 8. An information processing device according to claim 7.

[0179] (Appendix 10) The potential energy of a protein is calculated by processing a NN provided for each residue type constituting the protein in the plurality of nodes. 10. The information processing device according to any one of claims 7 to 9.

[0180] (Appendix 11) The process of calculating the execution time includes: calculating a predicted performance improvement rate for each of the plurality of execution environments based on a measurement result obtained by causing one of the plurality of nodes to execute a process related to the distributed learning; The execution time is calculated by reflecting the predicted performance improvement rate in the upper limit of the execution time. 10. The information processing device according to any one of claims 7 to 9, further comprising a process.

[0181] (Appendix 12) The process of calculating the waiting time includes: The waiting time is obtained by inputting processing state information at the node into a machine learning model. 10. The information processing device according to any one of claims 7 to 9, further comprising a process.

[0182] (Appendix 13) An information processing device that manages distributed learning using a plurality of nodes, calculating a waiting time until resources used in the distributed learning are secured and an execution time required for the distributed learning for each execution environment having a different number of nodes; calculating a score for each of the execution environments based on the waiting time and the execution time calculated for each of the execution environments; The number of nodes to be used in the distributed learning is determined based on the plurality of scores. 2. An information processing method comprising:

[0183] (Appendix 14) The determined number of nodes to be used in the distributed learning is the number of nodes corresponding to the execution environment with the smallest score among the multiple scores. 14. An information processing method according to claim 13. (Appendix 15) The scores include a first score related to execution time and a second score related to cost. 14. An information processing method according to claim 13.

[0184] (Appendix 16) The potential energy of a protein is calculated by processing a NN provided for each residue type constituting the protein in the plurality of nodes. 16. The information processing method according to any one of claims 13 to 15.

[0185] (Appendix 17) The process of calculating the execution time includes: calculating a predicted performance improvement rate for each of the plurality of execution environments based on a measurement result obtained by causing one of the plurality of nodes to execute a process related to the distributed learning; The execution time is calculated by reflecting the predicted performance improvement rate in the upper limit of the execution time. 16. The information processing method according to any one of claims 13 to 15, further comprising the steps of:

[0186] (Appendix 18) The process of calculating the waiting time includes: The waiting time is obtained by inputting processing state information at the node into a machine learning model. 16. The information processing method according to any one of claims 13 to 15, further comprising the steps of: [Explanation of symbols]

[0187] 1. Information Processing Systems 2 Network 3. Client terminal 4 Execution nodes 5 Execution Node 6 System Management Device 7 Distributed Learning Unit 10. Computers 10a processor 10b Graphics Processing Unit 10c Memory 10d storage section 10e IF section 10f IO section 10g Reading unit 10h Program 10i Recording media 10j Bus 11 Estimated calculation time information 12. Predicted cumulative calculation time information 13. Actual cumulative calculation time information 14 References 15 Expected performance improvement rate information 51 Training Execution Department 61 Job Deployment Department 62 Job Management Department 63 Prediction Department 64 Parallel number determination unit 65 Queue

Claims

1. A processor of an information processing device that manages distributed learning using a plurality of nodes calculating a waiting time until resources used in the distributed learning are secured and an execution time required for the distributed learning for each execution environment having a different number of nodes; calculating a score for each of the execution environments based on the waiting time and the execution time calculated for each of the execution environments; The number of nodes to be used in the distributed learning is determined based on the plurality of scores. An information processing program for causing a process to be executed.

2. 2. The information processing program according to claim 1, wherein the determined number of nodes to be used in the distributed learning is the number of nodes corresponding to the execution environment with the smallest score among the plurality of scores.

3. The scores include a first score related to execution time and a second score related to cost.

2. The information processing program according to claim 1 .

4. The potential energy of the protein is calculated by processing a neural network provided for each of the plurality of nodes, which is provided for each of the residue types constituting the protein.

4. The information processing program according to claim 1, wherein the information processing program is a program for processing a plurality of information.

5. The process of calculating the execution time includes: calculating a predicted performance improvement rate for each of the plurality of execution environments based on a measurement result obtained by causing one of the plurality of nodes to execute a process related to the distributed learning; The execution time is calculated by reflecting the predicted performance improvement rate in the upper limit of the execution time.

4. The information processing program according to claim 1, further comprising:

6. The process of calculating the waiting time includes: The waiting time is obtained by inputting processing state information at the node into a machine learning model.

4. The information processing program according to claim 1, further comprising:

7. An information processing device for managing distributed learning using a plurality of nodes, calculating a waiting time until resources used in the distributed learning are secured and an execution time required for the distributed learning for each execution environment having a different number of nodes; calculating a score for each of the execution environments based on the waiting time and the execution time calculated for each of the execution environments; The number of nodes to be used in the distributed learning is determined based on the plurality of scores. An information processing device comprising a control unit that executes processing.

8. An information processing device that manages distributed learning using a plurality of nodes, calculating a waiting time until resources used in the distributed learning are secured and an execution time required for the distributed learning for each execution environment having a different number of nodes; calculating a score for each of the execution environments based on the waiting time and the execution time calculated for each of the execution environments; The number of nodes to be used in the distributed learning is determined based on the plurality of scores.

2. An information processing method comprising:

Images