Information processing system, information processing apparatus, information processing method, scheduling method, and scheduling program

Abstract

To improve a training speed when executing model training processing.SOLUTION: A system that executes training processing of an NN using first and second workers, wherein the first worker generates a first output by executing Fwd processing on first data using a first parameter of the NN, generates a second output by executing Fwd processing on second data using the first parameter, and the second worker generates a third output by executing Fwd processing based on the first output using a second parameter of the NN, generating a fourth output by executing Fwd processing based on the second output using the second parameter, executing first processing on gradient information of the second parameter generated based on the third and fourth outputs, and updating the second parameter based on an execution result, wherein the first worker executes second processing using the updated second parameter received from the second worker.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system, an information processing device, an information processing method, a scheduling method, and a scheduling program. [Background technology]

[0002] Data parallelism and pipeline parallelism are known as techniques for improving the training speed when training a neural network. Generally, when training is performed by combining data parallelism and pipeline parallelism, a worker corresponding to each pipeline stage performs a ReduceScatter process on gradient information and an AllGather process on weight parameters.

[0003] These processes are executed in each worker using the network within the worker after the gradient information is calculated. Therefore, in order to improve the training speed, it is desirable to schedule the network bandwidth of each worker so that it can be used efficiently. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Samyyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He, "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models", arXiv:1910.02054 [cs.LG], 4 Oct 2019, [Retrieved 2024.3.13], Internet,<URL:https: / / arxiv.org / abs / 1910.02054> Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure improves the training speed when performing the model training process. [Means for solving the problem]

[0006] An information processing system according to an aspect of the present disclosure has, for example, the following configuration: An information processing system that performs a neural network training process using at least a first worker and a second worker, The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing forward processing on second data using the first parameters to generate second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating first gradient information of the second parameter based on the third output and the fourth output; performing a first process on the first gradient information; updating the second parameter based on a result of executing the first process on the first gradient information; Sending the updated second parameters to the first worker; The first worker: A second process is executed using the updated second parameters received from the second worker. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an example of a system configuration of an information processing system. [Figure 2] FIG. 2 illustrates an example of a hardware configuration of an information processing device. [Figure 3]FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing apparatus. [Figure 4] FIG. 10 is a diagram illustrating an example of scheduling constraints. [Figure 5A] FIG. 1 is a first diagram showing a specific example of scheduling information. [Figure 5B] FIG. 2 is a second diagram showing a specific example of scheduling information. [Figure 6A] FIG. 1 is a first diagram showing scheduling in a comparative example. [Figure 6B] FIG. 10 is a second diagram showing scheduling in a comparative example. [Figure 7] FIG. 10 is a diagram for explaining details of the ReduceScatter process of each worker. [Figure 8] FIG. 10 is a diagram for explaining details of the Allgather process of each worker. [Figure 9] FIG. 2 is a diagram illustrating an overview of scheduling by the information processing device according to the first embodiment. [Figure 10A] FIG. 11 is a first diagram showing a specific example of scheduling backward weight calculation by a worker with worker name="Worker3". [Figure 10B] FIG. 10 is a first diagram showing a specific example of scheduling of ReduceScatter processing and Allgather processing. [Figure 11A] FIG. 11 is a first diagram showing a specific example of scheduling backward weight calculation by a worker with worker name="Worker2". [Figure 11B] FIG. 2 is a second diagram showing a specific example of scheduling of ReduceScatter processing and Allgather processing. [Figure 12A] FIG. 10 is a first diagram showing a specific example of backward weight calculation by a worker with worker name="Worker1". [Figure 12B] FIG. 10 is a third diagram showing a specific example of scheduling of ReduceScatter processing and Allgather processing. [Figure 13A]FIG. 10 is a first diagram showing a specific example of backward weight calculation by a worker with worker name="Worker0". [Figure 13B] FIG. 4 is a fourth diagram showing a specific example of scheduling of ReduceScatter processing and Allgather processing. [Figure 14A] FIG. 2 is a second diagram showing a specific example of backward weight calculation using worker name="Worker0". [Figure 14B] FIG. 5 is a fifth diagram showing a specific example of scheduling ReduceScatter processing and Allgather processing. [Figure 15A] FIG. 2 is a second diagram showing a specific example of backward weight calculation by a worker with worker name="Worker1". [Figure 15B] FIG. 6 is a sixth diagram showing a specific example of scheduling ReduceScatter processing and Allgather processing. [Figure 16A] FIG. 2 is a second diagram showing a specific example of backward weight calculation by a worker with worker name="Worker2". [Figure 16B] FIG. 7 is a seventh diagram showing a specific example of scheduling ReduceScatter processing and Allgather processing. [Figure 17A] FIG. 2 is a second diagram showing a specific example of backward weight calculation by a worker with worker name="Worker3". [Figure 17B] FIG. 8 is an eighth diagram showing a specific example of scheduling ReduceScatter processing and Allgather processing. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0009] [First embodiment] <System configuration of information processing system> First, a system configuration of an information processing system according to the first embodiment will be described. Fig. 1 is a diagram showing an example of the system configuration of the information processing system. As shown in Fig. 1, the information processing system 100 according to the first embodiment has a plurality of server devices (a server device group 110) and an information processing device 120.

[0010] The server device group 110 executes training processing for a model to be trained (for example, a neural network, but is not limited to a neural network and may be a model other than a neural network). The training processing by the server device group 110 is executed based on a schedule (a training processing schedule that combines data parallelism and pipeline parallelism) generated by the information processing device 120.

[0011] The information processing device 120 performs data parallelization and pipeline parallelization on the training process for the model to be trained, and generates a schedule for efficiently executing the process on multiple workers. Note that in this embodiment, the term "worker" refers to multiple servers included in the server device group 110. In other words, one worker includes multiple servers.

[0012] However, the definition of a worker is not limited to this, and a worker may refer to one or more servers included in the server device group 110. Also, one worker may be one or more servers, or one or more information processing devices. In more general terms, a worker may be one device or a group of devices identified as a destination to which a schedule is assigned.

[0013] Alternatively, a worker may refer to multiple accelerators included in one server. In other words, one worker may include multiple accelerators. Alternatively, a worker may refer to one accelerator included in one server. In other words, one worker may equal one accelerator. Note that although an accelerator is given as an example here, an accelerator may also be read as a GPU (Graphics Processing Unit). Alternatively, an accelerator may also be read as a processor. In more general terms, a worker may be one component or a group of components identified as a destination for schedule allocation.

[0014] In this embodiment, during training processing, the processing that each worker performs on each micro-batch of training data includes forward calculation, backward calculation, ReduceScatter processing, and Allgather processing.

[0015] That is, the information processing device 120 Accepts scheduling information; - Generate a schedule for forward and backward calculations, Generate schedules for ReduceScatter and Allgather processes.

[0016] Specifically, the information processing device 120 stores, as scheduling information, for example, Configuration information indicating the configuration of the model being trained; The number of microbatches of training data used in the training process, The number of workers used in the training process, The execution order of each micro-batch when performing the training process. · The memory capacity of each worker used for training processing, ·Network bandwidth between each worker used for training processing, Accepts input such as:

[0017] Furthermore, when generating the schedules for forward calculation and backward calculation, the information processing device 120 generates forward calculation identifiers and backward calculation identifiers in numbers corresponding to the number of micro-batches included in the scheduling information.

[0018] In the information processing system 100 according to the first embodiment, when each worker is made to execute a backward calculation, the backward calculation is divided into a backward data calculation and a backward weight calculation. The backward data calculation refers to the part of the backward calculation that, for example, calculates the gradient of activation (data that is not a parameter). The backward weight calculation refers to the part of the backward calculation that, for example, calculates the gradient of a parameter. However, the method of dividing the backward calculation is not limited to this. For example, a part of the backward weight calculation may be calculated as a part of the backward data calculation, and the method of dividing the backward calculation is arbitrary.

[0019] Therefore, the information processing device 120 divides the generated backward calculation identifier into a backward data calculation identifier and a backward weight calculation identifier.

[0020] Next, the information processing device 120 places the generated forward calculation identifier, backward data calculation identifier, and backward weight calculation identifier at positions that indicate the execution timing of each worker based on the scheduling information. This allows the information processing device 120 to schedule the execution timing of the forward calculation, backward data calculation, and backward weight calculation when each micro-batch is input. Note that the information processing device 120 schedules the forward calculation, backward data calculation, and backward weight calculation so as to satisfy a pre-stored constraint (a first constraint on the execution order of the forward calculation, backward data calculation, and backward weight calculation).

[0021] Next, the information processing device 120 executes the following at the execution timings of the forward calculation, backward data calculation, and backward weight calculation when each micro-batch is input: ReduceScatter processing in each worker, and Allgather processing in each worker, The information processing device 120 schedules the execution procedure of the above. Note that the information processing device 120 schedules the procedure so as to satisfy a pre-stored constraint (a second constraint regarding the execution order of the ReduceScatter process and the Allgather process).

[0022] The information processing device 120 transmits the generated schedule to the server device group 110. This allows the server device group 110 to execute training processing based on the schedule generated by the information processing device 120.

[0023] As an example of the training process executed by each worker in the server device group 110, for example, when the model to be trained is a neural network (NN), · Worker name = Worker0: 1st layer of NN, · Worker name = Worker1: Second layer of NN, ··· For example, each worker may perform a training process for a corresponding layer, such as: Worker name = Worker0:NN, from the 1st layer to the Nth layer, Worker name = Worker1: NN (N+1)th layer to 2Nth layer, For example, each worker may perform training for multiple layers, such as in the following example. In other words, the NN may be divided as evenly as possible, and each worker may be responsible for training the layers closest to the input.

[0024] However, if the number of layers in the NN is not divisible by the number of workers, the number of layers that some workers are responsible for when performing the training process may be less than the number of layers that other workers are responsible for when performing the training process.Alternatively, if special calculations are included in the layers around the input and the layers around the output, the calculation load may become unbalanced among the workers.

[0025] <Hardware configuration of information processing device> Next, the hardware configuration of the information processing device 120 will be described. Fig. 2 is a diagram showing an example of the hardware configuration of the information processing device. The information processing device 120 has, as components, a processor 201, a main storage device 202 (memory), an auxiliary storage device 203 (memory), a network interface 204, and a device interface 205. The information processing device 120 may be realized as a computer in which these components are connected via a bus 206. Note that, in the example of Fig. 2, the information processing device 120 is shown as having one of each component, but the information processing device 120 may have multiple of the same component.

[0026] Various calculations of the information processing device 120 may be executed in parallel using one or more processors. Furthermore, various calculations may be distributed to multiple processing cores in the processor 201 and executed in parallel. Furthermore, some or all of the processes, means, etc. disclosed herein may be executed by an external device 230 (at least one of a processor and a storage device) provided on a cloud that can communicate with the information processing device 120 via the network interface 204.

[0027] The processor 201 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.). The processor 201 may also be a semiconductor device including a dedicated processing circuit. The processor 201 is not limited to an electronic circuit using electronic logic elements, and may be realized by an optical circuit using optical logic elements. The processor 201 may also include an arithmetic function based on quantum computing.

[0028] The processor 201 performs various calculations based on various data and commands input from each device, etc., in the internal configuration of the information processing device 120, and outputs the calculation results and control signals to each device, etc. The processor 201 controls each component included in the information processing device 120 by executing an OS (Operating System), applications, etc.

[0029] Furthermore, processor 201 may refer to one or more electronic circuits arranged on a single chip, or to one or more electronic circuits arranged on two or more chips or devices. When multiple electronic circuits are used, the electronic circuits may communicate with each other via wires or wirelessly.

[0030] The main memory device 202 is a memory device that stores instructions executed by the processor 201 and various data, and the various data stored in the main memory device 202 is read by the processor 201. The auxiliary memory device 203 is a memory device other than the main memory device 202. Note that these memory devices refer to any electronic component that can store various data (for example, the first constraint and second constraint stored in the constraint storage unit 310, which will be described later), and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. The memory device for saving various data in the information processing device 120 may be realized by the main memory device 202 or the auxiliary memory device 203, or may be realized by an internal memory built into the processor 201.

[0031] Furthermore, multiple processors 201 may be connected (coupled) to one main memory device 202, or a single processor 201 may be connected. Alternatively, multiple main memory devices 202 may be connected (coupled) to one processor 201. When the information processing device 120 is configured with at least one main memory device 202 and multiple processors 201 connected (coupled) to this at least one main memory device 202, it may include a configuration in which at least one processor of the multiple processors 201 is connected (coupled) to at least one main memory device 202.

[0032] The network interface 204 is an interface for connecting to a communication network 220 wirelessly or via a wired connection.

[0033] The device interface 205 is an interface such as a USB that directly connects to an external device 240 .

[0034] The external device 240 may be, for example, an input device. In this embodiment, the input device is, for example, an electronic device such as a camera, a microphone, various sensors, a keyboard, a mouse, or a touch panel, and provides acquired information to the information processing device 120.

[0035] Furthermore, the external device 240 may be, for example, an output device. In this embodiment, the output device may be, for example, a display device such as an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube), a PDP (Plasma Display Panel), or an organic EL (Electro Luminescence) panel, or may be a speaker that outputs sound or the like.

[0036] The external device 240 may also be a storage device (memory). For example, the external device 240 may be a network storage or the like, or may be a storage such as an HDD.

[0037] Furthermore, the external device 240 may be a device that has some of the functions of the components of the information processing device 120. In other words, the information processing device 120 may transmit and receive processing results to and from the external device 240.

[0038] Note that the hardware configuration of the information processing device 120 has been described here, and no mention has been made of the hardware configuration of each of the multiple server devices included in the server device group 110. However, at least one server device included in the server device group 110 may have the same hardware configuration as the information processing device 120.

[0039] <Functional configuration of information processing device> Next, a description will be given of the functional configuration of the information processing device 120. Fig. 3 is a diagram showing an example of the functional configuration of the information processing device. A scheduling program is installed in the information processing device 120, and by executing this program, the information processing device 120 functions as an identifying unit 301, a dividing unit 302, a scheduling unit 303, and a transmitting unit 304.

[0040] The identification unit 301 accepts input of scheduling information. Details of the scheduling information accepted by the identification unit 301 have already been explained using FIG. 1, and therefore will not be explained here. The identification unit 301 notifies the scheduling unit 303 of the accepted input of the scheduling information. Furthermore, the identification unit 301 generates forward calculation identifiers and backward calculation identifiers in numbers corresponding to the "number of micro-batches" included in the accepted input of the scheduling information. Furthermore, the identification unit 301 notifies the dividing unit 302 of the generated forward calculation identifiers and backward calculation identifiers.

[0041] The dividing unit 302 further divides the backward calculation identifiers into backward data calculation identifiers and backward weight calculation identifiers among the forward calculation identifiers and backward calculation identifiers, the number of which corresponds to the "number of micro-batches" notified from the specifying unit 301. The dividing unit 302 notifies the scheduling unit 303 of the forward calculation identifiers, the number of which corresponds to the number of micro-batches, and the backward data calculation identifiers and backward weight calculation identifiers, the number of which corresponds to the number of micro-batches.

[0042] The scheduling unit 303 acquires the scheduling information notified by the specifying unit 301, and the forward calculation identifier, backward data calculation identifier, and backward weight calculation identifier notified by the dividing unit 302. Furthermore, based on the scheduling information and the first constraint read from the constraint storage unit 310, the scheduling unit 303 - Forward calculation identifiers, the number of which corresponds to the number of micro-batches; -Backward data calculation identifiers, the number of which corresponds to the number of micro-batches; - Backward weight calculation identifiers, the number of which corresponds to the number of micro-batches; By placing the at a position that indicates the execution timing of each worker, the execution procedures of forward calculation, backward data calculation, and backward weight calculation in the training process using each micro-batch are scheduled.

[0043] Furthermore, the scheduling unit 303 determines, based on the scheduling information notified by the specifying unit 301 and the second constraint condition read from the constraint condition storage unit 310, · The execution procedure for each worker in the ReduceScatter process, and Execution procedure for each worker in the Allgather process, The scheduling unit 303 generates schedules for the ReduceScatter process and the Allgather process based on the schedules generated for the forward calculation, backward data calculation, and backward weight calculation.

[0044] The transmission unit 304 transmits the schedule generated by the scheduling unit 303 to the server device group 110 .

[0045] <Example of constraints> Next, a description will be given of the details of the first and second constraint conditions stored in the constraint condition storage unit 310. Fig. 4 is a diagram showing an example of the constraint conditions.

[0046] When scheduling the execution procedures of the forward calculation, backward data calculation, and backward weight calculation of each micro-batch, the information processing device 120 performs the scheduling so as to satisfy the first constraint condition. As shown in FIG. 4, the first constraint condition is as follows. 1) The forward computation in the training process using each micro-batch is executed among the workers in a specific execution order. 2) Each worker performs backward data calculations in the training process using each micro-batch after forward calculations in the training process using each micro-batch. 3) The backward data calculations in the training process using each micro-batch are executed among workers in the reverse order of the specific execution order mentioned above. 4) Each worker performs backward weight calculations in the training process using each micro-batch after backward data calculations in the training process using each micro-batch.

[0047] In the information processing device 120, the scheduling unit 303 searches for an arrangement that minimizes the training time, for example, while arranging each calculation identifier notified by the dividing unit 302 at a position that indicates the execution timing of each worker so as to satisfy the first constraint. Note that the scheduling unit 303 may search for an arrangement that minimizes the training time by solving an optimization problem.

[0048] Furthermore, when scheduling the execution procedures of the ReduceScatter process and the Allgather process in the training process using each micro-batch in each worker, the information processing device 120 schedules them so as to satisfy the second constraint. As shown in FIG. 4, the second constraint is that "each worker executes the ReduceScatter process and the Allgather process in parallel with the backward weight calculation in the training process using each micro-batch, or after the backward weight calculation." Note that "parallel" refers to a state in which multiple processes are executed with at least a partial overlap in time. Also, the ReduceScatter process refers to a reduction process of information (e.g., gradient information) by collective communication, and the Allgather process refers to a parallel collection process of parameters (e.g., weight parameters) by collective communication. Note that the gradient information refers to information necessary for updating each weight parameter, Information before ReduceScatter processing (gradient information of weight parameters calculated using part of the input data), and Information after ReduceScatter processing (the gradient information transformed in some way, typically compressed), Includes:

[0049] In the information processing device 120, the scheduling unit 303 schedules the execution procedures of the ReduceScatter process and the Allgather process for each micro-batch so as to improve the training speed while satisfying the second constraint. Specifically, in the first embodiment, the scheduling unit 303 improves the training speed during training by each worker by scheduling so as to effectively utilize the network bandwidth within each worker.

[0050] <Specific examples of scheduling> Next, a specific example of scheduling by scheduling section 303 will be described.

[0051] (1) Scheduling information First, a specific example of scheduling information that is input when scheduling section 303 performs scheduling will be described. Figures 5A and 5B are first and second diagrams showing a specific example of scheduling information.

[0052] As shown in Figures 5A and 5B, the neural network, which is the model to be trained, has four layers, "NN0" to "NN3." Of these, Figure 5A shows a case where microbatches 0 to 3 are input when a training process is executed on the model to be trained. Also, Figure 5B shows a case where microbatches 4 to 7 are input when a training process is executed on the model to be trained.

[0053] The example in Figure 5A also shows a situation in which a worker with worker name = "Worker0" is assigned to the training process for the "NN0" layer, and a worker with worker name = "Worker1" is assigned to the training process for the "NN1" layer. The example in Figure 5A also shows a situation in which a worker with worker name = "Woker2" is assigned to the training process for the "NN2" layer, and a worker with worker name = "Worker3" is assigned to the training process for the "NN3" layer.

[0054] In the example of FIG. 5A, each worker is one server device, and each server device has four accelerators.

[0055] Similarly, the example in Figure 5B shows a situation in which a worker with worker name = "Worker3" is assigned to the training process for the "NN0" layer, and a worker with worker name = "Worker2" is assigned to the training process for the "NN1" layer. Also, the example in Figure 5B shows a situation in which a worker with worker name = "Woker1" is assigned to the training process for the "NN2" layer, and a worker with worker name = "Worker0" is assigned to the training process for the "NN3" layer.

[0056] In the example of Figure 5B, each worker is the same as each worker shown in the example of Figure 5A (each is a server device, and each server device has four accelerators), but the allocation destinations are different from each worker shown in the example of Figure 5A.

[0057] Based on the scheduling information shown in FIGS. 5A and 5B, in specific examples ((5) and (6)) described later, the scheduling unit 303 performs the following processing based on a combination of data parallelism and pipeline parallelism: Scheduling to assign forward calculations, backward data calculations, and backward weight calculations in the training process using each microbatch (microbatch 0 to 3) to each worker with worker name = "Worker0" to "Worker3"; Scheduling to assign the ReduceScatter process and Allgather process in the training process using each microbatch (microbatch 0 to 3) to each worker with worker name = "Worker0" to "Worker3"; Scheduling to assign forward calculations, backward data calculations, and backward weight calculations in the training process using each microbatch (microbatch 4 to 7) to each worker with worker name = "Worker3" to "Worker0"; Scheduling to assign the ReduceScatter process and Allgather process in the training process using each microbatch (microbatch 4 to 7) to each worker with worker name = "Worker3" to "Worker0"; This section explains how to do this.

[0058] However, the scheduling of the forward calculation, backward data calculation, and backward weight calculation in the training process using each microbatch (microbatch 0 to 3, 4 to 7) will show the schedule after generation. In other words, the explanation of the allocation process of each calculation identifier during scheduling will be omitted.

[0059] (2) Scheduling of Comparative Example Next, a schedule generated by a scheduling unit of a comparative example based on the scheduling information shown in Fig. 5A will be described. The scheduling unit of a comparative example is a general term for a scheduling unit for clarifying the difference between the schedule generated by the scheduling unit 303 of the information processing device 120 according to the first embodiment and a general schedule. Fig. 6A is a first diagram showing scheduling of a comparative example.

[0060] In Fig. 6A, reference numeral 600 denotes an example of a schedule generated by the scheduling unit of the comparative example for the forward calculation, backward data calculation, and backward weight calculation in the training process using each micro-batch (micro-batch 0 to 3). The scheduling unit of the comparative example generates the schedule indicated by reference numeral 600 by scheduling based on the scheduling information shown in Fig. 5A so as to satisfy the first constraint condition shown in Fig. 4.

[0061] In the graph 600, the horizontal axis indicates time and the vertical axis indicates the worker name of each worker. Each calculation identifier is placed in the area where each time and each worker name intersect.

[0062] Of the calculation identifiers placed in the schedule shown in reference numeral 600, Fwd0 to Fwd3 are the forward calculation identifiers in the training process using microbatch 0 to the forward calculation identifiers in the training process using microbatch 3, BD0 to BD3 are the backward data calculation identifiers in the training process using micro batch 0 to the backward data calculation identifiers in the training process using micro batch 3, BW0 to BW3 are the backward weight calculation identifiers in the training process using micro batch 0 to the backward weight calculation identifiers in the training process using micro batch 3, The length of each calculation identifier along the time axis represents the time required for each calculation. Also, the time when no calculation identifier is assigned represents the time when the worker is not executing the process.

[0063] Also, in FIG. 6A, reference numeral 604 indicates the start timing when the worker with worker name="Worker0" executes the ReduceScatter process and Allgather process in the training process using microbatches 0 to 3. The ReduceScatter process and Allgather process can be executed after the final backward data calculation ("BW3") is completed. Therefore, the scheduling unit in the comparative example schedules the ReduceScatter process and Allgather process by the worker with worker name="Worker0" to start at the timing indicated by reference numeral 604.

[0064] Similarly, in FIG. 6A, reference numeral 603 indicates the start timing when a worker with worker name="Worker1" executes ReduceScatter processing and Allgather processing in training processing using microbatches 0 to 3. The ReduceScatter processing and Allgather processing can be executed after the final backward data calculation ("BW3") is completed. For this reason, the scheduling unit in the comparative example schedules the ReduceScatter processing and Allgather processing by the worker with worker name="Worker1" to start at the timing indicated by reference numeral 603.

[0065] Similarly, in FIG. 6A, reference numeral 602 indicates the start timing when a worker with worker name="Worker2" executes ReduceScatter processing and Allgather processing in training processing using microbatches 0 to 3. The ReduceScatter processing and Allgather processing can be executed after the final backward data calculation ("BW3") is completed. For this reason, the scheduling unit in the comparative example schedules the ReduceScatter processing and Allgather processing by the worker with worker name="Worker2" to start at the timing indicated by reference numeral 602.

[0066] Similarly, in FIG. 6A, reference numeral 601 indicates the start timing when a worker with worker name="Worker3" executes ReduceScatter processing and Allgather processing in training processing using microbatches 0 to 3. The ReduceScatter processing and Allgather processing can be executed after the final backward data calculation ("BW3") is completed. For this reason, the scheduling unit in the comparative example schedules the ReduceScatter processing and Allgather processing by the worker with worker name="Worker3" to start at the timing indicated by reference numeral 601.

[0067] 6B is a second diagram showing scheduling in a comparative example. The scheduling unit in the comparative example Each worker performs the ReduceScatter process on the gradient information calculated during the training process using each microbatch (microbatch 0-3, 4-7) (reducing the gradient information held by each rank (each accelerator)). Each worker updates each weight parameter based on the information obtained by executing the ReduceScatter process, and then executes the Allgather process for each updated weight parameter. The updated weight parameters include not only the updated weight parameters themselves, but also those compressed from the updated weight parameters and those converted from the updated weight parameters by any method (those that can restore or obtain the updated weight parameters).

[0068] Specifically, the scheduling unit of the comparative example generates the following schedule as a schedule for executing the ReduceScatter process and the Allgather process in the training process using microbatches 0 to 3, as shown in FIG. 6B. i) The worker with worker name="Worker3" executes the ReduceScatter process on each gradient information. The worker with worker name="Worker3" also updates each weight parameter based on the information obtained by executing the ReduceScatter process, and executes the Allgather process on each updated weight parameter. ii) The worker with worker name = "Worker2" executes the ReduceScatter process for each gradient information. The worker with worker name = "Worker2" also updates each weight parameter based on the information obtained by executing the ReduceScatter process, and executes the Allgather process for each updated weight parameter. iii) The worker with worker name = "Worker1" executes the ReduceScatter process for each gradient information. The worker with worker name = "Worker1" also updates each weight parameter based on the information obtained by executing the ReduceScatter process, and executes the Allgather process for each updated weight parameter. iv) The worker with worker name = "Worker0" executes the ReduceScatter process on each gradient information. The worker with worker name = "Worker0" also updates each weight parameter based on the information obtained by executing the ReduceScatter process, and executes the Allgather process on each updated weight parameter.

[0069] Similarly, the scheduling unit of the comparative example generates the following schedule as a schedule for executing the ReduceScatter process and the Allgather process in the training process using microbatches 4 to 7, as shown in FIG. 6B. v) The worker with worker name = "Worker0" executes the ReduceScatter process for each gradient information. The worker with worker name = "Worker0" also updates each weight parameter based on the information obtained by executing the ReduceScatter process, and executes the Allgather process for each updated weight parameter. vi) The worker with worker name = "Worker1" executes the ReduceScatter process for each gradient information. The worker with worker name = "Worker1" also updates each weight parameter based on the information obtained by executing the ReduceScatter process, and executes the Allgather process for each updated weight parameter. vii) The worker with worker name = "Worker2" executes the ReduceScatter process on each gradient information. The worker with worker name = "Worker2" also updates each weight parameter based on the information obtained by executing the ReduceScatter process, and executes the Allgather process on each updated weight parameter. viii) The worker with worker name="Worker3" executes the ReduceScatter process on each gradient information. The worker with worker name="Worker3" also updates each weight parameter based on the information obtained by executing the ReduceScatter process, and executes the Allgather process on each updated weight parameter.

[0070] Note that the processes described in i) to iv) are executed in parallel and do not indicate the order of execution. Similarly, the processes described in v) to viii) are executed in parallel and do not indicate the order of execution.

[0071] The schedule generated in this manner is transmitted to the server device group 110 and distributed to each worker, as described above. The method of distribution to each worker is arbitrary; for example, if a schedule is generated in a server device other than each worker, the other server device distributes the schedule to each worker. Also, if a schedule is generated in one of the workers, the one worker distributes it to the other workers. Also, if the same schedule is generated in each worker, each worker extracts a corresponding portion of the schedule.

[0072] (3) ReduceScatter processing and Allgather processing in comparison examples Next, the ReduceScatter process and Allgather process scheduled by the scheduling unit 303 of the comparative example will be described in detail.

[0073] 7 is a diagram for explaining the details of the ReduceScatter process of each worker. As described above, the worker with worker name="Worker3" executes the ReduceScatter process at the start timing of reference numeral 601.

[0074] Specifically, before the start timing of reference numeral 601, each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker 3" performs the training process using each microbatch (microbatch 0 to 3): Backward data calculation, and Backward weight calculation, As a result, as shown in FIG. 7, each accelerator (Accelerator0 to 3) owned by the worker with worker name="Worker3" acquires each gradient information (IN_0 to IN_3). Note that each gradient information (IN_0 to IN_3) contains a number of gradient information corresponding to the number of accelerators (4 in the example of FIG. 7). The dotted lines included in each of IN_0 to IN_3 in FIG. 7 indicate that IN_0 to IN_3 each contains four gradient information. Then, at the start timing of reference numeral 601, the worker with worker name="Worker3" contracts the gradient information acquired by each accelerator between each accelerator, and one of the accelerators holds the contraction results. For example, the worker with worker name="Worker3" contracts the first-stage gradient information of the four gradient information included in each of IN_0 to IN_3 between each accelerator, and Accelerator0 holds the contraction results (note that the second to fourth stages are processed in the same way). As a result, each accelerator of the worker with worker name="Worker3" acquires the reduction results (OUT0 to OUT3).

[0075] Similarly, the worker with the worker name="Worker2" executes the ReduceScatter process at the start timing of reference numeral 602.

[0076] Specifically, before the start timing of reference numeral 602, each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker 2" performs the training process using each microbatch (microbatch 0 to 3): Backward data calculation, and Backward weight calculation, As a result, as shown in FIG. 7, each accelerator (Accelerator0 to 3) owned by the worker with worker name = "Worker2" acquires each gradient information (IN_0 to IN_3). Note that each gradient information (IN_0 to IN_3) contains a number of gradient information corresponding to the number of accelerators (4 in the example of FIG. 7). The dotted lines included in each of IN_0 to IN_3 in FIG. 7 indicate that IN_0 to IN_3 each contains four gradient information. Then, at the start timing of reference numeral 602, the worker with worker name = "Worker2" contracts the gradient information acquired by each accelerator between each accelerator, and one of the accelerators holds the contraction results. For example, the worker with worker name = "Worker2" contracts the first-stage gradient information of the four gradient information included in each of IN_0 to IN_3 between each accelerator, and Accelerator0 holds the contraction results (note that the second to fourth stages are processed in the same way). As a result, each accelerator of the worker with worker name="Worker2" acquires the reduction results (OUT0 to OUT3).

[0077] Similarly, the worker with the worker name="Worker1" executes the ReduceScatter process at the start timing of reference numeral 603.

[0078] Specifically, before the start timing of reference numeral 603, each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker 1" performs the training process using each microbatch (microbatch 0 to 3), Backward data calculation, and Backward weight calculation, As a result, as shown in FIG. 7, each accelerator (Accelerator0 to 3) owned by the worker with worker name = "Worker1" acquires each gradient information (IN_0 to IN_3). Note that each gradient information (IN_0 to IN_3) contains a number of gradient information corresponding to the number of accelerators (4 in the example of FIG. 7). The dotted lines included in each of IN_0 to IN_3 in FIG. 7 indicate that IN_0 to IN_3 each contains four gradient information. Then, at the start timing of reference numeral 603, the worker with worker name = "Worker1" contracts the gradient information acquired by each accelerator between each accelerator, and one of the accelerators holds the contraction results. For example, the worker with worker name = "Worker1" contracts the first-stage gradient information of the four gradient information included in each of IN_0 to IN_3 between each accelerator, and Accelerator0 holds the contraction results (note that the second to fourth stages are processed in the same way). As a result, each accelerator of the worker with worker name="Worker1" acquires the reduction results (OUT0 to OUT3).

[0079] Similarly, the worker with the worker name="Worker0" executes the ReduceScatter process at the start timing of reference numeral 604.

[0080] Specifically, before the start timing of reference numeral 604, each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker 0" performs the training process using each microbatch (microbatch 0 to 3): Backward data calculation, and Backward weight calculation, As a result, as shown in FIG. 7, each accelerator (Accelerator0 to 3) owned by the worker with worker name = "Worker0" acquires each gradient information (IN_0 to IN_3). Note that each gradient information (IN_0 to IN_3) contains a number of gradient information corresponding to the number of accelerators (four in the example of FIG. 7). The dotted lines included in each of IN_0 to IN_3 in FIG. 7 indicate that IN_0 to IN_3 each contains four gradient information. Then, at the start timing of reference numeral 604, the worker with worker name = "Worker0" contracts the gradient information acquired by each accelerator between each accelerator, and one of the accelerators holds the contraction results. For example, the worker with worker name = "Worker0" contracts the first-stage gradient information of the four gradient information included in each of IN_0 to IN_3 between each accelerator, and Accelerator0 holds the contraction results (note that the second to fourth stages are processed in the same way). As a result, each accelerator of the worker with worker name="Worker0" acquires the reduction results (OUT0 to OUT3).

[0081] In the example of Fig. 7, the ReduceScatter process is described as being executed on each piece of gradient information calculated in the training process using microbatches 0 to 3. However, the same applies when the ReduceScatter process is executed on each piece of gradient information calculated in the training process using microbatches 4 to 7. Therefore, a description thereof will be omitted here.

[0082] 8 is a diagram for explaining the details of the Allgather process of each worker. When the ReduceScatter process of the worker with the worker name="Worker3" is completed, each accelerator (Accelerator0 to 3) of the worker with the worker name="Worker3" is Using the gradient information for which the ReduceScatter process was executed, the optimizer state (for example, 32 bits) of NN3 owned by the worker with worker name = "Worker3" is updated, Convert the weight parameters included in the updated optimizer state (for example, convert 32-bit weight parameters to 16-bit weight parameters).

[0083] After that, the worker with worker name="Worker3" executes the Allgather process as shown in Figure 8. Note that the optimizer state includes various values ​​required for optimization in addition to the weight parameters, but after being converted (for example, to 16 bits), the Allgather process is executed only on the weight parameters.

[0084] Specifically, the worker with worker name="Worker3" collects weight parameters (IN_0 to IN_3). The worker with worker name="Worker3" distributes the collected weight parameters to the accelerators (Accelerator0 to 3) owned by the worker with worker name="Worker3".

[0085] Similarly, when the ReduceScatter process in the worker with worker name = "Worker2" is completed, each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker2" is Using the gradient information for which the ReduceScatter process was executed, the optimizer state (for example, 32 bits) of NN2 owned by the worker with worker name = "Worker2" is updated, Convert the weight parameters included in the updated optimizer state (for example, convert 32-bit weight parameters to 16-bit weight parameters).

[0086] After that, the worker with worker name = "Worker2" executes the Allgather process as shown in Figure 8. Note that the optimizer state includes various values ​​required for optimization in addition to the weight parameters, but after being converted (for example, to 16 bits), the Allgather process is executed only on the weight parameters.

[0087] Specifically, the worker with worker name="Worker2" collects weight parameters (IN_0 to IN_3). The worker with worker name="Worker2" distributes the collected weight parameters to the accelerators (Accelerator0 to 3) owned by the worker with worker name="Worker2".

[0088] Similarly, when the ReduceScatter process in the worker with worker name = "Worker1" is completed, each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker1" is Using the gradient information for which the ReduceScatter process was executed, the optimizer state (for example, 32 bits) of NN1 owned by the worker with worker name = "Worker1" is updated, Convert the weight parameters included in the updated optimizer state (for example, convert 32-bit weight parameters to 16-bit weight parameters).

[0089] After that, the worker with worker name = "Worker1" executes the Allgather process as shown in Figure 8. Note that the optimizer state includes various values ​​required for optimization in addition to the weight parameters, but after being converted (for example, to 16 bits), the Allgather process is executed only on the weight parameters.

[0090] Specifically, the worker with worker name="Worker1" collects weight parameters (IN_0 to IN_3). The worker with worker name="Worker1" distributes the collected weight parameters to the accelerators (Accelerator0 to 3) owned by the worker with worker name="Worker1".

[0091] Similarly, when the ReduceScatter process in the worker with worker name = "Worker0" is completed, each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker0" is Using the gradient information for which the ReduceScatter process was executed, the optimizer state (for example, 32 bits) of NN0 owned by the worker with worker name = "Worker0" is updated, Convert the weight parameters included in the updated optimizer state (for example, convert 32-bit weight parameters to 16-bit weight parameters).

[0092] After that, the worker with worker name = "Worker0" executes the Allgather process as shown in Figure 8. Note that the optimizer state includes various values ​​required for optimization in addition to the weight parameters, but after being converted (for example, to 16 bits), the Allgather process is executed only on the weight parameters.

[0093] Specifically, the worker with worker name="Worker0" collects weight parameters (IN_0 to IN_3). The worker with worker name="Worker0" distributes the collected weight parameters to the accelerators (Accelerator0 to 3) owned by the worker with worker name="Worker0".

[0094] As is clear from the explanation of FIGS. 7 and 8, when the ReduceScatter process and the Allgather process are executed, communication (reduction, gathering, distribution) occurs between the accelerators in each worker.

[0095] On the other hand, the start timing of the ReduceScatter process and the start timing of the Allgather process differ between workers, which means that the communication timing between accelerators that occurs when the ReduceScatter process and the Allgather process are executed also differs between workers.

[0096] For example, when a worker with worker name="Worker2" starts ReduceScatter processing of gradient information, a worker with worker name="Worker0" has not yet started ReduceScatter processing of gradient information.

[0097] In other words, the worker with worker name="Worker0" is still in a state where the network bandwidth usage rate within the worker is low at the time when the worker with worker name="Worker2" starts the ReduceScatter processing of the gradient information.

[0098] In the information processing device 120 according to the first embodiment, the scheduling unit 303 focuses on this point and generates a schedule that improves the training speed by effectively utilizing the network bandwidth within each worker. Below, an overview of the scheduling performed by the scheduling unit 303 and a specific example of the generated schedule will be described.

[0099] (4) Overview of scheduling by the scheduling unit 303 First, an overview of scheduling by scheduling section 303 will be described. FIG. 9 is a diagram showing an overview of scheduling. As shown in FIG. 9, scheduling section 303 The first worker performs a ReduceScatter operation on each gradient. The first worker transmits information obtained by executing the ReduceScatter operation to the second worker; The second worker receives, from the first worker, information obtained by executing the ReduceScatter operation. The second worker performs an allgather process on the weight parameters. The first worker is a worker that is scheduled to execute backward data calculations in the training process using each microbatch (microbatches 0 to 3, 4 to 7) earlier than the second worker.

[0100] Specifically, as shown in FIG. 9, the scheduling unit 303 generates the following schedule as a schedule for executing the ReduceScatter process and the Allgather process in the training process using microbatches 0 to 3. i) The worker with worker name="Worker3" executes the ReduceScatter process for each gradient information. The worker with worker name="Worker3" also updates each weight parameter based on the information obtained by executing the ReduceScatter process and sends it to the worker with worker name="Worker0". The worker with worker name="Worker0" executes the Allgather process for each updated weight parameter. ii) The worker with worker name="Worker2" executes the ReduceScatter process for each gradient information. The worker with worker name="Worker2" also updates each weight parameter based on the information obtained by executing the ReduceScatter process and sends it to the worker with worker name="Worker1". The worker with worker name="Worker1" executes the Allgather process for each updated weight parameter. iii) The worker with worker name="Worker1" executes the ReduceScatter process for each gradient information. The worker with worker name="Worker1" also updates each weight parameter based on the information obtained by executing the ReduceScatter process and sends it to the worker with worker name="Worker2". The worker with worker name="Worker2" executes the Allgather process for each updated weight parameter. iv) The worker with worker name="Worker0" executes the ReduceScatter process for each gradient information. The worker with worker name="Worker0" also updates each weight parameter based on the information obtained by executing the ReduceScatter process and sends it to the worker with worker name="Worker3". The worker with worker name="Worker3" executes the Allgather process for each updated weight parameter.

[0101] Similarly, as shown in FIG. 9, the scheduling unit 303 generates the following schedule as a schedule for executing the ReduceScatter process and the Allgather process in the training process using microbatches 4 to 7. v) The worker with worker name="Worker0" executes the ReduceScatter process for each gradient information. The worker with worker name="Worker0" also sends the information obtained by executing the ReduceScatter process to the worker with worker name="Worker3". The worker with worker name="Worker3" updates each weight parameter based on the sent information and executes the Allgather process for each updated weight parameter. vi) The worker with worker name="Worker1" executes the ReduceScatter process for each gradient information. The worker with worker name="Worker1" also sends the information obtained by executing the ReduceScatter process to the worker with worker name="Worker2". The worker with worker name="Worker2" updates each weight parameter based on the sent information and executes the Allgather process for each updated weight parameter. vii) The worker with worker name="Worker2" executes the ReduceScatter process for each gradient information. The worker with worker name="Worker2" also sends the information obtained by executing the ReduceScatter process to the worker with worker name="Worker1". The worker with worker name="Worker1" updates each weight parameter based on the sent information and executes the Allgather process for each updated weight parameter. viii) The worker with worker name="Worker3" executes the ReduceScatter process for each gradient information. The worker with worker name="Worker3" also sends the information obtained by executing the ReduceScatter process to the worker with worker name="Worker0". The worker with worker name="Worker0" updates each weight parameter based on the sent information and executes the Allgather process for each updated weight parameter.

[0102] Note that the processes described in i) to iv) are executed in parallel and do not indicate the order of execution. Similarly, the processes described in v) to viii) are executed in parallel and do not indicate the order of execution.

[0103] (5) Details of scheduling by the scheduling unit 303, part 1 Next, we will explain the details of scheduling by the scheduling unit 303 when executing the ReduceScatter process and the Allgather process in the training process using microbatches 0 to 3. Here, we will also explain the scheduling of the backward weight calculation that is executed in parallel with the ReduceScatter process and the Allgather process.

[0104] (5-1) Processing by worker with worker name = "Worker3" First, the scheduling of backward weight calculation by a worker with worker name="Worker3" will be described. Fig. 10A is a first diagram showing a specific example of scheduling backward weight calculation by a worker with worker name="Worker3".

[0105] As shown in FIG. 10A, the scheduling unit 303 generates the following schedule for backward weight calculation after completing backward data calculation in the training process using micro-batch 3. Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker3" performs backward weight calculations for layer 0 included in NN3 during the training process using each microbatch (microbatch 0 to 3). Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker3" performs backward weight calculations for layer 1 included in NN3 during the training process using each microbatch (microbatch 0 to 3). Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker3" performs backward weight calculations for layer 2 included in NN3 during the training process using each microbatch (microbatch 0 to 3). Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker3" performs backward weight calculations for layer 3 included in NN3 during the training process using each microbatch (microbatch 0 to 3).

[0106] Note that layer0 to layer3 included in NN3 refer to layers within the backward weight calculation. After completing the backward data calculation in the training process using microbatch 3, the worker with worker name="Worker3" performs ReduceScatter processing sequentially each time it calculates gradient information that can be ReduceScatter processed. Specifically, the worker with worker name="Worker3" may perform ReduceScatter processing, for example, each time it completes the backward weight calculation for each layer. That is, it may perform ReduceScatter processing each time it calculates gradient information for one layer as gradient information that can be ReduceScatter processed. Alternatively, the worker with worker name="Worker3" may perform ReduceScatter processing, for example, each time it performs backward weight calculation for half of one layer. That is, it may perform ReduceScatter processing each time it calculates gradient information for half of one layer as gradient information that can be ReduceScatter processed.

[0107] Next, a specific example of scheduling the ReduceScatter process and the Allgather process, which are executed in parallel with the backward weight calculation by the worker with the worker name="Worker3", will be described. Fig. 10B is a first diagram showing a specific example of scheduling the ReduceScatter process and the Allgather process.

[0108] As shown in FIG. 10B, the scheduling unit 303 generates the following schedule for the ReduceScatter process and Allgather process after the backward data calculation is completed in the training process using microbatch 3. II)-i)-1: Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker3" acquires each gradient information based on backward data calculation and backward weight calculation in the training process using each microbatch (microbatch 0 to 3). In addition, the worker with worker name = "Worker3" executes ReduceScatter processing for each gradient information acquired by each accelerator (Accelerator 0 to 3). II)-i)-2: The worker with worker name="Worker3" updates the optimizer state of NN3 using each gradient information obtained by executing the ReduceScatter process (an example of "information obtained by executing the ReduceScatter process"). The worker with worker name="Worker3" also converts each weight parameter included in the updated optimizer state. II)-i)-3: The worker with worker name="Worker3" sends each updated weight parameter to the worker with worker name="Worker0". II)-i)-4: The worker with worker name="Worker0" executes Allgather processing for each updated weight parameter. As a result, the worker with worker name="Worker0" collects each updated weight parameter and distributes the collected weight parameters to each accelerator (Accelerator0 to 3) owned by the worker with worker name="Worker0".

[0109] (5-2) Processing by worker with worker name = "Worker2" Next, a description will be given of scheduling of backward weight calculation by a worker with worker name="Worker2". Fig. 11A is a first diagram showing a specific example of scheduling of backward weight calculation by a worker with worker name="Worker2".

[0110] As shown in FIG. 11A, the scheduling unit 303 generates the following schedule for backward weight calculation after the backward data calculation is completed in the training process using micro-batch 3. Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker2" performs backward weight calculations for layer 0 included in NN2 during the training process using each microbatch (microbatch 2 to 3). Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker2" performs backward weight calculations for layer 1 included in NN2 during the training process using each microbatch (microbatch 2 to 3). Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker2" performs backward weight calculations for layer 2 included in NN2 during the training process using each microbatch (microbatch 2 to 3). Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker2" performs backward weight calculations for layer 3 included in NN2 during the training process using each microbatch (microbatch 2 to 3).

[0111] Note that layer0 to layer3 included in NN2 refer to layers within the backward weight calculation. After completing the backward data calculation during training processing using microbatch 3, the worker with worker name="Worker2" performs ReduceScatter processing sequentially each time it calculates gradient information that can be ReduceScatter processed. Specifically, the worker with worker name="Worker2" may perform ReduceScatter processing, for example, each time it completes the backward weight calculation for each layer. That is, it may perform ReduceScatter processing each time it calculates gradient information for one layer as gradient information that can be ReduceScatter processed. Alternatively, the worker with worker name="Worker2" may perform ReduceScatter processing, for example, each time it performs backward weight calculation for half of one layer. That is, it may perform ReduceScatter processing each time it calculates gradient information for half of one layer as gradient information that can be ReduceScatter processed.

[0112] Next, a specific example of scheduling the ReduceScatter process and the Allgather process, which are executed in parallel with the backward weight calculation by the worker with the worker name="Worker2", will be described. Fig. 11B is a second diagram showing a specific example of scheduling the ReduceScatter process and the Allgather process.

[0113] As shown in FIG. 11B, the scheduling unit 303 generates the following schedule for the ReduceScatter process and Allgather process after the backward data calculation is completed in the training process using microbatch 3. II)-ii)-1: Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker2" acquires each gradient information based on backward data calculation and backward weight calculation in the training process using each microbatch (microbatch 2 to 3). In addition, the worker with worker name = "Worker2" executes ReduceScatter processing for each gradient information acquired by each accelerator (Accelerator 0 to 3). II)-ii)-2: The worker with worker name="Worker2" updates the optimizer state of NN2 using each gradient information obtained by executing the ReduceScatter process (an example of "information obtained by executing the ReduceScatter process"). The worker with worker name="Worker2" also converts each weight parameter included in the updated optimizer state. II)-ii)-3: The worker with worker name="Worker2" sends each updated weight parameter to the worker with worker name="Worker1". II)-ii)-4: The worker with worker name="Worker1" executes Allgather processing for each updated weight parameter. As a result, the worker with worker name="Worker1" collects each updated weight parameter and distributes the collected weight parameters to each accelerator (Accelerator0 to 3) owned by the worker with worker name="Worker1".

[0114] (5-3) Processing by the worker with worker name = "Worker1" Next, a description will be given of scheduling of backward weight calculation by a worker with worker name="Worker1". Fig. 12A is a first diagram showing a specific example of scheduling of backward weight calculation by a worker with worker name="Worker1".

[0115] As shown in FIG. 12A, the scheduling unit 303 generates the following schedule for backward weight calculation after the backward data calculation is completed in the training process using micro-batch 3. Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker1" performs backward weight calculations for layer 0 included in NN1 in the training process using a microbatch (microbatch 3). Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker1" performs backward weight calculations for layer 1 included in NN1 during training processing using a microbatch (microbatch 3). Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker1" performs backward weight calculations for layer 2 included in NN1 during training processing using a microbatch (microbatch 3). Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker1" performs backward weight calculations for layer 3 included in NN1 during training processing using a microbatch (microbatch 3).

[0116] Note that layers 0 to 3 included in NN1 refer to layers within the backward weight calculation. After completing the backward data calculation during training processing using microbatch 3, the worker with worker name = "Worker1" performs ReduceScatter processing sequentially each time it calculates gradient information that can be ReduceScatter processed. Specifically, the worker with worker name = "Worker1" may perform ReduceScatter processing, for example, each time it completes the backward weight calculation for each layer. That is, it may perform ReduceScatter processing each time it calculates gradient information for one layer as gradient information that can be ReduceScatter processed. Alternatively, the worker with worker name = "Worker1" may perform ReduceScatter processing, for example, each time it performs backward weight calculation for half of one layer. That is, it may perform ReduceScatter processing each time it calculates gradient information for half of one layer as gradient information that can be ReduceScatter processed.

[0117] Next, a specific example of scheduling the ReduceScatter process and the Allgather process, which are executed in parallel with the backward weight calculation by the worker with the worker name="Worker1", will be described. Fig. 12B is a third diagram showing a specific example of scheduling the ReduceScatter process and the Allgather process.

[0118] As shown in FIG. 12B, the scheduling unit 303 generates the following schedule for the ReduceScatter process and Allgather process after the backward data calculation is completed in the training process using microbatch 3. II)-iii)-1: Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker1" acquires each gradient information based on backward data calculation and backward weight calculation in the training process using a microbatch (microbatch 3). In addition, the worker with worker name = "Worker1" executes ReduceScatter processing on each gradient information acquired by each accelerator (Accelerator 0 to 3). II)-iii)-2: The worker with worker name="Worker1" updates the optimizer state of NN1 using each gradient information obtained by executing the ReduceScatter process (an example of "information obtained by executing the ReduceScatter process"). The worker with worker name="Worker1" also converts each weight parameter included in the updated optimizer state. II)-iii)-3: The worker with worker name="Worker1" sends each updated weight parameter to the worker with worker name="Worker2". II)-iii)-4: The worker with worker name="Worker2" executes Allgather processing for each updated weight parameter. As a result, the worker with worker name="Worker2" collects each updated weight parameter and distributes the collected weight parameters to each accelerator (Accelerator0 to 3) owned by the worker with worker name="Worker2".

[0119] (5-4) Processing by the worker with worker name = "Worker0" Next, a description will be given of scheduling of backward weight calculation by a worker with worker name="Worker0". Fig. 13A is a first diagram showing a specific example of scheduling of backward weight calculation by a worker with worker name="Worker0".

[0120] As shown in FIG. 13A, the scheduling unit 303 generates the following schedule for backward weight calculation after the backward data calculation is completed in the training process using micro-batch 3. Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker0" performs backward weight calculations for layer0 included in NN0 in the training process using a microbatch (microbatch 3). Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker0" performs backward weight calculations for layer 1 included in NN0 in the training process using a microbatch (microbatch 3). Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker0" performs backward weight calculations for layer 2 included in NN0 during training processing using a microbatch (microbatch 3). Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker0" performs backward weight calculations for layer 3 included in NN0 in the training process using a microbatch (microbatch 3).

[0121] Note that layer0 to layer3 included in NN0 refer to layers within the backward weight calculation. After completing the backward data calculation during training processing using microbatch 3, the worker with worker name="Worker0" performs ReduceScatter processing sequentially each time it calculates gradient information that can be ReduceScatter processed. Specifically, the worker with worker name="Worker0" may perform ReduceScatter processing, for example, each time it completes the backward weight calculation for each layer. That is, it may perform ReduceScatter processing each time it calculates gradient information for one layer as gradient information that can be ReduceScatter processed. Alternatively, the worker with worker name="Worker0" may perform ReduceScatter processing, for example, each time it performs backward weight calculation for half of one layer. That is, it may perform ReduceScatter processing each time it calculates gradient information for half of one layer as gradient information that can be ReduceScatter processed.

[0122] Next, a specific example of scheduling the ReduceScatter process and the Allgather process, which are executed in parallel with the backward weight calculation by the worker with the worker name="Worker0", will be described. Fig. 13B is a fourth diagram showing a specific example of scheduling the ReduceScatter process and the Allgather process.

[0123] As shown in FIG. 13B, the scheduling unit 303 generates the following schedule for the ReduceScatter process and Allgather process after the backward data calculation is completed in the training process using microbatch 3. II)-iv)-1: Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker0" acquires each gradient information based on backward data calculation and backward weight calculation in the training process using a microbatch (microbatch 3). In addition, the worker with worker name = "Worker0" executes ReduceScatter processing on each gradient information acquired by each accelerator (Accelerator0 to 3). II)-iv)-2: The worker with worker name="Worker0" updates the optimizer state of NN0 using each gradient information obtained by executing the ReduceScatter process (an example of "information obtained by executing the ReduceScatter process"). The worker with worker name="Worker0" also converts each weight parameter included in the updated optimizer state. II)-iv)-3: The worker with worker name="Worker0" sends each updated weight parameter to the worker with worker name="Worker3". II)-iv)-4: The worker with worker name="Worker3" executes Allgather processing for each updated weight parameter. As a result, the worker with worker name="Worker3" collects each updated weight parameter and distributes the collected weight parameters to each accelerator (Accelerator0 to 3) owned by the worker with worker name="Worker3".

[0124] (6) Details of scheduling process by scheduling unit 303, part 2 Next, details of scheduling by the scheduling unit 303 when executing the ReduceScatter process and the Allgather process in the training process using microbatches 4 to 7 will be described.

[0125] Note that the training process using microbatches 4 to 7 is basically the same as the training process using microbatches 0 to 3. However, as is clear from the above explanation, the worker with worker name="Worker3" holds the updated weight parameters of NN0 through the training process using microbatches 0 to 3. Therefore, when scheduling the training process using microbatches 4 to 7, the scheduling unit 303 schedules the worker with worker name="Worker3" to start the forward calculation first. As a result, in the backward data calculation in the training process using microbatch 7, the worker with worker name="Worker0" finishes first, and the worker with worker name="Worker0" starts the ReduceScatter process first.

[0126] Furthermore, the optimizer state of each worker is an optimizer state that should be updated using each gradient information on which the ReduceScatter process has been executed by the source worker when the worker becomes a destination (see FIG. 5B). Therefore, in the case of training processing using microbatches 4 to 7, the scheduling unit 303 schedules each worker to transmit each gradient information on which the ReduceScatter process has been executed to the destination worker without using it to update the optimizer state.

[0127] (6-1) Processing by the worker with worker name = "Worker0" First, a description will be given of scheduling of backward weight calculation by a worker with worker name="Worker0". Fig. 14A is a second diagram showing a specific example of scheduling of backward weight calculation by a worker with worker name="Worker0".

[0128] As shown in FIG. 14A, the scheduling unit 303 generates the following schedule for backward weight calculation after the backward data calculation is completed in the training process using micro-batch 7. Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker0" performs backward weight calculations for layer 0 included in NN3 during the training process using each microbatch (microbatch 4 to 7). Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker0" performs backward weight calculations for layer 1 included in NN3 during the training process using each microbatch (microbatch 4 to 7). Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker0" performs backward weight calculations for layer 2 included in NN3 during the training process using each microbatch (microbatch 4 to 7). Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker0" performs backward weight calculations for layer 3 included in NN3 during the training process using each microbatch (microbatch 4 to 7).

[0129] Note that layer0 to layer3 included in NN3 refer to layers within the backward weight calculation. After completing backward data calculation in the training process using microbatch 7, the worker with worker name="Worker0" performs ReduceScatter processing sequentially each time it calculates gradient information that can be ReduceScatter processed. Specifically, the worker with worker name="Worker0" may perform ReduceScatter processing, for example, each time it completes backward weight calculation for each layer. That is, it may perform ReduceScatter processing each time it calculates gradient information for one layer as gradient information that can be ReduceScatter processed. Alternatively, the worker with worker name="Worker0" may perform ReduceScatter processing, for example, each time it performs backward weight calculation for half of one layer. That is, it may perform ReduceScatter processing each time it calculates gradient information for half of one layer as gradient information that can be ReduceScatter processed.

[0130] Next, a specific example of scheduling the ReduceScatter process and the Allgather process, which are executed in parallel with the backward weight calculation by the worker with the worker name="Worker0", will be described. Fig. 14B is a fifth diagram showing a specific example of scheduling the ReduceScatter process and the Allgather process.

[0131] As shown in FIG. 14B, the scheduling unit 303 generates the following schedule for the ReduceScatter process and Allgather process after the backward data calculation is completed in the training process using microbatch 7. II)-v)-1: Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker 0" acquires gradient information based on backward data calculation and backward weight calculation in the training process using each microbatch (microbatch 4 to 7). In addition, the worker with worker name = "Worker 0" executes ReduceScatter processing for each gradient information acquired by each accelerator (Accelerator 0 to 3). II)-v)-2: Each accelerator (Accelerator0 to 3) of the worker with worker name="Worker0" transmits each piece of gradient information on which the ReduceScatter process has been executed to the worker with worker name="Worker3". II)-v)-3: The worker with worker name="Worker3" updates the optimizer state of NN3 using the gradient information on which the ReduceScatter process was executed. By updating the optimizer state, various parameters, including each weight parameter, included in the optimizer state are updated. In addition, the worker with worker name="Worker3" converts each weight parameter included in the updated optimizer state. II)-v)-4: The worker with worker name="Worker3" executes Allgather processing for each updated weight parameter. As a result, the worker with worker name="Worker3" collects each updated weight parameter and distributes the collected weight parameters to each accelerator (Accelerator0 to 3) owned by the worker with worker name="Worker3".

[0132] (6-2) Processing by worker with worker name = "Worker1" Next, a description will be given of scheduling of backward weight calculation by a worker with worker name="Worker1". Fig. 15A is a second diagram showing a specific example of scheduling of backward weight calculation by a worker with worker name="Worker1".

[0133] As shown in FIG. 15A, the scheduling unit 303 generates the following schedule for backward weight calculation after the backward data calculation is completed in the training process using micro-batch 7. Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker1" performs backward weight calculations for layer 0 included in NN2 during the training process using each microbatch (microbatch 6, 7). Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker1" performs backward weight calculations for layer 1 included in NN2 during the training process using each microbatch (microbatch 6, 7). Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker1" performs backward weight calculations for layer 2 included in NN2 during the training process using each microbatch (microbatch 6, 7). Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker1" performs backward weight calculations for layer 3 included in NN2 during the training process using each microbatch (microbatch 6, 7).

[0134] Note that layer0 to layer3 included in NN2 refer to layers within the backward weight calculation. After completing the backward data calculation during training processing using microbatch 7, the worker with worker name="Worker1" performs ReduceScatter processing sequentially each time it calculates gradient information that can be ReduceScatter processed. Specifically, the worker with worker name="Worker1" may perform ReduceScatter processing, for example, each time it completes the backward weight calculation for each layer. That is, it may perform ReduceScatter processing each time it calculates gradient information for one layer as gradient information that can be ReduceScatter processed. Alternatively, the worker with worker name="Worker1" may perform ReduceScatter processing, for example, each time it performs backward weight calculation for half of one layer. That is, it may perform ReduceScatter processing each time it calculates gradient information for half of one layer as gradient information that can be ReduceScatter processed.

[0135] Next, a specific example of scheduling the ReduceScatter process and the Allgather process that are executed in parallel with the backward weight calculation by the worker with the worker name="Worker1" will be described. Fig. 15B is a sixth diagram showing a specific example of scheduling the ReduceScatter process and the Allgather process.

[0136] As shown in FIG. 15B, the scheduling unit 303 generates the following schedule for the ReduceScatter process and Allgather process after the backward data calculation is completed in the training process using microbatch 7. II)-vi)-1: Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker1" acquires each gradient information based on backward data calculation and backward weight calculation in the training process using each microbatch (microbatch 6 to 7). In addition, the worker with worker name = "Worker1" executes ReduceScatter processing for each gradient information acquired by each accelerator (Accelerator 0 to 3). II)-vi)-2: Each accelerator (Accelerator0 to 3) of the worker with worker name="Worker1" transmits each piece of gradient information on which the ReduceScatter process has been executed to the worker with worker name="Worker2". II)-vi)-3: The worker with worker name="Worker2" updates the optimizer state of NN2 using the gradient information on which the ReduceScatter process was executed. By updating the optimizer state, various parameters, including each weight parameter, included in the optimizer state are updated. In addition, the worker with worker name="Worker2" converts each weight parameter included in the updated optimizer state. II)-vi)-4: The worker with worker name="Worker2" executes Allgather processing for each updated weight parameter. As a result, the worker with worker name="Worker2" collects each updated weight parameter and distributes the collected weight parameters to each accelerator (Accelerator0 to 3) owned by the worker with worker name="Worker2".

[0137] (6-3) Processing by worker with worker name = "Worker2" Next, a description will be given of scheduling of backward weight calculation by a worker with worker name="Worker2". Fig. 16A is a second diagram showing a specific example of scheduling of backward weight calculation by a worker with worker name="Worker2".

[0138] As shown in FIG. 16A, the scheduling unit 303 generates the following schedule for backward weight calculation after the backward data calculation in the training process using micro-batch 7 is completed. Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker2" performs backward weight calculations for layer 0 included in NN1 in the training process using a microbatch (microbatch 7). Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker2" performs backward weight calculations for layer 1 included in NN1 during training processing using a microbatch (microbatch 7). Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker2" performs backward weight calculations for layer 2 included in NN1 during training processing using micro-batches (micro-batch 7). Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker2" performs backward weight calculations for layer 3 included in NN1 during training processing using micro-batches (micro-batch 7).

[0139] Note that layers 0 to 3 included in NN1 refer to layers within the backward weight calculation. After completing the backward data calculation during training processing using microbatch 7, the worker with worker name = "Worker2" performs ReduceScatter processing sequentially each time it calculates gradient information that can be ReduceScatter processed. Specifically, the worker with worker name = "Worker2" may perform ReduceScatter processing, for example, each time it completes the backward weight calculation for each layer. That is, it may perform ReduceScatter processing each time it calculates gradient information for one layer as gradient information that can be ReduceScatter processed. Alternatively, the worker with worker name = "Worker2" may perform ReduceScatter processing, for example, each time it performs backward weight calculation for half of one layer. That is, it may perform ReduceScatter processing each time it calculates gradient information for half of one layer as gradient information that can be ReduceScatter processed.

[0140] Next, a specific example of scheduling the ReduceScatter process and the Allgather process that are executed in parallel with the backward weight calculation by the worker with the worker name="Worker2" will be described. Fig. 16B is a seventh diagram showing a specific example of scheduling the ReduceScatter process and the Allgather process.

[0141] As shown in FIG. 16B, the scheduling unit 303 generates the following schedule for the ReduceScatter process and Allgather process after the backward data calculation is completed in the training process using microbatch 7. II)-vii)-1: Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker2" acquires each gradient information based on backward data calculation and backward weight calculation in the training process using a microbatch (microbatch 7). In addition, the worker with worker name = "Worker2" executes ReduceScatter processing for each gradient information acquired by each accelerator (Accelerator 0 to 3). II)-vii)-2: Each accelerator (Accelerator0 to 3) of the worker with worker name="Worker2" transmits each piece of gradient information on which the ReduceScatter process has been executed to the worker with worker name="Worker1". II)-vii)-3: The worker with worker name="Worker1" updates the optimizer state of NN1 using the gradient information on which the ReduceScatter process was executed. By updating the optimizer state, various parameters, including each weight parameter, included in the optimizer state are updated. In addition, the worker with worker name="Worker1" converts each weight parameter included in the updated optimizer state. II)-vii)-4: The worker with worker name="Worker1" executes Allgather processing for each updated weight parameter. As a result, the worker with worker name="Worker1" collects each updated weight parameter and distributes the collected weight parameters to each accelerator (Accelerator0 to 3) owned by the worker with worker name="Worker1".

[0142] (6-4) Processing by worker with worker name = "Worker3" Next, a description will be given of scheduling of backward weight calculation by a worker with worker name="Worker3". Fig. 17A is a second diagram showing a specific example of scheduling of backward weight calculation by a worker with worker name="Worker3".

[0143] As shown in FIG. 17A, the scheduling unit 303 generates the following schedule for backward weight calculation after the backward data calculation is completed in the training process using micro-batch 7. Each accelerator (Accelerator0 to 3) of the worker with worker name = "Worker3" performs backward weight calculations for layer0 included in NN0 in the training process using a microbatch (microbatch7). Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker 3" performs backward weight calculations for layer 1 included in NN0 in the training process using a microbatch (microbatch 7). Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker 3" performs backward weight calculations for layer 2 included in NN0 in the training process using a microbatch (microbatch 7). Each accelerator (Accelerator 0 to 3) of the worker with worker name = "Worker 3" performs backward weight calculations for layer 3 included in NN0 in the training process using a microbatch (microbatch 7).

[0144] Note that layer0 to layer3 included in NN0 refer to layers within the backward weight calculation. After completing the backward data calculation during training processing using microbatch 7, the worker with worker name="Worker3" performs ReduceScatter processing sequentially each time it calculates gradient information that can be ReduceScatter processed. Specifically, the worker with worker name="Worker3" may perform ReduceScatter processing, for example, each time it completes the backward weight calculation for each layer. That is, it may perform ReduceScatter processing each time it calculates gradient information for one layer as gradient information that can be ReduceScatter processed. Alternatively, the worker with worker name="Worker3" may perform ReduceScatter processing, for example, each time it performs backward weight calculation for half of one layer. That is, it may perform ReduceScatter processing each time it calculates gradient information for half of one layer as gradient information that can be ReduceScatter processed.

[0145] Next, a specific example of scheduling the ReduceScatter process and the Allgather process that are executed in parallel with the backward weight calculation by the worker with the worker name="Worker3" will be described. Fig. 17B is an eighth diagram showing a specific example of scheduling the ReduceScatter process and the Allgather process.

[0146] As shown in FIG. 17B, the scheduling unit 303 generates the following schedule for the ReduceScatter process and Allgather process after the backward data calculation is completed in the training process using microbatch 7. II)-viii)-1: Each accelerator (Accelerator 0 to 3) of the worker with worker name="Worker3" acquires each gradient information based on the backward data calculation and backward weight calculation of the microbatch (microbatch 7). In addition, the worker with worker name="Worker3" executes ReduceScatter processing for each gradient information acquired by each accelerator (Accelerator 0 to 3). II)-viii)-2: Each accelerator (Accelerator0 to 3) of the worker with worker name="Worker3" transmits each piece of gradient information on which the ReduceScatter process has been executed to the worker with worker name="Worker0". II)-viii)-3: The worker with worker name="Worker0" updates the optimizer state of NN0 using the gradient information on which the ReduceScatter process was executed. By updating the optimizer state, various parameters, including each weight parameter, included in the optimizer state are updated. In addition, the worker with worker name="Worker0" converts each weight parameter included in the updated optimizer state. II)-viii)-4: The worker with worker name="Worker0" executes Allgather processing for each updated weight parameter. As a result, the worker with worker name="Worker0" collects each updated weight parameter and distributes the collected weight parameters to each accelerator (Accelerator0 to 3) owned by the worker with worker name="Worker0".

[0147] <Summary> As is clear from the above description, the information processing system 100 according to the first embodiment: · Execute a neural network training process using at least a first worker (e.g., worker name="Worker0") and a second worker (e.g., worker name="Worker3"). The first worker (e.g., worker name="Worker0") performing forward processing on first data (e.g., data processed by a first worker in a training run using microbatch 0) using first parameters of the neural network to generate a first output; Perform forward processing on second data (e.g., data processed by the first worker in the training process using microbatch 1) using the first parameters to generate second output. The second worker (e.g., worker name="Worker3") performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing forward processing based on the second output using second parameters to generate a fourth output; Generate gradient information of the second parameter based on the third output and the fourth output (for example, generate gradient information of the second parameter by backpropagating the error between the information generated in the forward processing of each worker and the ground truth data), performing a first operation (e.g., a ReduceScatter operation) on the gradient information and updating a second parameter (e.g., a weight parameter); The updated second parameter (for example, the updated weight parameter) is sent to the first worker (for example, worker name="Worker0"). The first worker (e.g., worker name="Worker0") The second process (e.g., Allgather process) is executed using the updated second parameters (e.g., updated weight parameters) received from the second worker (e.g., worker name="Worker3"). However, the first to fourth outputs are information generated by executing the forward process, and may be, for example, intermediate data. Furthermore, the third output and the fourth output may be output data from the final layer.

[0148] Moreover, the information processing system 100 according to the first embodiment · Execute a neural network training process using at least a first worker (e.g., worker name="Worker3") and a second worker (e.g., worker name="Worker0"). The first worker (for example, worker name="Worker3") performing forward processing on first data (e.g., data processed by a first worker in a training run using microbatch 4) using first parameters of the neural network to generate a first output; Perform forward processing on second data (e.g., data processed by the first worker in a training process using microbatch 5) using the first parameters to generate a second output. The second worker (for example, worker name="Worker0") performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing forward processing based on the second output using second parameters to generate a fourth output; Generate gradient information of the second parameter based on the third output and the fourth output (for example, generate gradient information of the second parameter by backpropagating the error between the information generated in the forward processing of each worker and the ground truth data), Execute a first process (e.g., ReduceScatter process) on the gradient information, and send the execution result of the first process to a first worker (e.g., worker name="Worker3"); The first worker (e.g., worker name="Worker3"): Update the first parameter (e.g., weight parameter) using the execution result received from the second worker (e.g., worker name="Worker0"); The second process (e.g., Allgather process) is executed using the updated first parameters (e.g., updated weight parameters). However, the process of sending the execution results of the first process to the first worker includes the following: -Transmitting the gradient information itself, - The process of transmitting compressed gradient information (in this case, the decompression process is performed on the receiving side), A process of transmitting information that can restore or acquire gradient information (in this case, the restoration or acquisition process is performed on the receiving side), Includes:

[0149] In this way, in the information processing system 100 according to the first embodiment, the ReduceScatter process and the Allgather process are executed by different workers. As a result, the information processing system 100 according to the first embodiment makes it possible to effectively utilize the network bandwidth within the workers when executing model training processing by combining data parallelism and pipeline parallelism. As a result, the information processing system 100 according to the first embodiment makes it possible to improve the training speed when executing model training processing by combining data parallelism and pipeline parallelism.

[0150] [Second embodiment] In the first embodiment, each worker has an optimizer state of the corresponding "NN" and updates the optimizer state when updating weight parameters in a training process using each microbatch. In contrast, when a worker does not have an optimizer state of the corresponding "NN," the worker may acquire the optimizer state of the corresponding "NN" from another worker. In this case, each worker may update the weight parameters by updating the optimizer state acquired from another worker in a training process using each microbatch. Alternatively, each worker may instruct another worker to update the weight parameters in a training process using each microbatch and acquire the update results. The other workers referred to here may be workers other than the workers with worker name = Worker0 to worker name = Worker3, or may be one or more of the workers with worker name = Worker0 to worker name = Worker3.

[0151] [Third embodiment] In the above embodiments, the information processing device 120 generates schedules for ReduceScatter processing and Allgather processing. However, the targets for scheduling by the information processing device 120 are not limited to ReduceScatter processing and Allgather processing.

[0152] For example, similar scheduling may be performed for processes similar to ReduceScatter processing and Allgather processing. Here, ReduceScatter processing can be said to be processing in which M pieces of data per node held by each of N nodes (for example, a unit such as one accelerator or one server device) are reduced, and each node holds M / N pieces of the results. Therefore, processing similar to ReduceScatter processing includes processing in which the reduction method or storage method is modified. Specifically, There are N nodes, · Each node has M pieces of data, ·N nodes cooperate to perform calculations and communications to obtain M reduction results. The M reduction results are stored at one of the N nodes. ·There is arbitrariness in the correspondence between the M reduction results and the N nodes. Based on this assumption, for example, the process may be such that M pieces of data are compressed among N nodes, and N / 2 nodes each hold 2M / N pieces of the compressed results, while the remaining N / 2 nodes do not hold the compressed results.

[0153] [Fourth embodiment] In the above embodiments, the case where each worker is made to execute the backward calculation by dividing it into a backward data calculation and a backward weight calculation has been described. However, the scheduling method described in the above embodiments can also be applied to a case where the backward calculation is executed without dividing it into a backward data calculation and a backward weight calculation.

[0154] In the first embodiment, the example of FIG. 7 illustrates a case where a backward calculation identifier is divided into a backward data calculation identifier and a backward weight calculation identifier. However, the division method by the division unit 302 is not limited to this. For example, the division unit 302 may divide the backward calculation identifier into a first calculation identifier indicating a calculation that should be executed before other calculations and a second calculation identifier indicating a calculation that should be executed after the first calculation, among multiple calculations included in the backward calculation. Specifically, when the model to be trained is a Transformer, it is possible to classify the backward weight calculation of a normalization process (such as layer normalization or RMS normalization) as a backward data calculation. This is because memory usage can be reduced by performing the backward weight calculation of the normalization process first.

[0155] In addition, in each of the above embodiments, when assigning each worker with worker name="Worker0" to "Worker3" to each layer of "NN0" to "NN3", - Training process using micro batches 0 to 3, - Training process using microbatches 4 to 7, However, when applying the scheduling method described in each of the above embodiments, it is not necessary to switch the allocation destination between the case of training processing using microbatches 0 to 3 and the case of training processing using microbatches 4 to 7.

[0156] In addition, in the above embodiments, the ReduceScatter process is described as being executed sequentially each time gradient information that can be ReduceScatter processed is calculated. However, the target of sequential processing is not limited to this, and the processing after the ReduceScatter process may also be configured to be processed sequentially. For example, the information processing system 100 may be configured to perform communication between workers each time a portion of the ReduceScatter process is completed. This allows the start timing of the Allgather process to be advanced.

[0157] In addition, in each of the above embodiments, it has been described that communication takes place between a worker with worker name="Worker0" and a worker with worker name="Worker3". It has also been described that communication takes place between a worker with worker name="Worker1" and a worker with worker name="Worker2". However, the combination of workers that communicate is not limited to this, and may be changed by pipeline parallel scheduling.

[0158] In addition, in each of the above embodiments, the case where there is an even number of workers communicating has been described. In contrast to this, when there is an odd number of workers communicating, communication may be performed by a combination of the even number of workers excluding one worker, and the one worker may be scheduled not to be combined with other workers (i.e., not to communicate with other workers).

[0159] Although the above embodiments do not mention details of the method for updating the optimizer state, the method for updating the optimizer state may be, for example, adaptive moment estimation (ADAM). Alternatively, the optimizer state may be updated by a method other than ADAM.

[0160] [Fifth embodiment] In the above embodiments, the scheduling has been described for a case where training processing is performed by combining data parallelism and pipeline parallelism, but the application of the scheduling is not limited to this combination. For example, the scheduling may be applied to a case where training processing is performed by combining sequence parallelism and pipeline parallelism, or to a case where training processing is performed by combining data parallelism, sequence parallelism, and pipeline parallelism.

[0161] [Sixth embodiment] In each of the above embodiments, the case where the training process includes the ReduceScatter process and the Allgather process has been described.

[0162] However, instead of the ReduceScatter process, an intermediate process between the ReduceScatter process and the Reduce process may be included. For example, in the case of a ReduceScatter process of size N (N is an even number), N accelerators perform communication and calculation, and the result is divided into N equal parts and stored. Also, in the case of a Reduce process of size N (N is an even number), N accelerators perform communication and calculation, and only one accelerator stores the result. An example of the intermediate process is a process in which N accelerators perform communication and calculation, the result is divided into N / 2 equal parts, N / 2 accelerators store the parts, and the remaining N / 2 accelerators do not store the results.

[0163] Similarly, instead of Allgather processing, intermediate processing between Allgather processing and Broadcast processing may be included. The intermediate processing here may be, for example, a process in which, from an initial state in which N / 2 accelerators have N / 2 equally divided results, all of the results are transferred to a state in which N accelerators hold all of the results via communication.

[0164] [Seventh embodiment] In the above embodiments, the allgather processing is described as being performed on updated weight parameters. However, in order to speed up the allgather processing, the allgather processing may be performed after compressing the updated weight parameters. On the other hand, when compressing the updated weight parameters, if the compression rate is increased, the accuracy of the weight parameters may be degraded when they are restored. Therefore, instead of compressing the updated weight parameters, the allgather processing may be performed after compressing the difference between the updated weight parameters and the weight parameters before the update.

[0165] Here, each accelerator to which the difference is distributed by the allgather process retains the weight parameters before the update, and even if the accuracy of the distributed difference is degraded, the accuracy of the result of adding up the difference does not necessarily deteriorate. In other words, by performing a compression process on the difference between the updated weight parameters and the weight parameters before the update and then performing allgather processing, it is possible to avoid deterioration of the weight parameters even when the compression rate is increased. The seventh embodiment will be described below, focusing on the differences from the above embodiments.

[0166] <General Allgather processing flow> First, we will explain the general flow of allgather processing. In distributed learning that at least partially employs data parallelism, the following processes are executed serially or in parallel: Multiple nodes (accelerators) included in the worker execute forward and backward calculations. Executes ReduceScatter processing of gradient information among multiple nodes, resulting in distributed calculation of gradient information. · Execute update processing such as Adam using the result of the ReduceScatter processing and the optimizer state. - Perform Allgather processing on the results of the update processing.

[0167] Furthermore, the results of the update process may be compressed before the all-gather process is performed, for example by quantizing them to about 4 bits per element. This is because a higher compression ratio reduces communication traffic, shortening the time required for all-gather processing and ultimately speeding up the entire training process.

[0168] <Flow of Allgather Processing in This Embodiment> Here, it is noted that all nodes (accelerators) participating in the ReduceScatter process and the Allgather process hold the weight parameters before updating. Specifically, each node: - Holds the weight parameters used in forward and backward calculations. The weight parameters are distributed before the first iteration. In the second and subsequent iterations, the weight parameters are updated using the methods of the above embodiments. Note that, in general, the update process does not significantly change the weight parameter values, so the difference between the weight parameters before and after the update is small. Therefore, if allgather processing is performed on the low-precision difference and each node adds the difference resulting from allgather processing to the weight parameters before the update, each node can maintain the precision of the weight parameters after the update. In other words, as a result, it is possible to perform high-precision allgather processing on weight parameters that have been compressed at a high compression rate.

[0169] Thus, according to the seventh embodiment, By performing all-gather processing on the differences before and after the update, all-gather processing of large data can be replaced with all-gather processing of small data. -The amount of communication for allgather processing within each worker can be reduced. - Can improve training speed, It has the following effects.

[0170] <Details of the Allgather Processing Procedure in the Present Embodiment> Next, a detailed description will be given of the procedure for the Allgather process in the seventh embodiment. In the seventh embodiment, the Allgather process is executed according to the following procedure. 1) rank0 updates the weight parameters and the optimizer state using gradient information of the weight parameters of the neural network and the optimizer state. Note that the weight parameters may be included in the optimizer state. However, rank0 contains the optimizer state needed to update the weight parameters. ·Let the updated weight parameter (high precision) be x. ·Let x' be the weight parameter (high precision) before updating. ·The updated weight parameters (low precision) are x_lp. ·The weight parameters (low precision) before updating are defined as x_lp'. 2) Rank0 calculates the difference d between the updated weight parameter and the weight parameter before the update (d=x-x_lp'). 3) Rank0 compresses the difference and calculates the difference (d') after compression (d'=compress(d)). 4) Rank0 performs Allgather processing on the difference (d') after compression and distributes it to rank1 to rank3. 5) For rank1 to rank3, the difference (d') after compression processing is decompressed to obtain the difference (d") (d" = decompress(d')). 6) For rank1 to rank3, the updated weight parameters (x_lp) are calculated using the decompressed difference (d") and the weight parameters (x_lp') before the update (x_lp=x_lp'+d"). However, for rank1 to rank3, the weight parameters (x_lp') before the update are retained until this step is executed. 7) The same process as above is executed in parallel for ranks 1 to 3.

[0171] <Specific example of compression processing> Next, a specific example of compression processing equivalent to the above compress(d) will be described. For example, if x_lp and x_lp' are 16 bits per element, each element is converted to an 8-bit floating point (FP8) by d'=compress(d). As a result, what was 16-bit communication per element in the conventional method becomes 8-bit communication per element. Note that instead of compression processing, low-rank approximation of matrices used in, for example, PowerSGD and GaLore may be used.

[0172] <Variations of the Processing Procedure of Allgather Processing in the Present Embodiment> In the above processing procedure, in 6), x_lp is calculated using only d"; however, in actual training processing, d" does not necessarily change significantly with each iteration. For this reason, the previous d" may be used. Furthermore, the compression processing may be performed on d = x - x'. Furthermore, instead of using the previous d", the weight parameters of the two iterations before may be used.

[0173] In the above description, all updated weight parameters are subjected to difference compression processing and then all-gather processing. However, for some weight parameters, difference compression processing may not be performed and all-gather processing may be performed on the differences. Alternatively, for some weight parameters, differences may not be calculated and all-gather processing may be performed on the weight parameters.

[0174] For example, when the model to be trained is a large-scale language model (LLM), the weight parameters of the linear layer in the Transformer block are dominant, and therefore, there is a significant benefit in performing difference compression processing on these weight parameters. On the other hand, there is not much benefit in performing difference compression processing on weight parameters other than the Transformer block or weight parameters related to normalization in the Transformer block. Therefore, for some of these weight parameters, difference compression processing may not be performed and allgather processing may be performed on the differences. Alternatively, for some of these weight parameters, allgather processing may be performed on the weight parameters without calculating the differences.

[0175] The allgather processing in this embodiment may be applied to the allgather processing in each of the above embodiments. In this case, each rank holds the weight parameters before the update. Alternatively, the weight parameters before the update may be obtained from another worker as needed. When applying the allgather processing in this embodiment to the allgather processing in each of the above embodiments, it may be applied to updating some of the weight parameters among multiple weight parameter updates. Note that the compression processing in this embodiment is not limited to the above embodiments, and may be applied to model training processing using other embodiments.

[0176] [Eighth embodiment] The training process schedules shown in the above embodiments are merely examples, and it goes without saying that other schedules may be generated. For example, when executing forward processing of a neural network, a schedule may be generated in which forward processing by one or more other workers is executed between forward processing by a first worker and forward processing by a second worker.

[0177] Furthermore, the configuration of the neural network shown in each of the above embodiments is merely an example, and other configurations are also possible. For example, in the neural network, other parameters corresponding to other layers may be included before or after the first parameter corresponding to the first layer, or before or after the second parameter corresponding to the second layer.

[0178] Furthermore, the schedule of the training process shown in each of the above embodiments includes various variations.

[0179] For example, "a first worker performing forward processing on first data using first parameters of a neural network" includes performing forward processing using the results of forward processing on the first data performed by another worker.

[0180] For example, "generating first gradient information of the second parameter based on the third output and the fourth output" includes generating first gradient information of the second parameter based on an output from the final layer generated using the third output and an output from the final layer generated using the fourth output.

[0181] In each of the above embodiments, the execution of a predetermined process by a predetermined worker includes, but is not limited to, the execution of a predetermined process by one or more accelerators among a plurality of accelerators included in the predetermined worker. Alternatively, the execution of a predetermined process by a predetermined worker includes, but is not limited to, the execution of a predetermined process by a plurality of accelerators included in the predetermined worker each executing a different process.

[0182] [Other embodiments] In the above embodiments, when generating a schedule for backward weight calculation after completing backward data calculation in the training process, the schedule for backward weight calculation for each of layer 0 to layer 3 included in the NN is shown. Layer 0 to layer 3 included in the NN are examples of layers included in a neural network, and in the case of a model other than a neural network, a schedule for backward weight calculation is generated for each layer included in the model.

[0183] Furthermore, in each of the above embodiments, a schedule has been described in which the allocation of workers to each layer (NN0 to NN3) of the neural network is swapped once. However, the swapping of worker allocation is not limited to once, and a schedule may be generated so that it is repeated. For example, in each of the above embodiments, the allocation of workers is swapped when microbatches 4 to 7 are input, but the allocation of workers may also be swapped when microbatches 8 to 11 are input. Specifically, the allocation may be returned to the allocation when microbatches 0 to 3 were input. In other words, a schedule may be generated in which the allocations are swapped alternately.

[0184] In addition, in each of the above embodiments, a microbatch has been described as an example of the processing unit (batch) of training data executed by each worker during training processing, but the batch of training data is not limited to a microbatch and may be a minibatch. Furthermore, one batch may be one of multiple divisions of training data, or may include one or more data included in the training data.

[0185] In addition, in the above embodiments, when each worker executes backward calculation, the backward calculation is divided into backward data calculation and backward weight calculation and executed. However, in scheduling of some workers (for example, a worker with worker name="Worker0"), the backward data calculation and the backward weight calculation may be scheduled as a single unit without being divided.

[0186] Furthermore, in each of the above embodiments, the case where the information processing device 120 applies the scheduling method to the training process has been described, but the information processing device 120 may also apply the scheduling method to processes other than the training process. In other words, the information processing device 120 may also apply the scheduling method to data other than training data.

[0187] In addition, in each of the above embodiments, the information processing device 120 is provided as a separate entity from the server device group 110. However, the information processing device 120 may be integrated with the server device group 110.

[0188] Specifically, all functions of the information processing device 120 may be implemented by some of the servers in the server device group 110. That is, the information processing system 100 may have N server device groups 110 and one information processing device 120, or (N-1) server device groups 110 and one server device. Alternatively, the information processing device 120 itself may be a worker or a part of a worker.

[0189] Furthermore, in each of the above embodiments, the information processing system 100 has been described as including one information processing device 120, but the information processing device 120 may be configured with a plurality of devices.

[0190] In this specification (including the claims), when the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used, it includes any of a, b, c, ab, ac, bc, or abc. It may also include multiple instances of any element, such as aa, abb, aabbcc, etc. Furthermore, it also includes the addition of elements other than the enumerated elements (a, b, and c), such as having d, as in abcd.

[0191] Furthermore, in this specification (including claims), when expressions such as "using data as input / based on / according to / in response to" (including similar expressions) are used, unless otherwise specified, this includes cases where various data itself is used as input, or where various data that has been processed in some way (e.g., noise-added, normalized, intermediate representation of various data, etc.) is used as input. Furthermore, when it is stated that a result is obtained "based on / according to / in response to data," this includes cases where the result is obtained based solely on the data in question, as well as cases where the result is obtained as a result of being influenced by other data, factors, conditions, and / or states other than the data in question. Furthermore, when it is stated that "data is output," unless otherwise specified, this includes cases where various data itself is used as output, or where various data that has been processed in some way (e.g., noise-added, normalized, intermediate representation of various data, etc.) is output.

[0192] Furthermore, when the terms "connected" and "coupled" are used in this specification (including the claims), they are intended as open-ended terms that include any of direct connection / coupling, indirect connection / coupling, electrically connection / coupling, communicatively connection / coupling, functionally connection / coupling, and physically connection / coupling. These terms should be interpreted appropriately depending on the context in which they are used, but any form of connection / coupling that is not intentionally or naturally excluded should be interpreted as being included in these terms without any restrictions.

[0193] Furthermore, in this specification (including the claims), when the expression "A configured to B" is used, it may include the physical structure of element A having a configuration capable of performing operation B, and the permanent or temporary setting / configuration of element A being configured / set to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and is configured to actually perform operation B by setting a permanent or temporary program (instruction). Furthermore, if element A is a dedicated processor or dedicated arithmetic circuit, it is sufficient that the circuit structure of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0194] Furthermore, when words implying containing or possessing (such as "comprising / including" and "having") are used in this specification (including the claims), they are intended to be open-ended terms that include cases where things other than the object indicated by the object of the term are contained or possessed. When the object of such words implying containing or possessing does not specify a quantity or suggests a singular number (such as an expression using the article "a" or "an"), the expression should be construed as not being limited to a specific number.

[0195] Furthermore, in this specification (including the claims), even if expressions such as "one or more" or "at least one" are used in some places and expressions that do not specify a quantity or suggest a singular number (expressions using the articles "a" or "an") are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or suggest a singular number (expressions using the articles "a" or "an") should be interpreted as not necessarily being limited to a specific number.

[0196] Furthermore, if a particular advantage / result is described in this specification as being obtained from a particular configuration of an embodiment, it should be understood that the same advantage / result can also be obtained from one or more other embodiments having the same configuration, unless otherwise stated. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or states, and that the effect is not necessarily obtained by the configuration. The effect is merely obtained by the configuration described in the embodiment when various factors, conditions, and / or states are satisfied, and the effect does not necessarily occur in a claimed invention that defines the same or a similar configuration.

[0197] Furthermore, in this specification (including claims), when multiple pieces of hardware perform a predetermined process, the pieces of hardware may cooperate to perform the predetermined process, or some of the hardware may perform all of the predetermined process. Furthermore, some of the hardware may perform part of the predetermined process, and other hardware may perform the rest of the predetermined process. In this specification (including claims), when an expression such as "one or more pieces of hardware perform a first process, and the one or more pieces of hardware perform a second process" is used, the hardware performing the first process and the hardware performing the second process may be the same or different. In other words, it is sufficient that the hardware performing the first process and the hardware performing the second process are included in the one or more pieces of hardware. Note that hardware may include an electronic circuit, a device including an electronic circuit, or the like.

[0198] Furthermore, in this specification (including the claims), when multiple storage devices (memories) store data, each of the multiple storage devices (memories) may store only a portion of the data, or may store the entire data.

[0199] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, partial deletions, etc. are possible within the scope of the conceptual idea and spirit of the present invention derived from the content defined in the claims and their equivalents. For example, in all of the above-described embodiments, the numerical values ​​used in the explanations are shown as examples and are not limited to these. Furthermore, the order of each operation in the embodiments is shown as an example and is not limited to these.

[0200] The disclosed technology may take the following forms as described below. (Appendix 1) An information processing system that performs a neural network training process using at least a first worker and a second worker, The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing forward processing on second data using the first parameters to generate second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating first gradient information of the second parameter based on the third output and the fourth output; performing a first process on the first gradient information; updating the second parameter based on the execution result of the first process; Sending the updated second parameters to the first worker; The first worker: Executing a second process using the updated second parameters received from the second worker. Information processing system. (Appendix 2) An information processing system that performs a neural network training process using at least a first worker and a second worker, The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing forward processing on second data using the first parameters to generate second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating first gradient information of the second parameter based on the third output and the fourth output; performing a first process on the first gradient information; Sending an execution result of the first process to the first worker; The first worker: updating the first parameter using the execution result received from the second worker; executing a second process using the updated first parameters; Information processing system. (Appendix 3) The first process is a process of collecting the first gradient information within the second worker. 3. The information processing system according to claim 1 or 2. (Appendix 4) The first process is a ReduceScatter process of the first gradient information in the second worker. 3. The information processing system according to claim 1 or 2. (Appendix 5) The first processing is processing in which data included in the first gradient information, the number of which corresponds to a predetermined number of data per node, is reduced among nodes included in the second worker, and the reduction result is held in each node included in the second worker. 3. The information processing system according to claim 1 or 2. (Appendix 6) The second processing is a processing of distributing the updated second parameters among the first workers. 10. The information processing system of claim 1. (Appendix 7) The second processing is an allgather processing of the updated second parameters in the first worker. 10. The information processing system of claim 1. (Appendix 8) The second process is a process of distributing the updated first parameters among the first workers. 10. The information processing system of claim 2. (Appendix 9) The second processing is an allgather processing of the first parameters after updating in the first worker. 10. The information processing system of claim 2. (Appendix 10) The second worker: performing forward processing on third data using the updated first parameters of the neural network to generate a fifth output; performing a forward process on the fourth data using the updated first parameters to generate a sixth output; The first worker: generating a seventh output by performing forward processing based on the fifth output using the updated second parameters of the neural network; performing a forward process based on the sixth output using the updated second parameters to generate an eighth output; generating second gradient information of the updated second parameter based on the seventh output and the eighth output; performing a first process on the second gradient information; Sending the execution result of the first process to the second worker; The second worker: further updating the updated second parameters using the execution result of the first process received from the first worker; and executing a second process using the updated second parameters. 10. The information processing system of claim 1. (Appendix 11) The second worker: performing forward processing on third data using the updated first parameters of the neural network to generate a fifth output; performing a forward process on the fourth data using the updated first parameters to generate a sixth output; The first worker: performing forward processing based on the third output using the updated second parameters of the neural network to generate a seventh output; generating an eighth output by performing a forward process based on the sixth output using the updated second parameters; generating second gradient information of the updated second parameter based on the seventh output and the eighth output; performing a first process on the second gradient information; Further updating the updated second parameter based on the execution result of the first process; further transmitting the updated second parameters to the second worker; The second worker: executing a second process using the further updated second parameters received from the first worker; 10. The information processing system of claim 2. (Appendix 12) The second worker: performing the first processing on a portion of the first gradient information before a backward calculation based on the third output and the fourth output is completed in the second worker; 3. The information processing system according to claim 1 or 2. (Appendix 13) The second worker: updating a portion of the second parameters based on a result of executing the first processing on a portion of the first gradient information before the backward calculation based on the third output and the fourth output is completed in the second worker, and transmitting the updated portion of the second parameters to the first worker. 10. The information processing system of claim 1. (Appendix 14) The second worker: sending a result of executing the first processing on a portion of the first gradient information to the first worker before a backward calculation based on the third output and the fourth output is completed in the second worker; 10. The information processing system of claim 2. (Appendix 15) The first worker: executing the second process using difference information between the second parameter before the update and the second parameter after the update; 10. The information processing system of claim 1. (Appendix 16) The first worker: generating the difference information by compressing a difference between the second parameter before updating and the second parameter after updating; 16. The information processing system of claim 15. (Appendix 17) The first worker: calculating the updated second parameter using the difference information after the second process has been executed and the second parameter before the update; 17. The information processing system of claim 16. (Appendix 18) The second process is a process of distributing the difference information among the first workers. 17. The information processing system of claim 16. (Appendix 19) An information processing device that schedules execution of a neural network training process using at least a first worker and a second worker, The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing forward processing on second data using the first parameters to generate second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating gradient information of the second parameter based on the third output and the fourth output; performing a first process on the gradient information; updating the second parameter based on the execution result of the first process; Sending the updated second parameters to the first worker; The first worker: Executing a second process using the updated second parameters received from the second worker. Schedule accordingly. Information processing device. (Appendix 20) An information processing device that schedules execution of a neural network training process using at least a first worker and a second worker, The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing forward processing on second data using the first parameters to generate second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating gradient information of the second parameter based on the third output and the fourth output; performing a first process on the gradient information; Sending an execution result of the first process to the first worker; The first worker: updating the first parameter using the execution result received from the second worker; executing a second process using the updated first parameters; Schedule accordingly. Information processing device. (Appendix 21) An information processing method for performing a neural network training process using at least a first worker and a second worker, The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing forward processing on second data using the first parameters to generate second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating gradient information of the second parameter based on the third output and the fourth output; performing a first process on the gradient information; updating the second parameter based on the execution result of the first process; Sending the updated second parameters to the first worker; The first worker: Executing a second process using the updated second parameters received from the second worker. Information processing methods. (Appendix 22) An information processing method for performing a neural network training process using at least a first worker and a second worker, The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing forward processing on second data using the first parameters to generate second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating gradient information of the second parameter based on the third output and the fourth output; performing a first process on the gradient information; Sending an execution result of the first process to the first worker; The first worker: updating the first parameter using the execution result received from the second worker; executing a second process using the updated first parameters; Information processing methods. (Appendix 23) 1. A scheduling method for scheduling execution of a neural network training process using at least a first worker and a second worker, the method comprising: The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing forward processing on second data using the first parameters to generate second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating gradient information of the second parameter based on the third output and the fourth output; performing a first process on the gradient information; updating the second parameter based on the execution result of the first process; Sending the updated second parameters to the first worker; The first worker: Executing a second process using the updated second parameters received from the second worker. The processor schedules it as follows: Scheduling methods. (Appendix 24) 1. A scheduling method for scheduling execution of a neural network training process using at least a first worker and a second worker, the method comprising: The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing forward processing on second data using the first parameters to generate second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating gradient information of the second parameter based on the third output and the fourth output; performing a first process on the gradient information; Sending an execution result of the first process to the first worker; The first worker: updating the first parameter using the execution result received from the second worker; executing a second process using the updated first parameters; The processor schedules it as follows: Scheduling methods. (Appendix 25) A processor of an information processing device that schedules a neural network training process using at least a first worker and a second worker, The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing forward processing on second data using the first parameters to generate second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating gradient information of the second parameter based on the third output and the fourth output; performing a first process on the gradient information; updating the second parameter based on the execution result of the first process; Sending the updated second parameters to the first worker; The first worker: Executing a second process using the updated second parameters received from the second worker. A scheduling program that allows you to schedule tasks. (Appendix 26) A processor of an information processing device that schedules a neural network training process using at least a first worker and a second worker, The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing forward processing on second data using the first parameters to generate second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating gradient information of the second parameter based on the third output and the fourth output; performing a first process on the gradient information; Sending an execution result of the first process to the first worker; The first worker: updating the first parameter using the execution result received from the second worker; executing a second process using the updated first parameters; A scheduling program that allows you to schedule tasks.

Claims

1. An information processing system that performs a neural network training process using at least a first worker and a second worker, The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing a forward process on second data using the first parameters to generate a second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating first gradient information of the second parameter based on the third output and the fourth output; performing a first process on the first gradient information; updating the second parameter based on the execution result of the first process; Sending the updated second parameters to the first worker; The first worker: Executing a second process using the updated second parameters received from the second worker. Information processing system.

2. An information processing system that performs a neural network training process using at least a first worker and a second worker, The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing a forward process on second data using the first parameters to generate a second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating first gradient information of the second parameter based on the third output and the fourth output; performing a first process on the first gradient information; Sending an execution result of the first process to the first worker; The first worker: updating the first parameter using the execution result received from the second worker; executing a second process using the updated first parameters; Information processing system.

3. The first process is a process of collecting the first gradient information within the second worker.

3. The information processing system according to claim 1.

4. The first process is a ReduceScatter process of the first gradient information in the second worker.

3. The information processing system according to claim 1.

5. The first processing is processing in which data included in the first gradient information, the number of which corresponds to a predetermined number of data per node, is contracted between nodes included in the second worker, and the contraction result is held in each node included in the second worker.

3. The information processing system according to claim 1.

6. The second process is a process of distributing the updated second parameters among the first workers. The information processing system according to claim 1 .

7. The second process is an Allgather process of the updated second parameters in the first worker. The information processing system according to claim 1 .

8. The second process is a process of distributing the updated first parameters among the first workers. The information processing system according to claim 2 .

9. The second process is an Allgather process of the updated first parameters in the first worker. The information processing system according to claim 2 .

10. The second worker: performing forward processing on third data using the updated first parameters of the neural network to generate a fifth output; performing a forward process on the fourth data using the updated first parameters to generate a sixth output; The first worker: generating a seventh output by performing forward processing based on the fifth output using the updated second parameters of the neural network; performing a forward process based on the sixth output using the updated second parameters to generate an eighth output; generating second gradient information of the updated second parameter based on the seventh output and the eighth output; performing a first process on the second gradient information; Sending an execution result of the first process to the second worker; The second worker: further updating the updated second parameters using the execution result of the first process received from the first worker; and executing a second process using the updated second parameters. The information processing system according to claim 1 .

11. The second worker: performing forward processing on third data using the updated first parameters of the neural network to generate a fifth output; performing a forward process on the fourth data using the updated first parameters to generate a sixth output; The first worker: performing forward processing based on the third output using the updated second parameters of the neural network to generate a seventh output; generating an eighth output by performing a forward process based on the sixth output using the updated second parameters; generating second gradient information of the updated second parameter based on the seventh output and the eighth output; performing a first process on the second gradient information; Further updating the updated second parameter based on the execution result of the first process; further transmitting the updated second parameters to the second worker; The second worker: executing a second process using the further updated second parameters received from the first worker; The information processing system according to claim 2 .

12. The second worker: performing the first processing on a portion of the first gradient information before a backward calculation based on the third output and the fourth output is completed in the second worker; 3. The information processing system according to claim 1.

13. The second worker: updating a portion of the second parameters based on a result of executing the first processing on a portion of the first gradient information before the backward calculation based on the third output and the fourth output is completed in the second worker, and transmitting the updated portion of the second parameters to the first worker. The information processing system according to claim 1 .

14. The second worker: sending a result of executing the first processing on a portion of the first gradient information to the first worker before a backward calculation based on the third output and the fourth output is completed in the second worker; The information processing system according to claim 2 .

15. The first worker: executing the second process using difference information between the second parameter before the update and the second parameter after the update; The information processing system according to claim 1 .

16. The first worker: generating the difference information by compressing a difference between the second parameter before updating and the second parameter after updating; 16. The information processing system according to claim 15.

17. The first worker: calculating the updated second parameter using the difference information after the second process has been executed and the second parameter before the update; 17. The information processing system according to claim 16.

18. The second process is a process of distributing the difference information among the first workers.

17. The information processing system according to claim 16.

19. An information processing device that schedules execution of a neural network training process using at least a first worker and a second worker, The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing a forward process on second data using the first parameters to generate a second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating gradient information of the second parameter based on the third output and the fourth output; performing a first process on the gradient information; updating the second parameter based on the execution result of the first process; Sending the updated second parameters to the first worker; The first worker: Executing a second process using the updated second parameters received from the second worker. Schedule accordingly. Information processing device.

20. An information processing device that schedules execution of a neural network training process using at least a first worker and a second worker, The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing a forward process on second data using the first parameters to generate a second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating gradient information of the second parameter based on the third output and the fourth output; performing a first process on the gradient information; Sending an execution result of the first process to the first worker; The first worker: updating the first parameter using the execution result received from the second worker; executing a second process using the updated first parameters; Schedule accordingly. Information processing device.

21. An information processing method for performing a neural network training process using at least a first worker and a second worker, comprising: The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing a forward process on second data using the first parameters to generate a second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating gradient information of the second parameter based on the third output and the fourth output; performing a first process on the gradient information; updating the second parameter based on the execution result of the first process; Sending the updated second parameters to the first worker; The first worker: Executing a second process using the updated second parameters received from the second worker. Information processing methods.

22. An information processing method for performing a neural network training process using at least a first worker and a second worker, comprising: The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing a forward process on second data using the first parameters to generate a second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating gradient information of the second parameter based on the third output and the fourth output; performing a first process on the gradient information; Sending an execution result of the first process to the first worker; The first worker: updating the first parameter using the execution result received from the second worker; executing a second process using the updated first parameters; Information processing methods.

23. 1. A scheduling method for scheduling a neural network training process using at least a first worker and a second worker, the method comprising: The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing a forward process on second data using the first parameters to generate a second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating gradient information of the second parameter based on the third output and the fourth output; performing a first process on the gradient information; updating the second parameter based on the execution result of the first process; Sending the updated second parameters to the first worker; The first worker: Executing a second process using the updated second parameters received from the second worker. The processor schedules it as follows: Scheduling methods.

24. 1. A scheduling method for scheduling a neural network training process using at least a first worker and a second worker, the method comprising: The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing a forward process on second data using the first parameters to generate a second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating gradient information of the second parameter based on the third output and the fourth output; performing a first process on the gradient information; Sending an execution result of the first process to the first worker; The first worker: updating the first parameter using the execution result received from the second worker; executing a second process using the updated first parameters; The processor schedules it as follows: Scheduling methods.

25. A processor of an information processing device that schedules execution of a neural network training process using at least a first worker and a second worker, The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing a forward process on second data using the first parameters to generate a second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating gradient information of the second parameter based on the third output and the fourth output; performing a first process on the gradient information; updating the second parameter based on the execution result of the first process; Sending the updated second parameters to the first worker; The first worker: Executing a second process using the updated second parameters received from the second worker. A scheduling program that allows you to schedule tasks.

26. A processor of an information processing device that schedules execution of a neural network training process using at least a first worker and a second worker, The first worker: performing forward processing on first data using first parameters of the neural network to generate a first output; performing a forward process on second data using the first parameters to generate a second output; The second worker: performing forward processing based on the first output using second parameters of the neural network to generate a third output; performing a forward process based on the second output using the second parameters to generate a fourth output; generating gradient information of the second parameter based on the third output and the fourth output; performing a first process on the gradient information; Sending an execution result of the first process to the first worker; The first worker: updating the first parameter using the execution result received from the second worker; executing a second process using the updated first parameters; A scheduling program that allows you to schedule tasks.

Images