Information processing system, information processing device, information processing method, scheduling method, information processing program, and scheduling program

Abstract

Improves execution speed in parallel processing. [Solution] An information processing system comprising multiple memories and multiple processors, which executes parallel processing using a model, wherein the multiple processors execute communication processing resulting from calculation processing using at least a part of the model for first input data, and calculation processing using at least a part of the model for second input data, with at least a part of their respective processing times overlapping.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system, an information processing device, an information processing method, a scheduling method, an information processing program, and a scheduling program. [Background technology]

[0002] When training large-scale models using large amounts of data, intra-layer parallelism, such as tensor parallelism, is known as a method for improving training speed. However, to further improve execution speed, scheduling that enables further parallelization is required. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Ptrick LeGresley, Jared Casper, Bryan Catanzaro "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism", arXiv:1909.08053v4 [cs.CL], 13 Mar 2020, [Search date 2024.4.5], Internet,<URL:https: / / arxiv.org / abs / 1909.08053> Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure improves execution speed in parallel processing. [Means for solving the problem]

[0005] An information processing system according to an aspect of the present disclosure has, for example, the following configuration: An information processing system comprising a plurality of memories and a plurality of processors, and performing parallel processing using a model, The plurality of processors A communication process resulting from a calculation process using at least a part of the model for first input data and a calculation process using at least a part of the model for second input data are executed with at least a part of their respective processing times overlapping. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 illustrates an example of a system configuration of an information processing system. [Figure 2] FIG. 2 illustrates an example of a hardware configuration of an information processing device. [Figure 3] FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing apparatus. [Figure 4] FIG. 2 is a diagram illustrating an overview of scheduling by the information processing device according to the first embodiment. [Figure 5] FIG. 10 is a first diagram showing a specific example of scheduling by a scheduling unit of the comparative example. [Figure 6] FIG. 2 is a first diagram showing a specific example of scheduling by a scheduling unit of the information processing device according to the first embodiment. [Figure 7A] FIG. 2 is a second diagram showing a specific example of scheduling by the scheduling unit of the information processing device according to the first embodiment. [Figure 7B] FIG. 10 is a third diagram showing a specific example of scheduling by the scheduling unit of the information processing device according to the first embodiment. [Figure 7C] FIG. 4 is a fourth diagram showing a specific example of scheduling by the scheduling unit of the information processing device according to the first embodiment. [Figure 8] FIG. 10 is a diagram illustrating an overview of scheduling by an information processing device according to a second embodiment. [Figure 9A]FIG. 11 is a first diagram showing a specific example of scheduling by a scheduling unit of an information processing device according to a second embodiment. [Figure 9B] FIG. 10 is a second diagram showing a specific example of scheduling by the scheduling unit of the information processing device according to the second embodiment. [Figure 10] FIG. 11 is a diagram illustrating an overview of scheduling by an information processing device according to a third embodiment. [Figure 11] FIG. 10 is a second diagram showing a specific example of scheduling by the scheduling unit of the comparative example. [Figure 12A] FIG. 11 is a first diagram showing a specific example of scheduling by a scheduling unit of an information processing device according to a third embodiment. [Figure 12B] FIG. 11 is a second diagram showing a specific example of scheduling by the scheduling unit of the information processing device according to the third embodiment. [Figure 13] FIG. 10 is a diagram illustrating an overview of scheduling by an information processing device according to a fourth embodiment. [Figure 14A] FIG. 11 is a first diagram showing a specific example of scheduling by a scheduling unit of an information processing device according to a fourth embodiment. [Figure 14B] FIG. 20 is a second diagram showing a specific example of scheduling by the scheduling unit of the information processing device according to the fourth embodiment. [Figure 15] FIG. 13 is a diagram illustrating an outline of scheduling by an information processing device according to a sixth embodiment. [Figure 16] FIG. 13 is a diagram showing a specific example of scheduling by a scheduling unit of an information processing device according to the seventh embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0008] [First embodiment] <System configuration of information processing system> First, a system configuration of an information processing system according to the first embodiment will be described. Fig. 1 is a diagram showing an example of the system configuration of the information processing system. As shown in Fig. 1, the information processing system 100 according to the first embodiment has a plurality of server devices (a server device group 110) and an information processing device 120.

[0009] The server device group 110 executes training processing for a model to be trained (for example, a neural network, but is not limited to a neural network and may be a model other than a neural network). The training processing by the server device group 110 is executed based on a schedule (a schedule for training processing by intra-layer parallelism) generated by the information processing device 120.

[0010] The information processing device 120 generates a schedule for each worker to efficiently execute training processing using intra-layer parallelism for a model to be trained. In this embodiment, the term "worker" refers to a plurality of server devices included in the server device group 110. In other words, one worker includes a plurality of server devices.

[0011] However, the definition of a worker is not limited to this, and a worker may refer to one or more server devices included in the server device group 110. Also, one worker may be one or more server devices, or one or more information processing devices. In more general terms, one worker may be one device or a group of devices identified as a destination to which a schedule is assigned.

[0012] Alternatively, a worker may refer to multiple accelerators included in one server device. In other words, one worker may include multiple accelerators. Alternatively, a worker may refer to one accelerator included in one server. In other words, one worker may equal one accelerator. Note that although an accelerator is given as an example here, an accelerator may also be read as a GPU (Graphics Processing Unit). Alternatively, an accelerator may also be read as a processor. In more general terms, one worker may be one component or a group of components identified as a destination for schedule allocation.

[0013] In this embodiment, during the training process, the processing that each worker performs on each micro-batch of training data includes forward calculations and backward calculations.

[0014] That is, in this embodiment, the information processing device 120: Accepts input of information for intra-tier scheduling, Generate forward and backward calculation schedules.

[0015] The information processing device 120 may store, as information for intra-layer scheduling, for example: Configuration information indicating the configuration of the model being trained; The number of microbatches of training data used in the training process, The execution order of each micro-batch when performing the training process. · The memory capacity of each worker used for training processing, Accepts input such as:

[0016] The information processing device 120 transmits the generated schedule to the server device group 110. As a result, an information processing program for executing training processing based on the schedule generated by the information processing device 120 is stored in the server device group 110. As a result, each worker in the server device group 110 can execute training processing based on the generated schedule.

[0017] As an example of the training process executed by each worker in the server device group 110, for example, when the model to be trained is a neural network (NN), · Worker name = Worker0: 1st layer of NN, · Worker name = Worker1: Second layer of NN, ··· For example, each worker may perform a training process for a corresponding layer, such as: Worker name = Worker0:NN, from the 1st layer to the Nth layer, Worker name = Worker1: NN (N+1)th layer to 2Nth layer, For example, each worker may perform training for multiple layers, such as in the following example. In other words, the NN may be divided as evenly as possible, and each worker may be responsible for training the layers, starting from the layer closest to the input.

[0018] However, if the number of layers in the NN is not divisible by the number of workers, the number of layers that some workers are responsible for when performing the training process may be less than the number of layers that other workers are responsible for when performing the training process.Alternatively, if special calculations are included in the layers around the input and the layers around the output, the calculation load may become unbalanced among the workers.

[0019] <Hardware configuration of server device and information processing device> Next, a description will be given of the hardware configuration of each server device included in the server device group 110 and the hardware configuration of the information processing device 120. Note that each server device included in the server device group 110 and the information processing device 120 have similar hardware configurations, and therefore the hardware configuration of the information processing device 120 will be described here.

[0020] Fig. 2 is a diagram showing an example of the hardware configuration of an information processing device. The information processing device 120 has, as components, a processor 201, a main storage device 202 (memory), an auxiliary storage device 203 (memory), a network interface 204, and a device interface 205. The information processing device 120 may be realized as a computer in which these components are connected via a bus 206. Note that, in the example of Fig. 2, the information processing device 120 is shown as having one of each component, but the information processing device 120 may have multiple of the same component.

[0021] Various calculations of the information processing device 120 may be executed in parallel using one or more processors. Furthermore, various calculations may be distributed to multiple processing cores in the processor 201 and executed in parallel. Furthermore, some or all of the processes, means, etc. disclosed herein may be executed by an external device 230 (at least one of a processor and a storage device) provided on a cloud that can communicate with the information processing device 120 via the network interface 204.

[0022] The processor 201 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.). The processor 201 may also be a semiconductor device including a dedicated processing circuit. The processor 201 is not limited to an electronic circuit using electronic logic elements, and may be realized by an optical circuit using optical logic elements. The processor 201 may also include an arithmetic function based on quantum computing.

[0023] The processor 201 performs various calculations based on various data and commands input from each device, etc., in the internal configuration of the information processing device 120, and outputs the calculation results and control signals to each device, etc. The processor 201 controls each component included in the information processing device 120 by executing an OS (Operating System), applications, etc.

[0024] Furthermore, processor 201 may refer to one or more electronic circuits arranged on a single chip, or to one or more electronic circuits arranged on two or more chips or devices. When multiple electronic circuits are used, the electronic circuits may communicate with each other via wires or wirelessly.

[0025] The main memory device 202 is a memory device that stores instructions executed by the processor 201 and various data, and the various data stored in the main memory device 202 is read by the processor 201. The auxiliary memory device 203 is a memory device other than the main memory device 202. Note that these memory devices refer to any electronic component that can store various data, and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. The memory device for saving various data in the information processing device 120 may be realized by the main memory device 202 or the auxiliary memory device 203, or may be realized by an internal memory built into the processor 201.

[0026] Furthermore, multiple processors 201 may be connected (coupled) to one main memory device 202, or a single processor 201 may be connected. Alternatively, multiple main memory devices 202 may be connected (coupled) to one processor 201. When the information processing device 120 is configured with at least one main memory device 202 and multiple processors 201 connected (coupled) to this at least one main memory device 202, it may include a configuration in which at least one processor of the multiple processors 201 is connected (coupled) to at least one main memory device 202.

[0027] The network interface 204 is an interface for connecting to a communication network 220 wirelessly or via a wired connection.

[0028] The device interface 205 is an interface such as a USB that directly connects to an external device 240 .

[0029] The external device 240 may be, for example, an input device. In this embodiment, the input device is, for example, an electronic device such as a camera, a microphone, various sensors, a keyboard, a mouse, or a touch panel, and provides acquired information to the information processing device 120.

[0030] Furthermore, the external device 240 may be, for example, an output device. In this embodiment, the output device may be, for example, a display device such as an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube), a PDP (Plasma Display Panel), or an organic EL (Electro Luminescence) panel, or may be a speaker that outputs sound or the like.

[0031] The external device 240 may also be a storage device (memory). For example, the external device 240 may be a network storage or the like, or may be a storage such as an HDD.

[0032] Furthermore, the external device 240 may be a device that has some of the functions of the components of the information processing device 120. In other words, the information processing device 120 may transmit and receive processing results to and from the external device 240.

[0033] <Functional configuration of information processing device> Next, a description will be given of the functional configuration of the information processing device 120. Fig. 3 is a diagram showing an example of the functional configuration of the information processing device. A scheduling program is installed in the information processing device 120, and by executing this program, the information processing device 120 functions as an input unit 301, a scheduling unit 302, and a transmission unit 303.

[0034] The input unit 301 accepts input of information for intra-layer scheduling. Details of the information for intra-layer scheduling accepted as input by the input unit 301 have already been explained using Fig. 1, and therefore will not be explained here. The input unit 301 notifies the scheduling unit 302 of the information for intra-layer scheduling that has been accepted as input.

[0035] The scheduling unit 302 schedules the execution procedures of forward calculations and backward calculations based on the information for intra-layer scheduling notified from the input unit 301. At this time, the scheduling unit 302 schedules to execute forward calculations and backward calculations using intra-layer parallelism (tensor parallelism in the first embodiment). Note that tensor parallelism is distributed parallel processing of matrix multiplication within a layer. When scheduling, the scheduling unit 302 also schedules multiple sets of input data input to each accelerator of each worker as a group. Specifically, the scheduling unit 302 schedules, in each accelerator, -communication processing between accelerators after calculation processing in the accelerator of the first input data; In the training process, a calculation process is performed in the accelerator for second input data that is successive in processing order to the first input data; are scheduled to be executed in parallel. Here, "parallel" means that at least a portion of the processing time for the communication processing and a portion of the processing time for the calculation processing overlap. For example, the communication processing may be started when at least a portion of the calculation processing has been calculated. Also, "input data" here refers to micro-batch data or processed data obtained by processing the micro-batch, and refers to data input to the accelerator (data based on the micro-batch). Also, the first input data and the second input data may be at least partially the same data (or may be entirely the same data).

[0036] The transmission unit 303 transmits the schedule generated by the scheduling unit 302 to the server device group 110 .

[0037] <Specific examples of scheduling> Next, a specific example of scheduling by scheduling section 302 will be described.

[0038] (1) Overview of intra-layer scheduling First, an outline of intra-layer scheduling performed by the scheduling unit 302 will be described. Fig. 4 is a diagram showing an outline of scheduling by the information processing device according to the first embodiment.

[0039] As shown in FIG. 4, in the first embodiment, the model to be trained has four layers, "NN0" to "NN3". The example in FIG. 4 shows a situation in which a worker with worker name="Worker0" is assigned to the training process for the "NN0" layer, and a worker with worker name="Worker1" is assigned to the training process for the "NN1" layer. The example in FIG. 4 also shows a situation in which a worker with worker name="Woker2" is assigned to the training process for the "NN2" layer, and a worker with worker name="Worker3" is assigned to the training process for the "NN3" layer.

[0040] In the example of Figure 4, each worker is one server device (for example, Server 0) and has two accelerators (Accelerator 1 and 2). In the training process for the "NN0" layer, data based on microbatch 1 and data based on microbatch 2 are input as input data to the "NN0" layer, and the two accelerators perform the following calculation process as forward calculation. y=f(x@W1) z=y@W2 where: x: input data, W1, W2: Weight parameters of the "NN0" layer, @: matrix multiplication operator, f: a given activation function, where x is an L×C matrix, W1 is a C×4C matrix, W2 is a 4C×C matrix, y is an L×4C matrix, and z is an L×C matrix.

[0041] The scheduling unit 302 generates a schedule so that the accelerators (Accelerator 1, 2) execute forward calculations using tensor parallelism, using data based on microbatches 1 and 2 as input data. Note that, although the example in Fig. 4 shows a case where a schedule is generated within the "NN0" layer, the same applies to generating schedules within the "NN1" to "NN3" layers.

[0042] The schedule generated in this manner is transmitted to the server device group 110 and distributed to each worker, as described above. The method of distribution to each worker is arbitrary; for example, if a schedule is generated on a server device other than the workers, the other server device distributes the schedule to each worker. Also, if a schedule is generated on one of the workers, the one worker distributes it to the other workers. Also, if the same schedule is generated on each worker, each worker extracts a corresponding portion of the schedule.

[0043] (2) Scheduling by the scheduling unit of the comparative example Next, a schedule generated by a scheduling unit of a comparative example will be described based on the overview of intra-layer scheduling shown in Fig. 4. The scheduling unit of a comparative example is a general term for a general scheduling unit that generates a schedule so as to execute forward calculations using tensor parallelism. To clarify the difference between the schedule generated by the scheduling unit 302 of the information processing device 120 and a general schedule, scheduling by the scheduling unit of a comparative example will be described first. Fig. 5 is a first diagram showing a specific example of scheduling by the scheduling unit of a comparative example.

[0044] As indicated by reference numeral 510, in order to execute tensor parallelism, the scheduling unit of the comparative example generates data x1 (L×C matrix) based on microbatch 1 according to the number of accelerators (here, the number of accelerators generated = 2). This makes it possible to have each of the multiple accelerators process data x1 (L×C matrix) based on microbatch 1.

[0045] Furthermore, as indicated by reference numeral 510, in order to execute tensor parallelism, the scheduling unit of the comparative example divides the weight parameters W1 (C×4C matrix) and W2 (4C×C matrix) in accordance with the number of accelerators (here, the number of divisions is 2). This makes it possible to generate divided W1_1, W1_2 (C×2C matrix) and W2_1, W2_2 (2C×C matrix) to be assigned to each of the multiple accelerators.

[0046] Next, as indicated by reference numeral 511, the scheduling unit of the comparative example assigns processing to Accelerators 1 and 2. In the example indicated by reference numeral 511, Accelerators 1 is assigned y1_1=f(x1@W1_1), and ·z1_1=y1_1@W2_1, and assign the processing to Accelerator2. y1_2=f(x1@W1_2), and ·z1_2=y1_2@W2_2, The figure shows how the above processing is assigned. Note that y1_1 and y1_2 are L×2C matrices, and z1_1 and z1_2 are L×C matrices.

[0047] Moreover, the example of reference numeral 511 shows how z1 (L×C matrix) is calculated by adding z1_1 (L×C matrix) and z1_2 (L×C matrix) calculated by processing in Accelerator 1 and 2. For example, when z1_1 (L×C matrix) and z1_2 (L×C matrix) are added in Accelerator 1 and 2, z1_1 (L×C matrix) is transmitted from Accelerator 1 to Accelerator 2. Furthermore, z1_2 (L×C matrix) is transmitted from Accelerator 2 to Accelerator 1. In other words, communication processing occurs between the accelerators. Note that z1 is used as x1 in the next layer.

[0048] Next, the scheduling unit of the comparative example performs scheduling for micro-batch 2 in the same manner.

[0049] As indicated by reference numeral 520, in order to execute tensor parallelism, the scheduling unit of the comparative example generates data x2 (L × C matrix) based on microbatch 2 in accordance with the number of accelerators (here, the number of generations = 2). This makes it possible to have each of the multiple accelerators process data x2 (L × C matrix) based on microbatch 2.

[0050] Furthermore, as indicated by reference numeral 520, in order to execute tensor parallelism, the scheduling unit of the comparative example divides the weight parameters W1 (C×4C matrix) and W2 (4C×C matrix) in accordance with the number of accelerators (here, the number of divisions is 2). This makes it possible to generate divided W1_1, W1_2 (C×2C matrix) and W2_1, W2_2 (2C×C matrix) to be assigned to each of the multiple accelerators.

[0051] Next, as indicated by reference numeral 521, the scheduling unit of the comparative example assigns processing to Accelerators 1 and 2. In the example indicated by reference numeral 521, Accelerators 1 is assigned y2_1=f(x2@W1_1), and ·z2_1=y2_1@W2_1, and assigns the processing to the other accelerator (Accelerator2). y2_2=f(x2@W1_2), and ·z2_2=y2_2@W2_2, The figure shows how the above processing is assigned. Note that y2_1 and y2_2 are L×2C matrices, and z2_1 and z2_2 are L×C matrices.

[0052] Moreover, the example of reference numeral 521 shows how z2 (L×C matrix) is calculated by adding z2_1 (L×C matrix) and z2_2 (L×C matrix) calculated by processing in Accelerator 1 and 2. For example, when z2_1 (L×C matrix) and z2_2 (L×C matrix) are added in Accelerator 1 and 2, z2_1 (L×C matrix) is transmitted from Accelerator 1 to Accelerator 2. Furthermore, z2_2 (L×C matrix) is transmitted from Accelerator 2 to Accelerator 1. In other words, communication processing occurs between the accelerators. Note that z2 is used as x2 in the next layer.

[0053] (3) Scheduling by the scheduling unit 302, part 1 Next, a description will be given of a schedule generated by the scheduling unit 302 of the information processing device 120 according to the first embodiment. Fig. 6 is a first diagram showing a specific example of scheduling by the scheduling unit of the information processing device according to the first embodiment.

[0054] As indicated by reference numeral 610, in order to execute tensor parallelism, the scheduling unit 302 generates data x1 (L×C matrix) based on microbatch 1 in accordance with the number of accelerators (here, the number of generations=2).

[0055] At this time, the scheduling unit 302 schedules the processing of data based on two microbatches (microbatches 1 and 2) as a single unit. Therefore, the scheduling unit 302 generates data x2 (L × C matrix) based on microbatch 2 according to the number of accelerators (here, the number of generations = 2).

[0056] Furthermore, as indicated by reference numeral 610, in order to execute tensor parallelism, the scheduling unit 302 divides the weight parameters W1 (C×4C matrix) and W2 (4C×C matrix) in accordance with the number of accelerators (here, the number of divisions is 2). This makes it possible to generate divided W1_1, W1_2 (C×2C matrix) and W2_1, W2_2 (2C×C matrix) to be assigned to each of the multiple accelerators.

[0057] Next, as indicated by reference numeral 611, the scheduling unit 302 assigns processing to Accelerators 1 and 2. In the example indicated by reference numeral 611, y1_1=f(x1@W1_1), and ·z1_1=y1_1@W2_1, and assign the processing to Accelerator2. y1_2=f(x1@W1_2), and ·z1_2=y1_2@W2_2, The figure shows how the above processing is assigned. Note that y1_1 and y1_2 are L×2C matrices, and z1_1 and z1_2 are L×C matrices.

[0058] Moreover, the example of reference numeral 611 shows how z1 (L×C matrix) is calculated by adding z1_1 (L×C matrix) and z1_2 (L×C matrix) calculated by processing in Accelerator 1 and 2. For example, when z1_1 (L×C matrix) and z1_2 (L×C matrix) are added in Accelerator 1 and 2, z1_1 (L×C matrix) is transmitted from Accelerator 1 to Accelerator 2. Furthermore, z1_2 (L×C matrix) is transmitted from Accelerator 2 to Accelerator 1. In other words, communication processing occurs between the accelerators.

[0059] Here, the scheduling unit 302 schedules the accelerators so that, while communication processing is being executed between the accelerators, the accelerators execute calculation processing on data based on the next micro-batch (micro-batch 2) in parallel.

[0060] In the example of reference numeral 611, while communication processing is being executed between accelerators, y2_1=f(x2@W1_1), and ·z2_1=y2_1@W2_1, and assign the processing to Accelerator2. y2_2=f(x2@W1_2), and ·z2_2=y2_2@W2_2, The figure shows how the above processing is assigned. Note that y2_1 and y2_2 are L×2C matrices, and z2_1 and z2_2 are L×C matrices.

[0061] Moreover, the example of reference numeral 611 shows how z2 (L×C matrix) is calculated by adding z2_1 (L×C matrix) and z2_2 (L×C matrix), which are calculated by processing in Accelerator 1 and Accelerator 2. For example, when z2_1 (L×C matrix) and z2_2 (L×C matrix) are added in Accelerator 1 and Accelerator 2, z2_1 (L×C matrix) is transmitted from Accelerator 1 to Accelerator 2. Furthermore, z2_2 (L×C matrix) is transmitted from Accelerator 2 to Accelerator 1. In other words, communication processing occurs between the accelerators.

[0062] Here, the scheduling unit 302 schedules each accelerator to execute processing for data based on the next micro-batch in parallel while communication processing is being executed between the accelerators. Note that, due to space limitations, data based on the next micro-batch (micro-batch 3) is not shown in Fig. 6, but processing for data based on two micro-batches (micro-batches 2 and 3) is also scheduled as a single unit.

[0063] In this way, the scheduling unit 302 schedules the processing of data based on two consecutive micro-batches as a single unit, y=f(x@W1) z=y@W2 This makes it possible to execute the calculation process and the communication process between the accelerators in parallel. As a result, the scheduling unit 302 can improve the execution speed when executing the training process using tensor parallelism.

[0064] Although the example in FIG. 6 does not show the processing in the next layer, z1 (L×C matrix) and z2 (L×C matrix) calculated in Accelerator 1 and Accelerator 2 are used as x1 and x2 in the next layer. y=f(x@W1) z=y@W2 The calculation process and the communication process between the accelerators can be executed in parallel.

[0065] (4) Scheduling by the scheduling unit 302, part 2 Next, another schedule generated by the scheduling unit 302 of the information processing device 120 according to the first embodiment will be described.

[0066] In the above (3), it is assumed that the worker with the worker name "Worker0" is one server device (Server 0) and has two accelerators. On the other hand, here, a case where there are eight accelerators (referred to as Accelerator1_1 to 2_2) will be described. Therefore, in the above (3), 42 pieces of data x1 based on microbatch 1 were generated, and two pieces of data x2 based on microbatch 2 were generated, but here, four pieces of each are generated and each is divided into two. Similarly, in the above (3), the weight parameters W1 and W2 were each divided into two, but here, each is divided into four and then further divided into two.

[0067] 7A to 7C are second to fourth diagrams showing specific examples of scheduling by the scheduling unit of the information processing device according to the first embodiment.

[0068] As indicated by reference numeral 710, in order to execute tensor parallelism, the scheduling unit 302 generates data based on microbatch 1 in accordance with the number of accelerators (here, eight data are generated). Specifically, four pieces of data x1 (L×C matrix) based on microbatch 1 are generated, and each piece is divided into two to generate four pieces of x1_1 (L×(C / 2) matrix) and four pieces of x1_2 (L×(C / 2) matrix).

[0069] Similarly, the scheduling unit 302 generates data based on microbatch 2 according to the number of accelerators (here, eight data are generated). Specifically, four pieces of data x2 (L×C matrix) based on microbatch 2 are generated, and each piece is divided into two to generate four pieces of x2_1 (L×(C / 2) matrix) and four pieces of x2_2 (L×(C / 2) matrix).

[0070] Furthermore, as indicated by reference numeral 710, in order to execute tensor parallelism, the scheduling unit 302 divides the weight parameters W1 (a C×4C matrix) and W2 (a 4C×C matrix) in accordance with the number of accelerators (here, divided into four and then further divided into two). As a result, the scheduling unit 302 can generate divided weight parameters W1_1_1 to W1_1_4 (a (C / 2)×C matrix) and W1_2_1 to W1_2_4 (a (C / 2)×C matrix) to be assigned to each of the multiple accelerators. Furthermore, the scheduling unit 302 can generate weight parameters W2_1_1 to W2_1_4 (a C×(C / 2) matrix) and W2_2_1 to W2_2_4 (a C×(C / 2) matrix).

[0071] Next, the scheduling unit 302 allocates processing to the Accelerators 1_1 to 2_4 (reference numerals 711 to 724).

[0072] As shown in FIG. 7B, the examples of reference numerals 711, 712, 721, and 722 are assigned to Accelerator1_1. ·y1_1_1=x1_1@W1_1_1, ·y1_1=Allreduce[Accelerator1_1, 2_1], ·y2_1_1=x2_1@W1_1_1, ·y2_1=Allreduce[Accelerator1_1, 2_1], ·z1_1_1=f(y1_1@W2_1_1), ·z1_1=Allreduce[Accelerator1_1, 1_2, 1_3, 1_4], ·z2_1_1=f(y2_1@W2_1_1), ·z2_1=Allreduce[Accelerator1_1, 1_2, 1_3, 1_4], ·y1_1_1_nxt=z1_1@W1_1_1_nxt, and assign the processing to Accelerator1_2. ·y1_1_2=x1_1@W1_1_2, ·y1_1=Allreduce[Accelerator1_2, 2_2], ·y2_1_2=x2_1@W1_1_2, ·y2_1=Allreduce[Accelerator1_2, 2_2], ·z1_1_2=f(y1_1@W2_1_2), ·z1_1=Allreduce[Accelerator1_1, 1_2, 1_3, 1_4], ·z2_1_2=f(y2_1@W2_1_2), ·z2_1=Allreduce[Accelerator1_1, 1_2, 1_3, 1_4], ·y1_1_2_nxt=z1_1@W1_1_2_nxt, Assign the processing to Accelerator2_1, ·y1_2_1=x1_2@W1_2_1, ·y1_2=Allreduce[Accelerator1_1, 2_1], ·y2_2_1=x2_2@W1_2_1, ·y2_2=Allreduce[Accelerator1_1, 2_1], ·z1_2_1=f(y1_2@W2_2_1), ·z1_2=Allreduce[Accelerator2_1, 2_2, 2_3, 2_4], ·z2_2_1=f(y2_2@W2_2_1), ·z2_2=Allreduce[Accelerator2_1, 2_2, 2_3, 2_4], ·y1_2_1_nxt=z1_2@W1_2_1_nxt, Assign the processing to Accelerator2_2, ·y1_2_2=x1_2@W1_2_2, ·y1_2=Allreduce[Accelerator1_2, 2_2], ·y2_2_2=x2_2@W1_2_2, ·y2_2=Allreduce[Accelerator1_2, 2_2], ·z1_2_2=f(y1_2@W2_2_2), ·z1_2=Allreduce[Accelerator2_1, 2_2, 2_3, 2_4], ·z2_2_2=f(y2_2@W2_2_2), ·z2_2=Allreduce[Accelerator2_1, 2_2, 2_3, 2_4], ·y1_2_2_nxt=z1_2@W1_2_2_nxt, The figure shows how the processing of the above is assigned. Note that y1_1_1, y1_1_2, y1_2_1, y1_2_2 are L×C matrices. y2_1_1, y2_1_2, y2_2_1, y2_2_2 are L×C matrices. z1_1_1, z1_1_2, z1_2_1, z1_2_2 are (L×(C / 2)) matrices, and z1_1, z2_1 are (L×(C / 2)) matrices. z2_1_1, z2_1_2, z2_2_1, z2_2_2 are (L×(C / 2)) matrices, and z2_1, z2_2 are (L×(C / 2)) matrices. y1_1_1_next, y1_1_2_next, y1_2_1_next, y1_2_2_next are L×C matrices.

[0073] By allocating processes in this way, the scheduling unit 302 can, for example, A process of communicating y1_1_1 and y1_2_1 to calculate y1_1 by Allreduce; ·A process for calculating y2_1_1, can be executed in parallel, A process of communicating y2_1_1 and y2_2_1 to calculate y2_1 by Allreduce; ·Processing to calculate z1_1_1, and can be executed in parallel, · A process of communicating z1_1_1, z1_1_2, z1_1_3, and z1_1_4 to calculate z1_1 by Allreduce; ·Processing to calculate z2_1_1, and can be executed in parallel, · A process of communicating z2_1_1, z2_1_2, z2_1_3, and z2_1_4 to calculate z2_1 by Allreduce; · A process to calculate y1_1_1_nxt of the next layer using z1_1; can be executed in parallel (see reference numeral 711).

[0074] Similarly, according to the scheduling unit 302, for example, A process of communicating y1_1_2 and y1_2_2 to calculate y1_1 by Allreduce; ·Processing to calculate y2_1_2, and can be executed in parallel, A process of communicating y2_1_2 and y2_2_2 to calculate y2_1 by Allreduce; ·Processing to calculate z1_1_2, and can be executed in parallel, · A process of communicating z1_1_1, z1_1_2, z1_1_3, and z1_1_4 to calculate z1_1 by Allreduce; ·Processing to calculate z2_1_2, and can be executed in parallel, · A process of communicating z2_1_1, z2_1_2, z2_1_3, and z2_1_4 to calculate z2_1 by Allreduce; · A process to calculate y1_1_2_nxt of the next layer using z1_1; can be executed in parallel (see reference numeral 712).

[0075] Similarly, according to the scheduling unit 302, for example, A process of communicating y1_1_1 and y1_2_1 to calculate y1_2 by Allreduce; ·Processing to calculate y2_2_1, and can be executed in parallel, A process of communicating y2_1_1 and y2_2_1 to calculate y2_2 by Allreduce; ·Processing to calculate z1_2_1, and can be executed in parallel, · A process of communicating z1_2_1, z1_2_2, z1_2_3, and z1_2_4 to calculate z1_2 by Allreduce; ·Processing to calculate z2_2_1, and can be executed in parallel, · A process of communicating z2_2_1, z2_2_2, z2_2_3, and z2_2_4 to calculate z2_2 by Allreduce; · Processing to calculate y1_2_1_nxt of the next layer using z1_2, can be executed in parallel (see reference numeral 721).

[0076] Similarly, according to the scheduling unit 302, for example, A process of communicating y1_1_2 and y1_2_2 to calculate y1_2 by Allreduce; ·A process to calculate y2_2_2, can be executed in parallel, A process of communicating y2_1_2 and y2_2_2 to calculate y2_2 by Allreduce; ·Processing to calculate z1_2_2, and can be executed in parallel, · A process of communicating z1_2_1, z1_2_2, z1_2_3, and z1_2_4 to calculate z1_2 by Allreduce; ·Processing to calculate z2_2_2, and can be executed in parallel, · A process of communicating z2_2_1, z2_2_2, z2_2_3, and z2_2_4 to calculate z2_2 by Allreduce; · Processing to calculate y1_2_2_nxt of the next layer using z1_2, can be executed in parallel (see reference numeral 722).

[0077] Also, as shown in FIG. 7C, the examples of reference numerals 713, 714, 723, and 724 are assigned to Accelerator1_3 as follows: ·y1_1_3=x1_1@W1_1_3, ·y1_1=Allreduce[Accelerator1_3, 2_3], ·y2_1_3=x2_1@W1_1_3, ·y2_1=Allreduce[Accelerator1_3, 2_3], ·z1_1_3=f(y1_1@W2_1_3), ·z1_1=Allreduce[Accelerator1_1, 1_2, 1_3, 1_4], ·z2_1_3=f(y2_1@W2_1_3), ·z2_1=Allreduce[Accelerator1_1, 1_2, 1_3, 1_4], ·y1_1_3_nxt=z1_1@W1_1_3_nxt, and assign the processing to Accelerator1_4. ·y1_1_4=x1_1@W1_1_4, ·y1_1=Allreduce[Accelerator1_4, 2_4], ·y2_1_4=x2_1@W1_1_4, ·y2_1=Allreduce[Accelerator1_4, 2_4], ·z1_1_4=f(y1_1@W2_1_4), ·z1_1=Allreduce[Accelerator1_1, 1_2, 1_3, 1_4], ·z2_1_4=f(y2_4@W2_1_4), ·z2_1=Allreduce[Accelerator1_1, 1_2, 1_3, 1_4], ·y1_1_4_nxt=z1_1@W1_1_4_nxt, and assign the processing to Accelerator2_3. ·y1_2_3=x1_2@W1_2_3, ·y1_2=Allreduce[Accelerator1_3, 2_3], ·y2_2_3=x2_2@W1_2_3, ·y2_2=Allreduce[Accelerator1_3, 2_3], ·z1_2_3=f(y1_2@W2_2_3), ·z1_2=Allreduce[Accelerator2_1, 2_2, 2_3, 2_4], ·z2_2_3=f(y2_2@W2_2_3), ·z2_2=Allreduce[Accelerator2_1, 2_2, 2_3, 2_4], ·y1_2_3_nxt=z1_2@W1_2_3_nxt, and assign the processing to Accelerator2_4. ·y1_2_4=x1_2@W1_2_4, ·y1_2=Allreduce[Accelerator1_4, 2_4], ·y2_2_4=x2_2@W1_2_4, ·y2_2=Allreduce[Accelerator1_4, 2_4], ·z1_2_4=f(y1_2@W2_2_4), ·z1_2=Allreduce[Accelerator2_1, 2_2, 2_3, 2_4], ·z2_2_4=f(y2_2@W2_2_4), ·z2_2=Allreduce[Accelerator2_1, 2_2, 2_3, 2_4], ·y1_2_4_nxt=z1_2@W1_2_4_nxt, The figure shows how the processing of the above is assigned. Note that y1_1_3, y1_1_4, y1_2_3, y1_2_4 are L×C matrices. y2_1_3, y2_1_4, y2_2_3, y2_2_4 are L×C matrices. z1_1_3, z1_1_4, z1_2_3, z1_2_4 are (L×(C / 2)) matrices, and z1_1, z2_1 are (L×(C / 2)) matrices. z2_1_3, z2_1_4, z2_2_3, z2_2_4 are (L×(C / 2)) matrices, and z2_1, z2_2 are (L×(C / 2)) matrices. y1_1_3_next, y1_1_4_next, y1_2_3_next, y1_2_4_next are L×C matrices.

[0078] By allocating processes in this way, the scheduling unit 302 can, for example, A process of communicating y1_1_3 and y1_2_3 to calculate y1_1 by Allreduce; ·Processing to calculate y2_1_3, and can be executed in parallel, A process of communicating y2_1_3 and y2_2_3 to calculate y2_1 by Allreduce; ·Processing to calculate z1_1_3, and can be executed in parallel, · A process of communicating z1_1_1, z1_1_2, z1_1_3, and z1_1_4 to calculate z1_1 by Allreduce; ·Processing to calculate z2_1_3, and can be executed in parallel, · A process of communicating z2_1_1, z2_1_2, z2_1_3, and z2_1_4 to calculate z2_1 by Allreduce; · A process to calculate y1_1_3_nxt of the next layer using z1_1; can be executed in parallel (see reference numeral 713).

[0079] Similarly, according to the scheduling unit 302, for example, A process of communicating y1_1_4 and y1_2_4 to calculate y1_1 by Allreduce; ·Processing to calculate y2_1_4, and can be executed in parallel, A process of communicating y2_1_4 and y2_2_4 to calculate y2_1 by Allreduce; ·Processing to calculate z1_1_4, and can be executed in parallel, · A process of communicating z1_1_1, z1_1_2, z1_1_3, and z1_1_4 to calculate z1_1 by Allreduce; ·Processing to calculate z2_1_4, and can be executed in parallel, · A process of communicating z2_1_1, z2_1_2, z2_1_3, and z2_1_4 to calculate z2_1 by Allreduce; · Processing to calculate y1_1_4_nxt of the next layer using z1_1; can be executed in parallel (see reference numeral 714).

[0080] Similarly, according to the scheduling unit 302, for example, A process of communicating y1_1_3 and y1_2_3 to calculate y1_2 by Allreduce; ·Processing to calculate y2_2_3, and can be executed in parallel, A process of communicating y2_1_3 and y2_2_3 to calculate y2_2 by Allreduce; ·Processing to calculate z1_2_3, and can be executed in parallel, · A process of communicating z1_2_1, z1_2_2, z1_2_3, and z1_2_4 to calculate z1_2 by Allreduce; ·Processing to calculate z2_2_3, and can be executed in parallel, · A process of communicating z2_2_1, z2_2_2, z2_2_3, and z2_2_4 to calculate z2_2 by Allreduce; · Processing to calculate y1_2_3_nxt of the next layer using z1_2, can be executed in parallel (see reference numeral 723).

[0081] Similarly, according to the scheduling unit 302, for example, A process of communicating y1_1_4 and y1_2_4 to calculate y1_2 by Allreduce; ·Processing to calculate y2_2_4, and can be executed in parallel, A process of communicating y2_1_4 and y2_2_4 to calculate y2_2 by Allreduce; ·Processing to calculate z1_2_4, and can be executed in parallel, · A process of communicating z1_2_1, z1_2_2, z1_2_3, and z1_2_4 to calculate z1_2 by Allreduce; ·Processing to calculate z2_2_4, and can be executed in parallel, · A process of communicating z2_2_1, z2_2_2, z2_2_3, and z2_2_4 to calculate z2_2 by Allreduce; · Processing to calculate y1_2_4_nxt of the next layer using z1_2, can be executed in parallel (see reference numeral 724).

[0082] <Summary> As is clear from the above description, in the information processing device 120 according to the first embodiment, in scheduling when executing training processing by tensor parallelism for each layer of the NN, a communication process between the accelerators of a result of executing a calculation process using at least a part of the model for the first input data; computing the second input data using at least a portion of the model; The processing is assigned to multiple accelerators used for tensor parallelism so that the above processes are executed in parallel.

[0083] In addition, in the server device according to the first embodiment, when performing training processing using tensor parallelism for each layer of the NN, a communication process between the accelerators of a result of executing a calculation process using at least a part of the model for the first input data; computing the second input data using at least a portion of the model; are executed in parallel.

[0084] As a result, according to the first embodiment, the execution speed of the training process using tensor parallelism can be improved compared to when the calculation process of the previous input data is executed and the calculation process of the next input data is performed after the communication process between the accelerators is completed.

[0085] [Second embodiment] In the first embodiment, scheduling was described for a case where training processing is performed in tensor parallelism with data based on multiple microbatches as input data for each layer of a neural network. In contrast, in the second embodiment, scheduling will be described for a case where training processing is performed in tensor parallelism with data based on multiple microbatches as input data in a decoder unit of a transformer. Note that the second embodiment will be described mainly with respect to differences from the first embodiment.

[0086] <Specific examples of scheduling> First, a specific example of scheduling by the scheduling unit 302 of the information processing device according to the second embodiment will be described.

[0087] (1) Overview of intra-layer scheduling FIG. 8 is a diagram illustrating an overview of scheduling by an information processing device according to a second embodiment. In FIG. 8, reference numerals 810_1 to 810_n denote multiple Decoder units constituting a Transformer. The example in FIG. 8 illustrates a state in which a worker with a worker name="Worker0" is assigned to training processing in the Decoder unit denoted by reference numeral 810_1. In the example in FIG. 8, the worker is one server device (Server 0) and has two accelerators. The Decoder unit denoted by reference numeral 810_1 is divided into an Attention unit and an MLP unit. When data based on microbatch 1 and data based on microbatch 2 are input as input data via a preprocessing unit, the Attention unit of the Decoder unit denoted by reference numeral 810_1 performs the following calculation. ·q=x@Wq, ·k=x@Wk, ·v=x@Wv, a=MultiHeadAttention(q,k,v) o=x+(a@Wo) where: x: input data, Wq, Wk, Wv, Wo: Weight parameters for the attention part, @: matrix multiplication operator, where x, q, k, v, a, and o are L×C matrices, and Wq, Wk, Wv, and Wo are C×C matrices.

[0088] Also, assume that the above calculation is performed in the attention unit of the decoder unit indicated by reference numeral 810_1, and data o based on microbatches 1 and 2, which are processed data, is input as input data. In this case, the MLP unit of the decoder unit indicated by reference numeral 810_1 performs the following calculation. ·y=f(o@W1), z=o+(y@W2) where: f: a given activation function, W1, W2: Weight parameters of the MLP section, where W1 is a C×4C matrix, W2 is a 4C×C matrix, y is an L×4C matrix, and z is an L×C matrix.

[0089] As described above, in the second embodiment, the scheduling unit 302 uses data based on microbatches 1 and 2 as input data, and the Accelerators 1 and 2 perform the following: ·Attention Department, ·MLP Department, A schedule is generated to execute each calculation using tensor parallelism.

[0090] (2) Scheduling by the scheduling unit 302 Next, details of the schedule generated by the scheduling unit 302 of the information processing device 120 according to the second embodiment will be described. Figures 9A and 9B are first and second diagrams showing a specific example of scheduling by the scheduling unit of the information processing device according to the second embodiment.

[0091] As indicated by reference numeral 910 in FIG. 9A, in order to execute tensor parallelism, the scheduling unit 302 generates data x1 (L×C matrix) based on microbatch 1 in accordance with the number of accelerators (here, the number of generations=2).

[0092] At this time, the scheduling unit 302 schedules the processing of data based on two microbatches (microbatches 1 and 2) as a single unit. Therefore, the scheduling unit 302 generates data x2 (L × C matrix) based on microbatch 2 according to the number of accelerators (here, the number of generations = 2).

[0093] Furthermore, as indicated by reference numeral 910, in order to execute tensor parallelism, the scheduling unit 302 divides the weight parameters Wq (C×C matrix), Wk (C×C matrix), and Wv (C×C matrix) according to the number of accelerators (here, the number of divisions is 2). As a result, divided Wq_1, Wq_2 (C×(C / 2) matrix), Wk_1, Wk_2 (C×(C / 2 matrix)), Wv_1, Wv_2 (C×(C / 2) matrix)) to be assigned to each of the multiple accelerators. Furthermore, as indicated by reference numeral 910, in order to execute tensor parallelism, the scheduling unit 302 divides the weight parameter Wo (C×C matrix) according to the number of accelerators (here, the number of divisions is 2). As a result, divided Wo_1, Wo_2 ((C / 2)×C matrix) to be assigned to each of the multiple accelerators are generated.

[0094] Furthermore, as indicated by reference numeral 910, in order to execute tensor parallelism, the scheduling unit 302 divides the weight parameters W1 (C×4C matrix) and W2 (4C×C matrix) in accordance with the number of accelerators (here, the number of divisions is 2). As a result, post-division W1_1, W1_2 (C×2C matrix), W2_1, and W2_2 (2C×C matrix) are generated to be assigned to each of the multiple accelerators.

[0095] Next, as indicated by reference numeral 920 in Fig. 9B, the scheduling unit 302 allocates processing to the Attention unit. In the example indicated by reference numerals 921 and 922, the accelerator Accelerator1 is assigned ·q1_1=x1@Wq_1, k1_1=x1@Wk_1, v1_1=x1@Wv_1, ·a1_1=MultiHeadAttention(q1_1, k1_1, v1_1), ·o1_1=a1_1@Wo_1, Send o1_1 to Accelerator2, ·o1=x1+o1_1+o1_2, and assign the processing to Accelerator2. ·q1_2=x1@Wq_2, k1_2=x1@Wk_2, v1_2=x1@Wv_2, ·a1_2=MultiHeadAttention(q1_2, k1_2, v1_2), ·o1_2=a1_2@Wo_2, Send o1_2 to Accelerator1, ·o1=x1+o1_1+o1_2, In addition, the examples of reference numerals 921 and 922 show how the following processes are assigned to Accelerator1_1: ·q2_1=x2@Wq_1, k2_1=x2@Wk_1, v2_1=x2@Wv_1, ·a2_1=MultiHeadAttention(q2_1, k2_1, v2_1), ·o2_1=a2_1@Wo_1, ·Send o2_1 to Accelerator2, ·o2=x2+o2_1+o2_2, and assign the processing to Accelerator2. ·q2_2=x2@Wq_2, k2_2=x2@Wk_2, v2_2=x2@Wv_2, ·a2_2=MultiHeadAttention(q2_2, k2_2, v2_2), ·o2_2=a2_2@Wo_2, Send o2_2 to Accelerator1, ·o2=x2+o2_1+o2_2, The figure shows how the processing is assigned.

[0096] By allocating processes in this way, the scheduling unit 302 can, for example, A process of communicating o1_1 and o1_2 calculated using data based on the previous micro-batch as input data between Accelerator1 and Accelerator2; A process of calculating o2_1=a2_1@Wo_1 using data based on the next micro-batch as input data, or a process of calculating o2_2=a2_2@Wo_2; can be executed in parallel.

[0097] Next, as indicated by reference numeral 920, the scheduling unit 302 allocates the processing in the MLP unit to Accelerator 1 and 2. In the examples indicated by reference numerals 921 and 922, Accelerator 1 is assigned ·y1_1=f(o1@W1_1), ·z1_1=y1_1@W2_1, Send z1_1 to Accelerator2, ·z1=o1+z1_1+z1_2, and assign it to Accelerator2. ·y1_2=f(o1@W1_2), ·z1_2=y1_2@W2_2, Send z1_2 to Accelerator1, ·z1=o1+z1_1+z1_2, The figure shows how the processing is assigned.

[0098] By allocating processes in this way, the scheduling unit 302 can, for example, A process of communicating o2_1 and o2_2 calculated using data based on the next micro-batch as input data between Accelerator1 and Accelerator2; A process of calculating z1_1=y1_1@W2_1 or z1_2=y1_2@W2_2 using data based on the previous micro-batch as input data, can be executed in parallel.

[0099] Next, as indicated by reference numeral 920, the scheduling unit 302 allocates the processing in the MLP unit to Accelerator 1 and 2. In the examples indicated by reference numerals 921 and 922, Accelerator 1 is assigned ·y2_1=f(o2@W1_1), ·z2_1=y2_1@W2_1, · Send z2_1 to Accelerator2, ·z2=o2+z2_1+z2_2, and assign it to Accelerator2. ·y2_2=f(o2@W1_2), ·z2_2=y2_1@W2_2, Send z2_2 to Accelerator1, ·z2=o2+z2_1+z2_2, The figure shows how the processing is assigned.

[0100] By allocating processes in this way, the scheduling unit 302 can, for example, A process of communicating z1_1 and z1_2 calculated using data based on the previous micro-batch as input data between Accelerator1 and Accelerator2; - A process of calculating z2_1=y2_1@W2_1 using data based on the next micro-batch as input data, or a process of calculating z2_2=y2_1@W2_2; can be executed in parallel.

[0101] <Summary> As is clear from the above description, in the information processing device 120 according to the second embodiment, in the decoder unit of Transformer, in scheduling when executing training processing by tensor parallelism, - Data based on two consecutive micro-batches is used as a single input data. communication processing between accelerators as a result of executing a calculation process on the first input data in the attention unit; A calculation process for the second input data in the attention unit; The processing is assigned to multiple accelerators used for tensor parallelism so that the above processes are executed in parallel. The data based on two micro-batches is used as a single input data. communication processing between accelerators as a result of executing calculation processing on the second input data in the attention unit; A calculation process of the first input data in the MLP unit; The processing is assigned to multiple accelerators used for tensor parallelism so that the above processes are executed in parallel. The data based on two micro-batches is used as a single input data. communication processing between accelerators resulting from the execution of the calculation processing of the first input data in the MLP unit; A calculation process of the second input data in the MLP unit; The processing is assigned to multiple accelerators used for tensor parallelism so that the above processes are executed in parallel.

[0102] In addition, in the server device according to the second embodiment, when the tensor parallel training process is executed in the decoder unit of the transformer, - Data based on two consecutive micro-batches is used as a single input data. communication processing between accelerators as a result of executing a calculation process on the first input data in the attention unit; A calculation process for the second input data in the attention unit; are executed in parallel. - Data based on two consecutive micro-batches is used as a single input data. communication processing between accelerators as a result of executing calculation processing on the second input data in the attention unit; A calculation process of the first input data in the MLP unit; are executed in parallel. - Data based on two consecutive micro-batches is used as a single input data. communication processing between accelerators resulting from the execution of the calculation processing of the first input data in the MLP unit; A calculation process of the second input data in the MLP unit; are executed in parallel.

[0103] As a result, according to the second embodiment, the execution speed of the training process using tensor parallelism can be improved compared to when the calculation process of the previous input data is executed and the calculation process of the next input data is executed after the communication process between the accelerators is completed.

[0104] In this embodiment, the Transformer decoder performs training using tensor parallelism with input data based on multiple microbatches. However, if the Transformer includes an encoder, the encoder may perform training using tensor parallelism in a similar manner.

[0105] [Third embodiment] In the second embodiment, scheduling was described for a case where a Transformer decoder executes training processing using tensor parallelism with data based on multiple microbatches as input data. In contrast, the third embodiment describes scheduling for a Transformer decoder executes training processing using sequence parallelism with data based on multiple microbatches as input data. The third embodiment will be described focusing on differences from the second embodiment.

[0106] <Specific examples of scheduling> First, a specific example of scheduling by the scheduling unit 302 of the information processing device according to the third embodiment will be described.

[0107] (1) Overview of intra-layer scheduling FIG. 10 is a diagram illustrating an overview of scheduling by an information processing device according to a third embodiment. In FIG. 10, reference numerals 1010_1 to 1010_n denote multiple Decoder units constituting a Transformer. The example of FIG. 10 illustrates a state in which a worker with a worker name="Worker0" is assigned to training processing in the Decoder unit denoted by reference numeral 1010_1. In the example of FIG. 10, the worker is one server device (Server 0) and has two accelerators. The Decoder unit denoted by reference numeral 1010_1 is divided into an Attention unit and an MLP unit. When data based on microbatches 1 and 2 is input as input data via a preprocessing unit, the calculations performed in the Attention unit of the Decoder unit denoted by reference numeral 1010_1 are as described in the second embodiment. Furthermore, the calculations performed in the MLP unit of the Decoder unit denoted by reference numeral 1010_1 are also as described in the second embodiment.

[0108] In the second embodiment, the scheduling unit 302 receives data based on microbatches 1 and 2 as input data, and schedules the following: ·Attention Department, ·MLP Department, A schedule is generated to execute each calculation in sequence parallel.

[0109] (2) Scheduling by the scheduling unit of the comparative example Next, a schedule generated by a scheduling unit of a comparative example will be described based on the overview of intra-layer scheduling shown in Fig. 10. The scheduling unit of a comparative example is a general term for a general scheduling unit that generates a schedule so as to execute forward calculations in sequence parallelism. To clarify the difference between the schedule generated by the scheduling unit 302 of the information processing device 120 and a general schedule, the scheduling of the comparative example will be described first. Fig. 11 is a second diagram showing a specific example of scheduling by the scheduling unit of a comparative example.

[0110] As shown by reference numeral 1110, in order to execute sequence parallelism, the scheduling unit 302 of the comparative example divides data x1 (L × C matrix) based on microbatch 1 in accordance with the number of accelerators (here, the number of divisions = 2). As a result, the scheduling unit 302 of the comparative example generates x1_1 and x1_2 ((L / 2) × C matrix).

[0111] Next, as indicated by reference numeral 1111, the scheduling unit of the comparative example allocates the processing in the attention unit to Accelerators 1 and 2. In the example of reference numeral 1111, Accelerator 1 is assigned ·k1_1=x1_1@Wk, ·v1_1=x1_1@Wv, and assign the processing to Accelerator2. ·k1_2=x1_2@Wk, ·v1_2=x1_2@Wv, The figure shows how the processes are assigned. Note that k1_1, k1_2, v1_1, and v1_2 are (L / 2) × C matrices, and Wk and Wv are C × C matrices.

[0112] In addition, the example of reference numeral 1111 is a case where, after assigning the process of transmitting and receiving k1_1, v1_1 and k1_2, v1_2 calculated in Accelerator1 and Accelerator2 between the accelerators, ·k1=concat(k1_1, k1_2), ·v1=concat(v1_1, v1_2), The figure shows how the processes are assigned. Note that k1 and v1 are L × C matrices.

[0113] In addition, the example of code 1111 is Accelerator1, ·q1_1=x1_1@Wq, ·a1_1=MultiHeadAttention(q1_1,k1,v1), ·o1_1=x1_1+(a1_1@Wo), and assign the processing to Accelerator2. ·q1_2=x1_2@Wq, ·a1_2=MultiHeadAttention(q1_2,k1,v1), ·o1_2=x1_2+(a1_2@Wo), The figure shows how the processing of the above is assigned. Note that q1_2, a1_2, and o1_2 are (L / 2) × C matrices, and Wq and Wo are C × C matrices.

[0114] As indicated by reference numeral 1111, the scheduling unit of the comparative example allocates processing in the MLP unit to Accelerators 1 and 2. In the example of reference numeral 1111, Accelerators 1 is assigned ·y1_1=f(o1_1@W1), ·z1_1=o1_1+(y1_1@W2), and assign the processing to Accelerator2. ·y1_2=f(o1_2@W1), ·z1_2=o1_2+(y1_2@W2), The figure shows how the processing is assigned.

[0115] Next, the scheduling unit of the comparative example performs scheduling for micro-batch 2 in the same manner.

[0116] As indicated by reference numeral 1120, in order to execute sequence parallelism, the scheduling unit 302 of the comparative example divides data x2 (L × C matrix) based on microbatch 2 in accordance with the number of accelerators (here, the number of divisions = 2). As a result, the scheduling unit 302 of the comparative example generates x2_1 and x2_2 ((L / 2) × C matrix).

[0117] Next, as indicated by reference numeral 1121, the scheduling unit of the comparative example allocates the processing in the attention unit to Accelerators 1 and 2. In the example of reference numeral 1121, Accelerator 1 is assigned ·k2_1=x2_1@Wk, ·v2_1=x2_1@Wv, and assign the processing to Accelerator2. ·k2_2=x2_2@Wk, ·v2_2=x2_2@Wv, The figure shows how the processes of k2_1, k2_2, v2_1, and v2_2 are assigned. Note that k2_1, k2_2, v2_1, and v2_2 are (L / 2) × C matrices, and Wk and Wv are C × C matrices.

[0118] In addition, the example of reference numeral 1121 is a case where, after assigning the process of transmitting and receiving k2_1, v2_1 and k2_2, v2_2 calculated in Accelerator1 and Accelerator2 between the accelerators, ·k2=concat(k2_1, k2_2), v2=concat(v2_1, v2_2), The figure shows how the processes are assigned. Note that k2 and v2 are L × C matrices.

[0119] In addition, the example of code 1121 is Accelerator1, ·q2_1=x2_1@Wq, ·a2_1=MultiHeadAttention(q2_1,k2,v2), ·o2_1=x2_1+(a2_1@Wo), and assign the processing to Accelerator2. ·q2_2=x2_2@Wq, ·a2_2=MultiHeadAttention(q2_2,k2,v2), ·o2_2=x2_2+(a2_2@Wo), The figure shows how the processing of q2_2, a2_2, o2_2, Wq, and Wo are (L / 2) × C matrices.

[0120] As indicated by reference numeral 1121, the scheduling unit of the comparative example allocates processing in the MLP unit to Accelerators 1 and 2. In the example of reference numeral 1121, Accelerators 1 is assigned ·y2_1=f(o2_1@W1), ·z2_1=o2_1+(y2_1@W2), and assign the processing to Accelerator2. ·y2_2=f(o2_2@W1), ·z2_2=o2_2+(y2_2@W2), The figure shows how the processing is assigned.

[0121] (3) Scheduling by the scheduling unit 302 Next, a schedule generated by the scheduling unit 302 of the information processing device 120 according to the third embodiment will be described. Figures 12A and 12B are first and second diagrams showing a specific example of scheduling by the scheduling unit of the information processing device according to the third embodiment.

[0122] As indicated by reference numeral 1210 in FIG. 12A, in order to execute sequence parallelism, the scheduling unit 302 divides data x1 (an L×C matrix) based on microbatch 1 in accordance with the number of accelerators (here, the number of divisions is 2). As a result, the scheduling unit 302 generates x1_1 and x1_2 (an (L / 2)×C matrix). At this time, the scheduling unit 302 schedules processing of data based on two microbatches (microbatches 1 and 2) as a single unit. Therefore, the scheduling unit 302 divides data x2 (an L×C matrix) based on microbatch 2 in accordance with the number of accelerators (here, the number of divisions is 2) to generate x2_1 and x2_2 (an (L / 2)×C matrix).

[0123] Next, as indicated by reference numeral 1211 in FIG. 12B, the scheduling unit 302 assigns processing to the Attention unit. The example of reference numeral 1211 assigns the following to Accelerator 1: ·k1_1=x1_1@Wk, ·v1_1=x1_1@Wv, Send k1_1 and v1_1 to Accelerator2, and assign the processing to Accelerator2. ·k1_2=x1_2@Wk, ·v1_2=x1_2@Wv, Send k1_2 and v1_2 to Accelerator1, The figure shows how the above processes are assigned. Note that k1_1, k1_2, v1_1, and v1_2 are (L / 2) × C matrices. k1_1 and k1_2 are collectively referred to as k1 (L × C matrix), and v1_1 and v1_2 are collectively referred to as v1 (L × C matrix).

[0124] In addition, the example of code 1211 is Accelerator1, ·q1_1=x1_1@Wq, ·k2_1=x2_1@Wk, ·v2_1=x2_1@Wv, · k2_1, v2_1 sent to Accelerator2, and assign the processing to Accelerator2. ·q1_2=x1_2@Wq, ·k2_2=x2_2@Wk, ·v2_2=x2_2@Wv, Send k2_2 and v2_2 to Accelerator1, The figure shows how the above processes are assigned. Note that q1_1, k2_1, k2_2, v2_1, and v2_2 are (L / 2) × C matrices. k2_1 and k2_2 are collectively called k2 (L × C matrix), and v2_1 and v2_2 are collectively called v2 (L × C matrix).

[0125] By allocating processes in this way, the scheduling unit 302 can, for example, A process of communicating k1_1 and v1_1, and k1_2 and v1_2, which are calculated using data based on the previous micro-batch as input data, between Accelerator1 and Accelerator2; A process of calculating q1_1, k2_1, and v2_1, or a process of calculating q1_2, k2_2, and v2_2 using data based on the next micro-batch as input data; can be executed in parallel.

[0126] In addition, the example of reference numeral 1211 is configured to process the attention unit by Accelerator 1 as follows: ·q2_1=x2_1@Wq, ·k1=concat(k1_1,k1_2), ·v1=concat(v1_1,v1_2), ·a1_1=MultiHeadAttention(q1_1, k1, v1), ·o1_1=x1_1+(a1_1@Wo), ·k1_1_next=z1_1@Wk_next, ·v1_1_next=z1_1@Wv_next, Send k1_1_next and v1_1_next to Accelerator2, The processing of the Attention section is assigned to Accelerator2. ·q2_2=x2_2@Wq, ·k1=concat(k1_1,k1_2), ·v1=concat(v1_1,v1_2), ·a1_2=MultiHeadAttention(q1_2, k1, v1), ·o1_2=x1_2+(a1_2@Wo), ·k1_2_next=z1_2@Wk_next, ·v1_2_next=z1_2@Wv_next, Send k1_2_next and v1_2_next to Accelerator1, The processing of the MLP part is assigned to Accelerator1. ·y1_1=f(o1_1@W1), ·z1_1=o1_1+(y1_1@W2), The processing of the MLP part is assigned to Accelerator2. ·y1_2=f(o1_2@W1), ·z1_2=o1_2+(y1_2@W2), The figure shows how the processing of q2_1, q2_2, a1_1, a1_2, o1_1, o1_2 is an (L / 2) × C matrix.

[0127] By allocating processes in this way, the scheduling unit 302 can, for example, A process of communicating k2_1 and v2_1, and k2_2 and v2_2, calculated using data based on the next micro-batch as input data, between Accelerator1 and Accelerator2; Calculating o1_1 or o1_2 using data from the previous micro-batch as input data, and Calculating k1_1_next, v1_1_next, k1_2_next, and v1_2_next using data based on the next micro-batch as input data; can be executed in parallel.

[0128] Furthermore, by allocating processes in this way, the scheduling unit 302 can, for example, A process of communicating k2_1 and v2_1, and k2_2 and v2_2, calculated using data based on the next micro-batch as input data, between Accelerator1 and Accelerator2; A process of calculating y1_1 and z1_1, or a process of calculating y1_2 and z1_2 using data based on the previous micro-batch as input data; can be executed in parallel.

[0129] In addition, the example of reference numeral 1211 is configured to process the attention unit by Accelerator 1 as follows: ·q1_1_next=z1_1@Wq_next, ·k2=concat(k2_1,k2_2), v2=concat(v2_1,v2_2), ·a2_1=MultiHeadAttention(q2_1, k2, v2), ·o2_1=x2_1+(a2_1@Wo), ·k2_1_next=z2_1@Wk_next, ·v2_1_next=z2_1@Wv_next, The processing of the Attention section is assigned to Accelerator2. ·q1_2_next=z1_2@Wq_next, ·k2=concat(k2_1,k2_2), v2=concat(v2_1,v2_2), ·a2_2=MultiHeadAttention(q2_2, k2, v2), ·o2_2=x2_2+(a2_2@Wo), ·k2_2_next=z2_2@Wk_next, ·v2_2_next=z2_2@Wv_next, The processing of the MLP part is assigned to Accelerator1. ·y2_1=f(o2_1@W1), ·z2_1=o2_1+(y2_1@W2), The processing of the MLP part is assigned to Accelerator2. ·y2_2=f(o2_2@W1), ·z2_2=o2_2+(y2_2@W2), The figure shows how the processes of q1_1_next, q1_2_next, a2_1, a2_2, o2_1, and o2_2 are assigned. Note that q1_1_next, q1_2_next, a2_1, a2_2, o2_1, and o2_2 are (L / 2) × C matrices.

[0130] By allocating processes in this way, the scheduling unit 302 can, for example, A process of communicating k1_1_next and v1_1_next, and k1_2_next and v1_2_next, which are calculated using data based on the next micro-batch as input data, between Accelerator1 and Accelerator2; A process of calculating q1_1_next to v2_1_next or a process of calculating q1_2_next to v2_2_next using data based on the next microbatch as input data; can be executed in parallel.

[0131] Furthermore, by allocating processes in this way, the scheduling unit 302 can, for example, A process of communicating k1_1_next and v1_1_next, and k1_2_next and v1_2_next, which are calculated using data based on the next micro-batch as input data, between Accelerator1 and Accelerator2; A process of calculating y2_1 and z2_1, or a process of calculating y2_2 and z2_2, using data based on the next micro-batch as input data; can be executed in parallel.

[0132] <Summary> As is clear from the above description, in the information processing device 120 according to the third embodiment, in the Decoder unit of the Transformer, in scheduling when performing training processing by sequence parallelism, - Data based on two consecutive micro-batches is used as a single input data. communication processing between accelerators as a result of executing a calculation process on the first input data in the attention unit; A calculation process for the second input data in the attention unit; The processes are assigned to a plurality of accelerators used in sequence parallelism so that the processes are executed in parallel. - Data based on two consecutive micro-batches is used as a single input data. communication processing between accelerators as a result of executing calculation processing on the second input data in the attention unit; A calculation process for the third input data in the attention unit, and a calculation process for the first input data in the MLP unit; The processes are assigned to a plurality of accelerators used in sequence parallelism so that the processes are executed in parallel. - Data based on two consecutive micro-batches is used as a single input data. communication processing between accelerators as a result of executing calculation processing on the third input data in the attention unit; A calculation process for the fourth input data in the attention unit, and a calculation process for the second input data in the MLP unit; The processes are assigned to a plurality of accelerators used in sequence parallelism so that the processes are executed in parallel.

[0133] Furthermore, in the server device according to the third embodiment, when a sequence-parallel training process is executed in the decoder unit of the transformer, - Data based on two consecutive micro-batches is used as a single input data. communication processing between accelerators as a result of executing a calculation process on the first input data in the attention unit; A calculation process for the second input data in the attention unit; are executed in parallel. - Data based on two consecutive micro-batches is used as a single input data. communication processing between accelerators as a result of executing calculation processing on the second input data in the attention unit; A calculation process for the third input data in the attention unit, and a calculation process for the first input data in the MLP unit; are executed in parallel. - Data based on two consecutive micro-batches is used as a single input data. communication processing between accelerators as a result of executing calculation processing on the third input data in the attention unit; A calculation process for the fourth input data in the attention unit, and a calculation process for the second input data in the MLP unit; are executed in parallel.

[0134] As a result, according to the third embodiment, the execution speed of the training process can be improved when executing the training process in sequence parallelism, compared to when the calculation process of the previous input data is executed and the communication process between the accelerators is completed before the calculation process of the next input data is executed.

[0135] In this embodiment, the Transformer decoder performs training in sequence parallel using data based on multiple microbatches as input data. However, if the Transformer includes an encoder, the encoder may perform training in sequence parallel using a similar method.

[0136] [Fourth embodiment] In the second embodiment, a scheduling example was described in which a Transformer decoder executes training processing in tensor parallelism using data based on multiple microbatches as input data. In the third embodiment, a scheduling example was described in which a Transformer decoder executes training processing in sequence parallelism using data based on multiple microbatches as input data.

[0137] In contrast, in the fourth embodiment, a scheduling will be described for the case where a training process is executed by tensor parallelism and sequence parallelism using data based on multiple microbatches as input data in the decoder unit of a Transformer. Note that the fourth embodiment will be described mainly focusing on the differences from the second and third embodiments.

[0138] <Specific examples of scheduling> (1) Overview of intra-layer scheduling FIG. 13 is a diagram illustrating an overview of scheduling by an information processing device according to a fourth embodiment. In FIG. 13, reference numerals 1310_1 to 1310_n denote multiple Decoder units constituting a Transformer. The example of FIG. 13 illustrates a state in which a worker with a worker name="Worker0" is assigned to training processing in the Decoder unit denoted by reference numeral 1310_1. In the example of FIG. 13, the worker is one server device (Server 0) having four accelerators. The Decoder unit denoted by reference numeral 1310_1 is divided into an Attention unit and an MLP unit. When data based on microbatches 1 and 2 is input as input data via a preprocessing unit, the calculations performed in the Attention unit of the Decoder unit denoted by reference numeral 1310_1 are as described in the second and third embodiments. Furthermore, the calculations performed in the MLP unit of the Decoder unit denoted by reference numeral 1310_1 are also as described in the second and third embodiments.

[0139] In the fourth embodiment, the scheduling unit 302 receives data based on the microbatches 1 and 2 as input data, and the Accelerators 1_1 to 2_2 perform the following: ·Attention Department, ·MLP Department, A schedule is generated to execute each calculation using tensor parallelism and sequence parallelism.

[0140] (2) Scheduling by the scheduling unit 302 Next, a schedule generated by the scheduling unit 302 of the information processing device 120 according to the fourth embodiment will be described. Figures 14A and 14B are first and second diagrams showing a specific example of scheduling by the scheduling unit of the information processing device according to the fourth embodiment.

[0141] As indicated by reference numeral 1410 in FIG. 14A, in order to execute tensor parallelism and sequence parallelism, the scheduling unit 302 divides data x1 (an L×C matrix) based on microbatch 1 in accordance with the number of accelerators (here, the number of divisions is 2). As a result, the scheduling unit 302 generates x1_1 and x1_2 (an (L / 2)×C matrix). At this time, the scheduling unit 302 schedules processing of data based on two microbatches (microbatches 1 and 2) as a single unit. Therefore, the scheduling unit 302 divides data x2 (an L×C matrix) based on microbatch 2 in accordance with the number of accelerators (here, the number of divisions is 2) to generate x2_1 and x2_2 (an (L / 2)×C matrix).

[0142] Next, as indicated by reference numeral 1411 in FIG. 14B, the scheduling unit 302 assigns processing to the Attention unit. The example of reference numeral 1411 assigns the following to Accelerator1_1: ·k1_1_1=x1_1@Wk_1, ·v1_1_1=x1_1@Wv_1, · k1_1_1, v1_1_1 sent to Accelerator2_1, Assign the processing to Accelerator2_1, ·k1_2_1=x1_2@Wk_1, ·v1_2_1=x1_2@Wv_1, · k1_2_1, v1_2_1 sent to Accelerator1_1, and assign the processing to Accelerator1_2. ·k1_1_2=x1_1@Wk_2, ·v1_1_2=x1_1@Wv_2, · k1_1_2, v1_1_2 sent to Accelerator2_2, Assign the processing to Accelerator2_2, ·k1_2_2=x1_2@Wk_2, ·v1_2_2=x1_2@Wv_2, · k1_2_2, v1_2_2 sent to Accelerator1_2, The figure shows how the processes of the above are assigned. Note that k1_1_1, k1_2_1, k1_1_2, k1_2_2, v1_1_1, v1_2_1, v1_1_2, and v1_2_2 are (L / 2) × (C / 2) matrices. k1_1_1 and k1_2_1 are collectively referred to as k1_1 (L × (C / 2) matrix), and v1_1_1 and v1_2_1 are collectively referred to as v1_1 (L × (C / 2) matrix). k1_1_2 and k1_2_2 are collectively referred to as k1_2 (L × (C / 2) matrix), and v1_1_2 and v1_2_2 are collectively referred to as v1_2 (L × (C / 2) matrix).

[0143] In addition, the example of code 1411 is Accelerator1_1, ·q1_1_1=x1_1@Wq_1, ·k2_1_1=x2_1@Wk_1, ·v2_1_1=x2_1@Wv_1, · k2_1_1, send v2_1_1 to Accelerator2_1, Assign the processing to Accelerator2_1, ·q1_2_1=x1_2@Wq_1, ·k2_2_1=x2_2@Wk_1, ·v2_2_1=x2_2@Wv_1, · k2_2_1, v2_2_1 sent to Accelerator1_1, and assign the processing to Accelerator1_2. ·q1_1_2=x1_1@Wq_2, ·k2_1_2=x2_1@Wk_2, ·v2_1_2=x2_1@Wv_2, · k2_1_2, send v2_1_2 to Accelerator2_2, Assign the processing to Accelerator2_2, ·q1_2_2=x1_2@Wq_2, ·k2_2_2=x2_2@Wk_2, ·v2_2_2=x2_2@Wv_2, · k2_2_2, v2_2_2 sent to Accelerator1_2, The figure shows how the processes of the above are assigned. Note that q1_1_1, q1_2_1, q1_1_2, q1_2_2, k2_1_1, k2_2_1, k2_1_2, k2_2_2, v2_1_1, v2_2_1, v2_1_2, and v2_2_2 are (L / 2) × (C / 2) matrices. k2_1_1 and k2_2_1 are collectively referred to as k2_1 (L × (C / 2) matrix), and k2_1_2 and k2_2_2 are collectively referred to as k2_2 (L × (C / 2) matrix). v2_1_1 and v2_2_1 are collectively referred to as v2_1 (L × (C / 2) matrix), and v2_1_2 and v2_2_2 are collectively referred to as v2_2 (L × (C / 2) matrix).

[0144] By allocating processes in this way, the scheduling unit 302 can, for example, A process of communicating k1_1_1 and v1_1_1, and k1_2_1 and v1_2_1, which are calculated using data based on the previous micro-batch as input data, between Accelerator1_1 and Accelerator2_1; A process of calculating k2_1_1 and v2_1_1, or a process of calculating k2_2_1 and v2_2_1, using data based on the next micro-batch as input data; In addition, the scheduling unit 302 can execute the following in parallel, for example: A process of communicating k1_1_2 and v1_1_2, and k1_2_2 and v1_2_2, which are calculated using data based on the previous micro-batch as input data, between Accelerator1_2 and Accelerator2_2; A process of calculating k2_1_2 and v2_1_2, or a process of calculating k2_2_2 and v2_2_2, using data based on the next micro-batch as input data; can be executed in parallel.

[0145] In addition, the example of reference numeral 1411 is set to Accelerator1_1 as the processing of the Attention section. ·q2_1_1=x2_1@Wq_1, ·k1_1=concat(k1_1_1,k1_2_1), ·v1_1=concat(v1_1_1,v1_2_1), ·a1_1_1=MultiHeadAttention(q1_1_1, k1_1, v1_1), ·o1_1_1=a1_1_1@Wo_1, Send o1_1_1 to Accelerator1_2, · o1_1=x1_1+o1_1_1+o1_1_2, The processing of the Attention part is assigned to Accelerator2_1. ·q2_2_1=x2_2@Wq_1, ·a1_2_1=MultiHeadAttention(q1_2_1, k1_1, v1_1), ·o1_2_1=a1_2_1@Wo_1, Send o1_2_1 to Accelerator2_2, ·o1_2=x1_2+o1_2_1+o1_2_2, The processing of the Attention part is assigned to Accelerator1_2. ·q2_1_2=x2_1@Wq_2, ·k1_2=concat(k1_1_2,k1_2_2), ·v1_2=concat(v1_1_2,v1_2_2), ·a1_1_2=MultiHeadAttention(q1_1_2, k1_2, v1_2), ·o1_1_2=a1_1_2@Wo_2, Send o1_1_2 to Accelerator1_1, · o1_1=x1_1+o1_1_1+o1_1_2, The processing of the Attention part is assigned to Accelerator2_2. ·q2_2_2=x2_2@Wq_2, ·k1_2=concat(k1_1_2,k1_2_2), ·v1_2=concat(v1_1_2,v1_2_2), ·a1_2_2=MultiHeadAttention(q1_2_2, k1_2, v1_2), ·o1_2_2=a1_2_2@Wo_2, Send o1_2_2 to Accelerator2_1, ·o1_2=x1_2+o1_2_1+o1_2_2, The figure shows how the processing of the above is assigned. Note that q2_1_1, q2_2_1, q2_1_2, and q2_2_2 are (L / 2) × (C / 2) matrices. a1_1_1, a1_2_1, a1_1_2, and a1_2_2 are (L / 2) × (C / 2) matrices. o1_1_1, o1_2_1, o1_1_2, and o1_2_2 are (L / 2) × C matrices. o1_1 and o1_2 are (L / 2) × C matrices.

[0146] By allocating processes in this way, the scheduling unit 302 can, for example, A process of communicating k2_1_1 and v2_1_1 calculated using data based on the next micro-batch as input data, and k2_2_1 and v2_2_1 between Accelerator1_1 and Accelerator2_1; A process of calculating a1_1_1 and o1_1_1, or a process of calculating a1_2_1 and o1_2_1 using data based on the previous micro-batch as input data; In addition, the scheduling unit 302 can execute the following in parallel, for example: A process of communicating k2_1_2 and v2_1_2, k2_2_2 and v2_2_2 calculated using data based on the next micro-batch as input data, between Accelerator1_2 and Accelerator2_2; A process of calculating a1_1_2 and o1_1_2, or a process of calculating a1_2_2 and o1_2_2 using data based on the previous micro-batch as input data; can be executed in parallel.

[0147] In addition, the example of reference numeral 1411 is set to Accelerator1_1 as the processing of the Attention section. ·k2_1=concat(k2_1_1,k2_2_1), ·v2_1=concat(v2_1_1,v2_2_1), ·a2_1_1=MultiHeadAttention(q2_1_1, k2_1, v2_1), ·o2_1_1=a2_1_1@Wo_1, · Send o2_1_1 to Accelerator1_2, ·o2_1=x2_1+o2_1_1+o2_1_2, The processing of the Attention part is assigned to Accelerator2_1. ·k2_1=concat(k2_1_1,k2_2_1), ·v2_1=concat(v2_1_1,v2_2_1), ·a2_2_1=MultiHeadAttention(q2_2_1, k2_1, v2_1), ·o2_2_1=a2_2_1@Wo_1, · Send o2_2_1 to Accelerator2_2, ·o2_2=x2_2+o2_2_1+o2_2_2, The processing of the Attention part is assigned to Accelerator1_2. ·k2_2=concat(k2_1_2,k2_2_2), ·v2_2=concat(v2_1_2,v2_2_2), ·a2_1_2=MultiHeadAttention(q2_1_2, k2_2, v2_2), ·o2_1_2=a2_1_2@Wo_2, Send o2_1_2 to Accelerator1_1, ·o2_1=x2_1+o2_1_1+o2_1_2, The processing of the Attention part is assigned to Accelerator2_2. ·k2_2=concat(k2_1_2,k2_2_2), ·v2_2=concat(v2_1_2,v2_2_2), ·a2_2_2=MultiHeadAttention(q2_2_2, k2_2, v2_2), ·o2_2_2=a2_2_2@Wo_2, Send o2_2_2 to Accelerator2_1, ·o2_2=x2_2+o2_2_1+o2_2_2, The figure shows how the processing of the above is assigned. Note that a2_1_1, a2_2_1, a2_1_2, and a2_2_2 are (L / 2) × (C / 2) matrices. o2_1_1, o1_2_1, o2_1_2, and o2_2_2 are (L / 2) × C matrices. o2_1 and o2_2 are (L / 2) × C matrices.

[0148] By allocating processes in this way, the scheduling unit 302 can, for example, A process of communicating o1_1_1 and o1_1_2 calculated using data based on the previous micro-batch as input data between accelerators (between Accelerator1_1 and Accelerator1_2); A process of calculating a2_1_1 and o2_1_1, or a process of calculating a2_1_2 and o2_1_2 using data based on the next micro-batch as input data; In addition, the scheduling unit 302 can execute the following in parallel, for example: A process of communicating o1_2_1 and o1_2_2 calculated using data based on the previous micro-batch as input data between accelerators (between Accelerator2_1 and Accelerator2_2); A process of calculating a2_2_1 and o2_2_1, or a process of calculating a2_2_2 and o2_2_2, using data based on the next micro-batch as input data; can be executed in parallel.

[0149] In addition, the example of reference numeral 1411 is a process of the MLP unit, in which Accelerator1_1: ·y1_1_1=f(o1_1@W1_1), ·z1_1_1=o1_1+(y1_1_1@W2_1), Send z1_1_1 to Accelerator1_2, ·z1_1=z1_1_1+z1_1_2, The processing of the MLP part is assigned to Accelerator2_1. ·y1_2_1=f(o1_2@W1_1), ·z1_2_1=o1_2+(y1_2_1@W2_1), Send z1_2_1 to Accelerator2_2, ·z1_2=z1_2_1+z1_2_2, The processing of the MLP part is assigned to Accelerator1_2. ·y1_1_2=f(o1_1@W1_2), ·z1_1_2=o1_1+(y1_1_2@W2_2), Send z1_1_2 to Accelerator1_1, ·z1_1=z1_1_1+z1_1_2, The processing of the MLP part is assigned to Accelerator2_2. ·y1_2_2=f(o1_2@W1_2), ·z1_2_2=o1_2+(y1_2_2@W2_2), Send z1_2_2 to Accelerator2_1, ·z1_2=z1_2_1+z1_2_2, The figure shows how the processing of z1_1 and z1_2 is assigned. Note that z1_1 and z1_2 are (L / 2) × C matrices.

[0150] By allocating processes in this way, the scheduling unit 302 can, for example, A process of communicating o2_1_1 and o2_1_2 calculated using data based on the next micro-batch as input data between Accelerator1_1 and Accelerator1_2; A process of calculating y1_1_1 and z1_1_1, or a process of calculating y1_1_2 and z1_1_2 using data based on the previous micro-batch as input data; In addition, the scheduling unit 302 can execute the following in parallel, for example: A process of communicating o2_2_1 and o2_2_2 calculated using data based on the next micro-batch as input data between Accelerator1_1 and Accelerator1_2; A process of calculating y1_2_1 and z1_2_1, or a process of calculating y1_2_2 and z1_2_2 using data based on the previous micro-batch as input data; can be executed in parallel.

[0151] In addition, the example of reference numeral 1411 is an MLP process in Accelerator1_1. ·y2_1_1=f(o2_1@W1_1), ·z2_1_1=o2_1+(y2_1_1@W2_1), Send z2_1_1 to Accelerator1_2, ·z2_1=z2_1_1+z2_1_2, The processing of the MLP is assigned to Accelerator2_1. ·y2_2_1=f(o2_2@W1_1), ·z2_2_1=o2_2+(y2_2_1@W2_1), Send z2_2_1 to Accelerator2_2, ·z2_2=z2_2_1+z2_2_2, The processing of the MLP is assigned to Accelerator1_2. ·y2_1_2=f(o2_1@W1_2), ·z2_1_2=o2_1+(y2_1_2@W2_2), Send z2_1_2 to Accelerator1_1, ·z2_1=z2_1_1+z2_1_2, The processing of the MLP is assigned to Accelerator2_2. ·y2_2_2=f(o2_2@W1_2), ·z2_2_2=o2_2+(y2_2_2@W2_2), Send z2_2_2 to Accelerator2_1, ·z2_2=z2_2_1+z2_2_2, The figure shows how the processing of z2_1 and z2_2 is assigned. Note that z2_1 and z2_2 are (L / 2) × C matrices.

[0152] By allocating processes in this way, the scheduling unit 302 can, for example, A process of communicating z1_1_1 and z1_1_2 calculated using data based on the previous micro-batch as input data between Accelerator1_1 and Accelerator1_2; A process of calculating y2_1_1 and z2_1_1, or a process of calculating y2_1_2 and z2_1_2 using data based on the next micro-batch as input data; In addition, the scheduling unit 302 can execute the following in parallel, for example: A process of communicating z1_2_1 and z1_2_2 calculated using data based on the previous micro-batch as input data between Accelerator2_1 and Accelerator2_2; A process of calculating y2_2_1 and z2_2_1, or a process of calculating y2_2_2 and z2_2_2, using data based on the next micro-batch as input data; can be executed in parallel.

[0153] <Summary> As is clear from the above description, in the information processing device 120 according to the fourth embodiment, in the Decoder unit of the Transformer, in scheduling when the training process is executed by tensor parallelism and sequence parallelism, - Data based on two consecutive micro-batches is used as a single input data. communication processing between accelerators as a result of executing a calculation process on the first input data in the attention unit; A calculation process for the second input data in the attention unit; The processing is assigned to a plurality of accelerators used for tensor parallelism and sequence parallelism so that the above-mentioned processes are executed in parallel. - Data based on two consecutive micro-batches is used as a single input data. communication processing between accelerators as a result of executing calculation processing on the second input data in the attention unit; A calculation process for the first input data in the MLP unit; The processing is assigned to a plurality of accelerators used for tensor parallelism and sequence parallelism so that the above-mentioned processes are executed in parallel. - Data based on two consecutive micro-batches is used as a single input data. A communication process between accelerators for a result of executing a calculation process on the first input data in the MLP unit; A calculation process for the second input data in the MLP unit; The processing is assigned to a plurality of accelerators used for tensor parallelism and sequence parallelism so that the above-mentioned processes are executed in parallel.

[0154] Furthermore, in the server device according to the fourth embodiment, when a tensor parallel and sequence parallel training process is performed in the decoder unit of the transformer, - Data based on two consecutive micro-batches is used as a single input data. communication processing between accelerators as a result of executing a calculation process on the first input data in the attention unit; A calculation process for the second input data in the attention unit; are executed in parallel. - Data based on two consecutive micro-batches is used as a single input data. communication processing between accelerators as a result of executing calculation processing on the second input data in the attention unit; A calculation process for the first input data in the MLP unit; are executed in parallel. - Data based on two consecutive micro-batches is grouped together, A communication process between accelerators for a result of executing a calculation process on the first input data in the MLP unit; A calculation process for the second input data in the MLP unit; are executed in parallel.

[0155] As a result, according to the fourth embodiment, it is possible to improve the execution speed when executing the training process using tensor parallelism and sequence parallelism.

[0156] In this embodiment, the case where the Transformer decoder executes training processing using tensor parallelism and sequence parallelism with data based on multiple microbatches as input data has been described. However, if the Transformer has an encoder, the encoder may execute training processing using tensor parallelism and sequence parallelism in a similar manner.

[0157] [Fifth embodiment] In the above embodiments, specific examples of forward calculation have been mainly described, but the same applies to specific examples of backward data calculation and backward weight calculation. However, in a case where there is no recalculation in backward weight calculation, for example, Accelerator 1 ·gW1_1=gy1_1.transpose@x1_1, ·gW1_2=gy1_2.transpose@x1_2, The processing is assigned to Accelerator2. ·gW2_1=gy2_1.transpose@x2_1, ·gW2_2=gy2_2.transpose@x2_2, When the process is assigned, no communication occurs. Note that x1_1 and x1_2 represent the tensors for the first and second parameters in Accelerator1, and x2_1 and x2_2 represent the tensors for the first and second parameters in Accelerator2. Also, gW represents the weight gradient, gy represents the output gradient, and x represents the input data.

[0158] When training using tensor parallelism, either gy or x must be duplicated across accelerators. For example, in a matrix multiplication near the input of the MLP unit, x must be duplicated on each accelerator used for tensor parallelism.

[0159] In this case, in order to avoid storing tensors for a long time, a scheduling method can be considered in which the tensors are stored in a distributed manner and generated by performing calculations when needed.

[0160] In such cases, the backward weight calculation involves ·x1=Allgather(x1_1), ·gW1_1=gy1_1.transpose@x1, As shown above, the process includes a communication process for collecting tensors and a calculation process for calculating gradients. Therefore, as in the above embodiments, by treating data based on two micro-batches whose processing order is consecutive as a single set of input data, Processing can be assigned to multiple accelerators used for tensor parallelism so that communication processing using data based on microbatch 2 as input data and computation processing using data based on microbatch 1 as input data are executed in parallel. Processing can be assigned to multiple accelerators used for tensor parallelism so that communication processing using data based on microbatch 1 as input data and computation processing using data based on microbatch 2 as input data are executed in parallel.

[0161] As a result, the execution speed can be improved when performing training processing using tensor parallelism.

[0162] [Sixth embodiment] In the above embodiments, tensor parallelism and sequence parallelism have been cited as examples of intra-layer parallelism. However, other parallelism, such as expert parallelism in Mixture of Experts (MoE), can be cited. An expert parallelism system is configured to switch the DNN (Deep Neural Network) to be used for each input token. Therefore, by scheduling the transmission of the token to the destination and the processing of the token at the destination in parallel, the execution speed can be improved.

[0163] Fig. 15 is a diagram showing an overview of scheduling by an information processing device according to a sixth embodiment, in which each DNN, which is a model to be trained, is trained by expert parallelism. The example of Fig. 15 shows that a token for expert0 is sent to DNN0 and used in the training process of DNN0, and a token for expert1 is sent to DNN1 and used in the training process of DNN1. The example of Fig. 15 also shows that a token for expert2 is sent to DNN2 and used in the training process of DNN2, and a token for expert3 is sent to DNN3 and used in the training process of DNN3.

[0164] In this way, in the case of expert parallelism, the sending process of each token corresponding to each expert and the calculation process of each expert are executed. Therefore, by scheduling two tokens corresponding to the same expert and having consecutive processing orders as a group, Token sending process, Expert calculation processing and can be executed in parallel.

[0165] Although the above explanation did not mention the hardware configuration, the Router that sends the token and the DNN that performs the calculation processing of each expert may be realized in the same worker or in separate workers.

[0166] Furthermore, DNN0 to DNN3 may be realized in the same worker or in separate workers. When they are realized in separate workers, for example, DNN0 and DNN1 may be realized in a worker with the worker name "Worker0", and DNN2 and DNN3 may be realized in a worker with the worker name "Worker1".

[0167] [Seventh embodiment] In the first embodiment described above, the scheduling unit 302 schedules processing of data based on two microbatches as a single unit. However, the target of scheduling as a single unit is not limited to data based on two microbatches. Any other input data may be scheduled as a single unit. FIG. 16 is a diagram showing a specific example of scheduling by a scheduling unit of an information processing device according to the seventh embodiment. The difference from FIG. 6 described in the first embodiment above is that, in the case of FIG. 6, data x1, etc. is data based on a microbatch, but in the case of FIG. 16, data x1, etc. is any input data other than data based on a microbatch. Another difference from FIG. 6 is that, in the case of FIG. 6, scheduling of processing of data x1 and x2 is shown, but in FIG. 16, scheduling of processing of data x1 to x3 is shown.

[0168] As indicated by reference numeral 1610, the scheduling unit 302 generates input data x1 to x3 (L×C matrix) in accordance with the number of accelerators (here, the number of generated data=2) in order to execute tensor parallelism.

[0169] Furthermore, as indicated by reference numeral 1610, in order to execute tensor parallelism, the scheduling unit 302 divides the weight parameters W1 (C×4C matrix) and W2 (4C×C matrix) in accordance with the number of accelerators (here, the number of divisions is 2). This makes it possible to generate divided W1_1, W1_2 (C×2C matrix) and W2_1, W2_2 (2C×C matrix) to be assigned to each of the multiple accelerators.

[0170] Next, as indicated by reference numeral 1611, the scheduling unit 302 assigns processing to Accelerators 1 and 2. In the example indicated by reference numeral 1611, y1_1=f(x1@W1_1), and ·z1_1=y1_1@W2_1, and assign the processing to Accelerator2. y1_2=f(x1@W1_2), and ·z1_2=y1_2@W2_2, The figure shows how the above processing is assigned. Note that y1_1 and y1_2 are L×2C matrices, and z1_1 and z1_2 are L×C matrices.

[0171] Moreover, the example of reference numeral 1611 shows how z1 (L×C matrix) is calculated by adding z1_1 (L×C matrix) and z1_2 (L×C matrix) calculated by processing in Accelerator 1 and 2. For example, when z1_1 (L×C matrix) and z1_2 (L×C matrix) are added in Accelerator 1 and 2, z1_1 (L×C matrix) is transmitted from Accelerator 1 to Accelerator 2. Furthermore, z1_2 (L×C matrix) is transmitted from Accelerator 2 to Accelerator 1. In other words, communication processing occurs between the accelerators.

[0172] Here, the scheduling unit 302 schedules the accelerators so that they execute the calculation processes for the next input data in parallel while communication processes are being executed between the accelerators.

[0173] In the example of reference numeral 1611, while communication processing is being executed between accelerators, y2_1=f(x2@W1_1), and ·z2_1=y2_1@W2_1, and assign the processing to Accelerator2. y2_2=f(x2@W1_2), and ·z2_2=y2_2@W2_2, The figure shows how the above processing is assigned. Note that y2_1 and y2_2 are L×2C matrices, and z2_1 and z2_2 are L×C matrices.

[0174] Moreover, the example of reference numeral 1611 shows how z2 (L×C matrix) is calculated by adding z2_1 (L×C matrix) and z2_2 (L×C matrix) calculated by processing in Accelerator 1 and 2. For example, when z2_1 (L×C matrix) and z2_2 (L×C matrix) are added in Accelerator 1 and 2, z2_1 (L×C matrix) is transmitted from Accelerator 1 to Accelerator 2. Furthermore, z2_2 (L×C matrix) is transmitted from Accelerator 2 to Accelerator 1. In other words, communication processing occurs between the accelerators.

[0175] Here, the scheduling unit 302 schedules the accelerators so that they execute processing on the next input data in parallel while communication processing is being executed between the accelerators.

[0176] In the example of reference numeral 1611, while communication processing is being executed between accelerators, y3_1=f(x3@W1_1), and ·z3_1=y3_1@W2_1, and assign the processing to Accelerator2. y3_2=f(x3@W1_2), and ·z3_2=y3_2@W2_2, The figure shows how the above processing is assigned. Note that y3_1 and y3_2 are L×2C matrices, and z3_1 and z3_2 are L×C matrices.

[0177] In this way, the scheduling unit 302 schedules the processing of two consecutive input data as a single unit. y=f(x@W1) z=y@W2 This makes it possible to execute the calculation process and the communication process between the accelerators in parallel. As a result, the scheduling unit 302 can improve the execution speed when executing the training process using tensor parallelism.

[0178] [Other embodiments] In the above fourth embodiment, the model on which the training process using sequence parallelism is performed is described as a Transformer, but the model on which the training process using sequence parallelism is performed is not limited to a Transformer and may be a NN.

[0179] In addition, in each of the above embodiments, a microbatch has been described as an example of the processing unit (batch) of training data executed by each worker during training processing, but the batch of training data is not limited to a microbatch and may be a minibatch. Furthermore, one batch may be one of multiple divisions of training data, or may include one or more data included in the training data.

[0180] In the fifth embodiment, the backward calculation is divided into a backward data calculation and a backward weight calculation. However, in the scheduling of some workers (for example, a worker with a worker name of "Worker0"), the backward data calculation and the backward weight calculation may be scheduled as a single unit without being divided.

[0181] Furthermore, in each of the above embodiments, the case where the information processing device 120 applies the scheduling method to the training process has been described, but the information processing device 120 may also apply the scheduling method to processes other than the training process. In other words, the information processing device 120 may also apply the scheduling method to data based on data other than training data.

[0182] In addition, in each of the above embodiments, when executing the training process, data based on two micro-batches is grouped together, and the communication process between accelerators after the calculation process for the first input data and the calculation process for the second input data are executed in parallel. However, this execution method is not limited to when executing the training process, and may also be applied when executing the inference process. For example, when multiple sets of input data are input in the inference process, -communication processing between accelerators after calculation processing in the accelerator of the first input data; In the inference process, a calculation process is performed in the accelerator on second input data for the first input data; may be executed in parallel, thereby improving the execution speed when executing the inference process.

[0183] In this way, the parallel processing described in each of the above embodiments (tensor parallelism, sequence parallelism, expert parallelism, or a combination of these three parallelisms) is not limited to training processing, but may also be applied to inference processing. Furthermore, the parallel processing referred to here includes intra-layer parallelism, which refers to processing in which the calculation of one layer is performed by multiple workers.

[0184] Furthermore, in each of the above embodiments, examples have been given of communication processing and calculation processing being executed in parallel, but which communication processing and which calculation processing are executed in parallel are arbitrary and are not limited to the examples given in each of the above embodiments.

[0185] In each of the above embodiments, the model used in the training process or the inference process may be a machine learning model. The machine learning model here may be, for example, a generative model, a base model, or a neural network that generates various types of data such as audio, images, and videos. The machine learning model may also be multimodal.

[0186] In addition, in each of the above embodiments, data based on two microbatches is grouped together, one of which is defined as first input data and the other as second input data. However, the definition of the first input data and the second input data is not limited thereto. For example, the first input data and the second input data may be data used in the model training process or data used in the inference process. Furthermore, the units, divisions, etc. of the first input data and the second input data may be arbitrarily determined within the context. Furthermore, the first input data and the second input data may be input data obtained by dividing other first input data and other second input data, respectively. Furthermore, the first input data and the second input data may each be separate input data. Furthermore, the first input data and the second input data may each be data for which calculation processing is performed independently. Furthermore, the first input data and the second input data may be any data that can be input to a model. When the model is a large-scale language model, the first input data and the second input data may be text data, as a non-limiting example.

[0187] In addition, although the above embodiments have mentioned weight parameters among the model parameters, the model parameters used in parallel processing may include biases and other normalization parameters in addition to weight parameters. Note that when dividing the model parameters in parallel processing, the division may be either equal or unequal.

[0188] In the above embodiments, the information processing device 120 is provided as a separate entity from the server device group 110. However, the information processing device 120 may be integrated with the server device group 110.

[0189] Specifically, all of the functions of the information processing device 120 may be implemented by some of the servers in the server device group 110. That is, the information processing system 100 may have N server device groups 110 and one information processing device 120, or (N-1) server device groups 110 and one server device. Alternatively, the information processing device 120 itself may be a worker or a part of a worker. Alternatively, the information processing system 100 may be configured by a plurality of server devices (server device groups 110), or may be configured by an information processing device having a plurality of processors.

[0190] Furthermore, in each of the above embodiments, the information processing system 100 has been described as including one information processing device 120, but the information processing device 120 may be configured with a plurality of devices.

[0191] In this specification (including the claims), when the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used, it includes any of a, b, c, ab, ac, bc, or abc. It may also include multiple instances of any element, such as aa, abb, aabbcc, etc. Furthermore, it also includes the addition of elements other than the enumerated elements (a, b, and c), such as having d, as in abcd.

[0192] Furthermore, in this specification (including claims), when expressions such as "using data as input / based on / according to / in response to" (including similar expressions) are used, unless otherwise specified, this includes cases where various data itself is used as input, or where various data that has been processed in some way (e.g., noise-added, normalized, intermediate representation of various data, etc.) is used as input. Furthermore, when it is stated that a result is obtained "based on / according to / in response to data," this includes cases where the result is obtained based solely on the data in question, as well as cases where the result is obtained as a result of being influenced by other data, factors, conditions, and / or states other than the data in question. Furthermore, when it is stated that "data is output," unless otherwise specified, this includes cases where various data itself is used as output, or where various data that has been processed in some way (e.g., noise-added, normalized, intermediate representation of various data, etc.) is output.

[0193] Furthermore, when the terms "connected" and "coupled" are used in this specification (including the claims), they are intended as open-ended terms that include any of direct connection / coupling, indirect connection / coupling, electrically connection / coupling, communicatively connection / coupling, functionally connection / coupling, and physically connection / coupling. These terms should be interpreted appropriately depending on the context in which they are used, but any form of connection / coupling that is not intentionally or naturally excluded should be interpreted as being included in these terms without any restrictions.

[0194] Furthermore, in this specification (including the claims), when the expression "A configured to B" is used, it may include the physical structure of element A having a configuration capable of performing operation B, and the permanent or temporary setting / configuration of element A being configured / set to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and is configured to actually perform operation B by setting a permanent or temporary program (instruction). Furthermore, if element A is a dedicated processor or dedicated arithmetic circuit, it is sufficient that the circuit structure of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0195] Furthermore, when words implying containing or possessing (such as "comprising / including" and "having") are used in this specification (including the claims), they are intended to be open-ended terms that include cases where things other than the object indicated by the object of the term are contained or possessed. When the object of such words implying containing or possessing does not specify a quantity or suggests a singular number (such as an expression using the article "a" or "an"), the expression should be construed as not being limited to a specific number.

[0196] Furthermore, in this specification (including the claims), even if expressions such as "one or more" or "at least one" are used in some places and expressions that do not specify a quantity or suggest a singular number (expressions using the articles "a" or "an") are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or suggest a singular number (expressions using the articles "a" or "an") should be interpreted as not necessarily being limited to a specific number.

[0197] Furthermore, if a particular advantage / result is described in this specification as being obtained from a particular configuration of an embodiment, it should be understood that the same advantage / result can also be obtained from one or more other embodiments having the same configuration, unless otherwise stated. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or states, and that the effect is not necessarily obtained by the configuration. The effect is merely obtained by the configuration described in the embodiment when various factors, conditions, and / or states are satisfied, and the effect does not necessarily occur in a claimed invention that defines the same or a similar configuration.

[0198] Furthermore, in this specification (including claims), when multiple pieces of hardware perform a predetermined process, the pieces of hardware may cooperate to perform the predetermined process, or some of the hardware may perform all of the predetermined process. Furthermore, some of the hardware may perform part of the predetermined process, and other hardware may perform the rest of the predetermined process. In this specification (including claims), when an expression such as "one or more pieces of hardware perform a first process, and the one or more pieces of hardware perform a second process" is used, the hardware performing the first process and the hardware performing the second process may be the same or different. In other words, it is sufficient that the hardware performing the first process and the hardware performing the second process are included in the one or more pieces of hardware. Note that hardware may include an electronic circuit, a device including an electronic circuit, or the like.

[0199] Furthermore, in this specification (including the claims), when multiple storage devices (memories) store data, each of the multiple storage devices (memories) may store only a portion of the data, or may store the entire data.

[0200] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, partial deletions, etc. are possible within the scope of the conceptual idea and spirit of the present invention derived from the content defined in the claims and their equivalents. For example, in all of the above-described embodiments, the numerical values ​​used in the explanations are shown as examples and are not limited to these. Furthermore, the order of each operation in the embodiments is shown as an example and is not limited to these.

[0201] The disclosed technology may take the following forms as described below. (Appendix 1) An information processing system comprising a plurality of memories and a plurality of processors, and performing parallel processing using a model, The plurality of processors executing a communication process resulting from execution of a calculation process using at least a part of the model for first input data and a calculation process using at least a part of the model for second input data, with at least a part of their processing times overlapping; Information processing system. (Appendix 2) The parallel processing is processing using intra-layer parallelism. 10. The information processing system of claim 1. (Appendix 3) The first input data and the second input data are data based on two consecutive micro-batches in a training process of the model. 10. The information processing system according to claim 1 or 2. (Appendix 4) The first input data and the second input data are data used in an inference process using the model. 10. The information processing system according to claim 1 or 2. (Appendix 5) The information processing system executes at least tensor parallel processing as the parallel processing, each processor executes a calculation process using each model parameter after division and at least one of the first input data or the second input data; The divided model parameters are obtained by dividing the model parameters of the model based on the number of processors used for the tensor parallelism. An information processing system according to any one of Supplementary Note 1 to Supplementary Note 4. (Appendix 6) a first processor performs a calculation process using the first model parameters after the division and the first input data; a second processor performs a calculation process using the second model parameters after the division and the first input data; executing a communication process for transmitting and receiving a result of the calculation process performed by the first processor and a result of the calculation process performed by the second processor between the first processor and the second processor; the first processor executes a calculation process using the divided first model parameters and the second input data, overlapping the calculation process with at least a part of a processing time of the communication process; the second processor executes a calculation process using the divided second model parameters and the second input data, overlapping at least a part of the processing time of the communication process; 6. The information processing system according to claim 5. (Appendix 7) The information processing system executes at least sequential parallel processing as the parallel processing, Dividing the first input data and the second input data based on the number of processors used in the sequential parallel processing; each processor executes a calculation process using at least one of the divided first input data or the divided second input data and model parameters of the model; An information processing system according to any one of Supplementary Note 1 to Supplementary Note 4. (Appendix 8) a first processor performs a calculation process using one of the first input data after the division and the model parameters; a second processor performs a calculation process using another piece of the first input data after the division and the model parameters; executing a communication process for transmitting and receiving a result of the calculation process performed by the first processor and a result of the calculation process performed by the second processor between the first processor and the second processor; the first processor executes a calculation process using one of the second input data after the division and the model parameters, overlapping at least a part of a processing time of the communication process; the second processor executes a calculation process using another piece of the divided second input data and the model parameters, overlapping at least a part of the processing time of the communication process; 8. The information processing system of claim 7. (Appendix 9) The information processing system that executes processing by a combination of tensor parallelism and sequence parallelism as the parallel processing, each processor performs a calculation process using each of the divided model parameters and at least one of the divided first input data and the divided second input data; each of the divided model parameters is obtained by dividing the model parameters of the model based on the number of processors used in the combination of the tensor parallelism and the sequence parallelism, the divided first input data and the divided second input data are obtained by dividing the first input data and the second input data based on the number of processors used in the combination of tensor parallelism and sequence parallelism. 10. The information processing system of claim 1. (Appendix 10) The model is a neural network, The calculation processing and the communication processing are calculation processing and communication processing in the neural network. An information processing system according to any one of appendices 1 to 9. (Appendix 11) the neural network includes a Transformer; The calculation process and the communication process include at least an attention section of the encoder section included in the transformer; an MLP unit of the Encoder unit; an MLP unit of a decoder unit included in the transformer; Calculation processing and communication processing in any one of the above. 11. The information processing system of claim 10. (Appendix 12) An information processing system comprising a plurality of memories and a plurality of processors, and executing expert parallel processing using a plurality of experts, The plurality of processors a transmission process for transmitting each token corresponding to each expert to each expert and a calculation process for each token in each expert are executed so that at least a part of the processing time of each process overlaps; Information processing system. (Appendix 13) An information processing device that includes a plurality of memories and a plurality of processors and schedules execution of parallel processing using a model, The plurality of processors scheduling a communication process resulting from execution of a calculation process using at least a part of the model for first input data and a calculation process using at least a part of the model for second input data so that at least a part of their processing times overlap; Information processing device. (Appendix 14) A plurality of processors of an information processing device that executes parallel processing using a model, a communication process resulting from the execution of a calculation process using at least a part of the model for first input data and a calculation process using at least a part of the model for second input data are executed with at least a part of their processing times overlapping; Information processing methods. (Appendix 15) a plurality of processors of an information processing device that schedules execution of parallel processing using a model, scheduling a communication process resulting from execution of a calculation process using at least a part of the model for first input data and a calculation process using at least a part of the model for second input data so that at least a part of their processing times overlap; Scheduling methods. (Appendix 16) A processor of an information processing device that performs parallel processing using a model, An information processing program for executing a communication process resulting from a calculation process using at least a part of the model for first input data and a calculation process using at least a part of the model for second input data, with at least a part of the processing time of each process overlapping. (Appendix 17) A processor of an information processing device that schedules execution of parallel processing using a model, scheduling a communication process resulting from execution of a calculation process using at least a part of the model for first input data and a calculation process using at least a part of the model for second input data so that at least a part of their processing times overlap; A scheduling program for executing processes.

Claims

1. An information processing system comprising a plurality of memories and a plurality of processors, and performing parallel processing using a model, The plurality of processors executing a communication process resulting from execution of a calculation process using at least a part of the model for first input data and a calculation process using at least a part of the model for second input data, with at least a part of their processing times overlapping; Information processing system.

2. The parallel processing is processing using intra-layer parallelism. The information processing system according to claim 1 .

3. The first input data and the second input data are data based on two consecutive micro-batches in a training process of the model. The information processing system according to claim 1 .

4. the first input data and the second input data are data used in an inference process using the model; The information processing system according to claim 1 .

5. The information processing system executes at least tensor parallel processing as the parallel processing, each processor executes a calculation process using each model parameter after division and at least one of the first input data and the second input data; The divided model parameters are obtained by dividing the model parameters of the model based on the number of processors used for the tensor parallelism. The information processing system according to claim 1 .

6. the first processor executes a calculation process using the first model parameters after division and the first input data; a second processor performs a calculation process using the second model parameters after the division and the first input data; executing a communication process for transmitting and receiving a result of the calculation process performed by the first processor and a result of the calculation process performed by the second processor between the first processor and the second processor; the first processor executes a calculation process using the divided first model parameters and the second input data, overlapping at least a part of a processing time of the communication process; the second processor executes a calculation process using the divided second model parameters and the second input data, overlapping at least a part of a processing time of the communication process; The information processing system according to claim 5 .

7. The information processing system executes at least sequential parallel processing as the parallel processing, Dividing the first input data and the second input data based on the number of processors used in the sequential parallel processing; each processor executes a calculation process using at least one of the divided first input data or the divided second input data and model parameters of the model; The information processing system according to claim 1 .

8. a first processor performs a calculation process using one of the first input data after the division and the model parameters; a second processor performs a calculation process using another piece of the first input data after the division and the model parameters; executing a communication process for transmitting and receiving a result of the calculation process performed by the first processor and a result of the calculation process performed by the second processor between the first processor and the second processor; the first processor executes a calculation process using one of the second input data after division and the model parameters, overlapping at least a part of a processing time of the communication process; the second processor executes a calculation process using another piece of the divided second input data and the model parameters, overlapping at least a part of the processing time of the communication process; The information processing system according to claim 7 .

9. The information processing system that executes processing by a combination of tensor parallelism and sequence parallelism as the parallel processing, each processor executes a calculation process using each of the divided model parameters and at least one of the divided first input data and the divided second input data; each of the divided model parameters is obtained by dividing the model parameters of the model based on the number of processors used in the combination of the tensor parallelism and the sequence parallelism; the divided first input data and the divided second input data are obtained by dividing the first input data and the second input data based on the number of processors used in the combination of tensor parallelism and sequence parallelism; The information processing system according to claim 1 .

10. The model is a neural network, The calculation processing and the communication processing are calculation processing and communication processing in the neural network. The information processing system according to any one of claims 1 to 9.

11. the neural network includes a Transformer, The calculation process and the communication process include at least an Attention section of an Encoder section included in the Transformer; an MLP unit of the Encoder unit; an MLP unit of a decoder unit included in the transformer; Calculation processing and communication processing in any one of The information processing system according to claim 10.

12. An information processing system comprising a plurality of memories and a plurality of processors, and executing expert parallel processing using a plurality of experts, The plurality of processors a transmission process for transmitting each token corresponding to each expert to each expert and a calculation process for each token in each expert are executed so that at least a part of the processing time of each process overlaps; Information processing system.

13. An information processing device that includes a plurality of memories and a plurality of processors and schedules execution of parallel processing using a model, The plurality of processors scheduling a communication process resulting from execution of a calculation process using at least a part of the model for first input data and a calculation process using at least a part of the model for second input data so that at least a part of their processing times overlap; Information processing device.

14. A plurality of processors of an information processing device that executes parallel processing using a model, a communication process resulting from the execution of a calculation process using at least a part of the model for first input data and a calculation process using at least a part of the model for second input data are executed with at least a part of their processing times overlapping; Information processing methods.

15. a plurality of processors of an information processing device that schedules execution of parallel processing using a model, scheduling a communication process resulting from execution of a calculation process using at least a part of the model for first input data and a calculation process using at least a part of the model for second input data so that at least a part of their processing times overlap; Scheduling methods.

16. A processor of an information processing device that performs parallel processing using a model, An information processing program for executing a communication process resulting from a calculation process using at least a part of the model for first input data and a calculation process using at least a part of the model for second input data, with at least a part of the processing time of each process overlapping.

17. A processor of an information processing device that schedules execution of parallel processing using a model, scheduling a communication process resulting from execution of a calculation process using at least a part of the model for first input data and a calculation process using at least a part of the model for second input data so that at least a part of their processing times overlap; A scheduling program for executing processes.

Images