Methods and systems for performing multi-device based inference for large-scale language models
The multi-device-based inference system addresses the inefficiencies in large language model inference by employing intra-layer parallelism and simultaneous matrix operations, enhancing scalability and reducing latency.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- HYPERACCEL CO LTD
- Filing Date
- 2024-05-20
- Publication Date
- 2026-07-07
AI Technical Summary
The computational costs associated with training and inference of large language models are rising due to the lack of efficient accelerators for inference operations, despite the increasing scale of these models, and existing multi-device solutions are not optimized for inference tasks.
A multi-device-based inference execution system that separates matrices of each layer column by column using intra-layer parallelism, enabling simultaneous matrix multiplication, data transmission, and reception, and synchronization to reduce communication overhead and latency.
The system achieves high scalability and efficiency in inference execution by reducing communication overhead and latency through simultaneous matrix multiplication and All Gather operations, optimizing performance on multi-device infrastructure.
Smart Images
Figure 2026522371000001_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to a method and system for performing multi-device-based inference for large language models.
Background Art
[0002] A large language model (LLM) is a model that calculates the probability distribution of the existence of natural language sentences using an artificial neural network, and recently it has been widely used in language-related tasks such as answering questions and translation like ChatGPT.
[0003] Although the scale of large language models is increasing rapidly, there is a problem that the computational costs associated with training and inference of large language models are rising. As the scale of language models increases, the method of using multiple accelerators for their operations has become common. While GPUs (Graphics Processing Units) are efficiently used as accelerators for training, there is still no accelerator for efficiently advancing inference.
Summary of the Invention
Problems to be Solved by the Invention
[0004] It is possible to provide an inference execution method and system as a network technology that can effectively execute multi-device-based operations.
[0005] The technical problems of the present invention are not limited to those described above, and other technical problems not described will be clearly understood by those skilled in the art from the following description.
Means for Solving the Problems
[0006] The present invention provides a multi-device-based inference execution system, comprising a plurality of devices that map a Large Language Model (LLM) to partitions in which the matrices of each layer are separated column by column using an intra-layer parallelism scheme, and each of the plurality of devices is configured to synchronize data by sharing the sub-result of matrix multiplication on the data with other devices during the execution of the matrix multiplication.
[0007] According to one side, each of the plurality of devices may be characterized by including a matrix operation unit that performs matrix multiplication on data, a lower result storage unit that stores a first lower result calculated in real time by the matrix operation unit in memory during the execution of the matrix multiplication, a transmission unit that reads the first lower result stored in memory during the execution of the matrix multiplication and transmits it to at least one other device included in the inference execution system, a receiving unit that receives a second lower result calculated by each of the at least one other devices from the at least one other device and stores it in memory, and a synchronization unit that synchronizes the data using the first lower result and the second lower result during the execution of the matrix multiplication.
[0008] In other aspects, the system may be characterized by enabling the simultaneous execution of the matrix multiplication, the transmission of the first lower result, and the reception of the second lower result.
[0009] Another aspect of the matrix operation unit may be characterized by continuously calculating matrix multiplications between each row of the weighted value matrix stored in the register file and the input vector, and generating the first lower result for each matrix multiplication between each row and the input vector.
[0010] In another aspect, the synchronization unit may be characterized by loading the first lower result and the second lower result into the register file to synchronize the data.
[0011] The present invention provides a device included in a multi-device-based inference execution system, comprising: a matrix operation unit that performs matrix multiplication on data; a lower result storage unit that stores a first lower result calculated in real time by the matrix operation unit in memory during the execution of the matrix multiplication; a transmission unit that reads the first lower result stored in memory and transmits it to at least one other device included in the inference execution system during the execution of the matrix multiplication; a receiving unit that receives a second lower result calculated by each of the at least one other devices and stores it in memory; and a synchronization unit that synchronizes data using the first lower result and the second lower result during the execution of the matrix multiplication.
[0012] An inference execution method for a device included in a multi-device-based inference execution system is provided, which includes the step of performing matrix multiplication on data, wherein the step of performing matrix multiplication includes storing a first sub-result, which is calculated in real time, in memory during the execution of the matrix multiplication; reading the first sub-result stored in memory during the execution of the matrix multiplication and transmitting it to at least one other device included in the inference execution system; receiving a second sub-result calculated by each of the at least one other devices and storing it in memory; and synchronizing data using the first sub-result and the second sub-result during the execution of the matrix multiplication.
[0013] Specific details regarding other embodiments will be included in the detailed description and drawings. [Effects of the Invention]
[0014] This invention provides an inference execution method and system as a network technology that can effectively perform calculations on a multi-device infrastructure.
[0015] By performing matrix multiplication and All Gather simultaneously, communication overhead and latency can be reduced, providing extremely high scalability for inference execution systems.
[0016] The effects of the present invention are not limited to those described above, and any other effects not described will be clearly understood by those skilled in the art from the claims. [Brief explanation of the drawing]
[0017] [Figure 1] This figure shows an example of a multi-device hardware structure for realizing an inference execution system according to one embodiment of the present invention. [Figure 2] This figure shows an example of a timeline for a matrix multiplication operation using multiple devices in one embodiment of the present invention. [Figure 3] This figure shows an example of a sub-result in intralayer parallelism according to one embodiment of the present invention. [Figure 4] This figure illustrates an example of processing matrix multiplication operations using multiple devices in one embodiment of the present invention, where data synchronization is performed by overlapping. [Figure 5] This figure shows an example of the internal configuration of an inference execution system according to one embodiment of the present invention. [Figure 6] This figure shows an example of an inference execution method according to one embodiment of the present invention. [Modes for carrying out the invention]
[0018] The advantages and features of the present invention, and the method for achieving them, will become apparent by referring to the embodiments described in detail below together with the accompanying drawings. However, the present invention should not be limited to the embodiments disclosed below, and may be realized in various different forms. The embodiments are provided only to make the disclosure of the present invention complete and to fully inform those with ordinary knowledge in the technical field to which the present invention pertains of the scope of the invention. The present invention is only defined based on the claims. Throughout the specification, the same reference numerals indicate the same components.
[0019] When a component is described as being "connected to" or "coupled to" another component, it includes both the case of being directly connected or coupled to the other component and the case of having other components intervening therebetween. On the contrary, when a component is described as being "directly connected to" or "directly coupled to" another component, it means that no other component intervenes therebetween. "And / or" includes each of the items described and all combinations of one or more of them.
[0020] The terms used in this specification are only for the purpose of describing the embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated otherwise in the context. "Comprises" and / or "comprising" used in the specification mean that the described components, steps, operations, and / or elements do not exclude the existence or addition of one or more other components, steps, operations, and / or elements.
[0021] The terms such as "first", "second", etc. are used to describe various components, but these components should not be limited by these terms. These terms are only used to distinguish one component from another. Therefore, it is needless to say that the first component described below may become the second component within the technical idea of the present invention.
[0022] Unless otherwise defined, all terms (including technical and scientific terms) used in this specification shall be construed to have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Further, terms that are commonly used and defined in a dictionary shall not be construed to be ideally or overly interpreted unless specifically defined.
[0023] In the process of training for a Large Language Model (LLM), due to the large amount of input and a large number of operations, by using a GPU (Graphics Processing Unit) suitable for throughput operations, the operations on a multi-device platform can be efficiently processed. On the other hand, in the process of inference, since the amount of input is small and many memory accesses are required, if a GPU is used to execute operations on a multi-device platform, the efficiency will be significantly reduced.
[0024] For example, the communication operations required for training and inference are fundamentally different, and the computational features also differ, so naturally the optimal solutions will also differ. Training is a large-batch operation with many inputs, while inference is a small-batch operation with few inputs. Furthermore, training on a multi-device infrastructure requires four communication operations: 1) Reduce Scatter, 2) All Gather, 3) All Reduce, and 4) All-to-All, whereas inference requires only one communication operation: All Gather. Therefore, while it is possible to apply a training solution to inference, its performance will be significantly lower compared to a solution specifically designed for inference.
[0025] Furthermore, parallelism is necessary to compute large-scale language model inference on a multi-device infrastructure, and there are two main types of parallelism: data parallelism and model parallelism. Data parallelism is a method in which the same model is mapped to all devices without separating the models, and then the input is separated and sent to each device. Model parallelism is a method in which the model is separated and mapped to each device, and this type of model parallelism is divided into inter-layer parallelism and intra-layer parallelism. Inter-layer parallelism, also called pipeline parallelism, is a method in which the model is separated layer by layer without separating the layers of the model. This can reduce the size of the model mapped to each device and increase the amount of computation, but it cannot reduce the latency for a single request. On the other hand, intra-layer parallelism is a method in which each layer is separated. This method can increase processing power while simultaneously reducing latency, but it has the drawback of requiring communication between devices after calculations are performed on each device, due to the separation of each layer.
[0026] In embodiments of the present invention, an inference execution method and system are provided as a network technology capable of effectively performing calculations on a multi-device infrastructure.
[0027] Figure 1 shows an example of a multi-device hardware structure for implementing an inference execution system according to one embodiment of the present invention. The device 100 according to the embodiment of Figure 1 may include a Register File 110, a Matrix Unit 120, a Load / Store Unit 130, a Memory 140, and a P2P (Peer to Peer) Network 150. In this case, each of the multi-devices that implement the inference execution system may have the same or similar hardware structure as the device 100 according to the embodiment of Figure 1.
[0028] (1) The matrix multiplication process may be an example of a process in which device 100 reads data from register file 110 and performs matrix multiplication on the data through matrix unit 120.
[0029] (2) The Store process may be an example of a process in which the device 100 stores the sub-result in memory 140 through the load / store unit 130 in order to send the sub-result to the network in real time without waiting for all the results of the matrix multiplication operation.
[0030] (3) The transmission (TX) process may be an example of a process in which device 100 reads the lower result stored in memory 140 from memory 140 and transmits it to another device via the P2P network 150.
[0031] (4) The receiving (RX) process may be an example of a process in which device 100 gathers all the data (lower results) transmitted by each of the multiple devices and stores them in memory 140. In this case as well, device 100 may immediately store the lower results in memory 140 without waiting for all the data from the multiple devices.
[0032] (5) The loading process may be an example of a process in which device 100 reads the lower result stored in memory 140 in (4) from memory 140 and writes it back to register file 110. In this case, the lower result written back to register file 110 may include the lower result stored in memory 140 in the process of (2), in other words, the lower result of the matrix multiplication performed by device 100.
[0033] Figure 2 shows an example of a timeline for a matrix multiplication operation using multiple devices in one embodiment of the present invention. As shown in the embodiment of Figure 2, device 100 can reduce most of the communication overhead by executing the processes (1) to (5) described above in a continuous pipeline manner, thereby reducing latency excluding tail latency for the final calculation result. Since such tail latency is very small, the inference execution system according to this embodiment can consequently provide extremely high scalability.
[0034] Figure 3 shows an example of a lower result in intralayer parallelism according to one embodiment of the present invention. In the embodiment of Figure 3, an example is shown in which an output vector 330 is generated as a result of matrix multiplication between a shared input vector 310 and a partitioned weight matrix 320. In this case, each of the multi-devices included in the inference execution system according to this embodiment (for example, each of devices 1 to 4) may perform matrix multiplication between each partition of the partitioned weight matrix 320 and the shared input vector 310 for intralayer parallelism. In this case, a first lower result 350 may be generated as a result of device 1 performing matrix multiplication between the first column 340 of the partitioned weight matrix 320 and the shared input vector 310. In this case, device 1 may send the pre-generated first lower result 350 to each of the multi-devices without waiting for the results for the entire partitioned weight matrix 320 or the results for all of its partitions. In other words, each device in the inference execution system according to this embodiment can perform matrix multiplication and All Gather simultaneously, without waiting for the completion of matrix multiplication, while transmitting data based on lower-order results to other devices in real time.
[0035] For this reason, instruction fusion may be required. For example, a matrix multiplication instruction and an All Gather instruction may be fused. Furthermore, an architecture to support instruction fusion may be required. In other words, a structure is needed that can simultaneously perform matrix multiplication and transmission for All Gather. Moreover, it must be able to process reception at the device simultaneously. In other words, each of the multiple devices included in the inference execution system can have a structure that can simultaneously perform partitioned matrix multiplication and data transmission and reception in real time.
[0036] Figure 4 shows an example of processing matrix multiplication using multiple devices in one embodiment of the present invention, where data synchronization is processed by overlapping. In multi-device based inference operations, the most important thing is how to effectively perform partitioned matrix multiplication and the subsequent necessary data synchronization. It is common for processors to start All Gather after matrix multiplication is complete. In other words, since the target data of the first instruction, matrix multiplication, is the source data of the second instruction, All Gather, the dependency check logic ensures that All Gather is executed after matrix multiplication is complete. The embodiment in Figure 4 shows the latency when data synchronization (for example, data synchronization by All Gather) is performed after the matrix multiplication operation is completed on the timeline, and when data synchronization is processed by overlapping with matrix multiplication. In the inference execution system according to this embodiment, matrix multiplication and All Gather can be executed simultaneously on each of the multiple devices while sending lower result data to other devices in real time without waiting for the matrix multiplication to finish, and latency can be greatly reduced by such overlapping. For example, as explained with reference to Figure 3, it is easy to see that the effect of latency reduction increases as the size of the partitioned weight matrix 320 increases.
[0037] Figure 5 is a diagram showing an example of the internal configuration of an inference execution system according to one embodiment of the present invention, and Figure 6 is a diagram showing an example of an inference execution method according to one embodiment of the present invention.
[0038] The inference execution system 500 according to the embodiment shown in Figure 5 may include multiple devices (device 1 (510)) and other devices 520, which map the large-scale language model to partitions obtained by separating the matrices of each layer column by column using an intra-layer parallelism scheme. In this case, the inference execution system 500 may process inference using the large-scale language model through the multiple devices (510 and 520) mapped to the partitions of the large-scale language model.
[0039] In this case, device 1(510) may include a matrix operation unit 511, a lower result storage unit 512, a memory 513, a transmission unit 514, a reception unit 515, and a synchronization unit 516, as shown in Figure 5. Here, the matrix operation unit 511, the lower result storage unit 512, the transmission unit 514, the reception unit 515, and the synchronization unit 516 may be functional representations of the operation of a physical processor that can be included in device 1(510). Each of the other devices 520 may also have the same or similar components as device 1(510).
[0040] The inference execution method in Figure 6 may include step 610, and step 610 may include steps 611 to 614.
[0041] In step 610, the matrix operation unit 511 may perform matrix multiplication on the data. For example, the matrix operation unit 511 may continuously calculate matrix multiplication between each row of the weighted value matrix stored in the register file and the input vector, and continuously generate a first lower result for each matrix multiplication between each row and the input vector. In this case, steps 611 to 614 may be executed in the middle of the matrix multiplication being performed in step 610.
[0042] In step 611, the sub-result storage unit 512 may store the first sub-result, which is calculated in real time, in memory 513. For example, assuming that the matrix operation unit 511 calculates four sub-results from 1-1 to 1-4, the 1-1 sub-result may be stored in memory 513 during the process of calculating the 1-2 sub-result, and the 1-2 sub-result may be stored in memory 513 during the process of calculating the 1-3 sub-result. In subsequent steps 612 to 614, each sub-result may be processed in the calculation process of the next sub-result.
[0043] In step 612, the transmission unit 514 may read the first lower result stored in the memory 513 and transmit it to each of the other devices 520 included in the inference execution system 500. At this time, each of the other devices 520 may also calculate the second lower result during the execution of matrix multiplication and transmit it to each of the other devices.
[0044] In step 613, the receiving unit 515 may receive the second lower result calculated by each of the other devices 520 and store it in the memory 513. In this case, both the first lower result and the second lower result may be stored in the memory 513. Depending on the embodiment, the transmitting unit 514 and the receiving unit 515 may perform a process to associate the first lower result with the second lower result and store it again in the memory 513.
[0045] In step 614, the synchronization unit 516 may synchronize the data using the first lower result and the second lower result. For example, the synchronization unit 516 may synchronize the data by loading the first lower result and the second lower result into a register file.
[0046] Thus, according to embodiments of the present invention, it is possible to provide an inference execution method and system as a network technology that can effectively perform multi-device-based computations. Furthermore, by performing matrix multiplication and All Gather simultaneously, communication overhead and latency can be reduced, and the inference execution system can be provided with extremely high scalability.
[0047] As described above, the embodiments have been explained based on the limited embodiments and drawings, but those skilled in the art will be able to make various modifications and variations from the above description. Therefore, even different embodiments will fall within the scope of the attached claims as long as they are equivalent to the claims.
Claims
1. A device included in a multi-device-based inference execution system, A matrix operation unit that performs matrix multiplication on data, A sub-result storage unit stores in memory a first sub-result calculated in real time by the matrix operation unit during the execution of the matrix multiplication, A transmission unit that, during the execution of the matrix multiplication, reads the first lower result stored in the memory and transmits it to at least one other device included in the inference execution system, A receiving unit that receives a second lower result calculated by each of the at least one other devices and stores it in memory, A synchronization unit that synchronizes the data using the first lower result and the second lower result during the execution of the matrix multiplication, Includes, A device characterized in that it is implemented such that the execution of matrix multiplication, the transmission of the first lower result, and the reception of the second lower result can be performed simultaneously.
2. A device included in a multi-device-based inference execution system, A matrix operation unit that performs matrix multiplication on data, A sub-result storage unit stores in memory a first sub-result calculated in real time by the matrix operation unit during the execution of the matrix multiplication, A transmission unit that, during the execution of the matrix multiplication, reads the first lower result stored in the memory and transmits it to at least one other device included in the inference execution system, A receiving unit that receives a second lower result calculated by each of the at least one other devices and stores it in memory, A synchronization unit that synchronizes the data using the first lower result and the second lower result during the execution of the matrix multiplication, Includes, The aforementioned multi-device is mapped to partitions separated by an intra-layer parallelism scheme, where the Large Language Model (LLM) is located. The inference execution system processes inference using the large-scale language model through multiple devices mapped to the partition. A device characterized by the following.
3. A device included in a multi-device-based inference execution system, A matrix operation unit that performs matrix multiplication on data, A sub-result storage unit stores in memory a first sub-result calculated in real time by the matrix operation unit during the execution of the matrix multiplication, A transmission unit that, during the execution of the matrix multiplication, reads the first lower result stored in the memory and transmits it to at least one other device included in the inference execution system, A receiving unit that receives the second lower result calculated by each of the at least one other devices and stores it in memory, A synchronization unit that synchronizes the data using the first lower result and the second lower result during the execution of the matrix multiplication, Includes, The matrix operation unit continuously calculates matrix multiplication between each row of the weighted value matrix stored in the register file and the input vector, A first lower result is generated for each matrix multiplication between each row and the input vector. A device characterized by the following.
4. The synchronization unit loads the first lower result and the second lower result into the register file and synchronizes the data. The device according to claim 3, characterized by the following:
5. A method for performing inference on a device included in a multi-device-based inference execution system, Steps to perform matrix multiplication on data. Includes, The step of performing the aforementioned matrix multiplication is: The steps include storing the first sub-result, which is calculated in real time during the execution of the matrix multiplication, into memory, The steps include reading the first lower result stored in the memory during the execution of the matrix multiplication and transmitting it to at least one other device included in the inference execution system, The steps include receiving the second lower result calculated by each of the at least one other devices and storing it in memory, During the execution of the matrix multiplication, the data is synchronized using the first lower result and the second lower result, Includes, The device is designed to enable the simultaneous execution of matrix multiplication, transmission of the first lower result, and reception of the second lower result. An inference execution method characterized by the following.
6. A method for performing inference on a device included in a multi-device-based inference execution system, Steps to perform matrix multiplication on data. Includes, The step of performing the aforementioned matrix multiplication is: The steps include storing the first sub-result, which is calculated in real time during the execution of the matrix multiplication, into memory, The steps include reading the first lower result stored in the memory during the execution of the matrix multiplication and transmitting it to at least one other device included in the inference execution system, The steps include receiving the second lower result calculated by each of the at least one other devices and storing it in memory, During the execution of the matrix multiplication, the data is synchronized using the first lower result and the second lower result, Includes, The aforementioned multi-device is mapped to partitions separated by an intra-layer parallelism scheme, where the Large Language Model (LLM) is located. The inference execution system processes inference using the large-scale language model through multiple devices mapped to the partition. An inference execution method characterized by the following.
7. A method for performing inference on a device included in a multi-device-based inference execution system, Steps to perform matrix multiplication on data. Includes, The step of performing the aforementioned matrix multiplication is: The steps include storing the first sub-result, which is calculated in real time during the execution of the matrix multiplication, into memory, The steps include reading the first lower result stored in the memory during the execution of the matrix multiplication and transmitting it to at least one other device included in the inference execution system, The steps include receiving the second lower result calculated by each of the at least one other devices and storing it in memory, During the execution of the matrix multiplication, the data is synchronized using the first lower result and the second lower result, Includes, The step of performing the aforementioned matrix multiplication is: The step of continuously calculating matrix multiplications between each row of the weighted value matrix stored in the register file and the input vector, and generating the first lower result for each matrix multiplication between the row and the input vector. An inference execution method characterized by further including the following.
8. The aforementioned synchronization step is, During the execution of the matrix multiplication, the first lower result and the second lower result are loaded into the register file to synchronize the data. The inference execution method according to claim 7, characterized by the above.
9. A multi-device-based inference execution system, Multiple devices are mapped to partitions separated by an intra-layer parallelism scheme, where a Large Language Model (LLM) is used. Includes, Each of the aforementioned multiple devices is: The sub-result of matrix multiplication on the data is shared with other devices among the multiple devices during the execution of the matrix multiplication, thereby synchronizing the data. An inference execution system characterized by the following:
10. Each of the aforementioned multiple devices is: A matrix operation unit that performs matrix multiplication on the data, A lower result storage unit stores the first lower result, which is calculated in real time by the matrix operation unit during the execution of the matrix multiplication, in memory. A transmission unit that, during the execution of the matrix multiplication, reads the first lower result stored in the memory and transmits it to at least one other device included in the inference execution system, A receiving unit that receives a second lower result calculated by each of the at least one other devices and stores it in memory, A synchronization unit that synchronizes the data using the first lower result and the second lower result during the execution of the matrix multiplication, The inference execution system according to claim 9, characterized by including the following:
11. The matrix multiplication, the transmission of the first lower result, and the reception of the second lower result are to be implemented so that they can be performed simultaneously. The inference execution system according to claim 10, characterized by the above.
12. The matrix operation unit continuously calculates matrix multiplication between each row of the weighted value matrix stored in the register file and the input vector, The first lower result is generated for each matrix multiplication between each row and the input vector. The inference execution system according to claim 10, characterized by the above.
13. The synchronization unit loads the first lower result and the second lower result into the register file and synchronizes the data. The inference execution system according to claim 12, characterized by the above.