Method and system for executing multi-device-based reasoning for large language model
By utilizing a multi-device parallel system to synchronize data during the reasoning process of a large language model, the problem of high computational cost in existing technologies is solved, and efficient reasoning execution is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-20
- Publication Date
- 2026-03-24
AI Technical Summary
The reasoning process of existing large language models lacks effective accelerators, resulting in high computational costs and low efficiency.
A multi-device parallel inference execution system is adopted. By dividing the matrix of each layer of the large language model into partitions and synchronizing data among multiple devices, the sub-results of matrix multiplication are shared and synchronized.
It reduces communication overhead and latency, and improves the scalability and efficiency of the inference execution system.
Smart Images
Figure CN121729697A_ABST
Abstract
Description
Technical Field
[0001] Various embodiments of the present invention relate to a method and system for performing multi-device-based inference for large language models. Background Technology
[0002] Large Language Model (LLM) refers to a model that uses artificial neural networks to calculate the probability distribution of natural language sentences. Recently, it has been widely used in language-related tasks such as answering questions or translation, as exemplified by ChatGPT.
[0003] As the size of large language models continues to increase, the computational costs required for training and inference become extremely high. While multi-accelerator approaches are increasingly used for computation on these models, with GPUs being widely and effectively utilized for training, there remains a lack of effective accelerators for inference. Summary of the Invention
[0004] The problem that the invention aims to solve
[0005] The present invention can provide a reasoning execution method and system for network technology that can effectively perform multi-device-based computation.
[0006] The technical problems of this invention are not limited to those mentioned above. Those skilled in the art will clearly understand from the following description other technical problems not mentioned.
[0007] The measures taken to solve the problem
[0008] This invention provides an inference execution system, which is a multi-device inference execution system. The inference execution system is characterized in that it includes multiple devices, which are mapped to partitions obtained by dividing the matrices of each layer of the Large Language Model (LLM) in a column-based manner according to an intra-layer parallel approach. Each of the multiple devices is implemented such that it shares the sub-result of matrix multiplication of data with other devices in the multiple devices to synchronize data during the execution of the matrix multiplication.
[0009] According to one aspect of the present invention, the plurality of devices may each include: a matrix operation unit that performs matrix multiplication of data; a sub-result storage unit that, during the execution of the matrix multiplication, stores a first sub-result calculated in real time by the matrix operation unit into a memory; a transmission unit that, during the execution of the matrix multiplication, reads the first sub-result stored in the memory and transmits the first sub-result to at least one other device included in the inference execution system; a receiving unit that receives a second sub-result calculated by each of the at least one other device and stores the second sub-result into a memory; and a synchronization unit that, during the execution of the matrix multiplication, synchronizes data using the first sub-result and the second sub-result.
[0010] According to another aspect of the invention, it may be implemented in a manner that enables the simultaneous execution of the matrix multiplication, the transmission of the first sub-result, and the reception of the second sub-result.
[0011] According to another aspect of the present invention, the matrix operation unit continuously calculates the matrix multiplication between each row of the weight matrix stored in the register file and the input vector, and the matrix multiplication between each row and the input vector generates the first sub-result.
[0012] According to another aspect of the present invention, the synchronization unit may load the first sub-result and the second sub-result into the register file to synchronize the data.
[0013] The present invention provides an apparatus comprising a multi-device inference execution system, the apparatus comprising: a matrix operation unit that performs matrix multiplication on data; a sub-result storage unit that, during the matrix multiplication, stores a first sub-result calculated in real time by the matrix operation unit into a memory; a transmission unit that, during the matrix multiplication, reads the first sub-result stored in the memory and transmits the first sub-result to at least one other device included in the inference execution system; a receiving unit that receives a second sub-result calculated by each of the at least one other device and stores the second sub-result into the memory; and a synchronization unit that, during the matrix multiplication, synchronizes data using the first sub-result and the second sub-result.
[0014] This invention provides an inference execution method, which is an inference execution method for devices included in a multi-device inference execution system. The inference execution method includes the step of performing matrix multiplication on data. The step of performing matrix multiplication includes: storing a first sub-result calculated in real time into a memory during the matrix multiplication process; reading the first sub-result stored in the memory and transmitting the first sub-result to at least one other device included in the inference execution system during the matrix multiplication process; receiving a second sub-result calculated on each of the at least one other device and storing the second sub-result into a memory; and synchronizing data using the first sub-result and the second sub-result during the matrix multiplication process.
[0015] Specific details of other embodiments are included in the summary and drawings.
[0016] The effects of the invention
[0017] The present invention can provide a method and system for inference execution based on network technology with multi-device computing, which can effectively perform inference execution.
[0018] Because matrix multiplication and full collection operations are performed simultaneously, communication overhead and latency are reduced, thus providing very high scalability for the inference execution system.
[0019] The effects of this invention are not limited to those mentioned above, and those skilled in the art will clearly understand other effects not mentioned from the description in the claims. Attached Figure Description
[0020] Figure 1 This is a diagram illustrating an example of the hardware structure of a multi-device inference execution system according to an embodiment of the present invention.
[0021] Figure 2 This is a diagram illustrating an example of a timeline of a matrix multiplication operation performed using multiple devices in one embodiment of the present invention.
[0022] Figure 3 This is a diagram illustrating an example of a sub-result in intra-layer parallelism according to an embodiment of the present invention.
[0023] Figure 4 The illustration shows an example of overlapping data synchronization when using multiple devices to process matrix multiplication operations in one embodiment of the present invention.
[0024] Figure 5This is a diagram illustrating an example of the internal structure of a reasoning execution system according to an embodiment of the present invention.
[0025] Figure 6 This is a diagram illustrating an example of a reasoning execution method according to an embodiment of the present invention. Detailed Implementation
[0026] The advantages and features of the present invention, and the methods for implementing them, are described below with reference to the appendix. Figure 1 The details of the various embodiments will make it clear. However, the invention is not limited to the embodiments disclosed below, but can be embodied in various different ways. These embodiments are provided only to make the disclosure of the invention complete and to fully inform those skilled in the art of the scope of the invention, which is defined only by the scope of the claims. Throughout the specification, the same reference numerals refer to the same constituent elements.
[0027] When a component is referred to as being "connected to" or "coupled to" another component, it includes cases where the component is directly connected to or coupled to the other component, or cases where other components are involved between them. Conversely, when a component is referred to as being "directly connected to" or "directly coupled to" another component, it means that no other components are involved between them. "And / or" includes each of the mentioned items as well as all combinations of more than one item.
[0028] The terminology used in this specification is for illustrative purposes and is not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise. The terms "comprises" and / or "comprising" as used in this specification mean that the mentioned constituent elements, steps, operations, and / or components do not exclude the presence or addition of more than one other constituent element, step, operation, and / or component.
[0029] Although terms such as "first," "second," etc., are used to describe various constituent elements, these constituent elements are not limited by these terms. These terms are used merely to distinguish one constituent element from others. Therefore, the "first constituent element" mentioned below can certainly also be a "second constituent element" within the technical concept of this invention.
[0030] Unless otherwise defined, all terms used in this specification (including technical and scientific terms) may be used in the sense that can be commonly understood by one of ordinary skill in the art. Furthermore, terms defined in commonly used dictionaries should not be interpreted ideally or excessively unless explicitly defined otherwise.
[0031] During training for Large Language Models (LLMs), the large input volume and computational demands allow for efficient use of Graphics Processing Units (GPUs) to handle multi-device computations. Conversely, during inference, the smaller input volume and higher memory access requirements make GPUs inefficient for multi-device computations.
[0032] For example, the communication operations required for training and inference are fundamentally different, and their computational characteristics also differ, thus leading to different optimal solutions. Training involves large-batch operations with large input volumes, while inference involves small-batch operations with small input volumes. Furthermore, multi-device training requires four communication operations: 1) ReduceScatter; 2) All Gather; 3) All Reduce; and 4) All-to-All. Conversely, inference only requires the All Gather operation. Therefore, while solutions used for training can be applied to inference, their performance is significantly lower than solutions used for inference.
[0033] Furthermore, parallelism is required for reasoning on large language models across multiple devices. Parallelism can be broadly categorized into data parallelism and model parallelism. Data parallelism is a method that maps the same model across all devices without separating the model, then separates the input and sends it to each device. Model parallelism separates the model and maps it to each device. This type of model parallelism is further divided into inter-layer parallelism and intra-layer parallelism. Inter-layer parallelism, also known as pipelined parallelism, separates the model layer by layer without separating the model by individual layers. This method can reduce the size of the model mapped to each device and increase the throughput, but it cannot reduce latency for a single request. Conversely, intra-layer parallelism separates the layers. This method can both increase throughput and reduce latency, but because it separates the layers, it requires communication between devices after the computation within each device is completed.
[0034] Various embodiments of the present invention provide inference execution methods and systems for network technologies capable of effectively performing multi-device-based computations.
[0035] Figure 1 This is a diagram illustrating an example of the hardware structure of a multi-device inference execution system according to an embodiment of the present invention. Figure 1 The device 100 in one embodiment may include a register file 110, a matrix unit 120, a load / store unit 130, a memory 140, and a peer-to-peer (P2P) network 150. In this case, each of the multiple devices implementing the inference execution system may have [specific features / equivalents] according to [the specific implementation details]. Figure 1 The device 100 in the embodiments has the same or similar hardware structure.
[0036] (1) The process of matrix multiplication can be an example of the process by which device 100 reads data from register file 110 and performs matrix multiplication on the data through matrix unit 120.
[0037] (2) The storage process can be an example of a process in which the device 100 does not wait for the entire result of the matrix multiplication operation and sends the sub-result to the network in real time by storing the sub-result in the memory 140 through the load / store unit 130.
[0038] (3) The transmission (TX) process can be an example of the process by which device 100 reads a sub-result stored in memory 140 and transmits it to another device through peer-to-peer (P2P) network 150.
[0039] (4) The receiving (RX) process can be an example of the process by which device 100 collects all the data (sub-results) transmitted from each of the multiple devices and stores them in memory 140. In this case, device 100 may also directly store the sub-results in memory 140 without waiting for all the data from the multiple devices.
[0040] (5) The loading process can be an example of the process by which device 100 reads the sub-results stored in memory 140 in (4) from memory 140 and writes them back to register file 110. In this case, the sub-results written back to register file 110 can also include the sub-results stored in memory 140 in the process of (2), i.e., the sub-results of the matrix multiplication performed by device 100.
[0041] Figure 2 This is a diagram illustrating an example of a timeline of a matrix multiplication operation performed using multiple devices in one embodiment of the present invention. Figure 2 As shown in the embodiment, the device 100 performs the processes (1) to (5) described above in a pipeline manner, thereby virtually eliminating communication overhead and reducing latency other than the tail latency of the final computation result. This tail latency is very small, and ultimately, the inference execution system according to this embodiment can provide very high scalability.
[0042] Figure 3 This is a diagram illustrating an example of sub-results in intra-layer parallelism according to an embodiment of the present invention. Figure 3In one embodiment, an example is shown where the result of matrix multiplication between a shared input vector 310 and a partitioned weight matrix 320 generates an output vector 330. In this case, each of the multiple devices included in the inference execution system according to this embodiment (for example, devices 1 to 4 each) can perform matrix multiplication between each partition of the partitioned weight matrix 320 (partitioned for intra-layer parallelism) and the shared input vector 310. At this time, the result of matrix multiplication between the first column 340 of the partitioned weight matrix 320 and the shared input vector 310 performed by device 1 can generate a first sub-result 350. Device 1 can then transmit the first sub-result 350 generated earlier to each of the multiple devices without waiting for the result of the weight matrix 320 for the entire partition or the result for the corresponding partition as a whole. In other words, each device in the inference execution system according to this embodiment can simultaneously perform matrix multiplication and full collection operations while transmitting data based on the sub-results to other devices in real time without waiting for the matrix multiplication to complete.
[0043] Therefore, instruction fusion is required. As an example, matrix multiplication instructions and full collection instructions can be fused. Furthermore, an architecture capable of supporting instruction fusion is also needed. In other words, a structure capable of performing matrix multiplication and transmission for full collection is required simultaneously. Additionally, the devices should also be able to handle reception simultaneously. In other words, the multiple devices included in the inference execution system can each have a structure capable of simultaneously performing real-time partitioned matrix multiplication and data transmission, as well as data reception.
[0044] Figure 4 The illustration depicts an example of overlapping data synchronization during matrix multiplication operations using multiple devices, as shown in one embodiment of the invention. The most important aspect of multi-device-based inference operations is how to efficiently perform partitioned matrix multiplication and the subsequent required data synchronization. Generally, the processor can begin a full collection after the matrix multiplication is complete. In other words, the target data of the matrix multiplication instruction (the first instruction) is the source data for the full collection instruction (the second instruction), therefore, according to dependency check logic, a full collection can only be performed after the matrix multiplication is complete. Figure 4In the embodiments, the latency is shown on the timeline for the case where data synchronization is performed after the matrix multiplication operation is completed (as an example, through full collection data synchronization) and the case where data synchronization is overlapped with matrix multiplication. In the inference execution system according to this embodiment, matrix multiplication and full collection operations can be performed simultaneously by sending sub-result data to other devices in real time on each device without waiting for the matrix multiplication to finish, and this overlap can greatly reduce latency. For example, it can be easily understood that by... Figure 3 The larger the size of the partition weight matrix 320, the greater the effect of reducing latency.
[0045] Figure 5 This is a diagram illustrating an example of the internal structure of an inference execution system according to an embodiment of the present invention. Figure 6 This is a diagram illustrating an example of a reasoning execution method according to an embodiment of the present invention.
[0046] according to Figure 5 The inference execution system 500 of the embodiment may include multiple devices (device 1 510 and other devices 520) that are mapped to partitions obtained by dividing the matrix of each layer of a large language model in a column-by-column manner according to an intra-layer parallel approach. In this case, the inference execution system 500 can process inference using the large language model through the multiple devices 510, 520 mapped to the partitions of the large language model.
[0047] At this time, as Figure 5 As shown, device 1 (reference numeral 510) may include a matrix operation unit 511, a sub-result storage unit 512, a memory 513, a transmission unit 514, a receiving unit 515, and a synchronization unit 516. Here, the matrix operation unit 511, the sub-result storage unit 512, the transmission unit 514, the receiving unit 515, and the synchronization unit 516 may be functional representations of the operation of the physical processor that can be included in device 1 (reference numeral 510). Other devices 520 may each have the same or similar constituent elements as device 1 (reference numeral 510).
[0048] Figure 6 The reasoning execution method may include step 610, and step 610 may include steps 611 to 614.
[0049] In step 610, the matrix operation unit 511 can perform matrix multiplication on the data. As an example, the matrix operation unit 511 continuously calculates the matrix multiplication between each row of the weight matrix stored in the register file and the input vector, so that each row's matrix multiplication with the input vector can continuously generate a first sub-result. In this case, steps 611 to 614 can be performed during the matrix multiplication process in step 610.
[0050] In step 611, the sub-result storage unit 512 can store the first sub-result calculated in real time into the memory 513. For example, assuming that the matrix operation unit 511 calculates four sub-results from sub-result 1-1 to sub-result 1-4, sub-result 1-1 can be stored into the memory 513 during the calculation of sub-result 1-2, and sub-result 1-2 can be stored into the memory 513 during the calculation of sub-result 1-3. In subsequent steps 612 to 614, each sub-result can also be processed during the calculation of the next sub-result.
[0051] In step 612, the transmission unit 514 can read the first sub-result stored in the memory 513 and transmit the first sub-result to the other devices 520 included in the inference execution system 500. At this time, each of the other devices 520 can also calculate the second sub-result during the matrix multiplication process and transmit the second sub-result to devices other than itself.
[0052] In step 613, the receiving unit 515 may receive the second sub-result calculated by the other devices 520 and store the second sub-result in the memory 513. In this case, the memory 513 may store both the first sub-result and the second sub-result. According to an embodiment, the transmitting unit 514 and the receiving unit 515 may also handle the process of associating the first sub-result and the second sub-result and storing them again in the memory 513.
[0053] In step 614, the synchronization unit 516 can synchronize data using the first sub-result and the second sub-result. As an example, the synchronization unit 516 can load the first sub-result and the second sub-result into a register file to synchronize data.
[0054] Thus, according to embodiments of the present invention, a method and system for inference execution based on network technology capable of efficiently performing multi-device computation can be provided. Furthermore, since matrix multiplication and full collection operations are performed simultaneously, communication overhead and latency are reduced, thereby providing very high scalability for the inference execution system.
[0055] Methods for implementing the invention
[0056] While embodiments of the present invention have been described above with reference to the accompanying drawings, those skilled in the art will understand that the invention can be implemented in other specific ways without altering its technical concept or essential features. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting.
Claims
1. An apparatus comprising a multi-device-based inference execution system, the apparatus characterized in that it includes: The matrix operation unit performs matrix multiplication on the data; The sub-result storage unit stores the first sub-result calculated in real time by the matrix operation unit into the memory during the matrix multiplication process described above. The transmission unit, during the execution of the matrix multiplication described above, reads the first sub-result stored in the memory and transmits the first sub-result to at least one other device included in the inference execution system. A receiving unit that receives a second sub-result calculated in each of the at least one other device and stores the second sub-result in a memory; and, The synchronization unit, during the matrix multiplication process described above, synchronizes data using the first sub-result and the second sub-result. It is implemented in a way that allows the simultaneous execution of the matrix multiplication, the transmission of the first sub-result, and the reception of the second sub-result.
2. An apparatus comprising a multi-device-based inference execution system, the apparatus characterized in that it includes: The matrix operation unit performs matrix multiplication on the data; The sub-result storage unit stores the first sub-result calculated in real time by the matrix operation unit into the memory during the matrix multiplication process described above. The transmission unit, during the execution of the matrix multiplication described above, reads the first sub-result stored in the memory and transmits the first sub-result to at least one other device included in the inference execution system. A receiving unit that receives a second sub-result calculated in each of the at least one other device and stores the second sub-result in a memory; and, The synchronization unit, during the matrix multiplication process described above, synchronizes data using the first sub-result and the second sub-result. The aforementioned multi-device mapping is applied to partitions obtained by dividing the large language model according to the intra-layer parallel approach. The aforementioned inference execution system utilizes the aforementioned large language model for inference through multi-device processing mapped to the aforementioned partitions.
3. An apparatus comprising a multi-device-based inference execution system, the apparatus characterized in that it includes: The matrix operation unit performs matrix multiplication on the data; The sub-result storage unit stores the first sub-result calculated in real time by the matrix operation unit into the memory during the matrix multiplication process described above. The transmission unit, during the execution of the matrix multiplication described above, reads the first sub-result stored in the memory and transmits the first sub-result to at least one other device included in the inference execution system. A receiving unit that receives a second sub-result calculated in each of the at least one other device and stores the second sub-result in a memory; and, The synchronization unit, during the matrix multiplication process described above, synchronizes data using the first sub-result and the second sub-result. The matrix operation unit described above continuously calculates the matrix multiplications between each row of the weight matrix stored in the register file and the input vector. Furthermore, the matrix multiplication between each of the above rows and the above input vector generates the first sub-result mentioned above.
4. The device according to claim 3, characterized in that, The synchronization unit loads the first sub-result and the second sub-result into the register file to synchronize the data.
5. A reasoning execution method, which is a reasoning execution method for devices included in a multi-device reasoning execution system, characterized in that, This includes the step of performing matrix multiplication on the data. The steps for performing matrix multiplication described above include: The step of storing the first sub-result calculated in real time into memory during the matrix multiplication process described above; During the matrix multiplication process described above, the steps include reading the first sub-result stored in the memory and transmitting the first sub-result to at least one other device included in the inference execution system. The step of receiving a second sub-result calculated in each of the at least one other device and storing the second sub-result in a memory; as well as, During the matrix multiplication process described above, the step of synchronizing data using the first sub-result and the second sub-result described above is employed. Furthermore, the aforementioned device is implemented in a manner that enables the simultaneous execution of the aforementioned matrix multiplication, the transmission of the aforementioned first sub-result, and the reception of the aforementioned second sub-result.
6. A reasoning execution method, which is a reasoning execution method for devices included in a multi-device reasoning execution system, characterized in that, This includes the step of performing matrix multiplication on the data. The steps for performing matrix multiplication described above include: The step of storing the first sub-result calculated in real time into memory during the matrix multiplication process described above; During the matrix multiplication process described above, the steps include reading the first sub-result stored in the memory and transmitting the first sub-result to at least one other device included in the inference execution system. The step of receiving a second sub-result calculated in each of the at least one other device and storing the second sub-result in a memory; as well as, During the matrix multiplication process described above, the step of synchronizing data using the first sub-result and the second sub-result described above is employed. The aforementioned multi-device mapping is applied to partitions obtained by dividing the large language model according to the intra-layer parallel approach. The aforementioned inference execution system utilizes the aforementioned large language model for inference through multi-device processing mapped to the aforementioned partitions.
7. An inference execution method, which is an inference execution method for devices included in a multi-device inference execution system, characterized in that, This includes the step of performing matrix multiplication on the data. The steps for performing matrix multiplication described above include: The step of storing the first sub-result calculated in real time into memory during the matrix multiplication process described above; During the matrix multiplication process described above, the steps include reading the first sub-result stored in the memory and transmitting the first sub-result to at least one other device included in the inference execution system. The step of receiving a second sub-result calculated in each of the at least one other device and storing the second sub-result in a memory; as well as, During the matrix multiplication process described above, the step of synchronizing data using the first sub-result and the second sub-result described above is employed. The above-described steps of performing matrix multiplication further include the step of continuously calculating the matrix multiplication between each row of the weight matrix stored in the register file and the input vector, so that the matrix multiplication between each row and the input vector continuously generates the first sub-result.
8. The reasoning execution method according to claim 7, characterized in that, The synchronization steps described above involve loading the first sub-result and the second sub-result into the register file during the matrix multiplication process to synchronize the data.
9. An inference execution system, which is a multi-device inference execution system, characterized in that, This includes multiple devices, which are mapped to partitions obtained by dividing a large language model according to an intra-layer parallel approach. Each of the aforementioned devices is implemented such that the sub-results of matrix multiplication of data are shared with other devices among the aforementioned devices during the execution of the matrix multiplication process to synchronize the data.
10. The inference execution system according to claim 9, characterized in that, Each of the aforementioned devices includes: The matrix operation unit performs matrix multiplication on the data; The sub-result storage unit stores the first sub-result calculated in real time by the matrix operation unit into the memory during the matrix multiplication process described above. The transmission unit, during the execution of the matrix multiplication described above, reads the first sub-result stored in the memory and transmits the first sub-result to at least one other device included in the inference execution system. A receiving unit that receives a second sub-result calculated in each of the at least one other device and stores the second sub-result in a memory; and, The synchronization unit synchronizes data using the first sub-result and the second sub-result during the matrix multiplication process described above.
11. The inference execution system according to claim 10, characterized in that, It is implemented in a way that allows the execution of the matrix multiplication, the transmission of the first sub-result, and the reception of the second sub-result to be performed simultaneously.
12. The inference execution system according to claim 10, characterized in that, The matrix operation unit described above continuously calculates the matrix multiplications between each row of the weight matrix stored in the register file and the input vector. Furthermore, the matrix multiplication between each of the above rows and the above input vector generates the first sub-result mentioned above.
13. The inference execution system according to claim 12, characterized in that, The synchronization unit loads the first sub-result and the second sub-result into the register file to synchronize the data.