Method and system for performing multi-device-based inference for large language model

Abstract

A method and a system for performing multi-device-based inference for a large language model are provided. The multi-device-based inference performing system according to one embodiment may comprise a plurality of devices mapped to partitions obtained by separating a large language model (LLM) according to intra-layer parallelism. Each of the plurality of devices, according to the present embodiment, can be implemented to synchronize data by sharing a sub-result of matrix multiplication for the data with other devices of the plurality of devices while the matrix multiplication is performed.

Description

Method and system for performing multi-device based inference for large-scale language models

[0001] Embodiments of the present invention relate to a method and system for performing multi-device based inference for a large-scale language model.

[0002] A Large Language Model (LLM) is a model that uses an artificial neural network to calculate the probability distribution of natural language sentences. Recently, it has been widely used in language-related tasks such as question answering and translation, such as ChatGPT.

[0003] As the size of large-scale language models continues to increase, the computational costs required for training and inference are increasing significantly. As language model sizes increase, the use of multiple accelerators for computation is becoming more common. However, while GPUs (Graphics Processing Units) are being used efficiently for training, efficient accelerators for inference remain lacking.

[0004] A method and system for performing inference as a network technique capable of effectively performing multi-device-based operations can be provided.

[0005] The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0006] In a multi-device based inference execution system, a large language model (LLM) is provided, which includes a plurality of devices that are mapped to partitions in which matrices of each layer are separated by column units according to an intra-layer parallelism method, and each of the plurality of devices is implemented to synchronize data by sharing a sub-result of matrix multiplication on data with another device of the plurality of devices while the matrix multiplication is being performed.

[0007] According to one aspect, each of the plurality of devices may include: a matrix operation unit that performs matrix multiplication on data; a sub-result storage unit that stores a first sub-result calculated in real time by the matrix operation unit in a memory while the matrix multiplication is being performed; a transmission unit that reads the first sub-result stored in the memory while the matrix multiplication is being performed and transmits it to at least one other device included in the inference execution system; a reception unit that receives a second sub-result calculated in each of the at least one other device from the at least one other device and stores it in a memory; and a synchronization unit that synchronizes data using the first sub-result and the second sub-result while the matrix multiplication is being performed.

[0008] According to another aspect, it may be characterized in that the performance of the matrix multiplication, the transmission of the first sub-result, and the reception of the second sub-result are implemented so that they can be performed simultaneously.

[0009] According to another aspect, the matrix operation unit may be characterized in that it continuously calculates matrix multiplication between each row of a weight matrix stored in a register file and an input vector, and the first sub-result is generated for each matrix multiplication between each row and the input vector.

[0010] According to another aspect, the synchronization unit may be characterized in that it synchronizes data by loading the first sub-result and the second sub-result into the register file.

[0011] A device including a multi-device based inference execution system, the device includes: a matrix operation unit that performs matrix multiplication on data; a sub-result storage unit that stores a first sub-result calculated in real time by the matrix operation unit in a memory while the matrix multiplication is being performed; a transmission unit that reads the first sub-result stored in the memory while the matrix multiplication is being performed and transmits it to at least one other device included in the inference execution system; a reception unit that receives a second sub-result calculated in each of the at least one other device from the at least one other device and stores it in a memory; and a synchronization unit that synchronizes data using the first sub-result and the second sub-result while the matrix multiplication is being performed.

[0012] A method for performing inference in a device included in a multi-device-based inference performing system, comprising: performing matrix multiplication on data, wherein the step of performing the matrix multiplication comprises: storing a first sub-result calculated in real time in a memory while the matrix multiplication is being performed; reading the first sub-result stored in the memory while the matrix multiplication is being performed and transmitting it to at least one other device included in the inference performing system; receiving a second sub-result calculated in each of the at least one other device from the at least one other device and storing it in a memory; and synchronizing data using the first sub-result and the second sub-result while the matrix multiplication is being performed.

[0013] Specific details of other embodiments are included in the detailed description and drawings.

[0014] A method and system for performing inference as a network technique capable of effectively performing multi-device-based operations can be provided.

[0015] By performing matrix multiplication and all-gather simultaneously, communication overhead and latency are reduced, providing very high scalability for the inference execution system.

[0016] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

[0017] FIG. 1 is a diagram illustrating an example of a hardware structure of a multi-device that implements an inference execution system according to one embodiment of the present invention.

[0018] FIG. 2 is a diagram illustrating an example of a timeline of a matrix multiplication operation process using a multi-device in one embodiment of the present invention.

[0019] FIG. 3 is a diagram illustrating an example of a sub-result in an intra-layer parallelism according to an embodiment of the present invention.

[0020] FIG. 4 illustrates an example of overlapping data synchronization when processing matrix multiplication operations using multi-devices in one embodiment of the present invention.

[0021] FIG. 5 is a diagram illustrating an example of the internal configuration of an inference execution system according to one embodiment of the present invention.

[0022] FIG. 6 is a diagram illustrating an example of a method for performing inference according to one embodiment of the present invention.

[0023] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms. These embodiments are provided only to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Like reference numerals designate like elements throughout the specification.

[0024] When one component is referred to as being "connected to" or "coupled to" another component, it includes both cases where it is directly connected or coupled to the other component, or cases where there is another component intervening therebetween. Conversely, when one component is referred to as being "directly connected to" or "directly coupled to" another component, it indicates that there is no other component intervening therebetween. "And / or" includes each and any combination of one or more of the mentioned items.

[0025] The terminology used herein is for the purpose of describing embodiments only and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise. As used herein, the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, operations, and / or elements.

[0026] Although terms like "first" and "second" are used to describe various components, these components are not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it should be understood that a "first" component referred to below may also be a "second" component within the technical scope of the present invention.

[0027] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in their common sense to those of ordinary skill in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0028] During the training process for large language models (LLMs), the large input volume and high computational load make efficient multi-device computation possible using graphics processing units (GPUs), which excel at high-throughput computation. Conversely, inference requires small input volumes and extensive memory access, making multi-device computations performed on GPUs significantly inefficient.

[0029] For example, the communication operations required for training and inference are fundamentally different, and because their computational characteristics differ, the optimal solutions accordingly also differ. Training is a large-batch operation with a large input, while inference is a small-batch operation with a small input. Furthermore, multi-device-based training requires four communication operations: 1) Reduce Scatter, 2) All Gather, 3) All Reduce, and 4) All-to-All, whereas inference requires only a single communication operation: All-gather. Therefore, while solutions for training can be applied to inference, their performance is significantly lower than that of solutions for inference.

[0030] Furthermore, parallelism is required to compute inference for large-scale language models on a multi-device basis. Parallelism can be broadly divided into data parallelism and model parallelism. Data parallelism does not separate the models, but maps the same model to all devices and then separates the inputs and sends them to each device. Model parallelism separates the model and maps it to each device. This model parallelism can be further divided into inter-layer parallelism and intra-layer parallelism. Inter-layer parallelism, also called pipeline parallelism, separates the model layer by layer without separating the model layers. This reduces the size of the model mapped to each device and increases computational throughput, but does not reduce the latency for a single request. In contrast, intra-layer parallelism separates each layer. This is a method that can increase throughput and reduce latency, but it has the disadvantage of requiring communication between devices after calculations on each device by separating each layer.

[0031] Embodiments of the present invention provide a method and system for performing inference as a network technique capable of effectively performing multi-device-based operations.

[0032] FIG. 1 is a diagram illustrating an example of a hardware structure of a multi-device implementing an inference execution system according to an embodiment of the present invention. The device (100) according to the embodiment of FIG. 1 may include a register file (Register File, 110), a matrix unit (Matrix Unit, 120), a load / store unit (Load / Store Unit, 130), a memory (Memory, 140), and a P2P (Peer to Peer) network (P2P, 150). At this time, each of the multi-devices implementing the inference execution system may have a hardware structure identical or similar to that of the device (100) according to the embodiment of FIG. 1.

[0033] (1) The process of matrix multiplication may be an example of a process in which the device (100) reads data from a register file (110) and performs matrix multiplication on the data through a matrix unit (120).

[0034] (2) The process of storing (Store) may be an example of a process in which the device (100) stores sub-results in the memory (140) through the load / store unit (130) to send the sub-results to the network in real time without waiting for the entire operation result of the matrix multiplication.

[0035] (3) The process of transmission (TX) may be an example of a process in which the device (100) reads the sub-result stored in the memory (140) from the memory (140) and transmits it to another device (Other Device) through a P2P network (150).

[0036] (4) The process of receiving (RX) may be an example of a process in which the device (100) gathers all the data (sub-results) transmitted from each of the multi-devices and stores them in the memory (140). In this case, the device (100) can store the sub-results directly in the memory (140) without waiting for all the data of the multi-device.

[0037] (5) The process of loading may be an example of a process in which the device (100) reads the sub-results stored in the memory (140) in (4) from the memory (140) and writes them back to the register file (110). At this time, the sub-results written back to the register file (110) may also include the sub-results stored in the memory (140) in the process (2), that is, the sub-results of the matrix multiplication performed by the device (100).

[0038] FIG. 2 is a diagram illustrating an example of a timeline of a matrix multiplication operation process using a multi-device according to an embodiment of the present invention. As shown in the embodiment of FIG. 2, the device (100) can substantially reduce communication overhead by sequentially and pipelinely performing the processes (1) to (5) described above, and can reduce latency except for the tail latency for the final operation result. This tail latency is very small, and as a result, the inference execution system according to the present embodiment can provide very high scalability.

[0039] FIG. 3 is a diagram illustrating an example of a sub-result in intra-layer parallelism according to an embodiment of the present invention. The embodiment of FIG. 3 shows an example in which an output vector (330) is generated as a result of matrix multiplication between a shared input vector (Shared Input Vector) 310 and a partitioned weight matrix (Partitioned Weight Matrix) 320. At this time, each of the multi-devices (for example, each of Device 1 to Device 4) included in the inference execution system according to the present embodiment can perform matrix multiplication between each partition of the partitioned weight matrix (320) and the shared input vector (310) for intra-layer parallelism. At this time, a first sub-result (350) can be generated as a result of Device 1 performing matrix multiplication between the first column (340) of the partitioned weight matrix (320) and the shared input vector (310). At this time, device 1 can transmit the first sub-result (350) generated first to each multi-device without waiting for the result for the entire partitioned weight matrix (320) or the result for the entire partition. In other words, each device of the inference execution system according to the present embodiment can perform matrix multiplication and all-gather simultaneously while transmitting data according to the sub-result to other devices in real time without waiting for the matrix multiplication to be completed.

[0040] This may require instruction fusion. For example, matrix multiplication instructions and all-gather instructions can be fused. Furthermore, an architecture capable of supporting instruction fusion is also required. In other words, a structure capable of simultaneously transmitting matrix multiplication and all-gathering is required. Furthermore, the devices must be able to process reception simultaneously. In other words, each multi-device included in the inference execution system can have a structure capable of simultaneously performing partitioned matrix multiplication, data transmission, and data reception in real time.

[0041] FIG. 4 illustrates an example of overlapping data synchronization when processing matrix multiplication operations using multiple devices in one embodiment of the present invention. The most important thing in multi-device-based inference operations is how to effectively perform partitioned matrix multiplication and the necessary data synchronization thereafter. Typically, a processor can start all-gather after matrix multiplication is completed. In other words, since the target data of the first instruction, matrix multiplication, is the source data of the second instruction, all-gather, all-gather can be executed after matrix multiplication is completed due to dependency check logic. The embodiment of FIG. 4 illustrates the latency when data synchronization (for example, data synchronization through all-gather) is performed after matrix multiplication is completed on the timeline, and when data synchronization is overlapped and processed. The inference execution system according to the present embodiment can perform matrix multiplication and all-gather simultaneously, sending sub-result data from each multi-device to other devices in real time without waiting for matrix multiplication to complete. This overlapping significantly reduces latency. For example, it is easy to understand that the larger the size of the partitioned weight matrix (320) illustrated in FIG. 3, the greater the latency reduction effect.

[0042] FIG. 5 is a diagram illustrating an example of the internal configuration of an inference execution system according to an embodiment of the present invention, and FIG. 6 is a diagram illustrating an example of an inference execution method according to an embodiment of the present invention.

[0043] The inference execution system (500) according to the embodiment of FIG. 5 may include a plurality of devices (device 1 (510) and other devices (520)) that are mapped to partitions in which the matrix of each layer is divided into columns according to an intra-layer parallelism method of a large-scale language model. In this case, the inference execution system (500) may process inference using a large-scale language model through a plurality of devices (510 and 520) that are mapped to partitions of the large-scale language model.

[0044] At this time, device 1 (510) may include a matrix operation unit (511), a sub-result storage unit (512), a memory (513), a transmission unit (514), a reception unit (515), and a synchronization unit (516), as illustrated in FIG. 5. Here, the matrix operation unit (511), the sub-result storage unit (512), the transmission unit (514), the reception unit (515), and the synchronization unit (516) may be functional representations of the operation of a physical processor that device 1 (510) may include. Each of the other devices (520) may also have components identical or similar to those of device 1 (510).

[0045] The method for performing inference of FIG. 6 may include step (610), and step (610) may include steps (611) to (614).

[0046] In step (610), the matrix operation unit (511) may perform matrix multiplication on data. For example, the matrix operation unit (511) may continuously calculate matrix multiplication between each row of a weight matrix stored in a register file and an input vector, thereby continuously generating a first sub-result for each matrix multiplication between each row and the input vector. In this case, steps (611) to (614) may be performed while performing matrix multiplication in step (610).

[0047] In step (611), the sub-result storage unit (512) can store the first sub-result calculated in real time in the memory (513). For example, assuming that the matrix operation unit (511) calculates four sub-results from the 1-1st sub-result to the 1-4th sub-result, the 1-1st sub-result can be stored in the memory (513) during the process of calculating the 1-2nd sub-result, and the 1-2nd sub-result can be stored in the memory (513) during the process of calculating the 1-3rd sub-result. In the subsequent steps (612) to (614), each sub-result can also be processed in the process of calculating the next sub-result.

[0048] In step (612), the transmission unit (514) can read the first sub-result stored in the memory (513) and transmit it to each of the other devices (520) included in the inference execution system (500). At this time, each of the other devices (520) can also calculate the second sub-result during the performance of matrix multiplication and transmit it to each of the devices other than itself.

[0049] In step (613), the receiving unit (515) may receive the second sub-results calculated from each of the other devices (520) and store them in the memory (513). In this case, both the first sub-result and the second sub-result may be stored in the memory (513). Depending on the embodiment, the transmitting unit (514) and the receiving unit (515) may also process a process of linking the first sub-result with the second sub-result and storing them again in the memory (513).

[0050] In step (614), the synchronization unit (516) can synchronize data using the first sub-result and the second sub-result. For example, the synchronization unit (516) can synchronize data by loading the first sub-result and the second sub-result into a register file.

[0051] Thus, according to embodiments of the present invention, a method and system for performing inference as a network technique capable of effectively performing multi-device-based operations can be provided. Furthermore, by simultaneously performing matrix multiplication and all-gather, communication overhead and latency can be reduced, providing extremely high scalability for the inference system.

[0052] Although the embodiments of the present invention have been described with reference to the attached drawings, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without altering the technical concept or essential features thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

Claims

1. In a device including a multi-device based inference execution system, A matrix operation unit that performs matrix multiplication on data; A sub-result storage unit that stores a first sub-result calculated in real time by the matrix operation unit in memory while the matrix multiplication is being performed; A transmission unit that reads the first sub-result stored in the memory and transmits it to at least one other device included in the inference execution system while the matrix multiplication is being performed; A receiving unit that receives a second sub-result calculated in each of the at least one other device from the at least one other device and stores it in a memory; and A synchronization unit that synchronizes data using the first sub-result and the second sub-result while the above matrix multiplication is being performed. Including, A device characterized in that the performance of the matrix multiplication, the transmission of the first sub-result, and the reception of the second sub-result are implemented so that they can be performed simultaneously.

2. In a device including a multi-device based inference execution system, A matrix operation unit that performs matrix multiplication on data; A sub-result storage unit that stores a first sub-result calculated in real time by the matrix operation unit in memory while the matrix multiplication is being performed; A transmission unit that reads the first sub-result stored in the memory and transmits it to at least one other device included in the inference execution system while the matrix multiplication is being performed; A receiving unit that receives a second sub-result calculated in each of the at least one other device from the at least one other device and stores it in a memory; and A synchronization unit that synchronizes data using the first sub-result and the second sub-result while the above matrix multiplication is being performed. Including, The above multi-device is mapped to partitions separated by an intra-layer parallelism method of a large language model (LLM), The above inference execution system processes inference using the large-scale language model through multi-devices mapped to the above partitions. A device characterized by .

3. In a device including a multi-device based inference execution system, A matrix operation unit that performs matrix multiplication on data; A sub-result storage unit that stores a first sub-result calculated in real time by the matrix operation unit in memory while the matrix multiplication is being performed; A transmission unit that reads the first sub-result stored in the memory and transmits it to at least one other device included in the inference execution system while the matrix multiplication is being performed; A receiving unit that receives a second sub-result calculated in each of the at least one other device from the at least one other device and stores it in a memory; and A synchronization unit that synchronizes data using the first sub-result and the second sub-result while the above matrix multiplication is being performed. Including, The above matrix operation unit continuously calculates matrix multiplication between each row of the weight matrix stored in the register file and the input vector, The first sub-result is generated for each matrix multiplication between each row and the input vector. A device characterized by .

4. In paragraph 3, A device characterized in that the synchronization unit loads the first sub-result and the second sub-result into the register file to synchronize data.

5. In a method for performing inference on a device including a multi-device based inference performing system, Steps to perform matrix multiplication on data Including, The steps of performing the above matrix multiplication are: A step of storing a first sub-result calculated in real time in memory while the above matrix multiplication is being performed; A step of reading the first sub-result stored in the memory and transmitting it to at least one other device included in the inference execution system while the matrix multiplication is being performed; A step of receiving a second sub-result calculated in each of the at least one other device from the at least one other device and storing the same in a memory; and A step of synchronizing data using the first sub-result and the second sub-result while the above matrix multiplication is being performed. Including, A method for performing inference, characterized in that the device is implemented so that the performance of the matrix multiplication, the transmission of the first sub-result, and the reception of the second sub-result can be performed simultaneously.

6. In a method for performing inference on a device including a multi-device based inference performing system, Steps to perform matrix multiplication on data Including, The steps of performing the above matrix multiplication are: A step of storing a first sub-result calculated in real time in memory while the above matrix multiplication is being performed; A step of reading the first sub-result stored in the memory and transmitting it to at least one other device included in the inference execution system while the matrix multiplication is being performed; A step of receiving a second sub-result calculated in each of the at least one other device from the at least one other device and storing the same in a memory; and A step of synchronizing data using the first sub-result and the second sub-result while the above matrix multiplication is being performed. Including, The above multi-device is mapped to partitions separated by an intra-layer parallelism method of a large language model (LLM), The above inference execution system processes inference using the large-scale language model through multi-devices mapped to the above partitions. A method of performing inference characterized by .

7. In a method for performing inference on a device including a multi-device based inference performing system, Steps to perform matrix multiplication on data Including, The steps of performing the above matrix multiplication are: A step of storing a first sub-result calculated in real time in memory while the above matrix multiplication is being performed; A step of reading the first sub-result stored in the memory and transmitting it to at least one other device included in the inference execution system while the matrix multiplication is being performed; A step of receiving a second sub-result calculated in each of the at least one other device from the at least one other device and storing the same in a memory; and A step of synchronizing data using the first sub-result and the second sub-result while the above matrix multiplication is being performed. Including, The steps of performing the above matrix multiplication are: A step of continuously calculating matrix multiplication between each row of a weight matrix stored in a register file and an input vector, and continuously generating the first sub-result for each matrix multiplication between each row and the input vector. A method for performing inference, characterized in that it further includes.

8. In paragraph 7, The above synchronization steps are: A method for performing inference, characterized in that while the matrix multiplication is being performed, the first sub-result and the second sub-result are loaded into the register file to synchronize data.

9. In a multi-device based inference execution system, Multiple devices mapped to partitions separated by an intra-layer parallelism method of a large language model (LLM). Including, Each of the above multiple devices, It is implemented to synchronize data by sharing a sub-result of matrix multiplication on data with other devices of the plurality of devices while the matrix multiplication is being performed. An inference performing system characterized by .

10. In paragraph 9, Each of the above multiple devices, A matrix operation unit that performs matrix multiplication on data; A sub-result storage unit that stores the first sub-result calculated in real time by the matrix operation unit in memory while the matrix multiplication is being performed; A transmission unit that reads the first sub-result stored in the memory and transmits it to at least one other device included in the inference execution system while the matrix multiplication is being performed; A receiving unit that receives a second sub-result calculated in each of the at least one other device from the at least one other device and stores it in a memory; and A synchronization unit that synchronizes data using the first sub-result and the second sub-result while the above matrix multiplication is being performed. An inference performing system characterized by including:

11. In paragraph 10, An inference execution system characterized in that the execution of the matrix multiplication, the transmission of the first sub-result, and the reception of the second sub-result are implemented so that they can be performed simultaneously.

12. In paragraph 10, The above matrix operation unit continuously calculates matrix multiplication between each row of the weight matrix stored in the register file and the input vector, The first sub-result is generated for each matrix multiplication between each row and the input vector. An inference performing system characterized by .

13. In paragraph 12, An inference execution system characterized in that the synchronization unit synchronizes data by loading the first sub-result and the second sub-result into the register file.

Images