Efficient data forwarding method and system for accelerating large language model inference

Abstract

An efficient data forwarding method and system for accelerating large language model inference are provided. A data forwarding method, according to one embodiment, may comprise the steps of: adding, to an instruction, a bit indicating whether data associated with the instruction is to be forwarded during a compiling process; processing, by a processing apparatus, the instruction by using the data; and forwarding, by the processing apparatus, the data on the basis of the bit added to the instruction.

Description

An efficient data forwarding method and system for accelerating large-scale language model inference.

[0001] Embodiments of the present invention relate to an efficient data forwarding method and system for accelerating large-scale language model inference, and to a method for reducing the latency of large-scale language model inference having a sequential nature in which the output of a previous command must be used as the input of the next command.

[0002] Large-scale language model inference requiring low latency presents several key challenges.

[0003] 1) Large-scale tasks require significant hardware resources, which increases the physical distance between local on-chip memory on some processing units. This limitation requires several clock cycles to read and write data to reduce critical path and timing violations, significantly increasing latency.

[0004] 2) Due to the sequential nature of the model, there are many dependencies between instructions, requiring data forwarding. However, the original approach of identifying the source and destination for each pipeline stage and then mapping the path from each pipeline stage to each input source is wasteful for large language models because 1) each path is unnecessary and 2) the conditional logic for verifying forwarding can be extensive.

[0005] 3) Tasks like residuals require the output of several commands as input, which cannot be handled by the original forwarding method. Therefore, unless forwarding between intermediate commands is needed for other purposes, a method is needed to store (maintain) the data nearby rather than storing it in local memory.

[0006] A compiler-based data forwarding method and system can be provided that can reduce the latency of large-scale language model inference with sequential characteristics that require the output of a previous command to be used as the input of the next command.

[0007] The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0008] A data forwarding method is provided, comprising: a step of adding a bit to an instruction indicating whether data related to the instruction is forwarded during a compilation process; a step of processing the instruction using the data in a processing device; and a step of forwarding the data based on the bit added to the instruction in the processing device.

[0009] According to one aspect, the adding step may be characterized by adding to the instruction a first bit indicating whether data of a first source for the instruction is forwarded, a second bit indicating whether data of a second source for the instruction is forwarded, and a third bit indicating whether a processing result of the instruction is forwarded.

[0010] According to another aspect, the forwarding step may be characterized by forwarding at least one of data of the first source, data of the second source, and a processing result of the instruction through the first bit, the second bit, and the third bit.

[0011] According to another aspect, the data forwarding method may further include a step of configuring a forwarding buffer in a memory included in the processing device, and the forwarding step may be characterized by forwarding the data to the configured forwarding buffer.

[0012] According to another aspect, the forwarding buffer may be characterized as having the form of a FIFO (First In First Out) buffer having a depth equal to the length of the destination of the instruction.

[0013] According to another aspect, the forwarding buffer may include a scalar forwarding buffer for storing a scalar value and a vector forwarding buffer for storing a vector value, and the forwarding step may be characterized by storing the data in the scalar forwarding buffer or the vector forwarding buffer depending on whether the data is a scalar or a vector.

[0014] According to another aspect, the processing device may include a matrix processing device and a vector processing device, and the processing step may be characterized in that different instructions are processed in the matrix processing device and the vector processing device, respectively.

[0015] According to another aspect, the forwarding step may be characterized by including: a step of changing the precision of the data from a first precision to a second precision through a first converter included in the processing device; and a step of changing the precision of the data from the second precision back to the first precision after processing the instruction through a second converter included in the processing device.

[0016] According to another aspect, the first precision may include a precision of 16 bits of floating point, and the second precision may include a precision of 32 bits of floating point.

[0017] According to another aspect, the forwarding step may be characterized by including a step of further forwarding the data of the second precision to a register included in the processing device for reuse of the data changed to the second precision.

[0018] In a data forwarding system, a data forwarding system is provided, comprising: a bit addition unit that adds a bit to an instruction during a compilation process to indicate whether data related to the instruction is forwarded; and a processing device that processes the instruction using the data and forwards the data based on the bit added to the instruction.

[0019] Specific details of other embodiments are included in the detailed description and drawings.

[0020] A compiler-based data forwarding method and system can be provided that can reduce the latency of large-scale language model inference with sequential characteristics that require the output of a previous command to be used as the input of the next command.

[0021] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

[0022] Figure 1 is a drawing showing an example of an instruction processing process according to conventional technology.

[0023] FIG. 2 is a diagram illustrating an example of an instruction processing process according to one embodiment of the present invention.

[0024] FIG. 3 is an example for explaining data forwarding of sequential instructions in one embodiment of the present invention.

[0025] FIG. 4 is a diagram illustrating an example of an execution process of an instruction according to one embodiment of the present invention.

[0026] FIG. 5 is a diagram illustrating an example of the internal configuration of a data forwarding system according to one embodiment of the present invention.

[0027] FIG. 6 is a flowchart illustrating an example of a data forwarding method according to one embodiment of the present invention.

[0028] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms. These embodiments are provided only to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Like reference numerals designate like elements throughout the specification.

[0029] When one component is referred to as being "connected to" or "coupled to" another component, it includes both cases where it is directly connected or coupled to the other component, or cases where there is another component intervening therebetween. Conversely, when one component is referred to as being "directly connected to" or "directly coupled to" another component, it indicates that there is no other component intervening therebetween. "And / or" includes each and any combination of one or more of the mentioned items.

[0030] The terminology used herein is for the purpose of describing embodiments only and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise. As used herein, the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, operations, and / or elements.

[0031] Although terms like "first" and "second" are used to describe various components, these components are not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it should be understood that a "first" component referred to below may also be a "second" component within the technical scope of the present invention.

[0032] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in their common sense to those of ordinary skill in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0033] Embodiments of the present invention can provide a compiler-based data forwarding method and system that can reduce the latency of large-scale language model inference having sequential characteristics that require the output of a previous command to be used as the input of a next command.

[0034] Figure 1 is a drawing showing an example of an instruction processing process according to conventional technology.

[0035] In the prior art of FIG. 1, the instruction processing process may be composed of the processes of instruction fetch (hereinafter referred to as 'IF (110)'), instruction decoding (hereinafter referred to as 'DEC (120)'), execution (hereinafter referred to as 'EXE (130)'), and write back (hereinafter referred to as 'WB (140)'). For example, IF (110) may be a process in which a processing unit (e.g., a CPU (Central Processing Unit)) reads an instruction from a memory (e.g., 'MEM (150)') in order to execute the instruction. DEC (120) may be a process in which registers (e.g., a scalar register and a vector register of FIG. 1) required to perform an operation as indicated by the instruction are selected, and an ALU (Arithmetic Logic Unit) interprets which operation to perform. EXE (130) may be a process for executing an instruction, and WB (140) may be a process for storing the result of the operation in a target register. At this time, EXE (130) may be configured by dividing into a pipeline stage for a matrix processing unit (hereinafter referred to as 'MPU (131)') and a pipeline stage for a vector processing unit (hereinafter referred to as 'VPU (132)'), and each of MEM (150) and WB (140) may also be configured by dividing into a pipeline stage for storing or writing back the output of MPU (131) and VPU (132).

[0036] At this time, in the prior art of Fig. 1, data forwarding is required because there are many dependencies between instructions due to the sequential nature of the artificial intelligence model. In this prior art, a forwarding unit (hereinafter referred to as 'FWD Unit', 160) is required to identify the source and destination for each pipeline stage, which has a path from each pipeline stage to each input source of each pipeline. At this time, the FWD Unit (160) requires logic to compare the destination of an instruction with the source of the next instruction to determine whether to forward.

[0037] FIG. 2 is a diagram illustrating an example of an instruction processing process according to one embodiment of the present invention.

[0038] The instruction processing process according to the present embodiment may also be similar to the prior art of FIG. 1, and may be composed of the processes of instruction fetch (hereinafter referred to as 'IF (210)'), instruction decoding (hereinafter referred to as 'DEC (220)'), execution (hereinafter referred to as 'EXE (230)'), and writeback (hereinafter referred to as 'WB (140)'). In addition, the EXE (230) may be configured by dividing into a pipeline stage for a matrix processing unit (hereinafter referred to as 'MPU (231)') and a pipeline stage for a vector processing unit (hereinafter referred to as 'VPU (232)'), and each of the MEM (250) and the WB (240) may also be configured by dividing into a pipeline stage for storing or writing back the outputs of the MPU (231) and the VPU (232).

[0039] Meanwhile, in the present embodiment, a vector FIFO (First In First Out) buffer (hereinafter referred to as 'v_fwd FIFO (261, 271)') and a scalar FIFO buffer (hereinafter referred to as 's_fwd FIFO (262, 272)') having a large depth equal to the length of the destination of the instruction can be created, and can be arranged closest to each processing unit, MPU (231) and VPU (232). In the embodiment of Fig. 2, an example is shown in which v_fwd FIFO (261) and s_fwd FIFO (262) are arranged in conjunction with MPU (231), and v_fwd FIFO (271) and s_fwd FIFO (272) are arranged in conjunction with VPU (231). At this time, large depths may require fewer hardware resources because FPGAs (Field Programmable Gate Arrays) have built-in random access memory (RAM).

[0040] In this embodiment, dependencies between adjacent instructions, as well as a small number of adjacent instructions, can be analyzed during compilation. By including information on whether the next data should be passed to the instruction itself, unlike conventional techniques, logic for identifying the source, destination, and pipeline stage to which data should be transferred (e.g., the FWD Unit (160) of FIG. 1) is not required.

[0041] In practice, since the output of the VPU (232) is only transferred to the vector forwarding buffers v_fwd FIFO (261) and v_fwd FIFO (271), and the output of the MPU (231) is only transferred to the matrix forwarding buffers s_fwd FIFO (262) and s_fwd FIFO (272), all data forwarded during writeback can be forwarded to the appropriate buffer.

[0042] FIG. 3 is an example for explaining data forwarding of sequential instructions in one embodiment of the present invention. Whether the output of the MPU (231) and / or the VPU (232) is a vector or a scalar is also part of the instruction. Having a buffer is useful because it allows data to be forwarded without having to visit the register file when data forwarding is required for non-adjacent instructions but not for adjacent instructions. For example, in the table of FIG. 3, R3(S), which is an output of cycle 0, cannot be forwarded in the prior art because it is not used in the next instruction. On the other hand, in the embodiments of the present invention, by specifying that the output of R3(S), which is an output of cycle 0, is to be used later through 3 bits ("001" in cycle 0) indicating whether data is forwarded, R3(S) can be forwarded.

[0043] In other words, the compiler can ensure that the data passed to the buffer is used and then emptied before receiving other data. To this end, three bits can be added to the instruction as described above. One of the three bits (bit 0) can indicate whether the output is forwarded. Another of the three bits (bit 1) can indicate whether data from Source 2 is forwarded. The remaining one of the three bits (bit 2) can indicate whether data from Source 1 is forwarded. According to the table in Figure 3, since the three bits "111" in Cycle 2 confirm that the output R6(V) of Cycle 2 will be used later, the output R6(V) can be forwarded.

[0044] Furthermore, separating vectors and scalars allows for a combination of both types for inference, significantly increasing the opportunities for forwarding. For example, if an instruction outputs a vector, the next instruction may output a scalar, and the final instruction may use the vector and scalar values ​​from the previous two instructions as input. The forwarding buffer can quickly prepare data for the final instruction.

[0045] On the other hand, there is a drawback in that data forwarding is not allowed between different processing units. However, since switching between the MPU (231) and VPU (232) is minimal, the hardware overhead supporting this feature is not worth the overhead.

[0046] FIG. 4 is a diagram illustrating an example of an instruction execution process according to an embodiment of the present invention. As an additional forwarding method, a special forwarding buffer may be placed in a location where high precision is required. Since a specific task requires high precision, the data may be converted from a floating point 16-bit (FP16) to a floating point 32-bit (FP32) through an FP16-FP32 converter (FP16-FP32 Converter, 410 and 420) before entering the processing unit (VPU (232) in the embodiment of FIG. 4). At this time, the result of the instruction processed in the VPU (232) may be converted back to FP16 through an FP32-FP16 converter (FP32-FP16 Converter, 450). To this end, in the embodiment of FIG. 4, a 4-to-1 multiplexer (MUX), 430 and 440, may be used to allow a processing device to select and use one of the outputs of the s_fwd FIFO (272) and the v_fwd FIFO (271) for using data of FP16 precision, and the output of the FP16-FP32 converter (FP16-FP32 Converter, 410 and 420) for using data of FP32 precision.

[0047] Additionally, for certain instructions that are sensitive to scalar bit loss (e.g., division), a register (460) of depth 1 may be placed in the VPU (232) to maintain FP32 precision for this special case. In other words, FP32 precision data can be stored in the register (460) for reuse before being converted back to FP16 precision data through the FP32-FP16 Converter (450).

[0048] FIG. 5 is a diagram illustrating an example of the internal configuration of a data forwarding system according to an embodiment of the present invention, and FIG. 6 is a flowchart illustrating an example of a data forwarding method according to an embodiment of the present invention.

[0049] The data forwarding system (500) according to the present embodiment may include a bit addition unit (510) and a processing unit (520). The bit addition unit (510) may be a functional expression of a processing unit that processes compilation for an artificial intelligence model. In addition, the processing unit (520) is a hardware module that processes actual instructions and may include the MPU (231) and / or the VPU (232) described above. In addition, the processing unit (520) may include a forwarding buffer (530). For example, the forwarding buffer (530) may be implemented as a FIFO buffer in a memory (for example, a RAM built into the FPGA described above) included in the processing unit (520). Depending on the embodiment, the hardware device that constitutes the bit addition unit (510) and the hardware device that implements the processing unit (520) may be different from each other. In this case, the two hardware devices may communicate with each other through a network and perform a data forwarding method.

[0050] In step (610), the data forwarding system (500) can add a bit indicating whether data related to an instruction is forwarded to the instruction during the compiling process through the bit adding unit (510). For example, the bit adding unit (510) can add a first bit indicating whether data of a first source for the instruction is forwarded during the compiling process, a second bit indicating whether data of a second source for the instruction is forwarded, and a third bit indicating whether a processing result of the instruction is forwarded to the instruction. At this time, the bit adding unit (510) can generate a 3-bit value of the first bit, the second bit, and the third bit described above for each of a plurality of instructions for inference of an artificial intelligence model and add the generated 3-bit value to the corresponding instruction.

[0051] At step (620), the data forwarding system (500) may process instructions using data via the processing unit (520). As previously described, the instructions may be instructions for inferring an artificial intelligence model, and depending on the embodiment, the processing unit (520) may include an accelerator configured for the corresponding artificial intelligence model. In this case, the instructions may be processed via such an accelerator.

[0052] According to an embodiment, the processing unit (520) may include a matrix processing unit and a vector processing unit. In this case, the processing unit (520) may process different instructions in the matrix processing unit and the vector processing unit, respectively.

[0053] At step (630), the data forwarding system (500) may forward data based on bits added to the instruction via the processing unit (520). For example, the processing unit (520) may forward at least one of data from a first source, data from a second source, and a processing result of the instruction via the first bit, the second bit, and the third bit.

[0054] According to an embodiment, the data forwarding method may further include a step (not shown) of configuring a forwarding buffer in a memory (e.g., RAM built into an FPGA) included in the processing device (520). In this case, the processing device (520) may forward data to the forwarding buffer configured in the memory in step (630). This forwarding buffer may have the form of a FIFO buffer having a depth equal to the length of the destination of the instruction. In addition, the forwarding buffer may include a scalar forwarding buffer for storing scalar values ​​(e.g., s_fwd FIFOs (262, 272) of FIG. 2) and a vector forwarding buffer for storing vector values ​​(e.g., v_fwd FIFOs (261, 271) of FIG. 2). In this case, the processing device (520) may store the data in a scalar forwarding buffer or a vector forwarding buffer, depending on whether the data is a scalar or a vector, at step (630).

[0055] In addition, according to an embodiment, the data forwarding system (500) may change the precision of data from the first precision to the second precision through the first converter (for example, the FP16-FP32 converters (410 and 420) of FIG. 4) included in the processing device (520). In addition, the data forwarding system (500) may change the precision of data from the second precision back to the first precision through the second converter (for example, the FP32-FP16 converter (450) of FIG. 4) after processing the instruction included in the processing device (520). Here, the first precision may include a precision of 16 bits of floating point, and the second precision may include a precision of 32 bits of floating point.

[0056] Additionally, according to an embodiment, the data forwarding system (500) may further forward second precision data to a register (e.g., register (460) of FIG. 4) included in the processing device (520) for reuse of data converted to second precision. Accordingly, second precision data may be reused without reconversion to first precision through the register.

[0057] In this way, according to embodiments of the present invention, a compiler-based data forwarding method and system can be provided that can reduce the latency of large-scale language model inference having sequential characteristics that require the output of a previous command to be used as the input of the next command.

[0058] Although the embodiments of the present invention have been described with reference to the attached drawings, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without altering the technical concept or essential features thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

Claims

1. A step of adding a bit to the instruction to indicate whether data related to the instruction is forwarded during the compilation process; A step of processing the instruction using the data in the processing device; and A step of forwarding the data based on the bit added to the instruction in the processing device. Including, The steps to add above are: A data forwarding method characterized by adding to the instruction a first bit indicating whether data of a first source for the instruction is forwarded, a second bit indicating whether data of a second source for the instruction is forwarded, and a third bit indicating whether a processing result of the instruction is forwarded.

2. In paragraph 1, The above forwarding step is, A data forwarding method characterized in that at least one of data of the first source, data of the second source, and a processing result of the instruction is forwarded through the first bit, the second bit, and the third bit.

3. A step of adding a bit to the instruction to indicate whether data related to the instruction is forwarded during the compilation process; A step of processing the instruction using the data in the processing device; and A step of forwarding the data based on the bit added to the instruction in the processing device. Including, A step of configuring a forwarding buffer in a memory included in the above processing device. Including more, The above forwarding step is, A data forwarding method characterized by forwarding the data to the forwarding buffer configured above.

4. In paragraph 3, A data forwarding method, characterized in that the forwarding buffer has the form of a FIFO (First In First Out) buffer having a depth equal to the length of the destination of the instruction.

5. In paragraph 3, The above forwarding buffer includes a scalar forwarding buffer for storing scalar values ​​and a vector forwarding buffer for storing vector values, The above forwarding step is, Storing the data in the scalar forwarding buffer or the vector forwarding buffer depending on whether the data is a scalar or a vector. A data forwarding method characterized by .

6. A step of adding a bit to the instruction to indicate whether data related to the instruction is forwarded during the compilation process; A step of processing the instruction using the data in the processing device; and A step of forwarding the data based on the bit added to the instruction in the processing device. Including, The above processing device includes a matrix processing device and a vector processing device, The above processing steps are: Processing different instructions in the matrix processing unit and the vector processing unit, respectively A data forwarding method characterized by .

7. A step of adding a bit to the instruction to indicate whether data related to the instruction is forwarded during the compilation process; A step of processing the instruction using the data in the processing device; and A step of forwarding the data based on the bit added to the instruction in the processing device. Including, The above forwarding step is, A step of changing the precision of the data from a first precision to a second precision through a first converter included in the processing device; and A step of changing the precision of the data from the second precision back to the first precision after processing the instruction through the second converter included in the processing device. A data forwarding method characterized by including:

8. In paragraph 7, The above first precision includes a precision of 16 bits of floating point, The above second precision includes the precision of 32 bits of floating point. A data forwarding method characterized by .

9. In paragraph 7, The above forwarding step is, A step of further forwarding the data of the second precision to a register included in the processing device for reuse of the data changed to the second precision. A data forwarding method characterized by including:

10. In the data forwarding system, A bit addition unit that adds a bit to the instruction to indicate whether data related to the instruction is forwarded during the compilation process; and A processing device that processes the instruction using the above data and forwards the data based on the bits added to the instruction. Including, The above bit addition part is, A data forwarding system characterized in that a first bit indicating whether data of a first source for the instruction is forwarded, a second bit indicating whether data of a second source for the instruction is forwarded, and a third bit indicating whether a processing result of the instruction is forwarded are added to the instruction.

11. In the data forwarding system, A bit addition unit that adds a bit to the instruction to indicate whether data related to the instruction is forwarded during the compilation process; and A processing device that processes the instruction using the above data and forwards the data based on the bits added to the instruction. Including, The above processing device, memory; and a forwarding buffer configured in the memory, Forwarding the data to the forwarding buffer A data forwarding system featuring:

12. In paragraph 11, The above forwarding buffer includes a scalar forwarding buffer for storing scalar values ​​and a vector forwarding buffer for storing vector values, The processing device stores the data in the scalar forwarding buffer or the vector forwarding buffer depending on whether the data is scalar or vector. A data forwarding system featuring:

13. In the data forwarding system, A bit addition unit that adds a bit to the instruction to indicate whether data related to the instruction is forwarded during the compilation process; and A processing device that processes the instruction using the above data and forwards the data based on the bits added to the instruction. Including, The above processing device, A first converter that changes the precision of the above data from a first precision to a second precision; and A second converter that changes the precision of the data from the second precision back to the first precision after processing the above instruction. A data forwarding system comprising:

14. In paragraph 13, The above processing device, To reuse data changed to the second precision, a register that forwards data of the second precision A data forwarding system characterized by further including:

Images