Mixed-precision mac tree structure for maximizing memory bandwidth usage to accelerate operations of generative large language model

Abstract

Provided is a mixed-precision multiply-and-accumulate (MAC) tree structure for maximizing memory bandwidth usage in order to accelerate the operations of a generative large language model. A MAC tree-based computational device according to one embodiment may comprise: a plurality of floating-point multipliers that are connected in parallel and process multiplication operations for data transmitted from an external memory; a plurality of first converters for converting the output of each of the plurality of floating-point multipliers from floating-point to fixed-point; a fixed-point adder tree that is connected to the plurality of first converters and processes the addition of the multiplication results of the plurality of floating-point multipliers; a fixed-point accumulator for accumulating the outputs of the fixed-point adder tree; and a second converter for converting the output of the fixed-point accumulator from fixed point to floating point.

Description

A mixed-precision MAC tree structure to maximize memory bandwidth utilization for accelerating the computation of generative giant language models.

[0001] Embodiments of the present invention relate to a mixed-precision MAC tree structure for maximizing memory bandwidth usage for accelerating computation of a generative giant language model.

[0002] With the rapid development of generative large-scale language models, the size of these models is steadily increasing to achieve high accuracy, with model parameters ranging from millions to billions. Consequently, massive amounts of data must be retrieved from memory at once to perform computations without interruption.

[0003] Additionally, floating-point arithmetic units are required to support generative large-scale language model operations without loss of accuracy. However, these floating-point arithmetic units suffer from high logic complexity and, consequently, large area requirements.

[0004] It is possible to provide a hardware accelerator's computation unit that maximizes the use of the provided memory bandwidth for acceleration of generative large language models that are difficult to parallelize and have large amounts of data.

[0005] The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0006] A MAC (multiply-and-accumulation) tree-based calculator is provided, comprising: a plurality of floating-point multipliers connected in parallel to process a multiplication operation on data transferred from an external memory; a plurality of first converters for converting outputs of each of the plurality of floating-point multipliers from floating-point to fixed-point; a fixed-point adder tree connected to the plurality of first converters to process addition of multiplication results of the plurality of floating-point multipliers; a fixed-point accumulator for accumulating outputs of the fixed-point adder tree; and a second converter for converting outputs of the fixed-point accumulator from fixed-point to floating-point.

[0007] According to one side, the MAC tree-based operator may be characterized as corresponding to one of a plurality of MAC tree-based operators included in a hardware accelerator for acceleration of an artificial intelligence model.

[0008] According to another aspect, it may be characterized in that at least one of the number of MAC tree-based operators included in the hardware accelerator and the number of the plurality of floating-point multipliers included in the MAC tree-based operator is determined according to the memory bandwidth provided for the hardware accelerator.

[0009] According to another aspect, the plurality of MAC tree-based operators may be characterized by performing a matrix multiplication operation for at least one partition among the plurality of partitions implementing the artificial intelligence model.

[0010] According to another aspect, the external memory may be characterized by including a high-bandwidth memory in which the at least one partition is stored.

[0011] According to another aspect, each of the plurality of floating-point multipliers may be characterized by including a mixed-precision fixed-point exponent adder for addition of exponents; and a mixed-precision fixed-point mantissa multiplier for multiplication of mantissas.

[0012] According to another aspect, each of the plurality of floating-point multipliers may be characterized in that it processes a multiplication between a first operand and a second operand having the same bit precision when a high-precision mode is selected, thereby calculating a first result value.

[0013] According to another aspect, each of the plurality of floating-point multipliers may be characterized in that, when a high-performance mode is selected, the first multiplication between the third operand of the first bit precision and the 4-1 operand of the second bit precision and the second multiplication between the third operand and the 4-2 operand of the second bit precision are simultaneously processed to calculate the second result value of the first multiplication and the third result value of the second multiplication simultaneously.

[0014] According to another aspect, the first bit precision may include a 16-bit precision, and the second bit precision may include an 8-bit precision.

[0015] A method for operating a MAC (multiply-and-accumulation) tree-based calculator is provided, wherein the MAC tree-based calculator includes a plurality of floating-point multipliers connected in parallel, a plurality of first converters connected to the plurality of floating-point multipliers, a fixed-point adder tree, a fixed-point accumulator, and a second converter, the method comprising: processing a multiplication operation on data transferred from an external memory using the plurality of floating-point multipliers; converting a result of a multiplication operation of each of the plurality of floating-point multipliers from a floating-point to a fixed-point using the plurality of first converters; processing an addition of the converted results of the plurality of floating-point multipliers using the fixed-point adder tree; accumulating an output of the fixed-point adder tree using the fixed-point accumulator; and converting an output of the fixed-point accumulator from a fixed-point to a floating-point using the second converter.

[0016] Specific details of other embodiments are included in the detailed description and drawings.

[0017] It is possible to provide a hardware accelerator's computation unit that maximizes the use of the provided memory bandwidth for acceleration of generative large language models that are difficult to parallelize and have large amounts of data.

[0018] It is possible to provide an operator that can reduce the high hardware resource overhead of a floating-point-based MAC tree structure.

[0019] It is possible to provide an operator that can effectively support two-precision operations while minimizing hardware resource overhead.

[0020] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

[0021] FIG. 1 is a diagram illustrating an example of the structure of a latency processing unit according to an embodiment of the present invention.

[0022] FIGS. 2 to 5 are drawings illustrating examples of implementation models of LPUs according to embodiments of the present invention.

[0023] FIG. 6 is a diagram for explaining weight matrix data mapping of a high-bandwidth memory for matrix multiplication operation of a latency processing unit according to an embodiment of the present invention.

[0024] FIG. 7 is a diagram for explaining a high-bandwidth memory interface included in a latency processing unit according to one embodiment of the present invention.

[0025] FIG. 8 is a diagram illustrating a reconfigurable multi-function operation unit included in a latency processing unit according to one embodiment of the present invention.

[0026] FIG. 9 is a diagram for explaining the configuration of an address-based non-sequential multi-unit scheduler included in a latency processing unit according to one embodiment of the present invention.

[0027] FIG. 10 is a diagram illustrating an example of the internal configuration of a MAC tree in one embodiment of the present invention.

[0028] Fig. 11 is a drawing for explaining the performance of an operator according to one embodiment of the present invention.

[0029] FIG. 12 is a drawing for explaining a mixed-precision floating-point multiplier according to one embodiment of the present invention.

[0030] FIG. 12 is a diagram illustrating an example of the structure of a mixed-precision floating-point multiplier according to one embodiment of the present invention.

[0031] FIG. 13 is a diagram illustrating examples of two modes of a mixed-precision floating-point multiplier according to one embodiment of the present invention.

[0032] FIG. 14 is a diagram illustrating an example of a mixed-precision fixed-point mantissa multiplier according to an embodiment of the present invention.

[0033] FIG. 15 is a diagram illustrating an example of a mixed-precision fixed-point exponent adder according to an embodiment of the present invention.

[0034] Fig. 16 is a flowchart illustrating an example of an operation method of a MAC tree-based operator according to one embodiment of the present invention.

[0035] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms. These embodiments are provided only to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Like reference numerals designate like elements throughout the specification.

[0036] When one component is referred to as being "connected to" or "coupled to" another component, it includes both cases where it is directly connected or coupled to the other component, or cases where there is another component intervening therebetween. Conversely, when one component is referred to as being "directly connected to" or "directly coupled to" another component, it indicates that there is no other component intervening therebetween. "And / or" includes each and any combination of one or more of the mentioned items.

[0037] The terminology used herein is for the purpose of describing embodiments only and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise. As used herein, the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, operations, and / or elements.

[0038] Although terms like "first" and "second" are used to describe various components, these components are not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it should be understood that a "first" component referred to below may also be a "second" component within the technical scope of the present invention.

[0039] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in their common sense to those of ordinary skill in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0040] FIG. 1 is a diagram illustrating an example of the structure of a latency processing unit according to an embodiment of the present invention.

[0041] Referring to FIG. 1, an LPU (Latency Processing Unit) 100 according to an embodiment of the present invention may include an SMA (Streamlined Memory Access) 110, an OIU (Operand Issue Unit) 120, an SXE (Streamlined eXecution Engine) 130, a VXE (Vector eXecution Engine) 140, an LMU (Local Memory Unit) 150, an ISU (Instruction Scheduling Unit) 160, a PCIe (Peripheral Component Interconnect express) Interface 170, and a P2P (Peer to Peer) Interface 180.

[0042] The SMA (110) may be a specialized DMA (Direct Memory Access). For example, the SMA (110) may connect all channels (e.g., 32) of the HBM (200) to the execution engine (e.g., the SEE (130)) to transfer half-precision floating point (FP16) data at maximum bandwidth. The SMA (110) may be designed with a deep FIFO (First In First Out) to transfer sequential memory requests based on preloaded memory (MEM) instructions. Hardware-based memory mapping may reduce latency by eliminating matrix modification or transposition operations. Therefore, the SMA (110) may stream data received in a maximum burst size to the execution engine with minimal latency. The SMA (110) may also efficiently perform matrix transposes using a strobe signal. Streaming data may contain parameters for vector matrix operations (e.g., weights, biases) and other vector-related operations (e.g., gamma / beta, embedding).

[0043] The OIU (120) can coordinate data streamed from the SMA (110) (e.g., a first operand) with input from on-chip memory (e.g., a second operand) before issuing it to the execution engine. Based on the execute (EXE) instruction, the OIU (120) can generate microcode that configures the execution engine and determines the target engine of the operand. In addition, the OIU (120) can be provided with a reusable buffer to eliminate read latency of static operands (e.g., input vectors) and an asymmetric buffer to hold vectorized data used as scalars (e.g., biases). Therefore, appropriate operands are almost always prefetched and ready to be issued to the execution engine immediately.

[0044] The SXE (130) is the main computing hardware of the LPU (100), and can be designed to maximize the incoming bandwidth to execute vector-matrix multiplication (V·M) such as attention, 1D convolution, and feedforward network. The SXE (130) can include a number of MAC (multiply-and-accumulation) trees (MAC Trees, 131) that can match the reception bandwidth and the computation bandwidth of the HBM (200). For example, if 1024 elements are received from the HBM (200) per cycle, the reception bandwidth and the computation bandwidth can be matched through 16 MAC trees (131) each having 64 input vectors. In addition, the MAC tree (131) with 64 input vectors can be composed of 64 multipliers and 63 adders.

[0045] A plurality of MAC trees (131) can perform matrix multiplication operations and can be connected by channel through the HBM (200), which is a high-bandwidth memory and the SMA (110), which is an external memory. Specifically, one of the plurality of MAC trees (131) is connected to the HBM (200) through one channel, thereby maximizing the transmission bandwidth between the LPU (100) and the HBM (200), and performing the matrix multiplication operations required for the ultra-large artificial intelligence model without a bottleneck. Therefore, the number of the plurality of MAC trees (131) and the number of memory channels of the HBM (200) can be configured to be the same.

[0046] The matrix multiplication operation results of multiple MAC trees (131) can be provided to the VXE (140). The VXE (140) can be implemented using a custom low-latency ALU (Arithmetic Logic Unit) and can execute vector operations such as token embedding, softmax, normalization, and residual operations. Since these vector operations occur relatively infrequently, the fan-in from the OIU (120) to this path can be adjusted to reduce hardware resources with negligible performance loss. The VXE (140) can receive the operation results of multiple MAC trees (131) and receive activation values ​​from the LMU (150) to perform subsequent operations. The VXE (140) can be configured to include various combinations of operators by including multiple multi-function operation datapaths.

[0047] The LMU (150) can transmit activation values ​​to multiple MAC trees (131) and VXEs (140). At this time, the LMU (150) can copy and transmit the activation values ​​in order to transmit the same activation values ​​to multiple MAC trees (131). In addition, the LMU (150) can store the result values ​​calculated by the multiple MAC trees (131) and VXEs (140). In other words, the LMU (150) can function as an internal buffer corresponding to the HBM (200) within the LPU (100). At this time, the LPU (100) can store activation values ​​or model parameters with a high reuse rate in the LMU (150) and store weights with a low reuse rate in the HBM (200) in the matrix multiplication operation. The LMU (150) can be implemented as a 4MB multi-bank register file with scalar vector separation for fast, high-bandwidth access to input, output, and intermediate data. Additionally, the LMU (150) may be multi-ported to support simultaneous reads and writes in the write-back phase of the OIU (120) and the execution engine.

[0048] The ISU (160) can control the entire execution flow of the LPU (100). The ISU (160) can utilize the PIC (Parallel Instruction Chaining) method, and by using instruction chaining, dependent instructions can be executed sequentially. The PIC separates instructions requiring independent hardware into groups of dependent instructions (e.g., memory (MEM) instructions, execution (EXE) instructions, network (NET) instructions), so that all instructions can be executed in parallel with the instruction chain of each group, thereby achieving low control overhead and latency savings. The ISU (160) can also update control registers (e.g., token and layer number) for engine execution. The internal scheduler supports out-of-order execution of the SXE (130) and VXE (140) to maximize hardware utilization, and a powerful scoreboard can be designed to handle data risks. For example, the ISU (160) can schedule multiple MAC trees (131) and VXEs (140) to perform operations simultaneously. In addition, the ISU (160) can improve operation throughput and delay time by pre-executing instructions without dependencies to maximize parallel operations, thereby minimizing the idle time of each operation unit and memory access unit.

[0049] The LPU (100) can be connected to a host computer via a PCIe interface (170), and can receive commands required for the operation of the LPU (100) and input values ​​and weights of a large-scale artificial intelligence model from the host computer, perform calculations, and then transmit the results to the host computer.

[0050] The LPU (100) can be scaled out into a cluster of multiple LPUs connected via a P2P interface (180). The expanded cluster structure can further enhance the acceleration of computation of ultra-large artificial intelligence models.

[0051] FIGS. 2 to 5 are diagrams illustrating examples of implementation models of LPUs according to embodiments of the present invention. The embodiment of FIG. 1 previously described an example of an implementation model using external memory of HBM (200). Instead of HBM (200), DDR (Double Data Rate) may be used as the external memory. In this case, since it is difficult to store a large model in a single device, it may be divided into multiple partitions and stored in external memories for multiple devices (multiple LPUs) for each partition. In this case, synchronization between multiple devices may be required for inference of the large model.

[0052] The embodiment of FIG. 2 illustrates a plurality of external memories (320) storing a plurality of partitions (310) of a large model, similar to the embodiment described above in FIG. 1, and a plurality of LPUs (330) connected in parallel to the plurality of external memories (320). One LPU can be implemented in one FPGA (Field Programmable Gate Array), and one partition can be connected in parallel to one FPGA. The transformer structure includes multi-head attention, layer normalization, feed forward, etc. in the decoder layer, and the multi-head attention and feed forward can be model-parallelized. In this case, when the multi-head attention is terminated, one embedding vector can be output as a result. Since one device only has a portion of the embedding vector, synchronization may be required because multiple devices need to share each embedding vector in order to proceed to the next operation. At this time, considering scalability, a single LPU may be implemented with multiple external memories (e.g., two or four). For example, the embodiment of FIG. 1 illustrates an example in which two HBMs (200), each storing one partition, are used.

[0053] The embodiment of Fig. 3 illustrates an example of a PIM (Processing-in-Memory) model, in which one LPU is implemented as a PIM chip, and both the partition and the LPU computation unit are integrated into a single chip. The embodiment of Fig. 3 illustrates a plurality of LPUs (410), a plurality of partitions (310), and a plurality of LPU computation units (420), each of which can be implemented as a PIM chip. In this case, each of the plurality of LPUs (410) can include one partition and one LPU computation unit.

[0054] The embodiment of Fig. 4 illustrates an example of a Processing-near-Memory (PNM) model. It may be difficult to include a configuration for processing all LPU operations within a single PIM chip. The embodiment of Fig. 4 illustrates a model in which a plurality of partitions (310) are stored within a plurality of memory chips (510), and an LPU operation unit (521) for LPU operations is included within a buffer chip (520) such as a PNM chip.

[0055] The embodiment of Fig. 5 illustrates an example of a model combining PIM and PNM. For example, multiple partitions (310) may be stored in multiple memory chips (610). In addition, PIM-style LPU operation units (611) may be implemented as accumulation units, such as MAC trees, in each of the multiple memory chips (610). In this case, the LPU operation units (621) for the remaining high-level operations of the LPU may be implemented in the PNM manner in the buffer chip (620).

[0056] FIG. 6 is a diagram for explaining weight matrix data mapping of a high-bandwidth memory for matrix multiplication operation of a latency processing unit according to an embodiment of the present invention.

[0057] Referring to FIG. 6, the LPU (100) according to the present embodiment can store weight matrix data mapped to a high-bandwidth memory (610) such as HBM (200) so that the weight data can be retrieved without accessing another memory channel when performing matrix multiplication operation on each MAC tree, since the number of MAC trees constituting multiple MAC trees (131) and the number of memory channels of SMA (110) are the same.

[0058] Specifically, weight matrix data may be stored in the high-bandwidth memory (610) so that it is mapped to each channel (620-n) in the column direction (D1) of the weight matrix in the number of multiple MAC trees (131). Since the column direction in the weight matrix allows matrix multiplication operations to be performed in parallel, the multiple MAC trees (131) can read column direction data from their respective allocated memory channels (620-n) and perform matrix multiplication operations.

[0059] Next, multiple MAC trees (131) can map the weight matrix data so that they can accumulate in the row direction (D2) of the weight matrix to complete the final operation result. The number of row data mapped at once can be determined by the bandwidth of the high-bandwidth memory (610), and this can be determined by the size of the tiles that multiple MAC trees (131) can process at once.

[0060] FIG. 7 is a diagram for explaining a high-bandwidth memory interface included in a latency processing unit according to one embodiment of the present invention.

[0061] Referring to FIG. 7, the SMA (110) can connect the LMU (150), multiple MAC trees (131), and high-bandwidth memory (610). The SMA (110) is not connected to other computational units of the LPU (100), and thus, the high-bandwidth memory interface can be minimized in terms of hardware resources.

[0062] Multiple MAC trees (131) and memory channels (620-n) can be connected in a one-to-one correspondence, respectively. That is, multiple MAC trees (131) do not need to access channels other than the directly assigned channels, and thus can perform matrix multiplication operations without using complex interfaces such as cross-bars, which consume a lot of resources and have a high delay time.

[0063] SMA (110) can configure only a read interface for loading weight matrix data stored in high-bandwidth memory (610) of multiple MAC trees (131). In other words, as described later, since the result of the operation is stored in the high-bandwidth memory (610) through LMU (150), a write interface for the high-bandwidth memory (610) of multiple MAC trees (131) is not configured, and hardware resources can be reduced accordingly.

[0064] In contrast, the SMA (110) can only form a write interface between the LMU (150) and the high-bandwidth memory (610). The computational results stored in the LMU (150) as an internal buffer through the SMA (110) can be transmitted to be written to the high-bandwidth memory (610), and the memory channel of the writing target can be selected using the demultiplexer (710).

[0065] FIG. 8 is a diagram illustrating a reconfigurable multi-function operation unit included in a latency processing unit according to one embodiment of the present invention.

[0066] Referring to FIG. 8, the VXE (140) may include a plurality of multi-function operation data paths (810, 820), and the plurality of multi-function operation data paths (810, 820) may be connected to an operator / result value chain network (830) to form various combinations of operators.

[0067] As illustrated in FIG. 8, a plurality of multi-functional operation datapaths (810, 820) may include various operation units required for, for example, a lookup table-based nonlinear activation function, a masking operation, etc. However, the configuration of the operation units of the reconfigurable multi-functional operation datapaths (810, 820) illustrated in FIG. 8 is exemplary, and it goes without saying that additional operation units required for large-scale model operation may be included in the multi-functional operation datapaths (810, 820). The result calculated by the VXE (140) may be transmitted to the LMU (150).

[0068] FIG. 9 is a diagram for explaining the configuration of an address-based non-sequential multi-unit scheduler included in a latency processing unit according to one embodiment of the present invention.

[0069] Referring to FIG. 9, the ISU (160) as an address-based out-of-order multi-unit scheduler included in the LPU (100) according to the present embodiment may include an address-based instruction dependency determination and scheduling controller (910), a plurality of instruction issue controllers (921, 922, 923, 924), a multi-bank buffer address status table (950), an instruction buffer (960) and result address status update logic (970), and a multi-unit instruction dispatcher (980).

[0070] The ISU (160) can operate each operation unit and data movement unit simultaneously through the address-based instruction dependency judgment and scheduling controller (910) and multiple instruction issue controllers (921, 922, 923, 924). At this time, the ISU (160) can change the status of the operator address and result address of the multi-bank buffer address status table (950) for the instruction performed in each operation unit to 1.

[0071] The multi-bank buffer address status table (950) can change the status of the result address of a completed instruction to 0 through the result address status update logic (970).

[0072] The address-based instruction dependency determination and scheduling controller (910) can determine the dependency between the instruction to be executed and the instruction being executed, as well as the dependency between the instructions to be executed, by referencing the states of the addresses through the multi-bank buffer address state table (970). This enables preprocessing of instructions without dependencies, thereby minimizing the idle time of each operation unit and data movement unit.

[0073] The address-based instruction dependency determination and scheduling controller (910) included in the ISU (160) can load and process instructions from the instruction buffer (960). At this time, the address-based instruction dependency determination and scheduling controller (910) performs a loop instruction, and decodes other instructions to distinguish the instructions and transmit them to the device-to-device instruction issue controller (921), the direct memory access instruction issue controller (922), the MAC tree instruction issue controller (923), and the reconfigurable multi-function operation unit instruction issue controller (924) through the multi-unit instruction dispatcher (980).

[0074] The ISU (160) receives and stores commands of the LPU (100) from the host computer through the PCIe interface (170), and can store the current status of the LPU (100) in a status register. The host computer can check the status register through the PCIe interface (170).

[0075] FIG. 10 is a diagram illustrating an example of the internal configuration of a MAC tree according to an embodiment of the present invention. The MAC tree (1000) according to the present embodiment may correspond to the MAC tree (131) described above. The MAC tree (1000) may include a floating-point multiplier (1010), a fixed-point adder tree (1020), and a fixed-point accumulator (1030).

[0076] The floating-point multiplier (1010) can be implemented with a structure in which multiple floating-point multipliers are connected in parallel to seamlessly perform MAC operations with weight data obtained from an external memory such as a high-bandwidth memory (e.g., a high-bandwidth memory (610)) or an internal buffer (e.g., an LMU (150)).

[0077] In addition, the fixed-point adder tree (1020) and the fixed-point accumulator (1030) may be implemented in a form connected to the floating-point multiplier (1010) to accumulate the multiplication results of the floating-point multiplier (1010). At this time, the fixed-point adder tree (1020) may process the addition of the multiplication results of the floating-point multiplier (1010), and the fixed-point accumulator (1030) may accumulate the output of the fixed-point adder tree.

[0078] The MAC tree (1000) may further include an FP2FXP converter (FP-to-FXP converter, 1040) for converting the output of the floating-point multiplier (1010) from floating-point (FP) to fixed-point (FXP).

[0079] An arithmetic unit based on such a MAC tree (1000) can be designed to efficiently use all of the provided memory bandwidth by setting the number of multipliers or the number of MAC trees (1000) according to the provided memory bandwidth. At this time, the MAC tree (1000) may further include an FXP2FP converter (FXP-to-FP Converter, 1050) for converting the output of the fixed-point accumulator (1030) from fixed-point to floating-point.

[0080] To reduce the logic complexity and the resulting large area burden of floating-point operators while maintaining accuracy, multipliers were used as floating-point operators and adders were used as fixed-point operators.

[0081] FIG. 11 is a diagram illustrating the performance of an arithmetic unit according to an embodiment of the present invention. FIG. 11 shows the resource differences between when both multipliers and an adder tree in a MAC tree-based arithmetic unit are implemented as floating-point arithmetic units and when the multipliers are implemented as floating-point arithmetic units and the adder tree is implemented as a fixed-point arithmetic unit as in an embodiment of the present invention. In particular, when implemented in an FPGA, the fixed-point adder can replace the look-up table (LUT) logic resource with a DSP (Digital Signal Processing) so as not to use the LUT logic resource. Since the DSP is a circuit included in the FPGA, the area overhead can be reduced and the arithmetic unit can be implemented at a high frequency through the provided arithmetic unit structure. As an example, FIG. 11 shows the resource differences between an LUT, a flip-flop, and a DSP when implemented in an FPGA. Resources have been reduced in all three aspects: LUT, flip-flop, and DSP, and in particular, the LUT, which is responsible for most hardware resources, has seen a resource reduction of more than 50%.

[0082] Meanwhile, in the case of operations that use model weights as operands in language model operations (for example, operations that generate query, key, and value values ​​in attention operations), the operations can be performed with data that has relatively low precision. On the other hand, in the case of operations that use activation values ​​as operands in language model operations (for example, score operations that use query, key, and value as operands), the operations must be performed with data that has high precision. When using an operator that supports only one precision of high precision, when operations can be performed with relatively low precision data, efficient memory bandwidth use is not possible, and operations are performed with unnecessarily high precision.

[0083] Therefore, in order to efficiently support both high precision and low precision, a mixed-precision floating-point multiplier supporting two modes can be provided, and the two modes can be switched in real time. The first mode is a high-precision mode that can perform operations with high precision, and the second mode is a high-performance mode that can perform operations with relatively low precision but with high computational bandwidth. Such a mixed-precision floating-point multiplier can selectively perform the two modes with only one multiplier through preprocessing of the operands and postprocessing of the result value, thereby saving the logic overhead of the mixed-precision floating-point operator.

[0084] FIG. 12 is a diagram showing an example of the structure of a mixed-precision floating-point multiplier according to an embodiment of the present invention, and FIG. 13 is a diagram showing an example of two modes of a mixed-precision floating-point multiplier according to an embodiment of the present invention.

[0085] Referring to FIG. 12, a hardware shared mixed-precision floating-point operator (1200) may include a mixed-precision floating-point multiplier (1210), and the mixed-precision floating-point multiplier (1210) may include a mixed-precision fixed-point exponent adder (1211) and a mixed-precision fixed-point mantissa multiplier (1212).

[0086] Referring to FIGS. 12 and 13, the mixed-precision floating-point multiplier (1210) can support faster operations through high-performance modes such as 8-bit mode while also supporting high-precision modes such as 16-bit mode.

[0087] When the mixed-precision floating-point multiplier (1210) operates in high-precision mode, it can receive two operands (operand 1 and operand 2) with the same bit precision from the LMU (150) and high-bandwidth memory (610) and perform a multiplication operation with the maximum bit precision supported by the mixed-precision floating-point multiplier (1210).

[0088] In addition, when the mixed-precision floating-point multiplier (1210) operates in high-performance mode, it can perform two or more multiplications, and for example, as illustrated in FIG. 13, it can perform simultaneous multiplication of two operands (2-(a), 2-(b)) on one operand (operand 1) to obtain two multiplication results (result 1 and result 2). To this end, the hardware-shared mixed-precision floating-point multiplier (1200) can perform the operation by quantizing the parameters of the model with less precision.

[0089] FIG. 14 is a diagram illustrating an example of a mixed-precision fixed-point mantissa multiplier according to an embodiment of the present invention, and FIG. 15 is a diagram illustrating an example of a mixed-precision fixed-point exponent adder according to an embodiment of the present invention.

[0090] The mixed-precision fixed-point mantissa multiplier (1212) operates by differentiating the preprocessing of operators and the postprocessing of result values ​​according to the above-described mode, thereby enabling the mixed-precision floating-point multiplier (1210) to selectively perform high-precision and high-performance floating-point multiplication operations with only one fixed-point multiplier (the mixed-precision fixed-point mantissa multiplier (1212)), thereby saving hardware resources.

[0091] At this time, the mixed precision fixed-point mantissa multiplier (1212) implements two modes, a high-precision mode and a high-performance mode, through post-processing after performing two multiplications.

[0092] First, in high-precision mode, the first of the two multiplications is a multiplication between the mantissa of operand 1 and the lower bits of the mantissa of operand 2, and the second is a multiplication between the mantissa of operand 1 and the upper bits of the mantissa of operand 2. Afterwards, the mixed-precision fixed-point mantissa multiplier (1212) performs a shift on the first multiplication result (multiplication result 1) and performs sign extension on the second multiplication result (multiplication result 2) and adds them together to derive the final mantissa multiplication result.

[0093] Next, in high-performance mode, the first of the two multiplications is the multiplication between the mantissa of operand 1 and the mantissa of operand 2-(a), and the second is the multiplication between the mantissa of operand 1 and the mantissa of operand 2-(b). Thereafter, the mixed-precision fixed-point mantissa multiplier (1212) can derive two mantissa multiplication results by concatenating the respective multiplication results (multiplication result 1 and multiplication result 2) without post-processing.

[0094] In addition, not only the mixed-precision fixed-point mantissa multiplier (1212) but also the mixed-precision fixed-point exponent adder (1211) can selectively perform high-precision and high-performance floating-point multiplication operations through operator preprocessing and postprocessing, thereby saving hardware resources. The operation result of the mixed-precision floating-point multiplier (1210) can perform a partial sum operation through the fixed-point adder tree (1020) and the fixed-point accumulator (1030) of FIG. 10. The matrix multiplication operation result output from the MAC tree (131) is directly provided to the VXE (140), thereby minimizing the time required for memory access and the resulting delay time.

[0095] At this time, the mixed precision fixed-point exponential adder (1211) implements two modes through preprocessing of each operator and postprocessing after two additions.

[0096] First, here is an explanation in high-precision mode. The first operator of the first addition adds 0 to the most significant bit of the exponent of operand 1, and the second operator adds 0 to the least significant bit of the exponent of operand 2. The first operator of the second addition adds 0 to the least significant bit of the exponent of operand 1, and the second operator adds 0 to the least significant bit of the exponent of operand 2. The mixed-precision fixed-point exponent adder (1211) performs two additions and then adds the results of each operation to derive the final exponent addition result.

[0097] Here is an explanation in high-performance mode. The first operator of the first addition is the exponent of operand 1, and the second operator is the exponent of operand 2-(b). The first operator of the second addition is the exponent of operand 1, and the second operator is the sign extension of the exponent of operand 2-(a). The mixed-precision fixed-point exponent adder (1211) can derive two exponent addition results by performing two additions, truncating and concatenating the results of each operation.

[0098] Fig. 16 is a flowchart illustrating an example of an operating method of a MAC tree-based operator according to an embodiment of the present invention. The MAC tree-based operator may correspond to the MAC tree (1000) described above with reference to Fig. 10. This MAC tree-based operator may correspond to one of a plurality of MAC tree-based operators included in a hardware accelerator (e.g., LPU (100)) for accelerating an artificial intelligence model.

[0099] At this time, at least one of the number of MAC tree-based operators included in the hardware accelerator and the number of multiple floating-point multipliers included in the MAC tree-based operator may be determined based on the memory bandwidth provided to the hardware accelerator. In addition, the multiple MAC tree-based operators may perform a matrix multiplication operation for at least one partition among the multiple partitions implementing the artificial intelligence model.

[0100] This allows for efficient use of memory bandwidth to process large amounts of data for generative large-scale artificial intelligence models.

[0101] In step (1610), the MAC tree-based operator may process a multiplication operation on data transferred from an external memory using a plurality of floating-point multipliers connected in parallel. The external memory may include a high-bandwidth memory in which at least one partition is stored and a local memory unit (e.g., LMU (150)) included in a hardware accelerator. In addition, each of the plurality of floating-point multipliers may include a mixed-precision fixed-point exponent adder for addition of exponents (e.g., a mixed-precision fixed-point exponent adder (1211)) and a mixed-precision fixed-point mantissa multiplier for multiplication of mantissas (e.g., a mixed-precision fixed-point mantissa multiplier (1212)).

[0102] As described above, each of the plurality of floating-point multipliers can operate according to a high-precision mode and a high-performance mode. For example, each of the plurality of floating-point multipliers can process a multiplication between a first operand and a second operand having the same bit precision when the high-precision mode is selected, and calculate a first result value. In addition, each of the plurality of floating-point multipliers can process a first multiplication between a third operand having the first bit precision and a 4-1 operand having the second bit precision and a second multiplication between the third operand and a 4-2 operand having the second bit precision simultaneously when the high-performance mode is selected, and calculate a second result value of the first multiplication and a third result value of the second multiplication simultaneously. For example, the first bit precision can include a 16-bit precision, and the second bit precision can include an 8-bit precision.

[0103] In step (1620), the MAC tree-based operator may convert the results of multiplication operations of each of the plurality of floating-point multipliers from floating-point to fixed-point using a plurality of first converters connected to the plurality of floating-point multipliers. Here, the plurality of first converters may correspond to the FP2FXP converter (1040) described above with reference to FIG. 10.

[0104] In step (1630), the MAC tree-based operator can process the addition of the converted results of multiple floating-point multipliers using a fixed-point adder tree. Here, the fixed-point adder tree can correspond to the fixed-point adder tree (1020) described above with reference to FIG. 10.

[0105] In step (1640), the MAC tree-based operator can accumulate the output of the fixed-point adder tree using a fixed-point accumulator. Here, the fixed-point accumulator can correspond to the fixed-point accumulator (1030) described with reference to FIG. 10.

[0106] At step (1650), the MAC tree-based operator may convert the output of the fixed-point accumulator from fixed-point to floating-point using a second converter. Here, the second converter may correspond to the FXP2FP converter (1050) described above with reference to FIG. 10.

[0107] Thus, according to embodiments of the present invention, a hardware accelerator can be provided that maximizes the utilization of memory bandwidth for accelerating generative large-scale language models that are difficult to parallelize and have large amounts of data. Furthermore, a computing unit can be provided that reduces the high hardware resource overhead of floating-point-based MAC tree structures. Furthermore, a computing unit can be provided that effectively supports two-precision operations while minimizing hardware resource overhead.

[0108] Although the embodiments of the present invention have been described with reference to the attached drawings, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without altering the technical concept or essential features thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

Claims

1. In MAC (multiply-and-accumulation) tree-based operators, Multiple floating-point multipliers connected in parallel to perform multiplication operations on data transmitted from external memory; A plurality of first converters for converting the output of each of the plurality of floating-point multipliers from floating-point to fixed-point; A fixed-point adder tree connected to the plurality of first converters and processing addition of multiplication results of the plurality of floating-point multipliers; A fixed-point accumulator that accumulates the output of the fixed-point adder tree; and A second converter that converts the output of the fixed-point accumulator from fixed-point to floating-point. Including, The above MAC tree-based operator corresponds to one of a plurality of MAC tree-based operators included in a hardware accelerator for acceleration of an artificial intelligence model, At least one of the number of MAC tree-based operators included in the hardware accelerator and the number of the plurality of floating-point multipliers included in the MAC tree-based operator is determined according to the memory bandwidth provided for the hardware accelerator. A MAC tree-based operator featuring .

2. In paragraph 1, A MAC tree-based operator characterized in that the above plurality of MAC tree-based operators perform a matrix multiplication operation for at least one partition among the plurality of partitions implementing the artificial intelligence model.

3. In paragraph 2, A MAC tree-based computing device characterized in that the external memory includes a high-bandwidth memory in which at least one partition is stored and a local memory unit included in the hardware accelerator.

4. In paragraph 1, Each of the above plurality of floating-point multipliers, A mixed-precision fixed-point exponential adder for addition of exponents; and Mixed-precision fixed-point mantissa multiplier for mantissa multiplication A MAC tree-based operator characterized by including:

5. In paragraph 1, Each of the above plurality of floating-point multipliers, When high-precision mode is selected, the first result value is calculated by processing the multiplication between the first operand and the second operand with the same bit precision. A MAC tree-based operator featuring .

6. In MAC (multiply-and-accumulation) tree-based operators, Multiple floating-point multipliers connected in parallel to perform multiplication operations on data transmitted from external memory; A plurality of first converters for converting the output of each of the plurality of floating-point multipliers from floating-point to fixed-point; A fixed-point adder tree connected to the plurality of first converters and processing addition of multiplication results of the plurality of floating-point multipliers; A fixed-point accumulator that accumulates the output of the fixed-point adder tree; and A second converter that converts the output of the fixed-point accumulator from fixed-point to floating-point. Including, Each of the above plurality of floating-point multipliers, When the high-performance mode is selected, a first multiplication between a first operand of first bit precision and a second-first operand of second bit precision and a second multiplication between the first operand and a second-second operand of second bit precision are simultaneously processed, thereby calculating a first result value of the first multiplication and a second result value of the second multiplication simultaneously. A MAC tree-based operator featuring .

7. In paragraph 6, The above first bit precision includes 16 bit precision, The above second bit precision includes 8 bit precision. A MAC tree-based operator featuring .

8. In the operation method of a MAC (multiply-and-accumulation) tree-based operator, The MAC tree-based operator comprises a plurality of floating-point multipliers connected in parallel, a plurality of first converters connected to the plurality of floating-point multipliers, a fixed-point adder tree, a fixed-point accumulator, and a second converter, A step of processing a multiplication operation on data transmitted from an external memory using the plurality of floating-point multipliers; A step of converting the result of a multiplication operation of each of the plurality of floating-point multipliers from floating-point to fixed-point using the plurality of first converters; A step of processing the addition of the converted results of the plurality of floating-point multipliers using the fixed-point adder tree; A step of accumulating the output of the fixed-point adder tree using the fixed-point accumulator; and A step of converting the output of the fixed-point accumulator from fixed-point to floating-point using the second converter. Including, The above MAC tree-based operator corresponds to one of a plurality of MAC tree-based operators included in a hardware accelerator for acceleration of an artificial intelligence model, At least one of the number of MAC tree-based operators included in the hardware accelerator and the number of the plurality of floating-point multipliers included in the MAC tree-based operator is determined according to the memory bandwidth provided for the hardware accelerator. A method of operating a MAC tree-based operator characterized by .

Images