Mixed precision MAC tree structure for maximizing memory bandwidth usage to accelerate computation of large-scale generative language models

Abstract

This invention provides a multiply-and-accumulation (MAC) tree-based structure for maximizing memory bandwidth usage to accelerate computations in generative large-scale language models. One embodiment of the MAC tree-based arithmetic unit includes: a plurality of floating-point multipliers connected in parallel to process multiplication operations on data transmitted from external memory; a plurality of first converters for converting the output of each of the plurality of floating-point multipliers from floating-point to fixed-point; a fixed-point adder tree connected to the plurality of first converters for processing the summation of the multiplication results of the plurality of floating-point multipliers; a fixed-point accumulator for storing the output of the fixed-point adder tree; and a second converter for converting the output of the fixed-point accumulator from fixed-point to floating-point.

Description

Technical Field

[0001] Embodiments of the present invention relate to a mixed-precision MAC tree structure for maximizing the use of memory bandwidth for accelerating the operations of a generative large language model.

Background Art

[0002] In recent years, due to the rapid development of generative large language models, the scale of the models has been gradually expanding in order to obtain high accuracy, and they have come to have model parameters of at least millions, often reaching billions. As a result, a large amount of data must be called from memory at once to smoothly execute operations.

[0003] Also, in order to support the operations of a generative large language model without losing accuracy, a floating-point arithmetic unit is required. However, such a floating-point arithmetic unit has problems of high logic complexity and occupying a large area.

Summary of the Invention

Problems to be Solved by the Invention

[0004] It is possible to provide an arithmetic unit of a hardware accelerator that maximizes the use of memory bandwidth provided for accelerating a generative large language model that has a large amount of data while being difficult to parallelize.

[0005] The technical problems of the present invention are not limited to those described above, and other technical problems not described will be clearly understood by those skilled in the art from the following description.

Means for Solving the Problems

[0006] A MAC (Multiply-and-Accumulation) tree-based arithmetic unit is provided, comprising: a plurality of floating-point multipliers connected in parallel and processing multiplication operations on data transmitted from external memory; a plurality of first converters for converting the output of each of the plurality of floating-point multipliers from floating-point to fixed-point; a fixed-point adder tree connected to the plurality of first converters and processing the sum of the multiplication results of the plurality of floating-point multipliers; a fixed-point accumulator for storing the output of the fixed-point adder tree; and a second converter for converting the output of the fixed-point accumulator from fixed-point to floating-point.

[0007] According to one side, the MAC tree-based computing unit may be characterized by corresponding to one of several MAC tree-based computing units included in a hardware accelerator for accelerating artificial intelligence models.

[0008] In other aspects, the number of MAC tree-based arithmetic units included in the hardware accelerator and the number of floating-point multipliers included in the MAC tree-based arithmetic units may be determined depending on the memory bandwidth provided to the hardware accelerator.

[0009] In other respects, the plurality of MAC tree-based arithmetic units may be characterized by performing matrix multiplication for at least one of the plurality of partitions that realize the artificial intelligence model.

[0010] In other respects, the external memory may be characterized by including a high-bandwidth memory on which the at least one partition is stored.

[0011] In other respects, each of the plurality of floating-point multipliers may be characterized by including a mixed-precision fixed-point exponent adder for adding the exponents and a mixed-precision fixed-point mantissa multiplier for multiplying the mantissa.

[0012] In other respects, each of the multiple floating-point multipliers may be characterized by processing the multiplication between a first operand and a second operand having the same bit precision to calculate a first result value, by selecting a high-precision mode.

[0013] In other respects, each of the plurality of floating-point multipliers may be characterized by simultaneously processing a first multiplication between the third operand of the first bit precision and the 4-1 operand of the second bit precision, and a second multiplication between the third operand and the 4-2 operand of the second bit precision, by selecting a high-performance mode, thereby simultaneously calculating the second result of the first multiplication and the third result of the second multiplication.

[0014] In other respects, the first bit precision may be characterized by including a 16-bit precision, and the second bit precision may be characterized by including an 8-bit precision.

[0015] A method for operating a MAC (Multiply-and-Accumulation) tree-based arithmetic unit, wherein the MAC tree-based arithmetic unit includes a plurality of floating-point multipliers connected in parallel, a plurality of first converters connected to the plurality of floating-point multipliers, a fixed-point adder tree, a fixed-point accumulator, and a second converter, and the method for operating the MAC tree-based arithmetic unit includes the steps of: processing a multiplication operation on data transmitted from an external memory using the plurality of floating-point multipliers; converting the result of the multiplication operation of each of the plurality of floating-point multipliers from floating-point to fixed-point using the plurality of first converters; processing the addition of the converted results of the plurality of floating-point multipliers using the fixed-point adder tree; accumulating the output of the fixed-point adder tree using the fixed-point accumulator; and converting the output of the fixed-point accumulator from fixed-point to floating-point using the second converter.

[0016] Specific details regarding other embodiments will be included in the detailed description and drawings. [Effects of the Invention]

[0017] This provides a hardware accelerator arithmetic unit that maximizes the use of memory bandwidth provided for accelerating generative large-scale language models that have a large amount of data but are difficult to parallelize.

[0018] This provides an arithmetic unit that can reduce the high hardware source overhead of a floating-point based MAC tree structure.

[0019] It is possible to provide a arithmetic unit that can effectively support two precision calculations while minimizing hardware resource overhead.

[0020] The effects of the present invention are not limited to those described above, and any other effects not described will be clearly understood by those skilled in the art from the claims. [Brief explanation of the drawing]

[0021] [Figure 1] This figure shows an example of the structure of a latency processing unit according to an embodiment of the present invention. [Figure 2] This figure shows an example of an LPU implementation model according to an embodiment of the present invention. [Figure 3] This figure shows an example of an LPU implementation model according to an embodiment of the present invention. [Figure 4] This figure shows an example of an LPU implementation model according to an embodiment of the present invention. [Figure 5] This figure shows an example of an LPU implementation model according to an embodiment of the present invention. [Figure 6] This figure illustrates the weight matrix data mapping of a high-bandwidth memory for matrix multiplication in a latency processing unit according to one embodiment of the present invention. [Figure 7] This figure illustrates a high-bandwidth memory interface included in a latency processing unit according to an embodiment of the present invention. [Figure 8] A diagram for explaining a reconfigurable multifunctional arithmetic unit included in a latency processing unit according to an embodiment of the present invention. [Figure 9] A diagram for explaining the configuration of an address-based unordered multi-unit scheduler included in a latency processing unit according to an embodiment of the present invention. [Figure 10] A diagram showing an example of the internal configuration of a MAC tree in an embodiment of the present invention. [Figure 11] A diagram for explaining the performance of an arithmetic unit according to an embodiment of the present invention. [Figure 12] A diagram for explaining a mixed-precision floating-point multiplier in an embodiment of the present invention. [Figure 12] A diagram showing an example of the structure of a mixed-precision floating-point multiplier in an embodiment of the present invention. [Figure 13] A diagram showing examples of two modes of a mixed-precision floating-point multiplier in an embodiment of the present invention. [Figure 14] A diagram showing an example of a mixed-precision fixed-point adder according to an embodiment of the present invention. [Figure 15] A diagram showing an example of a mixed-precision fixed-point exponent adder according to an embodiment of the present invention. [Figure 16] A flowchart showing an example of an operation method of a MAC tree-based arithmetic unit according to an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0022] The advantages and features of the present invention, as well as methods for achieving them, will become apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be realized in a variety of different forms. The embodiments are provided only to complete the disclosure of the present invention and to fully inform a person ordinary skill in the art to which the invention pertains of the invention of the scope of the invention, and the present invention is defined solely by the claims. Throughout the specification, the same reference numerals indicate the same components.

[0023] When a component is described as “connected to” or “coupled to” another component, this includes both direct connection or coupling, and cases where the other component is an intermediary. Conversely, when a component is described as “directly connected to” or “directly coupled to” another component, this means that no other component is an intermediary. “And / or” includes each of the listed items, and all combinations of one or more of them.

[0024] The terms used herein are for illustrative purposes only and are not intended to limit the invention. In this specification, the singular form includes the plural form unless otherwise specified in the context. As used in this specification, “comprises” and / or “comprising” means that the components, steps, operations, and / or elements described herein do not exclude the presence or addition of one or more other components, steps, operations, and / or elements.

[0025] While terms such as "first," "second," etc., are used to describe various components, these components should not be limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it goes without saying that the first component described below may also become the second component within the technical concept of the present invention.

[0026] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall be used in a way that can be commonly understood by a person of ordinary skill in the art to which this invention pertains. Furthermore, commonly used, dictionary-defined terms shall not be interpreted ideally or excessively unless otherwise specified.

[0027] Figure 1 shows an example of the structure of a latency processing unit according to an embodiment of the present invention.

[0028] Referring to Figure 1, the LPU (Latency Processing Unit) 100 according to an embodiment of the present invention may include an SMA (Streamlined Memory Access) 110, an OIU (Operand Issue Unit) 120, an SXE (Streamlined eXecution Engine) 130, a VXE (Vector eXecution Engine) 140, an LMU (Local Memory Unit) 150, an ISU (Instruction Scheduling Unit) 160, a PCIe (Peripheral Component Interconnect express) Interface 170, and a P2P (Peer to Peer) Interface 180.

[0029] The SMA110 may be a special DMA (Direct Memory Access). For example, the SMA110 may connect all channels of the HBM200 (for example, 32) to an execution engine (for example, SEE130) and transmit FP16 (half precision floating point) data at maximum bandwidth. The SMA110 may be designed with a deep FIFO (First In First Out) to transmit contiguous memory requests based on preloaded memory (MEM) instruction words. Hardware-aware memory mapping may reduce latency by eliminating matrix modification or transposition operations. Thus, the SMA110 can stream data received at maximum burst size to the execution engine with minimal latency. Furthermore, the SMA110 may efficiently perform matrix transposition using strobe signals. The streaming data may include parameters for vector matrix execution (e.g., weights, bias) and other vector-related execution (e.g., gamma / beta, embedding).

[0030] The OIU120 may reconcile data streamed by the SMA110 (e.g., the first operand) with input in on-chip memory (e.g., the second operand) before issuing it to the execution engine. Based on the EXE instruction, the OIU120 may generate microcode to configure the execution engine and determine the target engine for the operands. The OIU120 may also have a reuse buffer to eliminate read wait times for static operands (e.g., input vectors) and an asymmetric buffer to maintain vectorized data used as scalars (e.g., biases). Thus, appropriate operands are almost always prefetched and ready to be immediately issued to the execution engine.

[0031] SXE130 is the primary computing hardware of LPU100 and may be designed to maximize the use of the bandwidth returning to perform vector-matrix multiplication (V·M) such as attention, 1D convolution, and feedforward networks. SXE130 may include a number of MAC (multiply-and-accumulation) trees 131 capable of matching the receiving bandwidth of HBM200 with the processing bandwidth. For example, if 1024 elements are received from HBM200 per cycle, 16 MAC trees 131 with 64 input vectors each may match the receiving bandwidth with the processing bandwidth. Furthermore, a MAC tree 131 with 64 input vectors may consist of 64 multipliers and 63 adders.

[0032] Multiple MAC trees 131 may perform matrix multiplication and may be connected channel by channel through the HBM200 and SMA110, which are external memory but high-bandwidth memory. Specifically, one of the multiple MAC trees 131 connects to the HBM200 through one channel to maximize the transmission bandwidth between the LPU 100 and the HBM200, thereby performing the matrix multiplication required for the ultra-large artificial intelligence model without bottlenecking. Therefore, the number of multiple MAC trees 131 and the number of memory channels in the HBM200 may be configured to be the same.

[0033] The matrix multiplication results of multiple MAC trees 131 may be provided to VXE140. VXE140 may be implemented using a user-specified low-latency ALU (Arithmetic Logic Unit) and may perform vector operations such as token embedding, softmax, normalization, and residual calculations. Since such vector operations occur relatively infrequently, hardware resources can be reduced with negligible performance loss by adjusting the fan-in to this path in OIU120. While VXE140 receives the calculation results of multiple MAC trees 131, it may also receive an activation value from LMU150 before performing subsequent calculations. VXE140 may be configured to include various combinations of arithmetic units by including multiple multifunction arithmetic data paths.

[0034] The LMU150 may transmit activation values ​​to multiple MAC trees 131 and VXE140. In this case, the LMU150 may copy and transmit the activation values ​​to multiple MAC trees 131 to transmit the same activation values. The LMU150 may also store the results of calculations performed by the multiple MAC trees 131 and VXE140. In other words, the LMU150 may function within the LPU100 as an internal buffer corresponding to the HBM200. In this case, the LPU100 may store activation values ​​or model parameters with high reuse rates in matrix multiplication in the LMU150, and weights with low reuse rates in the HBM200. The LMU150 may be implemented as a 4MB multibank register file with scalar vector separation for high-speed, high-bandwidth access to input, output, and intermediate data. The LMU150 may also be a multiport that simultaneously supports reading and writing during the write and storage phase of the OIU120 and execution engine.

[0035] The ISU160 may control the entire execution flow of the LPU100. The ISU160 may utilize the PIC (Parallel Instruction Chaining) method, which allows dependent instructions to be executed sequentially. PIC separates instructions requiring independent hardware into groups of dependent instructions (e.g., memory (MEM) instructions, execution (EXE) instructions, network (NET) instructions), so that all instructions are executed in parallel with each group's instruction chain, achieving low control overhead and latency savings. The ISU160 may also update control registers (e.g., tokens and hierarchy numbers) for engine execution. An internal scheduler may support the non-sequential execution of SXE130 and VXE140 to maximize hardware utilization, and a robust scoreboard may be designed to handle data hazards. For example, the ISU160 may schedule multiple MAC trees 131 and VXE140 to perform calculations simultaneously. Furthermore, the ISU160 may increase the amount of computation and delay time by pre-executing dependency-free instructions in order to maximize parallel processing, thereby minimizing the idle time of each arithmetic unit and memory access unit.

[0036] The LPU100 may be connected to a host computer via a PCIe interface 170, and may receive the necessary instructions, input values ​​and weights for the operation of the LPU100, a super-large-scale artificial intelligence model, from the host computer, perform calculations, and then transmit the results back to the host computer.

[0037] The LPU100 may be scaled out as a cluster of multiple LPUs connected via the P2P interface 180. The extended cluster structure can further accelerate the computation of ultra-large-scale artificial intelligence models.

[0038] Figures 2-5 show examples of LPU implementation models according to embodiments of the present invention. In the embodiment shown in Figure 1, an example of an implementation model using HBM200 external memory was described. Instead of HBM200, DDR (Double Data Rate) may be used as the external memory. In this case, since it is difficult to store a large-scale model on a single device, it may be divided into multiple partitions and stored in separate partitions in external memory for multiple devices (multiple LPUs). In this case, synchronization between multiple devices may be required for inference of the large-scale model.

[0039] In the embodiment shown in Figure 2, similar to the embodiment described in Figure 1, multiple external memories 320 are shown to store multiple partitions 310 of a large-scale model, and multiple LPUs 330 are shown connected in parallel to the multiple external memories 320. One LPU may be implemented on one FPGA (Field Programmable Gate Array), and one partition may be connected in parallel to one FPGA. The transformer structure includes multi-head attention, layer normalization, feedforward, etc., within the decoder layer, but multi-head attention and feedforward may be model-parallelized. In this case, when multi-head attention is completed, one embedding vector may be output as a result. Since one device only has the embedding vector portion, multiple devices need to share each embedding vector in order to move to the next operation, and this requires synchronization. In this case, considering scalability, one LPU may be implemented in a form that has multiple external memories (for example, two, four, etc.). As an example, the embodiment in Figure 1 shows an example in which two HBM200s, each storing one partition, are used.

[0040] In the embodiment shown in Figure 3, as an example of a PIM (Processing-in-Memory) model, one LPU is implemented on a PIM chip, and both the partition and the LPU arithmetic unit are integrated on a single chip. In the embodiment of Figure 3, multiple LPUs 410, multiple partitions 310, and multiple LPU arithmetic units 420 that can be implemented on a PIM chip are shown. In this case, each of the multiple LPUs 410 may include one partition and one LPU arithmetic unit.

[0041] The embodiment in Figure 4 shows an example of a PNM (Processing-near-Memory) model. It can be difficult to include the configuration for processing all LPU calculations within a single PIM chip. The embodiment in Figure 4 shows a model in which multiple memory chips 510 store multiple partitions 310, and a buffer chip 520, such as a PNM chip, includes an LPU calculation unit 521 for LPU calculations.

[0042] The embodiment shown in Figure 5 illustrates an example of a model in which PIM and PNM are combined. For example, multiple memory chips 610 may store multiple partitions 310. Furthermore, each of the multiple memory chips 610 may have a PIM-type LPU arithmetic unit 611, which is a storage unit such as a MAC tree. In this case, the buffer chip 620 may have an LPU arithmetic unit 621 for the remaining high-level operations of the LPU implemented in a PNM manner.

[0043] Figure 6 illustrates the weight matrix data mapping of a high-bandwidth memory for matrix multiplication in a latency processing unit according to one embodiment of the present invention.

[0044] Referring to Figure 6, in this embodiment, the LPU 100 has the same number of MAC trees as the number of MAC trees 131, and the same number of memory channels in the SMA 110. Therefore, weight matrix data mapped to a high-bandwidth memory 610 such as an HBM 200 can be stored so that weight data can be retrieved without accessing other memory channels during matrix multiplication of each MAC tree.

[0045] Specifically, the weight matrix data may be stored in the high-bandwidth memory 610 such that the column direction D1 of the weight matrix is ​​mapped to each channel 620-n for each of the multiple MAC trees 131. Since matrix multiplication can be performed in parallel in the column direction of the weight matrix, multiple MAC trees 131 may read the column direction data on their respective allocated memory channels 620-n and perform matrix multiplication.

[0046] Next, the weight matrix data may be mapped so that multiple MAC trees 131 can accumulate the weight matrix in the row direction D2 to complete the final calculation result. The number of rows of data mapped at one time is determined by the bandwidth of the high-bandwidth memory 610, which may be determined by the size of the tile that multiple MAC trees 131 can process at one time.

[0047] Figure 7 is a diagram illustrating a high-bandwidth memory interface included in a latency processing unit according to one embodiment of the present invention.

[0048] Referring to Figure 7, the SMA110 may connect the LMU150, multiple MAC trees 131, and high-bandwidth memory 610. The SMA110 is not connected to other computing units of the LPU100, and therefore the high-bandwidth memory interface can be minimized in terms of hardware resources.

[0049] Multiple MAC trees 131 and memory channels 620-n may be connected in a one-to-one correspondence. That is, multiple MAC trees 131 do not need to access channels other than the one directly assigned to them, and matrix multiplication can be performed without using complex interfaces such as crossbars, which use many resources and have high latency.

[0050] The SMA110 may consist only of a read interface for multiple MAC trees 131 to retrieve weight matrix data stored in the high-bandwidth memory 610. In other words, as described below, the results of the calculations are stored in the high-bandwidth memory 610 via the LMU150, so no write interface for multiple MAC trees 131 to the high-bandwidth memory 610 is configured, and hardware resources can be reduced accordingly.

[0051] Conversely, the SMA110 may only configure a write interface between the LMU150 and the high-bandwidth memory 610. The calculation results stored in the internal buffer LMU150 may be transmitted through the SMA110 to be recorded in the high-bandwidth memory 610, and the memory channel to be recorded may be selected using the demultiplexer 710.

[0052] Figure 8 is a diagram illustrating a reconfigurable multifunction computing unit included in a latency processing unit according to one embodiment of the present invention.

[0053] Referring to Figure 8, the VXE140 may include multiple multifunction arithmetic data paths 810 and 820, and these multiple multifunction arithmetic data paths 810 and 820 may be connected to an operator / result value chain network 830 to form various combinations of arithmetic units.

[0054] As shown in Figure 8, the multiple multi-function computation data paths 810 and 820 may include various computation units necessary for, for example, lookup table-based nonlinear activation functions and masking operations. However, the configuration of the computation units of the reconfigurable multi-function computation data paths 810 and 820 shown in Figure 8 is merely illustrative, and it goes without saying that any additional computation units necessary for large-scale model computations can be included in the multi-function computation data paths 810 and 820. The results calculated by the VXE140 may be transmitted to the LMU150.

[0055] Figure 9 is a diagram illustrating the configuration of an address-based non-sequential multi-unit scheduler included in a latency processing unit according to one embodiment of the present invention.

[0056] Referring to Figure 9, the ISU160, which is an address infrastructure non-sequential multi-unit scheduler included in the LPU100 according to this embodiment, may include an address infrastructure instruction dependency determination and scheduling controller 910, multiple instruction issue controllers 921, 922, 923, 924, a multi-bank buffer address state table 950, an instruction buffer 960, and a result address state update logic 970 and a multi-unit instruction dispatcher 980.

[0057] The ISU160 may operate each arithmetic unit and data transfer unit simultaneously through the address base instruction dependency determination and scheduling controller 910 and multiple instruction word issue controllers 921, 922, 923, and 924. In this case, the ISU160 may change the state of the operator address and result address in the multibank buffer address state table 950 for instructions executed by each arithmetic unit to 1.

[0058] The multibank buffer address state table 950 may change the state of the result address of an instruction that has finished executing to 0 through the result address state update logic 970.

[0059] The address infrastructure's instruction dependency determination and scheduling controller 910 may refer to the address status through the multibank buffer address status table 970 and determine the dependencies between instructions to be executed and instructions currently being executed, as well as the dependencies between instructions to be executed. This allows for pre-processing of instructions with no dependencies, thereby minimizing the idling time of each arithmetic unit and data transfer unit.

[0060] The address-based instruction dependency determination and scheduling controller 910 included in the ISU160 may load and process instruction words from the instruction word buffer 960. At this time, the address-based instruction dependency determination and scheduling controller 910 may execute loop instruction words, decode other instruction words to distinguish them, and transmit them through the multi-unit instruction word dispatcher 980 to the device-to-device instruction word issue controller 921, the direct memory access instruction word issue controller 922, the MAC tree instruction word issue controller 923, and the reconfigurable multi-function arithmetic unit instruction word issue controller 924.

[0061] The ISU160 may receive and store instructions for the LPU100 from the host computer via the PCIe interface 170, and store the current state of the LPU100 in a state register. The host computer may check the state register via the PCIe interface 170.

[0062] Figure 10 shows an example of the internal configuration of a MAC tree in one embodiment of the present invention. The MAC tree 1000 in this embodiment may correspond to the MAC tree 131 described above. The MAC tree 1000 may include a floating-point multiplier 1010, a fixed-point adder tree 1020, and a fixed-point accumulator 1030.

[0063] The floating-point multiplier 1010 may be implemented in a structure in which multiple floating-point multipliers are connected in parallel in order to smoothly perform MAC operations using weight data called from external memory such as a high-bandwidth memory (for example, a high-bandwidth memory 610) or an internal buffer (for example, an LMU 150).

[0064] Furthermore, the fixed-point adder tree 1020 and the fixed-point accumulator 1030 may be implemented in a configuration that connects to the floating-point multiplier 1010 to store the multiplication results of the floating-point multiplier 1010. In this case, the fixed-point adder tree 1020 may process the addition of the multiplication results of the floating-point multiplier 1010, and the fixed-point accumulator 1030 may store the output of the fixed-point adder tree.

[0065] The MAC tree 1000 may further include an FP-to-FXP converter 1040 for converting the output of the floating-point multiplier 1010 from floating-point (FP) to fixed-point (FXP).

[0066] Such MAC tree 1000-based arithmetic units may be designed to efficiently utilize all of the provided memory bandwidth by setting the number of multipliers or the number of MAC trees 1000 according to the provided memory bandwidth. In this case, the MAC tree 1000 may further include an FXP-to-FP converter 1050 for converting the output of the fixed-point accumulator 1030 from fixed-point to floating-point.

[0067] To reduce the logic complexity and associated area burden of floating-point arithmetic units while maintaining accuracy, multipliers were used as floating-point arithmetic units and adders as fixed-point arithmetic units.

[0068] Figure 11 is a diagram illustrating the performance of an arithmetic unit according to one embodiment of the present invention. Figure 11 shows the difference in resources between a MAC tree-based arithmetic unit where both the multiplier and adder tree are implemented using floating-point arithmetic units, and a MAC tree-based arithmetic unit where the multiplier is implemented using floating-point arithmetic units and the adder tree is implemented using fixed-point arithmetic units, as in one embodiment of the present invention. In particular, when implemented with an FPGA, the fixed-point adder can be made to not use LUT logic resources by replacing the Look-up Table (LUT) logic resource with a DSP (Digital Signal Processing) component. Since the DSP is a circuit included in the FPGA, it can be implemented at high frequencies by reducing area overhead through the arithmetic unit structure provided. As an example, Figure 11 shows the difference in resources for LUT, flip-flop, and DSP when implemented with an FPGA. Resources have decreased in all three aspects of LUT, flip-flop, and DSP, and in particular, it can be seen that the LUT, which accounts for the majority of hardware resources, has seen a reduction of more than 50%.

[0069] On the other hand, in language model calculations, operations that use model weights as operands (for example, operations that generate query, key, and value values ​​in attention operations) can be performed using relatively low-precision data. Conversely, in language model calculations that use activation values ​​as operands (for example, score operations that use query, key, and value as operands), operations must be performed using high-precision data. When using arithmetic units that only support high precision, memory bandwidth cannot be used efficiently when operations can be performed with relatively low-precision data, and unnecessary calculations are performed at high precision.

[0070] Therefore, to efficiently support both high and low precision, we propose a solution that provides a mixed-precision floating-point multiplier supporting two modes and that allows switching between the two modes in real time. The first mode is a high-precision mode that performs calculations with high precision, and the second mode is a high-performance mode that performs calculations with relatively lower precision but high computational bandwidth. Such a mixed-precision floating-point multiplier can selectively execute the two modes in a single multiplier through operand preprocessing and result value postprocessing, thereby saving the logic overhead of the mixed-precision floating-point arithmetic unit.

[0071] Figure 12 shows an example of the structure of a mixed-precision floating-point multiplier in one embodiment of the present invention, and Figure 13 shows examples of two modes of the mixed-precision floating-point multiplier in one embodiment of the present invention.

[0072] Referring to Figure 12, the hardware-shared mixed-precision floating-point arithmetic unit 1200 may include a mixed-precision floating-point multiplier 1210, and the mixed-precision floating-point multiplier 1210 may include a mixed-precision fixed-point exponent adder 1211 and a mixed-precision fixed-point mantissa multiplier 1212.

[0073] Referring to Figures 12 and 13, the mixed-precision floating-point multiplier 1210 can also support faster calculations through high-performance modes such as 8-bit mode, while supporting high-precision modes such as 16-bit mode.

[0074] When operating in high-precision mode, the mixed-precision floating-point multiplier 1210 may receive inputs of two operands (operand 1 and operand 2) with the same bit precision from the LMU 150 and the wideband memory 610, and perform a multiplication operation at the maximum bit precision supported by the mixed-precision floating-point multiplier 1210.

[0075] Furthermore, when the mixed-precision floating-point multiplier 1210 operates in high-performance mode, it may perform two or more multiplications. For example, as shown in Figure 13, it is possible to obtain two multiplication results (result value 1 and result value 2) by simultaneously performing two operand (2-(i), 2-(ii)) multiplications on one operand (operand 1). For this purpose, the hardware-shared mixed-precision floating-point arithmetic unit 1200 can quantize and perform calculations on the model parameters with less precision.

[0076] Figure 14 shows an example of a mixed precision fixed-point mantissa multiplier according to one embodiment of the present invention, and Figure 15 shows an example of a mixed precision fixed-point exponent adder according to one embodiment of the present invention.

[0077] The mixed-precision fixed-point mantissa multiplier 1212 operates by performing different pre-processing of operators and post-processing of result values ​​depending on the mode described above. As a result, the mixed-precision floating-point multiplier 1210 enables the selective execution of high-precision and high-performance floating-point multiplication operations with only one fixed-point multiplier (mixed-precision fixed-point multiplier 1212), thereby saving hardware resources.

[0078] In this case, the mixed precision fixed-point mantissa multiplier 1212 may perform multiplication twice and then, through post-processing, realize two modes: a high-precision mode and a high-performance mode.

[0079] First, in high-precision mode, the first of the two multiplications is the multiplication between the mantissa of operand 1 and the lower bits of the mantissa of operand 2, and the second multiplication is the multiplication between the fractional part of operand 1 and the upper bits of the fractional part of operand 2. Then, the mixed-precision fixed-point mantissa multiplier 1212 can derive the final mantissa multiplication result by performing a shift on the result of the first multiplication (multiplication result value 1) and performing a sign extension on the result of the second multiplication (multiplication result value 2) and adding them together.

[0080] Next, in high-performance mode, the first of the two multiplications is the multiplication between the mantissa of operand 1 and the mantissa of operand 2-(i), and the second is the multiplication between the mantissa of operand 1 and the mantissa of operand 2-(ro). Subsequently, the mixed-precision fixed-point mantissa multiplier 1212 can derive two mantissa multiplication results by concatenating each multiplication result (multiplication result value 1 and multiplication result value 2) without any post-processing.

[0081] Furthermore, not only the mixed-precision fixed-point mantissa multiplier 1212, but also the mixed-precision fixed-point exponential adder 1211 can selectively perform high-precision and high-performance floating-point multiplication operations through operator pre-processing and post-processing, thereby saving hardware resources. The calculation result of the mixed-precision floating-point multiplier 1210 may be partially summed via the fixed-point adder tree 1020 and fixed-point accumulator 1030 shown in Figure 10. The matrix multiplication result output from the MAC tree 131 is immediately provided to the VXE 140, thereby minimizing the time required for memory access and the resulting delay.

[0082] In this case, the mixed precision fixed-point exponent adder 1211 performs preprocessing for each operator and two additions, and then implements two modes through postprocessing.

[0083] First, let's explain the high-precision mode. The first operand of the first addition is obtained by adding a zero to the most significant bit of the lower bit of the exponent of operand 1, and the second operand is the lower bit of the exponent of operand 2. The first operand of the second addition is obtained by adding a zero to the least significant bit of the upper bit of the exponent of operand 1, and the second operand is obtained by adding a zero to the least significant bit of the upper bit of the exponent of operand 2. After performing the two additions, the mixed-precision fixed-point exponent adder 1211 adds the results of each operation to derive the final exponent sum value.

[0084] Next, let's describe the high-performance mode. In the first addition, the first operand is the exponent of operand 1, and the second operand is the exponent of operand 2-(i). In the second addition, the first operator is the exponent of operand 1, and the second operator is the sign-extended exponent of operand 2-(i). After performing the two additions, the mixed-precision fixed-point exponent adder 1211 can derive the two exponent sum results by cutting and concatenating the results of each operation.

[0085] Figure 16 is a flowchart showing an example of how a MAC tree-based computing unit operates according to one embodiment of the present invention. The MAC tree-based computing unit may correspond to the MAC tree 1000 described with reference to Figure 10. Such a MAC tree-based computing unit may correspond to one of several MAC tree-based computing units included in a hardware accelerator for accelerating artificial intelligence models (for example, an LPU 100).

[0086] At this time, depending on the memory bandwidth provided to the hardware accelerator, at least one of the number of MAC tree-based arithmetic units included in the hardware accelerator and the number of multiple floating-point multipliers included in the MAC tree-based arithmetic units may be determined. In addition, multiple MAC tree-based arithmetic units may perform matrix multiplication operations for at least one of the multiple partitions that realize the artificial intelligence model.

[0087] This allows for the effective use of memory bandwidth to process large amounts of data in generative large-scale artificial intelligence models.

[0088] In step 1610, the MAC tree base arithmetic unit may use a plurality of parallel-connected floating-point multipliers to process multiplication operations on data transmitted from external memory. The external memory may include a broadband memory containing at least one partition, and a local memory unit (for example, LMU150) included in the hardware accelerator. Each of the plurality of floating-point multipliers may also include a mixed-precision fixed-point exponent adder for exponent addition (for example, mixed-precision fixed-point exponent adder 1211) and a mixed-precision fixed-point mantissa multiplier for mantissa multiplication (for example, mixed-precision fixed-point mantissa multiplier 1212).

[0089] As described above, each of the multiple floating-point multipliers may operate according to high-precision mode and high-performance mode. For example, each of the multiple floating-point multipliers may, when high-precision mode is selected, process a multiplication between a first operand and a second operand with the same bit precision to calculate a first result value. Alternatively, each of the multiple floating-point multipliers may, depending on the selection of high-performance mode, simultaneously process a first multiplication between the third operand of the first bit precision and the 4-1 operand of the second bit precision, and a second multiplication between the third operand and the 4-2 operand of the second bit precision, to simultaneously calculate the second result value of the first multiplication and the third result value of the second multiplication. For example, the first bit precision may include 16-bit precision, and the second bit precision may include 8-bit precision.

[0090] In step 1620, the MAC tree base arithmetic unit may use multiple first converters connected to multiple floating-point multipliers to convert the results of each multiplication operation of the multiple floating-point multipliers from floating-point to fixed-point. Here, the multiple first converters may correspond to the FP2FXP converter 1040 described with reference to Figure 10.

[0091] In step 1630, the MAC tree-based arithmetic unit may use a fixed-point adder tree to process the addition of the converted results of multiple floating-point multipliers. Here, the fixed-point adder tree may correspond to the fixed-point adder tree 1020 described with reference to Figure 10.

[0092] In step 1640, the MAC tree base arithmetic unit may use a fixed-point accumulator to store the output of the fixed-point adder tree. Here, the fixed-point accumulator may correspond to the fixed-point accumulator 1030 described with reference to Figure 10.

[0093] In step 1650, the MAC tree base arithmetic unit may use a second converter to convert the output of the fixed-point accumulator from fixed-point to floating-point. Here, the second converter may correspond to the FXP2FP converter 1050 described with reference to Figure 10.

[0094] Thus, according to embodiments of the present invention, it is possible to provide a hardware accelerator arithmetic unit that maximizes the use of memory bandwidth provided for accelerating generative large-scale language models that have a large amount of data but are difficult to parallelize. Furthermore, it is possible to provide an arithmetic unit that can reduce the high hardware resource overhead of floating-point based MAC tree structures. Moreover, it is possible to provide an arithmetic unit that can minimize hardware resource overhead while effectively supporting two precision operations.

[0095] As described above, the embodiments have been explained based on the limited embodiments and drawings, but those skilled in the art will be able to make various modifications and variations from the above description. Therefore, even different embodiments fall within the scope of the attached claims as long as they are equivalent to the claims.

Claims

1. A MAC (Multiply-and-Accumulation) tree-based arithmetic unit, Multiple floating-point multipliers connected in parallel to process multiplication operations on data transmitted from external memory, Multiple first converters for converting the output of each of the multiple floating-point multipliers from floating-point to fixed-point, A fixed-point adder tree connected to the plurality of first converters, which processes the sum of the multiplication results of the plurality of floating-point multipliers, A fixed-point stabilizer that stores the output of the fixed-point adder tree, A second converter that converts the output of the fixed-point accumulator from fixed-point to floating-point, Includes, The MAC tree-based computing unit corresponds to one of several MAC tree-based computing units included in a hardware accelerator for accelerating artificial intelligence models. Depending on the memory bandwidth provided to the hardware accelerator, at least one of the number of MAC tree-based arithmetic units included in the hardware accelerator and the number of floating-point multipliers included in the MAC tree-based arithmetic units is determined. A MAC tree-based arithmetic unit characterized by the following:

2. The MAC tree-based arithmetic unit according to claim 1, characterized in that the plurality of MAC tree-based arithmetic units perform matrix multiplication for at least one partition among the plurality of partitions that realize the artificial intelligence model.

3. The MAC tree-based computing unit according to claim 2, characterized in that the external memory includes a high-bandwidth memory in which the at least one partition is stored, and a local memory unit included in the hardware accelerator.

4. Each of the aforementioned floating-point multipliers is: A mixed precision fixed-point exponent adder for adding exponents, A mixed precision fixed-point mantissa multiplier for mantissa multiplication, A MAC tree-based arithmetic unit according to claim 1, characterized by including the following:

5. Each of the aforementioned floating-point multipliers is: Depending on whether high-precision mode is selected, the first result value is calculated by processing the multiplication between a first operand and a second operand with the same bit precision. A MAC tree-based arithmetic unit according to claim 1, characterized by the above.

6. A MAC (Multiply-and-Accumulation) tree-based arithmetic unit, Multiple floating-point multipliers connected in parallel to process multiplication operations on data transmitted from external memory, Multiple first converters for converting the output of each of the multiple floating-point multipliers from floating-point to fixed-point, A fixed-point adder tree connected to the plurality of first converters, which processes the sum of the multiplication results of the plurality of floating-point multipliers, A fixed-point stabilizer that stores the output of the fixed-point adder tree, A second converter that converts the output of the fixed-point accumulator from fixed-point to floating-point, Includes, Each of the aforementioned floating-point multipliers is: Depending on the selection of high-performance mode, the first multiplication between the first operand of the first bit precision and the second-to-first operand of the second bit precision and the second multiplication between the first operand and the second-to-second operand of the second bit precision are processed simultaneously, and the first result value of the first multiplication and the second result value of the second multiplication are calculated simultaneously. A MAC tree-based arithmetic unit characterized by the following:

7. The first bit precision includes a 16-bit precision. The second bit precision includes an 8-bit precision. A MAC tree-based arithmetic unit according to claim 6, characterized by the above.

8. A method for operating a MAC (Multiply-and-Accumulation) tree-based arithmetic unit, The MAC tree-based arithmetic unit includes a plurality of floating-point multipliers connected in parallel, a plurality of first converters connected to the plurality of floating-point multipliers, a fixed-point adder tree, a fixed-point accumulator, and a second converter. The steps include: processing multiplication operations on data transmitted from external memory using the aforementioned multiple floating-point multipliers; The steps include: using the plurality of first converters to convert the result of the multiplication operation of each of the plurality of floating-point multipliers from floating-point to fixed-point; The steps include: using the fixed-point adder tree to process the sum of the converted results of the plurality of floating-point multipliers; The steps include: storing the output of the fixed-point adder tree using the fixed-point accumulator; The steps include using the second converter to convert the output of the fixed-point accumulator from fixed-point to floating-point, Includes, The MAC tree-based computing unit corresponds to one of several MAC tree-based computing units included in a hardware accelerator for accelerating artificial intelligence models. Depending on the memory bandwidth provided to the hardware accelerator, at least one of the number of MAC tree-based arithmetic units included in the hardware accelerator and the number of floating-point multipliers included in the MAC tree-based arithmetic units is determined. A method of operating a MAC tree-based arithmetic unit characterized by the following.

Images