Hybrid precision MAC tree structure for maximizing memory bandwidth usage to accelerate operation of generative large-scale language models

Abstract

The present invention provides a hybrid precision MAC (multiple-and-multiple-accumulation) tree structure for maximizing memory bandwidth usage to accelerate the operation of a generative large scale language model. The present invention provides a hybrid precision MAC (multiple-and-multiple-accumulation) tree structure for maximizing memory bandwidth usage to accelerate the operation of the generative large scale language model. A MAC tree-based arithmetic unit according to one embodiment may include: a plurality of floating point multipliers connected in parallel and processing multiplication of data transferred from an external memory; a plurality of first converters for converting the output of each of the plurality of floating point multipliers from a floating point to a fixed point; a fixed-point adder tree which is connected to the plurality of first converters and processes the addition of the multiplication results of the plurality of floating-point multipliers; a fixed-point accumulator that accumulates the output of the fixed-point adder tree; and a second converter that converts the output of the fixed-point accumulator from a fixed point to a floating point.

Description

TECHNICAL FIELD

[0001] Embodiments of the present invention relate to a mixed-precision MAC tree structure for maximizing memory bandwidth usage to accelerate operations of a generative large-scale language model. BACKGROUND

[0002] Recently, as generative large-scale language models are rapidly developed, the size of the model is gradually increasing in order to obtain high accuracy, thereby having a few million to several billion model parameters. Therefore, a large amount of data must be fetched from a memory at once to perform operations without obstruction.

[0003] In addition, a floating-point operation unit is required to support the generative large-scale language model operation without accuracy loss. However, such a floating-point operation unit has a high logic complexity, thereby having a problem of occupying a large area. SUMMARY

[0004] PROBLEMS TO BE SOLVED BY THE INVENTION

[0005] The present invention can provide an operation unit of a hardware accelerator that maximizes memory bandwidth usage provided to accelerate a generative large-scale language model that is difficult to be parallelized and has a large amount of data.

[0006] The technical problems of the present invention are not limited to the above-mentioned technical problems, and other technical problems not mentioned herein will be clearly understood by those skilled in the art from the following description.

[0007] MEANS FOR SOLVING PROBLEMS

[0008] The present invention provides an operation unit based on a MAC (multiply-and-accumulation) tree, the operation unit based on the MAC tree including: a plurality of floating-point multipliers connected in parallel and processing multiplication operations on data transferred from an external memory; a plurality of first converters for converting outputs of the respective floating-point multipliers from floating-point to fixed-point; a fixed-point adder tree connected to the plurality of first converters and processing addition of multiplication results of the plurality of floating-point multipliers; a fixed-point accumulator accumulating an output of the fixed-point adder tree; and a second converter for converting an output of the fixed-point accumulator from fixed-point to floating-point.

[0009] According to an aspect of the present invention, the operation unit based on the MAC tree can correspond to one of a plurality of operation units based on the MAC tree included in a hardware accelerator for accelerating an artificial intelligence model.

[0010] According to another aspect of the invention, the feature may be that the number of the MAC tree-based arithmetic units included in the hardware accelerator and at least one of the number of the plurality of floating-point multipliers included in the MAC tree-based arithmetic units are determined based on the memory bandwidth provided to the hardware accelerator.

[0011] According to another aspect of the invention, the plurality of MAC tree-based arithmetic units may perform matrix multiplication operations on at least one of the plurality of partitions for specifically implementing the artificial intelligence model described above.

[0012] According to another aspect of the invention, the external memory may include a high-bandwidth memory storing at least one of the aforementioned partitions.

[0013] According to another aspect of the invention, each of the plurality of floating-point multipliers comprises: a mixed-precision fixed-point exponent multiplier for adding the exponent; and a mixed-precision fixed-point mantissa multiplier for multiplying the mantissa.

[0014] According to another aspect of the invention, the plurality of floating-point multipliers each calculate a first result value as the high-precision mode is selected to process multiplication between a first operand and a second operand with the same bit precision.

[0015] According to another aspect of the invention, the plurality of floating-point multipliers, when a high-performance mode is selected, simultaneously process a first multiplication between a third operand of first bit precision and a fourth-first operand of second bit precision, and a second multiplication between the third operand and a fourth-second operand of second bit precision, and simultaneously calculate a second result value of the first multiplication and a third result value of the second multiplication.

[0016] According to another aspect of the present invention, the first bit precision may include 16-bit precision, and the second bit precision may include 8-bit precision.

[0017] This invention provides an operation method for an arithmetic unit based on a MAC (multiply-and-accumulation) tree. In this method, the MAC-based arithmetic unit includes multiple floating-point multipliers connected in parallel, multiple first converters connected to the multiple floating-point multipliers, a fixed-point adder tree, a fixed-point accumulator, and a second converter. The operation method includes: processing multiplication operations on data transferred from external memory using the multiple floating-point multipliers; converting the results of the multiplication operations of each of the multiple floating-point multipliers from floating-point to fixed-point using the multiple first converters; performing addition on the results of the conversion of the multiple floating-point multipliers using the fixed-point adder tree; accumulating the output of the fixed-point adder tree using the fixed-point accumulator; and converting the output of the fixed-point accumulator from fixed-point to floating-point using the second converter.

[0018] Specific details of other embodiments are included in the summary and drawings.

[0019] The effects of the invention

[0020] The present invention can provide a hardware accelerator that maximizes the use of memory bandwidth for accelerating generative large-scale language models that are difficult to parallelize and have large amounts of data.

[0021] This invention can provide an arithmetic unit that can reduce the high hardware resource overhead of floating-point MAC tree structures.

[0022] This invention provides an arithmetic unit that can effectively support two precision operations while minimizing hardware resource overhead.

[0023] The effects of this invention are not limited to those mentioned above, and those skilled in the art will clearly understand other effects not mentioned from the description in the claims. Attached Figure Description

[0024] Figure 1 This is a diagram illustrating an example of the structure of a delay processing unit according to an embodiment of the present invention.

[0025] Figures 2 to 5 This is a diagram illustrating examples of specific implementation models of each LPU according to various embodiments of the present invention.

[0026] Figure 6 This is a diagram illustrating the weight matrix data mapping of a high-bandwidth memory for matrix multiplication operations of a delay processing unit according to an embodiment of the present invention.

[0027] Figure 7This is a diagram illustrating a high-bandwidth memory interface included in a delay processing unit according to an embodiment of the present invention.

[0028] Figure 8 This is a diagram illustrating a reconfigurable multifunctional computing unit included in a delay processing unit according to an embodiment of the present invention.

[0029] Figure 9 This is a diagram illustrating the configuration of an address-based non-sequential multi-unit scheduler included in a delay processing unit according to an embodiment of the present invention.

[0030] Figure 10 This is a diagram illustrating an example of the internal structure of a MAC tree in one embodiment of the present invention.

[0031] Figure 11 This is a diagram used to illustrate the performance of an arithmetic unit according to an embodiment of the present invention.

[0032] Figure 12 This is a diagram illustrating a mixed-precision floating-point multiplier in one embodiment of the present invention.

[0033] Figure 12 This is a diagram illustrating an example of the structure of a mixed-precision floating-point multiplier in one embodiment of the present invention.

[0034] Figure 13 This is a diagram illustrating an example of two modes of a mixed-precision floating-point multiplier in one embodiment of the present invention.

[0035] Figure 14 This is a diagram illustrating an example of a mixed-precision fixed-point mantissa multiplier according to an embodiment of the present invention.

[0036] Figure 15 This is a diagram illustrating an example of a mixed-precision fixed-point exponential adder according to an embodiment of the present invention.

[0037] Figure 16 This is a flowchart illustrating an example of an operation method of a MAC tree-based arithmetic unit according to an embodiment of the present invention. Detailed Implementation

[0038] The advantages and features of the present invention, as well as the methods for implementing them, are described below with reference to the appendix. Figure 1 The details of the various embodiments will make it clear. However, the invention is not limited to the embodiments disclosed below, but can be embodied in various different ways. These embodiments are provided only to make the disclosure of the invention complete and to fully inform those skilled in the art of the scope of the invention, which is defined only by the scope of the claims. Throughout the specification, the same reference numerals refer to the same constituent elements.

[0039] When a component is referred to as being "connected to" or "coupled to" another component, it includes cases where the component is directly connected to or coupled to the other component, or cases where other components are involved between them. Conversely, when a component is referred to as being "directly connected to" or "directly coupled to" another component, it means that no other components are involved between them. "And / or" includes each of the mentioned items as well as all combinations of more than one item.

[0040] The terminology used in this specification is for illustrative purposes and is not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise. The terms "comprises" and / or "comprising" as used in this specification mean that the mentioned constituent elements, steps, operations, and / or components do not exclude the presence or addition of more than one other constituent element, step, operation, and / or component.

[0041] Although terms such as "first," "second," etc., are used to describe various constituent elements, these constituent elements are not limited by these terms. These terms are used merely to distinguish one constituent element from others. Therefore, the "first constituent element" mentioned below can certainly also be a "second constituent element" within the technical concept of this invention.

[0042] Unless otherwise defined, all terms used in this specification (including technical and scientific terms) may be used in the sense that is commonly understood by one of ordinary skill in the art. Furthermore, terms defined in commonly used dictionaries should not be interpreted ideally or excessively unless explicitly defined otherwise.

[0043] Figure 1 This is a diagram illustrating an example of the structure of a delay processing unit according to an embodiment of the present invention.

[0044] Reference Figure 1According to an embodiment of the present invention, the LPU (Latency Processing Unit) 100 may include an SMA (Streamlined Memory Access) 110, an OIU (Operand Issue Unit) 120, an SXE (Streamlined eXecution Engine) 130, a VXE (Vector eXecution Engine) 140, an LMU (Local Memory Unit) 150, an ISU (Instruction Scheduling Unit) 160, a PCIe interface (PCIe (Peripheral Component Interconnect express) interface, and a P2P (Peer to Peer) interface 180.

[0045] The SMA110 can be a special type of DMA (Direct Memory Access). As an example, the SMA110 can connect all channels of the HBM200 (32 channels for example) to the execution engine (SEE130 for example) to transmit FP16 (half-precision floating-point) data at maximum bandwidth. To continuously transfer memory requests based on pre-loaded memory (MEM) instructions, the SMA110 can be designed as a deep FIFO (First In First Out). Considering the memory mapping of the hardware, matrix changes or transpose operations can be eliminated to reduce latency. Therefore, the SMA110 can stream data received at maximum burst size to the execution engine with minimal latency. The SMA110 can also efficiently perform matrix transpose using strobe signals. Streaming data can include parameters for vector matrix operations (e.g., weights, biases) and other vector-related operations (e.g., gamma / beta, embedding).

[0046] The OIU120 can adjust the data streamed from the SMA110 (operand 1, for example) and the input from on-chip memory (operand 2, for example) before issuing them to the execution engine. Based on the Execute (EXE) instructions, the OIU120 can generate the microcode of the target engine that constitutes the execution engine and determines the operands. Furthermore, in the OIU120, multiplexed buffers can be configured to eliminate read latency for static operands (e.g., input vectors), and asymmetric buffers can be configured to retain vectorized data used as scalars (e.g., biases). Therefore, appropriate operands are almost always prefetched, ready to be immediately issued to the execution engine.

[0047] As the primary computing hardware of the LPU100, the SXE130 can be designed to maximize the utilization of the incoming bandwidth in order to perform vector-matrix multiplication (V·M) such as attention, one-dimensional convolution, and feedforward networks. The SXE130 can include a number of MAC (multiply-and-accumulation) trees 131 that allow the receive bandwidth of the HBM200 to match the computation bandwidth. For example, if 1024 elements are received from the HBM200 per cycle, 16 MAC trees 131 with 64 input vectors can make the receive bandwidth and computation bandwidth consistent. Furthermore, the MAC tree 131 with 64 input vectors can be composed of 64 multipliers and 63 adders.

[0048] Multiple MAC trees 131 can perform matrix multiplication operations and can be connected to the HBM200, which is both external memory and high-bandwidth memory, via SMA110 on a channel basis. Specifically, one of the multiple MAC trees 131 is connected to the HBM200 through a channel, thereby maximizing the transmission bandwidth between the LPU100 and the HBM200, and enabling the matrix multiplication operations required for ultra-large-scale artificial intelligence models to be performed without bottlenecks. Therefore, the number of multiple MAC trees 131 can be configured to be the same as the number of memory channels of the HBM200.

[0049] The matrix multiplication results of multiple MAC trees 131 can be provided to the VXE140. The VXE140 can be implemented using a user-specified low-latency ALU (Arithmetic Logic Unit) and can perform vector operations such as token embedding, softmax, normalization, and residual operations. Since these vector operations occur relatively infrequently, the fan-in of the OIU120 to this path can be adjusted to reduce hardware resource consumption to a negligible performance loss. While receiving the results of multiple MAC trees 131 operations, the VXE140 can also receive activation values ​​from the LMU150 for subsequent operations. The VXE140 includes multiple multi-functional operation data paths, enabling it to be configured with various combinations of arithmetic units.

[0050] The LMU150 can pass activation values ​​to multiple MAC trees 131 and VXE140. To pass the same activation value to multiple MAC trees 131, the LMU150 can copy and transfer the activation value. Additionally, the LMU150 can store the results of operations on multiple MAC trees 131 and VXE140. In other words, the LMU150 can function as an internal buffer within the LPU100, corresponding to the HBM200. The LPU100 can store frequently reused activation values ​​or model parameters in matrix multiplication operations in the LMU150, and less frequently reused weights in the HBM200. The LMU150 can be implemented as a 4MB multi-channel register file with scalar and vector separation for fast, high-bandwidth access to input, output, and intermediate data. Furthermore, the LMU150 can be a multi-port system supporting simultaneous reading and writing during the write-to-store phase of the OIU120 and the execution engine.

[0051] The ISU160 can control the entire execution flow of the LPU100. The ISU160 can utilize Parallel Instruction Chaining (PIC), which allows for the sequential execution of dependent instructions. PIC separates instructions requiring independent hardware into groups of dependent instructions (e.g., memory (MEM) instructions, execution (EXE) instructions, and network (NET) instructions). Therefore, all instructions are executed in parallel with their respective instruction chains, resulting in low control overhead and reduced latency. The ISU160 can also update control registers (e.g., word and level numbers) for the execution engine. To maximize hardware utilization, the internal scheduler supports non-sequential execution of the SXE130 and VXE140, and a robust scoreboard can be designed to handle data risks. For example, the ISU160 can schedule operations so that multiple MAC trees 131 and VXE140 can execute simultaneously. Furthermore, the ISU160 pre-executes non-dependent instructions to maximize parallel computation, thereby minimizing idle time for individual computing devices and memory access devices, increasing throughput and reducing latency.

[0052] The LPU100 can connect to the host computer via the PCIe interface 170, and receive the instructions required for the operation of the LPU100, the input values ​​and weights of the ultra-large-scale artificial intelligence model from the host computer, and transmit the results to the host computer after performing the calculations.

[0053] The LPU100 can be horizontally scaled out to a cluster of multiple LPUs connected via the P2P interface 180. This expanded cluster structure can further accelerate the computation of ultra-large-scale artificial intelligence models.

[0054] Figures 2 to 5 This is a diagram illustrating examples of specific implementation models of each LPU according to various embodiments of the present invention. Previously... Figure 1 The embodiments illustrate examples of specific implementation models using external memory with HBM200. Alternatively, DDR (Double Data Rate) can be used as external memory instead of HBM200. In this case, since large-scale models are difficult to store on a single device, they can be partitioned into multiple partitions and stored in external memory for multiple devices (multiple LPUs). In this scenario, synchronization between multiple devices may be required for inference of large-scale models.

[0055] exist Figure 2 In the previous embodiment, Figure 1The embodiments described similarly illustrate multiple external memories 320 storing multiple partitions 310 of a large-scale model, and multiple LPUs 330 connected in parallel with the multiple external memories 320. An LPU can be implemented on a single FPGA (Field Programmable Gate Array), and a partition can be connected in parallel with a single FPGA. The converter architecture includes multi-head attention, layer normalization, feedforward, etc., within the decoder layer, allowing for model parallelization of multi-head attention and feedforward. In this case, if multi-head attention ends, an embedding vector can be output as the result. Since a device only has a portion of the embedding vector, multiple devices need to share their respective embedding vectors to proceed to the next operation, thus requiring synchronization. At this point, considering scalability, an LPU can also be implemented with multiple external memories (for example, 2 or 4, etc.). As an example, in... Figure 1 In one embodiment, an example is shown using two HBM200s, each with a partition for its own storage.

[0056] exist Figure 3 In one embodiment, as an example of the PIM (Processing-in-Memory) model, an example is shown where the LPU is specifically implemented as a PIM chip, with both the partitioning and the LPU processing unit integrated on a single chip. Figure 3 In the embodiments shown, multiple LPUs 410, multiple partitions 310, and multiple LPU processing units 420, each of which can be specifically implemented as a PIM chip, are illustrated. In this case, each of the multiple LPUs 410 may include a partition and an LPU processing unit.

[0057] exist Figure 4 The embodiment illustrates an example of the PNM (Processing-near-Memory) model. It may be difficult to include the components for handling all LPU operations within a single PIM chip. Figure 4 In one embodiment, a model is shown in which multiple partitions 310 are stored in multiple memory chips 510 and an LPU arithmetic unit 521 for LPU operations is included in a buffer chip 520 such as a PNM chip.

[0058] exist Figure 5In the embodiments shown, an example of a model combining PIM and PNM is illustrated. As an example, multiple partitions 310 can be stored in multiple memory chips 610. Furthermore, each of the multiple memory chips 610 can specifically implement a PIM-based LPU arithmetic unit 611 as an accumulation unit such as a MAC tree. At this time, a PNM-based LPU arithmetic unit 621 for the remaining high-level operations of the LPU can be specifically implemented in the buffer chip 620.

[0059] Figure 6 This is a diagram illustrating the weight matrix data mapping of a high-bandwidth memory for matrix multiplication operations of a delay processing unit according to an embodiment of the present invention.

[0060] Reference Figure 6 Based on the fact that the number of MAC trees constituting multiple MAC trees 131 is the same as the number of memory channels of SMA 110, the LPU 100 of this embodiment can store the mapped weight matrix data in a high-bandwidth memory 610 such as HBM 200, so that the weight data can be loaded without accessing other memory channels when performing matrix multiplication operations of each MAC tree.

[0061] Specifically, the weight matrix data can be stored in the high-bandwidth memory 610 in a manner that maps the weight matrix along the column direction D1 to each channel 620-n in accordance with the number of multiple MAC trees 131. Since matrix multiplication can be performed in parallel along the column direction in the weight matrix, multiple MAC trees 131 can read the column direction data from their respective allocated memory channels 620-n to perform matrix multiplication.

[0062] Next, the weight matrix data can be mapped so that multiple MAC trees 131 accumulate along the row direction D2 of the weight matrix to complete the final calculation result. The number of rows of data mapped at one time can be determined by the bandwidth of the high-bandwidth memory 610, which can be determined by the size of the blocks that multiple MAC trees 131 can process at one time.

[0063] Figure 7 This is a diagram illustrating a high-bandwidth memory interface included in a delay processing unit according to an embodiment of the present invention.

[0064] Reference Figure 7 The SMA110 can connect to the LMU150, multiple MAC trees 131, and high-bandwidth memory 610. The SMA110 is not connected to other processing units of the LPU100, thus minimizing the high-bandwidth memory interface in terms of hardware resources.

[0065] Multiple MAC trees 131 and memory channels 620-n can be connected in a one-to-one correspondence. That is, multiple MAC trees 131 do not need to access channels other than those directly assigned to them, and thus matrix multiplication can be performed without using complex interfaces such as cross-bars that have long latency due to the use of large amounts of resources.

[0066] The SMA110 can serve as a read interface for loading weight matrix data stored in the high-bandwidth memory 610 onto multiple MAC trees 131. In other words, as explained below, since the computation results are stored in the high-bandwidth memory 610 via the LMU150, it does not serve as a write interface to the high-bandwidth memory 610 for multiple MAC trees 131, thereby reducing hardware resources accordingly.

[0067] In contrast, the SMA110 can serve as a write interface only between the LMU150 and the high-bandwidth memory 610. Through the SMA110, the computation results stored in the LMU150, which serves as an internal buffer, can be transmitted for recording into the high-bandwidth memory 610, and the memory channel of the recording target can be selected using the demultiplexer 710.

[0068] Figure 8 This is a diagram illustrating a reconfigurable multifunctional computing unit included in a delay processing unit according to an embodiment of the present invention.

[0069] Reference Figure 8 The VXE140 may include multiple multi-functional operation data paths 810 and 820, which can be connected to the operator / result value chain network 830 to form various combinations of arithmetic units.

[0070] like Figure 8 As shown, the multiple multifunctional computational data paths 810 and 820 can include various computational units required, such as lookup table-based nonlinear activation functions and mask operations, but... Figure 8 The configuration of the computational units in the reconfigurable multi-functional computational data paths 810 and 820 shown is exemplary; additional computational units required for large-scale model computations can, of course, be included in the multi-functional computational data paths 810 and 820. The results of computations performed by the VXE140 can be transferred to the LMU150.

[0071] Figure 9 This is a diagram illustrating the configuration of an address-based non-sequential multi-unit scheduler included in a delay processing unit according to an embodiment of the present invention.

[0072] Reference Figure 9The ISU160, which is an address-based non-sequential multi-unit scheduler included in the LPU100 according to this embodiment, may include an address-based instruction dependency judgment and scheduling controller 910, multiple instruction dispatch controllers 921, 922, 923, and 924, a multi-block buffer address status table 950, an instruction buffer 960, result address status update logic 970, and a multi-unit instruction dispatcher 980.

[0073] The ISU160 can simultaneously drive each arithmetic unit and data movement unit through the address-based instruction dependency judgment and scheduling controller 910 and multiple instruction issue controllers 921, 922, 923, and 924. At this time, the ISU160 can change the status of the operator address and result address in the multi-memory block buffer address status table 950 for instructions executed in each arithmetic unit to 1.

[0074] The multi-block buffer address status table 950 can change the status of the result address of an executed instruction to 0 through the result address status update logic 970.

[0075] The address-based instruction dependency determination and scheduling controller 910 can determine the dependencies between the instruction to be executed and the instruction currently being executed, as well as the dependencies between the instructions to be executed, by referring to the address status in the multi-block buffer address status table 970. This allows for the preprocessing of instructions without dependencies, thereby minimizing the idle time of each arithmetic unit and data movement unit.

[0076] The address-based instruction dependency determination and scheduling controller 910 included in the ISU160 can load instructions from the instruction buffer 960 for processing. At this time, the address-based instruction dependency determination and scheduling controller 910 executes loop instructions and can decode and distinguish other instructions. It can also pass instructions to the device-to-device instruction dispatch controller 921, the direct memory access instruction dispatch controller 922, the MAC tree instruction dispatch controller 923, and the reconfigurable multifunction arithmetic unit instruction dispatch controller 924 via the multi-unit instruction dispatcher 980.

[0077] The ISU160 can receive and store LPU100 instructions from the host computer via PCIe interface 170, and can also store the current state of LPU100 in a status register. The host computer can check this status register via PCIe interface 170.

[0078] Figure 10This is a diagram illustrating an example of the internal structure of a MAC tree in one embodiment of the present invention. The MAC tree 1000 according to this embodiment can correspond to the MAC tree 131 described above. The MAC tree 1000 may include a floating-point multiplier 1010, a fixed-point adder tree 1020, and a fixed-point accumulator 1030.

[0079] The floating-point multiplier 1010 can be specifically implemented as a structure that connects multiple floating-point multipliers in parallel, so as to perform MAC operations without hindrance using weight data obtained from external memory such as high-bandwidth memory (high-bandwidth memory 610 as an example) or internal buffer (LMU150 as an example).

[0080] Furthermore, the fixed-point adder tree 1020 and the fixed-point accumulator 1030 can be specifically implemented as connected to the floating-point multiplier 1010 to accumulate the multiplication results of the floating-point multiplier 1010. In this case, the fixed-point adder tree 1020 can handle the addition of the multiplication results of the floating-point multiplier 1010, and the fixed-point accumulator 1030 can accumulate the output of the fixed-point adder tree.

[0081] The MAC tree 1000 may further include an FP-to-FXP converter 1040 for converting the output of the floating-point multiplier 1010 from floating-point (FP) to fixed-point (FXP).

[0082] This arithmetic unit based on the MAC tree 1000 can be designed such that the number of multipliers or the number of MAC trees 1000 can be set according to the provided memory bandwidth, thereby enabling efficient use of the entire provided memory bandwidth. In this case, the MAC tree 1000 can further include an FXP-to-FP converter 1050 for converting the output of the fixed-point accumulator 1030 from fixed-point to floating-point.

[0083] In order to reduce the logical complexity of floating-point arithmetic units and the large overhead they bring while maintaining accuracy, multipliers are used as floating-point arithmetic units, while adders are used as fixed-point arithmetic units.

[0084] Figure 11 This is a diagram used to illustrate the performance of an arithmetic unit according to an embodiment of the present invention. Figure 11This illustrates the resource differences in a MAC tree-based arithmetic unit when both multipliers and adders are implemented as floating-point arithmetic units, versus when, as in one embodiment of the invention, the multipliers are implemented as floating-point arithmetic units while the adders are implemented as fixed-point arithmetic units. In particular, when implemented using an FPGA, the fixed-point adders can replace look-uptable (LUT) logic resources with DSP (Digital Signal Processing) to eliminate the use of LUT logic resources. Since the DSP is circuitry included within the FPGA, the provided arithmetic unit structure reduces area overhead and enables high-frequency implementation. As an example, Figure 11 The diagram illustrates the resource differences for LUTs, flip-flops, and DSPs implemented using an FPGA. Resources have been reduced in all three areas, with a reduction of over 50% particularly noticeable in LUTs, which handle the majority of hardware resources.

[0085] On the other hand, in language model computations where model weights are used as operands (for example, in attention computations where query, key, and value values ​​are generated), computations can be performed using data with relatively low precision. Conversely, in language model computations where activation values ​​are used as operands (for example, in score computations where query, key, and value are used as operands), computations must be performed using data with high precision. When using a processor that only supports high precision, memory bandwidth cannot be used efficiently when computations can be performed using relatively low precision data, and high precision is unnecessarily used for computation.

[0086] Therefore, to effectively support both high-precision and low-precision operations, a mixed-precision floating-point multiplier can be developed that supports both modes and can switch between them in real time. The first mode is a high-precision mode capable of high-precision operations, while the second mode is a high-performance mode that, although with relatively lower precision, can perform operations with high bandwidth. This mixed-precision floating-point multiplier, through operand preprocessing and result post-processing, can selectively execute both modes using only a single multiplier, thereby saving the logical overhead of the mixed-precision floating-point arithmetic unit.

[0087] Figure 12 This is a diagram illustrating an example of the structure of a mixed-precision floating-point multiplier in one embodiment of the present invention. Figure 13 This is a diagram illustrating an example of two modes of a mixed-precision floating-point multiplier in one embodiment of the present invention.

[0088] Reference Figure 12 The hardware-shared mixed-precision floating-point arithmetic unit 1200 may include a mixed-precision floating-point multiplier 1210, which may include a mixed-precision fixed-point exponent multiplier 1211 and a mixed-precision fixed-point mantissa multiplier 1212.

[0089] Reference Figure 12 and Figure 13 The mixed-precision floating-point multiplier 1210 can support not only high-precision modes such as 16-bit mode, but also high-performance modes such as 8-bit mode to support faster operations.

[0090] When operating in high-precision mode, the mixed-precision floating-point multiplier 1210 can receive two operands (operand 1 and operand 2) with the same bit precision from LMU 150 and high-bandwidth memory 610, and perform multiplication operations with the maximum bit precision supported by the mixed-precision floating-point multiplier 1210.

[0091] Furthermore, when the mixed-precision floating-point multiplier 1210 operates in high-performance mode, it can perform more than two multiplications, for example, such as Figure 13 As shown, two multiplications, 2-(a) and 2-(b), can be performed simultaneously on one operand (operand 1) to obtain two multiplication results (result value 1 and result value 2). For this purpose, the hardware-shared mixed-precision floating-point arithmetic unit 1200 can quantize the model's parameters for calculation with lower precision.

[0092] Figure 14 This is a diagram illustrating an example of a mixed-precision fixed-point mantissa multiplier according to an embodiment of the present invention. Figure 15 This is a diagram illustrating an example of a mixed-precision fixed-point exponential adder according to an embodiment of the present invention.

[0093] The mixed-precision fixed-point mantissa multiplier 1212 performs different operations on the preprocessing of the operator and the postprocessing of the result value according to the above pattern. Thus, the mixed-precision floating-point multiplier 1210 can selectively perform high-precision and high-performance floating-point multiplication operations with only one fixed-point multiplier (mixed-precision fixed-point mantissa multiplier 1212) and can save hardware resources.

[0094] At this point, the mixed-precision fixed-point mantissa multiplier 1212 performs two multiplications and then implements two modes—high-precision mode and high-performance mode—through post-processing.

[0095] First, in high-precision mode, the first multiplication in the two multiplications is the multiplication between the mantissa of operand 1 and the lower bits of the mantissa of operand 2, and the second multiplication is the multiplication between the mantissa of operand 1 and the higher bits of the mantissa of operand 2. Then, the mixed-precision fixed-point mantissa multiplier 1212 can perform a shift operation on the result of the first multiplication (multiplication result value 1) and a sign extension operation on the result of the second multiplication (multiplication result value 2) before adding them together to obtain the final mantissa multiplication result value.

[0096] Next, in high-performance mode, the first multiplication in the two multiplications is the multiplication between the mantissa of operand 1 and the mantissa of operand 2-(a), and the second multiplication is the multiplication between the mantissa of operand 1 and the mantissa of operand 2-(b). Afterwards, the mixed-precision fixed-point mantissa multiplier 1212 can concatenate the multiplication results (multiplication result value 1 and multiplication result value 2) without post-processing to obtain two mantissa multiplication result values.

[0097] Furthermore, not only the mixed-precision fixed-point mantissa multiplier 1212, but also the mixed-precision fixed-point exponent adder 1211 can selectively perform high-precision and high-performance floating-point multiplication operations through operator preprocessing and post-processing, while saving hardware resources. The operation result of the mixed-precision floating-point multiplier 1210 can be... Figure 10 The fixed-point adder tree 1020 and fixed-point accumulator 1030 perform partial sum operations. The matrix multiplication results output from the MAC tree 131 are directly provided to the VXE 140, thereby minimizing the time required for memory access and the resulting latency.

[0098] At this point, the mixed-precision fixed-point exponential adder 1211 implements two modes through post-processing after preprocessing each operator and performing two additions.

[0099] First, let's explain in high-precision mode. In the first addition, the first operator appends a 0 to the most significant bit of the least significant bit of the exponent of operand 1, while the second operator appends a 0 to the least significant bit of the exponent of operand 2. In the second addition, the first operator appends a 0 to the least significant bit of the most significant bit of the exponent of operand 1, while the second operator appends a 0 to the least significant bit of the most significant bit of the exponent of operand 2. The mixed-precision fixed-point exponent adder 1211 performs two additions and then adds the results of each operation to obtain the final exponent addition result.

[0100] The following is an explanation in high-performance mode. In the first addition, the first operator is the exponent of operand 1, and the second operator is the exponent of operand 2-(b). In the second addition, the first operator is the exponent of operand 1, and the second operator is the sign operator obtained by sign-extending the exponent of operand 2-(a). The mixed-precision fixed-point exponent adder 1211 truncates and concatenates the results after performing two additions, thus obtaining the two exponent addition result values.

[0101] Figure 16 This is a flowchart illustrating an example of an operation method for a MAC tree-based arithmetic unit according to an embodiment of the present invention. The MAC tree-based arithmetic unit can correspond to the previously described... Figure 10 The MAC tree 1000 is described. This MAC tree-based arithmetic unit can correspond to one of the multiple MAC tree-based arithmetic units included in a hardware accelerator (as an example, LPU100) used to accelerate artificial intelligence models.

[0102] At this point, based on the memory bandwidth provided to the hardware accelerator, at least one of the following can be determined: the number of MAC tree-based arithmetic units included in the hardware accelerator, and the number of multiple floating-point multipliers included in the MAC tree-based arithmetic units. Furthermore, the multiple MAC tree-based arithmetic units can perform matrix multiplication operations on at least one of the multiple partitions used to specifically implement the artificial intelligence model.

[0103] This allows for the efficient use of memory bandwidth to process the large amounts of data in generative large-scale artificial intelligence models.

[0104] In step 1610, the MAC tree-based arithmetic unit can utilize multiple floating-point multipliers connected in parallel to process the multiplication operations on data transferred from external memory. The external memory may include a high-bandwidth memory storing at least one partition and local memory units included in the hardware accelerator (e.g., LMU150). Additionally, each of the multiple floating-point multipliers may include a mixed-precision fixed-point exponent multiplier for adding the exponent (e.g., mixed-precision fixed-point exponent multiplier 1211) and a mixed-precision fixed-point mantissa multiplier for multiplying the mantissa (e.g., mixed-precision fixed-point mantissa multiplier 1212).

[0105] As previously explained, multiple floating-point multipliers can each operate in either a high-precision mode or a high-performance mode. As an example, when high-precision mode is selected, each of the multiple floating-point multipliers can process a multiplication between a first operand and a second operand of the same bit precision to calculate a first result value. Furthermore, when high-performance mode is selected, each of the multiple floating-point multipliers can simultaneously process a first multiplication between a third operand of the first bit precision and a fourth-first operand of the second bit precision, as well as a second multiplication between the third operand and a fourth-second operand of the second bit precision, simultaneously calculating a second result value for the first multiplication and a third result value for the second multiplication. As an example, the first bit precision can include 16 bits, and the second bit precision can include 8 bits.

[0106] In step 1620, the MAC tree-based arithmetic unit can utilize multiple first transformers connected to multiple floating-point multipliers to transform the results of the multiplication operations of each of the multiple floating-point multipliers from floating-point to fixed-point. Here, the multiple first transformers can correspond to those previously used... Figure 10 The FP2FXP converter 1040 is described.

[0107] In step 1630, the MAC tree-based arithmetic unit can utilize a fixed-point adder tree to process the addition of the results of transformations from multiple floating-point multipliers. Here, the fixed-point adder tree can correspond to the one previously processed by... Figure 10 The description of the fixed-point adder tree 1020.

[0108] In step 1640, the MAC tree-based arithmetic unit can use a fixed-point accumulator to accumulate the output of the fixed-point adder tree. Here, the fixed-point accumulator can correspond to... Figure 10 The description refers to the fixed-point accumulator 1030.

[0109] In step 1650, the MAC tree-based arithmetic unit can utilize the second transformer to transform the output of the fixed-point accumulator from fixed-point to floating-point. The second transformer can correspond to the previously used... Figure 10 The FXP2FP converter 1050 is described.

[0110] Thus, according to various embodiments of the present invention, an arithmetic unit for a hardware accelerator can be provided that maximizes memory bandwidth usage for accelerating generative large-scale language models that are difficult to parallelize and have large amounts of data. Additionally, an arithmetic unit capable of reducing the high hardware resource overhead of floating-point MAC tree structures can be provided. Furthermore, an arithmetic unit capable of effectively supporting two precision operations while minimizing hardware resource overhead can be provided.

[0111] Methods of implementing the invention

[0112] While embodiments of the present invention have been described above with reference to the accompanying drawings, those skilled in the art will understand that the invention can be implemented in other specific ways without altering its technical concept or essential features. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting.

Claims

1. A multiply-and-accumulation (MAC) tree-based operator, comprising: comprises: a plurality of floating point multipliers connected in parallel and processing multiplication operations on data transferred from an external memory; a plurality of first converters for converting outputs of the respective floating point multipliers into fixed point; a fixed point adder tree connected with the plurality of first converters and processing addition of multiplication results of the plurality of floating point multipliers; a fixed point accumulator accumulating an output of the fixed point adder tree; and a second converter for converting an output of the fixed point accumulator from fixed point into floating point, the MAC tree-based operator corresponds to one of a plurality of MAC tree-based operators included in a hardware accelerator for accelerating an artificial intelligence model, and at least one of a number of the MAC tree-based operators included in the hardware accelerator and a number of the plurality of floating point multipliers included in the MAC tree-based operator is determined according to a memory bandwidth provided to the hardware accelerator. 2.The MAC tree-based operator of claim 1, wherein the plurality of MAC tree-based operators perform a matrix multiplication operation for at least one of a plurality of partitions for implementing the artificial intelligence model. 3.The MAC tree-based operator of claim 2, wherein the external memory includes a high bandwidth memory in which the at least one partition is stored and a local storage unit included in the hardware accelerator. 4.The MAC tree-based operator of claim 1, wherein the plurality of floating point multipliers each include: a mixed precision fixed point exponent part adder for addition of an exponent part; and a mixed precision fixed point mantissa part multiplier for multiplication of a mantissa part. 5.The MAC tree-based operator of claim 1, wherein the plurality of floating point multipliers each calculate a first result value by processing multiplication between a first operand and a second operand having the same bit precision in a high precision mode.

6. A MAC (multiply-and-accumulation) tree-based operator, comprising: comprises: a plurality of floating point multipliers connected in parallel and processing multiplication operations on data transferred from an external memory; a plurality of first converters for converting outputs of the respective floating point multipliers into fixed point; a fixed point adder tree connected with the plurality of first converters and processing addition of multiplication results of the plurality of floating point multipliers; a fixed point accumulator accumulating an output of the fixed point adder tree; and a second converter for converting an output of the fixed point accumulator from fixed point into floating point, the plurality of floating point multipliers each calculate a first result value of a first multiplication between a first operand of a first bit precision and a second-1 operand of a second bit precision and a second result value of a second multiplication between the first operand and a second-2 operand of the second bit precision simultaneously when a high performance mode is selected. 7.The MAC tree-based operator of claim 6, wherein the first bit precision includes 16-bit precision, the second bit precision includes 8-bit precision.

8. A method of operating a multiply-and-accumulation (MAC) tree-based arithmetic unit, the method comprising: the MAC tree-based arithmetic unit including a plurality of floating-point multipliers connected in parallel, a plurality of first converters connected to the plurality of floating-point multipliers, a fixed-point adder tree, a fixed-point accumulator, and a second converter, the method including: processing, using the plurality of floating-point multipliers, multiplication operations on data transferred from an external memory; converting, using the plurality of first converters, results of the multiplication operations of the plurality of floating-point multipliers from floating-point to fixed-point; processing, using the fixed-point adder tree, addition of the converted results of the plurality of floating-point multipliers; accumulating, using the fixed-point accumulator, an output of the fixed-point adder tree; and converting, using the second converter, an output of the fixed-point accumulator from fixed-point to floating-point, the MAC tree-based arithmetic unit corresponding to one of a plurality of MAC tree-based arithmetic units included in a hardware accelerator for accelerating an artificial intelligence model, and at least one of a number of the MAC tree-based arithmetic units included in the hardware accelerator and a number of the plurality of floating-point multipliers included in the MAC tree-based arithmetic unit being determined based on a memory bandwidth provided to the hardware accelerator.

Images