Weight memory mapping method and system for streaming operation of massively generated artificial intelligence hardware

Abstract

Provided are a weight memory mapping method and a system for a streaming operation of massively generated artificial intelligence hardware. A weight memory mapping system according to an embodiment may comprise: a weight memory for storing a weight matrix for a pre-trained artificial intelligence model; an input register for storing a plurality of pieces of input data; a first hardware operator for processing a matrix multiplication operation between the plurality of pieces of input data and the weight matrix, and calculating a final sum in lane units during the matrix multiplication operation by reusing a partial sum of the matrix multiplication operation; and a second hardware operator for processing, in advance, a next matrix multiplication operation during the matrix multiplication operation by using the final sum.

Description

Weighted memory mapping method and system for streaming computation of large-scale generative artificial intelligence hardware

[0001] Embodiments of the present invention relate to a weighted memory mapping method and system for streaming operations of large-scale generative artificial intelligence hardware.

[0002] Most generative AI models adopt the Transformer architecture, demonstrating high performance. However, Transformers have limited computational parallelism, making latency a critical issue. To reduce hardware latency, an efficient weight mapping method is required.

[0003] A weight memory mapping method and system for streaming operations of large-scale generative artificial intelligence hardware can be provided.

[0004] The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0005] A weight memory mapping system is provided, comprising: a weight memory for storing a weight matrix for a pre-learned artificial intelligence model; an input register for storing a plurality of input data; a first hardware calculator for processing a matrix multiplication operation between the plurality of input data and the weight matrix, and calculating a final sum per lane during the matrix multiplication operation by reusing a partial sum of the matrix multiplication operation; and a second hardware calculator for pre-processing a next matrix multiplication operation during the matrix multiplication operation by using the final sum.

[0006] According to one side, the partial sum may include a result of matrix multiplication between one column of the weight matrix and the plurality of input data, and the final sum of the lane unit may include a value accumulated as a result of partial sums of matrix multiplication between each of the columns of the lane unit of the weight matrix and the plurality of input data.

[0007] According to another aspect, the first hardware operator may be characterized by including a plurality of MAC (multiply-and-accumulation) trees for processing the matrix multiplication operation in units of the lanes; a partial sum register for storing partial sums of the matrix multiplication operation; and a plurality of partial sum accumulators for accumulating partial sums of the partial sum registers to calculate a final sum in units of the lanes.

[0008] According to another aspect, the number of the plurality of MAC trees and the number of the plurality of partial sum accumulators may be characterized in that they each correspond to the number of lanes.

[0009] According to another aspect, the second hardware operator may be characterized in that it preprocesses the next operation using the final sum of the lane unit calculated by at least one partial sum accumulator among the plurality of partial sum accumulators.

[0010] According to another aspect, the artificial intelligence model may include a transformer model, the weight memory may store shared weight data for a token embedding operation for the transformer model and a Language Modeling (LM) head operation as the weight matrix, and the first hardware operator may be characterized in that, when performing the token embedding operation, it reads a weight of a specific column of the weight memory and processes the token embedding operation.

[0011] According to another aspect, the weight memory mapping system may further include a preprocessing routing unit that adjusts the positions of the values ​​of the weight matrix stored in the weight memory before the matrix multiplication operation so that the values ​​required for the next operation of the matrix multiplication operation are adjacent to each other.

[0012] According to another aspect, the artificial intelligence model may include a transformer model, and the following operation may be characterized by including a rotary embedding operation for the transformer model.

[0013] According to another aspect, the artificial intelligence model includes a transformer model, and the weight memory mapping system may further include a rotary embedding parameter processing unit that quantizes each of a sine value and a cosine value of one position of a rotary embedding parameter for a rotary embedding operation of the transformer model into a fixed-point 8-bit number and then stores a set of the quantized sine values ​​and cosine values ​​packed into 16 bits in the weight memory.

[0014] According to another aspect, the rotary embedding parameter processing unit may be characterized in that it stores the set in the weight memory or reads the set from the weight memory according to a value dividing the position of the set by the number of channels and a channel determined by a head number and an address determined by a remainder dividing the position of the set by the number of channels.

[0015] A weight memory mapping method is provided, comprising: a step of storing a weight matrix for a pre-learned artificial intelligence model in a weight memory; a step of storing a plurality of input data in an input register; a step of processing a matrix multiplication operation between the plurality of input data and the weight matrix through a first hardware operator, and calculating a final sum per lane during the matrix multiplication operation by reusing a partial sum of the matrix multiplication operation; and a step of pre-processing a next matrix multiplication operation during the matrix multiplication operation by using the final sum through a second hardware operator.

[0016] Specific details of other embodiments are included in the detailed description and drawings.

[0017] A weight memory mapping method and system for streaming operations of large-scale generative artificial intelligence hardware can be provided.

[0018] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

[0019] FIG. 1 is a diagram illustrating an example of the structure of a latency processing unit according to an embodiment of the present invention.

[0020] FIGS. 2 to 5 are drawings illustrating examples of implementation models of LPUs according to embodiments of the present invention.

[0021] FIG. 6 is a diagram for explaining weight matrix data mapping of a high-bandwidth memory for matrix multiplication operation of a latency processing unit according to an embodiment of the present invention.

[0022] FIG. 7 is a diagram for explaining a high-bandwidth memory interface included in a latency processing unit according to one embodiment of the present invention.

[0023] FIG. 8 is a diagram illustrating a reconfigurable multi-function operation unit included in a latency processing unit according to one embodiment of the present invention.

[0024] FIG. 9 is a diagram for explaining the configuration of an address-based non-sequential multi-unit scheduler included in a latency processing unit according to one embodiment of the present invention.

[0025] FIG. 10 and FIG. 11 are drawings for explaining the concepts of input reuse and partial sum (output) reuse in one embodiment of the present invention.

[0026] FIG. 12 is a diagram illustrating an example of a general appearance of a weighted memory mapping system based on partial sum reuse according to one embodiment of the present invention.

[0027] FIG. 13 is a diagram illustrating an example of data deduplication in one embodiment of the present invention.

[0028] FIG. 14 is a diagram illustrating an example of processing a matrix multiplication operation in one embodiment of the present invention.

[0029] FIG. 15 and FIG. 16 are drawings illustrating examples of a process for processing a rotary embedding operation in one embodiment of the present invention.

[0030] FIG. 17 is a block diagram illustrating an example of the internal configuration of a weighted memory mapping system according to one embodiment of the present invention.

[0031] FIG. 18 is a flowchart illustrating an example of a weighted memory mapping method according to one embodiment of the present invention.

[0032] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms. These embodiments are provided only to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Like reference numerals designate like elements throughout the specification.

[0033] When one component is referred to as being "connected to" or "coupled to" another component, it includes both cases where it is directly connected or coupled to the other component, or cases where there is another component intervening therebetween. Conversely, when one component is referred to as being "directly connected to" or "directly coupled to" another component, it indicates that there is no other component intervening therebetween. "And / or" includes each and any combination of one or more of the mentioned items.

[0034] The terminology used herein is for the purpose of describing embodiments only and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise. As used herein, the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, operations, and / or elements.

[0035] Although terms like "first" and "second" are used to describe various components, these components are not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it should be understood that a "first" component referred to below may also be a "second" component within the technical scope of the present invention.

[0036] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in their common sense to those of ordinary skill in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0037] FIG. 1 is a diagram illustrating an example of the structure of a latency processing unit according to an embodiment of the present invention.

[0038] Referring to FIG. 1, an LPU (Latency Processing Unit) 100 according to an embodiment of the present invention may include an SMA (Streamlined Memory Access) 110, an OIU (Operand Issue Unit) 120, an SXE (Streamlined eXecution Engine) 130, a VXE (Vector eXecution Engine) 140, an LMU (Local Memory Unit) 150, an ISU (Instruction Scheduling Unit) 160, a PCIe (Peripheral Component Interconnect express) Interface 170, and a P2P (Peer to Peer) Interface 180.

[0039] The SMA (110) may be a specialized DMA (Direct Memory Access). For example, the SMA (110) may connect all channels (e.g., 32) of the HBM (200) to the execution engine (e.g., the SEE (130)) to transfer half-precision floating point (FP16) data at maximum bandwidth. The SMA (110) may be designed with a deep FIFO (First In First Out) to transfer sequential memory requests based on preloaded memory (MEM) instructions. Hardware-based memory mapping may reduce latency by eliminating matrix modification or transposition operations. Therefore, the SMA (110) may stream data received in a maximum burst size to the execution engine with minimal latency. The SMA (110) may also efficiently perform matrix transposes using a strobe signal. Streaming data may contain parameters for vector matrix operations (e.g., weights, biases) and other vector-related operations (e.g., gamma / beta, embedding).

[0040] The OIU (120) can coordinate data streamed from the SMA (110) (e.g., a first operand) with input from on-chip memory (e.g., a second operand) before issuing it to the execution engine. Based on the execute (EXE) instruction, the OIU (120) can generate microcode that configures the execution engine and determines the target engine of the operand. In addition, the OIU (120) can be provided with a reusable buffer to eliminate read latency of static operands (e.g., input vectors) and an asymmetric buffer to hold vectorized data used as scalars (e.g., biases). Therefore, appropriate operands are almost always prefetched and ready to be issued to the execution engine immediately.

[0041] The SXE (130) is the main computing hardware of the LPU (100), and can be designed to maximize the incoming bandwidth to execute vector-matrix multiplication (V·M) such as attention, 1D convolution, and feedforward network. The SXE (130) can include a number of MAC (multiply-and-accumulation) trees (MAC Trees, 131) that can match the reception bandwidth and the computation bandwidth of the HBM (200). For example, if 1024 elements are received from the HBM (200) per cycle, the reception bandwidth and the computation bandwidth can be matched through 16 MAC trees (131) each having 64 input vectors. In addition, the MAC tree (131) with 64 input vectors can be composed of 64 multipliers and 63 adders.

[0042] A plurality of MAC trees (131) can perform matrix multiplication operations and can be connected by channel through the HBM (200), which is an external memory and a high-bandwidth memory, and the SMA (110). Specifically, one of the plurality of MAC trees (131) is connected to the HBM (200) through one channel, thereby maximizing the transmission bandwidth between the LPU (100) and the HBM (200), and performing the matrix multiplication operations required for the ultra-large artificial intelligence model without a bottleneck. Therefore, the number of the plurality of MAC trees (131) and the number of memory channels of the HBM (200) can be configured to be the same.

[0043] The matrix multiplication operation results of multiple MAC trees (131) can be provided to the VXE (140). The VXE (140) can be implemented using a custom low-latency ALU (Arithmetic Logic Unit) and can execute vector operations such as token embedding, softmax, normalization, and residual operations. Since these vector operations occur relatively infrequently, the fan-in from the OIU (120) to this path can be adjusted to reduce hardware resources with negligible performance loss. The VXE (140) can receive the operation results of multiple MAC trees (131) and receive activation values ​​from the LMU (150) to perform subsequent operations. The VXE (140) can be configured to include various combinations of operators by including multiple multi-function operation datapaths.

[0044] The LMU (150) can transmit activation values ​​to multiple MAC trees (131) and VXEs (140). At this time, the LMU (150) can copy and transmit the activation values ​​in order to transmit the same activation values ​​to multiple MAC trees (131). In addition, the LMU (150) can store the result values ​​calculated by the multiple MAC trees (131) and VXEs (140). In other words, the LMU (150) can function as an internal buffer corresponding to the HBM (200) within the LPU (100). At this time, the LPU (100) can store activation values ​​or model parameters with a high reuse rate in the LMU (150) and store weights with a low reuse rate in the HBM (200) in the matrix multiplication operation. The LMU (150) can be implemented as a 4MB multi-bank register file with scalar vector separation for fast, high-bandwidth access to input, output, and intermediate data. Additionally, the LMU (150) may be multi-ported to support simultaneous reads and writes in the write-back phase of the OIU (120) and the execution engine.

[0045] The ISU (160) can control the entire execution flow of the LPU (100). The ISU (160) can utilize the PIC (Parallel Instruction Chaining) method, and by using instruction chaining, dependent instructions can be executed sequentially. The PIC separates instructions requiring independent hardware into groups of dependent instructions (e.g., memory (MEM) instructions, execution (EXE) instructions, network (NET) instructions), so that all instructions can be executed in parallel with the instruction chain of each group, thereby achieving low control overhead and latency savings. The ISU (160) can also update control registers (e.g., token and layer number) for engine execution. The internal scheduler supports out-of-order execution of the SXE (130) and VXE (140) to maximize hardware utilization, and a powerful scoreboard can be designed to handle data risks. For example, the ISU (160) can schedule multiple MAC trees (131) and VXEs (140) to perform operations simultaneously. In addition, the ISU (160) can improve operation throughput and delay time by pre-executing instructions without dependencies to maximize parallel operations, thereby minimizing the idle time of each operation unit and memory access unit.

[0046] The LPU (100) can be connected to a host computer via a PCIe interface (170), and can receive commands required for the operation of the LPU (100) and input values ​​and weights of a large-scale artificial intelligence model from the host computer, perform calculations, and then transmit the results to the host computer.

[0047] The LPU (100) can be scaled out into a cluster of multiple LPUs connected via a P2P interface (180). The expanded cluster structure can further enhance the acceleration of computation of ultra-large artificial intelligence models.

[0048] FIGS. 2 to 5 are diagrams illustrating examples of implementation models of LPUs according to embodiments of the present invention. The embodiment of FIG. 1 previously described an example of an implementation model using external memory of HBM (200). Instead of HBM (200), DDR (Double Data Rate) may be used as the external memory. In this case, since it is difficult to store a large model in a single device, it may be divided into multiple partitions and stored in external memories for multiple devices (multiple LPUs) for each partition. In this case, synchronization between multiple devices may be required for inference of the large model.

[0049] The embodiment of FIG. 2 illustrates a plurality of external memories (320) storing a plurality of partitions (310) of a large model, similar to the embodiment described above in FIG. 1, and a plurality of LPUs (330) connected in parallel to the plurality of external memories (320). One LPU can be implemented in one FPGA (Field Programmable Gate Array), and one partition can be connected in parallel to one FPGA. The transformer structure includes multi-head attention, layer normalization, feed forward, etc. in the decoder layer, and the multi-head attention and feed forward can be model-parallelized. In this case, when the multi-head attention is terminated, one embedding vector can be output as a result. Since one device only has a portion of the embedding vector, synchronization may be required because multiple devices need to share each embedding vector in order to proceed to the next operation. At this time, considering scalability, a single LPU may be implemented with multiple external memories (e.g., two or four). For example, the embodiment of FIG. 1 illustrates an example in which two HBMs (200), each storing one partition, are used.

[0050] The embodiment of Fig. 3 illustrates an example of a PIM (Processing-in-Memory) model, in which one LPU is implemented as a PIM chip, and both the partition and the LPU computation unit are integrated into a single chip. The embodiment of Fig. 3 illustrates a plurality of LPUs (410), a plurality of partitions (310), and a plurality of LPU computation units (420), each of which can be implemented as a PIM chip. In this case, each of the plurality of LPUs (410) can include one partition and one LPU computation unit.

[0051] The embodiment of Fig. 4 illustrates an example of a Processing-near-Memory (PNM) model. It may be difficult to include a configuration for processing all LPU operations within a single PIM chip. The embodiment of Fig. 4 illustrates a model in which a plurality of partitions (310) are stored within a plurality of memory chips (510), and an LPU operation unit (521) for LPU operations is included within a buffer chip (520) such as a PNM chip.

[0052] The embodiment of Fig. 5 illustrates an example of a model combining PIM and PNM. For example, multiple partitions (310) may be stored in multiple memory chips (610). In addition, PIM-style LPU operation units (611) may be implemented as accumulation units, such as MAC trees, in each of the multiple memory chips (610). In this case, the LPU operation units (621) for the remaining high-level operations of the LPU may be implemented in the PNM manner in the buffer chip (620).

[0053] FIG. 6 is a diagram for explaining weight matrix data mapping of a high-bandwidth memory for matrix multiplication operation of a latency processing unit according to an embodiment of the present invention.

[0054] Referring to FIG. 6, the LPU (100) according to the present embodiment can store weight matrix data mapped to a high-bandwidth memory (610) such as HBM (200) so that the weight data can be retrieved without accessing another memory channel when performing matrix multiplication operation on each MAC tree, since the number of MAC trees constituting multiple MAC trees (131) and the number of memory channels of SMA (110) are the same.

[0055] Specifically, weight matrix data may be stored in the high-bandwidth memory (610) so that it is mapped to each channel (620-n) in the column direction (D1) of the weight matrix in the number of multiple MAC trees (131). Since the column direction in the weight matrix allows matrix multiplication operations to be performed in parallel, the multiple MAC trees (131) can read column direction data from their respective allocated memory channels (620-n) and perform matrix multiplication operations.

[0056] Next, multiple MAC trees (131) can map the weight matrix data so that they can accumulate in the row direction (D2) of the weight matrix to complete the final operation result. The number of row data mapped at once can be determined by the bandwidth of the high-bandwidth memory (610), and this can be determined by the size of the tiles that multiple MAC trees (131) can process at once.

[0057] FIG. 7 is a diagram for explaining a high-bandwidth memory interface included in a latency processing unit according to one embodiment of the present invention.

[0058] Referring to FIG. 7, the SMA (110) can connect the LMU (150), multiple MAC trees (131), and high-bandwidth memory (610). The SMA (110) is not connected to other computational units of the LPU (100), and thus, the high-bandwidth memory interface can be minimized in terms of hardware resources.

[0059] Multiple MAC trees (131) and memory channels (620-n) can be connected in a one-to-one correspondence, respectively. That is, multiple MAC trees (131) do not need to access channels other than the directly assigned channels, and thus can perform matrix multiplication operations without using complex interfaces such as cross-bars, which consume a lot of resources and have a high delay time.

[0060] SMA (110) can configure only a read interface for loading weight matrix data stored in high-bandwidth memory (610) of multiple MAC trees (131). In other words, as described later, since the result of the operation is stored in the high-bandwidth memory (610) through LMU (150), a write interface for the high-bandwidth memory (610) of multiple MAC trees (131) is not configured, and hardware resources can be reduced accordingly.

[0061] In contrast, the SMA (110) can only form a write interface between the LMU (150) and the high-bandwidth memory (610). The computational results stored in the LMU (150) as an internal buffer through the SMA (110) can be transmitted to be written to the high-bandwidth memory (610), and the memory channel of the writing target can be selected using the demultiplexer (710).

[0062] FIG. 8 is a diagram illustrating a reconfigurable multi-function operation unit included in a latency processing unit according to one embodiment of the present invention.

[0063] Referring to FIG. 8, the VXE (140) may include a plurality of multi-function operation data paths (810, 820), and the plurality of multi-function operation data paths (810, 820) may be connected to an operator / result value chain network (830) to form various combinations of operators.

[0064] As illustrated in FIG. 8, a plurality of multi-functional operation datapaths (810, 820) may include various operation units required for, for example, a lookup table-based nonlinear activation function, a masking operation, etc. However, the configuration of the operation units of the reconfigurable multi-functional operation datapaths (810, 820) illustrated in FIG. 8 is exemplary, and it goes without saying that additional operation units required for large-scale model operation may be included in the multi-functional operation datapaths (810, 820). The result calculated by the VXE (140) may be transmitted to the LMU (150).

[0065] FIG. 9 is a diagram for explaining the configuration of an address-based non-sequential multi-unit scheduler included in a latency processing unit according to one embodiment of the present invention.

[0066] Referring to FIG. 9, the ISU (160) as an address-based out-of-order multi-unit scheduler included in the LPU (100) according to the present embodiment may include an address-based instruction dependency determination and scheduling controller (910), a plurality of instruction issue controllers (921, 922, 923, 924), a multi-bank buffer address status table (950), an instruction buffer (960) and result address status update logic (970), and a multi-unit instruction dispatcher (980).

[0067] The ISU (160) can operate each operation unit and data movement unit simultaneously through the address-based instruction dependency judgment and scheduling controller (910) and multiple instruction issue controllers (921, 922, 923, 924). At this time, the ISU (160) can change the status of the operator address and result address of the multi-bank buffer address status table (950) for the instruction performed in each operation unit to 1.

[0068] The multi-bank buffer address status table (950) can change the status of the result address of a completed instruction to 0 through the result address status update logic (970).

[0069] The address-based instruction dependency determination and scheduling controller (910) can determine the dependency between the instruction to be executed and the instruction being executed, as well as the dependency between the instructions to be executed, by referencing the states of the addresses through the multi-bank buffer address state table (970). This enables preprocessing of instructions without dependencies, thereby minimizing the idle time of each operation unit and data movement unit.

[0070] The address-based instruction dependency determination and scheduling controller (910) included in the ISU (160) can load and process instructions from the instruction buffer (960). At this time, the address-based instruction dependency determination and scheduling controller (910) performs a loop instruction, and decodes other instructions to distinguish the instructions and transmit them to the device-to-device instruction issue controller (921), the direct memory access instruction issue controller (922), the MAC tree instruction issue controller (923), and the reconfigurable multi-function operation unit instruction issue controller (924) through the multi-unit instruction dispatcher (980).

[0071] The ISU (160) receives and stores commands of the LPU (100) from the host computer through the PCIe interface (170), and can store the current status of the LPU (100) in a status register. The host computer can check the status register through the PCIe interface (170).

[0072] Meanwhile, AI weights are learned in advance and are retrieved from external memory for computation during AI model inference. Therefore, if weights do not require runtime processing, they can be optimized as much as possible before being passed on to the hardware that processes AI model inference. Embodiments of the present invention provide a memory mapping method and system for large-scale generation AI hardware.

[0073] AI hardware must determine which inputs, weights, and outputs to reuse for computation. For example, Transformer-based models cannot reuse weights during inference, so they must decide which inputs and outputs to reuse to reduce hardware latency and ensure efficient computation.

[0074] Additionally, it's important to determine if there are any inefficient operations that can be performed at runtime and ensure that these can be handled during mapping. It's also important to utilize parameter distributions or their maximum and / or minimum values ​​to perform quantization.

[0075] The memory mapping method and system according to embodiments of the present invention are implemented with a focus on reducing hardware latency. The structure of a transformer, in which the output of the current operation is connected to the input of the next operation, cannot use methods that increase the utilization rate of the operator, such as out-of-order. Therefore, a method that quickly produces the final result value so that the next operation can be processed more quickly must be adopted. In embodiments of the present invention, a method that quickly produces the final sum based on an output reuse memory mapping method is used so that the next operation can be performed and the efficiency of the operator can be increased.

[0076] We also propose a mapping methodology for using the same weights in different ways, such as in token embedding and LM (Language Modeling) head operations. LM head operations involve matrix multiplication, while token embedding reads vectors from specific addresses, requiring different mapping methods. However, since the weight data itself is the same, mapping them to external memory differently would result in duplicate data being written, making external memory inefficient. To address this, we propose a method that utilizes token embedding in the matrix multiplication mapping method.

[0077] Furthermore, one embodiment of the present invention may provide a dedicated mapping method to reduce hardware routing complexity that may arise during rotary embedding. The weighted memory mapping method and system according to this embodiment eliminates routing in advance during weighted memory mapping, thereby reducing associated latency in runtime hardware.

[0078] Figures 10 and 11 are diagrams illustrating the concepts of input reuse and partial sum (output) reuse in one embodiment of the present invention. When assigning weights, one can decide whether to load the input once and reuse it, or store the output once and reuse it. When reusing inputs, as in Figure 10, the number of partial sums increases by the number of rows, requiring the use of as many FIFO (First In First Out) resources as the number of rows. Conversely, when reusing partial sums, as in Figure 11, resources must be used to store input data in registers equal to the number of columns. In other words, regardless of which method is used, the hardware resource usage is the same because the rows and columns in a transformer are generally the same size. However, there is a difference in the timing of the final result. When reusing inputs, the final sums are calculated sequentially at the end, whereas reusing partial sums allows for faster final calculation. Since the goal in embodiments of the present invention is to reduce latency, it is advantageous for the final sum to be calculated as quickly as possible. For example, if there is an operator that does not participate in the operation, the next operation can be performed in advance using the final sum that was calculated first.

[0079] FIG. 12 is a diagram illustrating an example of a general view of a weight memory mapping system based on partial sum reuse according to an embodiment of the present invention. The weight memory mapping system (1200) according to the present embodiment may include a weight memory (1210), an input register (1220), and a hardware operator (1230). Here, the hardware operator (1230) may correspond to the LPU (100) described above. In addition, the weight memory (1210) may correspond to the HBM (200) as a memory for storing the weight matrix of a pre-trained artificial intelligence model. As described above, a register is required to store input data equal to the number of columns (the number of columns of the weight matrix) for partial sum reuse, and the input register (1220) may correspond to a register for storing such input data.

[0080] In the embodiment of Fig. 12, an example is shown in which the number of lanes is 4. For this purpose, the hardware operator (1230) may include 4 MAC trees (1231) and 4 partial sum accumulators (1232), and may further include a partial sum register (1233) for storing the partial sums. The hardware operator (1230) may read data from the weight memory (1210) with a streamline, calculate partial sums directly through the MAC tree (1231), and store them in the partial sum register (1233), and accumulate the partial sums through the partial sum accumulator (1232) so that the final sums can be quickly calculated in the middle of the matrix multiplication operation. In this way, since the final sums are quickly calculated in the middle of the calculation rather than at the end as shown in Fig. 11, if there is a second hardware calculator (1240) that does not participate in the calculation, the second hardware calculator (1240) can first receive the calculated final sum from the partial sum accumulator (1232) and perform the next calculation in advance using the provided final sum, thereby reducing the latency of the hardware. For example, the second hardware calculator (1240) can correspond to another LPU.

[0081] FIG. 13 is a diagram illustrating an example of data deduplication in one embodiment of the present invention. In a transformer, there are token embedding (1310) operations and LM head (1320) operations. In the embodiment of FIG. 13, only the token embedding (1310) operations and the LM head (1320) operations are shown because the transformer itself is already well known. These token embedding (1310) operations and LM head (1320) operations share weight data. In other words, it is possible to store it in only one place in the HBM (200), read the vector at the corresponding address during the token embedding (1310) operation, and perform a matrix multiplication operation using the weights during the LM head (1320) operation. However, when performing different operations with the same data in this way, the GPU may view it as completely different data and store the weight data redundantly in the HBM (200), which results in waste of the HBM (200).

[0082] In this embodiment, token embedding (1310) can also be processed in memory mapping for matrix multiplication. As shown in Fig. 13, when memory mapping is processed using the partial sum reuse method, it can be seen that all data is gathered in a single line (one column). Therefore, when performing token embedding (1310), the required weight data can be easily read by simply determining which column of which lane to read the weight data from. In this case, although the bandwidth of the HBM (200) is not fully utilized, memory waste can be reduced because data does not need to be stored redundantly.

[0083] FIG. 14 is a diagram illustrating an example of processing a matrix multiplication operation in one embodiment of the present invention. This embodiment describes an example of a rotary embedding operation, but is not limited thereto. In the case of rotary embedding, after performing a matrix multiplication operation, an operation is performed between values ​​0 and 64 in the weight matrix, and an operation is performed between values ​​1 and 65, thereby processing operations between values ​​that are far apart in the weight matrix. When implementing this in hardware, not only does it congest the hardware routing, but if the values ​​to be operated on are too far apart, streaming processing may be difficult.

[0084] Therefore, by repositioning the values ​​in the weight matrix at the compiler level, operations between adjacent data can be processed without complex hardware routing. While preprocessing routing occurs during the preprocessing process, hardware routing can be eliminated at runtime, resulting in hardware benefits.

[0085] Also, even if the order of the values ​​in the weight matrix is ​​arbitrarily changed like this, there is no routing overhead at runtime because the next operation is an inner product operation that does not depend on the order.

[0086] FIGS. 15 and 16 are diagrams illustrating an example of a process for processing a rotary embedding operation according to an embodiment of the present invention. The rotary embedding operation performs sine (sin) and cosine (cos) operations. However, since the results of the sine and cosine operations range from -1 to 1, it is not efficient to use a general 16-bit floating-point number (float16). It was confirmed that there is no problem with the operation accuracy even if quantization is performed with a fixed-point number of 8 bits using a source code level simulator. Accordingly, in this embodiment, by packing two 8-bits and grouping the sine and cosine values ​​into a single set of 16-bits, the memory usage can be reduced. In addition, the sine and cosine values ​​exist as many as the number of heads and positions. These can be stored in a memory such as HBM (200) and read by determining the memory channel and address.

[0087] FIG. 17 is a block diagram illustrating an example of the internal configuration of a weighted memory mapping system according to an embodiment of the present invention, and FIG. 18 is a flowchart illustrating an example of a weighted memory mapping method according to an embodiment of the present invention.

[0088] A weight memory mapping system (1700) according to the present embodiment may include a weight memory (1710), an input register (1720), a preprocessing routing unit (1730), a rotary embedding parameter processing unit (1740), and a plurality of hardware calculators (1750). Depending on the embodiment, the weight memory (1710) may be a set of weight memories for each of the plurality of hardware calculators (1750). For example, each of the weight memories may correspond to the HBM (200) described above. In addition, depending on the embodiment, at least some of the components included in the weight memory mapping system (1700) may be implemented as physically different hardware devices. For example, each of the plurality of hardware calculators (1750) may be implemented as a separate hardware device, and the weight memory (1710), the input register (1720), the preprocessing routing unit (1730), and the rotary embedding parameter processing unit (1740) may be implemented as another hardware device. Each hardware device may be connected to a network and communicate with one another. Furthermore, at least some components may be excluded from the weight memory mapping system (1700), or additional components may be included. For example, for an embodiment that does not process rotary embedding operations for a transformer model, the rotary embedding parameter processing unit (1740) may be excluded from the weight memory mapping system (1700). As another example, the weight memory mapping system (1700) may further include an input / output interface for connection to an input / output device, or may further include a communication interface.

[0089] In step (1810), the weight memory mapping system (1700) may store a weight matrix for a pre-trained artificial intelligence model in the weight memory (1710). Here, the weight matrix stored in the weight memory (1710) may be a part of the entire weight matrix of the artificial intelligence model. For example, when a large artificial intelligence model is divided into multiple partitions and processed through multiple LPUs, weight memories may exist for each of the multiple LPUs, and the weight matrix to be processed by each LPU may be stored in the weight memory corresponding to the corresponding LPU. In the present embodiment, the weight memory (1710) may store a weight matrix for a first hardware operator (1751), which is one of multiple hardware operators (1750).

[0090] In step (1820), the weight memory mapping system (1700) can adjust the positions of the values ​​of the weight matrix stored in the weight memory (1710) through the preprocessing routing unit (1730) so that the values ​​required for the next operation of the matrix multiplication operation are adjacent to each other. For example, if the artificial intelligence model is a transformer model, the next operation may include, but is not limited to, a rotary embedding operation for the transformer model. If an operation after the matrix multiplication operation requires an operation between scattered values, the weight memory mapping system (1700) can adjust the positions of the values ​​of the weight matrix in advance for the operation. Depending on the embodiment, step (1820) may be omitted.

[0091] In step (1830), the weight memory mapping system (1700) can quantize each of the sine and cosine values ​​of one position of the rotary embedding parameter for the rotary embedding operation of the transformer model into fixed-point 8-bits through the rotary embedding parameter processing unit (1740), and then store the quantized sine and cosine values ​​as a set packed into 16 bits in the weight memory (1710). In this case, the rotary embedding parameter processing unit (1740) can store the set in the weight memory (1710) or read the set from the weight memory (1710) according to the address determined by the value obtained by dividing the position of the set by the number of channels and the channel determined by the head number and the remainder obtained by dividing the position of the set by the number of channels. This step (1830) is for the rotary embedding operation of the transformer model and may be omitted depending on the embodiment. In addition, the order of steps (1820) and (1830) may be changed.

[0092] In step (1840), the weight memory mapping system (1700) can store multiple input data in the input register (1720). In order to reuse partial sums, matrix multiplication operations between multiple input data and each column of the weight matrix can occur continuously. Accordingly, since multiple input data are repeatedly used as many times as the number of columns of the weight matrix, the weight memory mapping system (1700) can store these multiple input data in the input register (1720).

[0093] In step (1850), the weight memory mapping system (1700) processes a matrix multiplication operation between a plurality of input data and a weight matrix through a first hardware operator (1751), and can calculate the final sum per lane during the progress of the matrix multiplication operation by reusing the partial sum of the matrix multiplication operation. Here, the first hardware operator (1751) may correspond to the first hardware operator (1230) described through FIG. 12. In other words, the first hardware operator (1751) may include a plurality of MAC trees (1231) that process the matrix multiplication operation per lane, a partial sum register (1233) that stores the partial sum of the matrix multiplication operation, and a plurality of partial sum accumulators (1232) that accumulate the partial sums of the partial sum registers (1233) to calculate the final sum per lane. At this time, the number of the plurality of MAC trees (1231) and the number of the plurality of partial sum accumulators (1232) may each correspond to the number of lanes. For example, if the number of lanes is 4, the number of the plurality of MAC trees (1231) and the number of the plurality of partial sum accumulators (1232) may each be 4. Meanwhile, the partial sum may include the result of matrix multiplication between one column of the weight matrix and the plurality of input data. In addition, the final sum in lane units may include a value in which partial sums are accumulated as the result of matrix multiplication between each of the columns of the lane units of the weight matrix and the plurality of input data.

[0094] In step (1860), the weight memory mapping system (1700) can preprocess the next matrix multiplication operation during the progress of the matrix multiplication operation by using the final sum through the second hardware operator (1752). The second hardware operator (1752) may be an operator that is not currently participating in the operation among the plurality of hardware operators (1750). In other words, the weight memory mapping system (1700) can preprocess the next operation by transferring the quickly calculated final sum to the second hardware operator (1752), which is an operator that is not currently participating in the operation. At this time, the second hardware operator (1752) can preprocess the next operation by using the final sum of the lane unit calculated by at least one partial sum accumulator among the plurality of partial sum accumulators (1232).

[0095] In addition, according to an embodiment, when the artificial intelligence model is a transformer model, the token embedding operation and the LM head operation for the transformer model can share weight data. To this end, in step (1810), the weight memory mapping system (1700) can store the shared weight data as a weight matrix in the weight memory (1710). In addition, in step (1850), the weight memory mapping system (1700) can read the weight of a specific column of the weight memory and process the token embedding operation through the first hardware operator (1751) during the token embedding operation. For example, it has been described above that vector operations such as token embedding, softmax, normalization, and residual operations can be executed in the VXE (140) included in the LPU (100).

[0096] In this way, according to embodiments of the present invention, a weighted memory mapping method and system for streaming operations of large-scale generation artificial intelligence hardware can be provided.

[0097] Although the embodiments of the present invention have been described with reference to the attached drawings, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without altering the technical concept or essential features thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

Claims

1. In a weighted memory mapping system, Weight memory that stores weight matrices for pre-trained artificial intelligence models; An input register that stores multiple input data; A first hardware calculator that processes a matrix multiplication operation between the plurality of input data and the weight matrix, and calculates a final sum per lane during the matrix multiplication operation by reusing a partial sum of the matrix multiplication operation; and A second hardware operator that preprocesses the next matrix multiplication operation while the matrix multiplication operation is in progress using the final sum above. A weighted memory mapping system including.

2. In paragraph 1, The above partial sum includes the result of matrix multiplication between one column of the weight matrix and the plurality of input data, The final sum of the above lane units includes a value accumulated as a result of the matrix multiplication between each of the lane units of the above weight matrix and the plurality of input data. A weighted memory mapping system featuring:

3. In paragraph 1, The above first hardware operator is, A plurality of MAC (multiply-and-accumulation) trees that process the matrix multiplication operation in the above lane units; A partial sum register storing the partial sum of the above matrix multiplication operation; and A plurality of partial sum accumulators that accumulate partial sums of the partial sum registers and calculate the final sum of the lane unit. including A weighted memory mapping system featuring:

4. In paragraph 3, A weighted memory mapping system, characterized in that the number of the plurality of MAC trees and the number of the plurality of partial sum accumulators each correspond to the number of lanes.

5. In paragraph 3, A weighted memory mapping system, characterized in that the second hardware calculator preprocesses the next matrix multiplication operation using the final sum of the lane unit calculated by at least one partial sum accumulator among the plurality of partial sum accumulators.

6. In paragraph 1, The above artificial intelligence model includes a transformer model, The above weight memory stores shared weight data for token embedding operations and LM (Language Modeling) head operations for the transformer model as the weight matrix, The above first hardware operator reads the weight of a specific column of the weight memory and processes the token embedding operation during the token embedding operation. A weighted memory mapping system featuring:

7. In paragraph 1, A preprocessing routing unit that adjusts the positions of the values ​​of the weight matrix stored in the weight memory before the matrix multiplication operation so that the values ​​required for the next operation of the matrix multiplication operation are adjacent to each other. A weighted memory mapping system that further includes:

8. In paragraph 7, The above artificial intelligence model includes a transformer model, A weighted memory mapping system, characterized in that the above next operation includes a rotary embedding operation for the transformer model.

9. In paragraph 1, The above artificial intelligence model includes a transformer model, A rotary embedding parameter processing unit that quantizes each of the sine and cosine values ​​of one position of the rotary embedding parameter for the rotary embedding operation of the above transformer model into fixed-point 8-bits and then stores the set of quantized sine and cosine values ​​packed into 16 bits in the weight memory. A weighted memory mapping system that further includes:

10. In paragraph 9, A weight memory mapping system characterized in that the rotary embedding parameter processing unit stores the set in the weight memory or reads the set from the weight memory according to an address determined by a value obtained by dividing the position of the set by the number of channels and a channel determined by a head number and a remainder obtained by dividing the position of the set by the number of channels.

11. In the weighted memory mapping method, A step of storing a weight matrix for a pre-trained artificial intelligence model in a weight memory; A step of storing multiple input data in an input register; A step of processing a matrix multiplication operation between the plurality of input data and the weight matrix through a first hardware operator, and calculating a final sum per lane during the progress of the matrix multiplication operation by reusing a partial sum of the matrix multiplication operation; and A step of preprocessing the next matrix multiplication operation during the matrix multiplication operation using the final sum through a second hardware operator. A weighted memory mapping method including:

12. In paragraph 11, The above partial sum includes the result of matrix multiplication between one column of the weight matrix and the plurality of input data, The final sum of the above lane units includes a value accumulated as a result of the matrix multiplication between each of the lane units of the above weight matrix and the plurality of input data. A weighted memory mapping method characterized by .

13. In paragraph 11, The above first hardware operator is, A plurality of MAC (multiply-and-accumulation) trees that process the matrix multiplication operation in the above lane units; A partial sum register storing the partial sum of the above matrix multiplication operation; and A plurality of partial sum accumulators that accumulate partial sums of the partial sum registers and calculate the final sum of the lane unit. including A weighted memory mapping method characterized by .

14. In paragraph 11, The above artificial intelligence model includes a transformer model, The above saving step is, Store the shared weight data for the token embedding operation and the LM head operation for the above transformer model in the weight memory as the weight matrix, The steps for calculating the final sum above are: When performing the token embedding operation through the first hardware operator, the weight of a specific column of the weight memory is read and the token embedding operation is processed. A weighted memory mapping method characterized by .

15. In paragraph 11, A step of adjusting the positions of the values ​​of the weight matrix stored in the weight memory before the matrix multiplication operation so that the values ​​required for the next operation of the matrix multiplication operation are adjacent to each other. A weighted memory mapping method including more.

16. In paragraph 11, The above artificial intelligence model includes a transformer model, A step of quantizing each of the sine and cosine values ​​of one position of the rotary embedding parameter for the rotary embedding operation of the above transformer model into fixed-point 8-bits and then storing a set of the quantized sine and cosine values ​​packed into 16 bits in the weight memory. A weighted memory mapping method including more.

Images