Memory mapping method and system for rotary position embedding operation

Abstract

Provided are a memory mapping method and system for a rotary position embedding operation. The memory mapping system according to one embodiment may include: a preprocessing unit for adjusting the positions of the values in a weight matrix for a pretrained artificial intelligence model such that the values of the elements required in a rotary position embedding operation, which is the next operation after a matrix multiplication operation, are adjacent to one another; a weight memory for storing the weight matrix with the positions of the values adjusted; and a hardware operator for generating a query vector or a key vector by processing the matrix multiplication operation of an input vector and the weight matrix and for processing the rotary position embedding operation of the query vector or the key vector by using the values of the elements adjacent to one another.

Description

Memory mapping method and system for rotary position embedding operations

[0001] Embodiments of the present invention relate to a memory mapping method and system for rotary position embedding operations to minimize inference delay speed and optimize computational resources of various large-scale generative artificial intelligence models.

[0002] Recently, a large-scale generative AI model based on Transformer has emerged and is attracting a large number of users. Transformer uses position embedding to incorporate token location information into its calculations. The newly introduced Rotary Position Embedding can more effectively incorporate location information.

[0003] However, the rotary position embedding operation has a problem in that the elements of the required query or key vector are far apart from each other, requiring access to the on-chip buffer and / or external memory. For example, the rotary position embedding operation obtains a position-encoded query vector or a position-encoded key vector using the position value in the query or key vector. For this operation, an operation is performed between the elements included in the query vector or the elements included in the key vector. At this time, since the locations of the elements required for the operation are far apart from each other, access to the on-chip buffer and external memory is required.

[0004] In addition, when supporting streaming rotary position embedding operations and non-linear activation function operations in a special function unit, there is a problem that more computational units are required in the special function unit. For example, among the operations of the transformer model, there are cases where the rotary position embedding operation or the non-linear activation function operation follows the matrix-vector multiplication operation. In this case, if the rotary position embedding operation and the non-linear activation function operation are processed with separate logic, the required computational resources increase.

[0005] A memory mapping method and system for rotary position embedding operations can be provided to minimize the inference delay speed of various large-scale generative artificial intelligence models and optimize computational resources.

[0006] The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0007] In a memory mapping system, a preprocessing unit for adjusting the positions of values ​​of a weight matrix for a pre-learned artificial intelligence model so that the values ​​of elements required for a rotary position embedding operation, which is a subsequent operation of a matrix multiplication operation, are adjacent to each other; a weight memory for storing a weight matrix in which the positions of the values ​​are adjusted; and a hardware operator for generating a query vector or a key vector by processing the matrix multiplication operation between an input vector and the weight matrix, and processing a rotary position embedding operation for the query vector or the key vector using the values ​​of the adjacent elements.

[0008] According to one aspect, the preprocessing unit may be characterized by adjusting the positions of values ​​of the weight matrix so that the values ​​of the elements are adjacent to each other before the matrix multiplication operation.

[0009] According to another aspect, the weight memory may store sine and cosine values ​​as parameters of the rotary position embedding operation, and the hardware operator may process the rotary position embedding operation through a multiplication operation and an addition operation between the sine and cosine values ​​read from the weight memory and the values ​​of adjacent elements.

[0010] According to another aspect, the hardware operator may be characterized by including a special function unit including an operator for the multiplication operation and the addition operation.

[0011] According to another aspect, the hardware operator may be characterized in that the operator included in the special function unit is shared and used for processing the rotary position embedding operation and processing the non-linear activation function operation.

[0012] According to another aspect, the hardware operator may be characterized in that it processes the non-linear activation function operation using the multiplication operation and the addition operation according to linear interpolation.

[0013] According to another aspect, the hardware operator may be characterized by including a rotary position embedding buffer for storing the sine and cosine values ​​read from the weight memory for reusing the sine and cosine values ​​during a single token generation process.

[0014] According to another aspect, the hardware operator may be characterized in that it processes the rotary position embedding operation by generating a position-encoded query vector for the query vector or a position-encoded key vector for the key vector.

[0015] In a memory mapping method, a memory mapping method is provided, comprising: a step of adjusting the positions of values ​​of a weight matrix for a pre-learned artificial intelligence model through a preprocessing unit so that the values ​​of elements required for a rotary position embedding operation, which is a subsequent operation of a matrix multiplication operation, are adjacent to each other; a step of storing a weight matrix in which the positions of the values ​​are adjusted in a weight memory; a step of processing the matrix multiplication operation between an input vector and the weight matrix through a hardware operator to generate a query vector or a key vector; and a step of processing a rotary position embedding operation for the query vector or the key vector through the hardware operator using values ​​of elements adjacent to each other.

[0016] Specific details of other embodiments are included in the detailed description and drawings.

[0017] A memory mapping method and system for rotary position embedding operations can be provided to minimize the inference delay speed of various large-scale generative artificial intelligence models and optimize computational resources.

[0018] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

[0019] FIG. 1 is a diagram illustrating an example of the structure of a latency processing unit according to an embodiment of the present invention.

[0020] FIGS. 2 to 5 are drawings illustrating examples of implementation models of LPUs according to embodiments of the present invention.

[0021] FIG. 6 is a diagram for explaining weight matrix data mapping of a high-bandwidth memory for matrix multiplication operation of a latency processing unit according to an embodiment of the present invention.

[0022] FIG. 7 is a diagram for explaining a high-bandwidth memory interface included in a latency processing unit according to one embodiment of the present invention.

[0023] FIG. 8 is a diagram illustrating a reconfigurable multi-function operation unit included in a latency processing unit according to one embodiment of the present invention.

[0024] FIG. 9 is a diagram for explaining the configuration of an address-based non-sequential multi-unit scheduler included in a latency processing unit according to one embodiment of the present invention.

[0025] FIG. 10 is a diagram illustrating an example of memory mapping for rotary position embedding operations in one embodiment of the present invention.

[0026] FIG. 11 and FIG. 12 are diagrams illustrating examples of memory mapping of rotary position embedding parameters in one embodiment of the present invention.

[0027] FIG. 13 is a diagram illustrating an example of a rotary position embedding operation according to one embodiment of the present invention.

[0028] FIGS. 14 to 16 are diagrams showing examples of sine and cosine values ​​mapped to memory in one embodiment of the present invention.

[0029] FIG. 17 is a diagram illustrating an example of the internal configuration of a memory mapping system according to one embodiment of the present invention.

[0030] FIG. 18 is a flowchart illustrating an example of a memory mapping method according to one embodiment of the present invention.

[0031] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms. These embodiments are provided only to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Like reference numerals designate like elements throughout the specification.

[0032] When one component is referred to as being "connected to" or "coupled to" another component, it includes both cases where it is directly connected or coupled to the other component, or cases where there is another component intervening therebetween. Conversely, when one component is referred to as being "directly connected to" or "directly coupled to" another component, it indicates that there is no other component intervening therebetween. "And / or" includes each and any combination of one or more of the mentioned items.

[0033] The terminology used herein is for the purpose of describing embodiments only and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise. As used herein, the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, operations, and / or elements.

[0034] Although terms like "first" and "second" are used to describe various components, these components are not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it should be understood that a "first" component referred to below may also be a "second" component within the technical scope of the present invention.

[0035] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in their common sense to those of ordinary skill in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0036] Below, the structure of an LPU (Latency Processing Unit) as an example of an accelerator for an artificial intelligence model such as a large-scale generative artificial intelligence model is described through FIGS. 1 to 9.

[0037] Referring to FIG. 1, the LPU (100) according to an embodiment of the present invention may include an SMA (Streamlined Memory Access) 110, an OIU (Operand Issue Unit) 120, an SXE (Streamlined eXecution Engine) 130, a VXE (Vector eXecution Engine) 140, an LMU (Local Memory Unit) 150, an ISU (Instruction Scheduling Unit) 160, a PCIe (Peripheral Component Interconnect express) Interface 170, and a P2P (Peer to Peer) Interface 180.

[0038] The SMA (110) may be a specialized DMA (Direct Memory Access). For example, the SMA (110) may connect all channels (e.g., 32) of the HBM (200) to the execution engine (e.g., the SEE (130)) to transfer half-precision floating point (FP16) data at maximum bandwidth. The SMA (110) may be designed with a deep FIFO (First In First Out) to transfer sequential memory requests based on preloaded memory (MEM) instructions. Hardware-based memory mapping may reduce latency by eliminating matrix modification or transposition operations. Therefore, the SMA (110) may stream data received in a maximum burst size to the execution engine with minimal latency. The SMA (110) may also efficiently perform matrix transposes using a strobe signal. Streaming data may contain parameters for vector matrix operations (e.g., weights, biases) and other vector-related operations (e.g., gamma / beta, embedding).

[0039] The OIU (120) can coordinate data streamed from the SMA (110) (e.g., a first operand) with input from on-chip memory (e.g., a second operand) before issuing it to the execution engine. Based on the execute (EXE) instruction, the OIU (120) can generate microcode that configures the execution engine and determines the target engine of the operand. In addition, the OIU (120) can be provided with a reusable buffer to eliminate read latency of static operands (e.g., input vectors) and an asymmetric buffer to hold vectorized data used as scalars (e.g., biases). Therefore, appropriate operands are almost always prefetched and ready to be issued to the execution engine immediately.

[0040] The SXE (130) is the main computing hardware of the LPU (100), and can be designed to maximize the incoming bandwidth to execute vector-matrix multiplication (V·M) such as attention, 1D convolution, and feedforward network. The SXE (130) can include a number of MAC (multiply-and-accumulation) trees (MAC Trees, 131) that can match the reception bandwidth and the computation bandwidth of the HBM (200). For example, if 1024 elements are received from the HBM (200) per cycle, the reception bandwidth and the computation bandwidth can be matched through 16 MAC trees (131) each having 64 input vectors. In addition, the MAC tree (131) with 64 input vectors can be composed of 64 multipliers and 63 adders.

[0041] A plurality of MAC trees (131) can perform matrix multiplication operations and can be connected by channel through the HBM (200), which is an external memory and a high-bandwidth memory, and the SMA (110). Specifically, one of the plurality of MAC trees (131) is connected to the HBM (200) through one channel, thereby maximizing the transmission bandwidth between the LPU (100) and the HBM (200), and performing the matrix multiplication operations required for the ultra-large artificial intelligence model without a bottleneck. Therefore, the number of the plurality of MAC trees (131) and the number of memory channels of the HBM (200) can be configured to be the same.

[0042] The matrix multiplication operation results of multiple MAC trees (131) can be provided to the VXE (140). The VXE (140) can be implemented using a custom low-latency ALU (Arithmetic Logic Unit) and can execute vector operations such as token embedding, softmax, normalization, and residual operations. Since these vector operations occur relatively infrequently, the fan-in from the OIU (120) to this path can be adjusted to reduce hardware resources with negligible performance loss. The VXE (140) can receive the operation results of multiple MAC trees (131) and receive activation values ​​from the LMU (150) to perform subsequent operations. The VXE (140) can be configured to include various combinations of operators by including multiple multi-function operation datapaths.

[0043] The LMU (150) can transmit activation values ​​to multiple MAC trees (131) and VXEs (140). At this time, the LMU (150) can copy and transmit the activation values ​​in order to transmit the same activation values ​​to multiple MAC trees (131). In addition, the LMU (150) can store the result values ​​calculated by the multiple MAC trees (131) and VXEs (140). In other words, the LMU (150) can function as an internal buffer corresponding to the HBM (200) within the LPU (100). At this time, the LPU (100) can store activation values ​​or model parameters with a high reuse rate in the LMU (150) and store weights with a low reuse rate in the HBM (200) in the matrix multiplication operation. The LMU (150) can be implemented as a 4MB multi-bank register file with scalar vector separation for fast, high-bandwidth access to input, output, and intermediate data. Additionally, the LMU (150) may be multi-ported to support simultaneous reads and writes in the write-back phase of the OIU (120) and the execution engine.

[0044] The ISU (160) can control the entire execution flow of the LPU (100). The ISU (160) can utilize the PIC (Parallel Instruction Chaining) method, and by using instruction chaining, dependent instructions can be executed sequentially. The PIC separates instructions requiring independent hardware into groups of dependent instructions (e.g., memory (MEM) instructions, execution (EXE) instructions, network (NET) instructions), so that all instructions can be executed in parallel with the instruction chain of each group, thereby achieving low control overhead and latency savings. The ISU (160) can also update control registers (e.g., token and layer number) for engine execution. The internal scheduler supports out-of-order execution of the SXE (130) and VXE (140) to maximize hardware utilization, and a powerful scoreboard can be designed to handle data risks. For example, the ISU (160) can schedule multiple MAC trees (131) and VXEs (140) to perform operations simultaneously. In addition, the ISU (160) can improve operation throughput and delay time by pre-executing instructions without dependencies to maximize parallel operations, thereby minimizing the idle time of each operation unit and memory access unit.

[0045] The LPU (100) can be connected to a host computer via a PCIe interface (170), and can receive commands required for the operation of the LPU (100) and input values ​​and weights of a large-scale artificial intelligence model from the host computer, perform calculations, and then transmit the results to the host computer.

[0046] The LPU (100) can be scaled out into a cluster of multiple LPUs connected via a P2P interface (180). The expanded cluster structure can further enhance the acceleration of computation of ultra-large artificial intelligence models.

[0047] FIGS. 2 to 5 are diagrams illustrating examples of implementation models of LPUs according to embodiments of the present invention. The embodiment of FIG. 1 previously described an example of an implementation model using external memory of HBM (200). Instead of HBM (200), DDR (Double Data Rate) may be used as the external memory. In this case, since it is difficult to store a large model in a single device, it may be divided into multiple partitions and stored in external memories for multiple devices (multiple LPUs) for each partition. In this case, synchronization between multiple devices may be required for inference of the large model.

[0048] The embodiment of FIG. 2 illustrates a plurality of external memories (320) storing a plurality of partitions (310) of a large model, similar to the embodiment described above in FIG. 1, and a plurality of LPUs (330) connected in parallel to the plurality of external memories (320). One LPU can be implemented in one FPGA (Field Programmable Gate Array), and one partition can be connected in parallel to one FPGA. The transformer structure includes multi-head attention, layer normalization, feed forward, etc. in the decoder layer, and the multi-head attention and feed forward can be model-parallelized. In this case, when the multi-head attention is terminated, one embedding vector can be output as a result. Since one device only has a portion of the embedding vector, synchronization may be required because multiple devices need to share each embedding vector in order to proceed to the next operation. At this time, considering scalability, a single LPU may be implemented with multiple external memories (e.g., two or four). For example, the embodiment of FIG. 1 illustrates an example in which two HBMs (200), each storing one partition, are used.

[0049] The embodiment of Fig. 3 illustrates an example of a PIM (Processing-in-Memory) model, in which one LPU is implemented as a PIM chip, and both the partition and the LPU computation unit are integrated into a single chip. The embodiment of Fig. 3 illustrates a plurality of LPUs (410), a plurality of partitions (310), and a plurality of LPU computation units (420), each of which can be implemented as a PIM chip. In this case, each of the plurality of LPUs (410) can include one partition and one LPU computation unit.

[0050] The embodiment of Fig. 4 illustrates an example of a Processing-near-Memory (PNM) model. It may be difficult to include a configuration for processing all LPU operations within a single PIM chip. The embodiment of Fig. 4 illustrates a model in which a plurality of partitions (310) are stored within a plurality of memory chips (510), and an LPU operation unit (521) for LPU operations is included within a buffer chip (520) such as a PNM chip.

[0051] The embodiment of Fig. 5 illustrates an example of a model combining PIM and PNM. For example, multiple partitions (310) may be stored in multiple memory chips (610). In addition, PIM-style LPU operation units (611) may be implemented as accumulation units, such as MAC trees, in each of the multiple memory chips (610). In this case, the LPU operation units (621) for the remaining high-level operations of the LPU may be implemented in the PNM manner in the buffer chip (620).

[0052] FIG. 6 is a diagram for explaining weight matrix data mapping of a high-bandwidth memory for matrix multiplication operation of a latency processing unit according to an embodiment of the present invention.

[0053] Referring to FIG. 6, the LPU (100) according to the present embodiment can store weight matrix data mapped to a high-bandwidth memory (610) such as HBM (200) so that the weight data can be retrieved without accessing another memory channel when performing matrix multiplication operation on each MAC tree, since the number of MAC trees constituting multiple MAC trees (131) and the number of memory channels of SMA (110) are the same.

[0054] Specifically, weight matrix data may be stored in the high-bandwidth memory (610) so that it is mapped to each channel (620-n) in the column direction (D1) of the weight matrix in the number of multiple MAC trees (131). Since the column direction in the weight matrix allows matrix multiplication operations to be performed in parallel, the multiple MAC trees (131) can read column direction data from their respective allocated memory channels (620-n) and perform matrix multiplication operations.

[0055] Next, multiple MAC trees (131) can map the weight matrix data so that they can accumulate in the row direction (D2) of the weight matrix to complete the final operation result. The number of row data mapped at once can be determined by the bandwidth of the high-bandwidth memory (610), and this can be determined by the size of the tiles that multiple MAC trees (131) can process at once.

[0056] FIG. 7 is a diagram for explaining a high-bandwidth memory interface included in a latency processing unit according to one embodiment of the present invention.

[0057] Referring to FIG. 7, the SMA (110) can connect the LMU (150), multiple MAC trees (131), and high-bandwidth memory (610). The SMA (110) is not connected to other computational units of the LPU (100), and thus, the high-bandwidth memory interface can be minimized in terms of hardware resources.

[0058] Multiple MAC trees (131) and memory channels (620-n) can be connected in a one-to-one correspondence, respectively. That is, multiple MAC trees (131) do not need to access channels other than the directly assigned channels, and thus can perform matrix multiplication operations without using complex interfaces such as cross-bars, which consume a lot of resources and have a high delay time.

[0059] SMA (110) can configure only a read interface for loading weight matrix data stored in high-bandwidth memory (610) of multiple MAC trees (131). In other words, as described later, since the result of the operation is stored in the high-bandwidth memory (610) through LMU (150), a write interface for the high-bandwidth memory (610) of multiple MAC trees (131) is not configured, and hardware resources can be reduced accordingly.

[0060] In contrast, the SMA (110) can only form a write interface between the LMU (150) and the high-bandwidth memory (610). The computational results stored in the LMU (150) as an internal buffer through the SMA (110) can be transmitted to be written to the high-bandwidth memory (610), and the memory channel of the writing target can be selected using the demultiplexer (710).

[0061] FIG. 8 is a diagram illustrating a reconfigurable multi-function operation unit included in a latency processing unit according to one embodiment of the present invention.

[0062] Referring to FIG. 8, the VXE (140) may include a plurality of multi-function operation data paths (810, 820), and the plurality of multi-function operation data paths (810, 820) may be connected to an operator / result value chain network (830) to form various combinations of operators.

[0063] As illustrated in FIG. 8, a plurality of multi-function operation datapaths (810, 820) may include various operation units required for, for example, lookup table-based non-linear activation function operation, masking operation, etc. However, the configuration of the operation units of the reconfigurable multi-function operation datapaths (810, 820) illustrated in FIG. 8 is exemplary, and it goes without saying that additional operation units required for large model operation may be included in the multi-function operation datapaths (810, 820). The result calculated by the VXE (140) may be transmitted to the LMU (150).

[0064] FIG. 9 is a diagram for explaining the configuration of an address-based non-sequential multi-unit scheduler included in a latency processing unit according to one embodiment of the present invention.

[0065] Referring to FIG. 9, the ISU (160) as an address-based out-of-order multi-unit scheduler included in the LPU (100) according to the present embodiment may include an address-based instruction dependency determination and scheduling controller (910), a plurality of instruction issue controllers (921, 922, 923, 924), a multi-bank buffer address status table (950), an instruction buffer (960) and result address status update logic (970), and a multi-unit instruction dispatcher (980).

[0066] The ISU (160) can operate each operation unit and data movement unit simultaneously through the address-based instruction dependency judgment and scheduling controller (910) and multiple instruction issue controllers (921, 922, 923, 924). At this time, the ISU (160) can change the status of the operator address and result address of the multi-bank buffer address status table (950) for the instruction performed in each operation unit to 1.

[0067] The multi-bank buffer address status table (950) can change the status of the result address of a completed instruction to 0 through the result address status update logic (970).

[0068] The address-based instruction dependency determination and scheduling controller (910) can determine the dependency between the instruction to be executed and the instruction being executed, as well as the dependency between the instructions to be executed, by referencing the states of the addresses through the multi-bank buffer address state table (970). This enables preprocessing of instructions without dependencies, thereby minimizing the idle time of each operation unit and data movement unit.

[0069] The address-based instruction dependency determination and scheduling controller (910) included in the ISU (160) can load and process instructions from the instruction buffer (960). At this time, the address-based instruction dependency determination and scheduling controller (910) performs a loop instruction, and decodes other instructions to distinguish the instructions and transmit them to the device-to-device instruction issue controller (921), the direct memory access instruction issue controller (922), the MAC tree instruction issue controller (923), and the reconfigurable multi-function operation unit instruction issue controller (924) through the multi-unit instruction dispatcher (980).

[0070] The ISU (160) receives and stores commands of the LPU (100) from the host computer through the PCIe interface (170), and can store the current status of the LPU (100) in a status register. The host computer can check the status register through the PCIe interface (170).

[0071] Meanwhile, embodiments of the present invention can provide a memory mapping method and system for rotary position embedding operations that minimize on-chip buffer and external memory access. For example, in the LPU (100) structure as an accelerator described above, the operation result of the MAC tree (131) can be used as an input value of a special function unit to perform rotary position embedding operations or non-linear activation function operations in a streaming manner.

[0072] FIG. 10 is a diagram illustrating an example of memory mapping for a rotary position embedding operation according to an embodiment of the present invention. In the case of rotary position embedding, operations are performed between values ​​of elements 0 and 64 of the result vector after a matrix multiplication operation, and between values ​​of elements 1 and 65, etc., operations are processed between values ​​of elements that are far apart in the result vector. When this is implemented in hardware, not only will it congest the hardware routing, but if the elements to be operated on are too far apart, streaming processing may be difficult.

[0073] Therefore, by changing the positions of the elements of the weight matrix (the positions between the values ​​of the elements) at the compiler level, operations between data right next to each other can be processed without complex hardware routing. For example, the column order of the query weight matrix and / or the column order of the key weight matrix can be changed and stored. At this time, the order can be changed and stored from the time the weight matrix is ​​first stored. Through this, when obtaining the position-encoded query vector (query') or the position-encoded key vector (key') shown in Fig. 10, it is possible to create a query vector (query) and a key vector (key) in which the elements used are located in an adjacent (or neighboring) order.

[0074] This requires preprocessing routing, but hardware routing can be eliminated at runtime, providing hardware benefits. Furthermore, even if the order of the weight matrix values ​​is arbitrarily changed, the subsequent operation is an inner product operation, regardless of order, so there is no routing overhead at runtime.

[0075] Figures 11 and 12 are diagrams illustrating examples of memory mapping of rotary position embedding parameters in one embodiment of the present invention. Rotary position embedding operations can perform sine (sin) and cosine (cos) operations. However, since the results of sine and cosine operations range from -1 to 1, it is not efficient to use a general 16-bit floating point (float16). It was confirmed that there is no problem with the operation accuracy even if quantization is performed with fixed-point 8-bits using a source code level simulator. Accordingly, in this embodiment, memory usage can be reduced by packing two 8-bits and grouping the sine and cosine values ​​into a single set of 16-bits. In addition, the sine and cosine values ​​exist as the product of the dimension of one head and the number of positions. This can be stored in a memory such as HBM (200) and read by determining the memory channel and address.

[0076] Fig. 13 is a diagram illustrating an example of a rotary position embedding operation according to an embodiment of the present invention. Fig. 13 shows an example in which the positions of elements q0 and q64 within a weight matrix are changed to be adjacent to each other. Through this, after generating a query vector (query) or a key vector (key) through a matrix multiplication operation between an input vector and a weight matrix, a rotary position embedding operation can be performed streamingly without accessing an on-chip buffer and / or external memory, thereby obtaining a position-encoded query vector (query') or a position-encoded key vector (key').

[0077] In addition, embodiments of the present invention can share resources for special function unit operations for streaming rotary position embedding operations and non-linear activation function operations. As described above, in the LPU (100) structure as an accelerator, the operation results of the MAC tree (131) can be used as input values ​​of the special function unit to perform streaming rotary position embedding operations or non-linear activation function operations.

[0078] FIGS. 14 to 16 are diagrams illustrating examples of sine and cosine values ​​mapped to memory according to an embodiment of the present invention. Here, the memory may correspond to the HBM (200) described above. The sine and cosine values ​​may have different values ​​depending on the current token position, and DIM_HEAD / 2 values ​​corresponding to each token position are mapped to two channels (CH) of the HBM (200). Therefore, when a command to read sine and cosine values ​​is executed, two channels of the HBM (200) are read at once in the form shown in the embodiment of FIG. 15. The sine and cosine values ​​read from the HBM (200) can be reused in the rotary position embedding buffer during one token generation process. Since the on-chip buffer and external memory are far away from the location where the rotary position embedding is calculated, access to the on-chip buffer and external memory can be minimized by storing the sine and cosine values ​​in a reusable buffer near the calculator. At this time, an accelerator such as LPU (100) can perform non-linear activation function operations using multiplication and addition with linear interpolation, and can also perform rotary position embedding operations using multiplication and addition.

[0079] At this time, by sharing the multipliers and adders within the special function unit for the multiplication and addition operations required for rotary position embedding operations and non-linear activation function operations, the two operations can be effectively supported while saving hardware resources. These multiplication and addition operations were previously described with reference to Figure 13.

[0080] FIG. 17 is a diagram illustrating an example of the internal configuration of a memory mapping system according to an embodiment of the present invention, and FIG. 18 is a flowchart illustrating an example of a memory mapping method according to an embodiment of the present invention.

[0081] A memory mapping system (1700) according to the present embodiment may include a weight memory (1710), a preprocessing unit (1720), and a hardware operator (1730), and the hardware operator (1730) may include a rotary position embedding buffer (1731) and a special function unit (1732). According to the embodiment, the weight memory (1710) may correspond to the HBM (200) described above, and the hardware operator (1730) may correspond to the LPU (100).

[0082] Additionally, depending on the embodiment, at least some of the components included in the memory mapping system (1700) may be implemented as physically different hardware devices. For example, the hardware calculator (1730) may be implemented as a separate hardware device, and the weight memory (1710) and preprocessing unit (1720) may be implemented as separate hardware devices. In this case, each hardware device may be connected to a network and communicate with each other.

[0083] In step (1810), the memory mapping system (1700) can adjust the positions of the values ​​of the weight matrix for the pre-learned artificial intelligence model through the preprocessing unit (1720) so that the values ​​of the elements required for the rotary position embedding operation, which is the next operation of the matrix multiplication operation, are adjacent to each other. At this time, the preprocessing unit (1720) can adjust the positions of the values ​​of the weight matrix so that the values ​​of the elements are adjacent to each other before the matrix multiplication operation to be described later. Here, the artificial intelligence model may be a transformer model for inference of a large-scale generative artificial intelligence model. In addition, the weight matrix may be a part of the entire weight matrix of the artificial intelligence model.

[0084] In step (1820), the memory mapping system (1700) may store a weight matrix with adjusted value positions in the weight memory (1710). For example, when a large-scale artificial intelligence model is divided into multiple partitions and processed through multiple LPUs, weight memories may exist for each of the multiple LPUs. In this case, the weight matrix to be processed by each LPU may be stored in the weight memory corresponding to the LPU. In the present embodiment, the weight memory (1710) may store a weight matrix for one hardware operator (1730). In this case, the weight matrix stored in the weight memory (1710) may be a weight matrix with adjusted value positions so that the values ​​of the elements required for the rotary position embedding operation in step (1810) are adjacent to each other.

[0085] In step (1830), the memory mapping system (1700) can store sine and cosine values ​​as parameters of the rotary position embedding operation in the weight memory (1710). These parameters have been described in detail previously with reference to FIGS. 11 and 12.

[0086] At step (1840), the memory mapping system (1700) can generate a query vector or a key vector by processing a matrix multiplication operation between an input vector and a weight matrix through a hardware operator (1730). The matrix multiplication operation can be performed, for example, through a MAC tree (131) included in an LPU (100) as a hardware operator (1730).

[0087] In step (1850), the memory mapping system (1700) can process a rotary position embedding operation for a query vector or a key vector using values ​​of adjacent elements through a hardware operator (1730). In this case, the hardware operator (1730) can process the rotary position embedding operation by generating a position-encoded query vector for the query vector or a position-encoded key vector for the key vector. For example, the hardware operator (1730) can process the rotary position embedding operation through multiplication and addition operations between sine and cosine values ​​read from the weight memory (1710) and values ​​of adjacent elements. As described above, since the values ​​of the weight matrix have been adjusted so that the values ​​of the elements for the rotary position embedding operation (values ​​of the elements of the result vector of the matrix multiplication operation) are already adjacent to each other, the hardware operator (1730) can process the operation between adjacent values ​​without adjusting the positions of the values ​​through hardware routing.

[0088] In addition, as described above, the hardware operator (1730) may include a special function unit (1732) including an operator for a multiplication operation and an addition operation. In this case, the hardware operator (1730) may process a non-linear activation function operation using a multiplication operation and an addition operation according to a linear interpolation method. To this end, the hardware operator (1730) may share and use the operators (multipliers and adders) included in the special function unit (1732) for processing the rotary position embedding operation and the non-linear activation function operation. In other words, the memory mapping system (1700) may further process the non-linear activation function operation through the special function unit (1732).

[0089] Additionally, as described above, the hardware operator (1730) may further include a rotary position embedding buffer (1731) for storing sine and cosine values ​​in order to reuse the sine and cosine values ​​read from the weight memory (1710) during a single token generation process.

[0090] In this way, according to embodiments of the present invention, a memory mapping method and system for rotary position embedding operations can be provided to minimize the inference delay speed of various large-scale generative artificial intelligence models and optimize computational resources.

[0091] Although the embodiments of the present invention have been described with reference to the attached drawings, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without altering the technical concept or essential features thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

Claims

1. In a memory mapping system, A preprocessing unit that adjusts the positions of values ​​of a weight matrix for a pre-trained artificial intelligence model so that the values ​​of elements required for the rotary position embedding operation, which is the next operation of the matrix multiplication operation, are adjacent to each other; A weight memory that stores a weight matrix with adjusted positions of the above values; and A hardware calculator that processes the matrix multiplication operation between the input vector and the weight matrix to generate a query vector or a key vector, and processes a rotary position embedding operation for the query vector or the key vector using values ​​of adjacent elements. Including, A memory mapping system characterized in that the preprocessing unit adjusts the positions of the values ​​of the weight matrix so that the values ​​of the elements are adjacent to each other before the matrix multiplication operation.

2. In paragraph 1, The above weight memory stores sine and cosine values ​​as parameters of the above rotary position embedding operation, The above hardware calculator processes the rotary position embedding operation through multiplication and addition operations between the sine and cosine values ​​read from the weight memory and the values ​​of adjacent elements. A memory mapping system featuring .

3. In paragraph 2, A memory mapping system, characterized in that the hardware operator includes a special function unit including an operator for the multiplication operation and the addition operation.

4. In paragraph 3, A memory mapping system characterized in that the hardware operator shares the operator included in the special function unit for processing the rotary position embedding operation and processing the non-linear activation function operation.

5. In paragraph 4, A memory mapping system characterized in that the hardware operator processes the non-linear activation function operation using the multiplication operation and the addition operation according to linear interpolation.

6. In paragraph 2, A memory mapping system characterized in that the hardware calculator includes a rotary position embedding buffer for storing the sine and cosine values ​​read from the weight memory for reusing the sine and cosine values ​​during a single token generation process.

7. In paragraph 1, A memory mapping system characterized in that the hardware operator processes the rotary position embedding operation by generating a position-encoded query vector for the query vector or a position-encoded key vector for the key vector.

8. In the memory mapping method, A step of adjusting the positions of values ​​of a weight matrix for a pre-learned artificial intelligence model through a preprocessing unit so that the values ​​of elements required for the rotary position embedding operation, which is the next operation of the matrix multiplication operation, are adjacent to each other; A step of storing a weight matrix with adjusted positions of the above values ​​in a weight memory; A step of generating a query vector or a key vector by processing the matrix multiplication operation between the input vector and the weight matrix through a hardware calculator; and A step of processing a rotary position embedding operation for the query vector or the key vector using the values ​​of adjacent elements through the hardware calculator. Including, The above saving step is, A memory mapping method characterized in that the positions of values ​​of the weight matrix are adjusted so that the values ​​of the elements are adjacent to each other before the matrix multiplication operation through the hardware operator.

9. In paragraph 8, A step of storing sine and cosine values ​​as parameters of the above rotary position embedding operation in the weight memory. Including more, The step of processing the above rotary position embedding operation is: Processing the rotary position embedding operation through multiplication and addition operations between the sine and cosine values ​​read from the above weight memory and the values ​​of adjacent elements. A memory mapping method characterized by .

10. In paragraph 9, The above hardware operator includes a special function unit including an operator for the above multiplication operation and addition operation, A step of further processing a non-linear activation function operation through the above special function unit. Including more A memory mapping method characterized by .

11. In paragraph 9, A memory mapping method characterized in that the hardware calculator includes a rotary position embedding buffer that stores the sine and cosine values ​​read from the weight memory for reusing the sine and cosine values ​​during a single token generation process.

12. In paragraph 8, The step of processing the above rotary position embedding operation is: A memory mapping method characterized in that the rotary position embedding operation is processed by generating a position-encoded query vector for the query vector or a position-encoded key vector for the key vector.

Images