Apparatus for generating Query and Key pairs of the Quaternion Transformer for Minimizing outliers in LLM Quantization

Abstract

The present invention relates to a query-key quaternion pair generation device for a quaternion transformer for minimizing outliers in LLM quantization, wherein quantized 4-bit input data is converted into a quaternion, the quaternion is rotated and projected onto a plane, the quaternion is rotated twice in different directions to calculate a quaternion pair having a predetermined sequence, and a weight is applied to the calculated quaternion pair to generate a query and a key, thereby limiting the occurrence of outliers.

Description

Technology Field

[0001] The present invention relates to Large Language Model (LM) quantization technology, and in particular to a query-key quaternion pair generation device of a quaternion transformer for minimizing outliers in LLM quantization. Background Technology

[0002] A quaternion is a hyper-complex number that extends complex numbers. A quaternion q is expressed as qw + iqx + jqy + kqz, and consists of four components: a real part qw and imaginary parts qx, qy, and qz. Compared to using matrices, quaternions allow for a more concise representation of an object's rotation in space and enable faster computation. As a result, they are used in computer graphics, control theory, signal processing, attitude control, physics, bioinformatics, molecular dynamics, computer simulation, and orbital mechanics.

[0003] Recently, there have been reports of cases where inter-channel correlations (e.g., RGB, multisensor) are treated as a single dataset using quaternion 4-dimensional hypercomplex numbers (real + 3 imaginary numbers), and performance is maintained or improved while reducing parameters to one-quarter of their original level through weight sharing via a core operation (Hamilton product). These cases demonstrate distinct advantages in problems that are "tend to have internal 4-dimensional structures," such as color / spectral imaging, time series (speech / prediction), graphs, and 3D pose / rotation. The representative architectures and application trends in the field of artificial intelligence recently are as follows.

[0004] ① QCNN (Quaternion CNN): Reports efficiency and accuracy advantages over standard CNNs by naturally processing channel combination in color images (ECCV'18, etc.). Extended to spectral and heterogeneous images.

[0005] ② QRNN / QLSTM: Combines Mel filters into a single quaternion and applies it to speech recognition (CTC-based) to improve PER with fewer parameters.

[0006] ③ Quaternion Transformer:

[0007] QTN: Designed a Quaternion Self Attention Transformer in Spectroscopic (HSI) and demonstrated SoTA-level performance (IEEE TCSVT'24).

[0008] Quatformer: Combines the concept of quaternion rotation with attention in time series forecasting with complex periodicity (KDD'22 work).

[0009] ④ QGNN (Quaternion GNN): Introduced quaternion embeddings to graph representation learning, such as node / graph classification, demonstrating SOTA-level performance in benchmarks (ACML'21). Derived applications in recommendation and knowledge graphs.

[0010] ⑤ 3D Rotation / Pose: Quaternions were incorporated into the design of learning targets / losses as rotation representations (ICCV'23). Recently, integration with estimation pipelines such as dual quaternions and particle filters is also underway.

[0011] In addition, in research on communication systems, quaternion modulation techniques that use a multidimensional modulation scheme with quaternions and transmit via a dual-polarized antenna are being studied in the fields of satellite communication, optical communication, and mobile communication. In particular, in mobile communication, methods are being proposed to transmit quaternion modulated signals through a multiple-input multiple-output (MIMO) antenna after Caley-Dickson decomposition.

[0012] Meanwhile, in systems operating on limited power, such as Internet of Things (IoT) terminals, mobile communication terminals, or satellite communication devices, the specifications of the High Power Amplifier (HPA), which consumes the most power, are determined based on Back-Off, which establishes the linear operating point. Since the installed HPA unit affects transmission power even with small amounts of transmitted data or slow transmission speeds, analyzing the Peak to Average Power Ratio (PAPR) during the design phase is crucial. As the Back-Off value increases, linearity improves but HPA power efficiency decreases; conversely, as the Back-Off value decreases, power efficiency improves but non-linearity increases. Non-linear characteristics also increase Out-Of-Band (OOB) spurious emissions, degrading overall system performance; therefore, this is important not only for the terminal but also for the system as a whole.

[0013] Recently, the importance of On-Device sLLM (small / streamlined Large Language Model) has emerged in the era of artificial intelligence. The trend of running Large Language Models (LLM) on-device—such as smartphones, IoT devices, medical devices, robots, and vehicles—rather than on a server is important for several reasons, including the following.

[0014] 1. The Importance of On-Device sLLM

[0015] (1) Privacy and security

[0016] Processing data locally without transmitting it to cloud servers → Enhanced privacy protection. Essential in sensitive data environments such as healthcare, finance, and national defense.

[0017] (2) Reduction of latency

[0018] Instant response possible without network round-trip time. Advantageous for applications where response speed is critical, such as real-time voice assistants, AR / VR, and autonomous driving.

[0019] (3) Cost reduction

[0020] Reduce cloud GPU / TPU usage costs and perform inference locally. Reduce server load for services requiring a large number of concurrent connections (e.g., chatbots, translators).

[0021] (4) Reduction of network dependency

[0022] Usable even in environments with unstable network connections. AI features available even in offline mode.

[0023] In other words, On-Device sLLM is establishing itself as an increasingly important strategy in terms of security, speed, cost, and independence.

[0024] 2. Necessity and Problems of 4-bit Quantization

[0025] To run sLLM on a device, the original model, which amounts to tens to hundreds of GB, must be lightweighted, and the core technology for this is quantization.

[0026] (1) Advantages of 4-bit Quantization

[0027] ① Representing parameters in 4-bit instead of 16-bit / 32-bit reduces memory usage by approximately 4 to 8 times.

[0028] ② Reduced computational load improves inference speed.

[0029] ③ Suitable for low-power and low-memory environments of mobile / edge devices.

[0030] (2) Major problems

[0031] However, if reduced to the extreme level of 4 bits, various side effects occur.

[0032] (a) Loss of expressiveness

[0033] ① If a 32-bit or 16-bit real number is reduced to 4 bits, the quantization resolution of the representable value is lowered.

[0034] ② Model accuracy declines due to inability to distinguish subtle differences in weights.

[0035] (b) Range Selection Problem (Dynamic Range Collapse)

[0036] ① Certain layers have a narrow weight distribution, while others have a wide distribution.

[0037] ② Since 4-bit has a limited range coverage, some values ​​become saturated or information is lost.

[0038] (c) Increase in computational noise

[0039] ① Quantization noise increases during the 4-bit conversion process, particularly affecting attention operations and gradient-based calculations.

[0040] ② As a result, the model is unstable or output quality deteriorates.

[0041] (d) Outlier problem

[0042] ① If there are rare large values ​​(outliers) in the weight distribution, severe distortion occurs in 4-bit scaling.

[0043] ② This problem is particularly important in GPT and LLaMA-based models.

[0044] (e) Inference instability

[0045] ① Smaller models (sLLM) have less parameter margin, so quantization error is reflected more significantly.

[0046] ② Ultimately, this can lead to "inaccurate answers, failure to maintain context, and increased hallucinations." Prior art literature

[0047] Republic of Korea Registered Patent No. 10-2177663 (Published Nov. 11, 2020) The problem to be solved

[0048] The present invention aims to provide a query-key quaternion pair generation device of a quaternion transformer for minimizing outliers in LLM quantization, which converts quantized 4-bit input data into a quaternion, rotates the quaternion to project it onto a plane, rotates the quaternion twice in different directions to calculate a pair of quaternions having a predetermined sequence, and generates a query and a key by applying weights to the calculated pair of quaternions. means of solving the problem

[0049] According to one aspect of the present invention for achieving the above-mentioned purpose, a query-key quaternion pair generation device of a quaternion transformer for outlier minimization in LLM quantization comprises: a data acquisition unit that acquires four bit information (b0, b1, b2, b3) as input data; and a vector quaternion (q) based on the input data acquired by the data acquisition unit. s A quaternion data conversion unit that calculates ); and a vector quaternion (q) calculated by the quaternion data conversion unit s ) and using the most significant bit information (b0) among the input data obtained by the data collection unit, a projection quaternion (q p Calculate ) and projected quaternion(q p A quaternion rotation processing unit that calculates whether to project ) onto a first plane or a second plane; and a vector quaternion (q s A first plane-projected quaternion (q) projected onto a first plane ip The first query quaternion(q) from ) i1 ) and the first key quaternion(q i2 A first planar query-key quaternion generator that produces a pair of ); and a vector quaternion (q s A second-plane projected quaternion (q) projected onto a second plane kp From ) the second query quaternion(qk1 ) and the second key quaternion(q k2 A second planar query-key quaternion generator that produces a pair of ); and a first query quaternion (q) produced by the first planar query-key quaternion generator. i1 ) and the first key quaternion(q i2 ) For each, the first query weight (w iq ) and the first key weight (w ik A first plane query-key generation unit that generates a query and a key of the first plane by multiplying by ); and a second query quaternion (q) calculated by a second plane query-key quaternion generation unit k1 ) and the second key quaternion(q k2 ) For each, the second query weight (w kq ) and the second key weight (w kk It includes a second plane query-key generation unit that generates a query and a key of the second plane by multiplying by ).

[0050] According to an additional aspect of the present invention, a quaternion data conversion unit sets the value of the real part to 0, and sets the remaining three bit information (b1, b2, b3) excluding the most significant bit information (b0) among the input data obtained by the data collection unit as the imaginary part value, thereby forming a Vector Quaternion (q s It may include a vector quaternion conversion unit that generates ).

[0051] According to an additional aspect of the present invention, a quaternion rotation processing unit comprises a multiplication unit that multiplies three bit information (b0, b1, b3) among input data obtained by a data collection unit; and a vector quaternion (q) calculated by a quaternion data conversion unit. s Using ) and the most significant bit information (b0) among the input data, a projected quaternion (q pA vector quaternion rotation unit that calculates ); and a projection quaternion (q) calculated by the vector quaternion rotation unit depending on whether the value obtained by multiplying three bit information (b0, b1, b3) by the multiplication unit is a positive or negative value. p It may include a plane projection switch that determines whether to project ) onto a first plane or a second plane.

[0052] According to an additional aspect of the present invention, if the value obtained by multiplying three bit information (b0, b1, b3) by the multiplication unit of the planar projection switch unit is a positive value, the projection quaternion (q p ) is the first planar projection quaternion(q ip Separated by ), and if the value obtained by multiplying the 3 bit information (b0, b1, b3) by the multiplication part is a negative value, then the projection quaternion (q p ) is the second planar projection quaternion(q kp It can be implemented to be distinguished by ).

[0053] According to an additional aspect of the present invention, a first planar query-key quaternion generator has a first planar projection quaternion (q) with respect to a first planar rotator. ip Rotate ) in different directions to obtain the first query quaternion (q i1 ) and the first key quaternion(q i2 It can be implemented to produce ) pairs.

[0054] According to an additional aspect of the present invention, a second planar query-key quaternion generator generates a second planar projected quaternion (q) with respect to a second planar rotator. kp Rotate ) in different directions to obtain the second query quaternion (q k1 ) and the second key quaternion(q k2 It can be implemented to produce ) pairs.

[0055] According to an additional aspect of the present invention, a first planar query-key generator produces a first query quaternion (q) by a first planar query-key quaternion generator. i1 ) 1st query weight(wiq A first query-weight multiplication unit that generates a first plane query by multiplying by ) and outputs the generated first plane query; and a first key quaternion (q) calculated by a first plane query-key quaternion generation unit i2 ) 1st key weight(w ik It may include a first key-weight multiplication unit that generates a first plane key by multiplying by ) and outputs the generated first plane key.

[0056] According to an additional aspect of the present invention, a second planar query-key generator produces a second query quaternion (q) by a second planar query-key quaternion generator. k1 ) 2nd query weight(w kq A second query-weight multiplication unit that generates a second plane query by multiplying by ) and outputs the generated second plane query; and a second key quaternion (q) calculated by a second plane query-key quaternion generation unit k2 ) 2nd key weight(w kk It may include a second key-weight multiplication unit that generates a second plane key by multiplying by ) and outputs the generated second plane key.

[0057] According to an additional aspect of the present invention, a query-key quaternion pair generating device of a quaternion transformer for minimizing outliers in LLM quantization may further include a value generating unit that generates a value.

[0058] According to an additional aspect of the present invention, a value generation unit has a Value Quaternion (q v Value weights (w) in ) v It can be implemented to generate a Value by multiplying by ) and output the generated Value.

[0059] According to an additional aspect of the present invention, a quaternion data conversion unit, based on input data obtained by a data collection unit, obtains a Value Quaternion (q v It may further include a value quaternion converter that calculates ).

[0060] According to an additional aspect of the present invention, a value generation unit applies a value weight (w) to input data (X) obtained by a data collection unit. v It can be implemented to generate a Value by multiplying by ) and output the generated Value.

[0061] According to an additional aspect of the present invention, the query of the first plane and the query of the second plane, and the key of the first plane and the key of the second plane, generated by the first plane query-key generation unit and the second plane query-key generation unit, may be quaternions.

[0062] According to an additional aspect of the present invention, the value generated by the value generation unit may be a quaternion or a scalar.

[0063] According to an additional aspect of the present invention, a first query weight (w iq ) and, the first key weight (w ik ) and, the second query weight (w kq ) and, the second key weight (w kk ) and value weights (w v ) can be a Quaternion or a Scalar. Effects of the invention

[0064] The present invention can limit the occurrence of outliers by rotating a quaternion in quaternion space to project it onto a complex plane, and then calculating a pair of quaternions (query and key quaternions) having a predetermined sequence through a decomposition process in which the quaternion projected from the complex plane is rotated twice in different directions. Additionally, since all but one of the quaternion components become zero, and even when multiplied by weights, each component becomes a quaternion with 1 bit, it has the effect of preventing severe scaling distortion in weighted matrix multiplication. Brief explanation of the drawing

[0065] FIG. 1 is a block diagram illustrating a first embodiment of a query-key quaternion pair generation device of a quaternion transformer for minimizing outliers in LLM quantization according to the present invention. FIG. 2 is a block diagram illustrating a second embodiment of a query-key quaternion pair generation device of a quaternion transformer for minimizing outliers in LLM quantization according to the present invention. FIG. 3 is a diagram illustrating a three-dimensional quaternion space applied in a query-key quaternion pair generation device of a quaternion transformer for outlier minimization in LLM quantization according to the present invention. FIG. 4 is a diagram illustrating the formation of two quaternion pairs based on vector quaternions in a query-key quaternion pair generation device of a quaternion transformer for outlier minimization in LLM quantization according to the present invention. Specific details for implementing the invention

[0066] The foregoing and additional aspects are embodied through embodiments described with reference to the attached drawings. It is understood that the components of each embodiment may be combined in various ways within the embodiment or with components of other embodiments, unless otherwise stated or contradicted. Based on the principle that the inventor may appropriately define the concepts of terms to best describe his invention, the terms used in this specification and claims shall be interpreted in a meaning and concept consistent with the description or proposed technical idea.

[0067] Blocks referred to as 'circuits' in this specification may be composed of hardware such as dedicated semiconductors, gate arrays, FPGAs, or parts thereof. One or more blocks may be implemented as a single piece of hardware. As another example, these blocks may be implemented in software as an information processing device in which a computing element executes program instructions stored in a memory element. Multiple blocks may be implemented as part of a program executed on the same computing element.

[0068] As another example, these blocks may be implemented in a hybrid form where part of the individual circuit is hardware and part is software. Additionally, in the software implementation, the computational elements may include digital signal processors, dedicated computational processors, artificial intelligence processing engines, dedicated AI processors, and graphics processors, or a combination thereof to the extent possible.

[0069] Hereinafter, the present invention will be described in detail through preferred embodiments described with reference to the attached drawings so that those skilled in the art can easily understand and reproduce it. Although specific embodiments are illustrated in the drawings and detailed descriptions are provided, this is not intended to limit the various embodiments of the present invention to a specific form.

[0070] In describing the present invention, if it is determined that a detailed description of related known functions or configurations could unnecessarily obscure the essence of the embodiments of the present invention, such detailed description will be omitted.

[0071] When it is stated that one component is "connected" or "joined" to another component, it should be understood that while it may be directly connected or joined to that other component, there may also be other components in between.

[0072] On the other hand, when it is stated that one component is "directly connected" or "directly coupled" to another component, it should be understood that no other component exists in between.

[0073] FIG. 1 is a block diagram illustrating a first embodiment of a query-key quaternion pair generation device of a quaternion transformer for minimizing outliers in LLM quantization according to the present invention, and FIG. 2 is a block diagram illustrating a second embodiment of a query-key quaternion pair generation device of a quaternion transformer for minimizing outliers in LLM quantization according to the present invention.

[0074] A query-key quaternion pair generation device (100) of a quaternion transformer for minimizing outliers in LLM quantization according to the present invention converts quantized 4-bit input data into a quaternion, rotates the quaternion and projects it onto a plane, rotates the quaternion twice in different directions to calculate a quaternion pair having a predetermined sequence, and generates a query and a key by applying weights to the calculated quaternion pair.

[0075] As illustrated in FIGS. 1 and 2, a query-key quaternion pair generation device (100) of a quaternion transformer for minimizing outliers in LLM quantization according to the present invention includes a data collection unit (110), a quaternion data conversion unit (120), a quaternion rotation processing unit (130), a first planar query-key quaternion generation unit (140), a second planar query-key quaternion generation unit (150), a first planar query-key generation unit (160), and a second planar query-key generation unit (170).

[0076] The data collection unit (110) acquires four bit information (b0, b1, b2, b3) as input data. At this time, each bit information may be composed of 1 or -1, and the data collection unit (110) may be implemented to receive four quantized bit information as input data from an external server (not shown in the drawing) or an external terminal (not shown in the drawing).

[0077] The quaternion data conversion unit (120) converts a Vector Quaternion (q) based on the input data obtained by the data collection unit (110). s Calculate ). The quaternion data conversion unit (120) performs the operation of calculating a quaternion vector and a rotation angle using four bit information (b0, b1, b2, b3). At this time, the quaternion data conversion unit (120) may include a vector quaternion conversion unit (121).

[0078] The vector quaternion conversion unit (121) sets the real part value of the quaternion expressed as qw + iqx + jqy + kqz (real part qw and imaginary parts qx, qy, qz) to 0, and sets the remaining three bit information (b1, b2, b3) excluding the most significant bit information (b0) among the input data obtained by the data collection unit (110) as imaginary part values, thereby forming a vector quaternion (q s) generates. For example, the vector quaternion conversion unit (121) uses Equation 1 to generate a vector quaternion (q s It can be implemented to generate ).

[0079]

[0080] (q s : Vector quaternion, b1, b2, b3: 3-bit information included in the input data)

[0081] The quaternion rotation processing unit (130) is a vector quaternion (q) calculated by the quaternion data conversion unit (120). s ) and using the most significant bit information (b0) among the input data obtained by the data collection unit (110), a projection quaternion (q p Calculate ) and projected quaternion(q p Calculate whether to project ) onto a first plane or onto a second plane. At this time, the quaternion rotation processing unit (130) may include a multiplication unit (131), a vector quaternion rotation unit (132), and a plane projection switch unit (133).

[0082] The multiplication unit (131) multiplies three bit information (b0, b1, b3) among the input data obtained by the data collection unit (110). The value obtained by multiplying the three bit information (b0, b1, b3) among the input data (R = b0 × b1 × b3) is used to determine whether to project onto a first plane or a second plane.

[0083] The vector quaternion rotation unit (132) is a vector quaternion (q) calculated by the quaternion data conversion unit (120). s Using ) and the most significant bit information (b0) among the input data, a projected quaternion (q p ) calculates. For example, the vector quaternion rotation part (132) calculates the vector quaternion (q) using mathematical formula 2. s ) and using the most significant bit information (b0) among the input data, a projection quaternion (q pIt can be implemented to calculate ).

[0084]

[0085] (q p : Projected quaternion, q j (φ): j-axis rotor, q j * (φ): q j (φ) conjugate quaternion)

[0086] The planar projection switch unit (133) calculates a projection quaternion (q) by the vector quaternion rotation unit (132) depending on whether the value obtained by multiplying three bit information (b0, b1, b3) by the multiplication unit (131) is a positive or negative value. p Determine whether to project ) onto the first plane or the second plane.

[0087] For example, if the value obtained by multiplying three bit information (b0, b1, b3) by the multiplication unit (131) of the planar projection switch unit (133) is a positive value, then the projection quaternion (q p ) is the first planar projection quaternion(q ip Separated by ), and if the value obtained by multiplying the three bit information (b0, b1, b3) by the multiplication unit (131) is a negative value, the projection quaternion (q p ) is the second planar projection quaternion(q kp It can be implemented to be distinguished by ).

[0088] Projected quaternion(q p The rotation direction is determined based on whether the value obtained by multiplying three bit information (b0, b1, b3) is positive or negative, and depending on the rotation direction, it is projected onto a first plane or a second plane to form a first plane projected quaternion (q ip ) or second planar projection quaternion(q kp It is expressed as ).

[0089] The planar projection switch unit (133) uses mathematical formula 3 to produce a first planar projection quaternion (q) when the value obtained by multiplying three bit information (b0, b1, b3) is a positive value.ip Calculate ), and if the value obtained by multiplying the three bit information (b0, b1, b3) is negative, use Equation 4 to obtain the second planar projection quaternion (q kp Calculate )

[0090]

[0091] (q ip : First planar projection quaternion, q j (φ): j-axis rotor, q j * (φ): q j (φ) conjugate quaternion)

[0092]

[0093] (q kp : Second planar projection quaternion, q j (φ): j-axis rotor, q j * (φ): q j (φ) conjugate quaternion)

[0094] FIG. 3 is a diagram illustrating a three-dimensional quaternion space applied in a query-key quaternion pair generation device of a quaternion transformer for outlier minimization in LLM quantization according to the present invention. FIG. 3(a) illustrates a mutually perpendicular first plane (IJ plane) and a second plane (KJ plane) in the three-dimensional quaternion space, and FIG. 3(b) illustrates a projected quaternion (q p ) is projected onto the first plane (IJ plane) to form a first plane projected quaternion (q ip It illustrates what is indicated by ).

[0095] The first planar query-key quaternion generator (140) is a vector quaternion (q s A first plane-projected quaternion (q) projected onto a first plane ip The first query quaternion(q) from ) i1 ) and the first key quaternion(q i2 Produces ) pairs.

[0096] At this time, the first plane query-key quaternion generator (140) generates the first plane projected quaternion (q) based on the first plane rotator. ip Rotate ) in different directions to obtain the first query quaternion (q i1 ) and the first key quaternion(q i2 It can be implemented to produce ) pairs.

[0097] For example, the first planar query-key quaternion generator (140) is the first query quaternion (q i1 A first planar query quaternion generator (141) that calculates ), and a first key quaternion (q i2 It may include a first planar key quaternion generator (142) that calculates ).

[0098] The first planar query quaternion generator (141) and the first planar key quaternion generator (142) use Equation 5 to generate the first planar projection quaternion (q ip Rotate ) in different directions to obtain the first query quaternion (q i1 ) and the first key quaternion(q i2 It can be implemented to calculate ).

[0099]

[0100] (q i1 : First query quaternion, q i2 : 1st key quaternion, b3 : 4th bit of input data, R = b0×b1×b3, : 1st plane rotor, : Conjugate quaternion of the first planar rotor)

[0101] The second planar query-key quaternion generator (150) is a vector quaternion (q s A second-plane projected quaternion (q) projected onto a second plane kp From ) the second query quaternion(q k1 ) and the second key quaternion(q k2 Produces ) pairs.

[0102] At this time, the second plane query-key quaternion generator (150) generates the second plane projected quaternion (q) based on the second plane rotator. kp Rotate ) in different directions to obtain the second query quaternion (q k1 ) and the second key quaternion(q k2 It can be implemented to produce ) pairs.

[0103] For example, the second planar query-key quaternion generator (150) generates the second query quaternion (q k1 A second planar query quaternion generator (151) that calculates ), and a second key quaternion (q k2 It may include a second planar key quaternion generator (152) that calculates ).

[0104] The second planar query quaternion generator (151) and the second planar key quaternion generator (152) use Equation 6 to generate the second planar projection quaternion (q kp Rotate ) in different directions to obtain the second query quaternion (q k1 ) and the second key quaternion(q k2 It can be implemented to calculate ).

[0105]

[0106] (q k1 : Second query quaternion, q k2 : Second key quaternion, b1 : Second bit of input data, R = b0×b1×b3, : Second plane rotor, : Conjugate quaternion of the second planar rotor)

[0107] The first planar query-key generation unit (160) is a first query quaternion (q) produced by the first planar query-key quaternion generation unit (140). i1 ) and the first key quaternion(q i2 ) For each, the first query weight (w iq ) and the first key weight (w ikA query and a key of the first plane are generated by multiplying by ). At this time, the first plane query-key generation unit (160) may include a first query-weight multiplication unit (161) and a first key-weight multiplication unit (162).

[0108] The first query-weight multiplication unit (161) is a first query quaternion (q) produced by the first planar query-key quaternion generation unit (140). i1 ) 1st query weight(w iq Generate a first plane query by multiplying by ), and output the generated first plane query.

[0109] The first key-weight multiplication unit (162) is the first key quaternion (q) produced by the first planar query-key quaternion generation unit (140). i2 ) 1st key weight(w ik Generate a first plane key by multiplying by ), and output the generated first plane key.

[0110] The second planar query-key generation unit (170) is a second query quaternion (q) produced by the second planar query-key quaternion generation unit (150). k1 ) and the second key quaternion(q k2 ) For each, the second query weight (w kq ) and the second key weight (w kk A query and a key of the second plane are generated by multiplying by ). At this time, the second plane query-key generation unit (170) may include a second query-weight multiplication unit (171) and a second key-weight multiplication unit (172).

[0111] The second query-weight multiplication unit (171) is a second query quaternion (q) produced by the second planar query-key quaternion generation unit (150). k1 ) 2nd query weight(w kq Generate a second plane query by multiplying by ), and output the generated second plane query.

[0112] The second key-weight multiplication unit (172) is a second key quaternion (q) produced by the second planar query-key quaternion generation unit (150). k2 ) 2nd key weight(w kk Generate a second plane key by multiplying by ), and output the generated second plane key.

[0113] Meanwhile, the query of the first plane and the query of the second plane generated by the first plane query-key generation unit (160) and the second plane query-key generation unit (170) may be quaternions.

[0114] Additionally, the key of the first plane and the key of the second plane generated by the first plane query-key generation unit (160) and the second plane query-key generation unit (170) may be quaternions.

[0115] With reference to FIG. 4, we will examine the process of generating query quaternion and key quaternion pairs of a query-key quaternion pair generation device of a quaternion transformer for minimizing outliers in LLM quantization according to the present invention implemented as described above.

[0116] FIG. 4 is a diagram illustrating the formation of two quaternion pairs based on vector quaternions in a query-key quaternion pair generation device of a quaternion transformer for outlier minimization in LLM quantization according to the present invention.

[0117] When four bit information (b0, b1, b2, b3) is acquired as input data by the data collection unit (110) and input (210), the quaternion data conversion unit (120) converts a Vector Quaternion (q) based on the input data. s Calculate ) (220).

[0118] And, the quaternion rotation processing unit (130) is a vector quaternion (q s ) and using the most significant bit information (b0) among the input data, a projection quaternion (q pCalculate ) (230), and depending on whether the value obtained by multiplying three bit information (b0, b1, b3) of the input data is a positive or negative value, calculate the projection quaternion (q p Project (240) the ) onto the first plane or the second plane.

[0119] If, projected quaternion(q p When ) is projected onto the first plane, the first plane query-key quaternion generator (140) generates a vector quaternion (q s A first plane-projected quaternion (q) projected onto a first plane ip The first query quaternion(q) from ) i1 ) and the first key quaternion(q i2 ) Produces pairs (250).

[0120] Then, the first planar query-key generation unit (160) generates the first query quaternion (q i1 ) and the first key quaternion(q i2 ) For each, the first query weight (w iq ) and the first key weight (w ik Generate the query and key of the first plane by multiplying by ).

[0121] If, projected quaternion(q p When ) is projected onto the second plane, the second plane query-key quaternion generator (150) generates a vector quaternion (q s A second-plane projected quaternion (q) projected onto a second plane kp From ) the second query quaternion(q k1 ) and the second key quaternion(q k2 ) Produces pairs (260).

[0122] Then, the second planar query-key generation unit (170) generates the second query quaternion (q k1 ) and the second key quaternion(q k2 ) For each, the second query weight (w kq ) and the second key weight (w kk Generate the query and key of the second plane by multiplying by ).

[0123] Table 1 below illustrates a query and a key generated by a query-key quaternion pair generation device (100) of a quaternion transformer for minimizing outliers in LLM quantization according to the present invention, using four bit information as input data.

[0124]

[0125] When the weights are 4-bit quantized quaternions, the product between the query and the weight, or between the key and the weight, is a multiplication between 4-bit quaternions, and the distribution of the multiplication values ​​exists within the range [-4, 4].

[0126] When all components of each quaternion are ±1, each component of the quaternion product consists of the sum of four ±1 terms and always falls within the range [-4, 4]. Therefore, the result of the multiplication operation of a 2048-dimensional input vector and a weight matrix results in each component of the output vector being distributed within the range [-8192, 8192].

[0127] On the other hand, a query quaternion or key quaternion is a pure quaternion (vector quaternion) in which the scalar part is 0 and only one of the vector parts is ±1. When the same components are multiplied together, the result is -1 (scalar), and when different components are multiplied together, the result is another pure quaternion of magnitude ±1.

[0128] When the weights are 4-bit quantized quaternions, the distribution of the multiplication values ​​of the query quaternion or key quaternion and the 4-bit weight quaternion lies in the range [-1, 1], and when multiplied by a vector of length 2048, the distribution of the resulting vector components is in the range [-2048, 2048].

[0129] Therefore, since the distribution of query quaternion and key quaternion values ​​of the present invention is at most 2048 and at least -2048, the decomposed query quaternion and key quaternion pairs can reduce errors when quantized during subsequent computation processes. As lightweight language models (sLLM) have less parameter margin, they can resolve inference instability where quantization errors are reflected more significantly. In addition, battery life can be extended by reducing the Peak to Average Power Ratio (PAPR), which is a key performance indicator of the On-Device (not shown).

[0130] By implementing it in this way, the present invention can limit the occurrence of outliers by rotating a quaternion in quaternion space to project it onto a complex plane, and then calculating a pair of quaternions (query and key quaternions) having a predetermined sequence through a decomposition process in which the quaternion projected from the complex plane is rotated twice in different directions. Additionally, since all but one of the quaternion components become zero, and even when multiplied by weights, each component becomes a quaternion with 1 bit again, severe scaling distortion in weighted matrix multiplication can be prevented.

[0131] Meanwhile, according to an additional aspect of the invention, a query-key quaternion pair generating device (100) of a quaternion transformer for minimizing outliers in LLM quantization may further include a value generating unit (180). The value generating unit (180) generates a value.

[0132] At this time, the value generated by the value generation unit (180) may be a quaternion or a scalar value. If the value is a quaternion, the value quaternion (q v) can be defined as in mathematical formula 7, and if the value is a scalar value, the input data itself is the value.

[0133] As illustrated in FIG. 1, the value generation unit (180) generates a Value Quaternion (q v Value weights (w) in ) v It can be implemented to generate a Value by multiplying by ) and output the generated Value.

[0134]

[0135] In this case, the quaternion data conversion unit (120) may further include a value quaternion conversion unit (122). The value quaternion conversion unit (122) is based on input data obtained by the data collection unit (110) and a value quaternion (q v Calculate )

[0136] In contrast, as illustrated in FIG. 2, the value generation unit (180) applies a value weight (w) to the input data (X) obtained by the data collection unit (110). v It can be implemented to generate a Value by multiplying by ) and output the generated Value.

[0137] The query, key, and value generated by the query-key quaternion pair generation device (100) of the quaternion transformer for minimizing outliers in LLM quantization according to the present invention are concepts used in the attention mechanism of the quaternion transformer architecture.

[0138] The Query is the information currently being focused on (what one wants to find?), the Key is the information representing the characteristics of each token (what information I am about?), and the Value is the information actually to be conveyed (the actual content to be extracted). The attention mechanism calculates the similarity between the Query and the Key to obtain a Weight, and applies this to the Value to generate the final output.

[0139] By implementing as described above, the present invention can limit the occurrence of outliers by rotating a quaternion in quaternion space to project it onto a complex plane, and then calculating a pair of quaternions (query and key quaternions) having a predetermined sequence through a decomposition process in which the quaternion projected from the complex plane is rotated twice in different directions. Additionally, since all but one of the quaternion components become zero, and even when multiplied by weights, each component becomes a quaternion with 1 bit, severe scaling distortion in weighted matrix multiplication can be prevented.

[0140] The various embodiments disclosed in this specification and drawings are provided merely as specific examples to aid understanding and are not intended to limit the scope of the various embodiments of the invention.

[0141] Accordingly, the scope of the various embodiments of the present invention should be interpreted as including all modifications or variations derived based on the technical concept of the various embodiments of the present invention, in addition to the embodiments described herein. Industrial applicability

[0142] The present invention is industrially applicable in the field of technology related to LLM (Large Language Model) quantization and the field of technology for its application. Explanation of the symbols

[0143] 100 : Quaternion Transformer Query-Key Quaternion Pair Generator 110: Data Collection Unit 120 : Quaternion data conversion section 121 : Vector-Quantion Conversion Unit 122 : Value-Quantion Convertor 130 : Quaternion rotation processing unit 131 : Multiplication part 132 : Vector quaternion rotation 133 : Planar projection switch section 140 : First Planar Query-Key Quaternion Generator 141 : First Planar Query Quaternion Generator 142 : First planar key quaternion generator 150 : Second Planar Query-Key Quaternion Generator 151 : Second Planar Query Quaternion Generator 152 : Second planar key quaternion generator 160 : First Planar Query-Key Generation Unit 161 : 1st Query-Weight Multiplication Unit 162 : 1st key-weighted multiplication unit 170 : Second Planar Query-Key Generation Unit 171 : 2nd Query-Weighted Multiplication Unit 172 : 2nd Key-Weighted Multiplication Unit 180 : Value creation section

Claims

Claim 1 A data acquisition unit that acquires four bit information (b0, b1, b2, b3) as input data; and a Vector Quaternion (q) based on the input data acquired by the data acquisition unit s A quaternion data conversion unit that calculates ); and a vector quaternion (q) calculated by the quaternion data conversion unit s ) and using the most significant bit information (b0) among the input data obtained by the data collection unit, a projection quaternion (q p Calculate ) and projected quaternion(q p A quaternion rotation processing unit that calculates whether to project ) onto a first plane or a second plane; and a vector quaternion (q s A first plane-projected quaternion (q) projected onto a first plane ip The first query quaternion(q) from ) i1 ) and the first key quaternion(q i2 A first planar query-key quaternion generator that produces a pair of ); and a vector quaternion (q s A second-plane projected quaternion (q) projected onto a second plane kp From ) the second query quaternion(q k1 ) and the second key quaternion(q k2 A second planar query-key quaternion generator that produces a pair of ); and a first query quaternion (q) produced by the first planar query-key quaternion generator. i1 ) and the first key quaternion(q i2 ) For each, the first query weight (w iq ) and the first key weight (w ik A first plane query-key generation unit that generates a query and a key of a first plane by multiplying by ); and a second query quaternion (q) calculated by a second plane query-key quaternion generation unit k1 ) and the second key quaternion(q k2 ) For each, the second query weight (w kq ) and the second key weight (w kk A query-key quaternion pair generation device of a quaternion transformer for outlier minimization in LLM quantization, comprising a second plane query-key generation unit that generates a query and a key of the second plane by multiplying by ). Claim 2 In claim 1, the quaternion data conversion unit sets the value of the real part of a quaternion represented as qw + iqx + jqy + kqz (real part qw and imaginary parts qx, qy, qz) to 0, and sets the remaining three bit information (b1, b2, b3), excluding the most significant bit information (b0) among the input data obtained by the data collection unit, as the imaginary part value of the quaternion, thereby forming a Vector Quaternion (q s A query-key quaternion pair generation device of a quaternion transformer for outlier minimization in LLM quantization including a vector quaternion transform unit that generates ). Claim 3 In claim 1, the quaternion rotation processing unit comprises: a multiplication unit that multiplies three bit information (b0, b1, b3) among the input data obtained by the data collection unit; and a vector quaternion (q) calculated by the quaternion data conversion unit. s Using ) and the most significant bit information (b0) among the input data, a projected quaternion (q p A vector quaternion rotation unit that calculates ); and a projection quaternion (q) calculated by the vector quaternion rotation unit depending on whether the value obtained by multiplying 3 bit information (b0, b1, b3) by the multiplication unit is a positive or negative value. p A query-key quaternion pair generation device of a quaternion transformer for outlier minimization in LLM quantization, comprising a plane projection switch portion that determines whether to project ) onto a first plane or a second plane. Claim 4 In claim 3, the plane projection switch part: if the value obtained by multiplying the three bit information (b0, b1, b3) by the multiplication part is a positive value, the projection quaternion (q p ) is the first planar projection quaternion(q ip Separated by ), and if the value obtained by multiplying the 3 bit information (b0, b1, b3) by the multiplication part is a negative value, then the projection quaternion (q p ) is the second planar projection quaternion(q kp Query-key quaternion pair generator of a quaternion transformer for outlier minimization in LLM quantization distinguished by ). Claim 5 In claim 1, the first plane query-key quaternion generator: a first plane projected quaternion (q) based on the first plane rotator. ip Rotate ) in different directions to obtain the first query quaternion (q i1 ) and the first key quaternion(q i2 Query-key quaternion pair generator of a quaternion transformer for outlier minimization in LLM quantization yielding ) pairs. Claim 6 In claim 1, the second plane query-key quaternion generator: a second plane projected quaternion (q) based on the second plane rotator kp Rotate ) in different directions to obtain the second query quaternion (q k1 ) and the second key quaternion(q k2 Query-key quaternion pair generator of a quaternion transformer for outlier minimization in LLM quantization yielding ) pairs. Claim 7 In claim 1, the first planar query-key generator: a first query quaternion (q) produced by the first planar query-key quaternion generator i1 ) 1st query weight(w iq A first query-weight multiplication unit that generates a first plane query by multiplying by ) and outputs the generated first plane query; and a first key quaternion (q) calculated by a first plane query-key quaternion generation unit i2 ) 1st key weight(w ik A query-key quaternion pair generation device of a quaternion transformer for outlier minimization in LLM quantization, comprising: a first key-weight multiplier that generates a first plane key by multiplying and outputs the generated first plane key. Claim 8 In claim 1, the second planar query-key generator: a second query quaternion (q) produced by the second planar query-key quaternion generator k1 ) 2nd query weight(w kq A second query-weight multiplication unit that generates a second plane query by multiplying by ) and outputs the generated second plane query; and a second key quaternion (q) calculated by a second plane query-key quaternion generation unit k2 ) 2nd key weight(w kk A query-key quaternion pair generation device of a quaternion transformer for outlier minimization in LLM quantization, comprising: a second key-weight multiplier that generates a second plane key by multiplying and outputs the generated second plane key. Claim 9 In any one of claims 1 to 8, the query-key quaternion pair generating device of a quaternion transformer for outlier minimization in LLM quantization further comprises: a value generating unit for generating a value; a query-key quaternion pair generating device of a quaternion transformer for outlier minimization in LLM quantization. Claim 10 In Clause 9, Value generating part: Value Quaternion(q v Value weights (w) in ) v A query-key quaternion pair generator of a quaternion transformer for outlier minimization in LLM quantization that generates a value by multiplying and outputs the generated value. Claim 11 In claim 1, the quaternion data conversion unit: based on input data obtained by the data collection unit, a Value Quaternion (q v A query-key quaternion pair generation device of a quaternion transformer for outlier minimization in LLM quantization, further comprising a value quaternion transform unit that calculates ). Claim 12 In claim 9, the value generation unit: a value weight (w) to the input data (X) obtained by the data collection unit. v A query-key quaternion pair generator of a quaternion transformer for outlier minimization in LLM quantization that generates a value by multiplying and outputs the generated value. Claim 13 A query-key quaternion pair generation device of a quaternion transformer for outlier minimization in LLM quantization, wherein the query of the first plane and the query of the second plane and the key of the first plane and the key of the second plane generated by the first plane query-key generation unit and the second plane query-key generation unit are quaternions. Claim 14 In claim 9, a query-key quaternion pair generation device of a quaternion transformer for outlier minimization in LLM quantization, wherein the value generated by the value generation unit is a quaternion or a scalar. Claim 15 In Clause 10, the first query weight (w iq ) and, the first key weight (w ik ) and, the second query weight (w kq ) and, the second key weight (w kk ) and value weights (w v ) is a query-key quaternion pair generator of a quaternion transformer for outlier minimization in LLM quantization that is a quaternion or a scalar.

Images