Text processing method and text processing equipment

Abstract

The invention discloses a text processing method and text processing equipment. The text processing method includes inputting text data to a large language model configured to generate answers corresponding to the text data based on the text data and generating activation data as input data through at least one layer of the large language model; obtaining statistical information for quantization by applying an orthogonal matrix to the input data; determining a scaling factor based on the statistical information; performing scaling on the input data by the scaling factor; performing dynamic quantization based on a result of the scaling; and generating, by a subsequent layer of the at least one layer of the large language model, an answer corresponding to the text data based on the dynamically quantized scaled input data.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2024-0168607, filed on November 22, 2024, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to text processing methods and text processing devices. Background Technology

[0003] Large Language Models (LLMs) are becoming one of the models used in deep learning. LLMs typically take queries in text form and output the corresponding answers, and can have variables ranging in size from billions to over 100 billion units.

[0004] Compared to the size of a large LLM, the capacity of the dynamic random access memory (DRAM) used to operate an LLM can be relatively small. As a way to overcome such hardware limitations, techniques for reducing the model size by applying quantization to the LLM are widely used to implement practical services using LLMs. However, when processing text data, LLMs to which existing quantization methods are applied can have relatively low accuracy in outputting correct answers and / or relatively slow processing speeds. Summary of the Invention

[0005] The present invention is provided in a simplified form to introduce the choice of concepts further described below in the detailed embodiments. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0006] In one general aspect, a text processing method is performed by a computing device and includes: inputting text data into a large language model and generating activation data as input data through at least one layer of the large language model, wherein the large language model is configured to generate an answer corresponding to the text data based on the text data; applying an orthogonal matrix to the input data and obtaining statistical information for quantization based on the result of applying the orthogonal matrix to the input data; determining a scaling factor based on the statistical information; scaling the input data using the scaling factor; performing dynamic quantization on the scaled input data; and generating an answer corresponding to the text data based on the dynamically quantized scaled input data through subsequent layers of the at least one layer of the large language model.

[0007] In one general aspect, a text processing device includes: one or more processors; and a memory storing instructions configured to cause the one or more processors to: input text data into a large language model and generate activation data as input data through at least one layer of the large language model, wherein the large language model is configured to generate an answer corresponding to the text data based on the text data; obtain statistical information for quantization by applying an orthogonal matrix to the input data; determine a scaling factor based on the statistical information; perform scaling on the input data using the scaling factor; perform dynamic quantization on the scaled input data; and generate an answer corresponding to the text data based on the dynamically quantized scaled input data through subsequent layers of the at least one layer of the large language model.

[0008] In one general aspect, a text processing method includes: inputting text data into a large language model by one or more processors and generating activation data as input data through at least one layer of the large language model, wherein the large language model is configured to generate an answer corresponding to the text data based on the text data; storing the input data in a memory; generating an input matrix by the one or more processors by multiplying the input data by an orthogonal matrix; determining a reduction factor by the one or more processors based on statistics obtained from the input matrix; reducing the input matrix by the one or more processors using the reduction factor; generating a quantized input matrix by the one or more processors by reducing the bit size of the elements of the reduced input matrix; amplifying the quantized input matrix by the one or more processors using the reciprocal of the reduction factor; generating a quantized version of the input data by the one or more processors by applying the transpose of the orthogonal matrix to the amplified quantized input matrix; and generating an answer corresponding to the text data by the one or more processors through subsequent layers of the at least one layer of the large language model based on the quantized version of the input data.

[0009] In one general aspect, a dynamic quantization method performed by a computing device includes: applying an orthogonal matrix to input data and obtaining statistical information for quantization based thereon; determining a scaling factor based on the statistical information; scaling the input data using the scaling factor; and performing dynamic quantization on the scaled input data.

[0010] An orthogonal matrix can be generated, and the step of applying the orthogonal matrix to the input data may include multiplying the orthogonal matrix by the input data, and statistical information can be obtained by determining the statistics of the result of the multiplication.

[0011] Statistical information may include statistics on the results of applying an orthogonal matrix to the input data, and the statistics may be the mean, maximum, or variable.

[0012] An orthogonal matrix can be formed from {-1, 0, 1}.

[0013] The input data may include quantization units, and the step of determining the scaling factor may include: determining the scaling factor differently for each quantization unit.

[0014] The steps to perform scaling may include: determining whether the input data is in floating-point (FP) or integer (INT) form, and selectively using a floating-point (FP) arithmetic unit or a shifter to scale the input data according to the determined form of the input data.

[0015] The input data may include quantization units, and the scaling step may include scaling for each quantization unit.

[0016] The step of scaling input data for a quantization unit may include: scaling by performing an operation between a scalar value corresponding to the channel unit and a scalar value corresponding to the input data in response to the quantization unit being a channel unit; and scaling by performing an operation between a vector value corresponding to the block unit and a scalar value corresponding to the input data in response to the quantization unit being a block unit.

[0017] Each quantization unit can be a channel unit, a block unit, or a tensor unit.

[0018] The step of performing dynamic quantization on scaled input data can also be based on a target quantization value set by the user or automatically set by the algorithm.

[0019] The dynamic quantization method may further include: using the inverse of a scaling factor and the transpose of an orthogonal matrix to recover the dynamically quantized scaled input data; and multiplying the recovered dynamically quantized scaled input data by a weight and outputting the result.

[0020] The steps for recovering dynamically quantized scaled input data may include: calculating first recovered dynamically quantized scaled input data by applying the reciprocal of the scaling factor to the dynamically quantized scaled input data; and multiplying the transpose of the orthogonal matrix with the first recovered quantized data.

[0021] The transpose and weights of orthogonal matrices can be pre-calculated offline.

[0022] The dynamic quantization method may further include: in response to the input data being in floating-point (FP) form, converting the FP-form input data into integer (INT) form before applying the orthogonal matrix to the input data.

[0023] A non-transitory computer-readable storage medium storage instruction, which, when executed by one or more processors, causes the one or more processors to perform any of the dynamic quantization methods.

[0024] In another general aspect, a dynamic quantization device includes: one or more processors; and a memory storing instructions configured to cause the one or more processors to: obtain statistics for quantization by applying an orthogonal matrix to input data; determine a scaling factor based on the statistics; perform scaling on the input data by the scaling factor; and perform dynamic quantization on the scaled input data.

[0025] Orthogonal matrices can be generated, and applying orthogonal matrices to the processing of input data may include multiplying the orthogonal matrices by the input data, and statistical information can be obtained by determining the statistics of the result of the multiplication.

[0026] The instructions are also configured to cause the one or more processors to: determine whether the input data is in floating-point (FP) or integer (INT) form, and to perform scaling on the input data by selectively using a floating-point (FP) arithmetic unit or a shifter according to the determined form of the input data.

[0027] The input data may include quantization units, the scaling factor may be determined differently for each quantization unit, and each quantization unit may be a channel unit, a block unit, or a tensor unit.

[0028] In another general aspect, a method for quantizing input data includes: storing the input data in memory; generating an input matrix by one or more processors by multiplying the input data by an orthogonal matrix; determining a reduction factor by the one or more processors based on statistics obtained from the input matrix; reducing the input matrix by the one or more processors using the reduction factor; generating a quantized input matrix by the one or more processors by reducing the bit size of the elements of the reduced input matrix; amplifying the quantized input matrix by the one or more processors using the reciprocal of the reduction factor; and generating a quantized version of the input data by the one or more processors by applying the transpose of the orthogonal matrix to the amplified quantized input matrix.

[0029] Other features and aspects will become clear from the following detailed description, drawings and claims. Attached Figure Description

[0030] Figure 1A Examples are shown of the distribution of statistical outliers of activation functions and weights in a neural network model according to one or more embodiments.

[0031] Figure 1B Examples of distributions before and after applying an orthogonal matrix to an activation function are shown, according to one or more embodiments.

[0032] Figure 2Examples of dynamic quantization methods according to one or more embodiments are shown.

[0033] Figure 3 Examples of dynamic quantization methods according to one or more embodiments are shown.

[0034] Figure 4 Examples of dynamic quantization methods according to one or more embodiments are shown.

[0035] Figure 5 Examples of methods for obtaining statistical information for quantification according to one or more embodiments are shown.

[0036] Figure 6 An example is shown of a method for scaling input data for each quantization unit according to one or more embodiments.

[0037] Figure 7 An example is shown of a method for scaling input data for each quantization unit according to one or more embodiments.

[0038] Figure 8 Examples of dynamic quantization methods according to one or more embodiments are shown.

[0039] Figure 9 Examples of dynamic quantization methods according to one or more embodiments are shown.

[0040] Figure 10 An example of a dynamic quantization device according to one or more embodiments is shown.

[0041] Figure 11 An example of a dynamic quantization device according to one or more embodiments is shown.

[0042] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same or similar reference numerals will be understood to refer to the same or similar elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated. Detailed Implementation

[0043] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and brevity, descriptions of features known upon understanding this disclosure may be omitted.

[0044] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will be clear upon understanding the disclosure of this application.

[0045] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. As used herein, the term “and / or” includes any one and any combination of any two or more of the associated listed items. As a non-limiting example, the terms “comprising,” “including,” and “having” indicate the presence of the features, quantities, operations, components, elements, and / or combinations thereof stated, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0046] Throughout this specification, when a component or element is described as being "connected to," "joined to," or "attached to" another component or element, that component or element may be directly "connected to," "joined to," or "attached to" that other component or element, or there may reasonably be one or more other components or elements in between. When a component or element is described as being "directly connected to," "directly joined to," or "directly attached to" another component or element, there may not be any other components or elements in between. Similarly, expressions such as "between" and "immediately between," and "adjacent to" and "closely adjacent to" may be interpreted as described above.

[0047] Although terms such as “first,” “second,” and “third,” or A, B, (a), (b), etc., may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts are not limited by these terms. Each of these terms is not used to define, for example, the nature, order, or sequence of the corresponding component, assembly, region, layer, or part, but only to distinguish the corresponding component, assembly, region, layer, or part from other components, assemblies, regions, layers, or parts. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0048] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and as commonly understood based on an understanding of the disclosure of this application. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having the same meaning as they have in the relevant artistic context and in the disclosure of this application, and shall not be interpreted in an idealized or overly formal sense. The use of the term “may” herein in reference to examples or embodiments (e.g., regarding what an example or embodiment may include or implement) implies the existence of at least one example or embodiment that includes or implements such a feature, and not that all examples are limited thereto.

[0049] Figure 1A Examples are shown of the distribution of statistical outliers for the activation functions and weights of neural network models according to one or more embodiments. Figure 1A In the figure, Figure 110 shows the distribution of statistical outliers of the activation of the network model, and Figure 130 shows the distribution of outliers of the weights of the network model.

[0050] Quantization typically involves mapping the values ​​of activations (activations of activation functions) and weights of a neural network model (or deep learning model) to the range of integers (INT). In one example, the neural network model could be a large language model used to process text data. The large language model receives queries in text form (e.g., text data), generates intermediate data (e.g., activations) through multiple layers of the large language model, and finally outputs an answer corresponding to the text data. In the case of activations, as shown in Figure 110, a small number of activations can be relatively very large compared to the average, making quantization difficult; the information loss from quantization can be significant. In particular, abnormally large activations can lead to quantization errors when quantization is performed by uniformly dividing the entire range of the variable. As mentioned above, abnormally large values ​​can be referred to as outliers. Depending on how outliers are handled, the performance of the neural network model can vary after quantization, where poor quantization leads to degradation of the network model (e.g., relatively low accuracy in outputting correct answers and / or relatively slow processing speed).

[0051] As mentioned above, one of the important aspects of applying quantization to neural network models is the method of handling outliers. The activation outliers shown in Figure 110 can be higher than the weight outliers shown in Figure 130. Therefore, quantizing the activations shown in Figure 110 can be more difficult than quantizing the weights shown in Figure 130. Furthermore, activation outliers can have a greater impact on the quantization error of the neural network model.

[0052] To handle outliers, the specific channels of the activations and weights to which the outliers belong can be pre-classified individually through model profiling, quantized and processed with full precision or a relatively high number of bits, and then reordered via hardware (HW) awareness. Depending on the algorithm, smaller units (such as blocks or elements) can be used as quantization units in addition to using channels. However, since this method only considers the model's compression ratio and accuracy as performance metrics, without considering the characteristics of the associated HW device (e.g., bit width), the quantization latency can increase. See below. Figure 1B As shown, among the techniques for dealing with outliers, the technique of using orthogonal matrices can be used.

[0053] Figure 1B Examples of distributions before and after applying an orthogonal matrix to the activation function values ​​(i.e., activation) are shown according to one or more embodiments. Figure 1B In the diagram, Figure 150 shows the distribution of activations before the orthogonality matrix is ​​applied, and Figure 170 shows the distribution of activations after the orthogonality matrix is ​​applied. In Figure 150, it can be seen that the outlier activations are unevenly distributed before the orthogonality matrix is ​​applied, while after the orthogonality matrix is ​​applied, as shown in Figure 170, the distribution becomes somewhat more uniform.

[0054] The technique of using orthogonal matrices to handle outliers can distribute or disperse outliers uniformly across the entire dimension by multiplying the activations representing outliers by the orthogonal matrix and changing the dimensions.

[0055] However, when quantization is applied after outliers are distributed across different dimensions, there may still be values ​​or outliers that are too large to be processed with a small number of bits, even if the outliers are distributed across different dimensions.

[0056] In other words, even when using techniques that apply orthogonal matrices, it can still be difficult to express the maximum activation with a small number of bits. To address this difficulty, as shown in Figure 170, quantization can be applied (after distributing outliers across different dimensions) by applying orthogonal matrices within the layer units (e.g., for each layer) without processing outliers individually by reordering them. When quantization is applied after distributing outliers across different dimensions, the possibility that the channels of outliers obtained through model profiling may differ from the input data during actual testing and / or service can be reduced or even eliminated. Furthermore, when performing quantization with the same number of bits, the accuracy can be higher than when using methods that obtain and process outliers separately in channel units or value units. Additionally, the technique of using orthogonal matrices can be combined with methods that optimize the weights of outliers.

[0057] If the operation of multiplying orthogonal matrices is performed as a floating-point (FP) operation, the result can occupy a large amount of memory (assuming the orthogonal matrix is ​​large), and this processing can lead to high hardware costs compared to applying orthogonal matrices with INT operations.

[0058] Dynamic quantization can be performed efficiently and with good accuracy by using orthogonal matrices and simultaneously utilizing scaling factors determined from statistical information. Dynamic quantization may include dynamic scaling, which involves dynamically determining the scaling factor to be multiplied with the activations based on the range of data used in the neural network model. Dynamic scaling adjusts the amplification / scale (i.e., scaling factor) according to the characteristics of the dataset, allowing the neural network model to maintain optimal performance (e.g., relatively high accuracy in outputting correct answers and / or relatively fast processing speed).

[0059] Figure 2 Examples of dynamic quantization methods according to one or more embodiments are shown.

[0060] Most quantization techniques can be applied to deep learning models with memory footprint reduction and accuracy as two decisive criteria. However, quantization techniques that do not consider the implementation of hardware (HW) operations can actually increase latency and may not efficiently utilize HW resources. Furthermore, even implementing quantization techniques in dedicated HW devices can increase costs. For example, dynamic quantization methods according to one or more embodiments can be applied to LLM-based text processing methods, thereby enabling relatively high accuracy in outputting correct answers and / or relatively fast processing speeds when processing text data using text processing methods applying dynamic quantization.

[0061] Dynamic quantization devices can either directly generate orthogonal matrices or invoke and use stored orthogonal matrices. Regardless of how they are obtained, for example, given an orthogonal matrix, dynamic quantization devices can collect multiple statistical data points for dynamic quantization during the processing of the input data by applying the orthogonal matrix. In one example, the input data could be activation data generated through at least one layer of a large language model when processing text data in a large language model.

[0062] As mentioned above, applying an orthogonal matrix to input data can, for example, disperse outliers in the input data. However, even after this processing, trends with a wide range of values ​​may still exist. In such cases, dynamic quantization devices can use multiple statistical data obtained during (or immediately after) the application of the orthogonal matrix to perform dynamic quantization directly on the input data for each channel or each block / cell. Dynamic quantization accurately knows the range of values ​​in the input data, which can lead to better quantization performance than offline quantization. Dynamic quantization devices can reduce execution time by collecting multiple statistical data during (or immediately after) the operation using the orthogonal matrix. The orthogonal matrix can include, but is not limited to, rotation matrices, Hadamard matrices, or matrices using stochastic gradient descent (SGD) methods with Cayley transform.

[0063] Reference Figure 2 The dynamic quantization device can convert 16 bytes of input data 210 in FP form into input data in INT form. Figure 2 In the input data 210, the parts marked with double shading in bold are outliers.

[0064] Dynamic quantization devices can collect multiple statistical information for quantization by applying an orthogonal matrix O 220 to input data 210 (which can first be converted to INT form). These multiple statistical information can be obtained from, for example... Figure 11 The statistical information extraction circuit 1120 collects, but is not limited to, the data. Here, the location of outliers can be changed by applying an orthogonal matrix O 220 to the input data 210, which has been converted to the form of INT. The collected statistical information can be used to (i) determine the scaling factor used in the scaling process and / or (ii) determine the quantization unit in the quantization process. Here, the scaling factor may be referred to as a “scaling value” or “magnification ratio”. The scaling factor can be a vector or a scalar.

[0065] The dynamic quantization device calculates the required value for quantization (e.g., scaling factor, etc.) from multiple collected statistical data, performs scaling S 230, and then performs quantization 240. The dynamic quantization device can achieve a uniform distribution of the results (in...) through operations (e.g., multiplication) with orthogonal matrix O 220. Figure 2 In the matrix above “S 230”, scaling S 230 is performed to reduce the range of values.

[0066] Whether to perform scaling S 230 and / or the size of the scaling factor (or scaling unit) can be determined based on how many bits the dynamic quantization device performs quantization 240 on. The scaling unit may be the same as or different from the quantization unit. When the number of bits of the quantization target is sufficient, the dynamic quantization device may omit scaling S 230. The size of the scaling factor can be determined by a specific rule base. For example, when the statistics of the input data 210 (e.g., the maximum value) are x and the quantized bits are y, it can be predetermined as a rule whether scaling S 230 should be performed.

[0067] Here, the elements used as hyperparameters to determine the size of the scaling factor can be the size of the quantization unit (e.g., block unit, channel unit, and tensor unit) and the number of bits to which the data is quantized, but are not limited to these.

[0068] The dynamic quantization device can multiply the input data 210 (which has already been quantized by quantization 240) by the reciprocal of the reduction factor S. -1 250 (i.e., by amplification) to recover the range of the reduced values. Then, the dynamic quantization device can restore the range of values ​​by transposing the orthogonal matrix O220 into matrix O. T Multiplying the data by 260, whose value range has been restored, restores the data distribution dimension to its original dimension. Then, the result of multiplying the data (whose distribution dimension has been restored to its original dimension) by the weight W 270 can be output. Here, the transpose matrix O of the orthogonal matrix O 220 can be pre-computed offline or otherwise pre-obtained. T 260 and weight W270.

[0069] Using the above processing, dynamic quantization equipment can improve performance by reducing the implementation cost of the HW device performing quantization while reducing the latency during the quantization of the target HW.

[0070] Figure 3 Examples of dynamic quantization methods according to one or more embodiments are shown. Figure 300 illustrates the process of performing dynamic quantization on input data 310 when input data 310 in FP form is input to a dynamic quantization device.

[0071] Dynamic quantization devices can maintain accuracy while reducing computational complexity by converting input data 310 (e.g., activation functions) in the form of FP into input data 320 in the form of INT.

[0072] A dynamic quantization device can multiply input data 320 (in INT form) by an orthogonal matrix O 330 to distribute the values ​​of the input data 320 (in INT form) across the dimension of each quantization unit (e.g., block unit). The orthogonal matrix O 330 can be, for example, a Hadamard orthogonal matrix, but is not limited to it. The dimension N of the Hadamard orthogonal matrix can be a power of 2. Given a Hadamard orthogonal matrix of dimension N, for dimensions of powers of 2, the Hadamard orthogonal matrix can be freely generated to be less than or greater than N.

[0073] An orthogonal matrix O 330 can be generated by an orthogonal matrix generator (e.g., Figure 11 The orthogonal matrix generator 1110 generates the matrix. The orthogonal matrix generator can generate a diagonally set block-wise Hadamard orthogonal matrix based on the dimensions of the input data 320 (in INT form). Here, the orthogonal matrix O 330 multiplied with the input data 320 (in INT form) can distribute outliers included in the input data 320 across different dimensions. In this case, the dimension of the input data 320 in INT form can be the same as the dimension of the orthogonal matrix O 330, which is the target of the operation. The dynamic quantization device can transform to different dimensions by applying block-wise Hadamard orthogonal matrices of different sizes.

[0074] The dynamic quantization device can collect multiple pieces of information (e.g., statistical information) to be used for quantization from the result of applying an orthogonal matrix O 330 to input data 320 (in INT form) via an orthogonal matrix generator, and can dynamically apply the collected multiple pieces of information. The statistical information may include, for example, range, mean, maximum, and / or variable values, but is not limited to these. In one example, the statistical information may include statistics of the result of applying the orthogonal matrix to the input data, wherein the statistics may include at least one of range, mean, maximum, minimum, and variable. Simultaneously, the size and / or dimensionality of outliers can be dispersed by applying the orthogonal matrix O 330 to the input data 320 (in INT form).

[0075] Dynamic quantization devices can extract / obtain the values ​​required for quantization (e.g., scaling factors) from statistical information and can perform scaling 340. Scaling 340 can be achieved by, for example... Figure 11 The scaler 1130 is executed, but is not limited to it. The dynamic quantization device can dynamically determine the scaling factor based on statistics of the input data 320 (in INT form) with the applied orthogonal matrix O 330. When it is determined that it is still difficult to represent the input data 320 (in INT form) with a small number of bits even after dimension transformation, the scaling factor (e.g., the shrinking factor) can limit / shrink the range of values.

[0076] The dynamic quantization device can perform scaling 340 (e.g., shrinking) by multiplying the result of applying an orthogonal matrix O 330 to the input data 320 by a scaling factor determined based on statistical information.

[0077] In addition to scaling factors obtained from statistical information, dynamic quantization devices can also use, for example, the maximum, average, and variables of weights and / or activation functions as scaling factors to perform quantization 350.

[0078] For example, the dynamic quantization device can also apply scaling 340 to specific channels or specific blocks. For instance, when applying an orthogonal matrix O 330 to input data 320 (in INT form) using statistics after the dimensions of the input data 320 have been transformed, the dynamic quantization device can also apply scaling 340 to specific channels or specific blocks within the channels of the input data 320 that require scaling 340. For example, assuming 4,096 elements in the form of 8 blocks, each block can include 512 elements. Here, the same scaling factor can be applied to blocks of 512 elements. The scaling factor can be applied differently for each specific channel or each specific block. That is, a different scaling factor can be determined for each block.

[0079] The scaling factor can be freely used, such as FP or 2. n The dynamic quantization device is in the form of (a power of 2), and its configuration can vary depending on the type of scaling factor. For example, when the scaling factor is in the form of FP, the dynamic quantization device can use an FP operator (e.g., Figure 11 The FP arithmetic unit 1133 performs the calculation of the scaling factor (magnification), the operation between the scaling factor and the data, and the operation between the reciprocal of the scaling factor and the data (scaling down). On the other hand, when the scaling factor is in the form of INT, the dynamic quantization device can use a shifter (e.g., Figure 11 The shifter 1136 performs the calculation of the scaling factor (shrink), the operation between the scaling factor and the data, and the operation with the reciprocal of the scaling factor (enlarge).

[0080] The dynamic quantization device can pre-calculate the scaling factor and then determine and / or store it. When the scaling factor is pre-calculated, the dynamic quantization device can directly apply the scaling factor corresponding to the input data 320 (in INT form) during the inference process.

[0081] A dynamic quantization device can perform quantization 350 based on the result of scaling 340. Quantization 350 can be performed by, for example... Figure 11The quantizer 1140 can be used, but is not limited to it. Dynamic quantization devices can perform quantization 350 using the target digital system and can perform dynamic quantization using multiple statistical data collected during this process. For example, a dynamic quantization device can reduce the cost of HW by performing quantization 350 via a shifter and setting the scaling factor to a power of 2.

[0082] The dynamic quantization device can perform descaling 360, which multiplies the result of quantization 350 by the reciprocal of the corresponding previous scaling factor S. -1 This allows for the recovery of the value range of the result from quantization of 350. Descaling 360 can be achieved by, for example... Figure 11 The descaling function 1150 is executed, but it is not limited to this.

[0083] The scaling 340, quantization 350, and descaling 360 processes performed by the dynamic quantization device can be performed online (e.g., during operation).

[0084] Dynamic quantization devices can multiply data (whose range of values ​​has been recovered) by the transpose of the orthogonal matrix O 330. T 370 restores the dimensionality of the scattered data to its original dimension. Dynamic quantization devices can output by multiplying the data (whose values ​​have already been determined by the transpose matrix O) by... T The result of the 370 recovery is obtained by multiplying the weight W by 380. This generates the transpose matrix O. T 370 and transpose matrix O T The process of multiplying the recovered data by 370 and then multiplying the result by the weight W 380 can be performed by, for example... Figure 11 The dequantizer 1160 is executed, but it is not limited to this.

[0085] Given an orthogonal matrix O, can the orthogonal matrix O satisfy the condition O×O T =I. That is, the transpose matrix O T 370 can be the transpose of the orthogonal matrix O 330. Here, the transpose matrix O can be performed offline beforehand. T Multiplication between 370 and weight W 380.

[0086] When using calibration data (described below), the dynamic quantization device can fixate the range of values ​​for a specific channel after dimensional transformation. When multiple calibration data are input, the dynamic quantization device can record the statistical properties of each channel for each calibration data. The statistical properties can be, but are not limited to, absolute maximum (i.e., abs(max())), minimum-maximum (i.e., min-max), or standard deviation (i.e., std).

[0087] Dynamic quantization devices can use the corresponding statistical characteristics of a channel to identify the range of values ​​required for that channel. For example, in 1024 channels, when the range of values ​​for the fifth channel of all calibration data does not exceed 10 in the absolute maximum value abs(max()) and the scaling value is obtained later during quantization using the absolute maximum value abs(max()), the quantization scaling value for the fifth channel can be obtained using the absolute maximum value abs(max()) = 10.

[0088] Calibration data is a dataset used for static quantization and can typically be activation values. Calibration data can be generated by taking a portion of the training or validation data to identify statistical information about the values ​​of the input data 320 in INT form as it passes through each layer.

[0089] In this case, the transpose of the orthogonal matrix O 330, which is performed offline beforehand, can also be used. T The scaling values ​​that may not be integrated into the original weight W 380 are integrated in the same way as the calculations between weight W 370 and weight W 380. The scaling values ​​may not be integrated during the actual inference in the original dynamic quantization, but as mentioned above, they may be integrated into weight W 380 when the range of values ​​for a particular channel can be fixed.

[0090] The dynamic quantization device can analyze input data 310 in FP form offline and specify the channels where outliers occur. In this case, the dynamic quantization device can reduce computation and power consumption by bypassing the calculation by not generating an orthogonal matrix O 330 for the channels in the FP form input data 310 where no outliers occur. Here, it is feasible to perform analysis on the input data 310 in FP form offline, regardless of the size of the input data 310 in FP form.

[0091] Furthermore, when the orthogonal matrix O 330 is multiplied by a uniformly distributed value of the input data 310 (e.g., an activation function) in FP form, the distribution of the resulting value may be concentrated on one side, or outliers may be generated. Therefore, dynamic quantization devices can improve the computational speed and performance of neural network models by generating the orthogonal matrix O 330 and / or bypassing operations with the orthogonal matrix O 330 through channels of the input data 310 (e.g., an activation function) in FP form that do not have outliers or have a lower ratio of outliers than a specific reference.

[0092] When there are still values ​​that are difficult to quantize using low-bit quantization even after transforming the input data to 320 dimensions in INT form, dynamic quantization devices can collect multiple pieces of information (e.g., statistics) suitable for dynamic quantization, and can achieve fast execution time and high efficiency of quantization by using statistics to perform dynamic quantization.

[0093] Figure 4 Examples of dynamic quantization methods according to one or more embodiments are shown.

[0094] Reference Figure 4 The dynamic quantization device can perform dynamic quantization by operating 410 to 440.

[0095] In operation 410, the dynamic quantization device obtains statistical information (to be used for quantization) by applying an orthogonal matrix to the input data. The input data can be, for example, the activation of an activation function, but is not limited to this. For example, the input data can be activation data generated by at least one layer of a large language model. The input data can be, for example, in INT form, but is not limited to this. For example, when the input data is in FP form, the dynamic quantization device can apply an orthogonal matrix to the input data after converting the input data from FP form to INT form. Here, the orthogonal matrix can be generated by an orthogonal matrix generator included in the dynamic quantization device. As a non-limiting example, the orthogonal matrix can be, for example, a rotation matrix, a Hadamard matrix, or a matrix using the SGD method with Cayley transform. The orthogonal matrix can be formed, for example, {-1,0,1} or {-1,1}. Since the activation changes with the changing input value, the dynamic quantization device can perform quantization dynamically according to the changing activation.

[0096] In the following text, "statistical information" may refer to the statistical information of input data (in INT form) that has been subjected to orthogonal matrix manipulation. Statistical information may also be referred to as "statistical properties." Statistical information may include, but is not limited to, the mean, maximum, minimum, or variables of input data in INT form that have been subjected to orthogonal matrix manipulation. See also Figure 5 Describe a method for obtaining statistical information using dynamic quantization equipment.

[0097] In operation 420, the dynamic quantization device may determine a scaling factor based on statistical information obtained from operation 410. The dynamic quantization device may determine a different scaling factor for each quantization unit (a unit of data being quantized). Here, the quantization unit may include, but is not limited to, at least one of channel units, block units, and tensor units. For example, when the statistical information is the maximum value of the activation and the quantization unit is a block unit, the dynamic quantization device may determine a value less than (or less than or equal to) the maximum value of each block unit as the scaling factor for each block unit. Alternatively, when the quantization unit is a channel unit, the dynamic quantization device may determine a value less than the maximum value of each channel unit as the scaling factor for each channel unit. For example, when the input data is an activation function, the dynamic quantization device may perform scaling by using statistical information (e.g., the maximum value of the activation function) as the scaling factor.

[0098] In operation 430, the dynamic quantization device can perform scaling on the input data by applying a scaling factor determined in operation 420. Scaling can be performed selectively using different hardware devices (e.g., FP operators or shifters) depending on whether the scaling factor is in FP or INT form. Furthermore, the dynamic quantization device can perform scaling using different operation methods (e.g., scalar-to-scalar operations or vector-to-scalar operations) based on the quantization unit (e.g., channel unit or block unit). Additionally, the dynamic quantization device can determine a different scaling factor for each quantization unit. See reference. Figure 6 and Figure 7 Describes a method for scaling performed by a dynamic quantization device.

[0099] In operation 440, the dynamic quantization device can perform quantization on the scaling result obtained from operation 430. The dynamic quantization device can perform quantization on the scaling result based on a target value set by the user or automatically set by an algorithm. For example, an answer corresponding to the text data can be generated based on the result of the quantization performed through a subsequent layer of at least one layer of a large language model.

[0100] Figure 5 Examples of methods for obtaining statistical information for quantification according to one or more embodiments are shown. (Refer to...) Figure 5 The dynamic quantification equipment can detect and statistical information by operating 510 to 530.

[0101] In operation 510, the dynamic quantization device can generate or otherwise obtain an orthogonal matrix. The dynamic quantization device can generate an orthogonal matrix by using a Hadamard matrix generator. Here, as described below, the orthogonal matrix can distribute outliers included in the input data into different dimensions through operations with the input data.

[0102] In operation 520, the dynamic quantization device can perform the above operations (e.g., multiplication) by applying the orthogonal matrix generated in operation 510 to the input data.

[0103] In operation 530, the dynamic quantization device can detect statistical information from the computational results obtained from operation 520. Here, the statistical information may correspond to (i.e., be specific to) the quantization unit.

[0104] Figure 6 Examples of methods for performing scaling on the input data to be quantized for each quantization unit, according to one or more embodiments, are shown. (Refer to...) Figure 6 The dynamic quantization device can perform scaling through operations 610 to 630 using the scaling factor determined in operation 420.

[0105] In operation 610, the dynamic quantization device can determine whether the scaling factor determined in operation 420 is in the form of FP.

[0106] When the scaling factor is determined to be in the form of FP in operation 610, the dynamic quantization device can perform scaling on the input data by using the FP arithmetic unit in operation 620.

[0107] When it is determined in operation 610 that the scaling factor is not in FP form (that is, when it is determined that the scaling factor is in INT form), then in operation 630, the dynamic quantization device can perform scaling on the input data by using a shifter. The dynamic quantization device can perform scaling on the input data per quantization unit.

[0108] Figure 7 Examples of methods for scaling input data for each quantization unit, according to one or more embodiments, are shown. (Refer to...) Figure 7 The dynamic quantization device can perform scaling for each quantization unit by operating 710 and 720.

[0109] In operation 710, when the quantization unit is a channel unit, the dynamic quantization device can perform scaling by operating on the scalar value corresponding to the channel unit and the scalar value corresponding to the input data.

[0110] In operation 720, when the quantization unit is a block unit, the dynamic quantization device can perform scaling by operating between the vector value corresponding to the block unit and the scalar value corresponding to the input data.

[0111] Figure 8 Examples of dynamic quantization methods according to one or more embodiments are shown. (Refer to...) Figure 8 The dynamic quantization device can output the results of dynamic quantization by operating 810 to 860.

[0112] In operation 810, the dynamic quantization device obtains statistics for quantization by applying an orthogonal matrix to the input data in the form of INT. The input data can be, for example, an activation function (i.e., activation), but is not limited to this.

[0113] In operation 820, the dynamic quantization device can determine the scaling factor based on the statistical information obtained from operation 810.

[0114] In operation 830, the dynamic quantization device can perform scaling on the input data for each quantization unit using a corresponding scaling factor determined in operation 820. For example, when the input data is activation function data, the dynamic quantization device can perform scaling by using statistical information (e.g., the maximum value of the activation of the corresponding quantization unit) as the scaling factor. Here, the quantization unit can be a channel unit, a block unit, or a tensor unit, but is not limited to these. The dynamic quantization device can perform scaling by selectively using different means (e.g., an FP arithmetic unit or a shifter) depending on whether the scaling factor is in FP or INT form. Optionally, the dynamic quantization device can perform scaling using different arithmetic methods (e.g., scalar operation to scalar operation or vector operation to scalar operation) depending on the quantization unit (e.g., channel unit or block unit). Additionally, the dynamic quantization device can determine a different scaling factor for each quantization unit.

[0115] In operation 840, the dynamic quantization device may perform dynamic quantization based on the scaling result obtained from operation 830, and may do so based on a target value set by the user or a target value set automatically by the algorithm.

[0116] In operation 850, the dynamic quantization device can recover the data quantized in operation 840 using the reciprocal of the scaling factor determined in operation 820 and the transpose of the orthogonal matrix used in operation 810. The transpose of the orthogonal matrix can be pre-computed offline. The dynamic quantization device can compute the first recovered quantized data by applying the reciprocal of the scaling factor to the data quantized in operation 840. The dynamic quantization device can compute the quantized data whose dimensions have been recovered (obtaining a quantized version of the input data from operation 810) by multiplying the transpose of the orthogonal matrix by the first recovered quantized data.

[0117] In operation 860, the dynamic quantization device multiplies the quantized version of the input data (i.e., the data recovered in operation 850) by weights and outputs the result. Here, the weights can be pre-calculated offline. For example, an answer corresponding to the text data can be generated based on the quantized version of the input data through a subsequent layer of at least one layer of a large language model.

[0118] Figure 9 Examples of dynamic quantization methods according to one or more embodiments are shown. (Refer to...) Figure 9 The dynamic quantization device can perform dynamic quantization by operating 910 to 970.

[0119] In Operation 910, the dynamic quantization device can convert activations in FP form, which are input data, into activations in INT form.

[0120] In operation 920, the dynamic quantization device can generate an orthogonal matrix (e.g., a Hadamard matrix) from a matrix generator (e.g., a Hadamard matrix generator) based on the dimension of the input data (e.g., an activation function) (in INT form and as transformed in operation 910), and can perform operations between the orthogonal matrix and the input data in INT form.

[0121] In operation 920, the dynamic quantization device can detect and store statistics for each quantization unit from the result of the operation between an orthogonal matrix and input data in INT form. The dynamic quantization device obtains statistics used in the quantization process (more specifically, the scaling process for dynamic quantization) by applying an orthogonal matrix to the input data in INT form. The dynamic quantization device can detect and store statistics for each quantization unit (e.g., a channel unit or a block unit) from the result obtained through the matrix generator (i.e., the result of the operation between the orthogonal matrix and the input data in INT form).

[0122] In operation 930, the dynamic quantization device can perform scaling based on statistics stored in operation 920. Scaling can represent data with a smaller number of bits by reducing the range of values ​​in the quantization process to be performed.

[0123] In operation 940, the dynamic quantization device can perform quantization on the scaling result obtained from operation 930 using the target digital system. In short, quantization reduces the bit size of the values ​​in the quantized data (e.g., from 16 bits to 8 bits).

[0124] In operation 950, the dynamic quantization device can restore the range of the data (restore the range of the data to its original range) by multiplying the input data quantized in operation 940 by the reciprocal of the scaling factor used during scaling in operation 930.

[0125] In operation 960, the dynamic quantization device can recover the dispersed dimensions by multiplying the data recovered in operation 950 by the transpose of the orthogonal matrix (as in operation 920 with an operation on the orthogonal matrix, such as the Hadamard matrix).

[0126] In operation 970, the dynamic quantization device may, for example, multiply the data recovered in operation 960 by weights and output the result. Here, the transpose of the Hadamard matrix in operation 960 and the weights in operation 970 can be pre-computed offline. For example, an answer corresponding to the text data can be generated based on the data recovered in operation 960 through a subsequent layer of at least one layer of a large language model.

[0127] Although the quantization techniques disclosed herein refer to the descriptions of weights and activations, these techniques can be applied to any type of data and are not limited to neural network data. Furthermore, the source code can be formed using software engineering tools based on the descriptions herein, and the instructions compiled from the source code can cause the processor to perform the methods and operations described herein when executed by a processor.

[0128] Figure 10 Examples of dynamic quantization devices according to one or more embodiments are shown. (Refer to...) Figure 10 The dynamic quantization device 1000 may include a bus 1005, a communication interface 1010, a processor 1030, and a memory 1050.

[0129] The communication interface 1010 can receive input data. The input data can be in the form of INT, but is not limited to this.

[0130] Processor 1030 (in fact, possibly one or more processors of different types) obtains statistics for quantization by applying an orthogonal matrix to input data received via communication interface 1010. Processor 1030 determines a scaling factor based on the statistics. Processor 1030 performs scaling on the input data using the scaling factor. Processor 1030 performs scaling on the input data per quantization unit using the scaling factor. Processor 1030 performs quantization on the scaling result. Additionally, processor 1030 can recover the quantized data using the inverse of the scaling factor and the transpose of the orthogonal matrix. Processor 1030 can output the result of multiplying the recovered data by weights.

[0131] The memory 1050 may store various information generated during the processing of the processor 1030. Furthermore, the memory 1050 may store various types of data and programs. The memory 1050 may include volatile or non-volatile memory. The memory 1050 may include a high-capacity storage medium (such as a hard disk) for storing various types of data.

[0132] Furthermore, the processor 1030 can execute the above-mentioned functions as shown in Figure 1 to... Figure 9At least one method or an algorithm corresponding to at least one method is described. Processor 1030 can be a data processing device implemented by a HW (Hardware) having a physical structure for performing the desired operation. For example, the desired operation may include code or instructions included in a program. Processor 1030 may be implemented as, for example, a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processor (NPU). Dynamic quantization device 1000 implemented as an HW may include, for example, a microprocessor, CPU, processor core, multi-core processor, multiprocessor, application-specific integrated circuit (ASIC), and field-programmable gate array (FPGA).

[0133] The processor 1030 can execute programs and control the dynamic quantization device 1000. The code of the program executed by the processor 1030 can be stored in the memory 1050.

[0134] The dynamic quantization device 1000 can be implemented as various types of devices (such as personal computers (PCs), server devices, mobile devices, embedded devices, etc.), and can correspond to, for example, smartphones, tablets, augmented reality (AR) devices, Internet of Things (IoT) devices, and / or medical devices that perform speech recognition, image recognition, image classification, etc., based on neural networks, but examples are not limited thereto. Furthermore, the dynamic quantization device 1000 can correspond to a dedicated HW accelerator installed in the aforementioned devices, or can be an HW accelerator (such as an NPU, tensor processor (TPU), memory arithmetic unit, and / or neural engine as a dedicated module for operating neural networks), but examples are not limited thereto.

[0135] Figure 11 Examples of dynamic quantization devices according to one or more embodiments are shown. (Refer to...) Figure 11 The dynamic quantization device 1100 may include an orthogonal matrix generator 1110, a statistical information extraction circuit 1120, a scaler 1130, a quantizer 1140, a descaler 1150, and a dequantizer 1160.

[0136] The orthogonal matrix generator 1110 can generate (or store and provide) various types of orthogonal matrices (e.g., rotation matrices, Hadamard matrices, or matrices using the SGD method with Cayley transformation). The orthogonal matrix generator 1110 can be a Hadamard matrix generator, but is not limited to it.

[0137] The statistical information extraction circuit 1120 performs calculations by applying an orthogonal matrix generated by the orthogonal matrix generator 1110 to the input data, and can detect statistical information from the calculation results. The statistical information extraction circuit 1120 can detect statistical information corresponding to the quantization unit from the calculation results. Depending on the type of statistical information, the statistical information extraction circuit 1120 may include, for example, a maximum value detector, an average value detector, and / or a variable operator.

[0138] Scaler 1130 can perform dynamic scaling by determining a scaling factor based on statistical information. Scaler 1130 can also perform dynamic scaling by selectively using different means depending on the type of scaling factor. Scaler 1130 may include an FP (Optical Processing) arithmetic unit 1133 and a shifter 1136. For example, when the scaling factor is in FP form, scaler 1130 can perform scaling on the input data using FP arithmetic unit 1133. Alternatively, when the scaling factor is in INT form, scaler 1130 can perform scaling on the input data using shifter 1136.

[0139] The FP arithmetic unit 1133 and / or shifter 1136 can perform scaling on the input data for each quantization unit.

[0140] Quantizer 1140 can perform quantization on the scaling result of scaler 1130.

[0141] Descaler 1150 generates the inverse of the scaling factor and performs descaling by multiplying the quantization result by the inverse of the scaling factor.

[0142] Dequantizer 1160 generates the transpose of an orthogonal matrix and recovers the quantized data by multiplying the descaling result of descaler 1150 by the transpose of the orthogonal matrix. Dequantizer 1160 can then multiply the recovered data by weights and output the result.

[0143] Regarding Figure 1 to... Figure 11The described computing devices, electronic devices, processors, memories, information output systems and hardware, storage devices, and other devices, apparatuses, units, modules, and components are implemented by or represent hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by a processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. The hardware component can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms "processor" or "computer" are used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components, may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. The hardware components can have any one or more different processing configurations, examples of which include: single processor, discrete processor, parallel processor, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0144] Figure 1 to Figure 11The methods for performing the operations described in this application, as shown, are executed by computing hardware (e.g., one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations, may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation, or two or more operations.

[0145] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written in any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding description herein, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0146] Instructions or software used to control computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drives (HDDs), solid-state drives (SSDs), card storage devices (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state drives, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner, and to provide instructions or software and any associated data, data files, and data structures to one or more processors or computers, such that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system, such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0147] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered descriptive only and not for limiting purposes. The description of features or aspects in each example should be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents.

[0148] Therefore, in addition to the above disclosure, the scope of the disclosure may also be defined by the claims and their equivalents, and all variations within the scope of the claims and their equivalents shall be interpreted as included in the disclosure.

Claims

1. A text processing method, the text processing method being executed by a computing device and comprising: Text data is input into a large language model and activation data is generated as input data through at least one layer of the large language model, wherein the large language model is configured to generate answers corresponding to the text data based on the text data; Orthogonal matrices are applied to the input data, and statistical information for quantization is obtained based on the results of applying orthogonal matrices to the input data. Determine the scaling factor based on statistical information; Scaling of the input data is performed using a scaling factor; Perform dynamic quantization on the scaled input data; and Answers corresponding to text data are generated through subsequent layers of at least one layer of a large language model, based on dynamically quantized and scaled input data.

2. The text processing method as described in claim 1, wherein, Generating an orthogonal matrix, wherein the step of applying the orthogonal matrix to the input data includes multiplying the orthogonal matrix by the input data, wherein statistical information is obtained by determining a statistic of the result of the multiplication.

3. The text processing method as described in claim 1, wherein, The statistical information includes statistics of the results of applying an orthogonal matrix to the input data, wherein the statistics include at least one of range, mean, maximum, minimum and variable.

4. The text processing method as described in claim 1, wherein, An orthogonal matrix is ​​formed by {-1, 0, 1}.

5. The text processing method as described in claim 1, wherein, The input data includes quantization units, and the step of determining the scaling factor includes: determining the scaling factor differently for each quantization unit.

6. The text processing method as described in claim 1, wherein, The steps involved in performing scaling include: Determine whether the input data is in floating-point or integer form, and selectively use a floating-point arithmetic unit or a shifter to scale the input data based on the determined form of the input data.

7. The text processing method as described in claim 1, wherein, The input data includes quantization units, and the scaling step includes performing scaling for each quantization unit.

8. The text processing method as described in claim 7, wherein, The scaling steps performed for each quantization unit include: Since the quantization unit is a channel unit, scaling is performed through operations between the scalar value corresponding to the channel unit and the scalar value corresponding to the input data; and Since the quantization unit is a block unit, scaling is performed through operations between vector values ​​corresponding to the block unit and scalar values ​​corresponding to the input data.

9. The text processing method as described in claim 8, wherein, Each quantization unit is a channel unit, block unit, or tensor unit.

10. The text processing method according to any one of claims 1 to 9, wherein, The step of performing dynamic quantization on the scaled input data is based on a target quantization value set by the user or automatically by the algorithm.

11. The text processing method as described in claim 1, further comprising: The dynamically quantized scaled input data is recovered using the inverse of the scaling factor and the transpose of the orthogonal matrix. The step of generating the answer corresponding to the text data includes: generating the answer corresponding to the text data based on the recovered, dynamically quantized, and scaled input data through a subsequent layer of at least one layer of a large language model.

12. The text processing method as described in claim 11, wherein, The steps to recover dynamically quantized and scaled input data include: The first recovered dynamically quantized scaled input data is calculated by applying the reciprocal of the scaling factor to the dynamically quantized scaled input data; and Multiply the transpose of the orthogonal matrix by the scaled input data after the first recovery and dynamic quantization.

13. The text processing method as described in claim 11, wherein, The transpose and weights of orthogonal matrices are pre-calculated offline.

14. The text processing method according to any one of claims 1 to 13, further comprising: In response to the input data being in floating-point form, the input data in floating-point form is converted to integer form before applying the orthogonal matrix to the input data.

15. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the text processing method as described in any one of claims 1 to 14.

16. A text processing device, comprising: One or more processors; as well as Memory, storage instructions, said instructions being configured to cause the one or more processors to: Text data is input into a large language model and activation data is generated as input data through at least one layer of the large language model, wherein the large language model is configured to generate answers corresponding to the text data based on the text data; Statistical information for quantization is obtained by applying orthogonal matrices to the input data; Determine the scaling factor based on statistical information; Scaling of the input data is performed using a scaling factor; Perform dynamic quantization on the scaled input data; and Answers corresponding to text data are generated through subsequent layers of at least one layer of a large language model, based on dynamically quantized and scaled input data.

17. The text processing device of claim 16, wherein, An orthogonal matrix is ​​generated, wherein applying the orthogonal matrix to the input data processing includes multiplying the orthogonal matrix by the input data, and wherein statistical information is obtained by determining the statistics of the result of the multiplication.

18. The text processing device of claim 16, wherein, The instructions are also configured to cause the one or more processors to: determine whether the input data is in floating-point or integer form, and to perform scaling on the input data by selectively using a floating-point arithmetic unit or a shifter according to the form of the determined input data.

19. The text processing device of claim 16, wherein, The input data includes quantization units, the scaling factor is determined differently for each quantization unit, and each quantization unit is a channel unit, block unit, or tensor unit.

20. A text processing method, the text processing method comprising: Text data is input into a large language model by one or more processors and activation data is generated as input data through at least one layer of the large language model, wherein the large language model is configured to generate answers corresponding to the text data based on the text data; Store the input data in memory; The input matrix is ​​generated by the one or more processors by multiplying the input data by an orthogonal matrix; The reduction factor is determined by the one or more processors based on statistics obtained from the input matrix; The input matrix is ​​reduced by the one or more processors using a reduction factor; The quantized input matrix is ​​generated by the one or more processors by reducing the bit size of the elements of the reduced input matrix; The quantized input matrix is ​​amplified by the one or more processors using the reciprocal of the scaling factor; The one or more processors generate a quantized version of the input data by applying the transpose of an orthogonal matrix to the amplified and quantized input matrix; and The one or more processors generate answers corresponding to text data based on a quantized version of the input data through subsequent layers of at least one layer of a large language model.

Images