Large language model reasoning method and device, electronic equipment and storage medium

Abstract

The application provides a large language model reasoning method and device, electronic equipment and storage medium. The large language model comprises at least one Transformer architecture, and the Transformer architecture comprises a transformation module, a block quantization module, an Attention module and an FNN module. The method comprises the following steps: performing Hadamard transformation on input data through the transformation module to obtain transformed data; performing block quantization processing on the transformed data through the block quantization module to obtain quantized data; processing the quantized data and the quantized weight of the large language model through the Attention module to obtain a first processing result; and processing the first processing result through the FNN module to obtain an output result. According to the application, the input data is subjected to Hadamard transformation, and then the transformed data is subjected to block quantization, so that the influence of abnormal features in the large language model on reasoning accuracy is reduced, thereby improving the reasoning efficiency of the large language model while ensuring the accuracy of the large language model reasoning.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a method, apparatus, electronic device, and storage medium for reasoning using a large language model. Background Technology

[0002] With the increasing demand for large-scale inference models in consumer applications, reducing the cost of large-scale inference while ensuring its performance has become crucial for its widespread adoption. Various solutions have been developed to reduce the cost of large-scale inference, such as quantization techniques, sparsity methods, and key-value cache compression. However, the application of these techniques inevitably leads to a loss of accuracy. Anomalies are an important feature, and commonly used quantization techniques, such as GPTQ, LLM.int8(), and SmoothQuant, often do not handle anomalies well enough, resulting in performance degradation. Key-value cache compression techniques, such as steaming LLM and H2O, and quantization methods often perform poorly in long-context tests, leading to low accuracy in large language model inference. Summary of the Invention

[0003] The purpose of this application is to provide a large language model reasoning method, apparatus, electronic device, and storage medium to improve the reasoning speed of large language models while ensuring reasoning accuracy.

[0004] In a first aspect, embodiments of this application provide a large language model inference method. The large language model includes at least one Transformer architecture, which includes a transformation module, a block quantization module, an attention mechanism module, and a feedforward neural network (FNN) module. The method includes:

[0005] The input data is transformed using a transformation module to obtain the transformed data.

[0006] The transformed data is processed by block quantization module to obtain quantized data.

[0007] The Attention module processes the quantized data and the quantized weights of the large language model to obtain the first processing result; the quantized weights are obtained by performing block quantization on the weights of the large language model.

[0008] The FNN module processes the first processing result to obtain the output result.

[0009] This application embodiment reduces the impact of abnormal features on the inference accuracy of large language models by performing Hadamard transformation on the input data and then performing block quantization on the transformed data. Therefore, it improves the inference efficiency of large language models while ensuring the accuracy of inference.

[0010] In any embodiment, the transformed data is subjected to block quantization processing by a block quantization module to obtain quantized data, including:

[0011] The transformed data is divided into blocks according to the preset block division rules to obtain multiple blocks of data;

[0012] Quantization calculation is performed on each data block to obtain the quantized data block and the corresponding scaling factor; multiple quantized data blocks constitute the quantized data.

[0013] This application embodiment divides the input data of the large language model into blocks, so that the abnormal features in the large language model only affect the block in which they are located, thereby reducing the impact on the entire large language model and improving the accuracy of the large language model inference.

[0014] In any embodiment, quantization calculation is performed on each data block, including:

[0015] According to the formula Each data block undergoes quantization calculation;

[0016] Where Q(X) represents the quantized block data; X represents a block of data; S X S is the scaling factor. X =(max(abs(X)) / (2 B-1 -1); B is the maximum number of bits used by the scaling factor; Used for characterization The range of values ​​is [-2]. B-1 +1,2 B-1 -1].

[0017] This application embodiment reduces the computational load of matrix multiplication during inference by quantizing the data after it is segmented, thereby improving inference efficiency.

[0018] In any embodiment, the multiple data blocks include multiple levels of data blocks, and the data blocks of a lower level are obtained by further dividing the data blocks of the previous level.

[0019] The embodiments of this application employ multi-level block processing, which makes the data involved in model inference calculations more granular, further reducing the requirements for computing resources, storage, and communication, and improving model inference efficiency.

[0020] In any embodiment, the quantized weights include the first quantized weights in the Attention module; the Attention module processes the quantized data and the quantized weights of the large language model to obtain a first processing result, including:

[0021] The weights after the first quantization are inversely transformed using the inverse transform matrix corresponding to the Hadamard transform to obtain the weights after the first inverse transform.

[0022] The quantized data is multiplied by the weights after the first inverse transformation to obtain the first processing result.

[0023] This application embodiment uses an inverse transformation matrix to perform an inverse transformation on the first quantized weights, restoring the resulting data from the Hadamard domain to the original data domain.

[0024] In any embodiment, the quantized weights further include second quantized weights corresponding to the FNN module; the output result is obtained by processing the first processing result through the FNN module, including:

[0025] Perform a Hadamard transformation on the first processing result to obtain the transformed first processing result;

[0026] The transformed first processing result is quantized to obtain the quantized first processing result.

[0027] The weights after the second quantization are inversely transformed using the inverse transform matrix corresponding to the Hadamard transform to obtain the weights after the second inverse transform.

[0028] The first quantized result is multiplied by the weights after the second inverse transformation to obtain the second result.

[0029] The second processing result is processed by the activation layer in the FNN module to obtain the output result.

[0030] This application embodiment reduces the impact of abnormal features on the inference accuracy of large language models by performing Hadamard transformation on the input data and then performing block quantization on the transformed data. Therefore, it improves the inference efficiency of large language models while ensuring the accuracy of inference.

[0031] Secondly, embodiments of this application provide a large language model inference apparatus, wherein the large language model includes at least one Transformer architecture; the apparatus includes:

[0032] The transformation module is used to perform Hadamard transformation on the input data to obtain the transformed data.

[0033] The block quantization module is used to perform block quantization processing on the transformed data to obtain quantized data.

[0034] The Attention module is used to process the quantized data and the quantized weights of the large language model to obtain the first processing result; the quantized weights are obtained by performing block quantization on the weights of the large language model.

[0035] The FNN module is used to process the first processing result to obtain the output result.

[0036] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a bus, wherein,

[0037] The processor and the memory communicate with each other via the bus;

[0038] The memory stores program instructions that can be executed by the processor, and the processor can execute the method of the first aspect by calling the program instructions.

[0039] In any embodiment, the processor includes a data reading unit, a multiply-accumulate unit, and a post-processing unit; wherein:

[0040] The data reading unit is used to read the quantized data and corresponding scaling factor required by the multiply-accumulate unit from the memory;

[0041] The multiply-accumulate unit is used to perform multiply-accumulate calculations on the quantized data and the scaling factor.

[0042] The post-processing unit is used to perform inverse quantization on the multiplication-accumulation calculation results.

[0043] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium, comprising:

[0044] The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the method of the first aspect.

[0045] Fifthly, embodiments of this application provide a computer program product, including computer program instructions, which, when read and executed by a processor, perform the method of the first aspect.

[0046] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This application provides a schematic flowchart of a large language model reasoning method.

[0049] Figure 2 A schematic diagram illustrating the storage of quantized data and scaling factor provided in an embodiment of this application;

[0050] Figure 3 This is a schematic diagram of block quantization data provided in an embodiment of this application;

[0051] Figure 4 This is a schematic diagram of a large language model structure provided in an embodiment of this application;

[0052] Figure 5 A schematic diagram of a large language model inference device provided in an embodiment of this application;

[0053] Figure 6 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application;

[0054] Figure 7 A schematic diagram of a processor structure provided in an embodiment of this application;

[0055] Figure 8 A schematic diagram of the multiply-accumulate circuit for the block quantized data and scaling factor structure provided in the embodiments of this application;

[0056] Figure 9 This is a schematic diagram of a block quantized data and scaling factor structure multiplication and addition method provided in an embodiment of this application. Detailed Implementation

[0057] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0059] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0060] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0061] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0062] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0063] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0064] In recent years, large language models based on the Transformer architecture have demonstrated outstanding performance in multiple fields. These models have extremely large parameter scales, are well-suited for understanding and generating natural language, and are applicable to various downstream applications such as machine translation, content summarization, and sentiment analysis. However, the inference process of large language models has extremely high storage, bandwidth, and computing power requirements, limiting the number of users that service providers can serve simultaneously.

[0065] To reduce the inference cost of large models, techniques such as quantization, sparsity reduction, and key-value cache compression can be used. However, the application of these techniques inevitably leads to a loss of accuracy. This is mainly because anomalous features are important in large language models, and commonly used quantization techniques often cannot handle anomalous features well, thus causing a decline in the inference performance of large models.

[0066] To address the aforementioned technical problems, embodiments of this application provide a method, apparatus, electronic device, and storage medium for large language model inference. For the quantization of anomalous features, a block quantization strategy based on Hadamard transform is proposed, as detailed in the following embodiments.

[0067] It is understood that the large language model inference method provided in this application embodiment can be applied to electronic devices, including terminals and servers; wherein the terminal can specifically be a smartphone, tablet computer, computer, personal digital assistant (PDA), etc.; the server can specifically be an application server or a web server.

[0068] Figure 1 This is a schematic diagram of a large language model reasoning method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes:

[0069] Step 101: Perform Hadamard transformation on the input data using the transformation module to obtain the transformed data;

[0070] Step 102: Perform block quantization processing on the transformed data using the block quantization module to obtain the quantized data;

[0071] Step 103: Process the quantized data and the quantized weights of the large language model through the Attention module to obtain the first processing result; the quantized weights are obtained by performing block quantization on the weights of the large language model.

[0072] Step 104: Process the first processing result using the FNN module to obtain the output result.

[0073] In practical implementation, large language models are a type of deep learning model with a large number of parameters. They learn language patterns, syntax, and semantics by processing large amounts of text data, thereby understanding and generating human language. The large language model provided in this application embodiment can be applied to various scenarios, including but not limited to: machine translation, chatbots and AI assistants, automatic writing and text generation, information extraction, sentiment analysis, speech recognition, and social media monitoring. Furthermore, the large language model is based on the Transformer architecture and uses complex algorithms to analyze and process text data. A large language model can include multiple cascaded Transformer architectures. Each Transformer architecture mainly includes a transformation module, a block quantization module, an attention module, and an FNN module. In this application embodiment, steps 101-104 describe the data processing flow for one Transformer architecture. The output of one Transformer architecture is input into the next cascaded Transformer architecture until the last Transformer architecture outputs its result, which is the final result.

[0074] The transformation module performs Hadamard transformation on the input data. It should be noted that the input data can be embedded and mapped text content input by the user, the output of the previous Transformer architecture, or activations from a large language model during inference. The Hadamard transformation is a transformation method based on the Hadamard matrix. A Hadamard matrix is ​​an orthogonal square matrix composed of +1 and -1 elements, where any two rows (or two columns) are orthogonal. The elements in the Hadamard transformation matrix only contain +1 and -1, which simplifies the calculation process by involving only addition and subtraction operations, eliminating the need for multiplication. Furthermore, the Hadamard transformation possesses a certain degree of orthogonality, which helps preserve some important characteristics of the signal during the transformation process.

[0075] The Hadamard transform can smooth out anomalous features in input data, thus facilitating quantization. The Hadamard transform can include stochastic Hadamard transforms or Hadamard rotation matrices formed by learnable methods.

[0076] In this embodiment of the application, a Hadamard transformation is performed before each matrix multiplication calculation quantization during the large language model inference process.

[0077] The block quantization module is used to perform block quantization processing on the transformed data. The transformed data is in the form of a matrix, and it can be divided by row, column, or block. It can also be divided according to the channels of the large language model. The specific division method can be determined according to actual needs. Block division refers to using a smaller matrix, for example, dividing the transformed data into multiple 50*50 matrices. After block division, each block is quantized. The purpose of quantization is to convert high-precision data to low-precision data, thereby reducing computational load. For example, converting high-precision float16 data format to int8 data format. It should be noted that the weights corresponding to the large language model can also be pre-quantized in blocks, and the block quantization method can also be block quantization by row, column, block, or channel.

[0078] The Attention module is used to perform masking and self-attention calculations on the quantized data. During self-attention calculation, the quantized data and the quantized weights of the large language model are processed to obtain the first processing result. Specifically, based on the calculated attention weights, the values ​​are weighted and summed to obtain the output representation for each position.

[0079] The FNN module may include at least one fully connected layer (i.e., an FC layer). The first processing result output from the Attention module is input into the FNN module. In the FNN module, the first processing result is first subjected to a Hadamard transformation, and then multiplied with the weights of the inverse FC neural network to obtain the output of the FC layer. This output is then fed into the activation layer to obtain the final output result. It should be noted that if this Transformer architecture is the last layer, then the final output result is the result to be returned to the user; if this Transformer architecture is not the last layer, then the final output result needs to be input into the next Transformer architecture, and steps 101-104 need to be executed again.

[0080] This application embodiment reduces the impact of abnormal features on the inference accuracy of large language models by performing Hadamard transformation on the input data and then performing block quantization on the transformed data. Therefore, it improves the inference efficiency of large language models while ensuring the accuracy of inference.

[0081] Based on the above embodiments, the transformed data is processed by a block quantization module to obtain quantized data, including:

[0082] The transformed data is divided into blocks according to the preset block division rules to obtain multiple blocks of data;

[0083] Quantization calculation is performed on each data block to obtain the quantized data block and the corresponding scaling factor; multiple quantized data blocks constitute the quantized data.

[0084] In practice, data partitioning includes partitioning by block, column, or row, with each block corresponding to a scalar scaling factor. For example, if the original data is a 100x100 matrix, partitioning by block means treating each row vector as a block, resulting in 100 blocks. Similarly, partitioning by column means treating each column vector as a block, also yielding 100 blocks. Alternatively, data can be partitioned by block, for instance, dividing the original data into 25 20x20 blocks.

[0085] After data is segmented, each segment is quantized to obtain the quantized segment data and its corresponding scaling factor. Essentially, data quantization is a scaling process. For example, 1000 values ​​in the range [0, 255] can be compressed to the range [0, 1] by subtracting the mean from all values ​​and then multiplying by the variance. This process facilitates low-precision data representation.

[0086] It should be noted that after the transformed data is divided into blocks and quantized, the quantized data can be stored. During storage, the quantized data is stored in correspondence with the block index of each block. This allows for quick location and extraction of the quantized data and its corresponding scaling factor when reading the quantized data, using the corresponding block index. After the transformed data is divided into blocks and quantized, the quantized data and its corresponding scaling factor can be stored, such as... Figure 2 As shown, a Block is a piece of data (specifically, it can be a block, a row, or a column vector), and each Block corresponds to a scalar scaling factor (Scale). Blocks and Scales are stored in a cache in a one-to-one correspondence, and access to Blocks and Scales is performed synchronously.

[0087] This application embodiment divides the input data of the large language model into blocks, so that the abnormal features in the large language model only affect the block in which they are located, thereby reducing the impact on the entire large language model and improving the accuracy of the large language model inference.

[0088] Based on the above embodiments, quantization calculations are performed on each data block, including:

[0089] According to the formula Each data block undergoes quantization calculation;

[0090] Where Q(X) represents the quantized block data; X represents a block of data; SX S is the scaling factor. X =(max(abs(X)) / (2 B-1 -1); B is the maximum number of bits used by the scaling factor; Used for characterization The range of values ​​is [-2]. B-1 +1,2 B-1 -1]. Used to characterize rounding down.

[0091] It should be noted that, in order to accelerate the calculation of the scaling sparsity of the block quantization strategy, which is an exponential power of 2, the multiplication and division operations are converted into shift operations.

[0092] Furthermore, large language model quantization can reduce the computational cost of GEMM (Generic Matrix Multiplication). Data is quantized before the GEMM operator, the quantized data undergoes GEMM operations, and after computation, it can be dequantized and then fed into the next layer, such as the activation layer. The dequantization method for block quantization is as follows:

[0093]

[0094] Among them, S X It is in the form of a power of 2, Q(X)·S X Multiplication is simplified to bit shifting, Z X This indicates the bias in quantization.

[0095] This application embodiment reduces the computational load of matrix multiplication during inference by quantizing the data after it is segmented, thereby improving inference efficiency.

[0096] Based on the above embodiments, the multiple data blocks include multiple levels of data blocks, and the data blocks of the next level are obtained by further dividing the data blocks of the previous level.

[0097] In practical implementation, when dividing data into blocks, a multi-level block method can be adopted, where the block matrix W, along with the scaling factor, is represented as follows: in, This is the first-level quantization data for W. The first-level scaling factor is used; the second-level scaling factor is used to divide the single-machine block matrix. Further divide into blocks

[0098] For example, given an original 100x100 matrix, using a two-level block approach, the first level consists of 25 20x20 matrices. Each 20x20 block shares a scaling factor, resulting in 25 scaling factors for the first level. The second level further divides each 20x20 block into 25 4x4 blocks, and then assigns 25 scaling factors to each of the second-level blocks. If the original data consists of 100x100 float16 data points, the first-level block scaling factor has an int8 precision, and each sub-block uses fp8 data format, after first-level block quantization, the data size is compressed to approximately 50% of the original. Figure 3 This is a schematic diagram of block quantization data provided in an embodiment of this application, such as... Figure 3 As shown. The original data is high-precision float16 data, which can be divided into blocks using some method, such as by row, by column, or by block. Each block contains 0 elements. i After single-level quantization, each block of elements shares a scaling factor s, and the elements in each block after single-level quantization are... Original data O i High-precision numerical formats such as float16 are used, and the shared scaling factor after block quantization is int8. The quantized data can also use lower-precision numerical formats, such as float4, binary, ternary, etc. In the embodiments of this application, the quantized block data can use low-precision numerical formats such as binary, ternary, int2, int4, fp4, fp6, etc.; the scaling factor corresponding to the block data is stored in data formats such as int8. The low-precision numerical format significantly reduces the requirements for computing resources, storage, and communication, thereby accelerating the inference of large models.

[0099] Based on the above embodiments, the quantized weights include the first quantized weights in the Attention module; the Attention module processes the quantized data and the quantized weights of the large language model to obtain a first processing result, including:

[0100] The weights after the first quantization are inversely transformed using the inverse transform matrix corresponding to the Hadamard transform to obtain the weights after the first inverse transform.

[0101] The quantized data is multiplied by the weights after the first inverse transformation to obtain the first processing result.

[0102] The quantized weights also include the second quantized weights corresponding to the FNN module; the output results are obtained by processing the first processing result through the FNN module, including:

[0103] Perform a Hadamard transformation on the first processing result to obtain the transformed first processing result;

[0104] The transformed first processing result is quantized to obtain the quantized first processing result.

[0105] The weights after the second quantization are inversely transformed using the inverse transform matrix corresponding to the Hadamard transform to obtain the weights after the second inverse transform.

[0106] The first quantized result is multiplied by the weights after the second inverse transformation to obtain the second result.

[0107] The second processing result is processed by the activation layer in the FNN module to obtain the output result.

[0108] In the specific implementation process Figure 4 This is a schematic diagram of a large language model structure provided in an embodiment of this application, such as... Figure 4 As shown, before inference of the large model, the weights of the large model are first quantized using block quantization, forming the block quantization parameters of the neural network weights W of the large model and a scaling factor shared by the block quantization parameters. The quantization parameters can be in low-precision numerical format, and the scaling factor is int8. The inference process of the large model is as follows:

[0109] 1. The user's input question is vectorized into the first-layer Transformer structure X through the Embedding layer;

[0110] 2.1 Multiply the input X by the locally generated Hadamard matrix to obtain XH;

[0111] 2.2 Quantize XH into Q(XH) using block quantization. It is understood that the block quantization method is described in the above embodiment and will not be repeated here.

[0112] 2.3. Feed Q(XH) into the Attention module of the Transformer model; in the Attention module, H... T H is obtained by multiplying the weights of the query (Q), key (K), and value (V) projection layers respectively. T W q H T W k and H T W v The Q(XH) result is fed into the Q,K,V projection layers. In the Q,K,V projection layers, Q(XH) is compared with H respectively. T W q H T W k and H T W vMultiply, and feed the output of the Q and K projection layers into the softmax activation layer; multiply, and feed the H... T W v Multiply by a new Hadamard matrix H head H T W v H head It should be noted that the new Hadamard matrix can be generated in advance. Multiply the softmax activation layer output by H. T W v H head The input to the output layer is obtained, and the weights H of the output layer of this Attention layer are... head T W o This yields the output of the Attention module in the Transformer layer. It should be noted that the quantization strategy for the KV cache in the Attention module is as follows: key values ​​are quantized per-channel (column-based), and value values ​​are quantized per-token (row-based).

[0113] 3. The FFN module of the Transformer layer contains two or more FC layers. The following describes the calculation process of a single FC layer. The output X of the Attention module is multiplied by the locally generated Hadamard matrix to obtain XH. XH is then quantized into Q(XH) using a quantization strategy and fed into the first FC layer. The weights W of the FC layer neural network are then... FC Multiplied by H T H T W FC The input Q(XH) of the FC layer is multiplied by H T W FC The output of the FC layer is then fed into the active layer GeLU. The process is repeated for the other layer.

[0114] This large model has L layers of Transformers. The computation process of the other Transformer layers is the same as that of the first Transformer layer. After the L layers of Transformers have completed their computation, the output X is also processed by first performing a Hadamard transform, then quantization, and finally an inverse Hadamard transform to restore the result data from the Hadamard domain to the original data domain.

[0115] The above process obtains the first character of the user's question and answer; the output of other characters is similar.

[0116] This application embodiment performs Hadamard transformation on the weights or activations of each neural network layer in the Transformer architecture. The matrix multiplication operation neural network layer in the Transformer architecture accounts for a large portion of the Transformer's latency; therefore, the multi-level block quantization strategy of this application embodiment is adopted for neural network layers involving matrix multiplication. Due to the influence of anomalous features, in the standard Transformer architecture, neural network layers involving matrix multiplication operations undergo Hadamard transformation before using the quantization strategy.

[0117] Projecting layer X with q q W q Taking the calculation of X as an example, the others are similar. q W q It's a matrix operation layer, designed to reduce matrix multiplication and addition X. q W q The computational and data volume necessitates the use of low-precision numerical formats for X. q and W q Quantization, regardless of the quantization strategy, will inevitably lead to a loss of accuracy, especially for X. q and W q To mitigate the loss caused by quantization due to the presence of anomalous features, a Hadamard transformation is performed on activation X before quantization. Because the Hadamard transformation matrix has the property H·H T =I, then:

[0118] XW = XIW = XH·H T W

[0119] Therefore, when introducing the Hadamard transform into the input activation X, an inverse Hadamard transform needs to be performed to restore the result data from the Hadamard domain to the original data.

[0120] This application embodiment uses an inverse transformation matrix to perform an inverse transformation on the first quantized weights, restoring the resulting data from the Hadamard domain to the original data domain.

[0121] Figure 5 This is a schematic diagram of a large language model inference device provided in an embodiment of this application. The device can be a module, program segment, or code on an electronic device. It should be understood that this device is similar to the one described above. Figure 1 The method implementation corresponds to this and can be executed. Figure 1 The specific functions of the device involved in the method embodiment can be found in the description above; to avoid repetition, detailed descriptions are omitted here. The device includes: a transformation module 501, a block quantization module 502, an attention module 503, and an FNN module 504, wherein:

[0122] Transformation module 501 is used to perform Hadamard transformation on the input data to obtain the transformed data;

[0123] Block quantization module 502 is used to perform block quantization processing on the transformed data to obtain quantized data;

[0124] Attention module 503 is used to process the quantized data and the quantized weights of the large language model to obtain the first processing result; the quantized weights are obtained by performing block quantization on the weights of the large language model.

[0125] FNN module 504 is used to process the first processing result to obtain the output result.

[0126] Based on the above embodiments, the block quantization module 502 is specifically used for:

[0127] The transformed data is divided into blocks according to a preset block division rule to obtain multiple block data;

[0128] Each of the data blocks is quantized to obtain the quantized data block and the corresponding scaling factor; wherein, multiple quantized data blocks constitute the quantized data.

[0129] Based on the above embodiments, the block quantization module 502 is specifically used for:

[0130] According to the formula Each of the data blocks is quantized.

[0131] Where Q(X) is the quantized block data; X is a block of data; S X S is the scaling factor. X =(max(abs(X)) / (2 B-1 -1); B is the maximum number of bits used by the scaling factor; Used for characterization The range of values ​​is [-2]. B-1 +1,2 B-1 -1].

[0132] Based on the above embodiments, the multiple data blocks include multiple levels of data blocks, and the next level of data blocks is obtained by further dividing the data blocks of the previous level.

[0133] Based on the above embodiments, the quantized weights include the first quantized weights in the Attention module; the Attention module 503 is specifically used for:

[0134] The first quantized weights are inversely transformed using the inverse transform matrix corresponding to the Hadamard transform to obtain the first inverse-transformed weights.

[0135] The quantized data is multiplied by the weights after the first inverse transformation to obtain the first processing result.

[0136] Based on the above embodiments, the quantized weights further include second quantized weights corresponding to the FNN module; the FNN module 504 is specifically used for:

[0137] Perform a Hadamard transformation on the first processing result to obtain the transformed first processing result;

[0138] The transformed first processing result is quantized to obtain the quantized first processing result;

[0139] The second quantized weights are inversely transformed using the inverse transform matrix corresponding to the Hadamard transform to obtain the second inversely transformed weights.

[0140] The first quantized result is multiplied by the second inverse-transformed weight to obtain the second result.

[0141] The second processing result is processed by the activation layer in the FNN module to obtain the output result.

[0142] Figure 6 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this application, such as... Figure 6 As shown, the electronic device includes: a processor 601, a memory 602, and a bus 603; wherein,

[0143] The processor 601 and the memory 602 communicate with each other through the bus 603;

[0144] The processor 601 is used to call program instructions in the memory 602 to execute the methods provided in the above-described method embodiments, including, for example: performing a Hadamard transformation on the input data through the transformation module to obtain transformed data; performing block quantization processing on the transformed data through the block quantization module to obtain quantized data; processing the quantized data and the quantized weights of the large language model through the Attention module to obtain a first processing result; the quantized weights are obtained by performing block quantization processing on the weights of the large language model; and processing the first processing result through the FNN module to obtain an output result.

[0145] Processor 601 can be an integrated circuit chip with signal processing capabilities. The processor 601 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0146] The memory 602 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0147] Based on the above embodiments, the processor includes a data reading unit, a multiply-accumulate unit, and a post-processing unit; wherein:

[0148] The data read unit is used to read the quantized data and corresponding scaling factor required by the multiply-accumulate unit from memory;

[0149] The multiply-accumulate unit is used to perform multiply-accumulate calculations on the quantized data and the scaling factor;

[0150] The post-processing unit is used to perform dequantization on the results of multiplication and addition.

[0151] Figure 7 This is a schematic diagram of a processor structure provided in an embodiment of this application. Figure 8 A schematic diagram of the multiply-accumulate circuit for the block quantized data and scaling factor structure provided in the embodiments of this application is shown below. Figure 7 , Figure 8As shown, the data reading unit includes a controller, an address generator, and a cache manager. The controller loads the data required by the calculator from storage, including quantized block data and corresponding scaling factor data, into the cache manager. The address generator generates the address for each block data storage. In the cache manager, data is also stored in block quantization and corresponding scaling factor format. The two matrices in the cache (matrix A corresponding to the weights and matrix B corresponding to the activation) and the weight scaling factor (SA) and activation scaling factor (SB) are fed into a multiply-accumulator for multiplication and addition operations. The multiplication and addition operations of A and B are performed using the common GEMM operation method for block quantized data. The results of A*B = C and scaling factor SC = SA*SB are stored respectively. Then, C*SC is calculated. Since SA, SB, and SC are all powers of 2, C*SC is a shift operation on C, thus realizing the multiplication operation of the block quantized matrix. The multiplication result is sent to the post-processing module for dequantization, which involves shifting each data point of C according to the scaling factor SA*SB. Figure 8 This is the multiply-accumulate computation accelerator section of this application embodiment. The red arrow represents the neural network weight data W, and the blue arrow represents the input activation data X. Here, W and X are the values ​​after block quantization, and therefore include the quantized values ​​and scaling factors.

[0152] Figure 9 This application provides a schematic diagram of a method for multiplying and adding block-quantized data and scaling factor structures, as illustrated in an embodiment. Figure 9 As shown. A11 is a 16x16 matrix, and all elements of A11 share a scaling factor scalar.

[0153] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can perform the methods provided in the above-described method embodiments, such as: performing a Hadamard transformation on the input data through the transformation module to obtain transformed data; performing block quantization processing on the transformed data through the block quantization module to obtain quantized data; processing the quantized data and the quantized weights of the large language model through the Attention module to obtain a first processing result; wherein the quantized weights are obtained by performing block quantization processing on the weights of the large language model; and processing the first processing result through the FNN module to obtain an output result.

[0154] This embodiment provides a non-transitory computer-readable storage medium storing computer instructions that cause the computer to execute the methods provided in the above-described method embodiments. These instructions include, for example: performing a Hadamard transformation on the input data using the transformation module to obtain transformed data; performing block quantization processing on the transformed data using the block quantization module to obtain quantized data; processing the quantized data and the quantized weights of the large language model using the Attention module to obtain a first processing result; wherein the quantized weights are obtained by performing block quantization processing on the weights of the large language model; and processing the first processing result using the FNN module to obtain an output result.

[0155] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0156] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0158] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0159] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A reasoning method for large language models, characterized in that, The large language model includes at least one Transformer architecture, which comprises a transformation module, a block quantization module, an attention mechanism module, and a feedforward neural network (FNN) module; the method includes: The input data is transformed by the transformation module to obtain the transformed data; the transformation is based on the transformation method of the Hadamard matrix, which is an orthogonal square matrix composed of +1 and -1 elements. The transformed data is processed by the block quantization module to obtain quantized data. The Attention module processes the quantized data and the quantized weights of the large language model to obtain a first processing result; the quantized weights are obtained by performing block quantization on the weights of the large language model. The FNN module processes the first processing result to obtain the output result; The step of performing block quantization processing on the transformed data through the block quantization module to obtain quantized data includes: The transformed data is divided into blocks according to a preset block division rule to obtain multiple block data; Each of the data blocks is quantized to obtain quantized data blocks and corresponding scaling factors; wherein, multiple quantized data blocks constitute the quantized data.

2. The method according to claim 1, characterized in that, The quantization calculation for each of the data blocks includes: According to the formula Each of the data blocks is quantized. in, The quantized block data; For a block of data; Scaling factor ; The maximum number of bits used for the scaling factor; Used for characterization The range of values ​​is .

3. The method according to claim 1, characterized in that, The multiple data blocks include multiple levels of data blocks, and the data blocks of a lower level are obtained by further dividing the data blocks of the previous level.

4. The method according to claim 1, characterized in that, The quantized weights include the first quantized weights in the Attention module; the process of processing the quantized data and the quantized weights of the large language model through the Attention module to obtain the first processing result includes: The first quantized weights are inversely transformed using the inverse transform matrix corresponding to the Hadamard transform to obtain the first inverse-transformed weights. The quantized data is multiplied by the weights after the first inverse transformation to obtain the first processing result.

5. The method according to claim 1, characterized in that, The quantized weights also include the second quantized weights corresponding to the FNN module; The step of processing the first processing result through the FNN module to obtain the output result includes: Perform a Hadamard transformation on the first processing result to obtain the transformed first processing result; The transformed first processing result is quantized to obtain the quantized first processing result; The second quantized weights are inversely transformed using the inverse transform matrix corresponding to the Hadamard transform to obtain the second inversely transformed weights. The first quantized result is multiplied by the second inverse-transformed weight to obtain the second result. The second processing result is processed by the activation layer in the FNN module to obtain the output result.

6. A large language model reasoning device, characterized in that, The large language model includes at least one Transformer architecture; the device includes: The transformation module is used to perform Hadamard transformation on the input data to obtain the transformed data; the Hadamard transformation is a transformation method based on the Hadamard matrix, which is an orthogonal square matrix composed of +1 and -1 elements; The block quantization module is used to perform block quantization processing on the transformed data to obtain quantized data. The Attention module is used to process the quantized data and the quantized weights of the large language model to obtain a first processing result; the quantized weights are obtained by performing block quantization processing on the weights of the large language model. The FNN module is used to process the first processing result to obtain the output result; The block quantization module is specifically used for: The transformed data is divided into blocks according to a preset block division rule to obtain multiple block data; Each of the data blocks is quantized to obtain quantized data blocks and corresponding scaling factors; wherein, multiple quantized data blocks constitute the quantized data.

7. An electronic device, characterized in that, include: Processor, memory, and bus, among which, The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the method as described in any one of claims 1-5 by calling the program instructions.

8. The electronic device according to claim 7, characterized in that, The processor includes a data reading unit, a multiply-accumulate unit, and a post-processing unit; wherein: The data reading unit is used to read the quantized data and corresponding scaling factor required by the multiply-accumulate unit from the memory; The multiply-accumulate unit is used to perform multiply-accumulate calculations on the quantized data and the scaling factor. The post-processing unit is used to perform inverse quantization on the multiplication-accumulation calculation results.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions, which, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-5.

10. A computer program product, characterized in that, It includes computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-5.

Images