Large language model reasoning method and device, electronic equipment and storage medium

Abstract

The invention provides a big language model reasoning method and device, electronic equipment and a storage medium, a big language model comprises at least one Transform architecture, and the Transform architecture comprises a transformation module, a block quantization module, an Attention module and an FNN module; the method comprises the following steps: carrying out Hadamard transformation on input data through a transformation module to obtain transformed data; performing block quantization processing on the transformed data through a block quantization module to obtain quantized data; processing the quantized data and the quantized weight of the large language model through an Attention module to obtain a first processing result; and processing the first processing result through an FNN module to obtain an output result. According to the embodiment of the invention, the input data is subjected to Hadamard transformation, and then the transformed data is subjected to block quantization, so that the influence of abnormal features in the large language model on the reasoning precision is reduced, and therefore, the reasoning efficiency of the large language model is improved, and meanwhile, the reasoning precision of the large language model is ensured.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a large language model reasoning method, device, electronic device and storage medium. Background Art

[0002] The demand for inference of large models for consumer applications is growing. Reducing the cost of large model inference while ensuring the performance of large model inference has become the key to the implementation of large model inference. There have been a variety of solutions to reduce the cost of large model inference, such as quantization technology, sparse methods, and KV cache compression technology. The application of these technologies will inevitably lead to a loss of accuracy. Abnormal features are an important feature. Commonly used quantization technologies, such as GPTQ, LLM.int8(), SmoothQuant, etc., often do not handle abnormal features well enough, resulting in quantization performance loss. KV cache compression technologies such as steaming LLM, H20 and KV cache quantization methods often perform poorly in long context tests, resulting in low accuracy of large language model inference. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a large language model inference method, device, electronic device and storage medium, so as to improve the inference speed of the large language model while ensuring the inference accuracy.

[0004] In a first aspect, an embodiment of the present application provides a large language model reasoning method, wherein the large language model includes at least one Transformer architecture, and the Transformer architecture includes a transformation module, a block quantization module, an attention mechanism Attention module, and a feedforward neural network FNN module; the method includes:

[0005] Perform Hadamard transformation on the input data through the transformation module to obtain transformed data;

[0006] The transformed data is subjected to block quantization processing by a block quantization module to obtain quantized data;

[0007] The quantized data and the quantized weights of the large language model are processed by the Attention module to obtain a first processing result; the quantized weights are obtained by performing block quantization processing on the weights of the large language model;

[0008] The first processing result is processed by the FNN module to obtain an output result.

[0009] The embodiment of the present application performs Hadamard transform on the input data and then quantizes the transformed data in blocks, thereby reducing the impact of abnormal features in the large language model on the reasoning accuracy. Therefore, while improving the reasoning efficiency of the large language model, the accuracy of the reasoning of the large language model is guaranteed.

[0010] In any embodiment, performing block quantization processing on the transformed data by a block quantization module to obtain quantized data includes:

[0011] Divide the transformed data into blocks according to a preset block rule to obtain a plurality of block data;

[0012] A quantization calculation is performed on each block data to obtain quantized block data and a corresponding scaling factor; wherein a plurality of quantized block data constitute the quantized data.

[0013] The embodiment of the present application processes the input data of the large language model in blocks, so that abnormal features in the large language model only affect the blocks in which they are located, thereby reducing the impact on the entire large language model, thereby improving the accuracy of large language model reasoning.

[0014] In any embodiment, the quantization calculation is performed on each block of data, including:

[0015] According to the formula Each block of data is quantitatively calculated;

[0016] Among them, Q(X) is the quantized block data; X is a block data; S X is the scaling factor, S X =(max(abs(X)) / (2 B-1 -1); B is the maximum number of bits used for the scaling factor; For characterization The value range is [-2 B-1 +1,2 B-1 -1].

[0017] The embodiment of the present application reduces the amount of matrix multiplication calculations during the reasoning process and improves the reasoning efficiency by quantizing the divided data.

[0018] In any embodiment, the plurality of block data include block data of multiple levels, and the block data of a next level is obtained by further blocking the block data of a previous level.

[0019] The embodiment of the present application uses multi-level block processing to make the granularity of data involved in model reasoning calculations smaller, further reducing the demand for computing resources, storage and communication, and improving model reasoning efficiency.

[0020] In any embodiment, the quantized weight includes a first quantized weight in the Attention module; the quantized data and the quantized weight of the large language model are processed by the Attention module to obtain a first processing result, including:

[0021] Performing an inverse transformation on the first quantized weight using an inverse transformation matrix corresponding to the Hadamard transformation to obtain a first inverse transformed weight;

[0022] The quantized data is multiplied by the weight after the first inverse transformation to obtain a first processing result.

[0023] The embodiment of the present application performs an inverse transformation on the first quantized weights by using an inverse transformation matrix, thereby restoring the result data from the Hadamard domain to the original data domain.

[0024] In any embodiment, the quantized weight also includes a second quantized weight corresponding to the FNN module; and the first processing result is processed by the FNN module to obtain an output result, including:

[0025] Performing Hadamard transformation on the first processing result to obtain a transformed first processing result;

[0026] quantizing the transformed first processing result to obtain a quantized first processing result;

[0027] Performing an inverse transformation on the second quantized weight using an inverse transformation matrix corresponding to the Hadamard transformation to obtain a second inverse transformed weight;

[0028] Multiplying the quantized first processing result by the second inverse transformed weight to obtain a second processing result;

[0029] The second processing result is processed by the activation layer in the FNN module to obtain an output result.

[0030] The embodiment of the present application performs Hadamard transform on the input data and then quantizes the transformed data in blocks, thereby reducing the impact of abnormal features in the large language model on the reasoning accuracy. Therefore, while improving the reasoning efficiency of the large language model, the accuracy of the reasoning of the large language model is guaranteed.

[0031] In a second aspect, an embodiment of the present application provides a large language model reasoning device, wherein the large language model includes at least one Transformer architecture; the device includes:

[0032] A transformation module is used to perform Hadamard transformation on the input data to obtain transformed data;

[0033] A block quantization module is used to perform block quantization processing on the transformed data to obtain quantized data;

[0034] An Attention module is used to process the quantized data and the quantized weights of the large language model to obtain a first processing result; the quantized weights are obtained by performing block quantization processing on the weights of the large language model;

[0035] The FNN module is used to process the first processing result to obtain an output result.

[0036] In a third aspect, an embodiment of the present application provides an electronic device, including: a processor, a memory, and a bus, wherein:

[0037] The processor and the memory communicate with each other via the bus;

[0038] The memory stores program instructions that can be executed by the processor, and the processor can execute the method of the first aspect by calling the program instructions.

[0039] In any embodiment, the processor includes a data reading unit, a multiplication and addition unit, and a post-processing unit; wherein:

[0040] The data reading unit is used to read the quantized data and the corresponding scaling factor required by the multiplication and addition unit from the memory;

[0041] The multiplication and addition unit is used to perform multiplication and addition calculation on the quantized data and the scaling factor;

[0042] The post-processing unit is used to perform a dequantization operation on the multiplication and addition calculation result.

[0043] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium, including:

[0044] The non-transitory computer-readable storage medium stores computer instructions that cause the computer to execute the method of the first aspect.

[0045] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, wherein when the computer program instructions are read and executed by a processor, the method of the first aspect is executed.

[0046] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A schematic diagram of a large language model reasoning method flow chart provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of storing quantized data and scaling factors provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of block quantization data provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of a large language model structure provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of the structure of a large language model inference device provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application;

[0054] Figure 7 A schematic diagram of a processor structure provided in an embodiment of the present application;

[0055] Figure 8 A schematic diagram of a multiplication and addition circuit for block quantization data and scaling factor structure provided in an embodiment of the present application;

[0056] Fig. 9 A schematic diagram of a method for multiplying and adding block quantized data and a scaling factor structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0059] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0060] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0061] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0062] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0063] In the description of the embodiments of the present application, unless otherwise clearly specified and limited, technical terms such as "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0064] In recent years, large language models based on the Transformer architecture have shown excellent performance in many fields. These models have a very large parameter scale, can well understand and generate natural language, and are suitable for a variety of downstream application scenarios, such as machine translation, content summarization, sentiment analysis, etc. The reasoning process of large language models has extremely high storage, bandwidth and computing power requirements, which limits the scale of users that service providers can serve at the same time.

[0065] In order to reduce the cost of large model inference, quantization technology, sparse methods, and KV cache compression technology can be used. However, the application of these technologies will inevitably lead to a loss of accuracy. This is mainly manifested in that in large language models, abnormal features are an important feature, and commonly used quantization technologies often cannot handle abnormal features well, resulting in a decrease in the inference performance of large models.

[0066] In order to solve the above technical problems, the embodiments of the present application provide a large language model inference method, device, electronic device and storage medium. For the quantization of abnormal features, a block quantization strategy based on Hadamard transform is proposed, and the details are shown in the following embodiments.

[0067] It can be understood that the large language model inference method provided in the embodiment of the present application can be applied to electronic devices, which include terminals and servers; wherein the terminal can specifically be a smart phone, a tablet computer, a computer, a personal digital assistant (PDA), etc.; the server can specifically be an application server or a Web server.

[0068] Figure 1 A flow chart of a large language model inference method provided in an embodiment of the present application is as follows: Figure 1 As shown, the method includes:

[0069] Step 101: Performing Hadamard transformation on input data through a transformation module to obtain transformed data;

[0070] Step 102: performing block quantization processing on the transformed data through a block quantization module to obtain quantized data;

[0071] Step 103: Process the quantized data and the quantized weights of the large language model through the Attention module to obtain a first processing result; the quantized weights are obtained by performing block quantization processing on the weights of the large language model;

[0072] Step 104: Process the first processing result through the FNN module to obtain an output result.

[0073] In the specific implementation process, the large language model is a type of deep learning model with a large number of parameters, which learns language patterns, grammar and semantics by processing a large amount of text data, so as to understand and generate human language. The large language model provided in the embodiment of the present application can be applied in a variety of scenarios, including but not limited to: machine translation, chatbots and AI assistants, automatic writing and text generation, information extraction, sentiment analysis, speech recognition, social media monitoring, etc. And the large language model is based on the Transformer architecture, and the text data is processed by complex algorithm analysis. The large language model may include multiple cascaded Transformer architectures, each Transformer architecture mainly includes a transformation module, a block quantization module, an Attention module and an FNN module. In the embodiment of the present application, steps 101-104 are data processing flows for a Transformer architecture, and the output of the Transformer architecture will be input into the next Transformer architecture cascaded with it, until the last Transformer architecture outputs, and the result of the last Transformer architecture output is the final result.

[0074] Among them, the transformation module is used to perform Hadamard transformation on the input data. It should be noted that the input data can be the data after the text content entered by the user is embedded and mapped, or it can be the output result of the previous level Transformer architecture, or it can be the activation (Activations) of the large language model during the reasoning process. Hadamard transform is a transformation method based on the Hadamard matrix. The Hadamard matrix is ​​an orthogonal square matrix composed of +1 and -1 elements, and any two rows (or two columns) are orthogonal. The elements in the Hadamard transformation matrix only contain +1 and -1, which makes the transformation process only involve addition and subtraction operations, without the need for multiplication operations, thereby simplifying the calculation process. In addition, the Hadamard transform also has a certain degree of orthogonality, which helps to maintain some important characteristics of the signal during the transformation process.

[0075] The Hadamard transform can smooth out the abnormal features of the input data, thereby facilitating the quantization operation. The Hadamard transform can include a random Hadamard transform or a Hadamard rotation matrix formed by a learnable method.

[0076] In the embodiment of the present application, during the large language model inference process, Hadamard transform can be performed before each matrix multiplication calculation quantization.

[0077] The block quantization module is used to perform block quantization on the transformed data. The transformed data is in the form of a matrix. When the transformed data is divided into blocks, it can be divided by row, column or block. Of course, it can also be divided according to the channels of the large language model. The specific division method can be determined according to actual needs. Block division refers to the use of a smaller matrix for division, such as dividing the transformed data into multiple matrices of 50*50 size. After the block division is completed, each block is quantized. The purpose of quantization is to convert high-precision data into low-precision data, thereby reducing the amount of calculation. For example: convert the high-precision data format of float16 into the data format of int8. It should be noted that the weights corresponding to the large language model can also be pre-quantized in blocks, and its block quantization method can also be block quantized by row, column, block or channel.

[0078] The Attention module is used to perform mask calculation and self-attention calculation on the quantized data. In the self-attention calculation process, the quantized data and the quantized weights of the large language model can be processed to obtain the first processing result. Specifically, the value (Value) is weighted and summed according to the calculated attention weight to obtain the output representation of each position.

[0079] The FNN module may include at least one fully connected layer (i.e., FC layer). The first processing result output by the Attention module is input into the FNN module. In the FNN module, the first processing result is first subjected to Hadamard transformation, and then multiplied by the inversely transformed FC neural network weight to obtain the output of the FC layer, which is then sent to the activation layer to obtain the final output result. It should be noted that if the Transformer architecture is the last layer, the final output result is the result to be returned to the user; if the Transformer architecture is not the last layer, the final output result needs to be input into the next Transformer architecture, and steps 101 to 104 are performed again.

[0080] The embodiment of the present application performs Hadamard transform on the input data and then quantizes the transformed data in blocks, thereby reducing the impact of abnormal features in the large language model on the reasoning accuracy. Therefore, while improving the reasoning efficiency of the large language model, the accuracy of the reasoning of the large language model is guaranteed.

[0081] Based on the above embodiment, the transformed data is subjected to block quantization processing by a block quantization module to obtain quantized data, including:

[0082] Divide the transformed data into blocks according to a preset block rule to obtain a plurality of block data;

[0083] A quantization calculation is performed on each block data to obtain quantized block data and a corresponding scaling factor; wherein a plurality of quantized block data constitute the quantized data.

[0084] In the specific implementation process, the block data includes block data by block, by column or by row, and each block data corresponds to a scalar scaling factor. For example, if the original data is a 100x100 matrix, for the case of block division, each row vector in the original data can be used as a block data, and a total of 100 block data are obtained. For the case of column division, each column vector in the original data can be used as a block, and a total of 100 block data are obtained. It can also be divided by block, for example, the original data is divided into 25 20x20 block data.

[0085] After the block division is completed, each block data is quantized to obtain the quantized block data and the corresponding scaling factor. It can be understood that data quantization is essentially scaling the data. For example, a 1000 value in the range of [0,255] can be compressed to the range of [0,1] by subtracting the mean from all the values ​​and then multiplying by the variance. This processing facilitates low-precision data representation.

[0086] It should be noted that after the transformed data is divided into blocks and quantized, the quantized data can be stored. When storing, the quantized data is stored in correspondence with the block index of the corresponding block, so that when the quantized data is read, the corresponding block index can be used to quickly locate and extract the corresponding quantized data and the corresponding scaling factor. After the transformed data is divided into blocks and quantized, the quantized data and the corresponding scaling factor can be stored, such as Figure 2 As shown, Block is a block of data (specifically, it can be: a block, a row or a column of vectors), and each Block corresponds to a scalar scaling factor (Scale). Block and Scale are stored in the cache one by one, and Block and Scale are accessed synchronously.

[0087] The embodiment of the present application processes the input data of the large language model in blocks, so that abnormal features in the large language model only affect the blocks in which they are located, thereby reducing the impact on the entire large language model, thereby improving the accuracy of large language model reasoning.

[0088] Based on the above embodiment, each block of data is quantized and calculated, including:

[0089] According to the formula Each block of data is quantitatively calculated;

[0090] Among them, Q(X) is the quantized block data; X is a block data; SX is the scaling factor, S X =(max(abs(X)) / (2 B-1 -1); B is the maximum number of bits used for the scaling factor; For characterization The value range is [-2 B-1 +1,2 B-1 -1]. Used to represent rounding down.

[0091] It should be noted that in order to speed up the calculation of the block quantization strategy, the scaling sparsity is an exponential power of 2, and the multiplication and division operations are converted into shift operations.

[0092] In addition, large language model quantization can reduce the amount of matrix multiplication (GEMM) calculations. The data is quantized before the GEMM operator, and the quantized data is subjected to GEMM operations. After the calculation is completed, it can be dequantized and then sent to the next layer, such as the activation layer. The dequantization method of the block quantization method is:

[0093]

[0094] Among them, S X It is the exponential form of 2, Q(X)·S X The multiplication operation is simplified to a shift operation, Z X Represents the quantization bias.

[0095] The embodiment of the present application reduces the amount of matrix multiplication calculations during the reasoning process and improves the reasoning efficiency by quantizing the divided data.

[0096] On the basis of the above embodiment, the plurality of block data include block data of multiple levels, and the block data of the next level is obtained by further blocking the block data of the previous level.

[0097] In the specific implementation process, when dividing the data into blocks, a multi-level block method can be adopted, where and the scaling factor are in the form of a block matrix W, expressed as in, is the first-level quantitative data of W, is the first-level scaling factor; the second-level scaling is to divide the single machine into blocks Divide into blocks

[0098] For example: the original data is a 100x100 matrix. Taking two-level block as an example, the first-level block is 25 20x20 matrices. Each 20x20 block data shares a scaling factor, so the first-level block has 25 scaling factors; the second-level block is based on the first-level block, and each 20x20 small block is divided into 25 4x4 small blocks and 25 second-level block scaling factors. If the original data is 100x100 data with a precision of floap16, the precision of the first-level block scaling factor is int8, and each sub-block uses the fp8 data format. After the first-level block quantization, the data volume is compressed to about 50% of the original. Figure 3 A schematic diagram of block quantization data provided in an embodiment of the present application, such as Figure 3 The original data is high-precision float16 data, which can be divided into blocks by some block method, for example, by row, by column or by block. i , after single-level quantization, each block element shares a scaling factor s, and the elements in each block after single-level quantization are Original data i Using high-precision numerical formats such as float16, the shared scaling factor after block quantization uses int8, and the quantized data can use lower-precision numerical formats, such as float4, binary, ternary and other low-precision data. In the embodiment of the present application, the quantized block data can use low-precision numerical formats such as binary, ternary, int2, int4, fp4, fp6, etc.; the scaling factor corresponding to the block data is stored in data formats such as int8. The low-precision numerical format greatly reduces the demand for computing resources, storage and communication, thereby accelerating large model reasoning.

[0099] Based on the above embodiment, the quantized weight includes the first quantized weight in the Attention module; the quantized data and the quantized weight of the large language model are processed by the Attention module to obtain a first processing result, including:

[0100] Performing an inverse transformation on the first quantized weight using an inverse transformation matrix corresponding to the Hadamard transformation to obtain a first inverse transformed weight;

[0101] The quantized data is multiplied by the weight after the first inverse transformation to obtain a first processing result.

[0102] The quantized weight also includes a second quantized weight corresponding to the FNN module; the first processing result is processed by the FNN module to obtain an output result, including:

[0103] Performing Hadamard transformation on the first processing result to obtain a transformed first processing result;

[0104] quantizing the transformed first processing result to obtain a quantized first processing result;

[0105] Performing an inverse transformation on the second quantized weight using an inverse transformation matrix corresponding to the Hadamard transformation to obtain a second inverse transformed weight;

[0106] Multiplying the quantized first processing result by the second inverse transformed weight to obtain a second processing result;

[0107] The second processing result is processed by the activation layer in the FNN module to obtain an output result.

[0108] In the specific implementation process, Figure 4 A schematic diagram of a large language model structure provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, before the large model inference, the weight of the large model is first quantized by block quantization to form the block quantization parameters of the large model neural network weight W and the scaling factor shared by the block quantization parameters, where the quantization parameters can be in low-precision numerical format and the scaling factor is int8. The large model inference process is as follows:

[0109] 1. The question content entered by the user is vectorized into the input X of the first layer Transformer structure through the Embedding layer;

[0110] 2.1. Multiply the input X and the locally generated Hadamard matrix to obtain XH.

[0111] 2.2. Quantize XH into Q(XH) by using a block quantization method. It can be understood that the block quantization method is referred to the above embodiment and will not be described in detail here.

[0112] 2.3. Send Q(XH) to the Attention module of the Transformer model; in the Attention module, H T Multiply the weights of the query (Q), key (K), and value (V) projection layers to get H T W q , H T W k and H T W v , and send the Q(XH) result to the Q, K, V projection layer. In the Q, K, V projection layer, Q(XH) and H T W q , H T W k and H T W vMultiply, send the output of Q,K projection layer to softmax activation layer; T W v Multiply by a new Hadamard matrix H head Get H T W v H head It should be noted that the new Hadamard matrix can be generated in advance. Multiply the output of the softmax activation layer by H T W v H head Get the input of the output layer, the output layer weight of the Attention layer H head T W o , thus obtaining the output of the Attention module of the Transformer layer. It should be noted that the quantization strategy of the KV cache in the Attention module is: the key value is per-channel (quantized by column blocks), and the value is per-token (quantized by row blocks).

[0113] 3. There are two or more FC layers in the FFN module of the Transformer layer. The calculation process of one FC layer is introduced below. The output X of the Attention module is multiplied by the locally generated Hadamard matrix to obtain XH. The quantization strategy is used to quantize XH to Q(XH), which is sent to the first FC layer. The FC layer neural network weight W FC Multiply by H T Get H T W FC , the FC layer input Q(XH) multiplied by H T W FC , get the output of the FC layer and send it to the activation layer GeLU. The other layer is similar, repeat the process.

[0114] This large model has L layers of Transformers, and the calculation process of the other layers of Transformers repeats the first layer of Transformers. After completing the calculation of the L layers of Transformers, the output X also uses the process of first Hadamard transformation, quantization, and Hadamard inverse transformation to restore the result data from the Hadamard domain to the original data domain.

[0115] The first word of the answer to the question input by the user is obtained through the above process, and the output of other words is similar.

[0116] The embodiment of the present application performs Hadamard transformation on the weights or activations of each neural network layer of the Transformer architecture. The matrix multiplication operation neural network layer of the Transformer architecture occupies most of the Transformer delay, so the multi-level block quantization strategy of the embodiment of the present application is adopted for the neural network layer involving matrix multiplication; due to the influence of abnormal characteristics, the neural network layer involving matrix multiplication operation in the standard Transformer architecture performs Hadamard transformation before using the quantization strategy.

[0117] Project layer X with q q W q For example, the calculation of X q W q It is a matrix operation layer. In order to reduce the matrix multiplication and addition X q W q The amount of calculation and data requires the use of low-precision numerical formats for X q and W q Quantization, no matter which quantization strategy is used, will lead to accuracy loss, especially X q and W q There are abnormal features. In order to reduce the loss caused by quantization, Hadamard transform is performed on the activation X before quantization. Since the Hadamard transformation matrix has the property H·H T =I, then:

[0118] XW=XIW=XH·H T W

[0119] Therefore, when the Hadamard transform is introduced into the input activation X, the inverse Hadamard transform is required to restore the result data from the Hadamard domain to the original data.

[0120] The embodiment of the present application performs an inverse transformation on the first quantized weights by using an inverse transformation matrix, thereby restoring the result data from the Hadamard domain to the original data domain.

[0121] Figure 5 This is a schematic diagram of a large language model inference device provided in an embodiment of the present application. The device may be a module, program segment or code on an electronic device. It should be understood that the device is similar to the above-mentioned Figure 1 The method embodiment corresponds to and can be executed Figure 1 The various steps involved in the method embodiment and the specific functions of the device can be found in the above description. To avoid repetition, detailed description is appropriately omitted here. The device includes: a transformation module 501, a block quantization module 502, an Attention module 503 and an FNN module 504, wherein:

[0122] The transformation module 501 is used to perform Hadamard transformation on the input data to obtain transformed data;

[0123] The block quantization module 502 is used to perform block quantization processing on the transformed data to obtain quantized data;

[0124] The Attention module 503 is used to process the quantized data and the quantized weights of the large language model to obtain a first processing result; the quantized weights are obtained by performing block quantization processing on the weights of the large language model;

[0125] The FNN module 504 is used to process the first processing result to obtain an output result.

[0126] Based on the above embodiment, the block quantization module 502 is specifically used for:

[0127] Divide the transformed data into blocks according to a preset block rule to obtain a plurality of block data;

[0128] Perform quantization calculation on each of the block data to obtain the quantized block data and the corresponding scaling factor; wherein a plurality of the quantized block data constitute the quantized data.

[0129] Based on the above embodiment, the block quantization module 502 is specifically used for:

[0130] According to the formula Performing quantitative calculation on each of the block data;

[0131] Wherein, Q(X) is the quantized block data; X is a block data; S X is the scaling factor, S X =(max(abs(X)) / (2 B-1 -1); B is the maximum number of bits used for the scaling factor; For characterization The value range is [-2 B-1 +1,2 B-1 -1].

[0132] On the basis of the above embodiment, the plurality of block data include block data of multiple levels, and the block data of the next level is obtained by further blocking the block data of the previous level.

[0133] Based on the above embodiment, the quantized weight includes the first quantized weight in the Attention module; the Attention module 503 is specifically used for:

[0134] Performing an inverse transformation on the first quantized weight using an inverse transformation matrix corresponding to the Hadamard transformation to obtain a first inverse transformed weight;

[0135] The quantized data is multiplied by the weight after the first inverse transformation to obtain the first processing result.

[0136] On the basis of the above embodiment, the quantized weight also includes a second quantized weight corresponding to the FNN module; the FNN module 504 is specifically used for:

[0137] Performing Hadamard transformation on the first processing result to obtain a transformed first processing result;

[0138] quantizing the transformed first processing result to obtain a quantized first processing result;

[0139] Performing an inverse transformation on the second quantized weights using an inverse transformation matrix corresponding to the Hadamard transform to obtain second inverse transformed weights;

[0140] Multiplying the quantized first processing result and the second inverse transformed weight to obtain a second processing result;

[0141] The second processing result is processed by the activation layer in the FNN module to obtain the output result.

[0142] Figure 6 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the electronic device includes: a processor (processor) 601, a memory (memory) 602 and a bus 603; wherein,

[0143] The processor 601 and the memory 602 communicate with each other via the bus 603;

[0144] The processor 601 is used to call the program instructions in the memory 602 to execute the methods provided by the above-mentioned method embodiments, for example, including: performing Hadamard transform on the input data through the transformation module to obtain transformed data; performing block quantization processing on the transformed data through the block quantization module to obtain quantized data; processing the quantized data and the quantized weights of the large language model through the Attention module to obtain a first processing result; the quantized weights are obtained by performing block quantization processing on the weights of the large language model; and processing the first processing result through the FNN module to obtain an output result.

[0145] The processor 601 may be an integrated circuit chip with signal processing capabilities. The processor 601 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. It may implement or execute various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0146] The memory 602 may include but is not limited to random access memory (RAM), read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), etc.

[0147] Based on the above embodiment, the processor includes a data reading unit, a multiplication and addition unit and a post-processing unit; wherein:

[0148] The data reading unit is used to read the quantized data and the corresponding scaling factor required by the multiplication and addition unit from the memory;

[0149] The multiplication and addition unit is used to perform multiplication and addition calculations on the quantized data and the scaling factor;

[0150] The post-processing unit is used to perform a dequantization operation on the multiplication and addition calculation results.

[0151] Figure 7 A schematic diagram of a processor structure provided in an embodiment of the present application, Figure 8 The block quantization data and scaling factor structure multiplication and addition circuit diagram provided in the embodiment of the present application is as follows: Figure 7 , Figure 8As shown, the data reading unit includes a controller, an address generator and a cache manager. The controller is used to load the data required by the calculator from the storage, including the quantized block data and the corresponding scaling factor data to the cache manager; the address generator is used to generate the address for storing each block data. In the cache manager, the data is also saved in the form of block quantization and corresponding scaling factors; the two matrix data (respectively, the matrix A corresponding to the weight and the matrix B corresponding to the activation) in the cache and the weight scaling factor scale (SA for short), the activation scaling factor scale (SB for short) are sent to the multiplier to perform multiplication and addition operations, wherein the A and B multiplication and addition operations are performed on the block quantized data according to the commonly used GEMM operation mode, and the results of A*B=C and the scaling factor SC=SA*SB are saved respectively, and then C*SC is calculated. Since SA, SB, and SC are all exponential powers of 2, C*SC is a shift operation on C, thereby realizing the multiplication operation of the block quantization matrix; the multiplication operation result is sent to the post-processing module for dequantization operation, and the dequantization is to shift each data of C according to the scaling factor SA*SB. Figure 8 This is the multiplication-addition computing accelerator part of the embodiment of the present application. The red arrow represents the neural network weight W data, and the blue arrow represents the input activation X data. Where W and X are block quantized, so they include the quantized values ​​and scaling factors.

[0152] Fig. 9 A schematic diagram of a block quantization data and scaling factor structure multiplication and addition method provided in an embodiment of the present application, such as Fig. 9 As shown, A11 is a 16x16 matrix, and all elements of A11 share a scaling factor scalar of SA11.

[0153] The present embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: performing Hadamard transform on the input data through the transformation module to obtain transformed data; performing block quantization processing on the transformed data through the block quantization module to obtain quantized data; processing the quantized data and the quantized weights of the large language model through the Attention module to obtain a first processing result; the quantized weights are obtained by performing block quantization processing on the weights of the large language model; and processing the first processing result through the FNN module to obtain an output result.

[0154] The present embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable the computer to execute the methods provided by the above-mentioned method embodiments, for example, including: performing Hadamard transform on the input data through the transformation module to obtain transformed data; performing block quantization processing on the transformed data through the block quantization module to obtain quantized data; processing the quantized data and the quantized weights of the large language model through the Attention module to obtain a first processing result; the quantized weights are obtained by performing block quantization processing on the weights of the large language model; and processing the first processing result through the FNN module to obtain an output result.

[0155] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0156] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0157] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0158] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0159] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A large language model reasoning method, characterized in that: The large language model includes at least one Transformer architecture, and the Transformer architecture includes a transformation module, a block quantization module, an attention mechanism Attention module and a feedforward neural network FNN module; the method includes: Performing Hadamard transformation on the input data through the transformation module to obtain transformed data; Performing block quantization processing on the transformed data by the block quantization module to obtain quantized data; The quantized data and the quantized weights of the large language model are processed by the Attention module to obtain a first processing result; the quantized weights are obtained by performing block quantization processing on the weights of the large language model; The first processing result is processed by the FNN module to obtain an output result.

2. The method according to claim 1, characterized in that: The step of performing block quantization processing on the transformed data by the block quantization module to obtain quantized data includes: Divide the transformed data into blocks according to a preset block rule to obtain a plurality of block data; Perform quantization calculation on each of the block data to obtain quantized block data and a corresponding scaling factor; wherein a plurality of the quantized block data constitute the quantized data.

3. The method according to claim 2, characterized in that The performing quantization calculation on each of the block data includes: According to the formula Performing quantitative calculation on each of the block data; Wherein, Q(X) is the quantized block data; X is a block data; S X is the scaling factor, S X =(max(abs(X)) / (2 B-1 -1); B is the maximum number of bits used for the scaling factor; For characterization The value range is [-2 B-1 +1,2 B-1 -1].

4. The method according to claim 2, characterized in that: The plurality of block data include block data of multiple levels, and the block data of the next level is obtained by further blocking the block data of the previous level.

5. The method according to claim 1, characterized in that The quantized weight includes a first quantized weight in an Attention module; and the processing of the quantized data and the quantized weight of the large language model by the Attention module to obtain a first processing result includes: Performing an inverse transformation on the first quantized weight using an inverse transformation matrix corresponding to the Hadamard transformation to obtain a first inverse transformed weight; The quantized data is multiplied by the weight after the first inverse transformation to obtain the first processing result.

6. The method according to claim 1, characterized in that The quantized weights also include a second quantized weight corresponding to the FNN module; The step of processing the first processing result by the FNN module to obtain an output result includes: Performing Hadamard transformation on the first processing result to obtain a transformed first processing result; quantizing the transformed first processing result to obtain a quantized first processing result; Performing an inverse transformation on the second quantized weights using an inverse transformation matrix corresponding to the Hadamard transform to obtain second inverse transformed weights; Multiplying the quantized first processing result and the second inverse transformed weight to obtain a second processing result; The second processing result is processed by the activation layer in the FNN module to obtain the output result.

7. A large language model inference device, characterized in that: The large language model includes at least one Transformer architecture; the device includes: A transformation module is used to perform Hadamard transformation on the input data to obtain transformed data; A block quantization module, used for performing block quantization processing on the transformed data to obtain quantized data; An Attention module, used to process the quantized data and the quantized weights of the large language model to obtain a first processing result; the quantized weights are obtained by performing block quantization processing on the weights of the large language model; The FNN module is used to process the first processing result to obtain an output result.

8. An electronic device, characterized in that: include: processor, memory and bus, wherein, The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 6.

9. The electronic device according to claim 8, characterized in that: The processor includes a data reading unit, a multiplication and addition unit and a post-processing unit; wherein: The data reading unit is used to read the quantized data and the corresponding scaling factor required by the multiplication and addition unit from the memory; The multiplication and addition unit is used to perform multiplication and addition calculation on the quantized data and the scaling factor; The post-processing unit is used to perform a dequantization operation on the multiplication and addition calculation result.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 6.

11. A computer program product, characterized in that The method comprises computer program instructions, and when the computer program instructions are read and executed by a processor, the method according to any one of claims 1 to 6 is executed.

Images