Big language model acceleration system and method based on binary quantization

Abstract

The invention discloses a large language model acceleration system and method based on binary quantization, and the system comprises an extraction and quantization unit, a grouping packaging and splicing unit, an operation realization unit, and an inverse quantization and output unit, and comprises the specific steps: extracting parameters needing matrix / vector multiplication in a model; performing binary quantization processing on parameters of the model matrix / vector multiplication operation; splicing the grouped binary quantized parameters into single integer data bit by bit according to an order when matrix / vector multiplication is calculated by adopting a grouping and packaging algorithm; matrix / vector multiplication which is equivalent to original model reasoning and is friendly to hardware is executed; the result is converted back to the floating point data type through an inverse quantization algorithm, a final calculation result of the large language model in the matrix / vector multiplication operation is obtained, the final calculation result is transmitted into a subsequent network layer, and calculation acceleration for the matrix / vector multiplication operation in the large language model is completed. According to the method, the accuracy is ensured, and meanwhile, the speed is increased during large language model reasoning.

Description

Technical Field

[0001] The present invention belongs to the technical field of large language model calculation, and in particular relates to a large language model acceleration system and method based on binary quantization. Background Art

[0002] Large language models (LLMs) have demonstrated extraordinary performance in various fields, ushering in the era of artificial intelligence generated content (AIGC). Large language models are usually based on the Transformer architecture, and their key module is the multi-head self-attention mechanism, which contains three core component vectors: query, key, and value. Through a large number of matrix / vector multiplication operations, the attention score can be calculated from the above vectors, allowing the network to learn complex input relationships and long-range dependencies.

[0003] However, the design of a deployment acceleration framework for large language models currently needs to address the following three challenges and problems:

[0004] Problem 1: Large language models have a huge number of parameters and computational complexity

[0005] In the past five years, the size of large language models has exploded by 15,000 times, from BERT with 330 million parameters to the recent M6 model with 10 trillion parameters. At the same time, the computing power of computing platforms has not grown proportionally, which makes the acceleration of the deployment of large language models an important issue at present. Compared with traditional deep learning models, the unprecedented scale and computing volume of large language models lead to a large amount of storage and bandwidth overhead when inferring the model. Training a single large language model may take months and hundreds of high-performance GPUs, and generating a single word output may take more than ten seconds.

[0006] Problem 2: Existing system platforms cannot support the deployment of emerging large language models

[0007] With the increasing popularity of large language models, the application scenarios of large language models on various edge and mobile platforms are placing higher requirements on the flexibility of the reasoning framework, including supporting customized computing paradigms and storage optimization with scalable architecture and configurable parameters, rather than relying solely on CPU and GPU for reasoning in general scenarios.

[0008] Problem 3: Designing a system-on-chip for a large language model is time-consuming

[0009] Due to the scale of large language models, the difficulty, workload, and time required to design a dedicated acceleration framework for large language models have increased significantly, and the cost of chip verification, simulation, and performance testing often exceeds reasonable time and computing resource constraints. For example, simulating an accelerator may require millions of megabytes of waveform files, and an end-to-end simulation takes weeks. There is an urgent need to develop an efficient, scalable, and configurable dedicated acceleration framework for large language models.

[0010] In order to solve the problem of excessive model calculation and storage cost overhead in the design of large model deployment acceleration framework, the present invention creates an acceleration method for efficient deployment of large language models. The acceleration method is based on a binary quantization algorithm, which can effectively alleviate the problem of large language model calculation and large scale, and achieve speed improvement when inferring large language models. Summary of the invention

[0011] In order to make up for the shortcomings of the prior art, the purpose of the present invention is to provide a large language model acceleration system and method based on binary quantization, so as to solve the challenge that the current large language model has a huge computational scale and is difficult to accelerate and deploy, and achieve effective reduction of data overhead and computational overhead when inferring large language models.

[0012] The technical problem solved by the present invention can be achieved by adopting the following technical solutions:

[0013] On the one hand, the present invention provides a large language model acceleration method based on binary quantization, comprising the following steps:

[0014] S1. Extract the parameters that need to be processed by matrix / vector multiplication in the model, and use a fixed-point quantization algorithm to perform binary quantization on the parameters of the model matrix / vector multiplication;

[0015] S2. For the binary quantized parameters, a grouping and packing algorithm is used to splice the grouped binary quantized parameters into a single integer data bit by bit according to the order of calculating the matrix / vector multiplication;

[0016] S3. Based on the binary multiplication principle, perform hardware-friendly matrix / vector multiplication operations equivalent to the original model inference;

[0017] S4. Convert the result back to floating-point data type through the dequantization algorithm to obtain the final calculation result of the large language model in the matrix / vector multiplication operation, and pass it to the subsequent network layer to complete the calculation acceleration of the matrix / vector multiplication operation in the large language model.

[0018] Furthermore, the specific operation process of step S1 is as follows:

[0019] S11. Analyze the target large language model structure and calculation process, determine the parameter positions that need to be quantized, and the parameters that need to be quantized and accelerated are the floating-point format parameters of the linear layer and matrix multiplication operation layer in the target large language model that occupy the overall model reasoning calculation bottleneck;

[0020] S12. Based on the upper and lower bounds of the data value range before and after quantization, the model weights stored in the floating point data type are symmetrically quantized to obtain binary quantized data and its corresponding scaling factor;

[0021] S13. After binary quantization, in the tensor participating in the matrix / vector multiplication layer, each floating-point format data is represented by a single-bit value, and the quantized binary data in each layer of the tensor share the same quantization scaling factor.

[0022] Furthermore, the parameters that need to be quantized and accelerated are the floating-point format parameters of the linear layer and matrix multiplication operation layer in the target large language model, which occupy the bottleneck of the overall model inference calculation. Based on the symmetric quantization algorithm, the absolute value range of the value range before and after quantization is set with each tensor parameter layer l of size [B, C, H, W] as the unit and As the upper and lower bounds of the data during quantization, calculate the quantization scaling factor s and convert the original floating point data X float Transformed into binary data X quantized , save the quantized data and scaling factor. After binary quantization, the higher-precision floating-point data will be converted to a single-bit value representation. In addition, for each tensor parameter layer l, the scaling factor s when quantizing its parameters will be saved at the same time. l , used for the subsequent dequantization operation when restoring the floating-point data format.

[0023] Furthermore, in step S2, based on the upper and lower bounds of the data value range before and after quantization, the model weights stored in the floating-point data type are symmetrically quantized to obtain binary quantized data and its corresponding scaling factor. The symmetrical quantization process of converting floating-point values ​​into binary values ​​is shown in equations (1) and (2):

[0024]

[0025] Among them, x represents a floating-point parameter tensor of a certain layer of input for matrix / vector operation, s represents the scaling factor calculated by the absolute value of the value range before and after quantization, the rounding function round(·) rounds the division result, and the sign function sgn(·) is used as a truncation function during binary quantization. It judges the sign of the input value and outputs 1 when it is greater than 0 and -1 when it is less than 0. Therefore, the data distribution value after binary quantization is {-1, 1}. The obtained scaling factor s is calculated according to the data distribution of the floating-point parameter tensor of this layer. The quantized binary data in the same layer of tensors share the same scaling factor. Each input binary quantized parameter tensor layer has a corresponding scaling factor s, which will be used to restore the floating-point type result during the dequantization operation at the end of the operation.

[0026] The above binary quantization converts the data types of model parameters of a large number of matrix / vector multiplication operations that occupy the computing bottleneck in large language models, providing data type preparation for subsequent inference acceleration algorithms.

[0027] Furthermore, in the step S2, a data group size L is selected for packing, and L binary quantized data are bit-wise spliced ​​into a single integer data in the calculation order of matrix / vector multiplication operation; the data group size L can be set to a bit width of an integer data type that is more efficient than floating-point multiplication operation.

[0028] Furthermore, in the step S3, for the integer data obtained by grouping and packing, when implemented on the software algorithm side, according to the binary quantization algorithm, the operation is converted into a multiplication operation of two types of values ​​1 / -1; when implemented on the hardware deployment side of the large language model accelerator, the value of -1 is mapped to 0, and then based on the two-step binary bit operation of "XOR+Popcount", the matrix / vector multiplication is equivalently implemented.

[0029] Furthermore, when calculating binary vector multiplication, the calculation formula shown in the following equation (3) is used:

[0030]

[0031] Among them, x and y represent the input tensors, L represents the size of the data group to be packaged, Represents the number of bits set to 1 in the binary result obtained by statistics; in addition, the type of the output value is integer data, and its bit width is related to the size of the data group used.

[0032] Furthermore, in step S4, after the multiplication result is output, the layer-by-layer scaling factor s obtained during the previous quantization is used. l, the integer multiplication results of each layer are dequantized to obtain the final floating-point calculation results, and the results are sorted and passed to the subsequent network layers of the large language model network.

[0033] Furthermore, the calculation formula of the floating-point output after dequantization is shown in equation (4):

[0034] o float =x q-pack *y q-pack *s x s y (4)

[0035] Among them, x q-pack and q-pack are two quantized and packed integer multipliers, s x 、s y are the quantization scaling factors of the corresponding layers, o float It is the floating-point output data after dequantization.

[0036] On the other hand, the present invention provides a large language model acceleration system based on binary quantization, comprising:

[0037] The parameter extraction and quantization unit is used to extract and obtain the relevant floating-point parameters for matrix / vector multiplication operations in the model, and use a fixed-point quantization algorithm to perform binary quantization on the parameters of the model matrix / vector multiplication operations;

[0038] A grouping and splicing unit, used for splicing the grouped binary quantized parameters into a single integer data bit by bit according to the order of calculating the matrix / vector multiplication;

[0039] An operation realization unit, which is used to perform hardware-friendly matrix / vector multiplication operation based on the binary multiplication principle, which is equivalent to the original model reasoning;

[0040] The dequantization and output unit is used to convert the result back to the floating-point data type through the dequantization algorithm, obtain the final calculation result of the large language model in the matrix / vector multiplication operation, and pass it to the subsequent network layer.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] (1) The acceleration system of the invention includes four parts: an extraction and quantization unit, a grouping, packaging and splicing unit, an operation implementation unit, and a dequantization and output unit. The system quantizes several floating-point format data for matrix / vector multiplication and implements hardware-friendly equivalent operations using binary multiplication. While ensuring accuracy, it effectively reduces data overhead and computational overhead when inferring large language models.

[0043] (2) The acceleration method of the present invention is based on a binary quantization algorithm, which can effectively alleviate the problem of large language model computation and large scale, and achieve speed improvement when reasoning with a large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a module block diagram of the acceleration system of the present invention;

[0045] Figure 2 is a schematic diagram of the acceleration method of the present invention;

[0046] Figure 3 is a flow chart of the acceleration method of the present invention;

[0047] Figure 4 This is a diagram showing the analysis results of the reasoning process of the LLaMa model in Example 2 of the present invention;

[0048] Figure 5 This is a schematic diagram of a data processing flow for binary quantization and data packaging in Embodiment 2 of the present invention;

[0049] Figure 6 This is a schematic diagram of the calculation flow of the binary vector multiplication equivalent operation performed by the hardware deployment end in Embodiment 2 of the present invention;

[0050] Figure 7 This is a time comparison of Example 2 of the present invention and a commercial CPU / GPU platform benchmark on a large model reasoning task;

[0051] Figure 8 The inference process calculation and analysis results of the LLaMa model used in Example 2 of the present invention. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, a large language model acceleration system and method based on binary quantization of the present invention will be further described below in conjunction with the accompanying drawings.

[0053] Example 1

[0054] like Figure 1 As shown, this embodiment provides a large language model acceleration system based on binary quantization, including:

[0055] The parameter extraction and quantization unit is used to extract and obtain the relevant floating-point parameters for matrix / vector multiplication operations in the model, and use a fixed-point quantization algorithm to perform binary quantization on the parameters of the model matrix / vector multiplication operations;

[0056] A grouping and splicing unit, used for splicing the grouped binary quantized parameters into a single integer data bit by bit according to the order of calculating the matrix / vector multiplication;

[0057] An operation realization unit, which is used to perform hardware-friendly matrix / vector multiplication operation based on the binary multiplication principle, which is equivalent to the original model reasoning;

[0058] The dequantization and output unit is used to convert the result back to the floating-point data type through the dequantization algorithm, obtain the final calculation result of the large language model in the matrix / vector multiplication operation, and pass it to the subsequent network layer.

[0059] It can be understood that the extraction and quantization unit, the grouping, packaging and splicing unit, the operation implementation unit, and the inverse quantization and output unit will perform hardware-friendly equivalent operations on several floating-point format data for matrix / vector multiplication after binary quantization, thereby achieving effective reduction of data overhead and computational overhead during inference of large language models while ensuring accuracy.

[0060] Example 2

[0061] like Figure 2 and Figure 3 As shown, this embodiment provides a large language model acceleration method based on binary quantization, comprising the following steps:

[0062] (I) Extract the parameters that need to be multiplied by matrix / vector in the model, and use the fixed-point quantization algorithm to perform binary quantization on the parameters of the model matrix / vector multiplication. The specific operation process is as follows:

[0063] (1) Analyze the structure and calculation process of the target large language model to determine the parameter locations that need to be quantized. The parameters that need to be quantized and accelerated are the floating-point format parameters of the linear layer and matrix multiplication operation layer in the target large language model, which occupy the bottleneck of the overall model inference calculation.

[0064] (2) Based on the upper and lower bounds of the data value range before and after quantization, the model weights stored in the floating-point data type are symmetrically quantized to obtain binary quantized data and its corresponding scaling factor;

[0065] (3) After binary quantization, each floating-point format data in the tensor participating in the matrix / vector multiplication layer is represented by a single-bit value, and the quantized binary data in each layer of the tensor share the same quantization scaling factor.

[0066] The above steps involve applying parameter analysis and quantization at inference time on a large language model. Figure 4As shown in the figure, the analysis results of the reasoning process of the currently common LLaMa model are shown. It can be seen that the matrix / vector multiplication operation occupies a large amount of calculation and time overhead in the single inference of the large language model. Specifically, in this step, the parameters in the large language model structure are first identified and analyzed, and the parameter position of the matrix / vector multiplication operation layer is determined. Next, for the above-mentioned model weight parameters, based on the upper and lower bounds of the data value range before and after quantization, the model weights stored in the floating-point data type are symmetrically quantized to obtain binary quantized data and its corresponding scaling factor. The symmetrical quantization process of converting floating-point values ​​to binary values ​​is shown in equations (1) and (2):

[0067]

[0068] Among them, x represents a floating-point parameter tensor of a certain layer of input for matrix / vector operation, s represents the scaling factor calculated by the absolute value of the value range before and after quantization, the rounding function round(·) rounds the division result, and the sign function sgn(·) is used as a truncation function when performing binary quantization. It judges the sign of the input value and outputs 1 when it is greater than 0 and -1 when it is less than 0. Therefore, the data distribution value after binary quantization is {-1, 1}, and the obtained scaling factor s is calculated according to the data distribution of the floating-point parameter tensor of this layer. Therefore, the quantized binary data in the same layer of tensors share the same scaling factor. Each input binary quantized parameter tensor layer has a corresponding scaling factor s, which will be used to restore the floating-point type result during the dequantization operation at the end of the operation.

[0069] The above binary quantization converts the data types of model parameters of a large number of matrix / vector multiplication operations that occupy the computing bottleneck in large language models, providing data type preparation for subsequent inference acceleration algorithms.

[0070] (ii) For the binary quantized parameters, a grouping and packing algorithm is used to splice the grouped binary quantized parameters into a single integer data bit by bit according to the order of calculating the matrix / vector multiplication.

[0071] The process of the grouping and packing algorithm is as follows: select the data group size L to be packed, and splice the L binary quantized data into a single integer data bit by bit in the calculation order of matrix / vector multiplication operation. Figure 4 As shown, if L=64 is selected, a total of 64×N floating-point data will be quantized to obtain 64×N binary single-bit data. This batch of data is then grouped according to L=64 and spliced ​​bit by bit. Each group of 64 single-bit data forms an integer data of the int64 data type, totaling N integer data of the int64 data type.

[0072] The data group size L can be set to the bit width of an integer data type that is more efficient than floating-point multiplication operations. For example:

[0073] If L=8, 8 binary quantized floating-point parameter data are concatenated into 1 integer data of data type int8.

[0074] If L=16, 16 binary quantized floating-point parameter data are concatenated into one integer data of the data type int16.

[0075] If L=32, 32 binary quantized floating-point parameter data are concatenated into one integer data of the data type int32.

[0076] If L=64, 64 binary quantized floating-point parameter data are concatenated into one integer data of the data type int64.

[0077] The grouping and packing algorithm will decompose the matrix multiplication in the operation process into multiple vector multiplications without changing the output. For example, consider two floating-point data weight matrices with sizes of 1024×4096 and 4096×16384 for matrix multiplication. After binary quantization and grouping and packing (L=64), these two rectangles become matrices of int64 data type with sizes of 1024×64 and 64×16384 respectively.

[0078] (iii) Based on the principle of binary multiplication, perform hardware-friendly matrix / vector multiplication operations that are equivalent to the original model inference.

[0079] The specific operations for the integer data obtained by grouping and concatenating are as follows:

[0080] (1) Software algorithm end

[0081] When implemented on the software algorithm side, according to the binary quantization algorithm, the multiplication operation is actually a multiplication operation of two values ​​1 / -1, as shown in Table 1 below;

[0082] Table 1 Binary multiplication implementation on the software algorithm side

[0083] <![CDATA[x i ]]> <![CDATA[y i ]]> <![CDATA[x i ·and i ]]> -1 -1 +1 -1 +1 -1 +1 -1 -1 +1 +1 +1

[0084] (2) Hardware deployment

[0085] When implemented on the hardware deployment side, since +1 and -1 require 2 bits to represent, in order to facilitate hardware implementation, the value of -1 can be mapped to 0, and then based on the two-step binary bit operation of "xor+count Popcount", the matrix / vector multiplication is equivalently implemented to improve computing efficiency.

[0086] The xnor operation (symbolized by ⊙) can be used to represent the multiplication of binary values. Its calculation rules are shown in Table 2 below.

[0087] Table 2 XOR operation rules on the hardware deployment side

[0088]

[0089] Furthermore, when calculating binary vector multiplication, the calculation formula shown in the following equation (3) is used:

[0090]

[0091] like Figure 6 As shown, in this embodiment, the above calculation formula can be used on the hardware deployment end using XOR (XOR operation negation, symbol is ) and Popcount (a hardware-efficient function that counts the number of bits set to 1 in the binary representation of the data). Where x and y represent the input tensors, L represents the size of the data group to be packed, Represents the number of bits set to 1 in the binary result obtained by statistics; in addition, the type of the output value is integer data, and its bit width is related to the size of the data group used.

[0092] (IV) Convert the result back to floating point data type through the dequantization algorithm to obtain the large language model in matrix /

[0093] The final calculation result of the vector multiplication operation is passed to the subsequent network layer to complete the calculation acceleration of the matrix / vector multiplication operation in the large language model.

[0094] After outputting the multiplication result, the layer-by-layer scaling factor s obtained during the previous quantization is used l , the integer multiplication results of each layer are dequantized to obtain the final floating-point calculation results, and the results are sorted and passed to the subsequent network layers of the large language model network.

[0095] The matrix / vector multiplication output is an integer data, and its bit width is related to the size of the data group used. First, it is necessary to obtain the quantization scaling factor {s} corresponding to each multiplication operation layer. In order to keep the data type consistent when the large language model passes through different types of operation layers during inference, it is necessary to perform a dequantization operation on the matrix / vector multiplication output using binary quantization. The calculation formula for the quantized floating-point output is shown in equation (4):

[0096] o float =x q-pack *y q-pack *s x s y (4)

[0097] Among them, x q-pack and q-pack are two integer multipliers after quantization and data packing in step ②, s x ,s y are the quantization scaling factors of their corresponding layers, o float The above dequantization results ensure the consistency and accuracy of data types when inferring large language models.

[0098] After the inverse quantization is completed, the acceleration method of the invention passes the floating point output result to the subsequent network layer of the current matrix / vector operation layer, and continues to perform the subsequent inference process. The use of binary quantization in the invention effectively alleviates the problem of excessive data storage and computing overhead when inferring a large language model, and achieves better computing acceleration.

[0099] (V) Experimental verification

[0100] In order to verify the effectiveness and actual performance of the design proposed in the present invention, the proposed system-on-chip solution with a custom computing module is implemented based on Gemmini and Chipyard, and compiled using the 7-nanometer ASAP7 PDK. In the experiment, FP32, INT8, and INT32 are used as the benchmark performance of high-precision multiplication operations to verify the acceleration effect that can be brought by applying the binary quantization algorithm described in the present invention in the array accelerator hardware deployment for large language model inference.

[0101] When the scale of actual hardware deployment is expanded, the normalized growth change of the array accelerator calculation amount is shown in Table 3, where the values ​​of columns 1 to 3 represent the representation of dimensions [A, B] and [B, C] when performing data matrix multiplication. When developing a system-on-chip for a large language model, its design scale tends to expand exponentially compared to conventional models. When the design scale is expanded from 1 to 2048 and 16000, the array accelerator proposed by the present invention can effectively control the expansion of the calculation amount, because the binary quantization and data packing operations in the array accelerator save the high-dimensional matrix multiplication calculation amount in the large language model reasoning.

[0102] Table 3 Normalized power consumption of array accelerator when setting different data types

[0103]

[0104] The present invention has been compared with several typical and commonly used GPU and CPU platform devices deployed in large language reasoning performance experiments, including NVIDIA Jetson AGX Orin 64GB@2200MHz, NVIDIA Jetson NanoB01 4GB@1430MHz, Raspberry Pi 4B 4GB@1800MHz, Rockchip RK3588 SoC 16GB@1800MHz, to demonstrate the excellent performance of the invention. The entire system consists of a customized RISC-V Rocket core and two heterogeneous array accelerators (size 32×32), with a data packing length of 64 and an operating frequency of 1500MHz. The mainstream open source large language models selected in the experiment include the following four: LLaMa, Transformer, OpenAI GPT-1 and OPT. Figure 7 As shown, the latency of the platform of the design solution of the present invention and other CPU / GPU platforms when performing large language model inference, where the horizontal axis represents a set of settings for input token length and output token length.

[0105] Compared with Nvidia Nano, the acceleration method of the present invention can reach a maximum acceleration of 16.60 times. Even compared with the current best computing power Nvidia Orin, the design of the present invention is only about 3.34 times slower. Table 4 further lists the power consumption of each solution under the same large language model inference task. The design of the present invention is only less than one-tenth of Nvidia Orin. Compared with other edge platforms (such as RK3588 and Raspberry 4B), the power consumption efficiency is 3.5 times and 2.19 times that of them respectively, and the following are achieved: Figure 8 The large language model inference speed is shown to be 8.13 times and 11.70 times, which indicates that the design scheme of the present invention has extraordinary performance in terms of both latency and power consumption.

[0106] Table 4 Power consumption comparison with commercial CPU / GPU platform benchmarks on the same large model inference task

[0107]

[0108] After applying the acceleration method of the present invention, the reasoning of the LLaMa model was analyzed again, and the proportion of the matrix / vector multiplication part in the overall running time was significantly reduced, which effectively alleviated the bottleneck problem in the LLM reasoning process and proved the effectiveness of the solution.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A large language model acceleration method based on binary quantization, characterized in that: The following steps are involved: S1. Extract the parameters that need to be processed by matrix / vector multiplication in the model, and use a fixed-point quantization algorithm to perform binary quantization on the parameters of the model matrix / vector multiplication; S2. For the binary quantized parameters, a grouping and packing algorithm is used to splice the grouped binary quantized parameters into a single integer data bit by bit according to the order of calculating the matrix / vector multiplication; S3. Based on the binary multiplication principle, perform hardware-friendly matrix / vector multiplication operations equivalent to the original model inference; S4. Convert the result back to floating-point data type through the dequantization algorithm to obtain the final calculation result of the large language model in the matrix / vector multiplication operation, and pass it to the subsequent network layer to complete the calculation acceleration of the matrix / vector multiplication operation in the large language model.

2. The large language model acceleration method based on binary quantization according to claim 1, characterized in that: The specific operation process of step S1 is as follows: S11. Analyze the target large language model structure and calculation process, determine the parameter positions that need to be quantized, and the parameters that need to be quantized and accelerated are the floating-point format parameters of the linear layer and matrix multiplication operation layer in the target large language model that occupy the overall model reasoning calculation bottleneck; S12. Based on the upper and lower bounds of the data value range before and after quantization, the model weights stored in the floating point data type are symmetrically quantized to obtain binary quantized data and its corresponding scaling factor; S13. After binary quantization, in the tensor participating in the matrix / vector multiplication layer, each floating-point format data is represented by a single-bit value, and the quantized binary data in each layer of the tensor share the same quantization scaling factor.

3. The large language model acceleration method based on binary quantization according to claim 2, characterized in that: The parameters that need to be quantized and accelerated are the floating-point format parameters of the linear layer and matrix multiplication operation layer in the target large language model, which occupy the bottleneck of the overall model reasoning calculation. Based on the symmetric quantization algorithm, the absolute value range of the value range before and after quantization is set with each tensor parameter layer l of size [B, C, H, W]. and As the upper and lower bounds of the data during quantization, calculate the quantization scaling factor s and convert the original floating point data X float Transformed into binary data X quantized , save the quantized data and scaling factor. After binary quantization, the higher-precision floating-point data will be converted to a single-bit value representation. In addition, for each tensor parameter layer l, the scaling factor s when quantizing its parameters will be saved at the same time. l , used for the subsequent dequantization operation when restoring the floating-point data format.

4. The large language model acceleration method based on binary quantization according to claim 3 is characterized in that: In step S2, based on the upper and lower bounds of the data value range before and after quantization, the model weights stored in the floating-point data type are symmetrically quantized to obtain binary quantized data and its corresponding scaling factor. The symmetrical quantization process of converting floating-point values ​​into binary values ​​is shown in equations (1) and (2): Among them, x represents a floating-point parameter tensor of a certain layer of input for matrix / vector operation, s represents the scaling factor calculated by the absolute value of the value range before and after quantization, the rounding function round(·) rounds the division result, and the sign function sgn(·) is used as a truncation function during binary quantization. It judges the sign of the input value and outputs 1 when it is greater than 0 and -1 when it is less than 0. Therefore, the data distribution value after binary quantization is {-1, 1}. The obtained scaling factor s is calculated according to the data distribution of the floating-point parameter tensor of this layer. The quantized binary data in the same layer of tensors share the same scaling factor. Each input binary quantized parameter tensor layer has a corresponding scaling factor s, which will be used to restore the floating-point type result during the dequantization operation at the end of the operation.

5. The large language model acceleration method based on binary quantization according to claim 1, characterized in that: In the step S2, a data group size L is selected for packing, and the L binary quantized data are bit-wise spliced ​​into a single integer data in the calculation order of the matrix / vector multiplication operation; The data group size L may be set to a bit width of an integer data type that is more efficient than floating-point multiplication operations.

6. The large language model acceleration method based on binary quantization according to claim 1, characterized in that: In the step S3, when the integer data obtained by grouping and packaging is implemented on the software algorithm side, the operation is converted into a multiplication operation of two types of values ​​1 / -1 according to the binary quantization algorithm; when it is implemented on the hardware deployment side of the large language model accelerator, the value of -1 is mapped to 0, and then based on the two-step binary bit operation of "XOR+Popcount", the matrix / vector multiplication is equivalently implemented.

7. The large language model acceleration method based on binary quantization according to claim 6, characterized in that: When calculating binary vector multiplication, the calculation formula shown in equation (3) below is used: Among them, x and y represent the input tensors, L represents the size of the data group to be packaged, Represents the number of bits set to 1 in the binary result obtained by statistics; in addition, the type of the output value is integer data, and its bit width is related to the size of the data group used.

8. The large language model acceleration method based on binary quantization according to claim 1, characterized in that: In step S4, after the multiplication result is output, the layer-by-layer scaling factor s obtained during the previous quantization is used. l , the integer multiplication results of each layer are dequantized to obtain the final floating-point calculation results, and the results are sorted and passed to the subsequent network layers of the large language model network.

9. The large language model acceleration method based on binary quantization according to claim 8, characterized in that: The calculation formula of the floating-point output after dequantization is shown in equation (4): o float =x q-pack *y q-pack *s x s y (4) Among them, x q-pack and q-pack are two quantized and packed integer multipliers, s x 、s y are the quantization scaling factors of the corresponding layers, o float It is the floating-point output data after dequantization.

10. A large language model acceleration system based on binary quantization, characterized in that: include: The parameter extraction and quantization unit is used to extract and obtain the relevant floating-point parameters for matrix / vector multiplication operations in the model, and use a fixed-point quantization algorithm to perform binary quantization on the parameters of the model matrix / vector multiplication operations; A grouping and splicing unit, used for splicing the grouped binary quantized parameters into a single integer data bit by bit according to the order of calculating the matrix / vector multiplication; An operation realization unit, which is used to perform hardware-friendly matrix / vector multiplication operation based on the binary multiplication principle, which is equivalent to the original model reasoning; The dequantization and output unit is used to convert the result back to the floating-point data type through the dequantization algorithm, obtain the final calculation result of the large language model in the matrix / vector multiplication operation, and pass it to the subsequent network layer.

Images