A large language model quantization method based on orthogonal characteristics and accelerator architecture

Abstract

The application belongs to the technical field of large language model quantization, and particularly relates to a large language model quantization method based on orthogonal characteristics and an accelerator architecture. The quantization method divides the activation tensor of the large language model into multiple column blocks, and allocates an FP4 quantization format to the entire activation tensor with the column block as the granularity. The concept of the column block is defined as follows: the matrix of the activation tensor is divided into multiple segments with the same number of elements, wherein each element in the segment is arranged continuously in the same row in the first dimension of the matrix, and arranged in multiple continuous columns in the second dimension; the column block includes multiple columns in the second dimension, and the number of columns in each column block is consistent with the number of elements in the segment. The application overcomes the defects existing in the existing large language model grouping quantization technology, and solves the contradiction between the precision of the large language model and the hardware efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of large language model quantization, and particularly relates to a large language model quantization method based on orthogonal characteristics and an accelerator architecture. BACKGROUND

[0002] Large language models have been widely applied due to their excellent performance in various natural language processing tasks. However, the performance improvement of the model is often accompanied by a sharp increase in its size, which brings severe computing and memory resource challenges to the deployment of the model. In order to alleviate this burden, quantization has become a mainstream and effective technical solution. This technology represents the weights and activation values in the model with low-precision data types (such as 4-bit integers), which not only greatly reduces the memory occupation and bandwidth demand of the model, but also uses efficient low-precision computing units to accelerate the inference process.

[0003] Quantization can be performed at different granularities, from coarse to fine, including tensor-level, channel-level, and group-level. Among them, the group quantization processing method can significantly reduce the error introduced in the quantization process. However, when processing activation values, the existing technology generally faces a conflict between hardware efficiency and model accuracy, making it impossible for a fixed quantization format to effectively adapt to all group distributions, ultimately resulting in a decrease in accuracy.

[0004] To solve this orthogonal conflict, some existing technologies propose schemes based on reordering. Before performing group quantization, the numerical characteristics of each column in the activation tensor are analyzed, and then through physical data rearrangement, columns containing abnormal values are gathered together, while columns containing only normal values are gathered in another area. It breaks the match between the continuity of data on logical grouping and the regularity of physical storage, causing the data access pattern to become irregular and unpredictable. This dynamic rearrangement introduces a huge runtime overhead, which ultimately seriously affects the overall inference efficiency of the model.

[0005] In summary, the existing large language model activation value quantization technology has obvious technical problems: either sacrificing model accuracy or introducing non-negligible runtime overhead to maintain accuracy, making it difficult to achieve an ideal balance between model accuracy and hardware inference efficiency. Therefore, there is an urgent need for a new quantization method and hardware architecture that can achieve high accuracy and high efficiency at the same time. SUMMARY

[0006] Therefore, the application provides a large language model quantization method based on orthogonal characteristics, which is used to overcome the defects in the existing large language model group quantization technology and solve the contradiction between model accuracy and hardware efficiency.

[0007] In order to achieve the above technical purpose, the specific technical scheme adopted by the application is:

[0008] A large language model quantization method based on orthogonal characteristics, comprising: dividing an activation tensor of the large language model into a plurality of column blocks, and assigning an FP4 quantization format to the column blocks as a granularity; the column block division method is:

[0009] The matrix of the activation tensor is divided into a plurality of segments with the same number of elements; wherein each element in the segment is arranged continuously in the same row of the first dimension of the matrix, and arranged in a plurality of consecutive columns of the second dimension of the matrix; the column block includes a plurality of columns of the second dimension, and the number of columns in each column block is consistent with the number of elements in the segment;

[0010] Wherein, the method of assigning an FP4 quantization format to the column block as a granularity includes the following steps:

[0011] S101: for the selected column block, performing:

[0012] Obtain the set of all representable numerical values of the currently selected FP4 format, and use the set of numerical values to quantize all original data in the column block, and the quantization calculation formula used is:

[0013]

[0014] Wherein X represents the original data of a segment (group) in the current column block each time, T is the set of numerical values that the currently selected FP4 format can represent, s is the scaling factor, and Q(X) is the quantized data of the segment (group). After obtaining the quantized data, dequantization is performed:

[0015]

[0016] In the formula, X' is the dequantized data;

[0017] After quantizing and dequantizing the data of all segments (groups) in the selected column block based on the above formula using the currently selected FP4 format, the mean square error between the original data and the quantized data is calculated and recorded;

[0018] S102: traverse all candidate FP4 formats within the selected column block, specifically, select different candidate FP4 formats to execute S101, and determine the candidate FP4 format that produces the smallest mean square error as the optimal format index of the selected column block;

[0019] S103: performing S101-S102 for all column blocks of the activation tensor to obtain a format index array of the activation tensor.

[0020] Further, the candidate FP4 formats include: an E3M0 format, an E1M2 format, and an E2M1 format, for the E3M0, the set of representable values is {0, ±0.25, ±0.5, ±1, ±2, ±4, ±8, ±16}; for the E2M1, the set of representable values is {0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6}; for the E1M2, the set of representable values is {0, ±0.5, ±1, ±1.5, ±2, ±2.5, ±3, ±3.5}.

[0021] The application further provides a large language model quantization accelerator architecture for implementing the large language model quantization method.

[0022] a core computing array including a plurality of processing units arranged in an array, configured to perform matrix multiplication operations;

[0023] a data cache module configured to cache activation data in an FP4 format, weights in an INT4 format, quantization parameters, and output and format index caches;

[0024] a control and scheduling module including a top-level controller and an address generator (AG). The address generator, under the control of the top-level controller, first reads the format index of the current column block from the format index cache and places it in an index register, determines whether it is 0, if it is 0, then the address in the address register is incremented by 1 in the next cycle, and the next segment (group) of weights, activations, quantization parameters, and format indexes are controlled to be read and written; if it is not 0, then the data in the index register is decremented by 1 cycle by cycle, the address in the address register is incremented by 0 in the next cycle, and the weights, activations, quantization parameters of the current segment (group) are still controlled to be read and written until the data in the index register is decremented to 0.

[0025] a data processing and conversion module including a bit serial term generator and a vector unit; the bit serial term generator is configured to convert the activation data in the FP4 format into the bit serial format required by the core computing array; and the vector unit is configured to execute an activation function.

[0026] Further, the address generator generates an address for reading cache data from the data cache module, and the data corresponding to the address is transmitted directly or after processing to the core computing array; the bit serial term generator decomposes the activation data taken from the data cache module into bit serial terms, and transmits the bit serial terms, the weights in the INT4 format taken from the data cache module, and the quantization parameters to the core computing array; the core computing array transmits the calculation results to the vector unit, and the vector unit transmits the processed calculation results to the data cache module.

[0027] Further, the processing unit includes:

[0028] The group-serial multiplication accumulation unit internally contains multiple parallel calculation channels.

[0029] The scaling unit includes an FP16 multiplier and a subtractor, which are used to calculate the dequantization factor of each segment, multiply the scaling factor of the FP4 activation data and the scaling factor of the INT4 weight, and subtract the exponent part of the multiplication result from the format index to compensate for the implicit scaling introduced by the bit-serial calculation in the group-serial multiplication accumulation unit.

[0030] The cross-group accumulator includes a multiplier, a data format conversion unit, and an adder, which receive the calculation result of the group-serial multiplication accumulation unit and the dequantization factor of the scaling unit, perform a dequantization operation, accumulate the dequantized floating-point results of each segment with the accumulated results of the previous segments stored in the register, and store the result in the register for accumulation with the dequantized results of the next segment.

[0031] Further, the group-serial multiplication accumulation unit works in a bit-serial manner, receives one bit of the mantissa of the FP4 activation data and the complete INT4 weight in each clock cycle, multiplies them in the channel to generate a partial product, left-shifts the partial product according to the exponent of the FP4 activation data to align it, and sends the aligned product to the shift accumulator for accumulation.

[0032] Further, the bit-serial term includes a 1-bit sign bit, a 3-bit exponent, and a single bit of the mantissa to be processed in the current cycle.

[0033] The rule for indicating the correct position of the exponent bit based on the format index is as follows: if the format index is 0, the last three bits of the bit-serial term are the exponent, if it is 1, the middle two bits of the bit-serial term are the exponent, and if it is 2, the next highest bit of the bit-serial term is the exponent.

[0034] Further, the bit-serial term generator takes the last three bits of the activation data in FP4 format and puts them into the mantissa register.

[0035] The mantissa register performs implicit bit addition on the activation data in FP4 format according to the format index: if the format index is 0, the implicit bit is put into the lowest bit of the mantissa register, if it is 1, the implicit bit is put into the next highest bit, and if it is 2, the implicit bit is put into the highest bit, and then the mantissa bits are output in a stream.

[0036] The mantissa register controls the streaming out of mantissa bits according to an index register in the address generator; the signal of the index register is initialized according to the format index of the current processing group, and is decremented every clock cycle; in each cycle, the signal of the index register controls the data output on the corresponding bit in the mantissa register until the transmission of all mantissa bits required for the activation data of the current FP4 format is completed.

[0037] With the technical scheme, the application can bring the following beneficial effects:

[0038] The application can significantly improve the reasoning speed: unlike the prior art which needs to perform high-cost dynamic data rearrangement, the application directly operates on the original, hardware-friendly data layout, fundamentally avoiding the huge runtime overhead and calculation delay introduced by data rearrangement; and the bit-serial structure proposed by the application can efficiently process calculations of different FP4 formats, greatly reducing the overall delay of the system.

[0039] The application can achieve high model precision: the application achieves the same or even better precision level as the reordering scheme through an innovative way.

[0040] The application can greatly reduce system energy consumption: the energy efficiency advantage of the application is derived from the bit-serial structure which avoids the redundant operations and power waste caused by the mismatch between data bit widths when the traditional fixed-width processor processes low-precision data. In addition, the application is faster under the same task, so the overall energy consumption is reduced.

[0041] The application has extremely low hardware overhead: the hardware modules required for the implementation of the above-mentioned adaptive quantization function of the application are mainly the address generator (AG) and the bit-serial item generator (BSG). According to circuit synthesis evaluation, the hardware overhead introduced by these key modules is extremely small, accounting for less than 1% of the total chip area and about 1.12% of the total power consumption. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0043] Figure 1 A standard deviation distribution diagram of each segment in the activation tensor in the large language model quantization method based on orthogonal characteristics in the specific embodiment of the present application;

[0044] Figure 2A flowchart for determining a quantization format in a large language model quantization method based on orthogonal characteristics in the specific embodiment of the present application is shown in FIG. 1.

[0045] Figure 3 An accelerator architecture diagram for a large language model quantization accelerator in the specific embodiment of the present application is shown in FIG. 2.

[0046] Figure 4 A micro-architecture diagram of a processing unit in the specific embodiment of the present application is shown in FIG. 3.

[0047] Figure 5 A running mode diagram of an accelerator in the specific embodiment of the present application is shown in FIG. 4, where A i[j] represents segment A i the jth bit serial term. DETAILED DESCRIPTION

[0048] The embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0049] The embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0050] It should be noted that various aspects of the embodiments described below are within the scope of the appended claims. It should be apparent that the aspects described herein can be implemented in a wide variety of forms and that any specific structure and / or function described herein is merely illustrative. Based on the teachings provided herein one skilled in the art will appreciate the various ways in which the aspects described herein can be implemented in a variety of different environments and / or with a variety of different devices. Any and all such variations are considered to be within the scope of the present disclosure.

[0051] It is also necessary to note that the drawings in the following embodiments only illustrate the basic concept of the present disclosure in a schematic manner, and only show the components related to the present disclosure, not the number, shape and size of the components when actually implemented. The actual implementation of each component may be a random change, and the component layout pattern may be more complex.

[0052] In addition, in the following description, specific details are provided in order to facilitate a thorough understanding of examples. However, one skilled in the relevant art will understand that the aspects described can be practiced without these specific details.

[0053] In one embodiment of the present application, a large language model quantization method based on orthogonal features is provided, comprising: dividing the activation tensor of the large language model into multiple column blocks, and assigning an FP4 quantization format with the column block as the granularity;

[0054] The column block segmentation method is:

[0055] The matrix of the activation tensor is divided into multiple segments with the same number of elements, wherein each element in the segment is arranged continuously in the same row in the first dimension of the matrix, and arranged in multiple consecutive columns in the second dimension of the matrix; the column block includes multiple columns in the second dimension, and the number of columns in each column block is consistent with the number of elements in the segment;

[0056] The method of assigning an FP4 quantization format with the column block as the granularity includes the following steps:

[0057] S101: Perform the following steps for the selected column block:

[0058] Get the set of all representable values of the currently selected FP4 format. For E3M0, the set of representable values is {0, ±0.25, ±0.5, ±1, ±2, ±4, ±8, ±16}; for E2M1, the set of representable values is {0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6}; for E1M2, the set of representable values is {0, ±0.5, ±1, ±1.5, ±2, ±2.5, ±3, ±3.5}. Select a set of values to quantize all original data in the selected column block. The quantization calculation formula used is:

[0059]

[0060] Where X represents the original data of a segment (group) in the current column block each time, T is the set of representable values of the currently selected FP4 format, s is the scaling factor, and Q(X) is the quantized data of the segment (group). After obtaining the quantized data, dequantization is performed:

[0061]

[0062] After quantizing and dequantizing the data of all segments (groups) in the selected column block using the currently selected FP4 format based on the above formula, the mean square error between the original data and the quantized data is calculated and recorded;

[0063] S102: Traverse all candidate FP4 formats within the selected column block, specifically, select different candidate FP4 formats to execute S101, and determine the candidate FP4 format that produces the smallest mean square error as the optimal format index of the selected column block;

[0064] S103: Perform S101-S102 for all column blocks of the activation tensor to obtain the format index array of the activation tensor.

[0065] The core principle of the method of the embodiment is to utilize the "orthogonal characteristics" of the activation value distribution.

[0066] The embodiment is based on a key observation that the statistical distribution of the data divided by segments (groups) in the activation tensor of a large language model has "orthogonal characteristics".

[0067] The concepts of "group" and "column block" in the embodiment are as follows:

[0068] A column block is composed of several adjacent columns in the activation tensor, that is, the number of columns is consistent with the size of the group, or in other words, a column block is composed of "groups" in the same column.

[0069] For example, for an activation matrix of [2048, 4096], if the size of the group is 128 (the number of elements is 128) and the groups are continuously arranged in the first dimension, and if the activation matrix is divided into 4096 / 128=32 along the second dimension, then a column block refers to 32 continuous columns, and the column block size is [2048, 32].

[0070] The "group" and "column block" in the embodiment are two different granularity concepts: "column block" is the granularity of format data allocation. In the embodiment, "group" is the granularity of quantization, and the data format of the groups in each column block is consistent, and each group in the column block is quantized to the same data format.

[0071] As shown in FIG. 2, the data of the selected column block is quantized and dequantized using the currently selected FP4 format based on the above formula, and the mean square error between the original data and the quantized data is calculated and recorded. Figure 1As shown in the figure, the figure shows the standard deviation distribution of each group in the activation tensor. Each column in the figure represents the standard deviation value of a group. It can be clearly seen that the standard deviation shows obvious consistency along the row axis direction, but fluctuates greatly along the column axis direction. This indicates that the groups in the same "column block" (i.e. all groups sharing the same column index) have similar distribution characteristics; the distribution characteristics of groups in different column blocks are significantly different. The column block containing the column of the outlier tends to be Laplace distribution; the column block not containing the outlier is closer to uniform distribution. Therefore, the embodiment actually takes the "column block" as the selected granularity of the activation tensor data format.

[0072] At the same time, in order to adapt to different column block data distributions, the embodiment adopts FP4 as the expression format of the activation tensor, which can flexibly fit different data distributions by adjusting the allocation of the exponent and the mantissa bits. Among them, E3M0 (3-bit exponent, 0-bit mantissa): large dynamic range, low precision, suitable for representing column blocks containing outliers and similar to Laplace distribution; E1M2 (1-bit exponent, 2-bit mantissa): small dynamic range, but high precision and uniform distribution, equivalent to INT4 format, suitable for representing uniformly distributed column blocks; E2M1 (2-bit exponent, 1-bit mantissa): balanced between dynamic range and precision, suitable for representing column blocks between the two, similar to Gaussian distribution.

[0073] At the computer level, the language model quantization method of the embodiment determines the optimal FP4 format for each column block in the activation tensor through an offline calibration process, and the flow chart is as follows Figure 2 , and the specific steps are as follows:

[0074] Step one (initialization): provide the activation tensor A to be processed (its dimension is N x (D / G) x G, where N is the number of rows, D / G is the number of column blocks, and G is the group size), and the candidate FP4 format set F = {E3M0, E2M1, E1M2}.

[0075] Step two (traverse column blocks): traverse each column block index j (from 0 to D / G-1).

[0076] Step three (traverse candidate formats): in each column block, traverse three candidate formats F k (k from 0 to 2), where F0 represents E3M0, F1 represents E2M1, and F2 represents E1M2.

[0077] Step four (simulated quantization and error calculation): for the current column block A [:,j,:] and the current candidate format F k, first obtain the set of values that can be represented by the format, for E3M0, the set of values that can be represented is {0, ±0.25, ±0.5, ±1, ±2, ±4, ±8, ±16}; for E2M1, the set of values that can be represented is {0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6}; for E1M2, the set of values that can be represented is {0, ±0.5, ±1, ±1.5, ±2, ±2.5, ±3, ±3.5}. Using the current candidate format F k The set of values quantizes all original data in the column block, and the quantization calculation formula used is:

[0078]

[0079] Where X represents all original data of a segment (group) in the current column block each time, T is the set of values that can be represented by the current FP4 format, s is the scaling factor, and Q(X) is the quantized data. After obtaining the quantized data, dequantization is performed:

[0080]

[0081] After quantizing and dequantizing the data of all segments (groups) in the column block using the format based on the above formula, the dequantized data is obtained. [:,j,:] Finally, the mean square error between the original data A [:,j,:] and the dequantized data A j is calculated.

[0082] Step five (select the optimal format): compare the error generated by the current format with the minimum error recorded for the column block, if the current error is smaller, update the minimum error, and record the index k of the current format as the optimal format index I j of the column block.

[0083] Step six (generate format index): after completing the traversal of all candidate formats, the optimal format index I j of the current column block j is determined. Repeat steps two to five until all column blocks find their optimal formats.

[0084] Step seven (output): finally, return a one-dimensional array I containing the optimal format index corresponding to each column block.

[0085] In another embodiment of the application, a large language model quantization accelerator architecture is proposed for efficiently implementing the above-mentioned embodiment of the large language model quantization method based on orthogonal characteristics, comprising:

[0086] A core computing array including a plurality of processing units arranged in an array for performing matrix multiplication operations;

[0087] a data buffer module for buffering activation data in FP4 format, weights in INT4 format, quantization parameters, and output and format index buffers;

[0088] a control and scheduling module including a top-level controller and an address generator (AG). The address generator, under the control of the top-level controller, first reads the format index of the current column block from the format index buffer and places it in an index register. If the index is 0, the address in the address register is incremented by 1 in the next cycle, and the weights, activation, quantization parameters, and format index of the next segment (group) are controlled to be read and written. If the index is not 0, the data in the index register is decremented by 1 in each cycle, and the address in the address register is incremented by 0 in the next cycle. The weights, activation, quantization parameters of the current segment (group) are still controlled to be read and written until the data in the index register is decremented to 0.

[0089] a data processing and conversion module including a bit-serial term generator and a vector unit. The bit-serial term generator is used to convert the FP4 activation data into the bit-serial format required by the core computation array. The vector unit is used to execute the activation function.

[0090] wherein the address generator generates the address for reading the buffered data from the data buffer module, and the data corresponding to the address is transmitted directly or after processing to the core computation array; the bit-serial term generator decomposes the activation data taken from the data buffer module into bit-serial terms, and transmits the bit-serial terms and the weights in INT4 format and the quantization parameters taken from the data buffer module to the core computation array; the core computation array transmits the computation results to the vector unit, and the vector unit performs nonlinear operations on the data and then transmits the processed computation results to the data buffer module.

[0091] The processing unit includes:

[0092] a grouped bit-serial multiply-accumulate unit, which internally includes multiple parallel computation paths;

[0093] a scaling unit including FP16 multipliers and subtractors, which is used to calculate the dequantization factors of each segment, to multiply the scaling factor of the activation data in FP4 format and the scaling factor of the weights in INT4 format, and to subtract the exponent part of the multiplication result from the format index to compensate for the implicit scaling introduced by the bit-serial computation in the grouped bit-serial multiply-accumulate unit;

[0094] The cross-group accumulator, including a multiplier, a data format conversion unit and an adder, receives the calculation result of the group-serial multiplication accumulation unit and the dequantization factor of the scaling unit, performs a dequantization operation, and accumulates the dequantized floating point number result of each segment with the accumulated result of each segment stored in the register, and stores the result in the register for accumulation with the dequantized result of the next segment.

[0095] In the embodiment, the group-serial multiplication accumulation unit works in a bit-serial manner, receives one bit of the mantissa of the active data in FP4 format and the complete weight in INT4 format in each clock cycle, multiplies them in the pipeline to generate a partial product, generates the sign of the partial product according to the sign bit of the active data and the weight, left-shifts the partial product to align according to the exponent of the active data in FP4 format, and sends the aligned product to the shift accumulator for accumulation.

[0096] The bit-serial term of the embodiment includes one bit of the sign bit, three bits of the exponent and one bit of the mantissa of the single bit to be processed in the current cycle;

[0097] If the format index is 0, the last three bits of the bit-serial term are the exponent, if the format index is 1, the middle two bits of the bit-serial term are the exponent, and if the format index is 2, the second highest bit of the bit-serial term is the exponent.

[0098] The bit-serial term generator of the embodiment reads the highest bit of the data as the sign bit and indicates the correct position of the exponent bit according to the format index;

[0099] The bit-serial term generator of the embodiment takes the last three bits of the active data in FP4 format into the mantissa register;

[0100] The mantissa register performs implicit bit addition on the active data in FP4 format according to the format index, if the format index is 0, the implicit bit is put into the lowest bit of the mantissa register, if the format index is 1, the implicit bit is put into the second highest bit, and if the format index is 2, the implicit bit is put into the highest bit, and then the mantissa bits are output in a stream;

[0101] The mantissa register controls the output of the mantissa bits according to the index register in the address generator; the signal of the index register is initialized according to the format index of the group currently processed and decremented in each clock cycle; in each cycle, the signal of the index register controls the output of the data in the corresponding bit of the mantissa register until the transmission of all the mantissa bits required by the active data in FP4 format is completed.

[0102] Further description: accelerator overall architecture.

[0103] Please refer to the accompanying drawings Figure 3Figure 1 shows the overall architecture of the language model quantization accelerator of the embodiment, mainly composed of the following modules:

[0104] Core computation array, a 16x16 processing element (PE) array, for performing matrix multiplication operations;

[0105] Data cache module, including FP4 activation cache, INT4 weight cache, quantization parameter cache, output cache and format index cache;

[0106] Control and scheduling module, including top-level controller and address generator (AG). AG is responsible for generating accurate read-write addresses and control signals according to the format index, and is the key to achieving efficient data flow;

[0107] Data processing and conversion module, including bit-serial term generator (BSG) and vector unit. BSG is responsible for converting FP4 activation values into bit-serial format required by PE, and vector unit is responsible for performing activation function and other nonlinear operations.

[0108] The workflow of the language model quantization accelerator of the embodiment is as follows: first, the data such as weights, activation values, and format indexes are preloaded from external memory to on-chip cache. Then, AG generates addresses to fetch data from on-chip cache, BSG decomposes the fetched activation values into bit-serial terms, and flows into the PE array together with the fetched weights and quantization parameters. The calculation results are sent to the vector unit for processing, and then stored in the output cache, and finally written back to the external memory.

[0109] The microarchitecture of the core processing element (PE) of the embodiment is shown in Figure 2. Figure 4 Figure 2 shows the internal microarchitecture of a single PE in detail. Each PE is composed of three sub-modules:

[0110] Module 1) Grouped bit-serial multiply-accumulate unit. Structure: This unit contains 64 parallel computing paths inside, used to process a data group of size 64 at the same time. Working principle: This unit works in bit-serial mode. In each clock cycle, it receives one bit of the tail of the activation value and the complete INT4 weight. Inside the path, the two are multiplied to generate partial products, and then aligned by left shift according to the exponent of the activation value, and then sent to a shift accumulator for accumulation. According to the number of tail bits of the input FP4 format, this unit will take 1 to 3 cycles to complete the dot product calculation of one data group (E3M0 is 1 cycle, E2M1 is 2, and E1M2 is 3).

[0111] Module 2) Scaling unit. Structure: mainly composed of an FP16 multiplier and a subtractor. Working principle: this unit is responsible for calculating the dequantization factor of the group. It multiplies the scaling factor of the activation value and the scaling factor of the weight. At the same time, it subtracts the exponent part of the multiplication result from the format index to compensate for the implicit scaling introduced by the bit-serial calculation in module 1).

[0112] Module 3) Cross-group accumulator. Structure: composed of a multiplier, a data format conversion unit and an adder. Working principle: this unit receives the calculation result from module 1) and the dequantization factor from module 2), performs the dequantization operation, and accumulates the dequantized floating-point number result with the result of the previous data group.

[0113] The three modules work in a pipelined manner, i.e. while modules 1) and 2) process the current group, module 3) processes the previous group, improving the overall processing efficiency of the PE.

[0114] The structure and working principle of the bit-serial item generator (BSG) in this embodiment are described in detail in the following. Figure 3 The overall architecture diagram of the large language model quantization accelerator and the partial enlarged view on the right side thereof are shown in FIG. 1. The BSG is a key data conversion module connecting the "FP4 activation cache" and the "core calculation array". Its core function is to convert the FP4 format activation value stored in the "FP4 activation cache" into the bit-serial item required by the bit-serial calculation of the PE array on a cycle-by-cycle basis in real time. The item consists of three parts: a 1-bit sign bit, a 3-bit exponent, and a single-bit mantissa bit to be processed in the current cycle.

[0115] The working principle of the BSG is as follows:

[0116] Read the highest bit of the data as the sign bit;

[0117] According to the format index, the correct position of the exponent bit is indicated. If the format index is 0, the last three bits are the exponent, if it is 1, the middle two bits are the exponent, and if it is 2, the next highest bit is the exponent;

[0118] The last three bits are put into the mantissa register. For the mantissa, since it contains an implicit bit, it needs to be judged whether the implicit bit is 0 or 1 according to whether the data is a normalized number or a non-normalized number. For the normalized number, an implicit '1' needs to be added in front of the mantissa. For the non-normalized number (i.e. the case where the exponent part is all 0), the actual effective exponent should be added by 1, and an implicit '0' is added in front of the mantissa. The BSG will automatically judge and complete the addition of this implicit bit according to the format index. If the format index is 0, the implicit bit is put into the lowest bit of the mantissa register, if it is 1, it is put into the next highest bit, and if it is 2, it is put into the highest bit. Then the mantissa bit stream output is performed. The BSG controls the mantissa bit stream output through the "index register" in the AG. The register is initialized according to the format index of the current processed group, and the register is decremented every clock cycle. In each cycle, the signal of the index register controls the data output on the corresponding bit in the mantissa register until the transmission of all the mantissa bits required by the format is completed.

[0119] The data flow of the large language model quantization accelerator of the embodiment is described in Figure 5 .

[0120] The embodiment adopts a unique column-consistent pulsatile data flow, which is a soft and hardware collaborative design.

[0121] Software constraint: the above-mentioned large language model quantization method can be carried by software to ensure that all data groups in the same column block use the same FP4 format, which means that the number of calculation cycles required to process these data groups is completely the same and predictable.

[0122] Hardware utilization: the AG module of the large language model accelerator of the embodiment utilizes this predictability. Referring to Figure 3 the AG logic diagram in the upper right corner, the AG reads the format index of the current column block from the index cache and determines the number of cycles required for calculation according to the format index. If the format index is 0, the number of cycles is 1; if the format index is 1, the number of cycles is 2; if the format index is 2, the number of cycles is 3. Within these cycles, the AG continuously issues the same address signal to make the activation cache, weight cache and quantization parameter cache repeatedly provide the same operands until the current calculation is completed. The specific implementation method is: the address generator first reads the format index of the current column block from the format index cache under the control of the top-level controller and puts it into the index register. If it is 0, the address in the address register is added by 1 in the next cycle, and the weights, activations, quantization parameters and format indexes of the next segment (group) are controlled to be read and written. If it is not 0, the data in the index register is decremented by 1 every cycle, and the address in the address register is added by 0 in the next cycle, and the weights, activations, quantization parameters of the current segment (group) are still controlled to be read and written until the data in the index register is decremented to 0.

[0123] Effect: As shown in the accompanying Figure 5 This scheduling mechanism ensures that when the upstream PE finishes the computation and is ready to pass the data, the downstream PE also finishes the computation and is ready to receive the data. This makes the data propagate rhythmically and without pause in the PE array, so that the pipeline can be fully loaded when performing fine-grained mixed-precision tasks, avoiding pipeline bubbles caused by mismatched computation periods.

[0124] The following describes the experimental results of the quantization method and accelerator architecture described in this embodiment in terms of model accuracy, inference speed, and energy consumption.

[0125] To verify the effectiveness of the proposed quantization method in maintaining model performance, this embodiment conducts a systematic evaluation on multiple models on the Wikitext-2 dataset, and compares it with the original FP16 model, SmoothQuant, OmniQuant, Atom, ANT, Olive, and Tender (all are model-specific names in the field, usually not translated). The perplexity (PPL) results are shown in Table 1 (the smaller the PPL, the better the effect, and "-" indicates that no relevant data is provided). The accuracy of the model under two quantization settings in this embodiment is better than or equal to other methods.

[0126] Table 1 Model Accuracy Results

[0127]

[0128] To verify the effectiveness of the proposed quantization method and accelerator architecture in improving inference speed and reducing energy consumption, in the case of a sequence length of 2048, it is compared with existing mainstream software and hardware co-design frameworks (ANT, Olive, and Tender). In terms of performance, the accelerator of this embodiment achieves an average inference speedup of 4.95 times (compared to ANT), 4.38 times (compared to Olive), and 2.11 times (compared to Tender). In terms of energy efficiency, this embodiment saves 60.58% (compared to ANT), 60.61% (compared to Olive), and 26.42% (compared to Tender) of total energy consumption. And this significant performance and energy efficiency gain is achieved without sacrificing model accuracy. Unlike ANT, Olive, and Tender, which experience a significant drop in accuracy at 4-bit quantization, the model perplexity of this embodiment remains at an optimal level. Therefore, this embodiment achieves an excellent balance between model accuracy and hardware deployment efficiency.

[0129] The above merely provides the specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present disclosure, which should be covered in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A method for quantizing large language models based on orthogonal features, characterized in that, include: The activation tensor of the large language model is divided into multiple column blocks, and the FP4 quantization format is assigned at the granularity of the column blocks. The column block segmentation method is as follows: the matrix of the activation tensor is divided into multiple segments with the same number of elements; each element in the segment is arranged continuously in the same row of the first dimension of the matrix, and each element in the segment is arranged in multiple consecutive columns of the second dimension of the matrix; the column block includes multiple columns of the second dimension, and the number of columns in each column block is the same as the number of elements in the segment; The method for allocating FP4 quantization format at the column block level includes the following steps: S101: Execute on the selected column block: Obtain the set of all representable values ​​in the currently selected FP4 format, use the set of values ​​to quantize and dequantize all the original data in the column block to obtain dequantized data, and then calculate and record the mean square error between the original data and the dequantized data. S102: For the selected column block, each candidate FP4 format is selected and S101 is executed respectively, and the candidate FP4 format that produces the minimum mean square error is determined as the optimal format index of the selected column block. S103: Perform S101-S102 for all column blocks of the activated tensor to obtain the format index array of the activated tensor.

2. The large language model quantization method based on orthogonal features according to claim 1, characterized in that, The candidate FP4 formats include: E3M0 format, E1M2 format and E2M1 format.

3. A large language model quantization accelerator architecture for executing the large language model quantization method based on orthogonal features as described in claim 1 or 2, characterized in that, include: The core computing array consists of multiple processing units arranged in an array to perform matrix multiplication operations; The data caching module is used to cache activation data in FP4 format, weights in INT4 format, quantization parameters, and output and format index caches. The control and scheduling module includes a top-level controller and an address generator; under the control of the top-level controller, the address generator generates precise read / write addresses and control signals according to the format index. The data processing and conversion module includes a bit serial item generator and a vector unit; the bit serial item generator is used to convert FP4 format activation data into the bit serial format required by the core computing array; the vector unit is used to execute the activation function.

4. The large language model quantization accelerator architecture according to claim 3, characterized in that, The address generator generates an address for reading cached data from the data cache module, and the data corresponding to the address is transmitted to the core computing array; the bit serial item generator decomposes the FP4 format activation data retrieved from the data cache module into bit serial items, and transmits the bit serial items and the INT4 format weights and quantization parameters retrieved from the data cache module to the core computing array; the core computing array transmits the calculation results to the vector unit, and the vector unit transmits the processed calculation results to the data cache module.

5. The large language model quantization accelerator architecture according to claim 4, characterized in that, The processing unit includes: The grouped bit serial multiplication and accumulation unit contains multiple parallel computation paths. The scaling unit, including an FP16 multiplier and a subtractor, is used to calculate the dequantization factor of each segment, multiply the scaling factor of the FP4 format activation data and the scaling factor of the INT4 format weights, and then subtract the format index from the exponent part of the multiplication result, in order to compensate for the implicit scaling introduced by the bit serial calculation in the grouped bit serial multiplication accumulation unit. The cross-group accumulator includes a multiplier, a data format conversion unit, and an adder. After receiving the calculation result of the group bit serial multiplication and accumulation unit and the dequantization factor of the scaling unit, it performs a dequantization operation, accumulates the dequantized floating-point result of each segment with the accumulation result of the previous segment stored in the register, and stores the result in the register for accumulation with the dequantization result of the next segment.

6. The large language model quantization accelerator architecture according to claim 5, characterized in that, The grouped bit-serial multiply-accumulate unit operates in a bit-serial mode. In each clock cycle, it receives one bit of the mantissa of the FP4 format activation data and the complete weight in the INT4 format, multiplies them in the path to generate a partial product, generates the sign of the partial product according to the sign bits of the activation and weight, performs left shift alignment on the partial product according to the exponent of the FP4 format activation data, and sends the aligned product to the shift accumulator for accumulation.

7. The large language model quantization accelerator architecture according to claim 6, characterized in that, The bit serialization entry of the activation data in the FP4 format includes a 1-bit sign bit, a 3-bit exponent, and a mantissa bit representing the single bit to be processed in the current cycle.

8. The large language model quantization accelerator architecture according to claim 7, characterized in that, The bit serial item generator reads the highest bit of the data as the sign bit and indicates the correct position of the exponent bit according to the format index.

9. The large language model quantization accelerator architecture according to claim 8, characterized in that, The bit serial item generator uses the following rule to indicate the correct position of the exponent bit based on the format index: if the format index is 0, the last three bits of the FP4 format activation data are the exponent; if the format index is 1, the middle two bits of the FP4 format activation data are the exponent; and if the format index is 2, the second highest bit of the FP4 format activation data is the exponent.

10. The large language model quantization accelerator architecture according to claim 9, characterized in that, The bit serial item generator takes the last three bits of the FP4 format activation data and puts them into the mantissa register; The mantissa register performs implicit bit addition on the FP4 format activation data according to the format index. If the format index is 0, the implicit bit is placed in the least significant bit of the mantissa register; if it is 1, the implicit bit is placed in the second most significant bit; if it is 2, the implicit bit is placed in the most significant bit. Then, the mantissa bits are streamed out. The mantissa register controls the streaming output of mantissa bits according to the index register in the address generator; the signal of the index register is initialized according to the format index of the group being processed and decrements every clock cycle; in each cycle, the signal of the index register controls the data output of the corresponding bit in the mantissa register until all mantissa bits required for the current FP4 format activation data are transmitted.

Images