Model reasoning method and device, equipment and storage medium

Abstract

The invention provides a model reasoning method, device and equipment and a storage medium, and relates to the technical field of computers, in particular to the artificial intelligence fields of neural networks, model reasoning, model quantification and the like. According to the specific implementation scheme, a to-be-processed quantization weight matrix of a model is read into a register of a processor from a cache of the processor; performing inverse quantization fusion calculation based on elements of the quantization weight matrix to be processed in the register and an activation matrix of the model to obtain a reasoning result of a current operator of the model; and writing the reasoning result of the current operator into a memory of the processor for operators after the current operator to use. The calculation amount in the reasoning process can be reduced, and the calculation speed and efficiency of reasoning can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the field of artificial intelligence, including neural networks, model inference, and model quantization. Background Technology

[0002] Inference in neural network models primarily relies on high-precision floating-point weights and activations for computation. While this high-precision approach improves model accuracy, it incurs significant storage and computational overhead, slows inference speed, and suffers from poor hardware compatibility. Quantization schemes can be employed to compress the model's weights and activations, thereby reducing the storage and computational costs of the inference process. Summary of the Invention

[0003] This disclosure provides a model reasoning method, apparatus, device, and storage medium.

[0004] According to one aspect of this disclosure, a model reasoning method is provided, comprising:

[0005] The quantization weight matrix of the model to be processed is read from the processor's cache into the processor's register;

[0006] Based on the elements of the quantization weight matrix to be processed in the register and the activation matrix of the model, inverse quantization fusion calculation is performed to obtain the inference result of the current operator of the model;

[0007] The inference result of the current operator is written into the processor's memory for use by subsequent operators.

[0008] According to another aspect of this disclosure, a model inference apparatus is provided, comprising:

[0009] The quantization weight reading module is used to read the quantization weight matrix of the model to be processed from the processor's cache into the processor's register;

[0010] The dequantization fusion calculation module is used to perform dequantization fusion calculation based on the elements of the quantization weight matrix to be processed in the register and the activation matrix of the model, so as to obtain the inference result of the current operator of the model.

[0011] The storage module is used to write the inference result of the current operator into the processor's memory for use by subsequent operators.

[0012] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0013] At least one processor; and

[0014] The memory is communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods of any embodiment of the present disclosure.

[0016] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform a method according to any embodiment of this disclosure.

[0017] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a method according to any embodiment of this disclosure.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0020] Figure 1 This is a schematic flowchart of a model reasoning method according to an embodiment of the present disclosure;

[0021] Figure 2 This is a schematic flowchart of a model reasoning method according to another embodiment of the present disclosure;

[0022] Figure 3 This is a schematic flowchart of a model reasoning method according to another embodiment of the present disclosure;

[0023] Figure 4 This is a schematic flowchart of a model reasoning method according to another embodiment of the present disclosure;

[0024] Figure 5 This is a schematic flowchart of a model reasoning method according to another embodiment of the present disclosure;

[0025] Figure 6 This is a flowchart illustrating the process of using the Stream-K algorithm to partition a matrix by dimension.

[0026] Figure 7 This is a flowchart illustrating the process of using the Stream-K algorithm to partition the matrix into blocks;

[0027] Figure 8 This is a schematic flowchart of a model reasoning method according to another embodiment of the present disclosure;

[0028] Figure 9 This is a flowchart illustrating the model preprocessing and transformation process;

[0029] Figure 10 This is a flowchart illustrating the weight changes;

[0030] Figure 11 This is a flowchart illustrating the neural network inference and computation process;

[0031] Figure 12 This is an example diagram illustrating the computational process of a fusion matrix multiplication.

[0032] Figure 13 This is a flowchart illustrating the non-uniform 4-bit weighted dequantization lookup table algorithm.

[0033] Figure 14 This is a schematic block diagram of a model reasoning apparatus according to an embodiment of the present disclosure;

[0034] Figure 15 This is a schematic block diagram of a model reasoning apparatus according to another embodiment of the present disclosure;

[0035] Figure 16 This is a block diagram of an electronic device used to implement embodiments of the present disclosure. Detailed Implementation

[0036] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0037] The term "and / or" in this disclosure indicates that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The term "at least one" in this document means any combination of at least two of a plurality of options, such as including at least one of A, B, and C, which can mean including any one or more elements selected from the set of A, B, and C. The terms "first" and "second" in this document refer to and distinguish multiple similar technical terms, and do not imply a specific order or a limitation to only two. For example, "first feature" and "second feature" refer to two types / two features; the first feature can be one or more, and the second feature can also be one or more.

[0038] The model inference phase requires calculating weights and activations using high-precision floating-point numbers, such as 32-bit or 16-bit floating-point numbers. However, this method of using high-precision weights and activations results in a large model size, consuming significant storage space and increasing data transfer and processing costs, thus incurring high storage and computational overhead. High-precision calculations require more computing resources, including GPU memory and computing power, leading to a decrease in inference speed, especially on devices with limited hardware resources. High-precision models have high hardware requirements; some devices with limited resources may not be able to support the inference of large-scale neural network models, resulting in poor hardware compatibility.

[0039] To reduce storage and computational overhead, 8-bit or 4-bit quantization methods can be used to compress weights and activations. Compared to 8-bit quantization, 4-bit quantization saves more GPU memory and memory access overhead during computation. However, this method incurs a certain loss of quantization accuracy. Although pure 4-bit quantization can reduce storage requirements, in some cases, this quantization may lead to a significant decrease in the inference accuracy of the neural network, affecting model performance.

[0040] Figure 1 This is a flowchart illustrating a model inference method according to an embodiment of the present disclosure. The method may include:

[0041] S110. Read the quantization weight matrix of the model to be processed from the processor's cache into the processor's register;

[0042] S120. Based on the elements of the quantization weight matrix to be processed in the register and the activation matrix of the model, perform inverse quantization fusion calculation to obtain the inference result of the current operator of the model;

[0043] S130. Write the inference result of the current operator into the memory of the processor for use by subsequent operators.

[0044] In this embodiment of the disclosure, the model, such as a neural network model, may have nonlinear layers and matrix layers. Matrix layers are typically positioned between different nonlinear layers. For example, the nonlinear layers of the model may include normalization layers, fully connected layers, attention layers, pooling layers, etc. Nonlinear layers (or nonlinear layer operators) can output activation matrices. Matrix layers (or matrix layer operators) can have corresponding weight matrices. The computing unit of the processor, such as a stream multiprocessor (SM) of a GPU, can obtain the quantized weight matrix to be processed based on the weight matrix corresponding to the current operator (e.g., matrix layer) and write it to memory, such as a hard disk.

[0045] During model inference, the computation unit can read the quantization weight matrix to be processed by the current operator from memory into the processor's memory, such as the GPU's video memory. The computation unit can also read the activation matrix output by the previous operator (the operator of the nonlinear layer) or the quantized activation matrix obtained by quantization based on that activation matrix into memory. Furthermore, the computation unit can read a portion of the elements of the aforementioned matrices from memory into a cache (e.g., a layer-1 cache), and then read some or all of the elements from the cache into the processor's registers. In some examples, the layer-1 cache and registers can be located within the processor's computation unit.

[0046] During model inference, the computation unit can perform dequantization fusion calculations on the elements of the quantized weight matrix and the activation matrix (or quantized activation matrix) already read into the register to obtain the inference result of the current operator (e.g., a matrix layer operator), and write it into memory. This inference result can be used as the input features for the next operator (a nonlinear layer operator) to continue inference calculations. Dequantization fusion calculations can combine the lookup dequantization process and matrix multiplication calculations, and can also combine parallel processing methods such as the Stream-K algorithm to obtain the inference result of the current operator, improving the speed and efficiency of inference operations.

[0047] In some examples, the model may have multiple nonlinear layers and multiple matrix layers. The quantization weight matrices corresponding to the multiple matrix layers of the model can be written into memory separately. During inference, dequantization fusion calculations are performed based on the quantization weight matrices corresponding to the multiple matrix layers, and the inference results corresponding to these multiple matrix layers are written into memory separately for use by subsequent operators.

[0048] The embodiments disclosed herein perform inverse quantization fusion calculations based on the elements of the quantization weight matrix to be processed read into the register and the activation matrix of the model, which can reduce the amount of computation in the inference process and improve the speed and efficiency of inference.

[0049] Figure 2 This is a flowchart illustrating a model inference method according to another embodiment of the present disclosure. This embodiment may include one or more features from the above embodiments. In one implementation, the method further includes:

[0050] S210. The weight matrix of the model is calibrated by post-training quantization (PTQ) to obtain the initial quantized weight matrix and its corresponding scaling factor and lookup table.

[0051] S220. The initial quantization weight matrix is ​​interleaved to obtain the quantization weight matrix to be processed.

[0052] In this embodiment of the disclosure, after PTQ calibration of the weight matrix corresponding to a certain matrix layer of the trained model, an initial quantized weight matrix corresponding to that matrix layer can be obtained, along with a scaling factor and a lookup table corresponding to the initial quantized weight matrix. The scaling factor may include a quantization scaling factor and / or an inverse quantization scaling factor. The quantization scaling factor and the inverse quantization scaling factor may be the same or different, and there may be a conversion relationship between the quantization scaling factor and the inverse quantization scaling factor.

[0053] In this embodiment, S220 can be a step in the preprocessing stage, where the preprocessing results can be stored in memory for later use. For example, after S220, the computing unit can store the quantization weight matrix, scaling factor, and lookup table to be processed into memory. During the model inference stage, the preprocessing results, such as the quantization weight matrix to be processed, can be pre-loaded from memory into the processor's memory, and then read from memory into registers to participate in the model inference process. By pre-arranging the initial quantization weight matrix in an interleaved manner, the arranged quantization weight matrix to be processed can be written into memory for later use, thereby reducing the computational load during inference and improving the speed and efficiency of inference.

[0054] In one implementation, S220 involves interleaving the initial quantization weight matrix to obtain the quantization weight matrix to be processed, including:

[0055] After converting the initial quantization weight matrix blocks from the first arrangement to the second arrangement, a column of elements in the transformed matrix is ​​obtained, and the multiple columns of elements from the multiple blocks of the transformed matrix are sequentially mapped to the transformation result matrix.

[0056] The blocks of the transformation result matrix are reordered according to the data arrangement (instruction arrangement) specified by the instructions read from the cache into the register.

[0057] In this embodiment of the disclosure, during the preprocessing, the computing unit can first divide the initial quantization weight matrix into multiple blocks according to a certain dimension. For example, if the initial quantization weight matrix is ​​a K×N dimensional matrix, it can be divided into x blocks. The dimensions of different blocks can be the same or different. If the blocks of the initial quantization weight matrix are arranged in a row-major order (first arrangement), the row-major order can be converted to a column-major order (second arrangement). For example, if the first row of a block in memory contains 0123 and the second row contains 5678, after converting to a column-major order, the first and second row elements can be expanded in memory to the same row, resulting in 01235678. This expanded result can be considered as a column of elements in the transformed result matrix.

[0058] The transformation elements of all blocks of the initial quantization weight matrix are mapped sequentially to the transformation result matrix in a column-first, row-later arrangement (i.e., column-first arrangement) until all blocks are mapped. For example, if the initial quantization weight matrix is ​​divided into 12 blocks, each of which is an 8×4 dimension matrix, the transformation result matrix can be viewed as a 32×12 dimension matrix, which can occupy 12 rows of storage space.

[0059] Then, the blocks of the transformation result matrix (e.g., a column of elements as described above) can be reordered according to the data arrangement specified by the instructions read from the L1 cache into the registers, resulting in the quantization weight matrix to be processed, which is then saved to disk. During inference, it can be loaded from disk into the GPU's memory for later use as needed. When the quantization weight matrix to be processed is needed, the blocks can be read from memory into the L1 cache, and then from the L1 cache into the registers. Since the blocks of the quantization weight matrix to be processed in the L1 cache are reordered according to the instruction arrangement from the L1 cache into the registers, the blocks can be directly read from the L1 cache into the registers.

[0060] This embodiment of the disclosure preprocesses the quantization weight matrix by performing matrix partitioning and arrangement, and the preprocessing results can be stored in the memory in advance. During the inference stage, it is not necessary to recalculate the quantization weight matrix, which can improve the inference speed.

[0061] In one implementation, the method further includes:

[0062] S230. Perform a reciprocal operation on the scaling factor to obtain the result of the reciprocal operation. This step can also be a step in the preprocessing stage. Write the reciprocal operation results of the quantization scaling factor and / or dequantization scaling factor into memory for use in the inference results. By performing a reciprocal operation on the scaling factor, the preprocessed reciprocal operation results can be stored in memory in advance. During the inference stage, the reciprocal operation results in memory can be read for multiplication operations, reducing the computationally intensive division operations, thereby improving the speed and efficiency of inference operations.

[0063] Figure 3This is a flowchart illustrating a model inference method according to another embodiment of the present disclosure. This embodiment may include one or more features from the above embodiments. In one implementation, S120 performs dequantization fusion calculation based on the elements of the quantization weight matrix to be processed in the register and the activation matrix of the model to obtain the inference result of the current operator of the model, including: performing dequantization fusion calculation cyclically on the first block of the quantization activation matrix of the model and the second block of the quantization weight matrix to be processed, and after all block groups have been calculated, obtaining the inference result of the current operator of the model and exiting the dequantization fusion calculation.

[0064] In one implementation, such as Figure 3 As shown, the inverse quantization fusion calculation includes:

[0065] S310. After dequantizing the second block based on the lookup table, the lookup table dequantization result is obtained;

[0066] S320. Perform a multiplication calculation between the first block and the table lookup inverse quantization result to obtain the block multiplication result;

[0067] S330. Perform parallel accumulation of multiple block multiplication results to obtain a block accumulation result;

[0068] S340. Based on the block accumulation result and the scaling factor, the block inverse quantization result is calculated.

[0069] In this embodiment of the disclosure, S310 and S320 can be executed multiple times to obtain multiple block multiplication results. Then, the multiple block multiplication results are accumulated in parallel based on the Stream-K algorithm to obtain a block accumulation result.

[0070] First, the Stream-K algorithm can be planned for the processor's computing units. For example, based on the architecture of the GPU, matrix multiplication can be partitioned into K dimensions, allocating the computational range K to computing units in the GPU, such as Stream Multiprocessors (SMs). For instance, the two matrices involved in matrix multiplication might be an M×K dimensional activation matrix or quantization activation matrix (matrix A) and a K×N dimensional quantization weight matrix (matrix B). Based on matrices A and B, matrix multiplication is partitioned into K dimensions. A thread group within a computing unit can handle the matrix multiplication of a block of matrix A (matrix A', dimension m×k) with the dequantized lookup table result (matrix B', dimension k×n) of a block of matrix B (matrix B', dimension k×n). The blocks of the two matrices have a corresponding relationship in the K dimensions. The number of dimensions in different blocks of the same matrix in the K dimensions can be the same or different. In some examples, the Stream-K algorithm can be used to accumulate the results of multiple block multiplications in parallel, obtaining a block accumulation result in the K dimensions. Then, the block accumulation results in the K dimensions and the scaling factor are used to calculate the block dequantization results in the K dimensions (dimension m×n). Repeating calculation steps S310 to S340 for a set of K-dimensional blocks in the M dimensions and a set of K-dimensional blocks in the N dimensions yields the dequantization results in the M×N dimensions. By fusing dequantization and matrix multiplication, the kernel function of the dequantization matrix multiplication can be optimized. Furthermore, parallel accumulation using algorithms such as Stream-K can improve the utilization of processor resources, such as GPUs, and increase inference speed.

[0071] In one implementation, S310 dequantizes the second block based on the lookup table to obtain the lookup table dequantization result, including: reading the lookup table and the second block into corresponding registers respectively; performing bitwise operations based on the lookup table and the second block in different registers to obtain the lookup table dequantization result. In this embodiment of the disclosure, performing bitwise operations based on the lookup table and the second block in different registers can obtain the lookup table dequantization result, which can improve the speed of obtaining the lookup table dequantization result and increase the inference speed.

[0072] In one implementation, such as Figure 4 As shown, the lookup table and the second block are read into their respective registers, including:

[0073] S410: Read the first lookup table and the second lookup table into the first register and the second register, respectively;

[0074] S420: Read the second block into the third register;

[0075] S430: Use shift and masking to read a portion of the elements in the second block from the third register;

[0076] S440. Expand the elements in this part and write them into the corresponding register.

[0077] In this embodiment, the lookup table may include multiple lookup tables, such as a first lookup table and a second lookup table, which may reside in two different registers. The first and second lookup tables can be used to find weight values ​​corresponding to weight index values ​​within different ranges. For example, the first lookup table can find weight index values ​​with a weight index value less than or equal to 8. The second lookup table can find weight index values ​​with a weight index value greater than 8. In S420, some or all elements of the second block can be read into the register according to the register's computational capacity. If the number of elements in the second block exceeds the register's processing capacity at one time, the second block can be read into the register multiple times, thus allowing multiple executions of S420, S430, and S440. Two parts of elements can be read from the second block into the third register each time, and then these two parts of elements can be expanded, for example, from a 4-bit weight index value to an 8-bit weight index value. The format of the expanded elements can be determined based on the element format in the block of the activation matrix. For example, if the elements in the blocks of the activation matrix are in 8-bit integer format, and the first part of the elements read from the second block are in 4-bit integer format, the first part of the elements can be expanded to 8-bit integer format to facilitate block multiplication and accumulation operations in the inference process and improve the operation speed.

[0078] In one implementation, such as Figure 4 As shown, bitwise operations are performed based on the lookup table in different registers and the second block to obtain the lookup table inverse quantization result, including:

[0079] S450. Using bitwise operations, construct the corresponding binary mask based on the weight index value of the extended element.

[0080] S460. Based on the weight index value of the elements, look up the first lookup table to obtain the first lookup result, and perform a bitwise AND operation between the first lookup result and the binary mask to obtain the first inverse quantization result;

[0081] S470, Invert the binary mask;

[0082] S480. Based on the weight index value of this part of the elements, look up the second lookup table to obtain the second lookup result, and perform a bitwise AND operation between the second lookup result and the inverted result of the binary mask to obtain the second inverse quantization result;

[0083] S490. Perform a bitwise OR operation on the first dequantization result and the second dequantization result to obtain the lookup table dequantization result of this part of the elements.

[0084] In this embodiment of the disclosure, the elements in the second block may include the number of the weight index value. One number can correspond to one weight index value. For example, one number corresponds to a 4-bit weight index value, which can be expanded by padding the first 4 bits with zeros to obtain an 8-bit weight index value. If this 8-bit weight index value is less than or equal to 8, its binary mask value can be 0xFF (where 0x represents hexadecimal); if this 8-bit weight index value is greater than 8, its binary mask value can be 0x00. The value of the binary mask in the above example can also be changed, for example, interchanged.

[0085] In one implementation, in S430, multiple sets of partial elements can be read from the third register at once, for example, the first and second partial elements in the second block can be read from the third register. In S440, the first partial elements are expanded and stored in the fourth register, and the second partial elements are expanded and stored in the fifth register.

[0086] For the extended elements in the fourth register (see S450 to S490), the following steps can be performed: Using bitwise operations, construct corresponding binary masks based on the weight index values ​​of each element (corresponding to its number) in the fourth register. Look up the first lookup result of the first element in the first register based on the weight index value of the first element, and perform a bitwise AND operation between the first lookup result and the binary mask to obtain the first dequantized result of the first element. Invert the binary mask. Look up the second lookup result of the first element in the second register based on the weight index value of the first element, and perform a bitwise AND operation between the second lookup result and the inverted result of the binary mask to obtain the second dequantized result of the first element. Perform a bitwise OR operation between the first dequantized result and the second dequantized result to obtain the lookup dequantized result of the first element.

[0087] For the extended elements in the fifth register (see S450 to S490), the following steps can be performed: Using bitwise operations, construct corresponding binary masks based on the weight index values ​​of each element (corresponding to its number) in the second part of the fifth register. Based on the weight index values ​​of the second part elements, look up the first lookup table in the first register to obtain the first lookup result of the second part elements. Perform a bitwise AND operation between the first lookup result and the binary mask to obtain the first dequantized result of the second part elements. Invert the binary mask. Based on the weight index values ​​of the second part elements, look up the second lookup table in the second register to obtain the second lookup result of the second part elements. Perform a bitwise AND operation between the second lookup result and the inverted result of the binary mask to obtain the second dequantized result of the second part elements. Perform a bitwise OR operation between the first dequantized result and the second dequantized result to obtain the lookup table dequantized result of the second part elements.

[0088] In this embodiment of the disclosure, a binary mask can be used to find the weight value corresponding to different weight index values ​​in multiple lookup tables, which can improve the search speed.

[0089] In one implementation, such as Figure 5 As shown, S330 performs parallel accumulation of multiple block multiplication results to obtain a block accumulation result, including:

[0090] S510. Perform block accumulation on the block multiplication result within the current thread group to obtain a partial accumulation result;

[0091] S520. Write the accumulated result to the temporary working area of ​​this memory.

[0092] S530. After the block accumulation in the last thread group determined based on the partitioning dimension is completed, the last thread group performs a block accumulation operation on the partial accumulation results of each thread group in the temporary work area and the partial accumulation results in the current thread group to obtain the block accumulation result.

[0093] In this embodiment of the disclosure, referring to the Stream-K algorithm planning described above, based on the architectural characteristics of the GPU, matrix multiplication can be divided into K dimensions, allocating the computational range of K to the computing units in the GPU. For example, as Figure 6 As shown, partitioning matrix A by K dimensions yields matrix A', where the first A' matrix has a dimension of m. 0 0×k 0 0, the dimension of the second A' matrix is ​​m. 0 1×k 0 1. And so on. Similarly, partitioning matrix B along K dimensions yields matrix B', where the first B' matrix has dimension k. 0 0×n 00, the dimension of the second B' matrix is ​​k. 0 1×n 0 1. And so on. A thread group in the computing unit can be responsible for computing the matrix multiplication of a lookup table inverse quantization result (B~ matrix) of a matrix A' and a matrix B'.

[0094] See Figure 7 , an m 0 0×k 0 A zero-dimensional matrix partition (matrix A') can be further divided into multiple k-partitions, for example, m 0 0×k 00 m 0 0×k 01 ...and so on. A k 0 0×n 0 A zero-dimensional matrix (B' matrix) can be further divided into multiple k-partitions, for example, k 00 ×n 0 0, k 01 ×n 0 0...and so on. The result of block multiplication of two k-part blocks can include k... 00 The corresponding m 0 0×n 0 A zero-dimensional matrix, k 01 The corresponding m 0 0×n 0 A zero-dimensional matrix... and so on. A thread group can be responsible for summing the calculation results of k blocks to obtain m. 0 0×n 0 The zero-dimensional matrix is ​​a partial summation result of matrix multiplication of matrices A' and B'. Assume multiple thread groups can handle the matrix multiplication of a row of A' matrix with a row of B' matrix after lookup table inversion. The last thread group can also sum the partial summations of all the preceding thread groups to obtain the block-accumulated result m. 0 ×n 0 .

[0095] In one implementation, such as Figure 8 As shown, S340 calculates the block inverse quantization result based on the block accumulation result and the scaling factor, including:

[0096] S810. Convert the block accumulation result from the first format to the second format;

[0097] S820. Multiply the block accumulation result of the second format with the reciprocal of the scaling factor to obtain the block dequantization result;

[0098] S830: Write the block dequantization result into this memory.

[0099] For example, if the block accumulation result is in 32-bit integer format, it can be converted to 32-bit floating-point format. Then, the 32-bit floating-point block accumulation result is multiplied by the reciprocal of the dequantization scaling factor to obtain the block dequantization result. The reciprocal of the scaling factor can be pre-written into memory, loaded from the memory register into memory during inference, read from memory into the cache, and then into the register. After calculating the block dequantization result using the block accumulation result and its reciprocal, the block dequantization result can be written into memory. This block dequantization process can be executed cyclically, writing the block dequantization result corresponding to each block into memory until all blocks have been calculated. See also... Figure 6 A block can be understood as a row of matrix A' (A' matrix) and a column of matrix B' (B' matrix). Performing dequantization according to the block size and scaling factor can improve dequantization accuracy and inference speed.

[0100] In some examples, combining non-uniform 4-bit quantization with 16-bit floating-point activation allows 4-bit quantization to be multiplied by 16-bit floating-point activation (e.g., 16-bit float16 or bfloat16 activation) through uniform or non-uniform quantization and dequantization.

[0101] In some examples, a high-performance GPU inference implementation scheme with 4-bit non-uniform quantization weights and 8-bit quantization activation can be adopted. By using 4-bit non-uniform quantization weights, a high degree of compression of model weights can be achieved, thereby significantly reducing the model's memory access requirements and inference cost. This non-uniform quantization method can more accurately represent the actual distribution of weights, reduce quantization errors, and thus improve inference accuracy. High-performance computational optimization and tuning are performed for inference scenarios with 4-bit non-uniform quantization weights and 8-bit quantization activation. Through methods including weight interleaving preprocessing and dequantization process optimization, a high-performance inference scheme with 4-bit non-uniform quantization weights and 8-bit quantization activation is achieved.

[0102] In some examples, 4-bit quantization inference schemes typically employ a separate inference approach for the dequantization step and the matrix multiplication step. This approach is easy to implement but has lower performance. It can save space in the model but cannot improve the model's inference performance.

[0103] In some examples, the dequantization stage in the inference computation of a 4-bit non-uniform quantization model can be integrated into the matrix multiplication computation. Furthermore, the kernel function for integrating the dequantized matrix multiplication is optimized through weight pre-transformation, Stream-K dynamic k-splitting technology, and Parallel Thread Execution (PTX) assembly technology. Through these techniques, embodiments of this disclosure further optimize the dequantization process of non-uniform quantization weights, significantly improving the computation speed and efficiency of inference.

[0104] The following section uses a high-performance quantization inference system with non-uniform 4-bit weights and 8-bit activation as an example to illustrate this in detail. Other quantization inference formats are similar and not limited to any specific format. This system can achieve efficient operation of neural network models through innovative quantization strategies and optimized inference mechanisms.

[0105] I. Non-uniform 4-bit weighted quantization

[0106] This disclosure embodiment can employ a non-uniform 4-bit weight quantization strategy. This strategy dynamically adjusts the quantization interval according to the actual distribution of the weights. This strategy can more effectively preserve important information in the weights, thereby improving the accuracy of the quantized model.

[0107] During inference, non-uniform 4-bit weight quantization can be dequantized internally by looking up a table within the GPU. To maximize GPU inference performance, the quantized 4-bit weights can be interleaved beforehand. The interleaving result can be written to a hard drive or other storage device beforehand, reducing the dequantization process during inference.

[0108] The model reasoning scheme of this disclosure mainly includes the following:

[0109] 1. Obtain the high-precision floating-point format (e.g., F16 / BF16) weights (which can be in matrix form) of the matrix layer of the model, the non-uniform 4-bit quantized weights after Post Training Quantization (PTQ) calibration, and their corresponding quantization scaling factor, dequantization scale, and look-up table (LUT).

[0110] II. Model preprocessing and transformation, see [link / reference] Figure 9 :

[0111] 1. The non-uniform 4-bit quantization weight matrix is ​​interleaved. Based on the hardware characteristics of the target GPU, the row-major 4-bit quantization weight matrix is ​​transposed and rearranged to make it more memory-friendly during GPU computation and easier to perform 4-bit to 8-bit inverse quantization calculations. See also Figure 10 The weight change is divided into two steps: the first step is the matrix arrangement (Layer_out) transformation; the second step is the 4-bit element arrangement transformation in the 32-bit register.

[0112] Step 1: Layer_out transformation:

[0113] (1) Let ctaK be the number of blocks K calculated in each subsequent matrix multiplication calculation. Expand a block of 4 columns ctaK*2 rows in the original row-major weight matrix into a consecutive column in the resulting column-major matrix. Map the original weight matrix to the transformed result matrix (i.e., the quantization weight matrix to be processed) in a column-first, row-later manner.

[0114] (2) Reorder the blocks of the transformation result matrix according to the data arrangement specified by the instruction in the GPU architecture that reads a continuous small block matrix (e.g., a continuous column of elements of the block mapping of the transformation result matrix) from the L1 cache into the register.

[0115] Step 2: Rearrangement of 4-bit elements within the 32-bit register:

[0116] The eight 4-bit values ​​in a 32-bit register, which are in the order of 01234567, are reordered to 04152637, which facilitates fast dequantization calculations in matrix multiplication.

[0117] 2. Preprocess the quantization scale and dequantization scale by taking the reciprocal of the scale to avoid division calculations during inference. For example, the original quantization scale = S1, the original dequantization scale = S2, the preprocessed quantization scale = 127 / S1, and the preprocessed dequantization scale = (S1*S2) / (127*127).

[0118] The preprocessing process can be handled in the GPU or by other processors such as the CPU.

[0119] III. Perform neural network inference calculations with 4-bit weights and 8-bit activation:

[0120] See Figure 11In the reasoning process, this disclosure can rewrite the matrix multiplication, and perform the dequantization process of the non-uniform 4-bit quantization weight matrix (which can be simply referred to as the 4-bit weight matrix) and the integer matrix multiplication of the 8-bit quantization activation matrix (output by the non-quantization operator (OP)) by merging the calculation, while applying the Stream-K algorithm.

[0121] An example of a formula for inverse quantization fusion calculation is as follows:

[0122] C(float16 / bfloat16)=float32(A(8bit)*lookuptable(B(4bit)))*scale(float16 / bfloat16),

[0123] Where A represents an 8-bit activation matrix, B represents a 4-bit weight matrix, lookuptable() represents the lookup table result, scale represents the scale vector of float16 or bfloat16 (the inverse of the dequantization scaling factor), and C represents the float16 / bfloat16 result matrix.

[0124] See Figure 12 An example of a computational flow that integrates matrix multiplication includes:

[0125] S1201. If necessary, the activation values ​​of 16-bit floating-point numbers (float16 / bfloat16) can be quantized into int8 type activation values, denoted as matrix A. The 4-bit weights pre-loaded into GPU storage are denoted as matrix B.

[0126] S1202. Perform Stream-K algorithm planning: Based on the architecture characteristics of the GPU, perform K-dimensional partitioning of matrix multiplication and assign the K range of computation to each computing unit in the GPU, such as a Stream Multiprocessor (SM).

[0127] S1203. Read the lookup table (LUT) corresponding to the non-uniform 4-bit quantization into the GPU register.

[0128] S1204. Matrix Block Computation: Divide matrices A and B into blocks to obtain matrices A' and B'. Read matrices A' and B' into the GPU's cache (e.g., L1 cache). Multiple blocks of matrices A' and B' can be read depending on the GPU's computation / memory access pipeline order and other performance parameters.

[0129] S1205. Read a block of matrix A' and a block of matrix B' from the L1 cache into the register.

[0130] S1206. Bitwise operations are performed between the B' matrix in the register and the lookup table (LUT) in the register to dequantize the B' matrix into the B~ matrix. At this time, the B~ matrix is ​​an 8-bit integer (int8) type. The algorithm for non-uniform 4-bit weighted dequantization lookup table is described later.

[0131] S1207. Perform matrix multiplication on matrix A' and matrix B to obtain the block matrix result C~, where matrix C~ is of type int32.

[0132] S1208. Read the next A' and B' matrices in the pipeline into the L1 cache. This step can also be done after step S1210.

[0133] S1209. Perform different Stream-K operations based on the range of the k-block.

[0134] (1) Accumulate the C matrix in k blocks within the thread group;

[0135] (2) End the accumulation operation of k blocks in this thread group and write the C matrix to the temporary work area;

[0136] (3) After completing the accumulation of k blocks within the last thread group, the thread group performs a reduce operation on the calculation results of all k blocks. For example, the thread group reads the partial accumulation results (C-matrix) calculated by other thread groups from the temporary workspace and accumulates them with the C-matrix within its own thread group. For the blocks of the two matrices, the thread groups are divided according to the K-dimensional plane.

[0137] For example, a thread group needs to calculate a total of 64 k blocks, and each thread group performs the calculation of the C matrix for 16 k blocks at a time; the results of the four calculations are summed.

[0138] S1210. Perform matrix multiplication post-processing, convert the C~ matrix to a 32-bit floating-point (float32) type, and calculate the block matrix multiplication result C' matrix by calculating the reciprocal of the dequantization scaling factor (scale).

[0139] S1211. Write matrix C' into the GPU's memory for storage. Return to step 1203 to perform loop calculations until all block matrices have been calculated.

[0140] IV. Non-uniform 4-bit weighted dequantization lookup table algorithm (step S1206 in point III above), see Figure 13 :

[0141] S1301. Extract the eight 4-bit weight indices (part of the B' matrix) numbered 0123 and 4567 from the 4-bit quantization weights arranged in the 32-bit register according to the format 04152637 using the shift and mask method.

[0142] S1302. After expanding the eight 4-bit weight indices into eight 8-bit weight indices, place them in two 32-bit registers.

[0143] For example, numbers 0123 and 4567 can be understood as weight index numbers. Number 0 corresponds to a 4-bit weight index, such as 0x7 (where 0x represents hexadecimal), which expands to 0x07. Number 1 corresponds to another 4-bit weight index, such as 0xA, which expands to 0x0A. Number 2 corresponds to another 4-bit weight index, such as 0xB, which expands to 0x0B. Number 3 corresponds to another 4-bit weight index, such as 0xD, which expands to 0x0D.

[0144] S1303. Using bitwise operations, construct a binary mask based on the size of the 4-bit weight index. For example, the mask corresponding to 0x07 is 0xFF, the mask corresponding to 0x0A is 0x00, the mask corresponding to 0x0B is 0x00, and the mask corresponding to 0x0D is 0x00. The mask for number 0123 is 0xFF000000.

[0145] S1304. Find the first LUT based on the 4-bit weight index value, and perform a bitwise AND operation between the lookup result and the binary search mask. For example, the lookup result for 0x07 is 0xEC, for 0x0A it is 0x5B, for 0x0B it is 0xFC, and for 0x0D it is 0x04. The results of performing a bitwise AND operation with their respective masks are 0xEC, 0x00, 0x00, and 0x00, respectively.

[0146] S1305. Invert the binary mask. For example, in the example above, the inverted result of the mask corresponding to each index is 0x00FFFFFF.

[0147] S1306. Based on the 4-bit weight index value, look up the second LUT. Perform a bitwise AND operation between the lookup result and the inverted binary mask. For example, the lookup result for 0x07 is 0x8F, for 0x0A it is 0x37, for 0x0B it is 0x75, and for 0x0D it is 0xA7. The results of performing a bitwise AND operation with their respective masks are 0x00, 0x37, 0x75, and 0x0D, respectively.

[0148] S1307. Perform a bitwise OR operation on the bitwise AND result of S1305 and the bitwise AND result of S1303. For example, in the above example, the result of the bitwise OR is 0xEC37750D, which is the lookup table dequantization result corresponding to number 1234. After the above steps, a portion of the B matrix can be obtained.

[0149] For the other elements of matrix B', repeat the table lookup steps S1301 to S1306 above to obtain the complete matrix B'.

[0150] S1308. Complete table lookup.

[0151] This disclosure, through the adoption of a quantization strategy using non-uniform 4-bit weights and 8-bit activation, enables high-precision, high-performance fused quantization inference on processors such as GPUs. This approach, while maintaining inference accuracy, significantly reduces storage requirements, improves the utilization of processor resources such as GPUs, increases inference speed, enhances hardware compatibility, and provides strong support for the widespread deployment of large-scale neural network inference in practical applications.

[0152] This disclosure presents a high-performance GPU inference scheme that is both efficient and accurate by combining 4-bit non-uniform quantization weights, 8-bit activation, and PTX assembly techniques. This scheme not only significantly reduces the model's memory access requirements and inference costs but also further improves inference efficiency by optimizing the dequantization process, providing strong technical support for the widespread deployment of neural network models in practical applications. The model inference method of this disclosure is applicable to fields such as natural language processing, machine translation, and intelligent question-answering systems, and has broad market application prospects and practical value.

[0153] The model inference method provided in this disclosure can be used in a high-performance inference system with non-uniform 4-bit weights and 8-bit activations. This system can reduce the storage and computational costs of deep learning models through quantization techniques while maintaining high inference performance. Specifically, the system employs a non-uniform quantization method to quantize the weights to 4 bits and the activation values ​​(activation matrix) to quantize to 8 bits, thereby achieving efficient model compression and acceleration.

[0154] The solution of this disclosure embodiment has the following characteristics:

[0155] Non-uniform 4-bit weighted quantization: Compared with traditional uniform quantization methods, non-uniform quantization can more accurately represent the distribution of weights and reduce quantization errors. By employing specific quantization algorithms, weights are compressed from high bit width (such as 32-bit floating-point numbers) to 4 bits, thereby significantly reducing storage and transmission costs.

[0156] 8-bit activation quantization: Activation values ​​are also quantized using 8 bits, which further reduces storage and computational requirements while maintaining high precision. Combined with weight quantization, this enables efficient compression of deep learning models.

[0157] High-performance inference: By optimizing the quantization algorithm and inference engine, the system can maintain high inference performance while reducing accuracy. This enables fast and accurate inference even on resource-constrained devices.

[0158] Examples of application scenarios for this disclosure are as follows:

[0159] This technology is used for natural language processing, text-to-image generation, and inference acceleration of ultra-large-scale neural networks with multiple modalities. Ultra-large-scale neural networks have hundreds of billions of weights; using non-uniform 4-bit weight quantization can significantly reduce the storage footprint of weights and the memory access overhead for inference, thereby improving inference performance.

[0160] This technology accelerates inference for neural networks on low-end devices where weight storage is limited. Using non-uniform 4-bit weight quantization can significantly reduce the storage footprint of weights, enabling the execution of neural networks that were previously unable to function.

[0161] Figure 14 This is a schematic block diagram of a model inference apparatus according to an embodiment of the present disclosure, comprising:

[0162] The quantization weight reading module 1401 is used to read the quantization weight matrix of the model to be processed from the processor's cache into the processor's register;

[0163] The inverse quantization fusion calculation module 1402 is used to perform inverse quantization fusion calculation based on the elements of the quantization weight matrix to be processed in the register and the activation matrix of the model, so as to obtain the inference result of the current operator of the model.

[0164] The storage module 1403 is used to write the inference result of the current operator into the memory of the processor for use by subsequent operators.

[0165] Figure 15 This is a schematic block diagram of a model inference apparatus according to another embodiment of the present disclosure. The apparatus may include one or more features of the described apparatus. In one embodiment, the model inference apparatus further includes:

[0166] The post-training quantization module 1501 is used to calibrate the model's weight matrix through post-training quantization to obtain the initial quantized weight matrix and its corresponding scaling factor and lookup table.

[0167] The staggered arrangement module 1502 is used to obtain the quantization weight matrix to be processed by staggering the initial quantization weight matrix.

[0168] In one embodiment, the staggered arrangement module 1502 includes:

[0169] The arrangement submodule 1503 is used to convert the blocks of the initial quantization weight matrix from the first arrangement method to the second arrangement method, obtain a column of elements in the transformed matrix, and sequentially map the multiple columns of elements of the multiple blocks of the transformed matrix to the transformation result matrix.

[0170] The reordering submodule 1504 is used to reorder the blocks of the transformation result matrix according to the data arrangement specified by the instruction read from the cache into the register.

[0171] In one embodiment, the model inference device further includes:

[0172] The reciprocal operation module 1505 is used to perform a reciprocal operation on the scaling factor to obtain the reciprocal result of the scaling factor.

[0173] In one implementation, the inverse quantization fusion calculation module 1402 is used to perform inverse quantization fusion calculation cyclically on the first block of the quantization activation matrix of the model and the second block of the quantization weight matrix to be processed. After all the block groups have been calculated, the inference result of the current operator of the model is obtained, and the inverse quantization fusion calculation is exited.

[0174] The inverse quantization fusion calculation module 1402 includes: a lookup submodule 1506, used to inverse quantize the second block based on the lookup table to obtain the lookup inverse quantization result; a multiplication submodule 1507, used to multiply the first block with the lookup inverse quantization result to obtain the block multiplication result; an accumulation submodule 1508, used to accumulate multiple block multiplication results in parallel to obtain the block accumulation result; and a block inverse quantization submodule 1509, used to calculate the block inverse quantization result based on the block accumulation result and the scaling factor.

[0175] In one implementation, the lookup table submodule 1506 is used to read the lookup table and the second block into their respective registers; and to perform bitwise operations based on the lookup table and the second block in different registers to obtain the lookup table inverse quantization result.

[0176] In one implementation, the lookup table submodule 1506 is used to read the lookup table and the second block into their respective registers, including:

[0177] Read the first lookup table and the second lookup table into the first register and the second register, respectively;

[0178] Read the second block into the third register;

[0179] The first and second elements of the second block are read from the third register using a shift and masking method.

[0180] The first and second elements are expanded and written into the fourth and fifth registers, respectively.

[0181] In one implementation, the lookup table submodule 1506 is used to perform bitwise operations based on the lookup table in different registers and the second block to obtain the lookup table inverse quantization result, including:

[0182] Using bitwise operations, construct the corresponding binary mask based on the weight index values ​​of each extended element in the fourth and fifth registers;

[0183] The first lookup table is searched based on the weight index value of the first part of the elements to obtain the first lookup result, and the first lookup result is bitwise ANDed with the binary mask to obtain the first dequantization result;

[0184] Invert the binary mask;

[0185] The second lookup table is searched based on the weight index value of the second part of the elements to obtain the second lookup result. The second lookup result is then bitwise ANDed with the inverted result of the binary mask to obtain the second inverse quantization result.

[0186] Perform a bitwise OR operation on the first dequantization result and the second dequantization result to obtain the lookup table dequantization result.

[0187] In one implementation, the accumulation submodule 1508 is used to perform block accumulation within the current thread group on the block multiplication result to obtain a partial accumulation result; write the partial accumulation result into the temporary working area of ​​the memory; after the block accumulation within the last thread group determined based on the partitioning dimension is completed, the last thread group performs an accumulation operation on the partial accumulation results of each thread group in the temporary working area with the partial accumulation results within the current thread group to obtain the block accumulation result.

[0188] In one implementation, the block dequantization submodule is used to convert the block accumulation result from a first format to a second format; multiply the block accumulation result in the second format with the reciprocal of the scaling factor to obtain the block dequantization result; and write the block dequantization result into the memory.

[0189] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0190] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0191] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0192] Figure 16 A schematic block diagram of an example electronic device 1600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0193] like Figure 16 As shown, device 1600 includes a computing unit 1601, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1602 or a computer program loaded into random access memory (RAM) 1603 from storage unit 1608. The RAM 1603 may also store various programs and data required for the operation of device 1600. The computing unit 1601, ROM 1602, and RAM 1603 are interconnected via bus 1604. Input / output (I / O) interface 1605 is also connected to bus 1604.

[0194] Multiple components in device 1600 are connected to I / O interface 1605, including: input unit 1606, such as keyboard, mouse, etc.; output unit 1607, such as various types of monitors, speakers, etc.; storage unit 1608, such as disk, optical disk, etc.; and communication unit 1609, such as network card, modem, wireless transceiver, etc. Communication unit 1609 allows device 1600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0195] The computing unit 1601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1601 performs the various methods and processes described above, such as the methods in the above embodiments. For example, in some embodiments, the methods may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1600 via ROM 1602 and / or communication unit 1609. When the computer program is loaded into RAM 1603 and executed by the computing unit 1601, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 1601 may be configured to perform the methods in the above embodiments by any other suitable means (e.g., by means of firmware).

[0196] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0197] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0198] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0199] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0200] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0201] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0202] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0203] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A model reasoning method, comprising: The quantization weight matrix of the model to be processed is read from the processor's cache into the processor's register; Based on the elements of the quantization weight matrix to be processed in the register and the activation matrix of the model, inverse quantization fusion calculation is performed to obtain the inference result of the current operator of the model; The inference result of the current operator is written into the processor's memory for use by subsequent operators.

2. The method according to claim 1, wherein, The method further includes: The model's weight matrix is ​​quantized and calibrated after training to obtain the initial quantized weight matrix and its corresponding scaling factor and lookup table. The initial quantization weight matrix is ​​arranged in an interleaved manner to obtain the quantization weight matrix to be processed.

3. The method according to claim 2, wherein, The initial quantization weight matrix is ​​interleaved to obtain the quantization weight matrix to be processed, including: After converting the initial quantization weight matrix blocks from the first arrangement to the second arrangement, a column of elements in the transformed matrix is ​​obtained, and the multiple columns of elements from the multiple blocks of the transformed matrix are sequentially mapped to the transformation result matrix. The blocks of the transformation result matrix are reordered according to the data arrangement specified by the instructions read from the cache into the register.

4. The method according to any one of claims 1 to 3, wherein, The method further includes: The reciprocal of the scaling factor is taken to obtain the result of the reciprocal calculation of the scaling factor.

5. The method according to any one of claims 1 to 4, wherein, Based on the elements of the quantization weight matrix to be processed in the register and the activation matrix of the model, inverse quantization fusion calculation is performed to obtain the inference result of the current operator of the model, including: For the first block of the quantization activation matrix of the model and the second block of the quantization weight matrix to be processed, the inverse quantization fusion calculation is performed cyclically. After all block groups have been calculated, the inference result of the current operator of the model is obtained, and the inverse quantization fusion calculation is exited. The inverse quantization fusion calculation includes: inverse quantizing the second block based on a lookup table to obtain a lookup table inverse quantization result; multiplying the first block with the lookup table inverse quantization result to obtain a block multiplication result; accumulating multiple block multiplication results in parallel to obtain a block accumulation result; and calculating a block inverse quantization result based on the block accumulation result and a scaling factor.

6. The method according to claim 5, wherein, After dequantizing the second block based on the lookup table, the lookup table dequantization result is obtained, including: The lookup table and the second block are read into their respective registers; Bitwise operations are performed based on the lookup tables in different registers and the second block to obtain the lookup table inverse quantization result.

7. The method according to claim 6, wherein, The lookup table and the second block are read into their respective registers, including: Read the first lookup table and the second lookup table into the first register and the second register, respectively; The second block is read into the third register; The first and second elements of the second block are read from the third register using a shift and masking method; The first and second parts of the elements are expanded and written into the fourth and fifth registers, respectively.

8. The method according to claim 7, wherein, Bitwise operations are performed based on the lookup tables in different registers and the second block to obtain the lookup table inverse quantization result, including: Using bitwise operations, corresponding binary masks are constructed based on the weight index values ​​of each extended element in the fourth and fifth registers; The first lookup result is obtained by searching the first lookup table based on the weight index value of the first part of the elements, and the first lookup result is then bitwise ANDed with the binary mask to obtain the first inverse quantization result. Invert the binary mask; The second lookup result is obtained by searching the second lookup table based on the weight index value of the second part of the elements, and the second lookup result is then bitwise ANDed with the inverted result of the binary mask to obtain the second inverse quantization result. Perform a bitwise OR operation on the first dequantization result and the second dequantization result to obtain the lookup table dequantization result.

9. The method according to any one of claims 5 to 8, wherein, The results of multiple block multiplications are accumulated in parallel to obtain a block accumulation result, including: The block multiplication results are then accumulated within the current thread group to obtain a partial accumulation result. Write the partial accumulation result into the temporary working area of ​​the memory; After the block accumulation within the last thread group determined based on the segmentation dimension is completed, the last thread group performs a summation operation on the partial accumulation results of each thread group in the temporary work area and the partial accumulation results within its own thread group to obtain the block accumulation result.

10. The method according to any one of claims 5 to 9, wherein, Based on the block accumulation result and the scaling factor, the block inverse quantization result is calculated, including: Convert the block accumulation result from the first format to the second format; The block accumulation result of the second format is multiplied with the reciprocal of the scaling factor to obtain the block dequantization result; The block dequantization result is written into the memory.

11. A model reasoning device, comprising: The quantization weight reading module is used to read the quantization weight matrix of the model to be processed from the processor's cache into the processor's register; The dequantization fusion calculation module is used to perform dequantization fusion calculation based on the elements of the quantization weight matrix to be processed in the register and the activation matrix of the model to obtain the inference result of the current operator of the model. The storage module is used to write the inference result of the current operator into the memory of the processor for use by subsequent operators.

12. The apparatus according to claim 11, wherein, The model inference device further includes: The post-training quantization module is used to calibrate the model's weight matrix through post-training quantization, obtaining the initial quantized weight matrix and its corresponding scaling factor and lookup table. The staggered arrangement module is used to stagger the initial quantization weight matrix to obtain the quantization weight matrix to be processed.

13. The apparatus according to claim 12, wherein, The staggered arrangement module includes: The arrangement submodule is used to convert the blocks of the initial quantization weight matrix from the first arrangement method to the second arrangement method, obtain a column of elements in the transformed matrix, and map the multiple columns of elements of the multiple blocks of the transformed matrix to the transformation result matrix in sequence. The reordering submodule is used to reorder the blocks of the transformation result matrix according to the data arrangement specified by the instructions read from the cache into the register.

14. The apparatus according to any one of claims 11 to 13, wherein, The model inference device further includes: The reciprocal operation module is used to perform a reciprocal operation on the scaling factor to obtain the reciprocal operation result of the scaling factor.

15. The apparatus according to any one of claims 11 to 14, wherein, The inverse quantization fusion calculation module is used to perform inverse quantization fusion calculation cyclically on the first block of the quantization activation matrix of the model and the second block of the quantization weight matrix to be processed. After all block groups have been calculated, the inference result of the current operator of the model is obtained, and the inverse quantization fusion calculation is exited. The inverse quantization fusion calculation module includes: a lookup table submodule, used to inverse quantize the second block based on a lookup table to obtain a lookup table inverse quantization result; a multiplication submodule, used to multiply the first block with the lookup table inverse quantization result to obtain a block multiplication result; an accumulation submodule, used to accumulate multiple block multiplication results in parallel to obtain a block accumulation result; and a block inverse quantization submodule, used to calculate a block inverse quantization result based on the block accumulation result and a scaling factor.

16. The apparatus according to claim 15, wherein, The lookup table submodule is used to read the lookup table and the second block into the corresponding registers respectively; Bitwise operations are performed based on the lookup tables in different registers and the second block to obtain the lookup table inverse quantization result.

17. The apparatus according to claim 16, wherein, The lookup table submodule is used to read the lookup table and the second block into their respective registers, including: Read the first lookup table and the second lookup table into the first register and the second register, respectively; The second block is read into the third register; The first and second elements of the second block are read from the third register using a shift and masking method; The first and second parts of the elements are expanded and written into the fourth and fifth registers, respectively.

18. The apparatus according to claim 17, wherein, The lookup table submodule is used to perform bitwise operations based on the lookup tables in different registers and the second block to obtain the lookup table inverse quantization result, including: Using bitwise operations, based on the weights of the extended elements in the fourth and fifth registers... The values ​​are used to construct the corresponding binary masks; The first lookup result is obtained by searching the first lookup table based on the weight index value of the first part of the elements, and the first lookup result is then bitwise ANDed with the binary mask to obtain the first inverse quantization result. Invert the binary mask; The second lookup result is obtained by searching the second lookup table based on the weight index value of the second part of the elements, and the second lookup result is then bitwise ANDed with the inverted result of the binary mask to obtain the second inverse quantization result. Perform a bitwise OR operation on the first dequantization result and the second dequantization result to obtain the lookup table dequantization result.

19. The apparatus according to any one of claims 15 to 18, wherein, The accumulation submodule is used to perform block accumulation of the block multiplication result within the current thread group to obtain a partial accumulation result; and write the partial accumulation result into the temporary working area of ​​the memory. After the block accumulation within the last thread group determined based on the segmentation dimension is completed, the last thread group performs a summation operation on the partial accumulation results of each thread group in the temporary work area and the partial accumulation results within its own thread group to obtain the block accumulation result.

20. The apparatus according to any one of claims 15 to 19, wherein, The block dequantization submodule is used to convert the block accumulation result from the first format to the second format; and to multiply the block accumulation result in the second format with the reciprocal of the scaling factor to obtain the block dequantization result. The block dequantization result is written into the memory.

21. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.

23. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-10.

Images