A method for performing matrix multiplication operations using a processor that includes multiple computing units.

Abstract

This disclosure relates to the field of computer technology and provides a method for a processor comprising multiple computing units to perform matrix multiplication. In this method: a quantized parameter matrix is ​​divided into multiple parameter blocks, which are then sequentially and non-repeatingly assigned to multiple computing units, such that each computing unit is assigned a first number of consecutive parameter blocks, except for at most one computing unit which is assigned fewer than a first number of consecutive parameter blocks from the multiple parameter blocks; each computing unit dequantizes its assigned parameter blocks; each computing unit obtains the portion of the input matrix that should be multiplied by the assigned parameter blocks, and performs matrix multiplication on the dequantized assigned parameter blocks and the portion of the input matrix to complete the matrix multiplication operation between the input matrix and the model parameters. Therefore, globally, only one dequantization is required on the quantized parameter matrix, saving computational resources and improving inference efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to a method for performing matrix multiplication operations on a processor comprising multiple computing units. Background Technology

[0002] With the development of artificial intelligence, large models (hereinafter sometimes simply referred to as "models"), such as large language models (LLMs) based on the Transformer architecture, have been widely used in many fields. For example, large models can be used to process query tasks, generate natural language text, and generate images or videos. Generally, the inference process of large models may involve a large number of general matrix multiplication (GEMM) operations and may involve billions or more parameters (such as model weights), which will consume a lot of storage space and bring huge challenges to the actual deployment and application of the models.

[0003] Currently, to reduce the storage space occupied by model parameters, model parameter quantization algorithms can be used to reduce the bit width of model parameters. For example, the Adaptive Weighted Quantization (AWQ) algorithm can quantize the 16-bit parameters of a pre-trained model to 4 bits. For LLM, this reduces the number of parameters by about three-quarters, thereby greatly reducing the storage and memory access overhead caused by model parameters.

[0004] While quantizing and compressing model parameters using model parameter quantization algorithms can save storage space, during model inference, the quantized model parameters need to be dequantized (e.g., restored from 4 bits to 16 bits) before matrix multiplication of input data and model parameters. This results in lengthy matrix multiplication operations and reduces the model's inference efficiency. Therefore, improving the computational efficiency of matrix multiplication operations for quantized model parameters to enhance inference efficiency is a pressing issue. Summary of the Invention

[0005] The inventors of this disclosure noted that during the inference process of large models, due to the large size of both the input data (e.g., a 32×4096 matrix) and model parameters (e.g., a 4096×4096 matrix) requiring matrix multiplication, the input data and model parameters are typically divided into multiple blocks and allocated to multiple computing units (e.g., streaming multiprocessors (SMs) in a graphics processing unit (GPU). Each computing unit performs matrix multiplication on its assigned input data block and the corresponding model parameter block to obtain an output data block as the result of the operation. Furthermore, the output data blocks from all the computing units can form a complete output data block. In related technologies, the computational tasks of computing units are typically mapped to the output data; that is, for an output data block, the input data block capable of producing that output data block and the corresponding model parameter block are allocated to a computing unit. Since operations on different output data blocks often require the use of the same model parameter block, when the model parameters are quantized, computing units allocated the same model parameter block need to dequantize that same model parameter block before performing matrix multiplication with the corresponding input data block. This results in the need to repeatedly dequantize the model parameters globally when performing matrix multiplication operations between input data and model parameters, wasting computing resources, reducing processor utilization and computing speed, and slowing down the model's inference speed.

[0006] In view of this, this disclosure proposes a method for performing matrix multiplication operations on a processor comprising multiple computing units, which can avoid dequantizing the quantized model parameters multiple times during inference when the model parameters of a large model have been quantized, thereby improving the computational efficiency of matrix multiplication operations, accelerating the processor's computing speed, saving computing resources, and improving inference efficiency.

[0007] According to one aspect of this disclosure, a method is provided for a processor including multiple computing units to perform matrix multiplication operations, the method comprising:

[0008] The quantization parameter matrix is ​​divided into multiple parameter blocks, such that each parameter block has a preset first length and first width;

[0009] The plurality of parameter blocks are sequentially and non-repeatedly assigned to each of the plurality of computing units, such that each computing unit is assigned the first number of consecutive parameter blocks from the plurality of parameter blocks, except that at most one computing unit is assigned fewer than a first number of consecutive parameter blocks from the plurality of parameter blocks.

[0010] Each of the plurality of computing units dequantizes the assigned block of parameters;

[0011] Each of the plurality of computing units obtains the portion of the input matrix that should be multiplied with the assigned parameter block, and performs matrix multiplication on the dequantized assigned parameter block and the portion of the input matrix to complete the matrix multiplication operation between the input matrix and the model parameters of the large model.

[0012] The quantization parameter matrix is ​​a two-dimensional matrix obtained by quantizing the model parameters. The first quantity is calculated based on the length and width of the quantization parameter matrix, the first length and first width of each parameter block, and the number of the plurality of computing units.

[0013] The input matrix is ​​a two-dimensional matrix that serves as the input data for the decoding stage of the inference process of the large model.

[0014] In one possible implementation, the allocated parameter block of each of the plurality of computing units is distributed across the entire length of the quantization parameter matrix, and the portion of the input matrix acquired by each computing unit is the entire input matrix.

[0015] In one possible implementation, the matrix multiplication operation between the dequantized allocated parameter block and the portion of the input matrix includes:

[0016] Each of the plurality of computing units, for each parameter block in the dequantized assigned parameter block, uses a preset matrix multiplication operation instruction to sequentially and without repetition perform matrix multiplication operations on the sub-parameter block with the second length and the second width in the parameter block and the sub-input block with the third length and the third width in the input matrix that should be multiplied with the sub-parameter block, so as to complete the matrix multiplication operation between the assigned parameter block and the input matrix.

[0017] In one possible implementation, the method further includes:

[0018] When the length of the input matrix indicates the batch size of the input data and is smaller than the length and width of the multiplicand matrix specified by the matrix multiplication instruction, and the width of the multiplier matrix specified by the matrix multiplication instruction is smaller than the length and width of the multiplicand matrix and the length of the multiplier matrix specified by the matrix multiplication instruction,

[0019] When performing matrix multiplication of a sub-parameter block and a sub-input block using the matrix multiplication instruction, the sub-parameter block is used as the multiplicand matrix, and the sub-input block is used as the multiplier matrix.

[0020] The second length, the second width, and the third width are respectively equal to the length, width, and length of the multiplicand matrix specified in the matrix multiplication operation instruction, and the second width is equal to the third length.

[0021] In one possible implementation, when the length of the input matrix is ​​equal to or less than the width of the multiplier matrix specified by the matrix multiplication instruction, the third length is equal to the length of the input matrix.

[0022] In one possible implementation, the processor is a graphics processor, and the computing unit is a streaming multiprocessor.

[0023] Each computing unit runs a thread block, in which matrix multiplication operations between the sub-parameter block and the sub-input block are performed sequentially for each parameter block in the dequantized assigned parameter block.

[0024] In one possible implementation, the matrix multiplication instruction is a matrix multiplication accumulator (MMA) instruction in the PTX instruction set executed by a parallel thread.

[0025] According to another aspect of this disclosure, a processor comprising a plurality of computing units is provided for performing matrix multiplication operations, the processor being configured to:

[0026] The quantization parameter matrix is ​​divided into multiple parameter blocks, such that each parameter block has a preset first length and first width;

[0027] The plurality of parameter blocks are sequentially and non-repeatingly assigned to each of the plurality of computing units, such that, except for at most one computing unit being assigned a consecutive number of parameter blocks less than a first number from the plurality of parameter blocks, each computing unit is assigned a consecutive first number of parameter blocks from the plurality of parameter blocks, and

[0028] Each of the plurality of computing units is configured as follows:

[0029] Dequantize the assigned parameter block;

[0030] Obtain the portion of the input matrix that should be multiplied with the assigned parameter block, and perform matrix multiplication on the dequantized assigned parameter block and the portion of the input matrix to complete the matrix multiplication operation between the input matrix and the model parameters of the large model.

[0031] The quantization parameter matrix is ​​a two-dimensional matrix obtained by quantizing the model parameters. The first quantity is calculated based on the length and width of the quantization parameter matrix, the first length and first width of each parameter block, and the number of the plurality of computing units.

[0032] The input matrix is ​​a two-dimensional matrix that serves as the input data for the decoding stage of the inference process of the large model.

[0033] According to another aspect of this disclosure, an apparatus is provided for a processor including multiple computing units to perform matrix multiplication operations, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to perform the following operations:

[0034] The processor executes the computer program to perform the following operations:

[0035] The quantization parameter matrix is ​​divided into multiple parameter blocks, such that each parameter block has a preset first length and first width;

[0036] The plurality of parameter blocks are sequentially and non-repeatedly assigned to each of the plurality of computing units included in the processor, such that each computing unit is assigned the first number of consecutive parameter blocks from the plurality of parameter blocks, except that at most one computing unit is assigned a number of consecutive parameter blocks less than a first number from the plurality of parameter blocks.

[0037] Each of the plurality of computing units dequantizes the assigned parameter block;

[0038] Each of the plurality of computing units obtains the portion of the input matrix that should be multiplied with the assigned parameter block, and performs matrix multiplication on the dequantized assigned parameter block and the portion of the input matrix to complete the matrix multiplication operation between the input matrix and the model parameters of the large model.

[0039] The quantization parameter matrix is ​​a two-dimensional matrix obtained by quantizing the model parameters. The first quantity is calculated based on the length and width of the quantization parameter matrix, the first length and first width of each parameter block, and the number of the plurality of computing units.

[0040] The input matrix is ​​a two-dimensional matrix that serves as the input data for the decoding stage of the inference process of the large model.

[0041] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0042] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0043] In the method for performing matrix multiplication operations in a processor comprising multiple computing units according to the present disclosure, a quantization parameter matrix is ​​divided into multiple parameter blocks; the multiple parameter blocks are sequentially and non-repeatingly assigned to each of the multiple computing units, such that, except for at most one computing unit being assigned a consecutive number of fewer than a first number of parameter blocks from the multiple parameter blocks, each computing unit is assigned a consecutive first number of parameter blocks from the multiple parameter blocks; each of the multiple computing units dequantizes the assigned parameter blocks; each of the multiple computing units obtains the portion of the input matrix that should be multiplied by the assigned parameter blocks, and performs matrix multiplication operations on the dequantized assigned parameter blocks and the portion of the input matrix to complete the matrix multiplication operation of the input matrix and the model parameters of a large model. Since the quantization parameter blocks assigned to each computing unit are non-repeating, only one dequantization operation is needed globally on the quantization parameter matrix, avoiding multiple dequantization operations on the quantized model parameters during inference when the model parameters of a large model have been quantized, thus improving the computational efficiency of matrix multiplication operations, accelerating the processor's computation speed, saving computational resources, and improving inference efficiency.

[0044] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0045] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0046] Figure 1 A flowchart illustrating a method for a processor including multiple computing units to perform matrix multiplication operations according to an embodiment of the present disclosure;

[0047] Figure 2 A schematic diagram illustrating the process of allocating parameter blocks to a computing unit according to an embodiment of the present disclosure is shown.

[0048] Figure 3 A schematic diagram of matrix multiplication operation according to an embodiment of the present disclosure is shown;

[0049] Figure 4 A schematic diagram of matrix multiplication operation according to another embodiment of the present disclosure is shown;

[0050] Figure 5 A block diagram of an apparatus for performing matrix multiplication operations by a processor including a plurality of computing units, according to an embodiment of the present disclosure, is shown.

[0051] Figure 6 A block diagram is shown of an apparatus for performing matrix multiplication operations by a processor including a plurality of computing units, according to another embodiment of the present disclosure. Detailed Implementation

[0052] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0053] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0054] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0055] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0056] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0057] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0058] The inference process of a large model refers to the process of generating output data from input data using a trained model. Large models such as Generative Pre-trained Transformers (GPT), Open Pre-trained Transformers (OPT), and Meta AI Large Language Model (LLaMA) include two processes in their inference phase: prefilling and decoding. Prefilling is the computational process in the large model's inference process before generating the first output token after receiving the input token sequence. Decoding is the computational process after generating the first token in the prefilling phase, where the model iteratively generates subsequent tokens until an end marker is generated or the maximum sequence length is reached. In actual implementations, the inference phase of a large model may also include other processes, such as input processing before prefilling and post-processing after decoding, which will not be listed here.

[0059] In the inference phase of large models, matrix multiplication operations are time-consuming, sometimes accounting for 60% or more of the total inference time. To improve the computational efficiency of matrix multiplication, some implementations deploy large models on processors (e.g., GPUs) that include multiple computing units. Since the input data (e.g., a 32×4096 matrix) and model parameters (e.g., a 4096×4096 matrix) required for matrix multiplication are both large, they are typically divided into multiple blocks and allocated to multiple computing units within the processor. Each computing unit performs matrix multiplication on its assigned input data block and the corresponding model parameter block, producing an output data block. The output data blocks from all computing units can then be combined to form the complete output data. In related technologies, the computational tasks of computing units are typically mapped to the output data; that is, for each output data block, the input data block and the corresponding model parameter block that can be computed to produce that output data block are assigned to a computing unit. Since the same model parameter block is often used for operations on different output data blocks, when the model parameters are quantized, the computational units that are assigned the same model parameter block need to dequantize the same model parameter block before performing matrix multiplication with the corresponding input data block.

[0060] For example, as a processor capable of deploying large models, Nvidia GPUs boast a powerful Compute Unified Device Architecture (CUDA) ecosystem and extremely high computing power. The CUDA Kernel consists of functions executed in parallel on Nvidia GPUs, composed of multiple (e.g., 131,070) threads, with a fixed number (e.g., 512) of threads organized into thread blocks. Tensor Cores are dedicated computing units in Nvidia GPUs, focused on accelerating matrix multiply-accumulate (MMA) operations. They support mixed-precision computation (e.g., FP16 input, FP32 accumulation), and a single instruction can complete multiply-accumulate operations on small matrices (e.g., 16×16), achieving higher throughput than traditional CUDA Kernels. When model parameters are quantized, during matrix multiplication, the Nvidia GPU's CUDA kernel uses multi-stage caching to cache data in stages, loop tiling to divide the parameter matrix into blocks, and dequantization magic number optimization (e.g., restoring from 4-bit to 16-bit) to the dequantized parameter blocks. Then, it calls the Nvidia GPU's Tensor Core to perform matrix multiplication with the input data. When performing matrix multiplication through the CUDA kernel, as described above, the computational tasks of thread blocks are typically mapped to output data, repeatedly assigning the same model parameter blocks to multiple thread blocks. This results in multiple thread blocks needing to dequantize the same model parameter block before executing MMA. From a global perspective, the repeated dequantization of model parameters causes a significant amount of redundant work, wasting computational resources, reducing GPU utilization and computation speed, and slowing down model inference.

[0061] More specifically, taking the attention layer of the LLaMA2-7B model as an example, during matrix multiplication, the size (shape) of the input matrix A of the attention layer is batch_size × 4096, where batch_size is the number of samples input to the large language model each time, and the batch size is generally much smaller than 4096. The size of the weight matrix B of the attention layer is 4096 × 4096. When performing matrix multiplication C = A × B on the input matrix A and the weight matrix B, the resulting output matrix C has a size of batch_size × 4096.

[0062] Before performing matrix multiplication of input matrix A and weight matrix B, in related technologies, such as those following the computational logic of CUDA kernels, the input matrix is ​​typically divided into multiple 16×4096 input data blocks 'a', and the weight matrix is ​​divided into multiple 32×4096 model parameter blocks 'b'. The 16×32 output data block 'c' = a×b is obtained by multiplying input data block 'a' with model parameter block 'b'. In actual computation, the computational tasks of thread blocks are mapped to the output data; that is, the computational task for obtaining each output data block 'c' is assigned to each thread block, allowing each thread block to obtain the output data block 'c' by calculating c = a×b. If batch_size = 32, the size of the output matrix C is 32×4096, and the number of output data blocks 'c' is (32 / 16) × (4096 / 32) = 256, requiring 256 thread blocks. Based on the properties of matrix multiplication, different input data blocks 'a' need to be multiplied with the same model parameter block 'b' to obtain different output data blocks 'c'. Therefore, in the traditional approach of mapping computational tasks of thread blocks to output data, the same model parameter block b and different input data blocks a that need to be multiplied by the model parameter block are usually assigned to multiple different thread blocks.

[0063] Beyond the application scenarios of Nvidia GPUs, other processors that include multiple computing units often encounter situations where the entire processor repeatedly performs dequantization operations on the same model parameters multiple times. For example, in AMD GPUs, the computation task for obtaining each output data block c is typically assigned to each computing unit, resulting in multiple different computing units repeatedly performing dequantization operations on the same model parameter block b.

[0064] Therefore, in processors with large models, according to traditional methods, multiple different computing units will repeatedly perform dequantization operations on the same model parameter block. From a global perspective, the entire processor repeatedly performs dequantization operations on the same model parameters, which wastes computing resources, reduces GPU utilization and computing speed, and also reduces the inference speed of the model.

[0065] Based on the aforementioned technical problems, this disclosure proposes a method for performing matrix multiplication operations on a processor comprising multiple computing units. When the model parameters of a large model are quantized, during inference, the quantized parameter matrix is ​​divided into multiple parameter blocks and sequentially and non-repeatingly allocated to multiple computing units. This avoids multiple dequantizations of the quantized model parameters during inference, reducing the computational overhead caused by repeated dequantization in traditional methods, improving the computational efficiency of matrix multiplication operations, accelerating processor computation, saving computational resources, and improving inference efficiency. The method for performing matrix multiplication operations on a processor comprising multiple computing units provided in this disclosure will be described in detail below.

[0066] Figure 1 A flowchart illustrating a method for performing matrix multiplication operations using a processor comprising a plurality of computing units, according to an embodiment of the present disclosure, is shown. The method is used for a processor having N computing units, where N is an integer greater than 1.

[0067] In one example, the processor is an Nvidia GPU, and correspondingly, the computing units include Streaming Multiprocessors (SMs) within the Nvidia GPU. Optionally, the number of SMs is the same as the number of thread blocks executing CUDAKernel functions in the Nvidia GPU; in this case, one thread block is allocated on each SM. Thread blocks are used to dequantize parameter blocks and perform matrix multiplication operations using each dequantized parameter block.

[0068] In another example, the processor could also be an AMD GPU platform; accordingly, the compute units include compute units (CUs) in the AMD GPU, each CU being assigned a work group. The work group is used to dequantize the parameter blocks and perform matrix multiplication operations using each dequantized parameter block.

[0069] In practical applications, the processor can also be of other types. This disclosure does not limit the implementation of the processor, as long as the method for performing matrix multiplication operations on a processor including multiple computing units according to this disclosure can be applied.

[0070] like Figure 1 As shown, the method includes:

[0071] Step 101: Divide the quantization parameter matrix into multiple parameter blocks, such that each parameter block has a preset first length and first width.

[0072] The quantization parameter matrix is ​​a two-dimensional matrix obtained by quantizing the model parameters. For example, the quantization parameter matrix is ​​obtained by quantizing the model parameters in a large model based on a model parameter quantization algorithm. These model parameters can be, for example, the weight matrix of the attention layer of the large model, but are not limited to this; they can also be other parameter matrices that need to be multiplied with the input data of the decoding stage of the large model (described later). Optionally, the model parameter quantization algorithm can be the AWQ algorithm, uniform quantization, etc. This disclosure does not limit the type of model parameter quantization algorithm.

[0073] The length (e.g., number of rows), i.e., the first length, of each parameter block is less than the length (e.g., number of rows) of the quantization parameter matrix, and the width (e.g., number of columns), i.e., the first width, is less than the width (e.g., number of columns) of the quantization parameter matrix. The first length and the first width may be the same or different. Taking the first length and the first width being the same as an example, the first length and the first width can be 16, 32, 64, etc., and the size (shape) of each parameter block can be 16×16, 32×32, 64×64, etc. In actual implementation, the values ​​of the first length and the first width can also be other values, and the first length and the first width can also be different. This embodiment does not limit the implementation method of the first length and the first width. In addition, the first length and the first width can be fixed values, or they can be dynamically adjusted, for example, according to the decoding progress, processor load, etc.

[0074] Step 102: Assign multiple parameter blocks sequentially and without repetition to each of the multiple computational units, such that, except for at most one computational unit being assigned a consecutive number of fewer than a first number of parameter blocks from the multiple parameter blocks, each computational unit is assigned a consecutive first number of parameter blocks from the multiple parameter blocks. The first number can be calculated based on the length and width of the quantization parameter matrix, the first length and first width of each parameter block, and the number of the multiple computational units.

[0075] For example, the first quantity is determined by the following formula:

[0076] ;

[0077] Where K represents the length of the quantization parameter matrix; N represents the width of the quantization parameter matrix; k represents the length of the parameter block (i.e., the first length), n represents the width of the parameter block (i.e., the first width); and x represents the number of computational units. This indicates rounding up to the nearest integer.

[0078] Taking a quantization parameter matrix with column direction (vertical) as length and row direction (horizontal) as width as an example, multiple parameter blocks are sequentially and non-repeatingly assigned to each of the multiple computation units. This includes: starting with the parameter block in the first row and first column, continuously selecting a first number of parameter blocks along the column direction to obtain the first computation task; continuing along the next parameter block after the last selected parameter block, continuously selecting a first number of parameter blocks along the column direction to obtain the second computation task, and so on, until starting with the next parameter block after the last selected parameter block, continuously selecting all remaining parameter blocks along the column direction (the number of these parameter blocks is less than or equal to the first number) to obtain the xth computation task. During the process of selecting the first number of parameter blocks along the column direction, if all parameter blocks in a column are selected before the first number of selected parameter blocks is reached, the process jumps to the next column and continues selecting parameter blocks along the column direction until the first number is reached. The first to xth computation tasks are assigned to different computation units so that each computation unit can use the assigned parameter blocks to perform subsequent matrix multiplication calculations.

[0079] For example, refer to Figure 2 If the length and width of the quantization parameter matrix are both 4096, that is, the size of the quantization parameter matrix is ​​4096×4096, then K=4096 and N=4096. And let the first length and first width both be 32, that is, the size of the parameter block is 32×32, then k=32 and n=32. In this case, after dividing the data into blocks according to the first length and first width, we can obtain 128 parameter blocks in the row direction and 128 parameter blocks in the column direction (i.e.,...). Figure 2 Each rectangular block is referred to as a "small block" in the calculation. If the number of calculation units x = 80, then according to the method for determining the first number above, the first number is 128 × 128 / 80 = 205. That is, each calculation unit is responsible for calculating the matrix multiplication of 205 consecutive parameter blocks.

[0080] Assuming the 80 computational units are numbered from 0 to 79, then:

[0081] For the computation unit numbered 0, the computation task for this unit is: 128 parameter blocks in the first column + 77 parameter blocks in the second column, totaling 205 parameter blocks. Figure 2 (The green rectangle)

[0082] For computation unit numbered 1, the computation task for this unit is: the remaining 51 parameter blocks in the second column + the 128 parameter blocks in the third column + the 26 parameter blocks in the fourth column, totaling 205 parameter blocks. Figure 2 (The red rectangle in the middle).

[0083] ...

[0084] This cycle repeats;

[0085] For the computation unit numbered 79, the computation task of this computation unit is: all the remaining unselected parameter blocks, that is, 128×128-79×205=189 parameter blocks.

[0086] Step 103: Each of the multiple computing units dequantizes the assigned parameter block.

[0087] The computational unit uses a dequantization algorithm adapted to the model parameter quantization algorithm to dequantize the assigned parameter blocks. For example, if the model parameter quantization algorithm is AWQ, the dequantization algorithm is AWQ inverse quantization; if the model parameter quantization algorithm is linear quantization, the dequantization algorithm is linear inverse quantization. The input matrix is ​​never quantized, therefore dequantization is not required.

[0088] Step 104: Each of the multiple computing units obtains the part of the input matrix that should be multiplied with the assigned parameter block, and performs matrix multiplication operation on the dequantized assigned parameter block and part of the input matrix to complete the matrix multiplication operation of the input matrix and the model parameters of the large model.

[0089] For example, matrix multiplication of the input matrix with the model parameters of the large model is used to infer the inference request input to the large model, such as to infer the prompt word, and obtain the output word sequence as the inference result.

[0090] The input matrix is ​​a two-dimensional matrix of input data for the decoding stage of the large model's inference, and its size can be M×K. Here, M = batch_size, where batch_size represents the number of input vectors (i.e., batch size) input to the large model (e.g., the attention layer) during the decoding stage. It can be 64, 32, 16, 8, etc., and this embodiment does not limit the value of M. For example, the decoding stage of the large model's inference is used to generate a token-by-token output word sequence, and each input vector in the decoding stage can be the vector corresponding to the previous word inferred for each prompt word's word sequence input to the large model. Accordingly, the input matrix can be understood as a set of M vectors corresponding to the previous word inferred for each of the M prompt words' word sequences input to the large model, with each vector having a length of K. The value of K is the same as the length K of the K×N quantization parameter matrix.

[0091] In one example, the assigned parameter blocks for each of the multiple computation units are distributed across the entire length of the quantization parameter matrix (which can also be understood as the assigned parameter blocks including a complete column of parameter blocks), and the portion of the input matrix obtained by each computation unit is the entire input matrix.

[0092] for example: Figure 2 The computation unit numbered 0 is assigned parameter blocks distributed across the entire length of the quantization parameter matrix, that is, including all 128 parameter blocks in the first column. These 128 parameter blocks need to be multiplied by the entire input matrix. Assuming the input matrix size is 64×4096, the computation unit numbered 0 needs to obtain the entire 64×4096 input matrix. Furthermore, the computation unit numbered 0 is assigned parameter blocks ( Figure 2 The green rectangular block (in the middle) spans two columns of parameter blocks in the width direction, with a size of 64 in the width direction. Therefore, in the calculation unit numbered 0, a calculation result of size 64×64 can be obtained. In this 64×64 calculation result, the left half, which is 64×32, is the result of multiplying the entire input matrix (size 64×4096) with the 128 parameter blocks (size 4096×32) in the first column. This result can be used as part of the final result of the matrix multiplication operation between the input matrix and the parameter matrix, for example, directly output to global memory. In this 64×64 calculation result, the right half, with a size of 64×32, is the result of multiplying the left half of the input matrix (size 64×2464) by the upper 77 parameter blocks of the second column (size 2464×32). This result needs to be added to the left half of the calculation result from calculation unit number 1 (described below), which also has a size of 64×32, to obtain the multiplication result of the entire input matrix with the 128 parameter blocks of the second column. The calculation result of the right half of the 64×32 calculation from calculation unit number 0 can, for example, be stored in a buffer, waiting for the corresponding part of the calculation result from calculation unit number 1 to arrive and be accumulated with it.

[0093] For example: Figure 2 The parameter block assigned to computation unit number 1 is also distributed across the entire length of the quantization parameter matrix, that is, including all 128 parameter blocks in the third column. These 128 parameter blocks also need to be multiplied by the entire input matrix. Assuming the input matrix size is also 64×4096, computation unit number 1 needs to obtain the entire 64×4096 input matrix. Furthermore, the parameter block assigned to computation unit number 1 ( Figure 2The red rectangle (in the center) spans three columns of parameter blocks in the width direction, with a dimension of 96 in that direction. Therefore, in calculation unit number 1, a calculation result of size 64×96 can be obtained. In this 64×96 calculation result, the left part, with a size of 64×32, is the result of multiplying the right part of the input matrix (size 64×1632) by the lower 51 parameter blocks (size 1632×32) of the second column. This result can be stored, for example, in the aforementioned buffer, and accumulated with the calculation result of the right half of the 64×32 calculation result in calculation unit number 0. The middle part of the 64×32 calculation result is the result of multiplying the entire input matrix (size 64×4096) by all 128 parameter blocks (size 4096×32) of the third column. This result can be used as part of the final result of the matrix multiplication operation between the input matrix and the parameter matrix, for example, directly output to global memory. The right part of the calculation result, which is 64×32, is the result of multiplying the left part of the input matrix, which is 64×832, with the upper 26 parameter blocks (832×32) of the fourth column. It can be similarly stored in the buffer mentioned above, waiting to be added to the left part of the calculation result of the calculation unit numbered 2, which is 64×32, to obtain the result of multiplying the entire input matrix with the 128 parameter blocks of the fourth column.

[0094] Similarly, each of the processor's multiple computing units obtains the portion of the input matrix that should be multiplied by the assigned parameter block (where the entire input matrix is ​​obtained when the assigned parameter block is distributed across the entire length of the quantized parameter matrix), and performs matrix multiplication on the dequantized assigned parameter block and the portion of the input matrix, thereby completing the matrix multiplication operation between the input matrix and the model parameters of the large model.

[0095] In summary, the method for performing matrix multiplication in a processor comprising multiple computing units provided in this embodiment involves dividing the quantization parameter matrix into multiple parameter blocks; sequentially and non-repeatingly allocating these parameter blocks to each computing unit, such that each computing unit is allocated a first number of consecutive parameter blocks, except for at most one computing unit being allocated fewer than a first number of consecutive parameter blocks; each computing unit dequantizes its allocated parameter blocks; and each computing unit obtains the portion of the input matrix that should be multiplied by the allocated parameter blocks, and performs matrix multiplication on the dequantized allocated parameter blocks and the portion of the input matrix, thereby completing the matrix multiplication operation between the input matrix and the model parameters of a large model. Since different computing units are allocated different portions (different parameter blocks) of the quantization parameter matrix, dequantization is only required once globally for all parameter blocks. Compared with related technologies that require multiple dequantizations of the same parameter blocks, this method saves computational resources, improves the computational efficiency of matrix multiplication, accelerates processor computation, and improves inference efficiency.

[0096] In one example, matrix multiplication is performed on a portion of the dequantized assigned parameter block and the input matrix. This includes: each of the multiple computing units, for each parameter block in the dequantized assigned parameter block, using a preset matrix multiplication instruction, sequentially and without repetition performs matrix multiplication on the sub-parameter block with a second length and a second width in the parameter block and the sub-input block with a third length and a third width in the input matrix that should be multiplied by the sub-parameter block, to complete the matrix multiplication operation between the assigned parameter block and the input matrix.

[0097] In one example, the processor is a graphics processing unit (GPU), the computing unit is a streaming multiprocessor (SM), and each computing unit runs a thread block. Accordingly, each computing unit sequentially and without repetition performs matrix multiplication operations on sub-parameter blocks of a second length and second width within the assigned parameter block and sub-input blocks of a third length and third width within the input matrix, which should be multiplied by the sub-parameter blocks. This includes, within the thread block, sequentially performing matrix multiplication operations between the sub-parameter blocks and sub-input blocks for each parameter block in the dequantized assigned parameter block.

[0098] Matrix multiplication instructions are hardware-level instructions generated by the processor when matrix multiplication is required. For example, in a processor with multiple computing units, such as an Nvidia GPU, where each computing unit is a streaming multiprocessor (SM) and each unit runs a thread block, the matrix multiplication instruction can be a matrix multiply-accumulate (MMA) instruction from the Parallel Thread Execution (PTX) instruction set. The length of the sub-parameter block (the second length) is less than or equal to the length of the parameter block (the first length), and the width of the sub-parameter block (the second width) is less than or equal to the width of the parameter block (the first width). When the sub-parameter block is used as the multiplicand matrix (the matrix before the multiplication sign) and the sub-input block is used as the multiplier matrix (the matrix after the multiplication sign), the second width is equal to the third length. Conversely, when the sub-input block is used as the multiplicand matrix and the sub-parameter block is used as the multiplier matrix, the third width is equal to the second length. The specific values ​​of the second length, second width, third length, and third width can be set according to the specifications of the matrix multiplication instruction.

[0099] For example, let's continue with the example of computation unit numbered 0 mentioned earlier. In this computation unit, assuming that the second length and the second width are both set to 16, the size of the sub-parameter block is 16×16; and the third length is set to 8 and the third width is set to 16, the size of the sub-input block is 8×16. In this case, for example, for each dequantized parameter block, in each iteration step of, for example, the thread block, the computation unit sequentially and without repetition performs matrix multiplication operations on each 16×16 sub-parameter block in the parameter block and all the 8×16 sub-input blocks in the portion of the input matrix that should be multiplied by that sub-parameter block. For example, in Figure 3 In this case, we still assume that the size of the input matrix is ​​64×4096, the size of the parameter matrix is ​​4096×4096, and the computational unit numbered 0 is... Figure 2 The example in the text was similar, with 128 parameter blocks in the first column and 77 parameter blocks in the second column (each parameter block being 32×32 in size), as shown below. Figure 3 The green rectangular block is shown in the diagram. In the computational unit numbered 0, in the first iteration step, for the first parameter block, the four sub-parameter blocks of size 16×16 within that parameter block are calculated respectively (in...). Figure 3 The matrix (shown in dark green) performs matrix multiplication with all 8×16 sub-input blocks in the input matrix that should be multiplied by their corresponding sub-parameter blocks. Specifically, for Figure 3 In the top row of the matrix, for each of the two sub-parameter blocks, all the sub-input blocks in the input matrix that should be multiplied by that sub-parameter block are... Figure 3 The input matrix in the diagram is shown as the 8 sub-input blocks in the left column (yellow), and for... Figure 3 In the bottom row of the matrix, for each of the two sub-parameter blocks, all sub-input blocks in the input matrix that should be multiplied by that sub-parameter block are... Figure 3 The input matrix is ​​shown in yellow, representing the eight sub-input blocks in the right column. For example, the matrix multiplication instruction can be used to multiply the first sub-input block (8×16) in the left column by the top-left sub-parameter block (16×16). Then, the matrix multiplication instruction can be used to multiply the second sub-input block (8×16) in the left column by the top-left sub-parameter block (16×16), and so on, completing the matrix multiplication of the first sub-parameter block with each of the eight sub-input blocks in the left column. Next, for example, the matrix multiplication instruction can be used to sequentially multiply the first sub-input block in the right column by the bottom-left sub-parameter block, the second sub-input block in the right column by the bottom-left sub-parameter block, and so on, completing the matrix multiplication of the second sub-parameter block with each of the eight sub-input blocks in the right column. Next, similarly, matrix multiplication is performed on the top-right sub-parameter block and each of the eight sub-input blocks in the left column, and the bottom-right sub-parameter block and each of the eight sub-input blocks in the right column are also multiplied. Thus, in each computational unit, for each parameter block in the dequantized assigned parameter block, using preset matrix multiplication instructions, matrix multiplication is performed sequentially and without repetition on each sub-parameter block in the parameter block and all the sub-input blocks in the input matrix that should be multiplied by that sub-parameter block, thereby completing the matrix multiplication operation between the assigned parameter block and the input matrix.

[0100] When performing matrix multiplication on a sub-parameter block and a sub-input block using preset matrix multiplication instructions, the instructions typically specify the dimensions of the two matrices involved in the operation. That is, the matrix multiplication instructions can specify the values ​​of m, k, and n to multiply an m×k matrix by a k×n matrix. For example, the MMA instruction in the PTX instruction set can specify m=16, n=8, and k=16. In this case, refer to... Figure 4 The matrix multiplication instruction can calculate C = A × B to multiply a 16 × 16 matrix A by a 16 × 8 matrix B, resulting in a 16 × 8 matrix C.

[0101] In related technologies, when performing matrix multiplication operations on model parameter blocks and input data blocks using matrix multiplication instructions, the input data block (or its sub-blocks) is typically mapped to matrix A as the multiplicand matrix of the matrix multiplication instruction, and the model parameter block (or its sub-blocks) is mapped to matrix B as the multiplier matrix of the matrix multiplication instruction. In this case, if the length of the input data indicates the batch size (e.g., in the example above where the size of the input matrix A for the attention layer is batch_size × 4096), and the length of the input data (batch size) is less than the length m and width k of the multiplicand matrix specified by the matrix multiplication instruction, then when the input data block is used as the multiplicand matrix A and the model parameter block as the multiplier matrix B according to related technologies, the size of the input data block is smaller than the hardware-supported size of the multiplicand matrix A, leading to a waste of hardware resources. For example, when m = 16, n = 8, k = 16, and the batch size is 1 to 15, the maximum length of the input data block is the batch size 1 to 15, and the maximum size of the input data block is correspondingly 1 × 16 to 15 × 16. If an input data block with a maximum size of 1×16 to 15×16 is used as the multiplicand matrix A, then during actual computation, for example, the remaining 15×16 to 1×16 parts that are less than 16×16 will be padded with zeros to meet the size specified by the matrix multiplication instruction. Thus, in each operation, 15×16 to 1×16 units of data are invalid, wasting hardware resources. Furthermore, it can be seen that the smaller the batch size, the smaller the size of the valid data, and the larger the size of the invalid data, resulting in more wasted hardware resources.

[0102] To address this technical problem, in some embodiments of this disclosure, when the length of the input matrix indicates the batch size of the input data and is less than the length and width of the multiplicand matrix specified in the matrix multiplication instruction, and the width of the multiplier matrix specified in the matrix multiplication instruction is less than the length and width of the multiplicand matrix and the length of the multiplier matrix specified in the matrix multiplication instruction, when performing matrix multiplication of the sub-parameter block and the sub-input block using the matrix multiplication instruction, the sub-parameter block is used as the multiplicand matrix, the sub-input block is used as the multiplier matrix, and the second length, second width, and third width are respectively equal to the length and width of the multiplicand matrix and the length of the multiplier matrix specified in the matrix multiplication instruction, and the second width is equal to the third width. In other words, when n < m and n < k as specified in the matrix multiplication instruction, the sub-parameter block is mapped to the aforementioned matrix A of size m × k, and the sub-input block is mapped to the aforementioned matrix B of size k × n. The length m of the multiplicand matrix specified in the matrix multiplication instruction is determined as the length of the sub-parameter block, i.e., the second length. The width of the multiplicand matrix and the length k of the multiplier matrix specified in the matrix multiplication instruction are determined as the width of the sub-parameter block, i.e., the second width, and the width of the sub-input block, i.e., the third width. Regarding the length of the sub-input block, i.e., the third length, when the length of the input matrix, i.e., the batch size, is greater than the width n of the multiplier matrix specified in the matrix multiplication instruction, the input matrix can be divided into sub-input blocks with a maximum size of n × k. In this case, the length of the sub-input block, i.e., the third length, is equal to or less than the width n of the multiplier matrix specified in the matrix multiplication instruction. When the length of the input matrix, i.e., the batch size, is equal to or less than the width *n* of the multiplier matrix specified in the matrix multiplication instruction, the input matrix does not need to be divided along its length; it only needs to be divided along its width, resulting in a sub-input block of size *batch_size × k*. In this case, the length of the sub-input block, i.e., the third length, is equal to the length of the input matrix. By mapping the length of the input matrix indicating the batch size to the smallest dimension among the length and width of the multiplicand matrix and the multiplier matrix specified in the matrix multiplication instruction, the waste of hardware resources for matrix multiplication operations can be reduced when the batch size is small, thereby improving hardware utilization and computational efficiency.

[0103] For example, in the case where the size of the input matrix A of the attention layer above is batch_size × 4096, the length of the input matrix indicates the batch size of the input data. When m = 16, n = 8, and k = 16, n < m and n < k. In this case, if batch_size < m and batch_size < k (i.e., batch_size < 16), then the sub-parameter block is used as the multiplicand matrix A, and the sub-input block is used as the multiplier matrix B, such that the length of the sub-parameter block (second length) = m = 16, the width of the sub-parameter block (second width) = k = 16, and the width of the sub-input block (third width) = k = 16. When the length of the input matrix (batch size) is greater than the width n (8 < batch_size < 16) of the multiplier matrix specified by the matrix multiplication instruction, matrix multiplication can be performed on sub-input blocks with a maximum size of n × k = 8 × 16. For example, when batch_size=14, matrix multiplication instructions can be used to perform matrix multiplication operations on sub-parameter blocks of size 16×16 and sub-input blocks of size 8×16 or 6×16 each time. When the length of the input matrix, i.e., the batch size, is equal to or less than the width n of the multiplier matrix specified by the matrix multiplication instructions (batch_size≤8), matrix multiplication operations can be performed on sub-input blocks of size batch_size×k. For example, when batch_size=4, matrix multiplication instructions can be used to perform matrix multiplication operations on sub-parameter blocks of size 16×16 and sub-input blocks of size 4×16 each time. It should be understood that internally, processors sometimes have a mechanism that uses k as an internal dimension and automatically aligns the internal dimension during matrix multiplication operations. Therefore, there is no need to manually transpose the sub-input blocks; the k-dimensional alignment of the sub-parameter blocks and sub-input blocks can be achieved during actual computation.

[0104] For example, when performing matrix multiplication on a parameter block of size 32×32 as described above, with batch_size=14, each parameter block needs to be multiplied by a 14×32 portion of the input matrix. Using methods in related technologies, for a 32×32 parameter block, each matrix multiplication operation requires multiplying a 14×16 sub-input block as matrix A with a 16×8 sub-parameter block as matrix B, requiring a total of 8 matrix multiplication operations. As mentioned above, each matrix multiplication operation results in 2×16 units of invalid data, totaling 16×16 units of invalid data across the 8 operations, wasting hardware resources (computing power). Using the method proposed in this disclosure, for a 32×32 parameter block, each matrix multiplication operation requires multiplying a 16×16 sub-parameter block as matrix A with an 8×16 or 6×16 sub-input block as matrix B, also requiring 8 matrix multiplication operations. However, in these 8 matrix multiplication operations, 4 of them involve multiplying a 16×16 sub-parameter block with an 8×16 sub-input block, thus avoiding the generation of invalid data. Only in the 4 operations involving multiplying a 16×16 sub-parameter block with a 6×16 sub-input block are 2×16 invalid data generated each time, for a total of 8×16 invalid data across all 8 operations. Compared to methods in related technologies, this reduces hardware resource waste by half, improving hardware utilization and computational efficiency.

[0105] When batch_size < n, this advantage of the method disclosed herein becomes more pronounced. For example, when performing matrix multiplication on a parameter block of size 32×32 as described above, with batch_size = 4, each parameter block needs to be multiplied by a 4×32 portion of the input matrix. If the method in the related art is used, for a 32×32 parameter block, in each matrix multiplication operation, a 4×16 sub-input block as matrix A and a 16×8 sub-parameter block as matrix B need to be multiplied separately, requiring a total of 8 matrix multiplication operations. At this time, as mentioned above, each matrix multiplication operation will have 12×16 data points that are invalid, resulting in a total of 96×16 invalid data points in 8 operations, and the corresponding hardware resources (computing power) are wasted. If the method proposed in this disclosure is adopted, for a 32×32 parameter block, in each matrix multiplication operation, a 16×16 sub-parameter block as matrix A and a 4×16 sub-input block as matrix B need to be multiplied, requiring 4 matrix multiplication operations. In these four matrix multiplication operations, each operation generates 4×16 invalid data points, for a total of 16×16 invalid data points, which is only 1 / 6 of the amount generated by methods in related technologies. Compared with methods in related technologies, this significantly reduces the waste of hardware resources and improves hardware utilization and computational efficiency.

[0106] In summary, according to the embodiments of this disclosure, when the length of the input matrix indicates the batch size of the input data and is less than the length and width of the multiplicand matrix specified in the matrix multiplication instruction, and the width of the multiplier matrix specified in the matrix multiplication instruction is less than the length and width of the multiplicand matrix and the length of the multiplier matrix specified in the matrix multiplication instruction, when performing matrix multiplication of sub-parameter blocks and sub-input blocks using the matrix multiplication instruction, the sub-parameter blocks are used as the multiplicand matrix, and the sub-input blocks are used as the multiplier matrix. Furthermore, the second length, second width, and third width are equal to the length and width of the multiplicand matrix and the length of the multiplier matrix specified in the matrix multiplication instruction, respectively, and the second width is equal to the third width. Therefore, when the batch size is less than the length and width of the multiplicand matrix specified in the matrix multiplication instruction, especially less than either the length and width of the multiplicand matrix or the length and width of the multiplier matrix specified in the matrix multiplication instruction, hardware resource waste can be reduced, hardware utilization and computational efficiency can be improved, thereby increasing inference efficiency.

[0107] Figure 5 A block diagram is shown of an apparatus for performing matrix multiplication operations by a processor including multiple computing units, according to an embodiment of the present disclosure. Figure 5 As shown, the device includes: a block partitioning module 510, a block allocation module 520, a dequantization module 530, and a matrix operation module 540.

[0108] The block partitioning module 510 is used to divide the quantization parameter matrix into multiple parameter blocks, such that each parameter block has a preset first length and a first width;

[0109] The block allocation module 520 is used to sequentially and non-repeatedly allocate the plurality of parameter blocks to each of the plurality of computing units, such that, except for at most one computing unit being allocated a consecutive number of fewer than a first number of parameter blocks from the plurality of parameter blocks, each computing unit is allocated a consecutive number of the first number of parameter blocks from the plurality of parameter blocks.

[0110] The dequantization module 530 is used for each of the plurality of computing units to dequantize the assigned parameter block;

[0111] The matrix operation module 540 is used for each of the plurality of computing units to obtain the portion of the input matrix that should be multiplied with the assigned parameter block, and to perform matrix multiplication operations on the dequantized assigned parameter block and the portion of the input matrix to complete the matrix multiplication operation between the input matrix and the model parameters of the large model.

[0112] The quantization parameter matrix is ​​a two-dimensional matrix obtained by quantizing the model parameters. The first quantity is calculated based on the length and width of the quantization parameter matrix, the first length and first width of each parameter block, and the number of the plurality of computing units.

[0113] The input matrix is ​​a two-dimensional matrix that serves as the input data for the decoding stage of the inference process of the large model.

[0114] For details, please refer to the above method implementation examples.

[0115] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0116] This disclosure also provides a processor comprising multiple computing units for performing matrix multiplication operations. The processor (specifically, it may execute a computer program) is configured as follows:

[0117] The quantization parameter matrix is ​​divided into multiple parameter blocks, such that each parameter block has a preset first length and first width;

[0118] Multiple parameter blocks are sequentially and non-repeatingly assigned to each of the multiple computational units, such that, except for at most one computational unit being assigned a consecutive number of parameter blocks less than a first number from the multiple parameter blocks, each computational unit is assigned a consecutive first number of parameter blocks from the multiple parameter blocks, and

[0119] Each of the multiple computing units is configured as follows:

[0120] Dequantize the assigned parameter block;

[0121] Obtain the portion of the input matrix that should be multiplied by the assigned parameter block, and perform matrix multiplication operations on the dequantized assigned parameter block and a portion of the input matrix to complete the matrix multiplication operation between the input matrix and the model parameters of the large model.

[0122] The quantization parameter matrix is ​​a two-dimensional matrix obtained by quantizing the model parameters. The first quantity is calculated based on the length and width of the quantization parameter matrix, the first length and first width of each parameter block, and the number of multiple computational units.

[0123] The input matrix is ​​a two-dimensional matrix of input data used in the decoding phase of the inference process of a large model.

[0124] For details, please refer to the above method implementation examples.

[0125] This disclosure also provides an apparatus for a processor including multiple computing units to perform matrix multiplication operations, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0126] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0127] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0128] Figure 6 This is a block diagram illustrating an apparatus 1900 for performing matrix multiplication operations using a processor comprising multiple computing units, according to an exemplary embodiment. For example, apparatus 1900 may be provided as a server or terminal device. (Refer to...) Figure 6 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0129] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0130] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method and achieve the following operations:

[0131] The quantization parameter matrix is ​​divided into multiple parameter blocks, such that each parameter block has a preset first length and first width;

[0132] The plurality of parameter blocks are sequentially and non-repeatedly assigned to each of the plurality of computing units included in the processor, such that each computing unit is assigned the first number of consecutive parameter blocks from the plurality of parameter blocks, except that at most one computing unit is assigned a number of consecutive parameter blocks less than a first number from the plurality of parameter blocks.

[0133] Each of the plurality of computing units dequantizes the assigned parameter block;

[0134] Each of the plurality of computing units obtains the portion of the input matrix that should be multiplied with the assigned parameter block, and performs matrix multiplication on the dequantized assigned parameter block and the portion of the input matrix to complete the matrix multiplication operation between the input matrix and the model parameters of the large model.

[0135] The quantization parameter matrix is ​​a two-dimensional matrix obtained by quantizing the model parameters. The first quantity is calculated based on the length and width of the quantization parameter matrix, the first length and first width of each parameter block, and the number of the plurality of computing units.

[0136] The input matrix is ​​a two-dimensional matrix that serves as the input data for the decoding stage of the inference process of the large model.

[0137] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0138] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0139] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0140] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0141] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0142] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0144] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for performing matrix multiplication operations using a processor comprising multiple computing units, characterized in that, The method includes: The quantization parameter matrix is ​​divided into multiple parameter blocks, such that each parameter block has a preset first length and first width; The plurality of parameter blocks are sequentially and non-repeatedly assigned to each of the plurality of computing units, such that each computing unit is assigned the first number of consecutive parameter blocks from the plurality of parameter blocks, except that at most one computing unit is assigned fewer than a first number of consecutive parameter blocks from the plurality of parameter blocks. Each of the plurality of computing units dequantizes an assigned block of parameters, the assigned block of parameters being distributed across the entire length of the quantization parameter matrix; Each of the plurality of computing units acquires an input matrix. When the length of the input matrix indicates the batch size of the input data, and the batch size is less than the length and width of the multiplicand matrix specified in the matrix multiplication instruction, and the width of the multiplier matrix specified in the matrix multiplication instruction is less than the length and width of both the multiplicand and multiplier matrices specified in the matrix multiplication instruction, for the dequantized assigned parameter block, using the matrix multiplication instruction, the sub-parameter block with a second length and a second width within the parameter block is used as the multiplicand matrix, and the sub-input block with a third length and a third width in the input matrix that should be multiplied by the sub-parameter block is used as the multiplier matrix. Matrix multiplication is performed sequentially and without repetition on the sub-parameter block and the sub-input block. The second length, second width, and third width are equal to the length, width, and multiplier matrix specified in the matrix multiplication instruction, respectively, and the second width is equal to the third width. This allows the matrix multiplication of the input matrix with the model parameters of the large model to be completed in a single dequantization operation on the plurality of parameter blocks. The quantization parameter matrix is ​​a two-dimensional matrix obtained by quantizing the model parameters. The first quantity is calculated based on the length and width of the quantization parameter matrix, the first length and first width of each parameter block, and the number of the plurality of computing units. The input matrix is ​​a two-dimensional matrix that serves as the input data for the decoding stage of the inference process of the large model.

2. The method according to claim 1, characterized in that, When the length of the input matrix is ​​equal to or less than the width of the multiplier matrix specified by the matrix multiplication operation instruction, the third length is equal to the length of the input matrix.

3. The method according to claim 1, characterized in that, The processor is a graphics processing unit (GPU), and the computing unit is a streaming multiprocessor. Each computing unit runs a thread block, in which matrix multiplication operations between the sub-parameter block and the sub-input block are performed sequentially for each parameter block in the dequantized assigned parameter block.

4. The method according to claim 3, characterized in that, The matrix multiplication instruction is a matrix multiplication and accumulation (MMA) instruction in the PTX instruction set executed by a parallel thread.

5. A processor comprising multiple computing units for performing matrix multiplication operations, characterized in that, The processor is configured to: The quantization parameter matrix is ​​divided into multiple parameter blocks, such that each parameter block has a preset first length and first width; The plurality of parameter blocks are sequentially and non-repeatingly assigned to each of the plurality of computing units, such that, except for at most one computing unit being assigned a consecutive number of parameter blocks less than a first number from the plurality of parameter blocks, each computing unit is assigned a consecutive first number of parameter blocks from the plurality of parameter blocks, and Each of the plurality of computing units is configured as follows: The assigned parameter blocks are dequantized, and the assigned parameter blocks are distributed across the entire length of the quantization parameter matrix; The input matrix is ​​obtained. Given that the length of the input matrix indicates the batch size of the input data, and the batch size is less than the length and width of the multiplicand matrix specified in the matrix multiplication instruction, and the width of the multiplier matrix specified in the matrix multiplication instruction is less than the length and width of both the multiplicand and multiplier matrices specified in the matrix multiplication instruction, for the dequantized assigned parameter block, the matrix multiplication instruction is used to take the sub-parameter block with a second length and a second width within the parameter block as the multiplicand matrix, and the sub-input block with a third length and a third width in the input matrix that should be multiplied by the sub-parameter block as the multiplier matrix. Matrix multiplication is then performed sequentially and without repetition on the sub-parameter block and the sub-input block. The second length, second width, and third width are equal to the length, width, and multiplier matrix specified in the matrix multiplication instruction, respectively, and the second width is equal to the third width. This allows the matrix multiplication of the input matrix with the model parameters of the large model to be completed in a single dequantization operation on the multiple parameter blocks. The quantization parameter matrix is ​​a two-dimensional matrix obtained by quantizing the model parameters. The first quantity is calculated based on the length and width of the quantization parameter matrix, the first length and first width of each parameter block, and the number of the plurality of computing units. The input matrix is ​​a two-dimensional matrix that serves as the input data for the decoding stage of the inference process of the large model.

6. An apparatus for performing matrix multiplication operations using a processor comprising multiple computing units, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to perform the following operations: The quantization parameter matrix is ​​divided into multiple parameter blocks, such that each parameter block has a preset first length and first width; The plurality of parameter blocks are sequentially and non-repeatedly assigned to each of the plurality of computing units included in the processor, such that each computing unit is assigned the first number of consecutive parameter blocks from the plurality of parameter blocks, except that at most one computing unit is assigned a number of consecutive parameter blocks less than a first number from the plurality of parameter blocks. Each of the plurality of computing units dequantizes an assigned block of parameters, the assigned block of parameters being distributed across the entire length of the quantization parameter matrix; Each of the plurality of computing units acquires an input matrix. When the length of the input matrix indicates the batch size of the input data, and the batch size is less than the length and width of the multiplicand matrix specified in the matrix multiplication instruction, and the width of the multiplier matrix specified in the matrix multiplication instruction is less than the length and width of both the multiplicand and multiplier matrices specified in the matrix multiplication instruction, for the dequantized assigned parameter block, the matrix multiplication instruction is used to take the sub-parameter block with a second length and a second width within the parameter block as the multiplicand matrix, and the sub-input block with a third length and a third width in the input matrix that should be multiplied by the sub-parameter block as the multiplier matrix. Matrix multiplication is then performed sequentially and without repetition on the sub-parameter block and the sub-input block. The second length, second width, and third width are equal to the length, width, and multiplier matrix specified in the matrix multiplication instruction, respectively, and the second width is equal to the third width. This allows the matrix multiplication of the input matrix with the model parameters of the large model to be completed in a single dequantization operation on the plurality of parameter blocks. The quantization parameter matrix is ​​a two-dimensional matrix obtained by quantizing the model parameters. The first quantity is calculated based on the length and width of the quantization parameter matrix, the first length and first width of each parameter block, and the number of the plurality of computing units. The input matrix is ​​a two-dimensional matrix that serves as the input data for the decoding stage of the inference process of the large model.

7. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.

Images