Matrix computing apparatus, method, electronic device and storage medium

Abstract

This disclosure provides a matrix computing device, method, electronic device, and storage medium. The matrix computing device, used in an artificial intelligence processor, includes at least one dequantization calculation module and a dequantization accumulation module. Each dequantization calculation module includes at least one processing unit and an accumulation unit. This matrix computing device, through two-stage dequantization operations, can reduce overall accuracy loss and achieve seamless integration of the computational pipeline, thereby improving overall computational throughput.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of artificial intelligence processors, and more specifically to a matrix computing device, a matrix computing method, an electronic device, and a storage medium. Background Technology

[0002] Floating-point dequantization refers to converting low-precision floating-point formats (such as FP4, FP8, etc.) back to high-precision floating-point formats (such as FP16, BF16, FP32, etc.). High-precision floating-point formats can significantly improve the granularity of data representation, especially in the post-processing and output layers of artificial intelligence models during inference. However, directly using low-precision data for final calculations can affect the accuracy of the results, thus limiting the model's performance on high-resolution tasks such as generating text and images. Therefore, in the high-precision output process of artificial intelligence (AI) models, the dequantization step typically utilizes previously stored scaling factors or corresponding inverse transformations to restore low-precision data to the high-precision representation range, thereby maximizing the recovery of the model's original expressive power and output quality. Summary of the Invention

[0003] This disclosure provides at least one embodiment of a matrix computing device for an artificial intelligence processor. The matrix computing device includes: at least one dequantization calculation module and a dequantization accumulation module; wherein each of the dequantization calculation modules includes at least one processing unit and an accumulation unit, wherein the at least one processing unit is configured to receive at least one first sub-matrix and at least one second sub-matrix for calculation, and perform data processing on the at least one first sub-matrix and at least one second sub-matrix to obtain at least one target matrix; the accumulation unit is configured to receive at least one first sub-dequantization result, and accumulate the at least one first sub-dequantization result to obtain a first dequantization result; wherein the at least one first sub-dequantization result is obtained by performing a first dequantization process based on a combination of the at least one target matrix and at least one first scaling factor; and the dequantization accumulation module is configured to receive at least one second sub-dequantization result, and accumulate the at least one second sub-dequantization result to obtain a second dequantization result; wherein the at least one second sub-dequantization result is obtained by performing a second dequantization process based on a combination of at least one first dequantization result and at least one second scaling factor.

[0004] For example, in at least one embodiment of this disclosure, a matrix calculation apparatus is provided, wherein the at least one inverse quantization calculation module includes... The inverse quantization calculation module, the at least one processing unit includes... The processing units, wherein... and The integer is positive; wherein the at least one first submatrix and at least one second submatrix are obtained by dividing the first matrix A and the second matrix B into T first submatrixes based on a preset data path width. and T second submatrices ,in, Each of the first submatrices and each of the second submatrices Has the corresponding first index ,in, Integer and ≤t≤T.

[0005] For example, in at least one embodiment of this disclosure, a matrix calculation apparatus is provided, wherein the at least one first submatrix and the at least one second submatrix are further obtained by: dividing the first matrix A into T first submatrixes along the column dimensions of the first matrix based on the preset data path width. Based on the preset data path width, the second matrix B is divided into T second sub-matrices along the row dimensions of the second matrix. .

[0006] For example, in at least one embodiment of this disclosure, a matrix calculation apparatus is provided, wherein the at least one processing unit is configured to perform calculations according to the first index. For the first submatrix and the second submatrix Perform matrix multiplication and accumulation operations The target matrix is ​​obtained. .

[0007] For example, in at least one embodiment of this disclosure, a matrix calculation apparatus is provided, wherein the first submatrix and the second submatrix Based on the first index t, each has a corresponding first sub-scaling factor. and the corresponding second sub-scaling factor Wherein, the first sub-scaling factor and the second sub-scaling factor The first floating-point precision is defined; wherein the first scaling factor combination is obtained by means of the first index. The first sub-scaling factor and the second sub-scaling factor Perform outer product operation The first scaling factor combination is obtained .

[0008] For example, in at least one embodiment of this disclosure, a matrix computing apparatus is provided in which the at least one first sub-inverse quantization result includes T first sub-inverse quantization results. The at least one inverse quantization calculation module is further configured to: calculate the... The first sub-inverse quantization result Perform cumulative calculation get The first inverse quantization result , where each of the first inverse quantization results Associated with the second index k, where k is even and .

[0009] For example, in at least one embodiment of this disclosure, a matrix computing device is provided, wherein the T first sub-inverse quantization results This includes obtaining it through the following methods: based on the first index. For each of the first scaling factor combinations and each of the target matrices Perform element-wise multiplication Obtain the The first sub-inverse quantization result .

[0010] For example, in at least one embodiment of this disclosure, a matrix calculation device is provided, wherein each of the accumulation units is further configured to: for the τ first sub-inverse quantization results corresponding to the first index t = (k-1)τ + 1, (k-1)τ + 2, …, kτ respectively Perform an accumulation operation to obtain the first inverse quantization result corresponding to the accumulation unit. .

[0011] For example, in at least one embodiment of this disclosure, a matrix calculation device is provided, wherein the first submatrix and the second submatrix have corresponding third sub-scaling factors and corresponding fourth sub-scaling factors, respectively; wherein the second scaling factor combination is obtained by performing an outer product operation on the third sub-scaling factor and the fourth sub-scaling factor.

[0012] For example, in at least one embodiment of this disclosure, a matrix calculation apparatus is provided in which, when the third sub-scaling factor and the fourth sub-scaling factor have a second floating-point precision, the at least one combination of second scaling factors includes j combinations of second scaling factors. , where each of the second scaling factor combinations Associated with the second index k; wherein the second scaling factor combination is obtained by: based on the second index k, the third sub-scaling factor and the fourth sub-scaling factor Perform outer product operation The second scaling factor combination is obtained. .

[0013] For example, in at least one embodiment of this disclosure, a matrix computing apparatus is provided in which the at least one second sub-inverse quantization result includes j second sub-inverse quantization results. ; wherein, the j second sub-inverse quantization results This includes obtaining the first inverse quantization result based on the second index k in the following way. Combined with the corresponding second scaling factor Perform element-wise multiplication to obtain the j second sub-inverse quantization results. .

[0014] For example, in at least one embodiment of this disclosure, a matrix calculation apparatus is provided in which, when the third sub-scaling factor and the fourth sub-scaling factor have a third floating-point precision, the at least one combination of second scaling factors includes j / 2 combinations of second scaling factors. , where each of the second scaling factor combinations Associated with the second index k; wherein the second scaling factor combination is obtained by: based on the second index k, the third sub-scaling factor and the fourth sub-scaling factor Perform outer product operation The second scaling factor combination is obtained. .

[0015] For example, in at least one embodiment of this disclosure, a matrix computing apparatus is provided in which the at least one second sub-inverse quantization result includes j / 2 second sub-inverse quantization results. ; wherein, the j / 2 second sub-inverse quantization results This includes obtaining it by combining the second scaling factor. Compared with the first inverse quantization result And the first inverse quantization result Perform element-wise multiplication operations separately and add the results together to obtain j / 2 second sub-inverse quantization results. .

[0016] For example, in at least one embodiment of this disclosure, a matrix calculation device is provided, which further includes a bias processing module configured to add a bias matrix to the second inverse quantization result to obtain an output matrix.

[0017] For example, in at least one embodiment of this disclosure, a matrix computing device is provided, wherein the artificial intelligence processor includes a plurality of computing units, each computing unit including a tensor core, wherein the matrix computing device is disposed in the tensor core.

[0018] This disclosure provides at least one embodiment of a matrix calculation method applied to an artificial intelligence processor. The matrix calculation method includes: receiving at least one first sub-matrix and at least one second sub-matrix for calculation; performing data processing on the at least one first sub-matrix and the at least one second sub-matrix to obtain at least one target matrix; receiving at least one first sub-inverse quantization result; accumulating the at least one first sub-inverse quantization result to obtain a first inverse quantization result; wherein the at least one first sub-inverse quantization result is obtained by performing a first inverse quantization process based on a combination of the at least one target matrix and at least one first scaling factor; and receiving at least one second sub-inverse quantization result; accumulating the at least one second sub-inverse quantization result to obtain a second inverse quantization result; wherein the at least one second sub-inverse quantization result is obtained by performing a second inverse quantization process based on a combination of at least one first inverse quantization result and at least one second scaling factor.

[0019] At least one embodiment of this disclosure provides an electronic device, including: a memory that non-transitoryly stores computer-executable instructions; and a processor configured to run the computer-executable instructions, wherein the computer-executable instructions are executed by the processor to implement the matrix calculation method provided in the at least one embodiment described above.

[0020] At least one embodiment of this disclosure provides a non-transitory computer-readable storage medium storing computer-readable instructions thereon, wherein the computer-readable instructions, when executed by at least one processor, perform the matrix calculation method provided in the at least one embodiment described above.

[0021] In at least one embodiment of this disclosure, the two-stage dequantization operation is integrated within the matrix computing device. Dynamic dequantization and accumulation are performed within the device, avoiding the latency caused by external storage and computation of high-precision (e.g., BF16 or FP32) scaling factors. This allows for direct high-precision dequantization (e.g., BF16 or FP32) within the device, achieving seamless integration of the computational pipeline and significantly improving overall computational throughput. Furthermore, by decomposing the computational task into fine-grained sub-matrix processing and performing two-stage dequantization, the processing unit can match the dynamic range of each local data block at a fine-grained level, thereby effectively reducing accuracy loss caused by low-precision computation at the system level while maintaining performance advantages. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0023] Figure 1 This is a schematic diagram of the architecture of a general-purpose graphics processing unit (GPGPU).

[0024] Figure 2 A schematic structural diagram of a matrix computing device provided in at least one embodiment of this disclosure;

[0025] Figure 3A A schematic diagram of a matrix computing device provided in at least one embodiment of this disclosure;

[0026] Figure 3B This is a schematic diagram of the operation of a matrix computing device provided in at least one embodiment of the present disclosure;

[0027] Figure 4 A schematic flowchart illustrating a matrix calculation method provided in at least one embodiment of this disclosure;

[0028] Figure 5 A schematic structural diagram of an artificial intelligence processor provided in at least one embodiment of this disclosure;

[0029] Figure 6 A schematic block diagram of an electronic device provided for at least one embodiment of this disclosure;

[0030] Figure 7 A schematic block diagram of another electronic device provided for at least one embodiment of this disclosure;

[0031] Figure 8 This is a schematic block diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0033] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described object changes. To keep the following description of the embodiments of this disclosure clear and concise, detailed descriptions of some known functions and components are omitted.

[0034] Floating-point numbers (FP) are primarily used to represent decimals and typically consist of three parts: a sign bit, an exponent, and a mantissa. The exponent part can also be called the exponent code. For example, a floating-point number V can usually be represented in the following form:

[0035]

[0036] In this context, the sign bit s can be 1 bit, determining whether the floating-point number V is negative or positive; M represents the mantissa, which can include multiple bits and is a binary fractional form, defining the precision of the floating-point number; E represents the exponent (also called the exponent value), used to weight the floating-point number, reflecting the position of the decimal point in the floating-point number V, and defining the range of values ​​for the floating-point number.

[0037] Traditional high-precision floating-point numbers typically include three formats: half-precision floating-point numbers (FP16), single-precision floating-point numbers (FP32), and double-precision floating-point numbers (FP64), with different numbers of bits in their exponent and mantissa parts.

[0038] AI accelerators are widely used for training deep learning models. For convolution operations, common in deep learning models, both hardware and software designs have been specifically optimized to accelerate computation. For example, various floating-point data formats have been developed and optimized for fields such as artificial intelligence and deep learning, including BF16 (brain floating point 16, 16-bit width), BF24 (brain floating point 24, 24-bit width), and TF32 (Tensor Float 32, 19-bit width). These data formats can significantly reduce computational processing, especially the computational resources and power consumption required for matrix multiplication or convolution multiplication operations. In addition, the processor also supports some common floating-point types, such as half-precision floating-point numbers (FP16, 16-bit width) or single-precision floating-point numbers (FP32, 32-bit width).

[0039] Low-precision matrix multiplication is increasingly being used in the training and inference of large AI models due to its significant performance gains with acceptable precision loss. In GPUs (Graphics Processing Units) or GPGPUs, matrix multiplication is typically performed in hardware by tensor kernels. Low-precision tensor kernels are several times more computationally powerful than high-precision tensor kernels, resulting in higher computational efficiency. Furthermore, the data volume of a low-precision tensor is also only a few times that of a high-precision tensor, leading to higher data transfer efficiency. Therefore, the end-to-end efficiency improvement brought by low-precision tensor computation is almost several times greater. For example, the computational power of an FP4 tensor kernel can be 2 to 8 times that of FP8, 4 to 16 times that of FP16 / BF16, or even higher. Moreover, the data volume is half that of FP8 and one-quarter that of FP16 / BF16.

[0040] To reduce the loss of precision, low-precision matrix multiplication introduces a scaling factor to maximize the numerical expressive power of low-precision tensors.

[0041] For low-precision matrix multiplication, such as D = A * B + C, where A, B, C, and D are all high-precision tensors, the general matrix multiplication operation using a scaling factor can be described as follows:

[0042] D = (α⊙A') × (β⊙B') + σ⊙C' or D = (α⊙A') × (β⊙B') + σ⊙C' β)⊙(A'×B')+σ⊙C';

[0043] D→D',γ;

[0044] Where ⊙ represents element-wise multiplication. σ represents the outer product, × represents matrix multiplication, α is the scaling factor for tensor A, β is the scaling factor for tensor B, σ is the scaling factor for tensor C, and γ is the scaling factor for tensor D. Quantized tensors A', B', C', and D' are the low-precision tensors obtained by floating-point quantization of tensors A, B, C, and D, respectively. For example, the data dimensions of each parameter are shown below:

[0045] A: [m,s*k], B: [s*k,n], C: [m,n], D: [m,n], α: [m,s], β: [s,n], σ: [m,r] or [r,n], γ: [m,r] or [r,n];

[0046] Where m, n, s, k, and r are all positive integers. For example, for any row in tensor A, every k tensor elements in that row share a single scaling factor parameter; for any column in tensor B, every k tensor elements in that column share a single scaling factor parameter.

[0047] In low-precision matrix multiplication, dequantization is the operation of mapping the low-precision calculation result to the high-precision representation domain.

[0048] Specifically, dequantization can be achieved through two mathematical forms:

[0049] In the formula D = (α ⊙ A') × (β ⊙ B') + σ ⊙ C', dequantization occurs before matrix multiplication. The scaling factors α and β are first isoquantized tensors A' and B' respectively and multiplied element-wise. This operation immediately "restores" each low-precision tensor element to a high-precision intermediate value. Then, high-precision matrix multiplication and bias addition are performed to directly generate the high-precision output D.

[0050] In the formula D = (α) In the matrix multiplication β) ⊙ (A' × B') + σ ⊙ C', dequantization occurs after matrix multiplication. First, a low-precision matrix multiplication unit (such as a Tensor Core) is used to calculate M' = A' × B', obtaining a low-precision intermediate result; then, the cross product of the high-precision scaling factor α and β (α) is used. β) Generate a scaling matrix of the same dimension and multiply it element-wise with M'. This step is equivalent to dequantizing and correcting each element of the intermediate result. Finally, add it to the dequantized bias term σ ⊙ C' to obtain D.

[0051] For the bias term C, its dequantization is directly performed by σ ⊙ C', where σ is the scaling factor of the bias tensor and C' is its quantized version. The result is used in the summation in a high-precision form.

[0052] Figure 1 This is a schematic diagram of a general-purpose graphics processing unit (GPGPU).

[0053] like Figure 1 As shown, a general-purpose graphics processor is actually an array of programmable multiprocessors. For example, a programmable multiprocessor can be a streaming processor cluster (SPC), such as including... Figure 1 The diagram shows streaming processor clusters 1, ..., M, where M is a positive integer greater than 1. In a general-purpose graphics processor, one streaming processor cluster handles one computational task, or multiple streaming processor clusters handle one computational task. Multiple streaming processor clusters share data through a global cache or global memory.

[0054] like Figure 1 As shown, taking streaming processor cluster 1 as an example, one streaming processor cluster includes multiple computing units, such as... Figure 1 The system consists of Compute Unit 1, Compute Unit 2, ..., Compute Unit N, where N is a positive integer. Each Compute Unit (CU) is used to perform arithmetic and logical operations, such as accumulation, reduction, and regular addition, subtraction, multiplication, and division.

[0055] A computing unit includes multiple cores (also called computing kernels or computing cores). Figure 1 (Not shown in the image), each computing core includes an arithmetic logic unit (ALU), a floating-point unit (FPU), etc., and is used to perform specific computing tasks.

[0056] like Figure 1 As shown, each computational unit also provides a tensor core for performing tensor-related computations. For example, this tensor core can be a GEMM core, used for matrix multiplication operations using the GEMM operator. Tensors are a very important data structure in deep learning; they are high-dimensional generalizations of scalars, vectors, and matrices. Tensor operations are commonly used in the training and inference of current deep learning models, and tensor cores can accelerate matrix multiplication operations. Tensor cores in multiple computational units can be uniformly scheduled and controlled.

[0057] In addition, the computation unit includes a register file, shared memory, and a tensor core memory unit for storing source and destination data related to computation tasks. Shared memory within a computation unit is used to share data between cores within that unit. The tensor core memory unit is a storage resource closely related to the tensor core, used to store intermediate data when the tensor core performs tensor operations (such as matrix multiplication). Furthermore, it can perform data format processing on the tensor data to be used in tensor operations, ensuring that data loaded from external sources conforms to the tensor core's data format requirements.

[0058] In parallel computing, computational tasks are typically executed by multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block distribution module. Figure 1 (Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0059] In each computing unit, the thread beam scheduling / distribution module ( Figure 1 (Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple computing cores within the computing unit can run thread bundles. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can be executed concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory execution instructions are issued to shared memory within the computing unit or further issued to intermediate-level caches, global caches, or global memory for read and write operations, etc.

[0060] In current low-precision computing, the extremely low-precision data formats used for core computation (such as FP4 E2M1) and the medium-precision factor formats used for local scaling (such as FP8 E4m3) have numerical representation ranges much smaller than the dynamic range of the original high-precision data (such as BF16 or FP32). For example, the exponent E8 of FP32 can represent values ​​as high as approximately 3.4 * 10^34. In order to restore the computation results to the original high-precision range during dequantization, it is currently necessary to introduce a high-precision numerical format with a full E8 range (such as FP32) as a global scaling factor outside the matrix multiplication kernel (GEMMCORE), and perform the final dequantization computation on the Vector Core to obtain the original high-precision data.

[0061] The inventors of this disclosure have noted that the above method has at least the following limitations: First, the computational efficiency of dequantization to high precision is low. Since the matrix multiplication kernel (GEMM CORE) typically only supports operations up to FP8 or lower precision, it cannot directly handle scaling factors of FP32 precision. Therefore, the high-precision global scaling factor must be stored outside the GEMM CORE, and the high-precision dequantization operation must be performed using a relatively general-purpose Vector Core. This disrupts the continuity of the computational pipeline, requiring data to be moved and converted back and forth between different precision domains and computational units, introducing additional overhead and becoming a bottleneck for system performance. Second, the precision loss during the dequantization process is severe. Global dequantization implemented on the Vector Core typically adopts a per-tensor coarse-grained quantization strategy, where the entire tensor shares the same global scaling factor. This global dequantization approach cannot adapt to the differences in data distribution in different regions within the tensor: in regions with a large dynamic range, the scaling factor is forced to be amplified to cover the maximum value, resulting in a decrease in quantization resolution in small numerical regions, and a large amount of information is lost during the quantization-dequantization process; and vice versa.

[0062] In summary, in order to cover the dynamic range of high-precision data, the above methods are forced to use an external high-precision global scaling factor, which directly leads to efficiency loss due to the break in the calculation process, as well as the inherent accuracy loss of the coarse-grained metric strategy.

[0063] At least one embodiment of this disclosure provides a matrix calculation apparatus, comprising: at least one inverse quantization calculation module and an inverse quantization accumulation module. Each inverse quantization calculation module includes at least one processing unit and an accumulation unit. The at least one processing unit is configured to receive at least one first sub-matrix and at least one second sub-matrix for calculation, and perform data processing on the at least one first sub-matrix and at least one second sub-matrix to obtain at least one target matrix. The accumulation unit is configured to receive at least one first sub-inverse quantization result, and accumulate the at least one first sub-inverse quantization result to obtain a first inverse quantization result; wherein the at least one first sub-inverse quantization result is obtained by performing a first inverse quantization process based on a combination of at least one target matrix and at least one first scaling factor. The inverse quantization accumulation module is configured to receive at least one second sub-inverse quantization result, and accumulate the at least one second sub-inverse quantization result to obtain a second inverse quantization result; wherein the at least one second sub-inverse quantization result is obtained by performing a second inverse quantization process based on a combination of at least one first inverse quantization result and at least one second scaling factor.

[0064] For example, in at least one embodiment, the two-stage dequantization operation is integrated within the matrix computing device. Dynamic dequantization and accumulation are performed within the device, avoiding the latency caused by external storage and computation of high-precision (e.g., BF16 or FP32) scaling factors. This allows for direct high-precision dequantization (e.g., BF16 or FP32) within the device, achieving seamless integration of the computational pipeline and significantly improving overall computational throughput. For instance, by decomposing the computational task into fine-grained sub-matrix processing and performing two-stage dequantization, the processing unit can match the dynamic range of each local data block at a fine-grained level, thereby effectively reducing accuracy loss caused by low-precision computation at the system level while maintaining performance advantages.

[0065] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, but this disclosure is not limited to these specific embodiments.

[0066] Figure 2 A schematic structural diagram of a matrix computing device provided for at least one embodiment of the present disclosure.

[0067] For example, this matrix computing device can be applied to processors such as artificial intelligence processors. The processor can be any one of the following: Graphics Processing Unit (GPU), Tensor Processing Unit (TPU), Neural Network Processing Unit (NPU), Deep Learning Processing Unit (DPU), Accelerated Processing Unit (APU), and General-Purpose Graphics Processing Unit (GPGPU).

[0068] For example, in this disclosure, the matrix computing device is described using a graphics processor or a general-purpose graphics processor as an example, but of course, this disclosure is not limited thereto.

[0069] For example, such as Figure 2 As shown, the matrix computing device 100 includes at least one inverse quantization computing module 110 and an inverse quantization accumulation module 120.

[0070] For example, in each dequantization calculation module 110, each processing unit 111 may be connected to an accumulation unit 112. For example, each dequantization calculation module 110 may be connected to a dequantization accumulation module 120, and for example, each accumulation unit 112 may be connected to a dequantization accumulation module 120.

[0071] For example, processing unit 111 can be a dedicated matrix computation core, such as a systolic array or tensor computation core. For example, processing unit 111 can be a two-dimensional array of processing units (e.g., 8x8, 16x16, 32x32, etc.). For example, each processing unit 111 can include multipliers and adders for performing multiply-accumulate operations. For example, processing unit 111 can receive dynamically quantized low-precision data blocks (e.g., FP4 format) from a previous level cache (e.g., shared memory), such as a first submatrix. With the second submatrix The amount of data fed into the array each clock cycle is determined by a preset data path width. For example, in each processing unit 111, the first submatrix receives... The elements and the second submatrix Element-wise multiplication is performed, and the results are accumulated in the partial sum register of processing unit 111. For example, after several clock cycles, the entire data block is calculated, and the target matrix can be stored in the partial sum register. Target matrix They are collected in parallel into the output register.

[0072] For example, the accumulation unit 112 and the dequantization accumulation module 120 can be a multi-level, parallelized tree-structured adder network. For example, the accumulation unit 112 can receive at least one first sub-dequantization result. For example, the dequantization accumulation module 120 can receive at least one second sub-dequantization result. For example, at least one first sub-dequantization result and at least one second sub-dequantization result can be temporarily stored in a set of high-speed registers or shared memory. For example, the accumulation unit 112 and the dequantization accumulation module 120 can include a first-level adder and intermediate-level adders. For example, the first-level adder can perform parallel element-by-element addition of corresponding elements of two adjacent first or second sub-dequantization results. For example, for a matrix of [m, n], m*n adders can operate simultaneously, each processing one matrix position. For example, the intermediate-level adder can continue to feed the result of the first-level addition (half the number) into the next-level adder array for the same parallel element-by-element addition. This process is repeated in the form of a binary tree. For example, for a case where there are T partial results that need to be accumulated, the number of levels of the tree adder is log2(T) (rounded up).

[0073] For example, such as Figure 2 As shown, each inverse quantization calculation module 110 may include at least one processing unit 111 and an accumulation unit 112. For example, at least one processing unit 111 may be configured to receive at least one first submatrix and at least one second submatrix for calculation, and perform data processing on the at least one first submatrix and at least one second submatrix to obtain at least one target matrix.

[0074] For example, such as Figure 2 As shown, the accumulation unit 112 can be configured to receive at least one first sub-inverse quantization result, accumulate the at least one first sub-inverse quantization result, and obtain a first inverse quantization result. For example, the at least one first sub-inverse quantization result is obtained by performing a first inverse quantization process based on a combination of at least one target matrix and at least one first scaling factor.

[0075] For example, such as Figure 2 As shown, the inverse quantization accumulation module 120 can be configured to receive at least one second sub-inverse quantization result, accumulate the at least one second sub-inverse quantization result, and obtain a second inverse quantization result. For example, at least one second sub-inverse quantization result can be obtained by performing a second inverse quantization process based on at least one first inverse quantization result and at least one second scaling factor combination.

[0076] For example, accumulation unit 112 can be configured to perform parallel tree-like accumulation on at least one first sub-inverse quantization result. For example, inverse quantization accumulation module 120 can be configured to perform parallel tree-like accumulation on at least one second sub-inverse quantization result.

[0077] For example, dequantization processing includes first dequantization and second dequantization. Dequantization is an operation that maps low-precision calculation results to a high-precision representation domain. For example, first dequantization can be performed by element-wise multiplication of the target matrix and the corresponding first scaling factor combination to obtain the corresponding first sub-dequantization result. For example, second dequantization can be performed by element-wise multiplication of the first dequantization result and the corresponding second scaling factor combination to obtain the corresponding second sub-dequantization result.

[0078] For example, the floating-point precision of the first sub-dequantization result and the first dequantization result is lower than that of the second sub-dequantization result and the second quantization result. For example, the floating-point precision of the first sub-dequantization result and the first dequantization result is the same, which can be the first floating-point precision. For example, the first floating-point precision can be FP8. For example, the floating-point precision of the second sub-dequantization result and the second dequantization result is the same, which can be the second floating-point precision or the third floating-point precision. For example, the second floating-point precision can be BF16 or FP16, and the third floating-point precision can be FP32.

[0079] For example, in at least one embodiment, the artificial intelligence processor may include multiple computing units, each computing unit may include a tensor kernel. For example, the matrix computing device 100 may be disposed in the tensor kernel. For example, the tensor kernel may be... Figure 1The tensor kernel in the matrix can be a GEMM core, used to perform matrix multiplication operations using GEMM operators. For example, each tensor kernel can handle calculations of various floating-point formats with different precisions. For instance, the matrix computing device 100 can be located in the hardware path within the tensor kernel that handles calculations of low-precision formats (e.g., FP4 precision).

[0080] In at least one embodiment, the matrix calculation device provided in this disclosure can be integrated inside the tensor kernel, enabling two-level dequantization and accumulation to be performed within the tensor kernel. This avoids the latency caused by storing and calculating high-precision (e.g., BF16 or FP32) scaling factors outside the tensor kernel.

[0081] For example, the first and second dequantization processes described above can be executed by an arithmetic logic unit, which can be located within a tensor kernel. For example, the arithmetic logic unit can be located within the matrix computing device 100. For example, the arithmetic logic unit implementing the first dequantization process can be located in the dequantization calculation module 110, and the arithmetic logic unit implementing the second dequantization process can be located in the dequantization accumulation module 120. For example, the arithmetic logic unit can be located in the hardware path within the tensor kernel responsible for calculating low-precision formats (e.g., FP4 precision). Alternatively, the arithmetic logic unit can be located outside the matrix computing device 100, for example, in the hardware path within the tensor kernel responsible for calculating other precision formats; that is, the arithmetic logic unit can reuse existing hardware resources. This disclosure does not limit the arithmetic logic unit.

[0082] For example, in at least one embodiment, at least one inverse quantization calculation module may include A dequantization calculation module, at least one processing unit may include a dequantization calculation module, and at least one processing unit may include a dequantization calculation module. There are 10 processing units, among which... and It is a positive integer. For example, such as Figure 2 As shown, the matrix computing device 100 may include Each inverse quantization calculation module 110 may include an inverse quantization calculation module 110. One processing unit 111. For example, such as Figure 2 As shown, the matrix computing device 100 may include 111 processing units.

[0083] For example, in at least one embodiment, at least one first submatrix and at least one second submatrix can be obtained by dividing the first matrix A and the second matrix B into T first submatrixes based on a preset data path width. and T second submatrices ,in, Each first submatrix and each second submatrix Has the corresponding first index ,in, Integer and ≤t≤T. For example, in at least one embodiment, at least one first submatrix and at least one second submatrix may further include being obtained by dividing the first matrix A into T first submatrixes along the column dimensions of the first matrix based on a preset data path width. Based on the preset data path width, the second matrix B is divided into T second sub-matrices along the row dimensions of the second matrix. .

[0084] For example, the first matrix A and the second matrix B can be processed in blocks within the matrix computing device 100 to obtain the first submatrix and the second submatrix. Alternatively, the matrix computing device 100 can directly receive the first submatrix and the second submatrix. In this configuration, the block operation can be performed by dedicated hardware or a programmable unit outside the matrix computing device 100. For example, the floating-point precision of the first matrix A and the second matrix B can be lower than the first sub-inverse quantization result and the first inverse quantization result; for example, it can be a low-precision format, such as FP4. For example, assuming the preset data path is 16, the dimension of the first matrix A is [m, T*16], and the dimension of matrix B is [T*16, n], the first matrix A and the second matrix B can be divided into T first submatrices respectively. and T second submatrices Each The dimension is [m, 16], each The dimensions are [16, n]. For example, the first and second submatrices have the same floating-point precision as the first and second matrices, and can be in a low-precision format, such as FP4.

[0085] For example, in at least one embodiment, at least one processing unit may be configured to follow a first index. For the first submatrix Second submatrix Perform matrix multiplication and accumulation operations Obtain the target matrix For example, the target matrix There can be T targets, each with a target matrix. The dimension is [m, n]. For example... Figure 2 As shown, each processing unit 111 can process the corresponding first submatrix Second submatrix Perform matrix multiplication and accumulation operations to obtain the corresponding target matrix. .

[0086] For example, processing unit 111 can be a dedicated hardware unit for performing matrix multiplication on such a preset data path. For example, when the preset data path is 16, processing unit 111 can be a dedicated hardware unit for performing matrix multiplication with dimensions [m, 16] and [16, n]. Since the target matrix has m rows and n columns, and a total of m * n elements, the total number of multiplications performed by the processing unit in each data processing operation can be m × n × 16.

[0087] In at least one embodiment, the processing unit provided in this disclosure accelerates core computing tasks through hardware-level specialization and fixed-function design. By decoupling large matrix multiplication tasks into regular small data block operations, the same multiple processing units can process different data blocks in parallel, greatly expanding the overall parallel computing capability of the system.

[0088] For example, in at least one embodiment, the first submatrix Second submatrix Each index t can have a corresponding first sub-scaling factor. and the corresponding second sub-scaling factor For example, the first sub-scaling factor. Second sub-scaling factor This can be a first floating-point precision, such as FP8. For example, the first scaling factor combination can be obtained based on the first index. , the first sub-scaling factor Second sub-scaling factor Perform outer product operation Obtain the first scaling factor combination .

[0089] For example, the corresponding first sub-scaling factor and the corresponding second sub-scaling factor With the first submatrix Second submatrix They have the same first index t, and are thus stored in memory in a sequential or addressable manner. For example, when the first sub-scaling factor... The dimension is [m, 1], and the second sub-scaling factor When the dimension is [1, n], the first scaling factor combination after performing the outer product operation. The dimension is [m,n].

[0090] For example, the operation of obtaining the first scaling factor combination described above can be performed by an arithmetic logic unit, which can be located within the tensor kernel. For example, this arithmetic logic unit can be located within the matrix computing device 100; for instance, the arithmetic logic unit for calculating the first scaling factor combination can be located within the dequantization calculation module. For instance, this arithmetic logic unit can be located in the hardware path within the tensor kernel responsible for calculating low-precision formats (e.g., FP4 precision). Alternatively, this arithmetic logic unit can also be located outside the matrix computing device 100, for example, in the hardware path within the tensor kernel responsible for calculating other precision formats; that is, the arithmetic logic unit can reuse existing hardware resources. This disclosure does not limit its scope.

[0091] For example, in at least one embodiment, at least one first sub-inverse quantization result may include T first sub-inverse quantization results. At least one inverse quantization calculation module 110 can be further configured to: The first sub-inverse quantization result Perform cumulative calculation get The first inverse quantization result For example, each first inverse quantization result It can be associated with the second index k, where k is even and For example, each first sub-inverse quantization result can correspond to each processing unit.

[0092] For example, in at least one embodiment, each accumulation unit 112 can be further configured to: for the τ first sub-inverse quantization results corresponding to the first index t = (k-1)τ + 1, (k-1)τ + 2, …, kτ respectively Perform the accumulation operation to obtain the first dequantization result corresponding to the accumulation unit. For example, each first dequantization result can correspond to an accumulation unit.

[0093] For example, the accumulation unit can perform a parallel tree-structured accumulation operation on at least one first sub-dequantization result to obtain the first dequantization result. For instance, the accumulation unit can first group the τ first sub-dequantization results into pairs, and simultaneously add the two first sub-dequantization results within each group, producing approximately τ / 2 first-level partial sums; then, the first-level partial sums can be further grouped into pairs and simultaneously added, producing even fewer second-level partial sums. This process is repeated, halving the number of operands at each level, until finally merged into a single sum, which is the first dequantization result. .

[0094] For example, in at least one embodiment, T first sub-inverse quantization results This can be obtained through the following methods: based on the first index For each first scaling factor combination and each target matrix Perform element-wise multiplication get The first sub-inverse quantization result For example, when the first scaling factor combination and target matrix Given a dimension of [m, n], after performing element-wise multiplication... The dimension is [m,n].

[0095] For example, in at least one embodiment, the first submatrix and the second submatrix may each have a corresponding third sub-scaling factor and a corresponding fourth sub-scaling factor. For example, the second scaling factor combination may be obtained by performing an outer product operation on the third and fourth sub-scaling factors.

[0096] For example, the operation to obtain the second scaling factor combination described above can be performed by an arithmetic logic unit, which can be located within the tensor kernel. For example, this arithmetic logic unit can be located within the matrix computing device 100; for instance, the arithmetic logic unit for calculating the first scaling factor combination can be located within the inverse quantization accumulation module. For instance, this arithmetic logic unit can be located in the hardware path within the tensor kernel responsible for calculating low-precision formats (e.g., FP4 precision). Alternatively, this arithmetic logic unit can also be located outside the matrix computing device 100, for example, in the hardware path within the tensor kernel responsible for calculating other precision formats; that is, the arithmetic logic unit can reuse existing hardware resources. This disclosure does not limit its scope.

[0097] For example, in at least one embodiment, when the third sub-scaling factor and the fourth sub-scaling factor are of second floating-point precision, at least one combination of second scaling factors may include j combinations of second scaling factors. For example, each second scaling factor combination Associated with the second index k. For example, the second scaling factor combination includes obtaining the third sub-scaling factor based on the second index k. and the fourth sub-scaling factor Perform outer product operation Obtain the second scaling factor combination For example, the second floating-point precision can be BF16 or FP16. For example, each combination of second scaling factors can correspond to each first dequantization result. For example, when the third sub-scaling factor... The dimension is [m, 1 / τ], and the fourth sub-scaling factor is... When the dimension is [1 / τ,n], the combination of the second scaling factors after performing the outer product operation. The dimension is [m,n].

[0098] For example, in at least one embodiment, at least one second sub-inverse quantization result may include j second sub-inverse quantization results. For example, j second sub-inverse quantization results This can be obtained by: based on the second index k, the first inverse quantization result. Combined with the corresponding second scaling factor Perform element-wise multiplication to obtain j second sub-inverse quantization results. For example, each second sub-inverse quantization result can correspond to the first inverse quantization result obtained by each inverse quantization calculation module. For example, when the second scaling factor is combined... And the first inverse quantization result Given a dimension of [m, n], after performing element-wise multiplication... The dimension is [m,n].

[0099] For example, in at least one embodiment, when the third sub-scaling factor and the fourth sub-scaling factor are of third floating-point precision, at least one combination of second scaling factors may include j / 2 combinations of second scaling factors. For example, each second scaling factor combination It can be associated with a second index k. For example, the second scaling factor combination can include obtaining it by: based on the second index k, the third sub-scaling factor. and the fourth sub-scaling factor Perform outer product operation Obtain the second scaling factor combination For example, the third floating-point precision can be FP32. For example, each combination of second scaling factors can correspond to two first dequantization results. For example, when the third sub-scaling factor... The dimension is [m, 1 / 2τ], and the fourth sub-scaling factor is... When the dimension is [1 / 2τ,n], the combination of the second scaling factors after performing the outer product operation. The dimension is [m,n].

[0100] For example, in at least one embodiment, at least one second sub-inverse quantization result may include j / 2 second sub-inverse quantization results. For example, j / 2 second sub-inverse quantization results This can be obtained by combining the second scaling factor. Compared with the first inverse quantization result And the first inverse quantization result Perform element-wise multiplication operations separately and add the results together to obtain j / 2 second sub-inverse quantization results. For example, each second sub-inverse quantization result can correspond to the first inverse quantization result obtained by the two inverse quantization calculation modules respectively. For example, each second sub-inverse quantization result can be generated by merging two first inverse quantization results.

[0101] For example, the third index p can also be used to represent the correspondence between the third sub-scaling factor, the fourth sub-scaling factor, the combination of the second scaling factors, and the second sub-inverse quantization result. For example, when the third sub-scaling factor... and the fourth sub-scaling factor When the precision is third floating-point (e.g., FP32), at least one combination of second scaling factors includes j / 2, which can be denoted as... Here, p is the third index, 1 ≤ p ≤ j / 2. The mapping relationship between this third index p and the second index k is p = k / 2, indicating that every two consecutive second indices k share the same third index p, and thus correspond to the same combination of second scaling factors. .

[0102] For example, in at least one embodiment, the matrix computing device 100 may further include an bias processing module ( Figure 2 (Not shown in the image) is configured to add a bias matrix to the second dequantization result to obtain the output matrix. For example, the bias matrix C can be loaded from memory, and the dimensions of the bias matrix C can be [1, n] or [n] (i.e., a row vector). In the addition operation, this vector can be automatically expanded into a [m, n] matrix through a broadcast mechanism, meaning that each row is the same bias vector. For example, the second dequantization result D can be added element-wise to the bias matrix C to obtain the output matrix. For example, the bias matrix and the output matrix can be of second or third floating-point precision.

[0103] In at least one embodiment, the matrix computation apparatus provided in this disclosure further implements more fine-grained dynamic dequantization compared to the per-tensor strategy. In at least one embodiment of this disclosure, the processing unit decomposes the computation task into fine-grained sub-matrix blocks (such as DP16) for processing, each sub-matrix block having a corresponding first scaling factor combination and a second scaling factor combination. This means that the dequantization operation can be performed at a finer granularity, allowing the scaling factors to accurately match the dynamic range of each local data block, thereby reducing dequantization loss at the system level and ensuring high fidelity of the model output while maintaining high computational efficiency.

[0104] Figure 3A A schematic diagram of a matrix computing device provided in at least one embodiment of this disclosure; Figure 3B This is a schematic diagram of the operation of a matrix computing device provided in at least one embodiment of the present disclosure.

[0105] like Figure 3AAs shown, the matrix computing device includes Each has a dequantization calculation module, for example, corresponding to dequantization calculation module 1, dequantization calculation module 2, ..., dequantization calculation module. Each inverse quantization calculation module may include an accumulation unit, for example, corresponding to accumulation unit 1, accumulation unit 2, ..., accumulation unit 3 respectively. Each inverse quantization calculation module may include Each processing unit, for example, corresponds to processing unit 1, processing unit 2, ..., processing unit... .

[0106] like Figure 3B As shown, the dimensions of the first matrix A and the second matrix B are [m, T*16] and [T*16, n], respectively, and the dimension of the bias matrix C is [m, n]. The first matrix A can be divided into T first submatrices of dimension [m, 16], and the second matrix B can be divided into T second submatrices of dimension [16, n]. For example, the first and second subscaling factors can be of first floating-point precision, such as FP8. The corresponding first submatrix... The first sub-scaling factor of [m,16] The dimension is [m,1], corresponding to the second submatrix. The second sub-scaling factor of [16,n] The dimension is [1, n]. That is, the first sub-scaling factor corresponding to the first matrix [m, T*16] which includes T first sub-matrices can be concatenated into a matrix of dimension [m, T]; the second sub-scaling factor corresponding to the first matrix [T*16, n] which includes T second sub-matrices can be concatenated into a matrix of dimension [T, n].

[0107] like Figure 3A and 3B As shown, each inverse quantization calculation module can receive the corresponding first sub-matrix. Second submatrix And in the corresponding processing unit, the first sub-matrix Second submatrix Performing the outer product operation yields the target matrix with dimensions [m, n]. For example, the inverse quantization calculation module 1 can obtain the target matrix. ~ The inverse quantization calculation module 2 can obtain the target matrix. ~ Inverse quantization calculation module The target matrix can be obtained. ~ .

[0108] like Figure 3Aand 3B As shown, in the target matrix Combination with the corresponding first scaling factor , that is When performing the first dequantization process to obtain the first sub-dequantization result, the dequantization calculation module 1 can obtain the first sub-dequantization result. ~ The dequantization calculation module 2 can obtain the first sub-dequantization result. ~ Inverse quantization calculation module The first sub-inverse quantization result can be obtained. ~ .

[0109] like Figure 3A and 3B As shown, the accumulator unit 1 can... ~ Parallel tree-structured accumulation yields the first inverse quantization result. Accumulation unit 2 can ~ Parallel tree-structured accumulation yields the first inverse quantization result. Accumulation unit Can ~ Parallel tree-structured accumulation yields the first inverse quantization result. That is, to carry out Each accumulation unit can convert the first dequantized result. Transmitted to the inverse quantization accumulation module.

[0110] like Figure 3A and 3B As shown, this applies when the third and fourth sub-scaling factors are of the second floating-point precision, such as FP16 or BF16. In this case, the third sub-scaling factor is a matrix of dimensions T (m, 1 / τ), and the fourth sub-scaling factor is a matrix of dimensions T (1 / τ, n). This corresponds to the first sub-matrix. The third sub-scaling factor of [m,16] (i.e. The dimension of ) is [m, 1 / τ], corresponding to the second submatrix. The fourth sub-scaling factor of [16,n] The dimension is [1 / τ, n]. That is, corresponding to the first matrix [m, T*16] which includes T first sub-matrices, the third sub-scaling factor can be concatenated into a matrix of dimension [m, T / τ]; corresponding to the first matrix [T*16, n] which includes T second sub-matrices, the fourth sub-scaling factor can be concatenated into a matrix of dimension [T / τ, n]. At this point, there are j combinations of second scaling factors. That is, each combination of second scaling factors corresponds to a first dequantization result. For example, in the first dequantization result... and the corresponding second scaling factor combination When performing the second dequantization process to obtain the second sub-dequantization result, the first dequantization result can be... Combined with the corresponding second scaling factor Perform element-wise multiplication We obtain j second sub-inverse quantization results. .

[0111] like Figure 3A and 3B As shown, when the third and fourth sub-scaling factors are of the third floating-point precision, for example, FP32, the third sub-scaling factor is a matrix of dimensions T / 2 [m, 1 / τ], and the fourth sub-scaling factor is a matrix of dimensions T / 2 [1 / τ, n]. This corresponds to the first sub-matrix. The third sub-scaling factor of [m,16] (i.e. The dimension of ) is [m, 1 / τ], corresponding to the second submatrix. The fourth sub-scaling factor of [16,n] (i.e. The dimension of the first sub-scaling factor is [1 / τ, n]. That is, corresponding to the first matrix [m, T*16] which includes T first sub-matrices, the third sub-scaling factor can be concatenated into a matrix of dimension [m, T / 2τ]; corresponding to the first matrix [T*16, n] which includes T second sub-matrices, the fourth sub-scaling factor can be concatenated into a matrix of dimension [T / 2τ, n]. At this point, there are j / 2 combinations of second scaling factors. That is, each combination of second scaling factors corresponds to two first dequantization results. For example, in the first dequantization result... and the corresponding second scaling factor combination When performing the second dequantization process to obtain the second sub-dequantization result, the first dequantization result can be... And the first inverse quantization result Combined with the corresponding second scaling factor Perform element-wise multiplication We obtain j / 2 second sub-inverse quantization results. .

[0112] like Figure 3A and Figure 3B As shown, the dequantization accumulation module can convert j second sub-dequantization results. j / 2 second sub-inverse quantization results Performing tree-based accumulation yields the second inverse quantization result D, which is... or .

[0113] like Figure 3A and Figure 3B As shown, the second dequantization result D can be added to the bias matrix C to obtain the output matrix. The dimensions of both D and C are [m,n].

[0114] Figure 4 This is a schematic flowchart illustrating a matrix calculation method provided in at least one embodiment of the present disclosure.

[0115] like Figure 4 As shown, the matrix calculation method provided in at least one embodiment of this disclosure includes at least steps S110-S130. For example, this matrix calculation method can be applied to an artificial intelligence processor.

[0116] Step S110: Receive at least one first submatrix and at least one second submatrix for calculation, and perform data processing on the at least one first submatrix and at least one second submatrix to obtain at least one target matrix.

[0117] Step S120: Receive at least one first sub-inverse quantization result, accumulate the at least one first sub-inverse quantization result to obtain a first inverse quantization result; wherein, the at least one first sub-inverse quantization result is obtained by performing a first inverse quantization process based on a combination of at least one target matrix and at least one first scaling factor.

[0118] Step S130: Receive at least one second sub-inverse quantization result, accumulate the at least one second sub-inverse quantization result to obtain a second inverse quantization result; wherein, the at least one second sub-inverse quantization result is obtained by performing a second inverse quantization process based on at least one first inverse quantization result and at least one second scaling factor combination.

[0119] For example, in at least one embodiment of the matrix calculation method, before step S110, the method may include: dividing the first matrix A and the second matrix B into T first sub-matrices based on a preset data path width. and T second submatrices ,in, Each first submatrix and each second submatrix Has the corresponding first index ,in, Integer and ≤t≤T.

[0120] For example, in at least one embodiment of the matrix calculation method, before step S110, the method may include: dividing the first matrix A into T first sub-matrices along the column dimensions of the first matrix based on a preset data path width. Based on the preset data path width, the second matrix B is divided into T second sub-matrices along the row dimensions of the second matrix. .

[0121] For example, in at least one embodiment of the matrix calculation method, step S110 may include: according to the first index For the first submatrix Second submatrix Perform matrix multiplication and accumulation operations Obtain the target matrix .

[0122] For example, in at least one embodiment of the matrix calculation method, a first submatrix may be included before step S120. Second submatrix Based on the first index t, each has a corresponding first sub-scaling factor. and the corresponding second sub-scaling factor , where the first sub-scaling factor Second sub-scaling factor The first floating-point precision is defined; wherein the first scaling factor combination is obtained by means of: based on the first index. , the first sub-scaling factor Second sub-scaling factor Perform outer product operation Obtain the first scaling factor combination .

[0123] For example, in at least one embodiment of the matrix calculation method, at least one first sub-inverse quantization result in step S120 may include T first sub-inverse quantization results. Step S120 may include: placing The first sub-inverse quantization result Perform cumulative calculation get The first inverse quantization result , where each first inverse quantization result Associated with the second index k, where k is even and .

[0124] For example, in at least one embodiment of the matrix calculation method, step S120 may further include: based on the first index For each first scaling factor combination and each target matrix Perform element-wise multiplication get The first sub-inverse quantization result .

[0125] For example, in at least one embodiment of the matrix calculation method, step S120 may include: processing the τ first sub-inverse quantization results corresponding to the first index t = (k-1)τ + 1, (k-1)τ + 2, …, kτ respectively. Perform the accumulation operation to obtain the first dequantization result corresponding to the accumulation unit. .

[0126] For example, in at least one embodiment of the matrix calculation method, the first submatrix and the second submatrix have corresponding third sub-scaling factors and corresponding fourth sub-scaling factors, respectively; before step S130, the method may include: the second scaling factor combination is obtained by performing an outer product operation on the third sub-scaling factor and the fourth sub-scaling factor to obtain the second scaling factor combination.

[0127] For example, in at least one embodiment of the matrix calculation method, when the third sub-scaling factor and the fourth sub-scaling factor are of second floating-point precision, at least one combination of second scaling factors includes j combinations of second scaling factors. , where each second scaling factor combination Associated with the second index k; prior to step S130, the second scaling factor combination may include obtaining it by: based on the second index k, the third sub-scaling factor and the fourth sub-scaling factor Perform outer product operation Obtain the second scaling factor combination .

[0128] For example, in at least one embodiment of the matrix computation method, at least one second sub-inverse quantization result includes j second sub-inverse quantization results. Before step S130, the following may be included: j second sub-inverse quantization results. This includes obtaining the first inverse quantization result based on the second index k. Combined with the corresponding second scaling factor Perform element-wise multiplication to obtain j second sub-inverse quantization results. .

[0129] For example, in at least one embodiment of the matrix calculation method, when the third sub-scaling factor and the fourth sub-scaling factor are of third floating-point precision, at least one combination of second scaling factors includes j / 2 combinations of second scaling factors. , where each second scaling factor combination Associated with the second index k; prior to step S130, the second scaling factor combination may include obtaining it by: based on the second index k, the third sub-scaling factor and the fourth sub-scaling factor Perform outer product operation Obtain the second scaling factor combination .

[0130] For example, in at least one embodiment of the matrix computation method, at least one second sub-inverse quantization result includes j / 2 second sub-inverse quantization results. Before step S130, the following may be included: j / 2 second sub-inverse quantization results. This includes obtaining it by combining the second scaling factor. Compared with the first inverse quantization result And the first inverse quantization result Perform element-wise multiplication operations separately and add the results together to obtain j / 2 second sub-inverse quantization results. .

[0131] For example, in at least one embodiment of the matrix calculation method, step S140 may also be included. Step S140: Add a bias matrix to the second inverse quantization result to obtain the output matrix.

[0132] Figure 5 This is a schematic structural diagram of an artificial intelligence processor provided in at least one embodiment of the present disclosure.

[0133] like Figure 5 As shown, the artificial intelligence processor 400 may include multiple computing units, each including a tensor core. For example, the tensor core may be a matrix multiplication core (GEMM Core) for performing matrix multiplication operations using GEMM operators.

[0134] like Figure 5 As shown, each tensor core is equipped with a matrix calculation device 100. For example, each tensor core can be responsible for calculations of various floating-point formats with different precision. For example, the matrix calculation device 100 can be located in the hardware path of the tensor core responsible for calculating low-precision formats (e.g., FP4 precision).

[0135] The matrix calculation device 100 includes at least one inverse quantization calculation module and an inverse quantization accumulation module.

[0136] For example, each inverse quantization calculation module may include at least one processing unit and an accumulation unit.

[0137] For example, at least one processing unit may be configured to receive at least one first submatrix and at least one second submatrix for computation, and to perform data processing on the at least one first submatrix and at least one second submatrix to obtain at least one target matrix.

[0138] For example, the accumulation unit can be configured to receive at least one first sub-inverse quantization result, accumulate the at least one first sub-inverse quantization result, and obtain a first inverse quantization result. For example, the at least one first sub-inverse quantization result can be obtained by performing a first inverse quantization process based on a combination of at least one target matrix and at least one first scaling factor.

[0139] For example, the dequantization accumulation module can be configured to receive at least one second sub-dequantization result, accumulate the at least one second sub-dequantization result, and obtain a second dequantization result. For example, at least one second sub-dequantization result can be obtained by performing a second dequantization process based on at least one first dequantization result and at least one second scaling factor combination.

[0140] For a more detailed description of the matrix computing device 100, please refer to the relevant records of the matrix computing device 100 as described above, and repeated details will not be repeated here.

[0141] For more information about the matrix computing device and its interaction with other units in the artificial intelligence processor, please refer to the relevant description of the matrix computing device. Repeated descriptions will not be repeated here.

[0142] In at least one embodiment, the matrix computation device provided by this disclosure can be integrated inside a tensor kernel, enabling two-level dequantization and accumulation to be performed within the tensor kernel. This avoids the latency caused by storing and computing high-precision (e.g., BF16 or FP32) scaling factors outside the tensor kernel. In at least one embodiment, the matrix computation device provided by this disclosure can match the dynamic range of each local data block at a fine-grained level by decomposing the computation task into fine-grained sub-matrix processing and performing two-level dequantization operations, thereby effectively reducing the accuracy loss caused by low-precision computation at the system level while maintaining performance advantages.

[0143] In addition, it should be noted that Figure 5 The components of the AI ​​processor 400 shown are merely exemplary and not limiting. The AI ​​processor 400 may also have other components depending on the actual application requirements.

[0144] Figure 6This is a schematic block diagram of an electronic device provided in at least one embodiment of the present disclosure; for example, the electronic device can be used to implement the matrix calculation method provided in at least one embodiment of the present disclosure.

[0145] For example, such as Figure 6 As shown, the electronic device 500 includes at least one processor 501 and at least one memory 502. For example, the at least one memory 502 includes one or more computer program modules. For example, the one or more computer program modules are stored in the memory 502 and configured to be executed by the at least one processor 501. The one or more computer program modules include instructions for performing the matrix calculation method described above. When executed by the at least one processor 501, they can perform one or more steps in the matrix calculation method provided in at least one embodiment of this disclosure. The memory 502 and the processor 501 can be interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0146] For example, processor 501 can be a central processing unit (CPU), digital signal processor (DSP), graphics processing unit (GPU), general-purpose graphics processing unit (GPGPU), artificial intelligence (AI) accelerator, or other form of processing unit with data processing and / or program execution capabilities, such as a field-programmable gate array (FPGA); for example, the central processing unit (CPU) can be an x86, ARM, or RISC-V architecture. Processor 501 can be a general-purpose processor or a special-purpose processor, capable of controlling other components in electronic device 500 to perform desired functions.

[0147] For example, memory 502 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc.

[0148] Figure 7 This is a schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0149] The electronic devices in at least one embodiment of this disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0150] The electronic device includes at least one processor and a memory. The processor may be referred to as processing device 601 as described below, and the memory may include at least one of ROM 602, RAM 603, and storage device 608 as described below. The memory is used to store programs for performing the methods described in the various method embodiments above; the processor is configured to execute the programs stored in the memory. The processor may include a central processing unit (CPU) or other forms of processing unit having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0151] like Figure 7 As shown, the electronic device 500 may include a processing unit 601 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in ROM 602 or a program loaded from storage device 608 into RAM 603. RAM 603 also stores various programs and data required for the operation of the electronic device 500. The processing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interfaces are also connected to bus 604.

[0152] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, displays, speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0153] In particular, according to at least one embodiment of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, at least one embodiment of this disclosure includes a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the matrix calculation method of at least one embodiment of this disclosure.

[0154] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In at least one embodiment of this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In at least one embodiment of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, radio frequency (RF), etc., or any suitable combination thereof.

[0155] The aforementioned computer-readable medium may be included in the aforementioned electronic device 500; or it may exist independently and not assembled into the electronic device 500.

[0156] Figure 8 This is a schematic block diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure.

[0157] For example, such as Figure 8 As shown, a non-transitory computer-readable storage medium 700 stores computer-readable instructions 701, which, when executed by at least one processor, perform one or more steps of the matrix calculation method described above.

[0158] For example, the storage medium may include a memory card for a smartphone, a storage component for a tablet computer, a hard drive for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media. For example, the readable storage medium may also be... Figure 7 The memory 502 in the memory is described in the foregoing content and will not be repeated here.

[0159] Although the present disclosure has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to the embodiments of the present disclosure, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present disclosure are within the scope of protection claimed by the present disclosure.

[0160] The following points should be noted regarding this disclosure:

[0161] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0162] (2) For clarity, the thickness of layers or regions in the drawings used to describe embodiments of the present disclosure is enlarged or reduced, i.e., these drawings are not drawn to actual scale.

[0163] (3) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

Claims

1. A matrix calculation device, characterized in that, For use in artificial intelligence processors, the matrix calculation device includes: at least one inverse quantization calculation module and an inverse quantization accumulation module; wherein... Each of the inverse quantization calculation modules includes at least one processing unit and an accumulation unit, wherein, The at least one processing unit is configured to receive at least one first submatrix and at least one second submatrix for calculation, and to perform data processing on the at least one first submatrix and at least one second submatrix to obtain at least one target matrix; The accumulation unit is configured to receive at least one first sub-inverse quantization result, accumulate the at least one first sub-inverse quantization result, and obtain a first inverse quantization result; wherein the at least one first sub-inverse quantization result is obtained by performing a first inverse quantization process based on a combination of the at least one target matrix and at least one first scaling factor; and The inverse quantization accumulation module is configured to: receive at least one second sub-inverse quantization result, accumulate the at least one second sub-inverse quantization result to obtain a second inverse quantization result; wherein the at least one second sub-inverse quantization result is obtained by performing a second inverse quantization process based on a combination of at least one first inverse quantization result and at least one second scaling factor.

2. The matrix computing device according to claim 1, characterized in that, The at least one inverse quantization calculation module includes The inverse quantization calculation module, the at least one processing unit includes... The processing units, wherein... and It is a positive integer; Wherein, the at least one first submatrix and the at least one second submatrix are obtained by means of the following: Based on the preset data path width, the first matrix A and the second matrix B are divided into T first sub-matrices. and T second submatrices ,in, Each of the first submatrices and each of the second submatrices With the corresponding first index ,in, Integer and ≤t≤T.

3. The matrix computing device according to claim 2, characterized in that, The at least one first submatrix and at least one second submatrix are further obtained by means of: Based on the preset data path width, the first matrix A is divided into T first sub-matrices along the column dimensions of the first matrix. ; Based on the preset data path width, the second matrix B is divided into T second sub-matrices along the row dimension of the second matrix. .

4. The matrix computing device according to claim 2, characterized in that, The at least one processing unit is configured to, According to the first index For the first submatrix and the second submatrix Perform matrix multiplication and accumulation operations The target matrix is ​​obtained. .

5. The matrix computing device according to claim 2, characterized in that, First submatrix and the second submatrix Based on the first index t, each has a corresponding first sub-scaling factor. and the corresponding second sub-scaling factor Wherein, the first sub-scaling factor and the second sub-scaling factor This is the first floating-point precision; The first scaling factor combination is obtained through the following method: Based on the first index The first sub-scaling factor and the second sub-scaling factor Perform outer product operation The first scaling factor combination is obtained .

6. The matrix computing device according to claim 2, characterized in that, The at least one first sub-dequantization result includes T first sub-dequantization results. The at least one inverse quantization calculation module is further configured as follows: The The first sub-inverse quantization result Perform cumulative calculation get The first inverse quantization result , where each of the first inverse quantization results Associated with the second index k, where k is even and .

7. The matrix computing device according to claim 6, characterized in that, The T first sub-inverse quantization results This includes obtaining it through the following methods: Based on the first index For each first scaling factor combination and each target matrix Perform element-wise multiplication Obtain the The first sub-inverse quantization result .

8. The matrix computing device according to claim 6, characterized in that, Each of the aforementioned accumulation units is further configured as follows: For the first index t = (k-1)τ + 1, (k-1)τ + 2, …, kτ, respectively, the τ first sub-inverse quantization results are... Perform an accumulation operation to obtain the first inverse quantization result corresponding to the accumulation unit. .

9. The matrix computing device according to claim 6, characterized in that, The first submatrix and the second submatrix each have a corresponding third subscaling factor and a corresponding fourth subscaling factor; The second scaling factor combination is obtained through the following method: The second scaling factor combination is obtained by performing an outer product operation on the third and fourth sub-scaling factors.

10. The matrix computing device according to claim 9, characterized in that, When the third sub-scaling factor and the fourth sub-scaling factor have a second floating-point precision, the at least one combination of second scaling factors includes j combinations of second scaling factors. , where each of the second scaling factor combinations Associated with the second index k; The second scaling factor combination is obtained through the following method: Based on the second index k, the third sub-scaling factor and the fourth sub-scaling factor Perform outer product operation The second scaling factor combination is obtained. .

11. The matrix computing device according to claim 10, characterized in that, The at least one second sub-inverse quantization result includes j second sub-inverse quantization results. ; Among them, the j second sub-inverse quantization results This includes obtaining it through the following methods: Based on the second index k, the first dequantization result Combined with the corresponding second scaling factor Perform element-wise multiplication to obtain the j second sub-inverse quantization results. .

12. The matrix computing device according to claim 9, characterized in that, When the third sub-scaling factor and the fourth sub-scaling factor have a third floating-point precision, the at least one combination of second scaling factors includes j / 2 combinations of second scaling factors. , where each of the second scaling factor combinations Associated with the second index k; The second scaling factor combination is obtained through the following method: Based on the second index k, the third sub-scaling factor and the fourth sub-scaling factor Perform outer product operation The second scaling factor combination is obtained. .

13. The matrix computing device according to claim 12, characterized in that, The at least one second sub-dequantization result includes j / 2 second sub-dequantization results. ; Among them, the j / 2 second sub-inverse quantization results This includes obtaining it through the following methods: Combine the second scaling factor Compared with the first inverse quantization result And the first inverse quantization result Perform element-wise multiplication operations separately and add the results together to obtain j / 2 second sub-inverse quantization results. .

14. The matrix computing apparatus according to any one of claims 1-13, characterized in that, The matrix calculation device further includes: The bias processing module is configured to add a bias matrix to the second inverse quantization result to obtain the output matrix.

15. The matrix computing apparatus according to any one of claims 1-13, characterized in that, The artificial intelligence processor includes multiple computing units, each computing unit including a tensor core, wherein the matrix computing device is disposed in the tensor core.

16. A matrix calculation method, characterized in that, The matrix calculation method, applied to an artificial intelligence processor, includes: Receive at least one first submatrix and at least one second submatrix for calculation, and perform data processing on the at least one first submatrix and the at least one second submatrix to obtain at least one target matrix; Receive at least one first sub-inverse quantization result, accumulate the at least one first sub-inverse quantization result to obtain a first inverse quantization result; wherein, the at least one first sub-inverse quantization result is obtained by performing a first inverse quantization process based on a combination of the at least one target matrix and at least one first scaling factor; and Receive at least one second sub-inverse quantization result, and accumulate the at least one second sub-inverse quantization result to obtain a second inverse quantization result; wherein, the at least one second sub-inverse quantization result is obtained by performing a second inverse quantization process based on at least one first inverse quantization result and at least one second scaling factor combination.

17. An electronic device, characterized in that, The electronic device includes: Memory stores computer-executable instructions non-transiently; The processor is configured to run computer-executable instructions. The computer-executable instructions are executed by the processor to implement the matrix calculation method according to claim 16.

18. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the matrix calculation method according to claim 16.

Images