Matrix computing apparatus, method, electronic device, and storage medium

Abstract

This disclosure provides a matrix computing apparatus, method, electronic device, and storage medium. The matrix computing apparatus is used in an artificial intelligence processor and includes a computation cache module and a quantization encoding module. The computation cache module is configured to receive a first matrix, perform a logarithmic domain transformation on the first matrix to obtain a logarithmic matrix, and divide the logarithmic matrix into at least one quantization block according to the quantization granularity. The quantization encoding module is configured to calculate a scaling factor for each quantization block; and, based on the scaling factor of each quantization block, perform logarithmic quantization encoding on the elements within each quantization block to generate a logarithmic quantization result. This matrix computing apparatus can directly perform logarithmic quantization, achieving seamless integration of the computational pipeline and greatly improving the overall computational throughput.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of artificial intelligence processors, and more specifically to a matrix computing device, a matrix computing method, an electronic device, and a storage medium. Background Technology

[0002] Floating-point quantization refers to converting high-precision floating-point formats (such as FP16, BF16, FP32, etc.) into low-precision floating-point formats (such as FP4, FP8, etc.). Low-precision floating-point formats can significantly improve computational efficiency, especially for General Matrix Multiplication (GEMM) operations required by Artificial Intelligence (AI) models. However, low-precision matrix multiplication introduces precision loss, thus reducing the prediction and generation performance of AI models. Therefore, in the low-precision computation process of AI models, a scaling factor is typically calculated from the current data during floating-point quantization to scale the data's representation range, thereby reducing the precision loss in the computational results. Summary of the Invention

[0003] At least one embodiment of this disclosure provides a matrix computing device for an artificial intelligence processor. The matrix computing device includes: a computing cache module configured to receive a first matrix, perform a logarithmic domain transformation on the first matrix to obtain a logarithmic matrix, and divide the logarithmic matrix into at least one quantization block according to a quantization granularity; and a quantization encoding module configured to calculate a scaling factor for each quantization block, and perform logarithmic quantization encoding on the elements within each quantization block based on the scaling factor of each quantization block to generate a logarithmic result.

[0004] For example, in at least one embodiment of this disclosure, a matrix calculation device is provided, wherein the calculation cache module includes: a logarithm calculation module configured to: receive a first matrix, wherein the first matrix includes at least one first submatrix; and perform a logarithmic domain transformation on each first submatrix to obtain a logarithmic submatrix corresponding to each first submatrix.

[0005] For example, in at least one embodiment of this disclosure, a matrix calculation device is provided, wherein the logarithm calculation module is further configured to: extract the sign information of each element in each of the first submatrixes; perform absolute value calculation on each element in each of the first submatrixes to obtain an absolute value submatrix; and perform a base-2 logarithmic operation on each element in the absolute value submatrix to obtain the logarithmic submatrix.

[0006] For example, in at least one embodiment of this disclosure, a matrix calculation device is provided, wherein the calculation cache module further includes: a local value acquisition module, configured to: iteratively calculate the maximum and minimum values ​​of each quantization block in the logarithmic domain based on the logarithmic submatrix; calculate the scaling factor of each quantization block based on the maximum and minimum values ​​corresponding to each quantization block; and perform logarithmic quantization encoding on the elements in each quantization block based on the scaling factor of each quantization block, and obtain the logarithmic quantization result.

[0007] For example, in at least one embodiment of this disclosure, a matrix calculation device is provided, wherein the quantization encoding module is further configured to obtain a target minimum value for each quantization block based on the quantization precision, the maximum value corresponding to each quantization block, and the minimum value corresponding to each quantization block; and to calculate a scaling factor corresponding to each quantization block based on the maximum value and the target minimum value, and to perform the logarithmic quantization encoding based on the scaling factor.

[0008] For example, in at least one embodiment of this disclosure, a matrix computing device is provided, which is configured to output a first tensor and quantization information, wherein the first tensor includes the logarithmic quantization result and the symbol information, and the quantization information includes the target minimum value of each quantization block and the scaling factor corresponding to each quantization block.

[0009] For example, in at least one embodiment of this disclosure, a matrix computing device is provided, the matrix computing device further comprising: a quantization control unit configured to: select whether to perform logarithmic calculation on the first matrix; wherein, in response to performing logarithmic calculation on the first matrix, the first matrix is ​​transmitted to the computing cache module; and in response to not performing logarithmic calculation on the first matrix, the first matrix is ​​transmitted outside the matrix computing device.

[0010] For example, in at least one embodiment of this disclosure, a matrix computing apparatus is provided in which the artificial intelligence processor is configured to deploy a neural network model including a hybrid expert MoE layer, the first matrix including the output result after the expert model in the hybrid expert MoE layer has been computed.

[0011] For example, in at least one embodiment of this disclosure, a matrix computing device is provided, wherein the artificial intelligence processor includes a plurality of computing units, each computing unit including a tensor core, wherein the matrix computing device is disposed in the tensor core.

[0012] At least one embodiment of this disclosure provides a matrix calculation method applied to an artificial intelligence processor. The matrix calculation method includes: receiving a first matrix; performing a logarithmic domain transformation on the first matrix to obtain a logarithmic matrix; dividing the logarithmic matrix into at least one quantization block according to the quantization granularity; calculating a scaling factor for each quantization block; and performing logarithmic quantization encoding on the elements within each quantization block based on the scaling factor of each quantization block to generate a logarithmic result.

[0013] At least one embodiment of this disclosure provides an electronic device, including: a memory that non-transitoryly stores computer-executable instructions; and a processor configured to run the computer-executable instructions, wherein the computer-executable instructions are executed by the processor to implement the matrix calculation method provided in the at least one embodiment described above.

[0014] At least one embodiment of this disclosure provides a non-transitory computer-readable storage medium storing computer-readable instructions thereon, wherein the computer-readable instructions, when executed by at least one processor, perform the matrix calculation method provided in the at least one embodiment described above.

[0015] In at least one embodiment of this disclosure, the quantization process is integrated into the matrix computing device, thereby enabling quantization to be performed directly within the matrix computing device, achieving seamless integration of the computing pipeline, and greatly improving the overall computing throughput. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0017] Figure 1 This is a schematic diagram of the architecture of a general-purpose graphics processing unit (GPGPU).

[0018] Figure 2 A schematic structural diagram of a matrix computing device provided in at least one embodiment of this disclosure;

[0019] Figure 3 Another schematic structural diagram of the matrix computing device provided in at least one embodiment of this disclosure;

[0020] Figure 4 A schematic diagram of a matrix computing device provided in at least one embodiment of this disclosure;

[0021] Figure 5 A schematic flowchart illustrating a matrix calculation method provided in at least one embodiment of this disclosure;

[0022] Figure 6A schematic structural diagram of an artificial intelligence processor provided in at least one embodiment of this disclosure;

[0023] Figure 7 A schematic block diagram of an electronic device provided for at least one embodiment of this disclosure;

[0024] Figure 8 A schematic block diagram of another electronic device provided for at least one embodiment of this disclosure; and

[0025] Figure 9 This is a schematic block diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0027] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described object changes. To keep the following description of the embodiments of this disclosure clear and concise, detailed descriptions of some known functions and components are omitted.

[0028] Floating-point numbers (FP) are primarily used to represent decimals and typically consist of three parts: a sign bit, an exponent, and a mantissa. The exponent part can also be called the exponent code. For example, a floating-point number V can usually be represented in the following form:

[0029]

[0030] In this context, the sign bit l can be 1 bit, determining whether the floating-point number V is negative or positive; M represents the mantissa, which can include multiple bits and is a binary fractional form, defining the precision of the floating-point number; E represents the exponent (also called the exponent value), used to weight the floating-point number, reflecting the position of the decimal point in the floating-point number V, and defining the range of values ​​for the floating-point number.

[0031] Traditional high-precision floating-point numbers typically include three formats: half-precision floating-point numbers (FP16), single-precision floating-point numbers (FP32), and double-precision floating-point numbers (FP64), with different numbers of bits in their exponent and mantissa parts.

[0032] AI accelerators are widely used for training deep learning models. For convolution operations, common in deep learning models, both hardware and software designs have been specifically optimized to accelerate computation. For example, various floating-point data formats have been developed and optimized for fields such as artificial intelligence and deep learning, including BF16 (brain floating point 16, 16-bit width), BF24 (brain floating point 24, 24-bit width), and TF32 (Tensor Float 32, 19-bit width). These data formats can significantly reduce computational processing, especially the computational resources and power consumption required for matrix multiplication or convolution multiplication operations. In addition, the processor also supports some common floating-point types, such as half-precision floating-point numbers (FP16, 16-bit width) or single-precision floating-point numbers (FP32, 32-bit width).

[0033] Low-precision matrix multiplication is increasingly being used in the training and inference of large AI models due to its significant performance gains with acceptable precision loss. In GPUs (Graphics Processing Units) or GPGPUs, matrix multiplication is typically performed in hardware by tensor kernels. Low-precision tensor kernels are several times more computationally powerful than high-precision tensor kernels, resulting in higher computational efficiency. Furthermore, the data volume of a low-precision tensor is also only a few times that of a high-precision tensor, leading to higher data transfer efficiency. Therefore, the end-to-end efficiency improvement brought by low-precision tensor computation is almost several times greater. For example, the computational power of an FP4 tensor kernel can be 2 to 8 times that of FP8, 4 to 16 times that of FP16 / BF16, or even higher. Moreover, the data volume is half that of FP8 and one-quarter that of FP16 / BF16.

[0034] To reduce the loss of precision, low-precision matrix multiplication introduces a scaling factor to maximize the numerical expressive power of low-precision tensors.

[0035] For low-precision matrix multiplication, such as D = A × B + C, where A, B, C, and D are all high-precision tensors, the general matrix multiplication operation using a scaling factor can be described as follows:

[0036] D = (α⊙A') × (β⊙B') + σ⊙C' or D = (α⊙A') × (β⊙B') + σ⊙C' β)⊙(A'×B')+σ⊙C'

[0037] D→D',γ

[0038] Where ⊙ represents element-wise multiplication. σ represents the outer product, × represents matrix multiplication, α is the scaling factor for tensor A, β is the scaling factor for tensor B, σ is the scaling factor for tensor C, and γ is the scaling factor for tensor D. Quantized tensors A', B', C', and D' are the low-precision tensors obtained by floating-point quantization of tensors A, B, C, and D, respectively. For example, the data dimensions of each parameter are shown below:

[0039] A: [m,p×k], B: [p×k,n], C: [m,n], D: [m,n], α: [m,p], β: [p,n], σ: [m,r] or [r,n], γ: [m,r] or [r,n],

[0040] Where m, n, p, k, and r are all positive integers. For example, for any row in tensor A, every k tensor elements in that row share a single scaling factor parameter; for any column in tensor B, every k tensor elements in that column share a single scaling factor parameter.

[0041] Currently, Mixture of Experts (MoE) models are gaining increasing attention in large-scale deep learning applications. These models improve computational efficiency and scalability by processing different inputs in parallel through multiple expert modules. To support MoE models with a large number of experts but a small number of activated experts, the Expert Parallelism (EP) approach is widely adopted. This approach increases the overall system throughput by deploying experts in parallel across multiple devices.

[0042] In distributed deep learning inference systems, large-scale expert parallelism (LAP) is an efficient computational paradigm for hybrid expert models. It involves distributing numerous experts from the model across multiple AI processors on multiple computing nodes (servers), such as graphics processing units (GPUs) and neural network processing units (NPUs). By increasing the breadth of expert distribution, this approach reduces the size of model weights that need to be loaded on a single accelerator card, thereby reducing GPU memory usage. The freed-up GPU memory resources can then be used to support larger batch processing scales, thus improving the efficiency of matrix multiplication operations and the overall inference throughput of the cluster, while reducing latency.

[0043] The model architecture served by the large EP solution is, for example, the Mixture of Experts (MoE) model. The MoE model is a neural network architecture that balances model size and computational cost through sparse activation.

[0044] MoE models can include multiple structurally identical but parameter-independent expert networks, such as feedforward neural networks and gated networks. For example, a gated network, acting as a dynamic router, can compute a probability distribution for each input token and route computation tasks only to the one or a few experts with the highest probabilities. This conditional computation mechanism can greatly increase the total number of parameters in a MoE model, but the number of parameters actually activated in each inference is only a small fraction, achieving decoupling between model capacity and single-inference computation cost. For example, a computer system model has 256 experts. If this model is run on two servers (nodes) with a total of 16 AI processors (e.g., 16 general-purpose graphics processing units (GPGPUs, hereinafter referred to as "GPUs")), then through a large EP strategy, such as each GPU can support 16 experts, the experts can be distributed across different GPUs.

[0045] When deploying the MoE model using a large EP strategy, experts are distributed across multiple computing nodes, making low-latency, high-bandwidth communication between nodes crucial for system performance.

[0046] Low-precision quantization is a technique that reduces communication overhead by decreasing the precision of data representation, thereby effectively reducing the communication bandwidth and storage costs required for data transfer between processors or storage levels. For example, activation function data in BF16 format can be quantized into FP8 format, compressing the data size and improving data transmission efficiency.

[0047] The current mainstream quantization method is linear quantization, for example, defining a high-precision tensor block. Its corresponding low-precision quantized tensor Defined by the following formula:

[0048]

[0049] in, This refers to the type conversion to an n-bit floating-point format, where the scaling factor... The maximum value of the absolute value of the original high-precision tensor This is the maximum range of positive numbers that the target precision format can represent (e.g., 448 for FP8 E4m3); to prevent A value of zero leads to a division-by-zero error. In practice, a local minimum value ϵ is introduced (e.g., ϵ = 1e-8, 1e-9, …, 1e-12, etc.). That is, when When =0, let =ϵ.

[0050] The linear dequantization process approximates the recovery of the original data. The linear dequantization formula is as follows:

[0051] Linear quantization has the advantage of being simple to implement and directly reducing data volume (e.g., BF16 to FP8 can achieve a compression ratio of 2:1). However, its drawback is that a uniform quantization step size introduces a significant loss of accuracy, especially when the data dynamic range (the ratio of the maximum to the minimum value) is large, the relative error of quantization for small-amplitude data will increase significantly. Scaling factor Although it can preserve certain numerical distribution characteristics on a global scale, it cannot eliminate the error caused by linear mapping.

[0052] In contrast, logarithmic quantization, by introducing a nonlinear mapping, typically achieves lower precision loss than linear quantization. Its principle lies in utilizing the properties of the logarithmic function to transform the multiplicative relationship of data in the original domain into an additive relationship in the quantization domain, thus achieving a proportionally uniform quantization. This property allows it to provide higher quantization density for small values ​​and lower quantization density for large values ​​in the logarithmic domain.

[0053] The formula for quantification is as follows:

[0054]

[0055] Among them, log This usually refers to logarithmic operations with base 2. `min` and `step` represent the minimum value and quantization step size for logarithmic data, respectively, based on... The statistical range was calculated. This indicates rounding to an n-1 bit integer format. The operation restricts the value to a specified range, and the subsequent +1 operation is often used to handle zero values ​​or reserve special codes.

[0056] The main drawback of quantization is its higher computational cost compared to linear quantization. It requires additional operations such as logarithmics, scaling, rounding, and exponentiation (during dequantization). More importantly, addition is complex to implement in the logarithmic field because addition in the primitive field corresponds to transcendental operations in the logarithmic field, while multiplication, although simplified to addition in the logarithmic field, still relies on exponential and logarithmic transformations in its actual implementation. Therefore, quantization introduces additional overhead if the quantized data needs to be used immediately for subsequent arithmetic calculations (such as multiplication and addition).

[0057] However, in scenarios where only data transmission is required rather than real-time computation, the advantages of quantization become apparent. For example, in distributed training or large-scale hybrid expert models (MoE), intermediate activation values ​​need to be exchanged between different processors or computing units. In this case, after data compression and transmission, it is dequantized at the receiving end before participating in computation. In such a "quantization-transmission-dequantization-computation" pipeline, quantization, due to its higher accuracy preservation capability, can deliver higher data fidelity with the same communication bandwidth, or allow the use of lower bit widths at the same target accuracy, thereby further reducing communication overhead and achieving a better end-to-end accuracy and efficiency balance than linear quantization.

[0058] Figure 1 This is a schematic diagram of a general-purpose graphics processing unit (GPGPU).

[0059] like Figure 1 As shown, a general-purpose graphics processor is actually an array of programmable multiprocessors. For example, a programmable multiprocessor can be a streaming processor cluster (SPC), such as including... Figure 1 The diagram shows streaming processor clusters 1, ..., M, where M is a positive integer greater than 1. In a general-purpose graphics processor, one streaming processor cluster handles one computational task, or multiple streaming processor clusters handle one computational task. Multiple streaming processor clusters share data through a global cache or global memory.

[0060] like Figure 1 As shown, taking streaming processor cluster 1 as an example, one streaming processor cluster includes multiple computing units, such as... Figure 1The system consists of Compute Unit 1, Compute Unit 2, ..., Compute Unit N, where N is a positive integer. Each Compute Unit (CU) is used to perform arithmetic and logical operations, such as accumulation, reduction, and regular addition, subtraction, multiplication, and division.

[0061] A computing unit includes multiple cores (also called computing kernels or computing cores). Figure 1 (Not shown in the image), each computing core includes an arithmetic logic unit (ALU), a floating-point unit (FPU), etc., and is used to perform specific computing tasks.

[0062] like Figure 1 As shown, each computational unit also provides a tensor core for performing tensor-related computations. For example, this tensor core can be a GEMM core, used for matrix multiplication operations using the GEMM operator. Tensors are a very important data structure in deep learning; they are high-dimensional generalizations of scalars, vectors, and matrices. Tensor operations are commonly used in the training and inference of current deep learning models, and tensor cores can accelerate matrix multiplication operations. Tensor cores in multiple computational units can be uniformly scheduled and controlled.

[0063] In addition, the computation unit includes a register file, shared memory, and a tensor core memory unit for storing source and destination data related to computation tasks. Shared memory within a computation unit is used to share data between cores within that unit. The tensor core memory unit is a storage resource closely related to the tensor core, used to store intermediate data when the tensor core performs tensor operations (such as matrix multiplication). Furthermore, it can perform data format processing on the tensor data to be used in tensor operations, ensuring that data loaded from external sources conforms to the tensor core's data format requirements.

[0064] In parallel computing, computational tasks are typically executed by multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block distribution module. Figure 1 (Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0065] In each computing unit, the thread beam scheduling / distribution module ( Figure 1(Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple computing cores within the computing unit can run thread bundles. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can be executed concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory execution instructions are issued to shared memory within the computing unit or further issued to intermediate-level caches, global caches, or global memory for read and write operations, etc.

[0066] The inventors of this disclosure have noted that when the output data of the GEMM Core is of high precision type, such as BF16 or FP32, a general-purpose vector processing core (VectorCore) is typically used to perform the relevant operations to achieve log quantization. However, since logarithmic transformation is a non-linear operation, its calculation process involves a series of complex steps such as absolute value calculation, logarithmic function calculation, scaling, rounding, and clamping. This results in significant performance overhead for such software-based or general-purpose hardware implementations. On the one hand, it consumes a large amount of vector operation resources that could be used for parallel computing, reducing the throughput efficiency of the core computing unit; on the other hand, frequent non-linear function calculations also introduce additional latency and energy consumption, making it a performance bottleneck in applications that seek a balance between high-bandwidth data transmission and computational efficiency. Therefore, implementing logarithmic quantization in a more hardware-efficient manner is of great significance for improving overall system performance.

[0067] At least one embodiment of this disclosure provides a matrix computing apparatus for use in an artificial intelligence processor. The matrix computing apparatus includes a computation cache module and a quantization encoding module. The computation cache module is configured to receive a first matrix, perform a logarithmic domain transformation on the first matrix to obtain a logarithmic matrix, and divide the logarithmic matrix into at least one quantization block according to the quantization granularity. The quantization encoding module is configured to calculate a scaling factor for each quantization block; and perform logarithmic quantization encoding on the elements within each quantization block based on the scaling factor, generating a logarithmic quantization result.

[0068] For example, in at least one embodiment, the logarithmic process is integrated into the matrix computing device, thereby enabling the logarithmic process to be performed directly within the matrix computing device, achieving seamless integration of the computing pipeline and greatly improving the overall computing throughput.

[0069] For example, in at least one embodiment, after the expert model completes its matrix multiplication calculations on a single AI processor, its high-precision output can be directly converted into a logarithmic format within the GEMM Core. This not only improves the data communication efficiency of the multi-AI processor system when running the MoE model and reduces latency and bandwidth consumption, but also ensures numerical accuracy during communication through a better logarithmic quantization method. This minimizes the amount of communication data, alleviates the pressure on the interconnect bandwidth between AI processors, reduces the accuracy loss of quantization, and improves system performance.

[0070] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, but this disclosure is not limited to these specific embodiments.

[0071] Figure 2 This is a schematic structural diagram of a matrix computing device provided in at least one embodiment of the present disclosure.

[0072] For example, this matrix computing device can be applied to processors such as artificial intelligence processors. The processor can be any one of the following: Graphics Processing Unit (GPU), Tensor Processing Unit (TPU), Neural Network Processing Unit (NPU), Deep Learning Processing Unit (DPU), Accelerated Processing Unit (APU), and General-Purpose Graphics Processing Unit (GPGPU).

[0073] For example, in this disclosure, the matrix computing device is described using a graphics processor or a general-purpose graphics processor as an example, but of course, this disclosure is not limited thereto.

[0074] For example, such as Figure 2 As shown, the matrix computing device 100 includes a computing cache module 110 and a quantization encoding module 120.

[0075] For example, the computation cache module 110 can be configured to receive a first matrix, perform a logarithmic field transformation on the first matrix to obtain a logarithmic matrix, and divide the logarithmic matrix into at least one quantization block according to the quantization granularity.

[0076] For example, logarithmic domain transformation can perform element-wise nonlinear mathematical operations on the first matrix. For example, quantization granularity is the size of a data block that performs one independent quantization; the logarithmic matrix can be divided into multiple quantization blocks according to the quantization granularity. Each quantization block can compute and share the same quantization parameters in the logarithmic domain based on its internal data and configuration; these quantization parameters may include, for example, a scaling factor. For example, the computation cache module 110 can pre-configure the quantization granularity and, according to the quantization granularity, logically divide the written logarithmic matrix into multiple quantization blocks in the cache, and manage the starting address and boundaries of each quantization block.

[0077] For example, the quantization encoding module 120 can be configured to calculate the scaling factor for each quantization block; and based on the scaling factor of each quantization block, to perform logarithmic encoding on the elements within each quantization block to generate a logarithmic result.

[0078] For example, the scaling factor in logarithmic quantization can be referred to as the quantization step size, which can be determined based on quantization precision and other parameters. For example, the quantization step size can be defined as the numerical span represented by each integer encoded value in the logarithmic domain. For example, logarithmic encoding can include calculations such as normalization, rounding, clamping, and offsetting. For example, the logarithmic quantization result can be a matrix of the same size as the logarithmic matrix, where each element is an n-bit integer representing the mapped value of the original value in the logarithmic domain.

[0079] For example, the computation cache module 110 can be connected to the quantization encoding module 120. For example, the computation cache module 110 and the quantization encoding module 120 may include basic digital circuit elements such as arithmetic logic units (ALUs), multipliers, adders, registers, and caches.

[0080] For example, the floating-point precision format of the first matrix is ​​higher than that of the logarithmic quantization result. For example, the first matrix can be a high-precision format such as BF16 or FP32. For example, the logarithmic quantization result can be determined by the quantization precision. For example, in logarithmic quantization, the quantization precision is mainly determined by the number of quantization bits n, which can define the integer range used to represent the quantized values ​​in the logarithmic field (for example, when n=8, the integer range is 1~256), thereby determining the representation precision of the values ​​in the original data field.

[0081] For example, in at least one embodiment, the matrix computing device 100 is applied to an artificial intelligence processor, which can be configured to deploy a neural network model including a hybrid expert MoE layer. For example, the first matrix can be the output of the expert model in the hybrid expert MoE layer after computation. For example, the first matrix can be the output of matrix multiplication computation via GEMM Core.

[0082] For example, in at least one embodiment, the artificial intelligence processor may include multiple computing units, each computing unit including a tensor core. For example, a matrix computing device 100 may be disposed within the tensor core. For example, the tensor core may be... Figure 1 The tensor kernel in the matrix can be a GEMM core, used for matrix multiplication operations using GEMM operators. For example, each tensor kernel can handle calculations of various floating-point formats. For instance, the matrix computing device 100 can be located in a hardware path within the tensor kernel responsible for calculating input precision formats such as FP16, BF16, or TF32. Alternatively, the matrix computing device 100 can also be located in a hardware path within the tensor kernel responsible for calculating low-precision formats (e.g., INT8). For example, the front end of this hardware path can be responsible for dequantizing the input low-precision data into a high-precision format for calculation, while the matrix computing device 100 can be located at the back end of this hardware path to perform quantization operations on the high-precision output.

[0083] The matrix computation apparatus provided in at least one embodiment of this disclosure can be integrated inside a tensor kernel, so that logarithmic quantization can be completed inside the tensor kernel, avoiding the need for logarithmic quantization to be performed outside the kernel, such as in a general-purpose vector processing core (VectorCore), thereby achieving seamless connection of the computation pipeline and greatly improving the overall computation throughput.

[0084] Figure 3 Another schematic structural diagram of a matrix computing device provided in at least one embodiment of this disclosure.

[0085] like Figure 3 As shown, the matrix computing device 200 may include a computing cache module 210, a quantization encoding module 220, and a quantization control unit 230.

[0086] For example, such as Figure 3 As shown, the computation cache module 210 may include a logarithm computation module 211 and a local value acquisition module 212.

[0087] For example, the quantization control unit 230, the logarithm calculation module 211, the local value acquisition module 212, and the quantization encoding module 220 can be connected in sequence for communication.

[0088] For example, in at least one embodiment, the logarithm calculation module 211 may be configured to: receive a first matrix. For example, the first matrix may include at least one first submatrix; and perform a logarithmic field transformation on each first submatrix to obtain a logarithmic submatrix corresponding to each first submatrix.

[0089] For example, in at least one embodiment, the logarithmic calculation module 211 may be further configured to: extract the sign information of each element in each first submatrix; perform absolute value calculation on each element in each first submatrix to obtain an absolute value submatrix; and perform a base-2 logarithmic operation on each element in the absolute value submatrix to obtain a logarithmic submatrix.

[0090] For example, the logarithm calculation module 211 can receive a first matrix from the GEMM core, which has, for example, been divided into multiple first submatrices. Each first submatrix can be associated with a corresponding block index. For example, the logarithm calculation module 211 can perform logarithmic field transformations on each first submatrix in parallel to obtain the corresponding logarithmic submatrix. For example, the logarithm calculation module 211 can temporarily store the logarithmic submatrix corresponding to each first submatrix in an on-chip cache to reduce direct access to memory.

[0091] For example, the logarithmic calculation module 211 can extract the sign information of each element in each first submatrix. Since logarithmic operations are only defined for positive numbers, the logarithmic calculation module needs to retain the sign information for subsequent dequantization to recover the data. For example, for floating-point formats (such as FP16, BF16, FP32), the sign bit can be located in the most significant bit (MSB), so the logarithmic calculation module 211 can extract the sign information using a bitmask and output 1 bit of sign information (e.g., mapped to 0 / 1 or +1 / -1). For example, the above operation of extracting sign information can be implemented by a parallel sign extraction unit array, where each parallel sign extraction unit can process one element in the first submatrix, and the sign information can be stored in a dedicated sign buffer.

[0092] For example, the logarithm calculation module 211 can perform absolute value calculation on each element in each first submatrix to obtain an absolute value submatrix. For example, for floating-point format, absolute value calculation can be achieved by clearing the sign bit, which is completed at the same pipeline level as the aforementioned sign information extraction.

[0093] For example, the logarithm calculation module 211 can perform a base-2 logarithmic operation on each element of the absolute value submatrix to obtain a logarithmic submatrix. Alternatively, the logarithm calculation module 211 can segment the input range using a lookup table, pre-store the logarithmic value and slope in each segment, and improve accuracy through linear interpolation. Finally, it fits the data using a low-order polynomial (e.g., quadratic or cubic) within a specific interval. Curve. For example, for floating-point format, the logarithm calculation module 211 can use its exponent as the integer part of the logarithm, and the mantissa is calculated as the fractional part through a LUT or approximation circuit.

[0094] For example, in at least one embodiment, the local value acquisition module 212 can be configured to: iteratively calculate the maximum and minimum values ​​of each quantization block in the logarithmic domain based on the logarithmic submatrix; calculate the scaling factor of each quantization block based on the maximum and minimum values ​​corresponding to each quantization block; and perform logarithmic encoding on the elements in each quantization block based on the scaling factor of each quantization block, and obtain the logarithmic result.

[0095] For example, the quantization block is determined based on the quantization granularity. For instance, in at least one embodiment, the quantization granularity is defined as M×N, and the dimension of the first submatrix is ​​m×n, where M=p×m, N=q×n, M and N are the quantization granularity of rows and columns, respectively, and p and q are non-negative integers. When p or q is 0, it means that each row or column is directly output. Therefore, the supported granularities are 1 and integer multiples of m and n. That is, when p>0 and q>0, the basic unit of the quantization block can be an m×n first submatrix; when p=0 and q>0, the quantization block size is 1×N (N=q×n), and the basic unit can be a 1×n row vector block. When p>0 and q=0, the quantization block size is M×1 (M=p×m), and the basic unit can be an m×1 column vector block. When p=0 and q=0, the quantization block size is 1×1, i.e., a single element, where the maximum and minimum values ​​are the element itself.

[0096] For example, the local value acquisition module 212 may include a maximum value register and a minimum value register. For example, the local value acquisition module 212 may receive the at least one quantized block. For example, when p>0 and q>0, the local value acquisition module 212 may be configured to sequentially process each m×n first submatrix, calculate its local extrema, and update the maximum value register and minimum value register. For example, when p=0 and q>0, the local value acquisition module 212 may divide the quantized block into q row vector blocks of length n (each block being 1×n in size) as the basic unit of iteration, and sequentially process each 1×n row vector block, calculate its local extrema, and update the maximum value register and minimum value register. For example, when p>0 and q=0, the local value acquisition module 212 may divide the quantized block into p column vector blocks of length m (each block being m×1 in size) as the basic unit of iteration, and sequentially process each m×1 column vector block, calculate its local extrema, and update the maximum value register and minimum value register. For example, the maximum value register can initially store a minimum value, and the minimum value register can initially store a maximum value for subsequent iterations and updates.

[0097] For example, in at least one embodiment, the quantization encoding module 220 may be further configured to obtain a target minimum value for each quantization block based on the quantization precision, the maximum value and the minimum value corresponding to each quantization block; and to calculate a scaling factor for each quantization block based on the maximum value and the target minimum value, and to perform logarithmic quantization encoding based on the scaling factor.

[0098] For example, in logarithmic quantization, the quantization precision is mainly determined by the number of quantization bits n. The number of quantization bits n can be defined as the integer range used to represent the quantized values ​​in the logarithmic field (for example, when n=8, the integer range is 1~256), thereby determining the representation precision of the values ​​in the original data field.

[0099] For example, the quantization encoding module 220 can be further configured to obtain the target minimum value for each quantization block based on the quantization precision, the maximum value, and the minimum value corresponding to each quantization block. For instance, the quantization encoding module 220 can calculate the theoretical lower limit of the quantization block at the quantization precision based on the maximum value and the number of quantization bits n corresponding to each quantization block, and compare the theoretical lower limit of each quantization block with the minimum value of each quantization block, taking the larger of the theoretical lower limit and the minimum value as the target minimum value. This step is to limit the dynamic range of the quantized data, ensuring that it does not exceed the upper limit of the quantizer's ability to accurately represent it. For example, the quantization encoding module may include a comparator and a multiplexer; for example, the comparator can compare the theoretical lower limit and the minimum value, and the multiplexer can select the larger value based on the comparison result and output it as the target minimum value.

[0100] For example, in at least one embodiment, the quantization encoding module 220 may be further configured to calculate a scaling factor corresponding to each quantization block based on the maximum value and the target minimum value, and perform logarithmic quantization encoding based on the scaling factor. For example, the quantization encoding module 220 may perform logarithmic quantization encoding on the elements within each quantization block based on the scaling factor and the target minimum value corresponding to each quantization block to obtain a logarithmic result. For example, the quantization encoding module 220 may include parallel comparators, multiplexers, etc., to implement clamping and rounding calculation processes in the logarithmic quantization encoding.

[0101] For example, in at least one embodiment, the matrix computation device 200 can be configured to output a first tensor and quantization information. For example, the first tensor includes a log-quantization result and symbol information. For example, the quantization information includes a target minimum value for each quantization block and a scaling factor corresponding to each quantization block. For example, the log-quantization result and the symbol information extracted from the first matrix can be concatenated into the first tensor. For example, this first tensor and quantization information can be used for subsequent inverse quantization.

[0102] For example, in at least one embodiment, the matrix computing device 200 may further include a quantization control unit 230. For example, the quantization control unit 230 may be configured to select whether to perform logarithmic calculation on the first matrix. For example, in response to performing logarithmic calculation on the first matrix, the quantization control unit 230 may transfer the first matrix to a computation cache module. For example, in response to not performing logarithmic calculation on the first matrix, the quantization control unit 230 may transfer the first matrix outside the matrix computing device 200.

[0103] In at least one embodiment, the quantization control unit provided in this disclosure can be configured to allow the user to choose whether to perform logarithmic quantization calculations, thereby enabling dynamic switching of working modes and adapting to diverse computing scenarios.

[0104] Figure 4 This is a schematic diagram of a matrix computing device provided for at least one embodiment of the present disclosure.

[0105] For example, such as Figure 4 As shown, the matrix computation unit can be located within the matrix multiplication core (GEMM Core). For example, as... Figure 4 As shown, the input to GEMM Core can be matrices A, B, and C. Matrix A can include multiple submatrices of dimension [m,k], matrix B can include multiple submatrices of dimension [k,n], and matrix C can include multiple submatrices of dimension [m,n]. In GEMM Core, matrices A, B, and C can be used according to block indices to perform the following matrix operations to obtain the first matrix D: First matrix It can include multiple first submatrices, each with dimensions [m, n].

[0106] For example, such as Figure 4 As shown, the matrix computing device may include a quantization control unit, a logarithm calculation module, a local value acquisition module, and a quantization encoding module. For example, as Figure 4 As shown, the quantization control unit, logarithmic calculation module, local value acquisition module, and quantization encoding module are sequentially connected in communication.

[0107] like Figure 4 As shown, the quantization control unit can be configured to select whether to perform logarithmic calculation on the first matrix. If logarithmic calculation is performed on the first matrix, the quantization control unit can convert the first submatrix... The data is transferred to the logarithmic calculation module in the computation cache module. For example, in response to not performing logarithmic calculation on the first matrix, the quantization control unit can transfer the first matrix D outside the matrix computation device, i.e., outside the GEMM Core.

[0108] like Figure 4As shown, the logarithm calculation module can be configured to receive at least one first submatrix. For example, the logarithm calculation module can be configured to perform a logarithmic field transformation on each first submatrix to obtain the corresponding logarithmic submatrix. For example, the logarithm calculation module can implement the following steps:

[0109] (1) Extract the sign information of each element in each first submatrix, that is, implement the formula. .in, for The corresponding symbol information, sign(.) is the symbol information extraction operation.

[0110] (2) Perform absolute value calculation on each element in each first submatrix to obtain the absolute value submatrix, which is to implement the formula. Where |.| represents the absolute value calculation.

[0111] (3) Perform a base-2 logarithmic operation on each element of the absolute value submatrix to obtain the logarithmic submatrix, which is to realize the formula. .in, It is a logarithmic submatrix.

[0112] like Figure 4 As shown, the local value calculation acquisition module can be configured to receive the logarithmic submatrix. For example, the quantization granularity is M×N, where M=p×m and N=q×n are the quantization granularities of the rows and columns, respectively, and p and q are non-negative integers.

[0113] For example, the local value calculation and acquisition module can implement the following steps:

[0114] (1) Based on the logarithmic submatrix, iteratively calculate the maximum and minimum values ​​of each quantization block in the logarithmic domain, that is, implement the following formula:

[0115]

[0116] in, Let be the number of components of the first submatrix of dimension [m,n] in the rows and columns of the quantized block. For logarithmic submatrices, The minimum value for each quantization block. The maximum value for each quantization block. This indicates that the local value acquisition module can be configured to process the quantization blocks sequentially. indivual The first submatrix is ​​used to calculate its local extrema and obtain the maximum and minimum values ​​in each quantization block.

[0117] (2) Based on the quantization precision, the maximum value corresponding to each quantization block, and the minimum value corresponding to each quantization block, the target minimum value of each quantization block is obtained, which is to achieve the following formula:

[0118]

[0119] Where n is the number of quantization bits, This indicates the theoretical lower limit of the quantization precision for this quantization block. This indicates the operation of retrieving the maximum value.

[0120] (3) Calculate the scaling factor for each quantization block based on the maximum value and the target minimum value, i.e., implement the following formula:

[0121]

[0122] (4) Based on the scaling factor, the elements in each quantization block are quantized and the quantization result is obtained.

[0123]

[0124] in, The round function represents the operation of rounding to the nearest integer. This is a clamping operation used to restrict input values ​​to a closed interval consisting of a specified minimum and maximum value, preventing overflow or underflow of the quantized result. For example, adding 1 at the end is necessary because the sign information of the corresponding element can be stored in the position of 0.

[0125] For example, such as Figure 4 As shown, the matrix computing device can output the first tensor and quantization information, that is, [ For example, the first tensor includes the quantization results. and symbol information Quantization information includes the target minimum value for each quantization block. and the scaling factor corresponding to each quantization block .

[0126] It should be noted that the formulas mentioned above are merely illustrative descriptions of the methods and principles described, and are not intended to limit the scope of this technical solution. In actual hardware implementation, the specific calculation parameters and formula forms can be adjusted according to the encoding format, configuration parameters, and accuracy requirements.

[0127] Figure 5 This is a schematic flowchart illustrating a matrix calculation method provided in at least one embodiment of the present disclosure.

[0128] like Figure 5As shown, the matrix calculation method provided in at least one embodiment of this disclosure includes at least steps S110-S140. For example, this matrix calculation method can be applied to an artificial intelligence processor.

[0129] Step S110: Receive the first matrix, perform a logarithmic field transformation on the first matrix to obtain a logarithmic matrix.

[0130] Step S120: Divide the logarithmic matrix into at least one quantization block according to the quantization granularity.

[0131] Step S130: For each quantization block, calculate the scaling factor for each quantization block.

[0132] Step S140: Receive the first matrix, perform a logarithmic field transformation on the first matrix to obtain a logarithmic matrix.

[0133] For example, in at least one embodiment of the matrix calculation method, step S110 may include: receiving a first matrix, wherein the first matrix includes at least one first submatrix; and performing a logarithmic field transformation on each first submatrix to obtain a logarithmic submatrix corresponding to each first submatrix.

[0134] For example, in at least one embodiment of the matrix calculation method, step S110 may further include: extracting the sign information of each element in each first submatrix; performing absolute value calculation on each element in each first submatrix to obtain an absolute value submatrix; and performing a base-2 logarithmic operation on each element in the absolute value submatrix to obtain a logarithmic submatrix.

[0135] For example, in at least one embodiment of the matrix calculation method, step S130 may include: iteratively calculating the maximum and minimum values ​​of each quantization block in the logarithmic domain based on the logarithmic submatrix; calculating the scaling factor of each quantization block based on the maximum and minimum values ​​corresponding to each quantization block; and performing logarithmic encoding on the elements in each quantization block based on the scaling factor of each quantization block, and obtaining the logarithmic result.

[0136] For example, in at least one embodiment of the matrix calculation method, step S130 may further include: obtaining the target minimum value of each quantization block based on the quantization precision, the maximum value corresponding to each quantization block, and the minimum value corresponding to each quantization block; and calculating the scaling factor corresponding to each quantization block based on the maximum value and the target minimum value, and performing logarithmic quantization encoding based on the scaling factor.

[0137] For example, in at least one embodiment of the matrix calculation method, step S140 may also be included. Step S140 includes: outputting a first tensor and quantization information, wherein the first tensor includes the corresponding quantization result and symbol information, and the quantization information includes the target minimum value of each quantization block and the scaling factor corresponding to each quantization block.

[0138] For example, in at least one embodiment of the matrix computation method, the artificial intelligence processor is configured to deploy a neural network model including a hybrid expert MoE layer, the first matrix including the output result after the expert model in the hybrid expert MoE layer has been computed.

[0139] Figure 6 This is a schematic structural diagram of an artificial intelligence processor provided in at least one embodiment of the present disclosure.

[0140] like Figure 6 As shown, each computational unit includes a tensor kernel. For example, this tensor kernel could be a matrix multiplication kernel (GEMMCore), used to perform matrix multiplication operations using GEMM operators.

[0141] like Figure 6 As shown, each tensor core is provided with a matrix calculation device 401 according to any embodiment of the present disclosure. For example, the matrix calculation device 401 can be disposed in a hardware path within the tensor core responsible for calculating input precision formats such as FP16, BF16, or TF32. Alternatively, the matrix calculation device 401 can also be disposed in a hardware path within the tensor core responsible for calculating low-precision formats (e.g., INT8). For example, the front end of this hardware path can be responsible for inverse quantizing the input low-precision data into a high-precision format for calculation, while the matrix calculation device 401 can be located at the back end of the hardware path to perform quantization operations on the high-precision output.

[0142] The matrix computing device 401 may include a computing cache module and a quantization encoding module.

[0143] For example, the computation cache module can be configured to receive a first matrix, perform a logarithmic field transformation on the first matrix to obtain a logarithmic matrix, and divide the logarithmic matrix into at least one quantization block according to the quantization granularity.

[0144] For example, the quantization encoding module can be configured to calculate the scaling factor for each quantization block; and based on the scaling factor of each quantization block, to perform logarithmic encoding on the elements within each quantization block to generate a logarithmic result.

[0145] For a more detailed description of the matrix computing device 401, please refer to the relevant records of the matrix computing device 100 and matrix computing device 200 as described above, and the repeated parts will not be repeated.

[0146] For more information about the matrix computing device and its interaction with other units in the artificial intelligence processor, please refer to the relevant description of the matrix computing device. Repeated descriptions will not be repeated here.

[0147] In at least one embodiment, the logarithmic process is integrated within the matrix computing device, thereby enabling the logarithmic process to be performed directly within the matrix computing device, achieving seamless integration of the computing pipeline and greatly improving the overall computing throughput.

[0148] For example, the artificial intelligence processor 400 can be a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), a deep learning processing unit (DPU), an accelerated processing unit (APU), a general-purpose graphics processing unit (GPGPU), etc., and the embodiments disclosed herein are not limited thereto. Furthermore, it should be noted that... Figure 6 The components of the AI ​​processor 400 shown are merely exemplary and not limiting. The AI ​​processor 400 may also have other components depending on the actual application requirements.

[0149] Figure 7 This is a schematic block diagram of an electronic device provided in at least one embodiment of the present disclosure; for example, the electronic device can be used to implement the matrix calculation method provided in at least one embodiment of the present disclosure.

[0150] For example, such as Figure 7 As shown, the electronic device 500 includes at least one processor 501 and at least one memory 502. For example, the at least one memory 502 includes one or more computer program modules. For example, the one or more computer program modules are stored in the memory 502 and configured to be executed by the at least one processor 501. The one or more computer program modules include instructions for performing the matrix calculation method described above. When executed by the at least one processor 501, they can perform one or more steps in the matrix calculation method provided in at least one embodiment of this disclosure. The memory 502 and the processor 501 can be interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0151] For example, processor 501 can be a central processing unit (CPU), digital signal processor (DSP), graphics processing unit (GPU), general-purpose graphics processing unit (GPGPU), artificial intelligence (AI) accelerator, or other form of processing unit with data processing and / or program execution capabilities, such as a field-programmable gate array (FPGA); for example, the central processing unit (CPU) can be an x86, ARM, or RISC-V architecture. Processor 501 can be a general-purpose processor or a special-purpose processor, capable of controlling other components in electronic device 500 to perform desired functions.

[0152] For example, memory 502 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc.

[0153] Figure 8 This is a schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0154] The electronic devices in at least one embodiment of this disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, and fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0155] The electronic device includes at least one processor and a memory. The processor may be referred to as processing device 601 as described below, and the memory may include at least one of ROM 602, RAM 603, and storage device 608 as described below. The memory is used to store programs for performing the methods described in the various method embodiments above; the processor is configured to execute the programs stored in the memory. The processor includes a matrix computing device according to any embodiment of this disclosure, for example, it may include a central processing unit (CPU) or other forms of processing unit having data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions.

[0156] like Figure 8 As shown, the electronic device 600 may include a processing unit 601 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in ROM 602 or a program loaded from storage device 608 into RAM 603. RAM 603 also stores various programs and data required for the operation of the electronic device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interfaces are also connected to bus 604.

[0157] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, displays, speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0158] In particular, according to at least one embodiment of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, at least one embodiment of this disclosure includes a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the matrix calculation method of at least one embodiment of this disclosure is performed.

[0159] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In at least one embodiment of this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In at least one embodiment of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, radio frequency (RF), etc., or any suitable combination thereof.

[0160] The aforementioned computer-readable medium may be included in the aforementioned electronic device 600; or it may exist independently and not assembled into the electronic device 600.

[0161] Figure 9 This is a schematic block diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure.

[0162] For example, such as Figure 9 As shown, a non-transient computer-readable storage medium 700 stores a computer-readable instruction 701, which, when executed by at least one processor, performs the matrix calculation method described above.

[0163] For example, the storage medium may include a memory card for a smartphone, a storage component for a tablet computer, a hard drive for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media. For example, the readable storage medium may also be... Figure 7 The memory 502 in the memory is described in the foregoing content and will not be repeated here.

[0164] Although the present disclosure has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to the embodiments of the present disclosure, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present disclosure are within the scope of protection claimed by the present disclosure.

[0165] The following points should be noted regarding this disclosure:

[0166] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0167] (2) For clarity, the thickness of layers or regions in the drawings used to describe embodiments of the present disclosure is enlarged or reduced, i.e., these drawings are not drawn to actual scale.

[0168] (3) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

Claims

1. A matrix calculation device, characterized in that, An application is made in an artificial intelligence processor, wherein the artificial intelligence processor includes multiple computing units, each computing unit includes a tensor core, and a matrix computing device is disposed in the tensor core, the matrix computing device comprising: The compute cache module is configured as follows: Receive a first matrix, perform a logarithmic field transformation on the first matrix to obtain a logarithmic matrix, wherein the first matrix includes the output result of the matrix multiplication calculation in the tensor kernel; and Based on the quantization granularity, the logarithmic matrix is ​​divided into at least one quantization block; The quantization encoding module is configured as follows: For each quantization block, calculate the scaling factor for each quantization block; and Based on the scaling factor of each quantization block, the elements within each quantization block are log-quantized to generate a log-quantization result.

2. The matrix computing device according to claim 1, characterized in that, The computation cache module includes: The logarithm calculation module is configured as follows: Receive the first matrix, wherein the first matrix includes at least one first submatrix; and Perform a logarithmic field transformation on each of the first submatrices to obtain the logarithmic submatrix corresponding to each of the first submatrices.

3. The matrix computing device according to claim 2, characterized in that, The logarithm calculation module is further configured to, Extract the symbol information of each element in each of the first submatrices; Perform absolute value calculation on each element in each first submatrix to obtain an absolute value submatrix; as well as Perform a base-2 logarithmic operation on each element of the absolute value submatrix to obtain the logarithmic submatrix.

4. The matrix computing device according to claim 3, characterized in that, The computation cache module also includes: The local value acquisition module is configured as follows: Based on the logarithmic submatrix, the maximum and minimum values ​​of each quantization block in the logarithmic domain are calculated iteratively; Calculate the scaling factor for each quantization block based on its maximum and minimum values; and Based on the scaling factor of each quantization block, the elements in each quantization block are logarithmically encoded, and the logarithmization result is obtained.

5. The matrix calculation device according to claim 4, characterized in that, The quantization encoding module is further configured to, Based on the quantization precision, the maximum value corresponding to each quantization block, and the minimum value corresponding to each quantization block, the target minimum value of each quantization block is obtained; as well as Based on the maximum value and the target minimum value, calculate the scaling factor corresponding to each quantization block, and perform the logarithmic quantization encoding based on the scaling factor.

6. The matrix computing device according to claim 5, characterized in that, The matrix computing device is configured to output a first tensor and quantization information. The first tensor includes the logarithmic quantization result and the symbol information, and the quantization information includes the target minimum value of each quantization block and the scaling factor corresponding to each quantization block.

7. The matrix computing device according to claim 1, characterized in that, The matrix calculation device further includes: The quantization control unit is configured as follows: Choose whether to perform logarithmic calculations on the first matrix; In response to performing logarithmic calculations on the first matrix, the first matrix is ​​transmitted to the calculation cache module; in response to not performing logarithmic calculations on the first matrix, the first matrix is ​​transmitted outside the matrix calculation device.

8. The matrix computing apparatus according to any one of claims 1-7, characterized in that, The artificial intelligence processor is configured to deploy a neural network model including a hybrid expert MoE layer, wherein the first matrix includes the output results after the expert model in the hybrid expert MoE layer has been computed.

9. A matrix calculation method, characterized in that, An artificial intelligence processor is applied to an AI processor, wherein the AI ​​processor includes multiple computing units, each computing unit including a tensor kernel, and the matrix calculation method is executed in the tensor kernel, the matrix calculation method including: Receive a first matrix, perform a logarithmic field transformation on the first matrix to obtain a logarithmic matrix, wherein the first matrix includes the output result of the matrix multiplication calculation in the tensor kernel; Based on the quantization granularity, the logarithmic matrix is ​​divided into at least one quantization block; For each quantization block, calculate the scaling factor for each quantization block; and Based on the scaling factor of each quantization block, the elements within each quantization block are log-quantized to generate a log-quantization result.

10. An electronic device, characterized in that, The electronic device includes: Memory stores computer-executable instructions non-transiently; The processor is configured to run computer-executable instructions. The computer-executable instructions are executed by the processor to implement the matrix calculation method according to claim 9.

11. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the matrix calculation method according to claim 9.

Images