Tensor core, processor, data processing method, electronic device, and storage medium

Abstract

A tensor core, a processor, a data processing method, an electronic device and a storage medium are applied to the field of tensor processing. The tensor core comprises a first point multiplication unit and a scaling factor matrix multiplication processing module, the scaling factor matrix multiplication processing module comprises a second point multiplication unit, the first point multiplication unit and the second point multiplication unit support point multiplication operations of different floating point precisions, the tensor core is configured to receive a first tensor, a second tensor, a scaling factor of the first tensor, a scaling factor of the second tensor, an offset term of the first tensor and an offset term of the second tensor, perform a matrix multiplication operation using a scaling factor and an offset term by using the first point multiplication unit and the scaling factor matrix multiplication processing module, and obtain a matrix multiplication operation result of the first tensor and the second tensor. By multiplexing the point multiplication units of different precisions in the tensor core to perform the matrix multiplication operation using the scaling factor, the additional hardware area cost is reduced, and the existing hardware resources are fully multiplexed.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to a tensor core, a processor, a data processing method, an electronic device and a non-transitory computer-readable storage medium. BACKGROUND

[0002] Floating point quantization refers to format conversion of a high-precision floating point number format (e.g., FP16, BF16, FP32, etc.) to a low-precision floating point number format (e.g., FP4, FP8, etc.). The low-precision floating point number format can greatly improve the calculation efficiency, especially the general matrix multiplication (GEMM) operation required by an artificial intelligence (AI) model. However, the low-precision matrix multiplication will cause a loss of precision, thereby reducing the prediction and generation effect of the AI model. Therefore, in the low-precision calculation process of the AI model, a corresponding scaling factor is usually calculated from the current data in the process of floating point quantization to scale the expression range of the data, thereby reducing the loss of precision of the calculation result.

[0003] At present, for the matrix multiplication operation using the scaling factor, additional multipliers and adders need to be added inside the tensor core, which brings an additional hardware area cost. SUMMARY

[0004] At least one embodiment of the present application provides a tensor core, comprising a first point multiplication unit, a scaling factor matrix multiplication processing module, and a cross-precision point multiplication unit hardware path, the scaling factor matrix multiplication processing module comprising a second point multiplication unit, the first point multiplication unit and the second point multiplication unit being configured to support point multiplication operations of different floating point precisions, the tensor core being configured to receive a first tensor, a second tensor, a scaling factor of the first tensor, a scaling factor of the second tensor, an offset term of the first tensor, and an offset term of the second tensor, perform a matrix multiplication operation using scaling factors and offset terms by using the first point multiplication unit and the scaling factor matrix multiplication processing module to obtain a matrix multiplication operation result of the first tensor and the second tensor, wherein the first tensor and the second tensor are both of a first floating point precision, the offset term of the first tensor is used to adjust the numerical value distribution of elements in the first tensor, and the offset term of the second tensor is used to adjust the numerical value distribution of elements in the second tensor; wherein, in the process of performing the matrix multiplication operation using scaling factors and offset terms, the first point multiplication unit is configured to perform a matrix multiplication operation of the first floating point precision of the first tensor and the second tensor to obtain a first multiplication result, the cross-precision point multiplication unit hardware path is configured to transmit the first multiplication result to the scaling factor matrix multiplication processing module, and the scaling factor matrix multiplication processing module is configured to determine the matrix multiplication operation result by using the second point multiplication unit according to the first multiplication result, the scaling factor of the first tensor, the scaling factor of the second tensor, the offset term of the first tensor, and the offset term of the second tensor.

[0005] For example, in the tensor core provided by at least one embodiment of the present application, the scaling factor matrix multiplication processing module further comprises a first submodule, a second submodule, a third submodule, and an offset correction hardware path, the first submodule is configured to determine a second multiplication result of the scaling factor of the first tensor and the scaling factor of the second tensor, the second submodule is configured to determine a third multiplication result of the offset term of the first tensor and the offset term of the second tensor, and the third submodule is configured to determine a first correction result and a second correction result for offset term correction; the second point multiplication unit is configured to determine the matrix multiplication operation result according to the first multiplication result, the second multiplication result, the third multiplication result, the first correction result, and the second correction result, and the offset correction hardware path is configured to transmit the second multiplication result, the third multiplication result, the first correction result, and the second correction result to the second point multiplication unit.

[0006] For example, in the tensor core provided by at least one embodiment of the present application, the second point multiplication unit supports point multiplication operations of a second floating point precision higher than the first floating point precision.

[0007] For example, in the tensor core provided in at least one embodiment, the first tensor is divided into T first tensor blocks in the column direction, each first tensor block has a shape size of m x k, the second tensor is divided into T second tensor blocks in the row direction, each second tensor block has a shape size of k x n, and the first point multiplication unit is configured to perform a matrix multiplication operation between the T first tensor blocks and the T second tensor blocks to obtain T matrix multiplication intermediate results, wherein the first multiplication result includes the T matrix multiplication intermediate results, and T, m, k, and n are positive integers.

[0008] For example, in the tensor core provided in at least one embodiment, the scaling factor of the first tensor is divided into T first vectors in the column direction, each first vector has a shape size of m x 1, the scaling factor of the second tensor is divided into T second vectors in the row direction, each second vector has a shape size of 1 x n, and the first sub-module is configured to perform an outer product operation between the T first vectors and the T second vectors to obtain T first outer product operation results, wherein the second multiplication result includes the T first outer product operation results, and T, m, and n are positive integers.

[0009] For example, in the tensor core provided in at least one embodiment, the offset term of the first tensor is divided into T / τ third vectors in the column direction, each third vector has a shape size of m x 1, the offset term of the second tensor is divided into T / τ fourth vectors in the row direction, each fourth vector has a shape size of 1 x n, and the second sub-module is configured to perform an outer product operation between the T / τ third vectors and the T / τ fourth vectors to obtain T / τ second outer product operation results, wherein the third multiplication result includes the T / τ second outer product operation results, T, m, and n are positive integers, and τ is a positive integer less than or equal to T.

[0010] For example, in the tensor core provided in at least one embodiment, the second point multiplication unit is configured to obtain a first matrix and a second matrix, perform a point multiplication operation between the first multiplication result and the second multiplication result, a point multiplication operation between the first correction result and the first matrix, a point multiplication operation between the second correction result and the first matrix, and a point multiplication operation between the third multiplication result and the second matrix to obtain the matrix multiplication operation result, wherein the first matrix has a size of m x n and all elements in the first matrix have a value of 1, and the second matrix has a size of m x n and all elements in the second matrix have a value of τ x k, wherein m, n, τ, and k are positive integers.

[0011] For example, in the tensor core provided by at least one embodiment of the present application, the first multiplication result includes T intermediate results of matrix multiplication each having a shape size of m x n, the second multiplication result includes T first outer product operation results each having a shape size of m x n, the third multiplication result includes T / τ second outer product operation results each having a shape size of m x n, the first correction result includes T / τ first coset results each having a shape size of m x n, and the second correction result includes T / τ second coset results each having a shape size of m x n, T is a positive integer, and a plurality of the second point multiplication units are configured to perform a plurality of first operations in parallel to obtain the matrix multiplication operation result through m x n first operations, wherein the first operation includes performing a first point multiplication operation, a second point multiplication operation, a third point multiplication operation, and a fourth point multiplication operation, and determining the sum of the operation results of the first point multiplication operation, the second point multiplication operation, the third point multiplication operation, and the fourth point multiplication operation as the operation result of the first operation, wherein the operation result of the first operation is an element in the matrix multiplication operation result, wherein the first point multiplication operation includes performing a point multiplication operation between an element in the i-th row and the j-th column of the T intermediate results of matrix multiplication and an element in the i-th row and the j-th column of the T first outer product operation results, the second point multiplication operation includes performing a point multiplication operation between an element in the i-th row and the j-th column of the T / τ first matrices and an element in the i-th row and the j-th column of the T / τ first coset results, the third point multiplication operation includes performing a point multiplication operation between an element in the i-th row and the j-th column of the T / τ first matrices and an element in the i-th row and the j-th column of the T / τ second coset results, and the fourth point multiplication operation includes performing a point multiplication operation between an element in the i-th row and the j-th column of the T / τ second matrices and an element in the i-th row and the j-th column of the T / τ second outer product operation results, wherein i is a positive integer less than or equal to m, and j is a positive integer less than or equal to n.

[0012] For example, in the tensor core provided in at least one embodiment of the present application, the second point multiplication unit is configured to perform a first operation at a single time, and when performing the first operation, the second point multiplication unit includes performing a point multiplication operation between a first point multiplication object and a second point multiplication object to obtain an operation result of the first operation, wherein the first point multiplication object includes T+3×T / τ elements, the T+3×T / τ elements include an i-th row and j-th column element of each matrix multiplication intermediate result in the T matrix multiplication intermediate results, 2×T / τ 1s, and T / τ τ×k, the second point multiplication object includes T+3×T / τ elements, wherein the T+3×T / τ elements include an i-th row and j-th column element of each first outer product operation result in the T first outer product operation results, an i-th row and j-th column element of each first coset result in the T / τ first coset results, an i-th row and j-th column element of each second coset result in the T / τ second coset results, and an i-th row and j-th column element of each second outer product operation result in the T / τ second outer product operation results, and the operation result of the first operation is used as an i-th row and j-th column element in the matrix multiplication operation result.

[0013] For example, in the tensor core provided in at least one embodiment of the present application, a minimum point multiplication number N of a single point multiplication operation supported by the second point multiplication unit is greater than or equal to T+3×T / τ, N is a positive integer, when performing the point multiplication operation between the first point multiplication object and the second point multiplication object, the second point multiplication unit includes performing N point multiplication calculation operations and determining a sum of operation results of the N point multiplication calculation operations, and in response to N being greater than T+3×T / τ, filling 0 in parts of the first point multiplication object and the second point multiplication object other than the T+3×T / τ elements already included in the first point multiplication object and the second point multiplication object respectively, to expand the first point multiplication object and the second point multiplication object to include N elements, and performing the point multiplication operation between the expanded first point multiplication object and the expanded second point multiplication object by the second point multiplication unit.

[0014] In at least one embodiment of the present application, a processor is provided, which includes the tensor core described in any embodiment of the present application.

[0015] The data processing method provided in at least one embodiment of the present application is applied to a tensor core, wherein the tensor core includes a first point multiplication unit, a scaling factor matrix multiplication processing module, and a cross-precision point multiplication unit hardware channel, the first point multiplication unit and the second point multiplication unit are configured to support point multiplication operations of different floating-point precisions, and the data processing method includes the following steps: receiving a first tensor, a second tensor, a scaling factor of the first tensor, a scaling factor of the second tensor, an offset term of the first tensor, and an offset term of the second tensor; performing a matrix multiplication operation using scaling factors and offset terms by using the first point multiplication unit and the scaling factor matrix multiplication processing module to obtain a matrix multiplication operation result of the first tensor and the second tensor, wherein the first tensor and the second tensor are both of a first floating-point precision, the offset term of the first tensor is used to adjust the numerical value distribution of elements in the first tensor, and the offset term of the second tensor is used to adjust the numerical value distribution of elements in the second tensor; and performing the matrix multiplication operation using scaling factors and offset terms by using the first point multiplication unit and the scaling factor matrix multiplication processing module to obtain the matrix multiplication operation result of the first tensor and the second tensor includes the following steps: performing a matrix multiplication operation of the first floating-point precision of the first tensor and the second tensor by using the first point multiplication unit to obtain a first multiplication result; transmitting the first multiplication result to the scaling factor matrix multiplication processing module by using the cross-precision point multiplication unit hardware channel; and determining the matrix multiplication operation result by using the second point multiplication unit according to the first multiplication result, the scaling factor of the first tensor, the scaling factor of the second tensor, the offset term of the first tensor, and the offset term of the second tensor.

[0016] For example, in the data processing method provided in at least one embodiment of the present application, determining the matrix multiplication operation result by using the second point multiplication unit according to the first multiplication result, the scaling factor of the first tensor, the scaling factor of the second tensor, the offset term of the first tensor, and the offset term of the second tensor includes the following steps: determining a second multiplication result of the scaling factor of the first tensor and the scaling factor of the second tensor; determining a third multiplication result of the offset term of the first tensor and the offset term of the second tensor; determining a first correction result and a second correction result for offset term correction; and determining the matrix multiplication operation result by using the second point multiplication unit according to the first multiplication result, the second multiplication result, the third multiplication result, the first correction result, and the second correction result.

[0017] The electronic device provided in at least one embodiment of the present application includes a memory that non-transiently stores computer executable instructions, and a processor configured to run the computer executable instructions, wherein the computer executable instructions are run by the processor to implement the data processing method according to any one of the embodiments of the present application.

[0018] The non-transitory computer-readable storage medium stores computer executable instructions. When the computer executable instructions are executed by a processor, a data processing method according to any one of the embodiments of the present application is implemented.

[0019] The present application uses different precision point multiplication units in the tensor core to perform the matrix multiplication operation using the scaling factor, reduces the additional hardware area cost, fully utilizes the existing hardware resources, reduces the multiplier setting and does not need to additionally set the adder, reduces the hardware area cost, optimizes the system performance, reduces the hardware cost and power consumption, and improves the chip integration. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only related to some embodiments of the present disclosure, and not limited to the present disclosure.

[0021] Figure 1A It is a schematic structural diagram of a general-purpose graphics processor (GPGPU);

[0022] Figure 1A It is a schematic structural diagram of a tensor core;

[0023] Figure 2 It is a schematic structural diagram of the tensor core provided by at least one embodiment of the present disclosure;

[0024] Figure 3 It is a schematic structural diagram of the scaling factor matrix multiplication processing module 102 provided by at least one embodiment of the present disclosure;

[0025] Figure 4A It is a schematic diagram of the first tensor and the second tensor provided by at least one embodiment of the present disclosure;

[0026] Figure 4B It is a schematic diagram of the scaling factor of the first tensor and the scaling factor of the second tensor provided by at least one embodiment of the present disclosure;

[0027] Figure 4C It is a schematic diagram of the offset term of the first tensor and the offset term of the second tensor provided by at least one embodiment of the present disclosure;

[0028] Figure 5A It is a schematic diagram of the first point multiplication object and the second point multiplication object provided by at least one embodiment of the present disclosure;

[0029] Figure 5B It is a schematic diagram of the first point multiplication object and the second point multiplication object provided by at least one embodiment of the present disclosure;

[0030] Figure 6 A schematic structural diagram of a processor provided for at least one embodiment of the present disclosure is shown in FIG. 1;

[0031] Figure 7 A schematic flow chart of a data processing method provided for at least one embodiment of the present disclosure is shown in FIG. 2;

[0032] Figure 8 A schematic block diagram of an electronic device provided for an embodiment of the present disclosure is shown in FIG. 3;

[0033] Figure 9 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure is shown in FIG. 4. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. Based on the described embodiments of the present disclosure, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present disclosure.

[0035] Unless otherwise defined, technical terms or scientific terms used in the present disclosure should be understood as having the common meaning in the field of the present disclosure to which they belong. The terms “first”, “second” and similar terms used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. The terms “include”, “contain” and similar terms mean that the components or objects before the terms encompass the components or objects listed after the terms and their equivalents, and do not exclude other components or objects. The terms “connect” or “connected” and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms “up”, “down”, “left”, “right” and the like only represent relative positional relationships, which can change accordingly when the absolute positions of the described objects change. In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits the detailed description of some known functions and known components.

[0036] A floating point (FP) is mainly used to represent a decimal number, and is usually composed of three parts, i.e., a sign bit, an exponent part and a mantissa part. The exponent part can also be referred to as a characteristic part. For example, a floating point V can be represented in the following form:

[0037]

[0038] wherein, the sign bit s can be 1 bit, determining whether the floating-point number V is negative or positive; M represents the mantissa part, which can include multiple bit positions, is in the form of a binary fraction, and defines the precision of the floating-point number; E represents the exponent (also known as the characteristic value), which is used to weight the floating-point number and reflects the position of the decimal point in the floating-point number V, and defines the value range of the floating-point number.

[0039] Traditional floating-point numbers usually include three formats, namely half-precision floating-point numbers (FP16), single-precision floating-point numbers (FP32) and double-precision floating-point numbers (FP64), and the exponent part and the mantissa part of them have different bit numbers.

[0040] AI accelerators and the like have been widely used in deep learning model training. For the convolution operation commonly used in deep learning models, special optimization is made in software and hardware design to accelerate the calculation. For example, various data formats of floating-point numbers are developed for optimization in the fields of artificial intelligence or deep learning, such as BF16 (brain floating point 16, bit width of 16 bits), BF24 (brain floating point 24, bit width of 24 bits), TF32 (Tensor Float 32, bit width of 19 bits) and the like. These data formats can greatly reduce the required operation resources and power consumption, especially for matrix multiplication or convolution multiplication operations. In addition, the processor also supports some conventional floating-point number types, such as half-precision floating-point numbers (FP16, bit width of 16 bits) or single-precision floating-point numbers (FP32, bit width of 32 bits) and the like.

[0041] Figure 1A A schematic structural diagram of a general-purpose graphics processor (GPGPU).

[0042] As shown in Figure 1A , the general-purpose graphics processor is actually an array of programmable multi-processors, for example, the programmable multi-processors can be streaming processor clusters (SPC), for example, including streaming processor cluster 1,..., streaming processor cluster M as shown in Figure 1A , M is a positive integer greater than 1. In the general-purpose graphics processor, one streaming processor cluster processes one computing task, or multiple streaming processor clusters process one computing task. Multiple streaming processor clusters share data through a global cache or a global memory.

[0043] As shown in Figure 1A , taking streaming processor cluster 1 as an example, one streaming processor cluster includes multiple computing units, for example Figure 1AThe computation unit is 1, computation unit 2, ..., computation unit N, where N is a positive integer.

[0044] A computing unit comprises multiple cores (also called computing kernels or computing cores). Each computing core includes an arithmetic logic unit (ALU), a floating-point unit, etc., and is used to execute specific computational tasks. In addition, the computing unit also includes registers (e.g., ... Figure 1A The register file and shared memory in a computing unit are used to store source and destination data related to computing tasks in a hierarchical manner. The shared memory in a computing unit is used to share data between the cores of that computing unit.

[0045] like Figure 1A As shown, each computational unit also provides a tensor core for performing tensor-related computations, such as matrix multiplication operations using the GEMM operator. Tensors are a crucial data structure in deep learning; they are high-dimensional generalizations of scalars, vectors, and matrices. Tensor operations are commonly used in the training and inference of current deep learning models, and tensor cores can accelerate matrix multiplication. Tensor cores across multiple computational units can be uniformly scheduled and controlled.

[0046] For example, each computing unit can also provide a vector core engine. Figure 1A (Not shown in the image). The vector computation kernel is used to perform vector-related computations, such as vector-related arithmetic and logical operations, like accumulation, reduction, and regular addition, subtraction, multiplication, and division.

[0047] In parallel computing, computational tasks are typically executed by multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block distribution module. Figure 1A (Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0048] In each computing unit, the thread beam scheduling / distribution module ( Figure 1AThe thread blocks are scheduled, allocated, and dispatched to the multiple compute cores of the compute unit to run the thread blocks. According to the number of compute cores in the compute unit, the multiple thread blocks in one thread block can be executed simultaneously or in time-sharing. The multiple threads in each thread block execute the same instructions. The memory execution instructions are transmitted to the shared memory in the compute unit or further transmitted to the intermediate level cache (such as a vector operation unit cache area) or a global cache or a global memory for read-write operations and the like.

[0049] Low-precision matrix multiplication is gradually widely used in the training and inference of AI large models, which can bring huge performance benefits with acceptable precision loss. In GPU (Graphics Processing Unit) or GPGPU, matrix multiplication is usually executed by tensor cores in hardware. Among them, the low-precision tensor core is several times the power of the high-precision tensor core, which brings higher computing efficiency. In addition, the data amount of the low-precision tensor itself is also several times less than that of the high-precision tensor, which brings higher data transmission efficiency. Therefore, the end-to-end efficiency brought by low-precision tensor calculation is almost several times improved. For example, the tensor core power of FP4 can be 2-8 times that of FP8, 4-16 times that of FP16 / BF16, or even higher. And the data amount is 1 / 2 of FP8 and 1 / 4 of FP16 / BF16.

[0050] The point multiplication unit is a hardware structure in the tensor core for performing point multiplication operations, which can efficiently calculate the point multiplication operation of each element in two vectors and accumulate the operation results of the point multiplication operation to obtain the point multiplication operation result. Through the cooperative work of multiple point multiplication units, the tensor core realizes large-scale matrix multiplication operation. For example, the point multiplication units are organized into an array to efficiently perform matrix multiplication.

[0051] Although the efficiency of low-precision matrix calculation is high, the precision loss will become more and more obvious as the bit decreases. In order to reduce the precision loss, low-precision matrix multiplication introduces a scaling factor to maximize the numerical representation ability of low-precision tensors. For example, low-precision matrix multiplication includes matrix multiplication for low-precision (such as FP4, FP8, etc.) tensors.

[0052] For low-precision matrix multiplication, for example, D=A'×B'+C', A', B', C', and D are all high-precision tensors, and the general matrix multiplication operation using the scaling factor can be described as:

[0053] D=(α⊙A)×(β⊙B)+σ⊙C or D=(α β)⊙(A×B)+σ⊙C

[0054] Wherein, ⊙ represents element multiplication, denotes outer product, x denotes matrix multiplication, a is a scaling factor of tensor A', b is a scaling factor of tensor B', s is a scaling factor of tensor C', and g is a scaling factor of tensor D. The quantized tensor A, the quantized tensor B, the quantized tensor C, and the quantized tensor D are low-precision tensors after floating-point quantization of the tensor A', the tensor B', the tensor C', and the tensor D, respectively. For example, the data sizes of the respective parameters are as follows:

[0055] A: [m, Txk], B: [Txk, n], C: [m, n], D: [m, n], a: [m, T], b: [T, n], s: [m, r] or [r, n], and g: [m, r] or [r, n].

[0056] wherein m, n, s, k, and r are positive integers. For any row in the tensor A', each k tensor elements in the row share one scaling factor parameter, and for any column in the tensor B', each k tensor elements in the column share one scaling factor parameter.

[0057] In order to maintain the performance advantage brought by low-precision matrix calculation, the calculation of the scaling factor and the element multiplication of the low-precision tensor are placed together inside the tensor core. As shown in the following formula 1, the tensor core performs one round of matrix multiplication operation of A t and B t (the matrix multiplication operation includes m x n point multiplication operations, and each point multiplication operation includes k element point multiplication calculation operations), and the result needs to be multiplied by the scaling factor a t x b t element by element, which is performed T times in total:

[0058] (Formula 1)

[0059] t takes 0, 1, 2,..., T-1 in turn. The data size of A t is [m, k], indicating the (t+1)th first tensor block in T first tensor blocks obtained by dividing the matrix A in the column direction, the data size of B t is [k, n], indicating the (t+1)th second tensor block in T second tensor blocks obtained by dividing the matrix B in the row direction. The data size of a t is [m, 1], indicating the (t+1)th first vector in T first vectors obtained by dividing the scaling factor a of the tensor A in the column direction, and the data size of b t is [1, n], indicating the (t+1)th second vector in T second vectors obtained by dividing the scaling factor b of the tensor B in the column direction.

[0060] Here A t x B tThe multiplication operation of is responsible for the point multiplication unit in the tensor core, while t ×β t and (α t ×β t ) (A t ×B t ) all need additional multipliers to perform. In addition Additional adders are also needed to perform. Therefore, the current matrix multiplication using scaling factors needs to introduce more additional adders and multipliers inside the tensor core, which brings additional hardware area cost, increases hardware cost and power consumption, and reduces processor performance and reliability.

[0061] For example, in some cases, the values corresponding to the same scaling factor parameter in a row or a column of the tensor are highly close, and it is difficult to maximize the potential numerical expression range by relying on the scaling factor alone, resulting in a loss of accuracy. Therefore, in this case, the accuracy loss can be ensured within an acceptable range by introducing an additional offset term, thereby improving the numerical expression capability of the low-precision tensor.

[0062] The matrix multiplication operation using scaling factors and offset terms can be represented as the following formula 2:

[0063]

[0064] wherein, , a, b, c are offset terms of tensors A, B and C, respectively.

[0065] For example, the offset terms a, b, and c can be provided by the user or determined according to the numerical distribution of the corresponding tensor. For example, the offset terms can be determined in the following manner:

[0066]

[0067] or

[0068]

[0069] or

[0070] Here, max represents the maximum value function, min represents the minimum value function, mean represents the average value, and median represents the median. Of course, the present disclosure is not limited thereto, and other ways of determining the offset term can also be used.

[0071] For example, the relevant data size can be represented as: , offset term a: [m, T / τ], offset term b: [T / τ, n], τ = 1, 2, …, or T, offset term c: [m, r] or [r, n].

[0072] For example, one offset parameter can be shared by multiple scaling factor parameters, such as when τ = T, one offset parameter is used for all scaling factor parameters, and when τ = 1, one offset parameter is used for each scaling factor parameter.

[0073] When the tensor core performs matrix multiplication using scaling factors and considering offset parameters, the resulting matrix multiplication result is Instead of the actual required matrix multiplication result D, therefore, the matrix multiplication result directly obtained by the tensor core needs to be corrected for offset parameters.

[0074] The matrix multiplication result corrected for offset parameters is represented as

[0075] Equation 3

[0076] After derivation, the matrix multiplication operation with offset parameter correction is represented as

[0077] Equation 4

[0078] The data size thereof can be represented as

[0079] [m, n] = [m, n] + [m, 1] ([1, n] ⊙ [1, n]) + ([m, 1] ⊙ [m, 1]) [1, n] + [1, 1] × [m, 1] [1, n]

[0080] Alternatively, the matrix multiplication result corrected for offset parameters is represented as

[0081] Equation 5

[0082] The data size thereof can be represented as

[0083] [m, n] = [m, n] + ([m, 1] [1, n]) ⊙ [1, n] + ([m, 1] [1, n]) ⊙ [m, 1] + [1, 1] × [m, 1] [1, n]

[0084] In Equation 4 and Equation 5, represents the tensor the result of summation calculation of all elements in each column, with a shape size of [1, n], represents the tensor the result of summation calculation of all elements in each row, with a shape size of [m, 1]. ​

[0085] It should be noted that formula 4 and formula 5 are two possible configuration modes or multiplication optimization modes, wherein the multiplication amount in formula 4 is smaller, and the multiplication and addition in formula 5 can be operated in parallel, and a person skilled in the art can select as needed.

[0086] For example, formula 4 can be specifically expressed as formula 6 as follows:

[0087]

[0088]

[0089] For example, formula 5 can be specifically expressed as formula 7 as follows:

[0090]

[0091] Figure 1B is a schematic structural diagram of a tensor core. As shown in formula 6, Figure 1B the multiplication operation of is responsible for the point multiplication unit in the tensor core, and and all need additional multipliers to execute, that is, Figure 1B the point multiplication and accumulation module in executes; in addition, Figure 1B an additional adder is also needed to execute, which is also executed by the point multiplication and accumulation module in

[0092] As shown in formula 6 and formula 7, in addition to , the matrix multiplication operation using the scaling factor and the offset term also needs to be calculated by the vector calculation core to correct the offset, and in formula 4, the vector calculation core also needs to complete .

[0093] Therefore, as described above, the current tensor core does not support these additional vector calculations (such as determining the scaling factor and the offset term), and additional multipliers and adders are needed to implement the calculation of the scaling factor, and the vector calculation core is also needed to complete the calculation of the offset correction. In addition, since the vector calculation core is used to complete the calculation of the offset correction, but the vector calculation core also needs to execute other vector-related operators, the calculation of the offset correction will occupy the calculation resources of other vector-related operators, resulting in performance degradation; and the path transmission between the tensor core and the vector calculation core When the tensor data is transmitted, other tensor data transmission may be blocked, for example, the cache of the shared memory is occupied, and transmission bandwidth on a path needs to be competed for with other data, which causes the overall throughput to decrease and the delay to be high, reducing the performance benefit brought by the low-precision matrix multiplication. As shown in formula 6 or formula 7, due to the further addition of the offset term, more multiplication and addition operations are brought, and more multipliers and adders need to be introduced when the above calculation is performed in the tensor core, which brings greater hardware area overhead.

[0094] In at least one embodiment of the present disclosure, a tensor core is provided, which includes a first point multiplication unit and a scaling factor matrix multiplication processing module, the scaling factor matrix multiplication processing module including a second point multiplication unit, the first point multiplication unit and the second point multiplication unit being configured to support point multiplication operations of different floating-point precisions, the tensor core being configured to receive a first tensor, a second tensor, a scaling factor of the first tensor, a scaling factor of the second tensor, an offset term of the first tensor and an offset term of the second tensor, and perform a matrix multiplication operation using the scaling factor and the offset term by using the first point multiplication unit and the scaling factor matrix multiplication processing module to obtain a matrix multiplication operation result of the first tensor and the second tensor, wherein the first tensor and the second tensor are both of a first floating-point precision, and the offset term is used to adjust the element value distribution in the first tensor; and wherein in the process of performing the matrix multiplication operation using the scaling factor and the offset term, the first point multiplication unit is configured to perform a matrix multiplication operation of the first floating-point precision of the first tensor and the second tensor to obtain a first multiplication result, and the scaling factor matrix multiplication processing module is configured to determine the matrix multiplication operation result by using the second point multiplication unit according to the first multiplication result, the scaling factor of the first tensor, the scaling factor of the second tensor, the offset term of the first tensor and the offset term of the second tensor.

[0095] In at least one embodiment, the present disclosure introduces a scaling factor matrix multiplication processing module inside the tensor calculation core, so that when calculating the asymmetric (with offset) quantization matrix multiplication, the vector unit does not need to rely on the accumulation across different levels of memory, thus greatly reducing the delay and improving the overall efficiency; in addition, a cross-precision point multiplication unit hardware path is set to enable reuse of different precision point multiplication units in the tensor core to perform the matrix multiplication operation using the scaling factor and the offset term, reduce the additional hardware area cost, fully reuse the existing hardware resources, reduce the setting of multipliers and the need for additional adders, reduce the hardware area cost, optimize the system performance, reduce the hardware cost and power consumption, and improve the chip integration.

[0096] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.

[0097] Figure 2 A schematic structural diagram of the tensor core provided in at least one embodiment of the present disclosure is shown.

[0098] As Figure 2As shown, the tensor core 100 comprises a first point multiplication unit 101, a scaling factor matrix multiplication processing module 102, the scaling factor matrix multiplication processing module 102 comprises a second point multiplication unit 103, the first point multiplication unit 101 and the second point multiplication unit 103 support point multiplication operations of different floating-point number precisions.

[0099] It should be noted that, Figure 2 The first point multiplication unit 101 and the second point multiplication unit 103 shown are schematic, and a plurality of first point multiplication units 101 and a plurality of second point multiplication units 103 can be included in the tensor core, and the plurality of first point multiplication units 101 and the plurality of second point multiplication units 103 can be arranged in an array.

[0100] As Figure 2 As shown, the tensor core 100 is also provided with a cross-precision point multiplication unit hardware path between the first point multiplication unit 101 and the scaling factor matrix multiplication processing module 102, as shown by the black arrow, the cross-precision point multiplication unit hardware path is configured to transmit the first multiplication result Y to the scaling factor matrix multiplication processing module 102. If the tensor core comprises a plurality of first point multiplication units 101, the plurality of first point multiplication units 101 each have a respective cross-precision point multiplication unit hardware path for transmitting the point multiplication result calculated by each first point multiplication unit 101 to the scaling factor matrix multiplication processing module for subsequent processing.

[0101] For example, the first point multiplication unit 101 supports point multiplication operations of a first floating-point number precision, the second point multiplication unit 103 supports point multiplication operations of a second floating-point number precision, and the first floating-point number precision and the second floating-point number precision are different.

[0102] For example, the first floating-point number precision is lower than the second floating-point number precision, and the second point multiplication unit supports point multiplication operations of a second floating-point number precision higher than the first floating-point number precision.

[0103] For example, the first floating-point number precision is FP4, and the second floating-point number precision includes FP8 and floating-point number precisions higher than FP8 (such as FP16, FP32, etc.), for example, in this embodiment, the second point multiplication unit 103 supports point multiplication operations of multiple floating-point number precisions, but does not support point multiplication operations of the first floating-point number precision. Of course, the present disclosure is not limited to this, and the second point multiplication unit 103 can only support point multiplication operations of one floating-point number precision.

[0104] The first point multiplication unit 101 and the second point multiplication unit 103 are independent of each other when performing the point multiplication operation and are not multiplexed with each other when performing the point multiplication operation of the first floating point precision or the second floating point precision. For example, when the first point multiplication unit 101 performs the point multiplication operation of the first floating point precision, the second point multiplication unit 103 does not perform the point multiplication operation of the first floating point precision, for example, is originally in an idle state; similarly, when the second point multiplication unit 103 performs the point multiplication operation of the second floating point precision, the first point multiplication unit 101 does not perform the point multiplication operation of the second floating point precision.

[0105] For example, the minimum point multiplication number of the single point multiplication operation supported by the second point multiplication unit is N, N is a positive integer greater than 1.

[0106] For example, taking the point multiplication operation of FP8 precision supported by the second point multiplication unit as an example, the minimum point multiplication number of the single point multiplication operation supported by the second point multiplication unit is 32, that is, when the point multiplication operation of FP8 floating point precision is performed, at least 32 pairs of elements perform point multiplication calculation operation and add the point multiplication calculation operation of 32 pairs of elements, and the result of the point multiplication operation is the addition result. Taking the point multiplication operation of FP16 supported by the second point multiplication unit as an example, N is 16, and taking the point multiplication operation of FP32 supported by the second point multiplication unit as an example, N is 8.

[0107] Similar restrictions can also exist for the first point multiplication unit, for example, taking the point multiplication operation of FP4 supported by the first point multiplication unit as an example, the minimum point multiplication number of the single point multiplication operation supported by the second point multiplication unit is 16.

[0108] Of course, the present disclosure is not limited to this, and the minimum point multiplication number of the single point multiplication operation supported by the first point multiplication unit and the second point multiplication unit can be set according to hardware needs.

[0109] The tensor core 100 is configured to receive a first tensor , a second tensor , a scaling factor α of the first tensor, a scaling factor β of the second tensor, an offset term a of the first tensor, and an offset term b of the second tensor, perform a matrix multiplication operation using scaling factors and offset terms by using the first point multiplication unit 101 and the scaling factor matrix multiplication processing module 102, to obtain a matrix multiplication operation result E of the first tensor and the second tensor , wherein the first tensor and the second tensor are both of the first floating point precision.

[0110] The offset term a of the first tensor is used to adjust the numerical value distribution of the elements in the first tensor, the offset term b of the second tensor is used to adjust the numerical value distribution of the elements in the second tensor, and the offset term c of the third tensor is used to adjust the numerical value distribution of the elements in the third tensor. As described above, the offset terms a, b and c can be provided by a user or determined according to the numerical value distribution of the corresponding tensor, for example, can be obtained by taking the mean value, taking the median value, etc. The offset term can adjust the numerical value of the elements in a row or a column of the tensor which can be originally highly consistent, so that the numerical value distribution is more symmetrical (the center point is close to zero), the numerical value expression range of the scaling factor is improved, and the quantization precision is improved.

[0111] It should be noted that, Therefore, after obtaining the matrix multiplication operation result E by referring to the content described below, if C is not equal to 0, the tensor core is further configured to determine , and obtain the final operation result D according to the matrix multiplication operation result E and , and the specific process will not be described here.

[0112] In the process of performing the matrix multiplication operation using the scaling factor, the first point multiplication unit 101 is configured to perform the matrix multiplication operation of the first tensor and the second tensor with the first floating point precision to obtain the first multiplication result Y.

[0113] The first floating point precision is low precision, for example, FP4, and the first point multiplication unit 101 is configured to perform low-precision matrix multiplication, which has high computing power and high data transmission efficiency.

[0114] The scaling factor matrix multiplication processing module 102 is configured to determine the matrix multiplication operation result E by using the second point multiplication unit 103 according to the first multiplication result Y, the scaling factor a of the first tensor, the scaling factor b of the second tensor, the offset term a of the first tensor and the offset term b of the second tensor.

[0115] Figure 3 The scaling factor matrix multiplication processing module 102 provided by at least one embodiment of the present disclosure is shown in the schematic structural diagram.

[0116] As shown in Figure 3 , the scaling factor matrix multiplication processing module 102 further includes a first sub-module, a second sub-module and a third sub-module. For example, the first sub-module, the second sub-module and the third sub-module include a multiplier.

[0117] The first sub-module is configured to determine the second multiplication result g of the scaling factor a of the first tensor and the scaling factor b of the second tensor.

[0118] The second sub-module is configured to determine the third multiplication result d of the offset term a of the first tensor and the offset term b of the second tensor.

[0119] The third submodule is configured to determine the first and second correction results used for offset term correction.

[0120] The scaling factor matrix multiplication processing module is also equipped with an offset correction hardware path. The second multiplication result γ, the third multiplication result d, the first correction result, and the second correction result are transmitted to the second dot multiplication unit 103 through the offset correction hardware path. By setting up the offset correction hardware path, communication connections can be established between the first submodule, the second submodule, and the third submodule and the second dot multiplication unit, transmitting relevant multiplication results and correction results to the second dot multiplication unit for subsequent calculations.

[0121] At least one embodiment of this disclosure provides a tensor kernel that transfers offset correction-related calculations to the scaling factor matrix multiplication processing module within the tensor kernel. This eliminates the need for offset correction calculations to be performed through the vector computation kernel. Consequently, when calculating asymmetric (offset) quantized matrix multiplications, there is no need to rely on vector units for accumulation across different memory levels, thus significantly reducing latency and improving overall efficiency. This also alleviates the computational pressure on the vector computation kernel, improves the efficiency of other vector operators, and reduces the computational power of the vector computation kernel to save hardware area, thereby improving the overall execution efficiency of low-precision matrix multiplications using scaling factors and offset terms.

[0122] For example, the calculation of the first and second correction results requires the first tensor. Second tensor The scaling factor α, scaling factor β, offset term a, and offset term b are described in detail below.

[0123] The second dot multiplication unit 103 is configured to determine the matrix multiplication operation result E based on the second multiplication result γ, the first multiplication result Y, the third multiplication result d, the first correction result e, and the second correction result f.

[0124] Figure 4A This is a schematic diagram of a first tensor and a second tensor provided for at least one embodiment of this disclosure.

[0125] like Figure 4A As shown, the first tensor It is divided into T first-dimensional blocks in the column direction, for example Figure 4A The first block A0, the first block A1, ..., the first block A T-1 Each first tensor has a shape and size of m×k. The second tensor is divided into T second tensors in the row direction, for example... Figure 4A The second volume block B0, the second volume block B1, ..., the second volume block B T-1 The shape and size of each second block are k×n.

[0126] Figure 4B A schematic diagram of the scaling factor of the first tensor and the scaling factor of the second tensor provided for at least one embodiment of the present disclosure.

[0127] As shown in Figure 4B , the scaling factor a of the first tensor is divided into T first vectors in the column direction, for example Figure 4B , the first vector a0, the first vector a1,..., the first vector a T-1 , each first vector has a shape size of m x 1. The scaling factor b of the second tensor is divided into T second vectors in the row direction, each second vector has a shape size of 1 x n.

[0128] Figure 4C A schematic diagram of the offset term of the first tensor and the offset term of the second tensor provided for at least one embodiment of the present disclosure.

[0129] As shown in Figure 4C , the offset term a of the first tensor is divided into T / τ third vectors in the column direction, each third vector has a shape size of m x 1. The offset term b of the second tensor is divided into T / τ fourth vectors in the row direction, each fourth vector has a shape size of 1 x n.

[0130] For example, referring to the above formula 6 or formula 7, , the matrix multiplication operation result E can be represented as:

[0131]

[0132] , γ1,..., γ T-1 represent T first outer product operation results, Y0, Y1,..., Y T-1 represent T matrix multiplication intermediate results, e0, e1,..., e T / τ-1 represent T / τ first syndrome results, f0, f1,..., f T / τ-1 represent T / τ second syndrome results, d0, d1,..., d T / τ-1 represent T / τ second outer product operation results.

[0133] Of course, it needs to be explained that referring to the above formula 6 or formula 7, the specific content of the first correction result and the second correction result may be different, but the overall process of implementing the matrix multiplication operation using the scaling factor and the offset term is the same, and the present disclosure does not limit the specific calculation process of the first correction result and the second correction result.

[0134] The first point multiplication unit 101 is configured to perform a matrix multiplication operation between the T first tensor blocks and the T second tensor blocks to obtain T matrix multiplication intermediate results Y0, Y1,..., Y T-1The first multiplication result includes the T matrix multiplication intermediate results, and each matrix multiplication intermediate result has a shape size of mxn.

[0135] For example, the first sub-module is configured to perform an outer product operation between the T first vectors and the T second vectors to obtain T first outer product operation results γ0, γ1,... γ T-1 The second multiplication result includes the T first outer product operation results, and each first outer product operation result has a shape size of mxn.

[0136] For example, the second sub-module is configured to perform an outer product operation between the T / τ third vectors and the T / τ fourth vectors to obtain T / τ second outer product operation results d0, d1,... d T / τ-1 The third multiplication result includes the T / τ second outer product operation results, and each second outer product operation result has a shape size of mxn.

[0137] For example, the third sub-module is configured to determine a first correction result according to the offset term a, the scaling factor β, the second tensor The first correction result includes T / τ first correction sub-results e0, e1,... e T / τ-1 The third sub-module is also configured to determine a second correction result according to the scaling factor α, the second tensor The second correction result includes T / τ second correction sub-results f0, f1,... f T / τ-1 The shape size of the first correction sub-result and the second correction sub-result is mxn.

[0138] The shape size of the matrix multiplication operation result E is also mxn.

[0139] The execution process of each hardware module when performing a specific processing operation is described in detail below.

[0140] For example, each first point multiplication unit 101 is configured to perform a matrix multiplication operation between 1 first tensor block and a corresponding second tensor block to obtain 1 matrix multiplication intermediate result. A plurality of first point multiplication units perform a plurality of matrix multiplication operations between a plurality of first tensor blocks and a plurality of second tensor blocks in parallel.

[0141] For example, one first point multiplication unit 101 is configured to perform a matrix multiplication operation A0xB0 between a first tensor block A0 and a second tensor block B0 to obtain a matrix multiplication intermediate result Y0, and another first point multiplication unit 101 is configured to perform a matrix multiplication operation A1xB1 between a first tensor block A1 and a second tensor block B1 to obtain a matrix multiplication intermediate result Y1, and so on. For example, T first point multiplication units perform T matrix multiplication operations in parallel to obtain T matrix multiplication intermediate results, that is, Y0, Y1,... Y T-1 .

[0142] For example, when the first point multiplication unit 101 performs the matrix multiplication operation of the first tensor block and the corresponding second tensor block, it includes performing m x n second operations, where each second operation includes performing k point multiplication calculation operations of elements.

[0143] For example, when the first point multiplication unit 101 performs the matrix multiplication operation of the first tensor block A0 and the second tensor block B0 to obtain the matrix multiplication intermediate result Y0 with a data size of m x n, it includes performing m x n second operations, where each second operation includes performing k point multiplication calculation operations of elements and determining the sum of the operation results of the k point multiplication calculation operations. Specifically, the second operation includes performing point multiplication calculation operations between the k elements of the first row of the first tensor block A0 and the k elements of the first column of the second tensor block B0 to obtain the sum result as the element in the first row and the first column in the matrix multiplication intermediate result Y0.

[0144] For example, k represents the number of elements sharing the same scaling factor, and the larger k is, the smaller the effect of the scaling factor and the lower the calculation accuracy. T is related to k and the width of the first tensor, and the larger T is, the higher the parallelism is, but the larger the calculation overhead is. The selection of T and k can be set by considering the balance between the calculation accuracy requirement and the hardware resource overhead. For example, in an embodiment, k = 16, and according to the different width of the first tensor A, T can be set to different values, such as T = 4, 8, 16, etc., of course, the present disclosure does not make specific limitations thereto.

[0145] For example, the first sub-module includes one or more multipliers, and in response to the first sub-module including multiple multipliers, the multiple multipliers are configured to perform multiple outer product operations in parallel. For example, one multiplier performs an outer product operation between the first vector a0 and the second vector b0 to obtain a first outer product operation result g0, another multiplier performs an outer product operation between the first vector a1 and the second vector b1 to obtain a first outer product operation result g1, and so on. The second multiplication result g includes the T first outer product operation results.

[0146] For example, the second sub-module includes one or more multipliers, and in response to the second sub-module including multiple multipliers, the multiple multipliers are configured to perform multiple outer product operations in parallel. For example, one multiplier performs an outer product operation between the third vector a0 and the fourth vector b0 to obtain a second outer product operation result d0, another multiplier performs an outer product operation between the third vector a1 and the fourth vector b1 to obtain a first outer product operation result d1, and so on. The third multiplication result d includes the T / τ second outer product operation results.

[0147] For example, the third sub-module includes one or more multipliers configured to determine the first correction result and the second correction result. For example, taking formula 6 as an example, the first correction sub-result e p is represented as The second correction sub-result f p is represented as Of course, according to different optimization modes, the calculation methods of the first correction sub-result and the second correction sub-result are different, and the present disclosure does not make specific limitations on this.

[0148] Referring to formula 8 as above, for the multiplexed second point multiplication unit, formula 8 is adjusted as follows:

[0149]

[0150] Wherein, J1 represents a first matrix with shape size m x n, and all elements are 1. Using a second matrix representation, the shape size of the second matrix is also m x n, but the numerical values of all elements are τ x k.

[0151] The second point multiplication unit is configured to obtain the first matrix and the second matrix, and perform point multiplication operations between the first multiplication result and the second multiplication result, between the first correction result and the first matrix, between the second correction result and the first matrix, and between the third multiplication result and the second matrix, to obtain the matrix multiplication operation result E.

[0152] For example, a plurality of second point multiplication units are configured to perform a plurality of first operations in parallel, to obtain the matrix multiplication operation result through m x n first operations.

[0153] For example, the first operation includes performing a first point multiplication operation, a second point multiplication operation, a third point multiplication operation and a fourth point multiplication operation, and determining the cumulative sum of the operation results of the first point multiplication operation, the second point multiplication operation, the third point multiplication operation and the fourth point multiplication operation as the operation result of the first operation, wherein the operation result of the first operation is as an element in the matrix multiplication operation result.

[0154] The first dot multiplication operation includes performing dot multiplication operation between the element in the ith row and jth column of the T matrix multiplication intermediate results and the element in the ith row and jth column of the corresponding T first outer product operation results, the second dot multiplication operation includes performing dot multiplication operation between the element in the ith row and jth column of the T / τ first matrices and the element in the ith row and jth column of the T / τ first syndrome results, the third dot multiplication operation includes performing dot multiplication operation between the element in the ith row and jth column of the T / τ first matrices and the element in the ith row and jth column of the T / τ second syndrome results, and the fourth dot multiplication operation includes performing dot multiplication operation between the element in the ith row and jth column of the T / τ second matrices and the element in the ith row and jth column of the T / τ second outer product operation results, where i is a positive integer less than or equal to m, and j is a positive integer less than or equal to n.

[0155] For example, the matrix multiplication operation result E = DP1 + DP2 + DP3 + DP4, where DP1, DP2, DP3 and DP4 are determined according to the following manners respectively:

[0156]

[0157]

[0158] wherein, denotes the element in the ith row and jth column of the first outer product operation result γ0, denotes the element in the ith row and jth column of the matrix multiplication intermediate result Y0, denotes the element in the ith row and jth column of the outer product operation result γ1, denotes the element in the ith row and jth column of the matrix multiplication intermediate result Y1, and so on.

[0159] denotes the element in the ith row and jth column of the first syndrome result e0, denotes the element in the ith row and jth column of the second syndrome result f0, denotes the element in the ith row and jth column of the second outer product operation result d0, and so on.

[0160] For example, the first dot multiplication operation includes performing multiplication accumulation operation of the element in the ith row and jth column of the T matrix multiplication intermediate results and the element in the ith row and jth column of the T outer product operation results. Specifically, the first dot multiplication operation can be represented as:

[0161]

[0162] The second dot multiplication operation includes performing dot multiplication operation between the element in the ith row and jth column of the T / τ first matrices and the element in the ith row and jth column of the T / τ first syndrome results. Specifically, the second dot multiplication operation can be represented as:

[0163]

[0164] The third dot product operation involves performing a dot product between the elements in the i-th row and j-th column of the T / τ first matrices and the elements in the i-th row and j-th column of the T / τ second corrector results. Specifically, the third dot product operation can be represented as:

[0165]

[0166] The fourth dot product operation involves performing a dot product between the i-th row and j-th column element of T / τ second matrices and the i-th row and j-th column element of the result of T / τ second outer product operations. Specifically, the fourth dot product operation can be represented as:

[0167]

[0168] The sum of the results of the first, second, third, and fourth dot product operations is taken as the result E of the first operation. i,j :

[0169]

[0170] For example, the second dot product unit is configured to perform a first operation once. When performing the first operation, the second dot product unit includes performing a dot product operation on the first dot product object and the second dot product object to obtain the operation result of the first operation.

[0171] Figure 5A A schematic diagram of a first dot product object and a second dot product object provided for at least one embodiment of this disclosure.

[0172] like Figure 5A As shown, the first dot product object includes T+3×T / τ elements, which include the i-th row and j-th column element of each intermediate matrix multiplication result in the T intermediate matrix multiplication results, that is... In addition, it also includes 2×T / τ 1s and T / τ τ×k.

[0173] like Figure 5A As shown, the second dot product object includes T+3×T / τ elements, which include the i-th row and j-th column element of each of the T first outer product results, i.e. The element in the i-th row and j-th column of each of the T / τ first corrector results, i.e. The element in the i-th row and j-th column of each of the T / τ second corrector results, i.e. ; and the i-th row and j-th column element of each of the T / τ second outer product operations, that is .

[0174] For example, the first operation includes multiplying the elements at the corresponding positions in the first point multiplication object and the second point multiplication object, and adding the multiplication results to obtain an operation result of the first operation.

[0175] The operation result of the first operation is an element in the i-th row and the j-th column of the matrix multiplication operation result .

[0176] For example, the plurality of second point multiplication units perform the first operation in parallel, each first operation can obtain an element in the matrix multiplication operation result, and all elements in the matrix multiplication operation result E are obtained through m*n first operations, thereby obtaining the matrix multiplication operation result.

[0177] In the above embodiment, the accumulation and element multiplication operation in the matrix multiplication operation using the scaling factor is implemented by the second point multiplication unit, the second point multiplication unit in an idle state when performing the matrix multiplication operation of the first floating point precision is multiplexed, the existing hardware is fully utilized, the multiplier is reduced, the additional adder is avoided, and the hardware area is saved. Moreover, since the floating point precision of the point multiplication operation supported by the second point multiplication unit is higher than the first floating point precision, the use of the second point multiplication unit to perform the multiplication and addition operation can preserve the effective data to a greater extent, and the calculation precision is not affected.

[0178] As described above, the minimum number of point multiplications N of the single point multiplication operation supported by the second point multiplication unit is limited, and in at least one embodiment of the present disclosure, when selecting the second point multiplication unit, a point multiplication unit supporting a minimum number of point multiplications N of the single point multiplication operation greater than or equal to T+3*T / τ is selected as the second point multiplication unit, N is a positive integer, thereby maximizing the balance between hardware overhead and operator execution speed and efficiency.

[0179] For example, the second point multiplication unit includes performing N point multiplication calculation operations and determining the sum of the operation results of the N point multiplication calculation operations when performing the point multiplication operation of the first point multiplication object and the second point multiplication object. That is, due to the limitation of the minimum number of point multiplications, the minimum number of point multiplications of the single point multiplication operation of the second point multiplication unit is at least N, that is, at least N point multiplication calculation operations are required.

[0180] Figure 5B A schematic diagram of the expanded first point multiplication object and the expanded second point multiplication object provided in at least one embodiment of the present disclosure.

[0181] As Figure 5BAs shown, in response to N being greater than T+3xT / τ, the first point multiplication object and the second point multiplication object are padded with 0 in parts other than the T+3xT / τ elements of each of the above-mentioned first point multiplication object and the second point multiplication object, so as to expand the first point multiplication object and the second point multiplication object to include N elements, and the second point multiplication unit performs a point multiplication operation on the expanded first point multiplication object and the expanded second point multiplication object. For example, with reference to Figure 5B When the second point multiplication unit performs the point multiplication operation, each point multiplication calculation operation performs a point multiplication operation on two elements of the same column, and then adds operation results of N point multiplication calculation operations.

[0182] Therefore, when the first operation described above is performed using the second point multiplication unit, the second point multiplication unit still performs N point multiplication calculation operations in the form of supplementing 0, but the actual calculation result is the operation result of T+3xT / τ point multiplication calculation operations, thereby multiplexing the second point multiplication unit.

[0183] For example, in one specific embodiment, the first floating-point precision is FP4, the first point multiplication unit supports an FP4 point multiplication operation, and the second point multiplication unit supports an FP8 point multiplication operation and a point multiplication operation higher than the FP8 precision. For example, assuming that k=16, T is set according to the width of the first tensor, for example, T=4, 8, 16, etc.

[0184] For example, τ can be set as needed, for example, set to 2.

[0185] The first point multiplication unit performs a matrix multiplication operation of the first tensor and the second tensor in the FP4 floating-point precision, to obtain a first multiplication result, for example, T first point multiplication units perform a matrix multiplication operation between T first tensor blocks and T second tensor blocks in parallel, to obtain T matrix multiplication intermediate results, and the first multiplication result includes the T matrix multiplication intermediate results. For details, refer to the foregoing description, which will not be described here.

[0186] The first sub-module performs an outer product operation between the T first vectors and the T second vectors, for example, T multipliers are used to perform an outer product operation between the T first vectors and the T second vectors in parallel, to obtain T first outer product operation results.

[0187] The second sub-module performs an outer product operation between the T third vectors and the T fourth vectors, for example, T multipliers are used to perform an outer product operation between the T third vectors and the T fourth vectors in parallel, to obtain T second outer product operation results.

[0188] The third sub-module is configured to determine the first correction result and the second correction result using a plurality of multipliers.

[0189] For details of the first sub-module, the second sub-module, and the third sub-module, refer to the foregoing description, which will not be described here.

[0190] Multiple second-point multiplication units execute multiple first operations in parallel, and the matrix multiplication result is obtained through m×n first operations.

[0191] For example, in one embodiment, when the second dot product unit performs a dot product operation with FP8 precision, the minimum number of dot products supported in a single dot product operation is 32; when the second dot product unit performs a dot product operation with FP16 precision, the minimum number of dot products supported in a single dot product operation is 16; and when the second dot product unit performs a dot product operation with FP32 precision, the minimum number of dot products supported in a single dot product operation is 8.

[0192] Assuming T=4 and τ=2, the first operation can be performed using FP8 or FP16 precision dot product operations. For example, when using FP8 to perform the first operation, (32-10) zeros are added to both the first and second dot product objects. Similarly, when using FP16 to perform the first operation, (16-10) zeros are added to both the first and second dot product objects.

[0193] Considering accuracy requirements and computational overhead, in the above embodiment, the second dot multiplication unit performs the first operation optimally using FP16.

[0194] Assuming T=8 and τ=2, the first operation can be performed using a dot product operation with FP8 precision. For example, when using FP8 to perform the first operation, the first and second dot product objects are each padded with (32-20) zeros.

[0195] In the above embodiment, the dot product operation of the first multiplication result and the second multiplication result, the summation operation of the T rounds of the dot product operation results, and the T / τ round summation operation of the first correction result, the second correction result, and the third multiplication result are performed by reusing the second dot product unit in the tensor kernel, which was originally in an idle state. Since the second dot product unit is independent of the first dot product unit and was originally in an idle state when the first dot product unit performs a matrix multiplication operation with a first floating-point precision, reusing the second dot product unit can improve the utilization of hardware resources, make full use of existing hardware resources, reduce the multiplier setup required for performing the dot product operation of the first multiplication result and the second multiplication result, reduce the adder setup required for performing the summation operation of the T rounds of the dot product operation results, and reduce the adder setup required for performing the T / τ round summation operation, thereby reducing the hardware area cost of additional multipliers and adders required to implement low-precision matrix multiplication using scaling factors and offset terms.

[0196] Figure 6 This is a schematic structural diagram of a processor provided for at least one embodiment of the present disclosure. For example... Figure 6As shown, the processor 200 includes a tensor core 201.

[0197] For example, the tensor core includes a first point multiplication unit and a scale factor matrix multiplication processing module, the scale factor matrix multiplication processing module includes a second point multiplication unit, the first point multiplication unit and the second point multiplication unit are configured to support point multiplication operations of different floating point precisions.

[0198] The tensor core is configured to receive a first tensor, a second tensor, a scale factor of the first tensor, a scale factor of the second tensor, an offset term of the first tensor and an offset term of the second tensor, perform a matrix multiplication operation using the scale factors and the offset terms by using the first point multiplication unit and the scale factor matrix multiplication processing module, and obtain a matrix multiplication operation result of the first tensor and the second tensor, wherein the first tensor and the second tensor are both of a first floating point precision, and the offset term of the first tensor is used to adjust the numerical value distribution of the elements in the first tensor, and the offset term of the second tensor is used to adjust the numerical value distribution of the elements in the second tensor.

[0199] In the process of performing the matrix multiplication operation using the scale factors and the offset terms, the first point multiplication unit is configured to perform a matrix multiplication operation of the first floating point precision of the first tensor and the second tensor to obtain a first multiplication result, and the scale factor matrix multiplication processing module is configured to determine the matrix multiplication operation result by using the second point multiplication unit according to the first multiplication result, the scale factor of the first tensor, the scale factor of the second tensor, the offset term of the first tensor and the offset term of the second tensor.

[0200] The specific structure and functions of the tensor core 201 can refer to the related descriptions of the tensor core 100 in the foregoing embodiments, which will not be repeated here.

[0201] For example, the processor can include a graphics processor, which can refer to Figure 1A As shown in the structure of the graphics processor, the processor includes a plurality of stream processor clusters and a memory, each stream processor cluster includes a plurality of calculation units, and each calculation unit includes a tensor core.

[0202] Of course, the processor can also be other processors provided with a tensor core, such as a neural network processor, a digital signal processor, a central processing unit, etc., and the present disclosure does not make specific limitations thereto.

[0203] In at least one embodiment, the processor of the present disclosure performs a matrix multiplication operation using scale factors by multiplexing point multiplication units of different precisions in the tensor core, reduces the additional hardware area cost, fully multiplexes the existing hardware resources, reduces the setting of the multiplier and does not need to additionally set the adder, reduces the hardware area cost, optimizes the system performance, reduces the hardware cost and power consumption, and improves the chip integration.

[0204] In addition, it should be noted that Figure 6The components of the processor 200 shown are merely exemplary and are not limiting, and the processor 200 can also have other components according to actual application needs.

[0205] The data processing method provided by at least one embodiment of the present disclosure also provides a data processing method. Figure 7 The data processing method provided by at least one embodiment of the present disclosure also provides a data processing method.

[0206] The data processing method is applied to a tensor core. The tensor core includes a first point multiplication unit and a scaling factor matrix multiplication processing module, and the scaling factor matrix multiplication processing module includes a second point multiplication unit. The first point multiplication unit and the second point multiplication unit support point multiplication operations of different floating-point precisions. For more information about the tensor core, please refer to the related description of the tensor core 100 in the foregoing embodiments, which will not be repeated here.

[0207] As shown in Figure 7 The data processing method provided by at least one embodiment of the present disclosure includes steps S10-S20.

[0208] In S10, a first tensor, a second tensor, a scaling factor of the first tensor, a scaling factor of the second tensor, an offset term of the first tensor, and an offset term of the second tensor are received.

[0209] In S20, a matrix multiplication operation using the scaling factors and the offset terms is performed by using the first point multiplication unit and the scaling factor matrix multiplication processing module to obtain a matrix multiplication operation result of the first tensor and the second tensor.

[0210] The first tensor and the second tensor are both of a first floating-point precision, and the offset terms are used to adjust the element value distribution in the first tensor.

[0211] For example, S20 can include: performing a matrix multiplication operation of the first floating-point precision of the first tensor and the second tensor by using the first point multiplication unit to obtain a first multiplication result; and determining the matrix multiplication operation result by using the second point multiplication unit according to the first multiplication result, the scaling factor of the first tensor, the scaling factor of the second tensor, the offset term of the first tensor, and the offset term of the second tensor.

[0212] For example, the scaling factor matrix multiplication processing module further includes a scaling factor multiplication processing submodule, and the scaling factor multiplication processing submodule includes a plurality of multipliers, for example.

[0213] Determining the matrix multiplication operation result by using the second point multiplication unit according to the first multiplication result, the scaling factor of the first tensor, and the scaling factor of the second tensor can include: determining a second multiplication result of the scaling factor of the first tensor and the scaling factor of the second tensor by using the scaling factor multiplication processing submodule; and determining the matrix multiplication operation result by using the second point multiplication unit according to the second multiplication result and the first multiplication result.

[0214] For example, in some embodiments, the matrix multiplication operation result is determined by the second point multiplication unit according to the first multiplication result, the scaling factor of the first tensor, the scaling factor of the second tensor, the offset term of the first tensor and the offset term of the second tensor, including: determining a second multiplication result of the scaling factor of the first tensor and the scaling factor of the second tensor; determining a third multiplication result of the offset term of the first tensor and the offset term of the second tensor; determining a first correction result and a second correction result for offset term correction; determining the matrix multiplication operation result by the second point multiplication unit according to the first multiplication result, the second multiplication result, the third multiplication result, the first correction result and the second correction result.

[0215] For example, the first tensor is divided into T first tensor blocks in the column direction, each first tensor block has a shape size of m x k, and the second tensor is divided into T second tensor blocks in the row direction, each second tensor block has a shape size of k x n.

[0216] The scaling factor of the first tensor is divided into T first vectors in the column direction, each first vector has a shape size of m x 1, and the scaling factor of the second tensor is divided into T second vectors in the row direction, each second vector has a shape size of 1 x n.

[0217] The offset term of the first tensor is divided into T / τ third vectors in the column direction, each third vector has a shape size of m x 1, and the offset term of the second tensor is divided into T / τ fourth vectors in the row direction, each fourth vector has a shape size of 1 x n.

[0218] The first point multiplication unit is configured to perform a matrix multiplication operation between the T first tensor blocks and the T second tensor blocks to obtain T matrix multiplication intermediate results, wherein the first multiplication result includes the T matrix multiplication intermediate results.

[0219] The first sub-module is configured to perform an outer product operation between the T first vectors and the T second vectors to obtain T first outer product operation results, wherein the second multiplication result includes the T first outer product operation results.

[0220] The second sub-module is configured to perform an outer product operation between the T / τ third vectors and the T / τ fourth vectors to obtain T / τ second outer product operation results, wherein the third multiplication result includes the T / τ second outer product operation results.

[0221] Wherein, T, m, n, k are positive integers, and τ is a positive integer and less than or equal to T.

[0222] For example, the second point multiplication unit is configured to obtain the first matrix and the second matrix, and perform a point multiplication operation between the first multiplication result and the second multiplication result, a point multiplication operation between the first correction result and the first matrix, a point multiplication operation between the second correction result and the first matrix, and a point multiplication operation between the third multiplication result and the second matrix to obtain the matrix multiplication operation result, wherein the size of the first matrix is m x n, and the values of all elements in the first matrix are 1, the size of the second matrix is m x n, and the values of all elements in the second matrix are τ x k.

[0223] The specific process of the first operation can refer to the description of the related embodiments in the foregoing tensor core, which will not be described here.

[0224] The data processing method provided by at least one embodiment of the present disclosure performs the matrix multiplication operation using the scaling factor by multiplexing the point multiplication units with different precisions in the tensor core, reduces the additional hardware area cost, fully multiplexes the existing hardware resources, reduces the setting of the multiplier and does not need to additionally set the adder, reduces the hardware area cost, optimizes the system performance, reduces the hardware cost and power consumption, and improves the chip integration.

[0225] It should be noted that the first tensor and the second tensor in the present disclosure can have different physical meanings according to different application fields.

[0226] For example, in the field of speech processing, the first tensor and the second tensor can be any parameter used, input, or generated in tasks such as feature extraction, speech enhancement, speech recognition, etc., which needs to perform matrix multiplication operation, such as a speech feature vector, a filtering parameter, etc.

[0227] For example, in the field of image processing, the first tensor and the second tensor can be any parameter used, input, or generated in tasks such as image preprocessing, feature extraction, image segmentation, target detection, etc., which needs to perform matrix multiplication operation, such as an image feature vector, various edge detection operators (such as Sobel operator, Canny operator, Prewitt operator, etc.), image filtering operators (such as Gaussian filtering, median filtering, bilateral filtering, etc.), morphological operators (such as erosion, dilation, opening operation, closing operation, etc.), and the like.

[0228] For example, in the field of text processing, the first tensor and the second tensor can be any parameter used, input, or generated in tasks such as text classification, sentiment analysis, text generation, etc., which needs to perform matrix multiplication operation, such as a semantic feature vector of text, etc.

[0229] For example, in the field of video processing, the first tensor and the second tensor can be parameters used, input, generated in the field of image processing as described above, or specific to the field of video processing, such as optical flow operators (used to estimate the motion between video frames), target tracking operators (used to track a specific target in a video), etc.

[0230] Of course, the present disclosure is not limited thereto, and for other application scenarios or fields, such as technical fields of graphics, machine learning, data mining, signal processing, natural language processing, geographic information systems, oceanography, environmental science, etc., as long as matrix multiplication operations are required, the data processing method described in at least one embodiment of the present disclosure can be applied, which will not be repeated here.

[0231] Figure 8 A schematic block diagram of an electronic device according to an embodiment of the present disclosure is provided.

[0232] As shown in Figure 8 The electronic device 300 is suitable for implementing the data processing method provided by the embodiments of the present disclosure, for example. It should be noted that Figure 8 The components of the electronic device 300 shown are only exemplary and are not limiting, and the electronic device 300 can also have other components according to actual application needs.

[0233] As shown in Figure 8 The electronic device 300 can include a processing device 301, which includes the aforementioned processor 200, for example, which can perform various appropriate actions and processes according to non-transitory computer-readable instructions stored in the memory to achieve various functions. For example, the processing device 301 can also include a central processing unit (CPU), a tensor processor (TPU), etc. with instruction optimization capability and / or program execution capability. The central processing unit (CPU) can be X86, ARM, RISC-V architecture, etc. The GPU can be directly integrated into the SOC, directly integrated into the mainboard, or built into the north bridge chip of the mainboard.

[0234] For example, the computer-readable instructions executed by the processing device 301 can perform one or more steps of the data processing method according to any of the above embodiments. It should be noted that the detailed description of the processing process of the data processing method can refer to the related description in the above embodiments of the data processing method.

[0235] As shown in Figure 8As shown, for example, the memory can include any combination of one or more computer program products. The computer program product(s) can include various forms of computer-readable storage media, such as volatile and / or non-volatile computer-readable media. Volatile computer- readable media, for example, can include random access memory (RAM) 303 and / or cache, etc. For example, computer-readable instructions can be loaded into the random access memory (RAM) 303 from the storage 308 to run the computer-readable instructions. Non-volatile computer-readable media, for example, can include read only memory (ROM) 302, hard disk, erasable programmable read only memory (EPROM), compact disk read only memory (CD-ROM), USB memory, flash memory, etc. Various applications and various data, such as various data used and / or produced by the applications, etc., can also be stored in the computer-readable storage media.

[0236] For example, the processing device 301, the read only memory (ROM) 302, and the random access memory (RAM) 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0237] Generally, the following devices can be connected to the input / output (I / O) interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, a flash memory, etc.; and communication devices 309. The communication devices 309 can allow the electronic device 300 to communicate wirelessly or wired with other electronic devices to exchange data. Although Figure 8 The electronic device 300 is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or present, and the electronic device 300 can instead be implemented or present more or fewer devices. For example, the processing device 301 can control other components in the electronic device 300 to perform desired functions.

[0238] For example, the electronic device can be in the form of a server for various application scenarios such as deep learning and artificial intelligence, scientific computing, graphics rendering and video editing, virtual reality and game development, cloud services, etc.

[0239] The flow and block diagrams in the drawings represent possible architectural, functional, and operational architectures of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block can represent a module, a segment, or a portion of code that comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0240] The units described in the embodiments of the present disclosure can be implemented by software, or by hardware. In some cases, the name of the unit does not constitute a limitation on the unit itself.

[0241] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, non-limiting examples of exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0242] Figure 9 A schematic diagram of a non-transitory computer-readable storage medium according to at least one embodiment of the present disclosure is shown. For example, as shown in FIG. 4, a storage medium 400 can be a non-transitory computer-readable storage medium, and one or more computer-readable instructions 401 can be stored non-transitorily on the storage medium 400. For example, the computer-readable instructions 401, when executed by a processor, can perform one or more steps of a data processing method according to the above. Figure 9

[0243] For example, the storage medium 400 can be applied in the electronic device 300. For example, the storage medium 400 can include the storage apparatus 308 in the electronic device 300.

[0244] ​For example, the storage can include any combination of one or more computer program products. The computer program product can include various forms of computer-readable storage media, such as volatile and / or non-volatile memory. The volatile memory, for example, can include random access memory (RAM) and / or cache. Non-volatile memory, for example, can include read-only memory (ROM), hard disks, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, and / or the like. The computer-readable storage media can store one or more computer-readable instructions that, when executed by the processor, implement various functions of the processor. Various application programs and various data, and the like, can also be stored in the storage.

[0245] For example, the storage medium can include a memory card of a smart phone, a cache component of a tablet computer, a hard disk of a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage medium, and can be other applicable storage medium.

[0246] The above description is merely exemplary of the present disclosure and the application of the principles thereof and the scope of the disclosure is not limited to the specific embodiments described herein, but only by the claims that follow. It will be readily apparent to those skilled in the art that varying substitutions and modifications can be made to the embodiments described herein without departing from the scope and spirit of the disclosure. For example, the features described herein with respect to a particular implementation can be combined with the features described in connection with one or more other implementations, even if that feature or those features are not expressly described in connection with that one or more other implementations. Other implementations can be apparent to those of ordinary skill in the art and can be derived from the description of the present disclosure without departing from the scope and spirit of the disclosure. For example, the features described above with respect to a particular implementation can be combined with the features described in connection with one or more other implementations, even if that feature or those features are not expressly described in connection with that one or more other implementations.

[0247] Furthermore, while operations are depicted in a particular, sequential order, this should not be understood as requiring or implying that the operations are performed in the order illustrated. Rather, many of the operations can be performed in parallel or in any suitably ordered order. In addition, while a particular implementation can be discussed in the context of a single embodiment, this should not be understood as requiring or implying that the implementation is exclusive to that embodiment. Rather, the various implementations can be combined in any suitable sub-combination or in any suitable order. In addition, while several variations can be described above as coming from a particular implementation, this should not be understood as requiring or implying that the variations are exclusive to that implementation. Rather, the variations can be combined in any suitable sub-combination or in any suitable order.

[0248] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0249] For the present disclosure, the following points also need to be explained:

[0250] (1) The drawings of the embodiments of the present disclosure only involve the structures involved in the embodiments of the present disclosure, and other structures can be referred to the general design.

[0251] (2) In the case of no conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0252] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A tensor kernel, characterized in that, It includes a first dot product unit, a scaling factor matrix multiplication processing module, and a hardware path for cross-precision dot product units. The scaling factor matrix multiplication processing module includes a second dot product unit. The first and second dot product units are configured to support dot product operations with different floating-point precisions. The tensor kernel is configured to receive a first tensor, a second tensor, a scaling factor of the first tensor, a scaling factor of the second tensor, an offset term of the first tensor, and an offset term of the second tensor. It then uses the first dot product unit and the scaling factor matrix multiplication processing module to perform a matrix multiplication operation using the scaling factor and the offset term to obtain the matrix multiplication result of the first tensor and the second tensor. Here, both the first tensor and the second tensor are of first floating-point precision. The offset term of the first tensor is used to adjust the numerical distribution of the elements in the first tensor, and the offset term of the second tensor is used to adjust the numerical distribution of the elements in the second tensor. During the matrix multiplication operation using scaling factors and offset terms, the first point multiplication unit is configured to perform a matrix multiplication operation with the first floating-point precision on the first tensor and the second tensor, obtaining a first multiplication result. The hardware path of the cross-precision dot product unit is configured to transmit the first multiplication result to the scaling factor matrix multiplication processing module. The scaling factor matrix multiplication processing module is configured to determine the matrix multiplication operation result using the second dot multiplication unit based on the first multiplication result, the scaling factor of the first tensor, the scaling factor of the second tensor, the offset term of the first tensor, and the offset term of the second tensor. The second dot product unit supports dot product operations with a second floating-point precision higher than that of the first floating-point number.

2. The tensor kernel according to claim 1, characterized in that, The scaling factor matrix multiplication processing module further includes a first submodule, a second submodule, a third submodule, and an offset correction hardware path. The first submodule is configured to determine the second multiplication result of the scaling factors of the first tensor and the second tensor. The second submodule is configured to determine the third multiplication result of the offset terms of the first tensor and the offset terms of the second tensor. The third submodule is configured to determine a first correction result and a second correction result for offset term correction; The second dot product unit is configured to determine the matrix multiplication operation result based on the first multiplication result, the second multiplication result, the third multiplication result, the first correction result, and the second correction result. The offset correction hardware path is configured to transmit the second multiplication result, the third multiplication result, the first correction result, and the second correction result to the second dot multiplication unit.

3. The tensor kernel according to claim 2, characterized in that, The first tensor is divided into T first tensor blocks in the column direction, each with a shape and size of m×k. The second tensor is divided into T second tensor blocks in the row direction, each with a shape and size of k×n. The first dot product unit is configured to perform matrix multiplication operations between the T first tensor blocks and the T second tensor blocks to obtain T intermediate matrix multiplication results, wherein the first multiplication result includes the T intermediate matrix multiplication results. Where T, m, k, and n are positive integers.

4. The tensor kernel according to claim 2, characterized in that, The scaling factor of the first tensor is divided into T first vectors in the column direction, each with a shape size of m×1. The scaling factor of the second tensor is divided into T second vectors in the row direction, each with a shape size of 1×n. The first submodule is configured to perform outer product operations between the T first vectors and the T second vectors to obtain T first outer product operation results, wherein the second multiplication result includes the T first outer product operation results. Where T, m, and n are positive integers.

5. The tensor kernel according to claim 2, characterized in that, The offset term of the first tensor is divided into T / τ third vectors in the column direction, each third vector having a shape size of m×1. The offset term of the second tensor is divided into T / τ fourth vectors in the row direction, each fourth vector having a shape size of 1×n. The second submodule is configured to perform the outer product operation between the T / τ third vectors and the T / τ fourth vectors to obtain T / τ second outer product operation results, wherein the third multiplication result includes the T / τ second outer product operation results. Where T, m, and n are positive integers, and τ is a positive integer less than or equal to T.

6. The tensor kernel according to claim 2, characterized in that, The second dot product unit is configured to acquire a first matrix and a second matrix, perform a dot product operation between the first multiplication result and the second multiplication result, a dot product operation between the first correction result and the first matrix, a dot product operation between the second correction result and the first matrix, and a dot product operation between the third multiplication result and the second matrix, to obtain the matrix multiplication result. The first matrix has a size of m×n, and all elements in the first matrix have a value of 1. The second matrix has a size of m×n, and all elements in the second matrix have a value of τ×k. Where m, n, τ, and k are all positive integers.

7. The tensor kernel according to claim 6, characterized in that, The first multiplication result includes T intermediate matrix multiplication results, each with a shape size of m×n; the second multiplication result includes T first outer product operation results, each with a shape size of m×n; the third multiplication result includes T / τ second outer product operation results, each with a shape size of m×n; the first correction result includes T / τ first correction sub-results, each with a shape size of m×n; and the second correction result includes T / τ second correction sub-results, each with a shape size of m×n, where T is a positive integer. Multiple second point multiplication units are configured to execute multiple first operations in parallel to obtain the matrix multiplication result through m×n first operations. The first operation includes performing a first dot multiplication operation, a second dot multiplication operation, a third dot multiplication operation, and a fourth dot multiplication operation, and determining the sum of the results of the first dot multiplication operation, the second dot multiplication operation, the third dot multiplication operation, and the fourth dot multiplication operation as the result of the first operation, wherein the result of the first operation is used as an element in the result of the matrix multiplication operation. The first dot product operation includes a dot product operation between the element in the i-th row and j-th column of the intermediate result of T matrix multiplications and the element in the i-th row and j-th column of the result of T first outer product operations. The second dot product operation includes performing a dot product operation between T / τ elements in the i-th row and j-th column of the first matrix and the T / τ elements in the i-th row and j-th column of the first corrector results. The third dot product operation includes performing a dot product operation between T / τ elements in the i-th row and j-th column of the first matrix and T / τ elements in the i-th row and j-th column of the second corrector results. The fourth dot product operation includes performing a dot product between T / τ elements in the i-th row and j-th column of the second matrix and the i-th row and j-th column of the result of the T / τ second outer product operations. Where i is a positive integer less than or equal to m, and j is a positive integer less than or equal to n.

8. The tensor kernel according to claim 7, characterized in that, The second dot multiplication unit is configured to perform one first operation at a time. When the second dot product unit performs the first operation, it includes performing a dot product operation between the first dot product object and the second dot product object to obtain the result of the first operation. The first dot-multiplication object comprises T+3×T / τ elements, wherein the T+3×T / τ elements include the element in the i-th row and j-th column of each intermediate matrix multiplication result in the T intermediate matrix multiplication results, 2×T / τ ones, and T / τ τ×k ones. The second dot product object comprises T+3×T / τ elements, including the i-th row and j-th column element of each of the T first outer product operation results, the i-th row and j-th column element of each of the T / τ first corrector results, the i-th row and j-th column element of each of the T / τ second corrector results, and the i-th row and j-th column element of each of the T / τ second outer product operation results. The result of the first operation is used as the element in the i-th row and j-th column of the matrix multiplication result.

9. The tensor kernel according to claim 8, characterized in that, The minimum number of dot product operations N supported by the second dot product unit in a single dot product operation is greater than or equal to T + 3 × T / τ, where N is a positive integer. When the second dot product unit performs a dot product operation on the first and second dot product objects, it includes performing N dot product calculation operations and determining the sum of the results of the N dot product calculation operations. In response to N being greater than T+3×T / τ, all portions of the first and second dot product objects except for the T+3×T / τ elements already included in each object are filled with 0, thereby expanding the first and second dot product objects to include N elements. The dot product operation of the expanded first and second dot product objects is then performed by the second dot product unit.

10. A processor, characterized in that, Includes the tensor kernel as described in any one of claims 1-9.

11. A data processing method applied to tensor kernels, characterized in that, The tensor kernel includes a first dot product unit, a scaling factor matrix multiplication processing module, and a hardware path for cross-precision dot product units. The scaling factor matrix multiplication processing module includes a second dot product unit. The first and second dot product units are configured to support dot product operations with different floating-point precisions. The second dot product unit supports dot product operations with a second floating-point precision higher than the first floating-point precision. The data processing method includes: Receive a first tensor, a second tensor, a scaling factor of the first tensor, a scaling factor of the second tensor, an offset term of the first tensor, and an offset term of the second tensor; The matrix multiplication operation using scaling factor and offset term is performed by the first dot multiplication unit and the scaling factor matrix multiplication processing module to obtain the matrix multiplication result of the first tensor and the second tensor. The first tensor and the second tensor are both of the first floating-point precision. The offset term of the first tensor is used to adjust the numerical distribution of the elements in the first tensor, and the offset term of the second tensor is used to adjust the numerical distribution of the elements in the second tensor. Specifically, the matrix multiplication operation using a scaling factor and an offset term is performed using the first dot multiplication unit and the scaling factor matrix multiplication processing module to obtain the matrix multiplication result of the first tensor and the second tensor, including: The first dot product unit is used to perform a matrix multiplication operation with the first floating-point precision on the first tensor and the second tensor to obtain the first multiplication result; The first multiplication result is transmitted to the scaling factor matrix multiplication processing module using the cross-precision dot product unit hardware path. The matrix multiplication result is determined using the second dot product unit based on the first multiplication result, the scaling factor of the first tensor, the scaling factor of the second tensor, the offset term of the first tensor, and the offset term of the second tensor.

12. The data processing method according to claim 11, characterized in that, Based on the first multiplication result, the scaling factor of the first tensor, the scaling factor of the second tensor, the offset term of the first tensor, and the offset term of the second tensor, the matrix multiplication operation result is determined using the second dot product unit, including: The second multiplication result determines the scaling factor of the first tensor and the scaling factor of the second tensor; Determine the third multiplication result of the offset terms of the first tensor and the offset terms of the second tensor; Determine the first and second correction results used for offset term correction; The second multiplication result, the third multiplication result, the first correction result, and the second correction result are transmitted to the second dot multiplication unit through the offset correction hardware path; Using the second dot product unit, the matrix multiplication result is determined based on the first multiplication result, the second multiplication result, the third multiplication result, the first correction result, and the second correction result.

13. An electronic device, characterized in that, include: Memory stores computer-executable instructions non-transiently; The processor is configured to run computer-executable instructions. The computer-executable instructions are executed by the processor to implement the data processing method according to claim 11 or 12.

14. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the data processing method according to claim 11 or 12.

Images