Tensor core, processor, data processing method, electronic device and storage medium

Abstract

The invention discloses a tensor core, a processor, a data processing method, electronic equipment and a storage medium, and is applied to the field of tensor processing. The tensor kernel comprises a first dot multiplication unit and a scaling factor matrix multiplication processing module, the scaling factor matrix multiplication processing module comprises a second dot multiplication unit, and the first dot multiplication unit and the second dot multiplication unit support dot multiplication operations of different floating-point number precisions. The tensor core is configured to receive a first tensor, a second tensor, a scaling factor of the first tensor, a scaling factor of the second tensor, an offset term of the first tensor, and an offset term of the second tensor, perform a matrix multiplication operation using the scaling factor and the offset term using a first dot multiplication unit and a scaling factor matrix multiplication processing module, and obtaining a matrix multiplication operation result of the first tensor and the second tensor. Matrix multiplication operation using scaling factors is executed by multiplexing dot multiplication units with different precisions in a tensor kernel, extra hardware area cost is reduced, and existing hardware resources are fully multiplexed.

Description

Technical Field

[0001] Embodiments of this disclosure relate to a tensor core, a processor, a data processing method, an electronic device, and a non-transitory computer-readable storage medium. Background Technology

[0002] Floating-point quantization refers to converting high-precision floating-point formats (such as FP16, BF16, FP32, etc.) into low-precision floating-point formats (such as FP4, FP8, etc.). Low-precision floating-point formats can significantly improve computational efficiency, especially for General Matrix Multiplication (GEMM) operations required by Artificial Intelligence (AI) models. However, low-precision matrix multiplication introduces precision loss, thus reducing the prediction and generation performance of AI models. Therefore, in the low-precision computation process of AI models, a scaling factor is typically calculated from the current data during floating-point quantization to scale the data's representation range, thereby reducing the precision loss in the computational results.

[0003] Currently, matrix multiplication operations using scaling factors require additional multipliers and adders inside the tensor kernel, resulting in additional hardware area overhead. Summary of the Invention

[0004] This invention application provides at least one embodiment of a tensor kernel, including a first dot product unit, a scaling factor matrix multiplication processing module, and a hardware path for a cross-precision dot product unit. The scaling factor matrix multiplication processing module includes a second dot product unit. The first and second dot product units are configured to support dot product operations with different floating-point precisions. The tensor kernel is configured to receive a first tensor, a second tensor, a scaling factor of the first tensor, and a scaling factor of the second tensor. It then performs a matrix multiplication operation using the scaling factor using the first dot product unit and the scaling factor matrix multiplication processing module to obtain the matrix multiplication result of the first and second tensors. Both the first tensor and the second tensor have a first floating-point precision. During the matrix multiplication operation using a scaling factor, the first dot product unit is configured to perform a matrix multiplication operation of the first tensor and the second tensor with the first floating-point precision to obtain a first multiplication result. The hardware path of the cross-precision dot product unit is configured to transmit the first multiplication result to the scaling factor matrix multiplication processing module. The scaling factor matrix multiplication processing module is configured to determine the matrix multiplication operation result using the second dot product unit based on the first multiplication result, the scaling factor of the first tensor, and the scaling factor of the second tensor.

[0005] For example, in the tensor kernel provided in at least one embodiment of this application, the scaling factor matrix multiplication processing module further includes a scaling factor multiplication processing submodule, which is configured to determine a second multiplication result of the scaling factor of the first tensor and the scaling factor of the second tensor; the second dot product unit is configured to determine the matrix multiplication operation result based on the second multiplication result and the first multiplication result.

[0006] For example, in the tensor kernel provided in at least one embodiment of this application, the second dot product unit supports dot product operations with a second floating-point precision higher than the first floating-point precision.

[0007] For example, in the tensor kernel provided in at least one embodiment of this application, the first tensor is divided into T first tensor blocks in the column direction, each first tensor block having a shape size of m×k; the second tensor is divided into T second tensor blocks in the row direction, each second tensor block having a shape size of k×n; the scaling factor of the first tensor is divided into T first vectors in the column direction, each first vector having a shape size of m×1; the scaling factor of the second tensor is divided into T second vectors in the row direction, each second vector having a shape size of 1×n; and the first dot product unit is configured to perform the T first tensor blocks. A matrix multiplication operation is performed between one block and T second blocks to obtain T intermediate matrix multiplication results, wherein the first multiplication result includes the T intermediate matrix multiplication results; the scaling factor multiplication processing submodule is configured to perform an outer product operation between the T first vectors and the T second vectors to obtain T outer product operation results, wherein the second multiplication result includes the T outer product operation results; the second dot product unit is configured to determine the matrix multiplication operation result based on the T intermediate matrix multiplication results and the T outer product operation results, wherein m, n, k, and T are all positive integers.

[0008] For example, in the tensor kernel provided in at least one embodiment of this application, the shape and size of the matrix multiplication operation result, each intermediate matrix multiplication result, and each outer product operation result are all m×n. The plurality of second dot product units are configured to execute a plurality of first operations in parallel to obtain the matrix multiplication operation result through m×n first operations. The first operation includes performing a multiplication and accumulation operation on the element in the i-th row and j-th column of the T intermediate matrix multiplication results and the T outer product operation results to obtain the element in the i-th row and j-th column of the matrix multiplication operation result, where i is a positive integer less than or equal to m and j is a positive integer less than or equal to n.

[0009] For example, in the tensor kernel provided in at least one embodiment of this application, the second dot product unit is configured to perform a first operation at a time. When performing the first operation, the second dot product unit includes performing the following operations: determining the i-th row and j-th column element of each intermediate matrix multiplication result in the T matrix multiplication intermediate results to obtain T first elements; determining the i-th row and j-th column element of each outer product operation result in the T outer product operation results to obtain T second elements; and performing a dot product operation between the T elements and the T second elements to obtain the i-th row and j-th column element in the matrix multiplication operation result.

[0010] For example, in the tensor kernel provided in at least one embodiment of this application, the minimum number of dot products N supported by the second dot product unit in a single dot product operation is greater than or equal to T, where N is a positive integer. When the second dot product unit performs the dot product operation of the T first elements and the T second elements, it includes performing N dot product calculation operations and determining the sum of the operation results of the N dot product calculation operations. In response to N being greater than T, when performing the N dot product calculation operations, it performs the dot product calculation operation of the N first elements and the N second elements, where the N first elements include the T first elements and N zeros, and the N second elements include the T second elements and N zeros.

[0011] For example, in the tensor kernel provided in at least one embodiment of this application, the first floating-point precision is FP4, and the second dot product unit supports dot product operations with precision FP8 and higher.

[0012] For example, in the tensor kernel provided in at least one embodiment of this application, the second dot multiplication unit is configured to perform the process of determining the result of the matrix multiplication operation with FP16 precision.

[0013] For example, in the tensor kernel provided in at least one embodiment of this application, each first dot product unit is configured to perform a matrix multiplication operation between a first tensor block and the corresponding second tensor block to obtain an intermediate matrix multiplication result. Multiple first dot product units execute multiple matrix multiplication operations in parallel. When a first dot product unit performs a matrix multiplication operation between the first tensor block and the corresponding second tensor block, it includes performing m×n second operations, wherein each second operation includes performing a dot product calculation operation of k pairs of elements.

[0014] This application provides at least one embodiment of a processor, including a tensor core as described in any embodiment of this application.

[0015] This invention application provides at least one embodiment of a data processing method applied to a tensor kernel, wherein the tensor kernel includes a first dot product unit, a scaling factor matrix multiplication processing module, and a hardware path for a cross-precision dot product unit. The scaling factor matrix multiplication processing module includes a second dot product unit. The first and second dot product units are configured to support dot product operations with different floating-point precisions. The data processing method includes: receiving a first tensor, a second tensor, a scaling factor of the first tensor, and a scaling factor of the second tensor; performing a matrix multiplication operation using the scaling factor using the first dot product unit and the scaling factor matrix multiplication processing module to obtain a matrix multiplication result of the first tensor and the second tensor. Both the first tensor and the second tensor have a first floating-point precision. The matrix multiplication operation using a scaling factor, performed by the first dot product unit and the scaling factor matrix multiplication processing module, yields the matrix multiplication result of the first tensor and the second tensor. This includes: performing a matrix multiplication operation of the first tensor and the second tensor with the first floating-point precision using the first dot product unit to obtain a first multiplication result; transmitting the first multiplication result to the scaling factor matrix multiplication processing module using the cross-precision dot product unit hardware path; and determining the matrix multiplication result using the second dot product unit based on the first multiplication result, the scaling factor of the first tensor, and the scaling factor of the second tensor.

[0016] For example, in the data processing method provided in at least one embodiment of this application, the scaling factor matrix multiplication processing module further includes a scaling factor multiplication processing submodule, which determines the matrix multiplication operation result using the second dot multiplication unit based on the first multiplication result, the scaling factor of the first tensor, and the scaling factor of the second tensor, including: determining a second multiplication result of the scaling factor of the first tensor and the scaling factor of the second tensor using the scaling factor multiplication processing submodule; and determining the matrix multiplication operation result using the second dot multiplication unit based on the second multiplication result and the first multiplication result.

[0017] At least one embodiment of this application provides an electronic device, including: a memory that non-transitoryly stores computer-executable instructions; and a processor configured to run the computer-executable instructions, wherein the computer-executable instructions are executed by the processor to implement the data processing method according to any embodiment of this application.

[0018] This application provides at least one embodiment of a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the data processing method according to any embodiment of this application.

[0019] This invention performs matrix multiplication operations using scaling factors by reusing dot product units of different precisions in the tensor kernel, thereby reducing additional hardware area costs, making full use of existing hardware resources, reducing the number of multipliers and eliminating the need for additional adders, reducing hardware area costs, optimizing system performance, reducing hardware costs and power consumption, and improving chip integration. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0021] Figure 1A This is a schematic diagram of the architecture of a general-purpose graphics processing unit (GPGPU).

[0022] Figure 1B This is a schematic diagram of a tensor kernel structure.

[0023] Figure 2 A schematic structural diagram of a tensor kernel provided for at least one embodiment of this disclosure;

[0024] Figure 3 A schematic structural diagram of a scaling factor matrix multiplication processing module 102 provided in at least one embodiment of this disclosure;

[0025] Figure 4 A schematic diagram illustrating a first tensor, a second tensor, a scaling factor of the first tensor, and a scaling factor of the second tensor, provided for at least one embodiment of this disclosure;

[0026] Figure 5 A schematic diagram illustrating a first operation provided for at least one embodiment of this disclosure;

[0027] Figure 6 A schematic structural diagram of a processor provided for at least one embodiment of this disclosure;

[0028] Figure 7 A schematic flowchart illustrating a data processing method provided in at least one embodiment of this disclosure;

[0029] Figure 8 A schematic block diagram of an electronic device provided in one embodiment of this disclosure;

[0030] Figure 9 This is a schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0032] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described object changes. To keep the following description of the embodiments of this disclosure clear and concise, detailed descriptions of some known functions and components are omitted.

[0033] Floating-point numbers (FP) are primarily used to represent decimals and typically consist of three parts: a sign bit, an exponent, and a mantissa. The exponent part can also be called the exponent code. For example, a floating-point number V can usually be represented in the following form:

[0034]

[0035] In this context, the sign bit s can be 1 bit, determining whether the floating-point number V is negative or positive; M represents the mantissa, which can include multiple bits and is a binary fractional form, defining the precision of the floating-point number; E represents the exponent (also called the exponent value), used to weight the floating-point number, reflecting the position of the decimal point in the floating-point number V, and defining the range of values ​​for the floating-point number.

[0036] Traditional floating-point numbers typically include three formats: half-precision floating-point (FP16), single-precision floating-point (FP32), and double-precision floating-point (FP64), with different numbers of bits in their exponent and mantissa parts.

[0037] AI accelerators are widely used for training deep learning models. For convolution operations, common in deep learning models, both hardware and software designs have been specifically optimized to accelerate computation. For example, various floating-point data formats have been developed and optimized for fields such as artificial intelligence and deep learning, including BF16 (brain floating point 16, 16-bit width), BF24 (brain floating point 24, 24-bit width), and TF32 (Tensor Float 32, 19-bit width). These data formats can significantly reduce computational processing, especially the computational resources and power consumption required for matrix multiplication or convolution multiplication operations. In addition, the processor also supports some common floating-point types, such as half-precision floating-point numbers (FP16, 16-bit width) or single-precision floating-point numbers (FP32, 32-bit width).

[0038] Figure 1A This is a schematic diagram of a general-purpose graphics processing unit (GPGPU).

[0039] like Figure 1A As shown, a general-purpose graphics processor is actually an array of programmable multiprocessors. For example, a programmable multiprocessor can be a streaming processor cluster (SPC), such as including... Figure 1A The diagram shows streaming processor clusters 1, ..., M, where M is a positive integer greater than 1. In a general-purpose graphics processor, one streaming processor cluster handles one computational task, or multiple streaming processor clusters handle one computational task. Multiple streaming processor clusters share data through a global cache or global memory.

[0040] like Figure 1A As shown, taking streaming processor cluster 1 as an example, one streaming processor cluster includes multiple computing units, such as... Figure 1A The computation unit is 1, computation unit 2, ..., computation unit N, where N is a positive integer.

[0041] A computing unit comprises multiple cores (also called computing kernels or computing cores). Each computing core includes an arithmetic logic unit (ALU), a floating-point unit, etc., and is used to execute specific computational tasks. In addition, the computing unit also includes registers (e.g., ... Figure 1A The register file and shared memory in a computing unit are used to store source and destination data related to computing tasks in a hierarchical manner. The shared memory in a computing unit is used to share data between the cores of that computing unit.

[0042] like Figure 1AAs shown, each computational unit also provides a tensor core for performing tensor-related computations, such as matrix multiplication operations using the GEMM operator. Tensors are a crucial data structure in deep learning; they are high-dimensional generalizations of scalars, vectors, and matrices. Tensor operations are commonly used in the training and inference of deep learning models, and tensor cores can accelerate matrix multiplication. Tensor cores across multiple computational units can be uniformly scheduled and controlled.

[0043] For example, each computing unit can also provide a Vector Core Engine (not shown in the figure) for performing vector-related computations, such as vector-related arithmetic and logical operations, such as accumulation, reduction, and regular addition, subtraction, multiplication, and division.

[0044] In parallel computing, computational tasks are typically executed by multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block distribution module. Figure 1A (Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0045] In each computing unit, the thread beam scheduling / distribution module ( Figure 1A (Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple computing cores within the computing unit can run thread bundles. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can be executed concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory execution instructions are issued to shared memory within the computing unit or further issued to intermediate-level caches (such as vector operation unit caches), global caches, or global memory for read and write operations, etc.

[0046] Low-precision matrix multiplication is increasingly being used in the training and inference of large AI models due to its significant performance gains with acceptable precision loss. In GPUs (Graphics Processing Units) or GPGPUs, matrix multiplication is typically performed in hardware by tensor kernels. Low-precision tensor kernels are several times more computationally powerful than high-precision tensor kernels, resulting in higher computational efficiency. Furthermore, the data volume of a low-precision tensor is also only a few times that of a high-precision tensor, leading to higher data transfer efficiency. Therefore, the end-to-end efficiency improvement brought by low-precision tensor computation is almost several times greater. For example, the computational power of an FP4 tensor kernel can be 2 to 8 times that of FP8, 4 to 16 times that of FP16 / BF16, or even higher. Moreover, the data volume is half that of FP8 and one-quarter that of FP16 / BF16.

[0047] A dot product unit is a hardware structure within a tensor kernel used to perform dot product operations. It efficiently calculates the dot product of elements in two vectors and sums the results to obtain the final dot product. Tensor kernels achieve large-scale matrix multiplication through the collaborative work of multiple dot product units. For example, dot product units can be organized into an array for efficient matrix multiplication.

[0048] To reduce the loss of precision, low-precision matrix multiplication introduces a scaling factor to maximize the numerical expressive power of low-precision tensors. For example, low-precision matrix multiplication includes matrix multiplication for low-precision tensors (such as FP4, FP8, etc.).

[0049] For low-precision matrix multiplication, such as D = A' × B' + C', where A', B', C', and D are all high-precision tensors, the general matrix multiplication operation using scaling factors can be described as follows:

[0050]

[0051] Where ⊙ represents element-wise multiplication. σ represents the outer product, × represents matrix multiplication, α is the scaling factor of tensor A', β is the scaling factor of tensor B', σ is the scaling factor of tensor C', and γ is the scaling factor of tensor D. Quantized tensors A, B, C, and D are the low-precision tensors obtained by quantizing tensors A', B', C', and D' respectively after floating-point quantization. For example, the data dimensions of each parameter are shown below:

[0052] A: [m,T×k], B: [T×k,n], C: [m,n], D: [m,n], α: [m,T], β: [T,n], σ: [m,r] or [r,n], γ: [m,r] or [r,n].

[0053] Where m, n, s, k, and r are all positive integers. For any row in tensor A', every k tensor elements in that row share a single scaling factor parameter, and for any column in tensor B', every k tensor elements in that column share a single scaling factor parameter.

[0054] To maintain the performance advantages of low-precision matrix computation, the scaling factor calculation and the element-wise multiplication of the low-precision tensor are performed together within the tensor kernel. As shown in Equation 1 below, the tensor kernel performs one round of A... t and B t The matrix multiplication operation (which consists of m×n element-wise multiplications, each involving k element-wise multiplications) requires the result to be scaled by a scaling factor α. t ×β t The element-wise multiplication step is performed T times in total:

[0055] (Formula 1)

[0056] t takes values ​​of 0, 1, 2, ..., T-1 in sequence. A t The data size is [m,k], representing the (t+1)th first tensor block of the T first tensor blocks into which matrix A is divided in the column direction, B t The data size is [k,n], representing the (t+1)th second tensor block among the T second tensor blocks into which matrix B is divided in the row direction. α t The data size is [m, 1], representing the (t+1)th first vector among the T first vectors into which tensor A is divided along the column direction by the scaling factor α. β t The data size is [1, n], which represents the (t+1)th second vector among the T second vectors into which the scaling factor β of tensor B is divided in the column direction.

[0057] Figure 1B This is a schematic diagram of a tensor kernel structure. (Example:) Figure 1B As shown, A in Formula 1 t ×B t The multiplication operation is performed by the dot product unit in the tensor kernel, while α t ×β t and (α) t ×β t ) (A t ×B t Each of these requires an additional multiplier to perform the operation, that is... Figure 1B The dot product and accumulation modules in [the framework / platform]. In addition, An additional adder is needed to perform this, which is also... Figure 1BThe dot product and accumulation modules in the tensor kernel are executed. Therefore, current matrix multiplication using scaling factors requires the introduction of many additional adders and multipliers within the tensor kernel, resulting in additional hardware area, increased hardware cost and power consumption, and reduced processor performance and reliability.

[0058] At least one embodiment of this disclosure provides a tensor kernel, including a first dot product unit and a scaling factor matrix multiplication processing module. The scaling factor matrix multiplication processing module includes a second dot product unit. The first and second dot product units are configured to support dot product operations with different floating-point precisions. The tensor kernel is configured to receive a first tensor, a second tensor, a scaling factor of the first tensor, and a scaling factor of the second tensor. It performs a matrix multiplication operation using the scaling factor using the first dot product unit and the scaling factor matrix multiplication processing module to obtain a matrix multiplication result of the first and second tensors. The first and second tensors are both of first floating-point precision. During the matrix multiplication operation using the scaling factor, the first dot product unit is configured to perform a matrix multiplication operation of the first and second tensors with first floating-point precision to obtain a first multiplication result. The scaling factor matrix multiplication processing module is configured to determine the matrix multiplication result using the second dot product unit based on the first multiplication result, the scaling factor of the first tensor, and the scaling factor of the second tensor.

[0059] In at least one embodiment, this disclosure performs matrix multiplication operations using scaling factors by reusing dot multiplication units of different precisions in the tensor kernel, thereby reducing additional hardware area costs, fully utilizing existing hardware resources, reducing the need for multipliers and adding additional adders, reducing hardware area costs, optimizing system performance, reducing hardware costs and power consumption, and improving chip integration.

[0060] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, but this disclosure is not limited to these specific embodiments.

[0061] Figure 2 A schematic structural diagram of a tensor kernel provided for at least one embodiment of this disclosure.

[0062] like Figure 2 As shown, the tensor kernel 100 includes a first dot product unit 101 and a scaling factor matrix multiplication processing module 102. The scaling factor matrix multiplication processing module 102 includes a second dot product unit 103. The first dot product unit 101 and the second dot product unit 103 support dot product operations with different floating-point precisions.

[0063] It should be noted that, Figure 2 The first dot product unit 101 and the second dot product unit 103 shown are for illustrative purposes. The tensor kernel may include multiple first dot product units 101 and multiple second dot product units 103. The multiple first dot product units 101 and multiple second dot product units 103 may be arranged in an array.

[0064] like Figure 2 As shown, the tensor kernel 100 also includes a cross-precision multiplication unit hardware path, indicated by the black arrow, between the first dot product unit 101 and the scaling factor matrix multiplication processing module 102. This cross-precision multiplication unit hardware path is configured to transmit the first multiplication result Y to the scaling factor matrix multiplication processing module 102. If the tensor kernel includes multiple first dot product units 101, each of the multiple first dot product units 101 has its own cross-precision multiplication unit hardware path, used to transmit the dot product results calculated by each first dot product unit 101 to the scaling factor matrix multiplication processing module for subsequent processing.

[0065] For example, the first dot product unit 101 supports dot product operations with a first floating-point precision, and the second dot product unit 103 supports dot product operations with a second floating-point precision. The first floating-point precision and the second floating-point precision are different.

[0066] For example, if the precision of the first floating-point number is lower than that of the second floating-point number, the second dot product unit supports dot product operations with a precision of the second floating-point number that is higher than that of the first floating-point number.

[0067] For example, the first floating-point precision is FP4, and the second floating-point precision includes FP8 and higher floating-point precisions (e.g., FP16, FP32, etc.). For example, in this embodiment, the second dot product unit 103 supports dot product operations of multiple floating-point precisions, but does not support dot product operations of the first floating-point precision. Of course, this disclosure is not limited to this, and the second dot product unit 103 may also support only one floating-point precision dot product operation.

[0068] The first dot product unit 101 and the second dot product unit 103 operate independently and do not reuse each other when performing dot product operations with first or second floating-point precision. For example, when the first dot product unit 101 performs a dot product operation with first floating-point precision, the second dot product unit 103 will not perform a dot product operation with first floating-point precision, for example, it may be in an idle state; similarly, when the second dot product unit 103 performs a dot product operation with second floating-point precision, the first dot product unit 101 will not perform a dot product operation with second floating-point precision.

[0069] For example, the minimum number of dot products supported by the second dot product unit in a single dot product operation is N, where N is a positive integer greater than 1.

[0070] For example, taking the second dot product unit supporting FP8 precision dot product operations as an example, the minimum number of dot products supported in a single dot product operation by the second dot product unit is 32. That is, when performing a dot product operation with FP8 floating-point precision, at least 32 pairs of elements are multiplied and then summed. The result of the dot product operation is this sum. Taking the second dot product unit supporting FP16 dot product operations as an example, N is 16; taking the second dot product unit supporting FP32 dot product operations as an example, N is 8.

[0071] Similar restrictions may also exist for the first dot product unit. For example, if the first dot product unit supports FP4 dot product operations, the minimum number of dot products supported by the second dot product unit in a single dot product operation is 16.

[0072] Of course, this disclosure is not limited to this. The minimum number of dot products supported by the first dot product unit and the second dot product unit in a single dot product operation can be set according to hardware requirements.

[0073] Tensor kernel 100 is configured to receive a first tensor A, a second tensor B, a scaling factor α of the first tensor, and a scaling factor β of the second tensor. It then performs a matrix multiplication operation using the scaling factor using a first dot multiplication unit 101 and a scaling factor matrix multiplication processing module 102 to obtain the matrix multiplication result of the first tensor A and the second tensor B. In this context, both the first tensor A and the second tensor B have the first floating-point precision.

[0074] During the matrix multiplication operation using a scaling factor, the first dot multiplication unit 101 is configured to perform a matrix multiplication operation with a first floating-point precision for the first tensor A and the second tensor B, to obtain a first multiplication result Y.

[0075] The first floating-point precision is low, such as FP4. The first dot multiplication unit 101 is configured to perform low-precision matrix multiplication, which has high computing power and high data transmission efficiency.

[0076] The scaling factor matrix multiplication processing module 102 is configured to determine the matrix multiplication operation result E using the second dot multiplication unit 103 based on the first multiplication result Y, the scaling factor α of the first tensor, and the scaling factor β of the second tensor.

[0077] Figure 3 This is a schematic structural diagram of a scaling factor matrix multiplication processing module 102 provided in at least one embodiment of the present disclosure.

[0078] like Figure 3 As shown, the scaling factor matrix multiplication processing module 102 further includes a scaling factor multiplication processing submodule 104. For example, the scaling factor multiplication processing submodule 104 includes a multiplier.

[0079] The scaling factor multiplication processing submodule 104 is configured to determine the second multiplication result γ of the scaling factor α of the first tensor and the scaling factor β of the second tensor.

[0080] The second dot multiplication unit 103 is configured to determine the matrix multiplication operation result E based on the second multiplication result γ and the first multiplication result Y.

[0081] In the tensor kernel provided in at least one embodiment of this disclosure, reference is made to... Figure 1B and Figure 2 The dot product and accumulation modules have been eliminated, thereby reducing the need for multipliers and adders, saving on operations such as accumulating the results of the dot product unit calculations, and conserving hardware area and overhead. In addition, by adding pathways between dot product units of different precisions, i.e., hardware pathways across precision dot product units, the second dot product unit can be reused to perform dot product and accumulation operations other than low-precision matrix multiplication, reducing additional hardware area costs and making full use of existing hardware resources.

[0082] Figure 4 This is a schematic diagram of a first tensor, a second tensor, a scaling factor of the first tensor, and a scaling factor of the second tensor, provided for at least one embodiment of this disclosure.

[0083] like Figure 4 As shown, the first tensor A is divided into T blocks along the column direction, for example... Figure 4 The first block A0, the first block A1, ..., the first block A T-1 Each first tensor has a shape and size of m×k. The second tensor is divided into T second tensors in the row direction, for example... Figure 4 The second volume block B0, the second volume block B1, ..., the second volume block B T-1 The shape and size of each second block are k×n.

[0084] The scaling factor α of the first tensor is divided into T first vectors along the column direction, for example... Figure 4 The first vector α0, the first vector α1, ..., the first vector α T-1 Each first vector has a shape size of m×1. The scaling factor β of the second tensor divides it into T second vectors along the row direction, each second vector having a shape size of 1×n.

[0085] For example, the result E of a matrix multiplication operation can be expressed as:

[0086] (Formula 2)

[0087] The first dot multiplication unit 101 is configured to perform matrix multiplication operations between T first tensor blocks and T second tensor blocks to obtain T intermediate matrix multiplication results Y0, Y1, ... Y T-1 The first multiplication result includes the T intermediate results of matrix multiplication, and the shape and size of each intermediate result of matrix multiplication is m×n.

[0088] For example, the scaling factor multiplication processing submodule 104 is configured to perform outer product operations between T first vectors and T second vectors, obtaining T outer product operation results γ0, γ1, ... γ T-1 The second multiplication result includes the results of the T outer product operations, and the shape and size of each outer product operation result is m×n.

[0089] The shape and size of the result E of the matrix multiplication operation are also m×n.

[0090] The following describes in detail the execution process of each hardware module when performing specific processing operations.

[0091] For example, each first dot multiplication unit 101 is configured to perform a matrix multiplication operation between one first tensor block and its corresponding second tensor block, resulting in one intermediate matrix multiplication result. Multiple first dot multiplication units execute matrix multiplication operations between multiple first tensor blocks and multiple second tensor blocks in parallel.

[0092] For example, one first dot product unit 101 is configured to perform a matrix multiplication operation A0×B0 between the first tensor block A0 and the second tensor block B0, obtaining an intermediate matrix multiplication result Y0. Another first dot product unit 101 is configured to perform a matrix multiplication operation A1×B1 between the first tensor block A1 and the second tensor block B1, obtaining an intermediate matrix multiplication result Y1, and so on. For example, T first dot product units can execute T matrix multiplication operations in parallel, obtaining T intermediate matrix multiplication results, namely Y0, Y1, ... Y0. T-1 .

[0093] For example, when the first dot multiplication unit 101 performs a matrix multiplication operation on the first tensor block and the corresponding second tensor block, it includes performing m×n second operations, wherein each second operation includes performing dot multiplication calculation operations on k pairs of elements.

[0094] Taking the first block A0 and the second block B0 as an example, when the first dot multiplication unit 101 performs matrix multiplication operations on the first block A0 and the second block B0 to obtain an intermediate matrix multiplication result Y0 with a data size of m×n, it includes performing m×n second operations. Each second operation includes performing dot multiplication calculations on k pairs of elements and determining the sum of the results of the k dot multiplication calculations. Specifically, the second operation includes performing dot multiplication calculations between the k elements of the first row of the first block A0 and the k elements of the first column of the first block B0, and obtaining the sum as the element in the first row and first column of the intermediate matrix multiplication result Y0.

[0095] For example, k represents the number of elements sharing the same scaling factor. A larger k may reduce the effect of the scaling factor and decrease computational accuracy. T is related to k and the width of the first tensor. A larger T results in higher parallelism, but also higher computational overhead. The selection of T and k can be set to balance computational accuracy requirements and hardware resource overhead. For example, in one embodiment, k=16. Depending on the width of the first tensor A, T can be set to different values, such as T=4, 8, 16, etc. Of course, this disclosure does not impose specific limitations on this.

[0096] For example, the scaling factor multiplication processing submodule 104 includes multiple multipliers configured to perform multiple outer product operations in parallel. For instance, one multiplier performs an outer product operation between a first vector α0 and a second vector β0, obtaining the outer product result γ0; another multiplier performs an outer product operation between a first vector α1 and a second vector β1, obtaining the outer product result γ1, and so on. The second multiplication result γ includes the results of these T outer product operations.

[0097] The second dot product unit is configured to determine the matrix multiplication result E based on the intermediate results of T matrix multiplications and the results of T outer product operations.

[0098] For example, multiple second-point multiplication units are configured to execute multiple first operations in parallel to obtain the matrix multiplication result through m×n first operations.

[0099] For example, expanding formula 2 above yields formula 3 as follows:

[0100]

[0101] in, Let represent the element in the i-th row and j-th column of the result γ0 of the outer product operation. Let Y0 represent the element in the i-th row and j-th column of the intermediate result Y0 from matrix multiplication. This represents the element in the i-th row and j-th column of the result γ1 of the outer product operation. This represents the element in the i-th row and j-th column of the intermediate result Y1 of matrix multiplication, and so on.

[0102] For example, the first operation includes performing a multiplication and accumulation operation on the element in the i-th row and j-th column of the intermediate results of T matrix multiplications and the results of T outer product operations to obtain the element in the i-th row and j-th column of the matrix multiplication operation result, where i is a positive integer less than or equal to m and j is a positive integer less than or equal to n.

[0103] For example, the second dot product unit is configured to execute a first operation at a time. When executing the first operation, the second dot product unit includes the following operations: determining the element in the i-th row and j-th column of each intermediate matrix multiplication result in the T intermediate matrix multiplication results to obtain T first elements; determining the element in the i-th row and j-th column of each outer product operation result in the T outer product operation results to obtain T second elements; and performing a dot product operation between the T elements and the T second elements to obtain the element in the i-th row and j-th column of the matrix multiplication operation result.

[0104] Specifically, referring to Formula 3, the first operation can be expressed as Formula 4 as follows:

[0105]

[0106] For example, This represents the element in the i-th row and j-th column of the result of the matrix multiplication operation. The other parameters are defined as described above and will not be repeated here.

[0107] When the second dot multiplication unit 103 performs the first operation, it determines the element in the i-th row and j-th column of each intermediate matrix multiplication result in the T intermediate matrix multiplication results, obtaining the T first elements, which are represented as follows: ; Determine the i-th row and j-th column element of each of the T outer product results to obtain T second elements, denoted as follows: Perform dot product operations on T elements and T second elements to obtain the element in the i-th row and j-th column of the matrix multiplication result. Specifically, the dot product operation here, as shown in Formula 4, includes performing dot product calculations on each first element and its corresponding second element, and summing the results of the dot product calculations as the element in the i-th row and j-th column of the matrix multiplication result. .

[0108] For example, multiple second-point multiplication units can perform the first operation described above in parallel. Each first operation can obtain one element in the matrix multiplication result. By performing m×n first operations, all elements in the matrix multiplication result E can be obtained, thus obtaining the matrix multiplication result.

[0109] In the above embodiments, this disclosure implements the accumulation and element-wise multiplication operations in matrix multiplication using scaling factors through a second dot product unit. This reuses the second dot product unit, which is idle when performing matrix multiplication operations with first floating-point precision, making full use of existing hardware, reducing the number of multipliers, avoiding additional adders, and saving hardware area. Furthermore, since the floating-point precision of the dot product operations supported by the second dot product unit is higher than that of the first floating-point unit, using the second dot product unit to perform multiplication and addition operations can retain more valid data and keep the calculation accuracy unaffected.

[0110] As mentioned above, the minimum number of dot products N supported by the second dot product unit in a single dot product operation is limited. In at least one embodiment of this disclosure, when selecting the second dot product unit, it is necessary to select a dot product unit whose minimum number of dot products N supported in a single dot product operation is greater than or equal to T as the second dot product unit, where N is a positive integer. This ensures the balance between hardware overhead and operator execution speed and efficiency to the greatest extent.

[0111] For example, when the second dot product unit performs dot product operations on T first elements and T second elements, it includes performing N dot product calculation operations and determining the sum of the results of the N dot product calculation operations. In other words, due to the limitation on the minimum number of dot product operations, the minimum number of dot product operations in a single dot product operation of the second dot product unit is at least N, that is, at least N dot product calculation operations need to be performed.

[0112] Figure 5 This is a schematic diagram of a first operation provided for at least one embodiment of the present disclosure.

[0113] like Figure 5 As shown, in response to N being greater than T, when performing N dot product operations, N dot product operations of the first element and N dot product operations of the second element are performed. Here, the N first elements include T first elements, i.e. Furthermore, the N first elements include NT zeros; the N second elements include T second elements, that is... In addition, the N second elements also include N zeros. When the second dot product unit performs a dot product operation, each dot product operation performs the dot product of two elements in the same column, and then the results of the N dot product operations are added together.

[0114] Therefore, when using the second dot product unit to perform the first operation, by adding 0, the second dot product unit still performs N dot product calculation operations, but the actual calculation result is the result of the dot product operations of T first elements and T second elements, thus reusing the second dot product unit.

[0115] For example, in one specific embodiment, the first floating-point precision is FP4, the first dot product unit supports FP4 dot product operations, and the second dot product unit supports FP8 and dot product operations with precision higher than FP8. For example, assuming k=16, T is set according to the width of the first tensor, such as T=4, 8, 16, etc.

[0116] The first dot-matrix unit performs FP4 floating-point precision matrix multiplication operations on the first and second tensors to obtain the first multiplication result. For example, T first dot-matrix units perform matrix multiplication operations between T first tensor blocks and T second tensor blocks in parallel to obtain T intermediate matrix multiplication results. The first multiplication result includes the T intermediate matrix multiplication results. The specific process is described above and will not be repeated here.

[0117] The scaling factor multiplication processing submodule performs the outer product operation between T first vectors and T second vectors. For example, it uses T multipliers to perform the outer product operation between T first vectors and T second vectors in parallel to obtain the result of T outer product operation.

[0118] Multiple second-point multiplication units execute multiple first operations in parallel, and the matrix multiplication result is obtained through m×n first operations.

[0119] For example, after obtaining the matrix multiplication result E, if C is not equal to 0, the tensor kernel is also configured to add the matrix multiplication result E to σ⊙C to obtain the final matrix multiplication result.

[0120] For example, in one embodiment, when the second dot product unit performs a dot product operation with FP8 precision, the minimum number of dot products supported in a single dot product operation is 32; when the second dot product unit performs a dot product operation with FP16 precision, the minimum number of dot products supported in a single dot product operation is 16; and when the second dot product unit performs a dot product operation with FP32 precision, the minimum number of dot products supported in a single dot product operation is 8.

[0121] Assuming T=4, the first operation can be performed using a dot product operation with FP8, FP16, or FP32 precision. For example, when using FP8, the dot product operation fills all positions except the four first elements and four second elements with 0, i.e., the remaining 2 × (32-4) positions are filled with 0. Similarly, when using FP16, the dot product operation fills all positions except the four first elements and four second elements with 0, and the remaining 2 × (16-4) positions are filled with 0. Finally, when using FP32, the dot product operation fills all positions except the four first elements and four second elements with 0, and the remaining 2 × (8-4) positions are filled with 0.

[0122] Considering accuracy requirements and computational overhead, in the above embodiment, the second dot multiplication unit performs the first operation optimally using FP16.

[0123] If T=16, then the first operation can be performed using a dot product operation with FP8 or FP16 precision. If T=32, then the first operation can be performed using a dot product operation with FP8 precision.

[0124] In the above embodiment, the dot product operation of the first multiplication result and the second multiplication result, as well as the summation operation of the result of the T rounds of the dot product operation, are performed by reusing the second dot product unit in the tensor kernel, which was originally in an idle state. Since the second dot product unit is independent of the first dot product unit and was originally in an idle state when the first dot product unit performs a matrix multiplication operation with a first floating-point precision, reusing the second dot product unit can improve the utilization of hardware resources, make full use of existing hardware resources, reduce the multiplier setup required for performing the dot product operation of the first multiplication result and the second multiplication result, and reduce the adder setup required for performing the summation operation of the result of the T rounds of the dot product operation, thereby reducing the hardware area cost of additional multipliers and adders required to implement low-precision matrix multiplication using scaling factors.

[0125] Figure 6 This is a schematic structural diagram of a processor provided for at least one embodiment of the present disclosure. For example... Figure 6 As shown, processor 200 includes tensor core 100.

[0126] For example, the tensor kernel includes a first dot product unit and a scaling factor matrix multiplication processing module, the scaling factor matrix multiplication processing module includes a second dot product unit, and the first and second dot product units are configured to support dot product operations with different floating-point precisions.

[0127] The tensor kernel is configured to receive a first tensor, a second tensor, a scaling factor of the first tensor, and a scaling factor of the second tensor. It then performs a matrix multiplication operation using the scaling factor by utilizing the multiplication unit and the scaling factor matrix multiplication processing module to obtain the matrix multiplication result of the first tensor and the second tensor. Both the first tensor and the second tensor are of the first floating-point precision.

[0128] During the matrix multiplication operation using the scaling factor, the first dot multiplication unit is configured to perform a matrix multiplication operation of the first tensor and the second tensor with the first floating-point precision to obtain a first multiplication result. The scaling factor matrix multiplication processing module is configured to determine the matrix multiplication operation result using the second dot multiplication unit based on the first multiplication result, the scaling factor of the first tensor, and the scaling factor of the second tensor.

[0129] For details regarding the specific structure and function of tensor core 100, please refer to the relevant descriptions of tensor core 100 in the aforementioned embodiments, which will not be repeated here.

[0130] For example, the processor may include a graphics processor, reference Figure 1A The graphics processor architecture shown includes multiple streaming processor clusters and memory. Each streaming processor cluster includes multiple computing units, and each computing unit includes a tensor core.

[0131] Of course, the processor can also be other processors with tensor cores, such as neural network processors, digital signal processors, central processing units, etc., and this disclosure does not impose specific limitations on them.

[0132] In at least one embodiment, the processor of this disclosure performs matrix multiplication operations using scaling factors by reusing dot multiplication units of different precisions in the tensor kernel, thereby reducing additional hardware area costs, making full use of existing hardware resources, reducing the number of multipliers and eliminating the need for additional adders, reducing hardware area costs, optimizing system performance, reducing hardware costs and power consumption, and improving chip integration.

[0133] In addition, it should be noted that Figure 6 The components of the processor 200 shown are merely exemplary and not limiting. The processor 200 may have other components depending on the actual application requirements.

[0134] At least one embodiment of this disclosure also provides a data processing method. Figure 7 This is a schematic flowchart illustrating a data processing method provided in at least one embodiment of the present disclosure.

[0135] The data processing method is applied to the tensor kernel. The tensor kernel includes a first dot product unit and a scaling factor matrix multiplication processing module. The scaling factor matrix multiplication processing module includes a second dot product unit. The first and second dot product units support dot product operations with different floating-point precisions. For more information about the tensor kernel, please refer to the relevant description of tensor kernel 100 in the foregoing embodiments, which will not be repeated here.

[0136] like Figure 7 As shown, at least one embodiment of the present disclosure provides a data processing method including steps S10-S20.

[0137] In S10, the first tensor, the second tensor, the scaling factor of the first tensor, and the scaling factor of the second tensor are received.

[0138] In S20, the matrix multiplication operation using the scaling factor is performed using the first dot multiplication unit and the scaling factor matrix multiplication processing module to obtain the matrix multiplication result of the first tensor and the second tensor.

[0139] Both the first and second tensors have the first floating-point precision.

[0140] For example, S20 may include: performing a matrix multiplication operation of the first tensor and the second tensor with the first floating-point precision using the first dot multiplication unit to obtain a first multiplication result; and determining the matrix multiplication operation result using the second dot multiplication unit based on the first multiplication result, the scaling factor of the first tensor, and the scaling factor of the second tensor.

[0141] For example, the scaling factor matrix multiplication processing module also includes a scaling factor multiplication processing submodule, which may include multiple multipliers.

[0142] Based on the first multiplication result, the scaling factor of the first tensor, and the scaling factor of the second tensor, the matrix multiplication operation result is determined using the second dot multiplication unit. This can include: determining the second multiplication result using the scaling factor multiplication processing submodule to determine the scaling factors of the first tensor and the second tensor; and determining the matrix multiplication operation result using the second dot multiplication unit based on the second multiplication result and the first multiplication result.

[0143] For example, the first tensor is divided into T first tensor blocks along the column direction, each with a shape size of m×k. The second tensor is divided into T second tensor blocks along the row direction, each with a shape size of k×n. The scaling factor of the first tensor is divided into T first vectors along the column direction, each with a shape size of m×1. The scaling factor of the second tensor is divided into T second vectors along the row direction, each with a shape size of 1×n. Further descriptions of the first tensor blocks, second tensor blocks, first vectors, and second vectors can be found above and will not be repeated here.

[0144] For example, in some embodiments, performing a matrix multiplication operation with a first floating-point precision on a first tensor and a second tensor using a first dot multiplication unit to obtain a first multiplication result may include: performing matrix multiplication operations between T first tensor blocks and T second tensor blocks using the first dot multiplication unit to obtain T intermediate matrix multiplication results, wherein the first multiplication result includes the T intermediate matrix multiplication results.

[0145] For example, in some embodiments, the second multiplication result of determining the scaling factor of the first tensor and the scaling factor of the second tensor using the scaling factor multiplication processing submodule may include: performing an outer product operation between T first vectors and T second vectors using the scaling factor multiplication processing submodule to obtain T outer product operation results, wherein the second multiplication result includes the T outer product operation results.

[0146] For example, in some embodiments, determining the matrix multiplication operation result using a second dot product unit based on the second multiplication result and the first multiplication result may include: determining the matrix multiplication operation result using a second dot product unit based on T intermediate matrix multiplication results and T outer product operation results.

[0147] The data processing method provided in at least one embodiment of this disclosure performs matrix multiplication operations using scaling factors by reusing dot multiplication units of different precisions in the tensor kernel. This reduces additional hardware area costs, fully utilizes existing hardware resources, reduces the need for multipliers and adds, reduces hardware area costs, optimizes system performance, reduces hardware costs and power consumption, and improves chip integration.

[0148] It should be noted that, depending on the application domain, the first tensor and the second tensor can have different physical meanings.

[0149] For example, in the field of speech processing, the first and second tensors can be any parameters that require matrix multiplication operations and are used, input, or generated in tasks such as feature extraction, speech enhancement, and speech recognition, such as speech feature vectors and filtering parameters.

[0150] For example, in the field of image processing, the first and second tensors can be any parameters that require matrix multiplication operations and are used, input, or generated in tasks such as image preprocessing, feature extraction, image segmentation, and object detection. These parameters include image feature vectors, various edge detection operators (such as Sobel, Canny, and Prewitt operators), image filtering operators (such as Gaussian filtering, median filtering, and bilateral filtering), and morphological operators (such as erosion, dilation, opening, and closing operations).

[0151] For example, in the field of text processing, the first and second tensors can be any parameters that require matrix multiplication operations and are used, input, or generated in tasks such as text classification, sentiment analysis, and text generation, such as the semantic feature vector of the text.

[0152] For example, in the field of video processing, the first tensor and the second tensor can be parameters used in the field of image processing as described above, or parameters used, input, or generated by video processing-specific operators such as optical flow operators (used to estimate motion between video frames) and target tracking operators (used to track specific targets in video).

[0153] Of course, this disclosure is not limited to this. For other application scenarios or fields, such as computer graphics, machine learning, data mining, signal processing, natural language processing, geographic information systems, oceanography, environmental science and other technical fields, as long as matrix multiplication operations are required, the data processing method described in at least one embodiment of this disclosure can be applied, and will not be described in detail here.

[0154] Figure 8 This is a schematic block diagram of an electronic device provided in one embodiment of the present disclosure.

[0155] like Figure 8 As shown, the electronic device 300 is, for example, suitable for implementing the data processing method provided in the embodiments of this disclosure. It should be noted that... Figure 8 The components of the electronic device 300 shown are merely exemplary and not limiting. The electronic device 300 may have other components as needed for the actual application.

[0156] like Figure 8 As shown, the electronic device 300 may include a processing unit 301, such as the aforementioned processor 200, which can perform various appropriate actions and processes according to non-transitory computer-readable instructions stored in memory to achieve various functions. For example, the processing unit 301 may also include devices with instruction optimization capabilities and / or program execution capabilities, such as a central processing unit (CPU) or a tensor processor (TPU). The central processing unit (CPU) can be an x86, ARM, or RISC-V architecture, etc. The GPU can be directly integrated into the SOC, directly integrated into the motherboard, or built into the northbridge chip of the motherboard.

[0157] For example, when the computer-readable instruction processing device 301 executes, it can perform one or more steps of the data processing method according to any of the above embodiments. It should be noted that a detailed description of the processing procedure of the data processing method can be found in the relevant descriptions in the embodiments of the above data processing method.

[0158] like Figure 8 As shown, for example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 303 and / or cache memory, etc., for example, computer-readable instructions may be loaded from storage device 308 into RAM 303 to execute computer-readable instructions. Non-volatile memory may include, for example, read-only memory (ROM) 302, hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. Various applications and various data, such as various data used and / or generated by applications, may also be stored in the computer-readable storage medium.

[0159] For example, the processing device 301, the read-only memory (ROM) 302, and the random access memory (RAM) 303 are interconnected via a bus 304. The input / output (I / O) interface 305 is also connected to the bus 304.

[0160] Typically, the following devices can be connected to the input / output (I / O) interface 305: input devices 306 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 308 including, for example, magnetic tape, hard disk, flash memory, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 8 An electronic device 300 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 300 may alternatively implement or have more or fewer devices. For example, the processing device 301 may control other components in the electronic device 300 to perform desired functions.

[0161] For example, the electronic device can be in the form of a server and can be used in various application scenarios such as deep learning and artificial intelligence, scientific computing, graphics rendering and video editing, virtual reality and game development, and cloud services.

[0162] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0163] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0164] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0165] Figure 9 This is a schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. For example, such as Figure 9 As shown, storage medium 400 can be a non-transitory computer-readable storage medium on which one or more computer-readable instructions 401 can be stored non-transitory. For example, when the computer-readable instructions 401 are executed by a processor, one or more steps in the data processing method described above can be performed.

[0166] For example, the storage medium 400 can be used in an electronic device 300, such as the storage medium 400 including the storage device 308 in the electronic device 300.

[0167] For example, a storage device may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB storage, flash memory, etc. One or more computer-readable instructions may be stored on the computer-readable storage medium, and a processor may execute these instructions to perform various functions of the processor. Various application programs and various data may also be stored in the storage medium.

[0168] For example, the storage medium may include a memory card for a smartphone, a cache component for a tablet computer, a hard disk for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.

[0169] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0170] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0171] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0172] The following points should be noted regarding this disclosure:

[0173] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0174] (2) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0175] The above description is only a specific embodiment of this disclosure, but the protection scope of this disclosure is not limited thereto. The protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A tensor kernel, characterized in that, It includes a first dot product unit, a scaling factor matrix multiplication processing module, and a hardware path for cross-precision dot product units. The scaling factor matrix multiplication processing module includes a second dot product unit. The first and second dot product units are configured to support dot product operations with different floating-point precisions. The tensor kernel is configured to receive a first tensor, a second tensor, a scaling factor of the first tensor, and a scaling factor of the second tensor. It then performs a matrix multiplication operation using the scaling factor using the first dot product unit and the scaling factor matrix multiplication processing module to obtain the matrix multiplication result of the first tensor and the second tensor. Both the first tensor and the second tensor are of first floating-point precision. During the matrix multiplication operation using the scaling factor, the first point multiplication unit is configured to perform a matrix multiplication operation with the first floating-point precision on the first tensor and the second tensor, obtaining a first multiplication result. The hardware path of the cross-precision dot product unit is configured to transmit the first multiplication result to the scaling factor matrix multiplication processing module. The scaling factor matrix multiplication processing module is configured to determine the matrix multiplication operation result using the second dot multiplication unit based on the first multiplication result, the scaling factor of the first tensor, and the scaling factor of the second tensor.

2. The tensor kernel according to claim 1, characterized in that, The scaling factor matrix multiplication processing module also includes a scaling factor multiplication processing submodule. The scaling factor multiplication processing submodule is configured to determine the second multiplication result of the scaling factor of the first tensor and the scaling factor of the second tensor; The second dot product unit is configured to determine the result of the matrix multiplication operation based on the second multiplication result and the first multiplication result.

3. The tensor kernel according to claim 1, characterized in that, The second dot product unit supports dot product operations with a second floating-point precision higher than that of the first floating-point number.

4. The tensor kernel according to claim 2, characterized in that, The first tensor is divided into T first tensor blocks in the column direction, each with a shape and size of m×k. The second tensor is divided into T second tensor blocks in the row direction, each with a shape and size of k×n. The scaling factor of the first tensor is divided into T first vectors in the column direction, each with a shape size of m×1. The scaling factor of the second tensor is divided into T second vectors in the row direction, each with a shape size of 1×n. The first dot multiplication unit is configured to perform matrix multiplication operations between the T first tensor blocks and the T second tensor blocks to obtain T intermediate matrix multiplication results, wherein the first multiplication result includes the T intermediate matrix multiplication results; The scaling factor multiplication processing submodule is configured to perform an outer product operation between the T first vectors and the T second vectors to obtain T outer product operation results, wherein the second multiplication result includes the T outer product operation results; The second dot product unit is configured to determine the result of the matrix multiplication operation based on the T intermediate results of matrix multiplication and the T outer product operation results. Where m, n, k, and T are all positive integers.

5. The tensor kernel according to claim 4, characterized in that, The shape and size of the matrix multiplication result, the intermediate result of each matrix multiplication, and the result of each outer product operation are all m×n. Multiple second point multiplication units are configured to execute multiple first operations in parallel to obtain the matrix multiplication result through m×n first operations. The first operation includes performing a multiplication and accumulation operation on the element in the i-th row and j-th column of the T intermediate matrix multiplication results and the T outer product operation results to obtain the element in the i-th row and j-th column of the matrix multiplication operation result. Where i is a positive integer less than or equal to m, and j is a positive integer less than or equal to n.

6. The tensor kernel according to claim 5, characterized in that, The second dot multiplication unit is configured to perform one first operation at a time. When the second dot multiplication unit performs the first operation, it includes the following operations: Determine the element in the i-th row and j-th column of each of the T intermediate matrix multiplication results to obtain the T first elements; Determine the element in the i-th row and j-th column of each of the T outer product operation results to obtain the T second elements; Perform a dot product operation between the T first elements and the T second elements to obtain the element in the i-th row and j-th column of the matrix multiplication result.

7. The tensor kernel according to claim 6, characterized in that, The minimum number of dot product operations N supported by the second dot product unit in a single dot product operation is greater than or equal to T, where N is a positive integer. When the second dot product unit performs the dot product operation of the T first elements and the T second elements, it includes performing N dot product calculation operations and determining the sum of the results of the N dot product calculation operations. In response to N being greater than T, when performing the N dot product calculation operations, N first elements and N second elements are performed, and the N first elements include the T first elements and NT zeros, and the N second elements include the T second elements and NT zeros.

8. The tensor kernel according to any one of claims 1-7, characterized in that, The first floating-point number has a precision of FP4, and the second dot multiplication unit supports dot multiplication operations with a precision of FP8 and higher.

9. The tensor kernel according to claim 8, characterized in that, The second point multiplication unit is configured to perform the process of determining the result of the matrix multiplication operation with FP16 precision.

10. The tensor kernel according to claim 4, characterized in that, Each first dot multiplication unit is configured to perform a matrix multiplication operation between a first tensor block and its corresponding second tensor block, resulting in an intermediate matrix multiplication result. Multiple first point multiplication units perform multiple matrix multiplication operations in parallel. When the first dot product unit performs a matrix multiplication operation on the first tensor block and the corresponding second tensor block, it includes performing m×n second operations, wherein each second operation includes performing dot product calculations on k pairs of elements.

11. A processor, characterized in that, Includes the tensor kernel as described in any one of claims 1-10.

12. A data processing method applied to tensor kernels, characterized in that, The tensor kernel includes a first dot product unit, a scaling factor matrix multiplication processing module, and a hardware path for cross-precision dot product units. The scaling factor matrix multiplication processing module includes a second dot product unit. The first and second dot product units are configured to support dot product operations with different floating-point precisions. The data processing method includes: Receive a first tensor, a second tensor, a scaling factor for the first tensor, and a scaling factor for the second tensor; The matrix multiplication operation using the scaling factor is performed using the first dot multiplication unit and the scaling factor matrix multiplication processing module to obtain the matrix multiplication result of the first tensor and the second tensor, wherein the first tensor and the second tensor are both of the first floating-point precision; Specifically, the matrix multiplication operation using a scaling factor is performed using the first dot multiplication unit and the scaling factor matrix multiplication processing module to obtain the matrix multiplication result of the first tensor and the second tensor, including: The first dot product unit is used to perform a matrix multiplication operation with the first floating-point precision on the first tensor and the second tensor to obtain the first multiplication result; The first multiplication result is transmitted to the scaling factor matrix multiplication processing module using the cross-precision dot product unit hardware path. The matrix multiplication result is determined using the second dot product unit based on the first multiplication result, the scaling factor of the first tensor, and the scaling factor of the second tensor.

13. The data processing method according to claim 12, characterized in that, The scaling factor matrix multiplication processing module also includes a scaling factor multiplication processing submodule. Based on the first multiplication result, the scaling factor of the first tensor, and the scaling factor of the second tensor, the matrix multiplication operation result is determined using the second dot product unit, including: The scaling factor multiplication processing submodule is used to determine the second multiplication result of the scaling factor of the first tensor and the scaling factor of the second tensor; The matrix multiplication result is determined using the second dot product unit based on the second multiplication result and the first multiplication result.

14. An electronic device, characterized in that, include: Memory stores computer-executable instructions non-transiently; The processor is configured to run computer-executable instructions. The computer-executable instructions are executed by the processor to implement the data processing method according to claim 12 or 13.

15. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the data processing method according to claim 12 or 13.

Images