Processors, electronic devices

Abstract

A processor and electronic device. The processor comprises a computing unit and a memory, the computing unit comprises a tensor core configured to perform a matrix multiplication operation using a scaling factor, the computing unit further comprises a scaling factor processing module configured to determine and cache scaling factors of each tensor related to the matrix multiplication operation, and to quantize each tensor from a first floating-point number format to a second floating-point number format, the computing unit further comprises at least one storage module arranged on a data path between the tensor core and the memory, the at least one storage module is exclusively occupied by the tensor core when the tensor core performs a tensor-related operation, and the scaling factor processing module is arranged on the at least one storage module. The scaling factor and floating-point quantization are currently completed by the vector computing core, which causes performance degradation and high delay. The scaling factor processing module arranged on the storage module in the computing unit can improve the overall execution efficiency of low-precision matrix multiplication using the scaling factor.

Description

Technical Field

[0001] Embodiments of this disclosure relate to a processor and an electronic device. Background Technology

[0002] Floating-point quantization refers to converting high-precision floating-point formats (such as FP16, BF16, FP32, etc.) into low-precision floating-point formats (such as FP4, FP8, etc.). Low-precision floating-point formats can significantly improve computational efficiency, especially for General Matrix Multiplication (GEMM) operations required by Artificial Intelligence (AI) models. However, low-precision matrix multiplication introduces precision loss, thus reducing the prediction and generation performance of AI models. Therefore, in the low-precision computation process of AI models, a scaling factor is typically calculated from the current data during floating-point quantization to scale the data's representation range, thereby reducing the precision loss in the computational results.

[0003] Currently, scaling factors and floating-point quantization are handled by the vector computation kernel. However, the vector computation kernel also needs to execute other vector-related operators. Calculating the scaling factor preempts the computational resources of these other vector-related operators, leading to performance degradation. Furthermore, the path for transmitting the scaling factor to the tensor kernel can block the transmission of other tensor data, such as occupying shared memory caches and competing for bandwidth with other data on the path, resulting in decreased overall throughput and increased latency. Summary of the Invention

[0004] This invention provides a processor, comprising a computing unit and memory. The computing unit includes a tensor core configured to perform matrix multiplication operations using a scaling factor. The computing unit further includes a scaling factor processing module configured to determine and cache scaling factors for each tensor related to the matrix multiplication operation, and to quantize each tensor from a first floating-point format to a second floating-point format. The scaling factor is used to scale the data representation range when the corresponding tensor is converted from the first floating-point format to the second floating-point format. The floating-point precision of the first floating-point format is higher than that of the second floating-point format. The computing unit also includes at least one storage module disposed on the data path between the tensor core and the memory. The at least one storage module is exclusively used by the tensor core when the tensor core performs tensor-related operations. The scaling factor processing module is disposed on the at least one storage module.

[0005] For example, in the processor provided in this application, the scaling factor processing module includes an arithmetic logic unit and a cache. The arithmetic logic unit is configured to receive each tensor in the first floating-point format, perform a scaling factor calculation operation on each tensor, and perform floating-point quantization on each tensor to obtain a quantized tensor corresponding to each tensor after floating-point quantization, wherein the quantized tensor is in the second floating-point format; the cache is configured to cache the scaling factors of each tensor.

[0006] For example, in the processor provided in this application, for any tensor of the first floating-point format received by the scaling factor processing module, the tensor is divided into multiple tensor blocks, each tensor block includes multiple tensor elements, each tensor block corresponds to a scaling factor parameter, and the multiple tensor elements share the same scaling factor parameter. When the arithmetic logic unit receives the tensors and performs the scaling factor calculation operation for each tensor, it includes performing the following operations: for each tensor block: determining the sum of the maximum absolute value among the multiple tensor elements and a preset floating-point number; determining the product of the sum and a preset constant, wherein the preset constant is the quotient of 1 and the maximum value of the precision expression range of the second floating-point format; converting the product into the precision specified by the scaling factor of the tensor to obtain the scaling factor parameter corresponding to the tensor block.

[0007] For example, in the processor provided in this application, when the arithmetic logic unit performs floating-point quantization on each tensor to obtain the quantized tensor corresponding to each tensor after floating-point quantization, it includes performing the following operations: for the tensor block: determining a division lookup table based on the scaling factor parameter; determining the quotient of the plurality of tensor elements with the scaling factor parameter based on the division lookup table, and quantizing the quotient into the second floating-point number format to obtain the quantized tensor elements corresponding to the plurality of tensor elements respectively.

[0008] For example, in the processor provided in this application, the arithmetic logic unit and the cache are located on the same storage module.

[0009] For example, in the processor provided in this application, the matrix multiplication operation using the scaling factor includes combining the scaling factors of a first tensor and a second tensor, performing matrix multiplication of the first tensor and the second tensor, and combining the scaling factor of a third tensor to determine the result of the matrix multiplication operation and the sum of the third tensor to obtain a fourth tensor as the result of the matrix multiplication operation. The cache includes a first cache and a second cache. The first cache is configured to cache the scaling factors of the first tensor, the second tensor, and the third tensor. The second cache is configured to cache the scaling factor of the fourth tensor. The at least one storage module includes a first storage module and a second storage module. The first storage module is closer to the tensor kernel than the second storage module is closer to the tensor kernel. The arithmetic logic unit and the second cache are disposed on the second storage module, and the first cache is disposed on the first storage module.

[0010] For example, in the processor provided in this application, the scaling factors of the first tensor, the second tensor, and the third tensor calculated by the arithmetic logic unit are transmitted and cached in the first cache area via the data path between the second storage module and the first storage module. The arithmetic logic unit processes the first tensor, the second tensor, and the third tensor to obtain the first quantized tensor, the second quantized tensor, and the third quantized tensor respectively after floating-point quantization. These are transmitted and cached in the first storage module via the data path between the second storage module and the first storage module. The scaling factors of the first tensor, the second tensor, and the third tensor, as well as the first quantized tensor, the second quantized tensor, and the third quantized tensor, are transmitted to the tensor core via the data path between the first storage module and the tensor core for matrix multiplication using the scaling factors.

[0011] For example, in the processor provided in this application, the fourth tensor is transmitted to the first storage module via a data path between the first storage module and the tensor core, and then transmitted to the arithmetic logic unit of the second storage module via a data path between the first storage module and the second storage module. The arithmetic logic unit is further configured to determine the scaling factor of the fourth tensor based on the fourth tensor and cache it in the second cache area, and perform floating-point quantization on the fourth tensor to obtain the fourth quantized tensor in the second floating-point format corresponding to the fourth tensor after the floating-point quantization.

[0012] For example, in the processor provided in this application, the matrix multiplication operation using the scaling factor includes combining the scaling factors of the first tensor and the second tensor to perform matrix multiplication of the first tensor and the second tensor, and combining the scaling factor of the third tensor to determine the result of the matrix multiplication operation and the sum of the third tensor to obtain a fourth tensor as the result of the matrix multiplication operation, wherein the fourth tensor is directly transmitted to other computing units or the memory via the transmission path between the tensor core and the memory.

[0013] For example, in the processor provided in this application, the at least one storage module includes at least one of a tensor kernel data processing unit and shared memory in the computing unit. The tensor kernel data processing unit is dedicated to the preprocessing of related data of the tensor kernel, and the shared memory is a storage area shared by all threads in the computing unit.

[0014] This application provides an electronic device including a processor as described in the embodiments of this application.

[0015] In at least one embodiment, by setting an additional hardware module, namely a scaling factor processing module, in the computing unit, the transmission efficiency of the scaling factor is improved, the latency caused by insufficient cache capacity is reduced, the computing pressure of the vector computing core is alleviated, the efficiency of other vector operators is improved, and the computing power of the vector computing core is reduced to save hardware area, thereby improving the overall execution efficiency of low-precision matrix multiplication using scaling factors. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0017] Figure 1 This is a schematic diagram of the architecture of a general-purpose graphics processing unit (GPGPU).

[0018] Figure 2 This is a schematic diagram illustrating the data flow of a scaling factor within a processor.

[0019] Figure 3 A schematic structural diagram of a processor provided for at least one embodiment of this disclosure;

[0020] Figure 4 A schematic diagram of the internal structure of a scaling factor processing module provided in at least one embodiment of this disclosure;

[0021] Figure 5 A schematic structural diagram of a computing unit provided in an embodiment of this disclosure;

[0022] Figure 6 A schematic structural diagram of a graphics processor provided in at least one embodiment of this disclosure;

[0023] Figure 7 A schematic block diagram of an electronic device provided in one embodiment of this disclosure;

[0024] Figure 8 This is a schematic diagram of the specific structure of another electronic device provided in at least one embodiment of the present disclosure. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0026] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described object changes. To keep the following description of the embodiments of this disclosure clear and concise, detailed descriptions of some known functions and components are omitted.

[0027] Floating-point numbers (FP) are primarily used to represent decimals and typically consist of three parts: a sign bit, an exponent, and a mantissa. The exponent part can also be called the exponent code. For example, a floating-point number V can usually be represented in the following form:

[0028]

[0029] In this context, the sign bit s can be 1 bit, determining whether the floating-point number V is negative or positive; M represents the mantissa, which can include multiple bits and is a binary fractional form, defining the precision of the floating-point number; E represents the exponent (also called the exponent value), used to weight the floating-point number, reflecting the position of the decimal point in the floating-point number V, and defining the range of values ​​for the floating-point number.

[0030] Traditional floating-point numbers typically include three formats: half-precision floating-point (FP16), single-precision floating-point (FP32), and double-precision floating-point (FP64), with different numbers of bits in their exponent and mantissa parts.

[0031] AI accelerators are widely used for training deep learning models. For convolution operations, common in deep learning models, both hardware and software designs have been specifically optimized to accelerate computation. For example, various floating-point data formats have been developed and optimized for fields such as artificial intelligence and deep learning, including BF16 (brain floating point 16, 16-bit width), BF24 (brain floating point 24, 24-bit width), and TF32 (Tensor Float 32, 19-bit width). These data formats can significantly reduce computational processing, especially the computational resources and power consumption required for matrix multiplication or convolution multiplication operations. In addition, the processor also supports some common floating-point types, such as half-precision floating-point numbers (FP16, 16-bit width) or single-precision floating-point numbers (FP32, 32-bit width).

[0032] Low-precision matrix multiplication is increasingly being used in the training and inference of large AI models due to its significant performance gains with acceptable precision loss. In GPUs (Graphics Processing Units) or GPGPUs, matrix multiplication is typically performed in hardware by tensor kernels. Low-precision tensor kernels are several times more computationally powerful than high-precision tensor kernels, resulting in higher computational efficiency. Furthermore, the data volume of a low-precision tensor is also only a few times that of a high-precision tensor, leading to higher data transfer efficiency. Therefore, the end-to-end efficiency improvement brought by low-precision tensor computation is almost several times greater. For example, the computational power of an FP4 tensor kernel can be 2 to 8 times that of FP8, 4 to 16 times that of FP16 / BF16, or even higher. Moreover, the data volume is half that of FP8 and one-quarter that of FP16 / BF16.

[0033] To reduce the loss of precision, low-precision matrix multiplication introduces a scaling factor to maximize the numerical expressive power of low-precision tensors.

[0034] For the matrix multiplication D = A × B + C, A, B, C, and D are all high-precision tensors. The general matrix multiplication operation using a scaling factor for this matrix multiplication can be described as follows:

[0035]

[0036] D→D',γ

[0037] Where ⊙ represents element-wise multiplication. σ represents the outer product, × represents matrix multiplication, α is the scaling factor for tensor A, β is the scaling factor for tensor B, σ is the scaling factor for tensor C, and γ is the scaling factor for tensor D. Quantized tensors A', B', C', and D' are the low-precision tensors obtained by floating-point quantization of tensors A, B, C, and D, respectively. For example, the data dimensions of each parameter are shown below:

[0038] A: [m,s×k], B: [s×k,n], C: [m,n], D: [m,n], α: [m,s], β: [s,n], σ: [m,r] or [r,n], γ: [m,r] or [r,n].

[0039] Where m, n, s, k, and r are all positive integers. For example, for any row in tensor A, every k tensor elements in that row share a single scaling factor parameter; for any column in tensor B, every k tensor elements in that column share a single scaling factor parameter.

[0040] Figure 1 This is a schematic diagram of a general-purpose graphics processing unit (GPGPU).

[0041] like Figure 1 As shown, a general-purpose graphics processor is actually an array of programmable multiprocessors. For example, a programmable multiprocessor can be a streaming processor cluster (SPC), such as including... Figure 1 The diagram shows streaming processor clusters 1, ..., M, where M is a positive integer greater than 1. In a general-purpose graphics processor, one streaming processor cluster handles one computational task, or multiple streaming processor clusters handle one computational task. Multiple streaming processor clusters share data through a global cache or global memory.

[0042] like Figure 1 As shown, taking streaming processor cluster 1 as an example, one streaming processor cluster includes multiple computing units, such as... Figure 1 The computation unit is denoted as Computation Unit 1, Computation Unit 2, ..., Computation Unit N, where N is a positive integer. A computation unit comprises multiple cores (also called computation kernels or computational cores). Figure 1 (Not shown in the image), each computing core includes an arithmetic logic unit (ALU), a floating-point unit, etc., and is used to perform specific computing tasks.

[0043] like Figure 1 As shown, each computational unit also provides a tensor core for performing tensor-related computations, such as matrix multiplication operations using the GEMM operator. Tensors are a crucial data structure in deep learning; they are high-dimensional generalizations of scalars, vectors, and matrices. Tensor operations are commonly used in the training and inference of deep learning models, and tensor cores can accelerate matrix multiplication. Tensor cores across multiple computational units can be uniformly scheduled and controlled.

[0044] like Figure 1 As shown, each computing unit also provides a Vector Core Engine. The Vector Core Engine is used to perform vector-related computations, such as vector-related arithmetic and logical operations, such as accumulation, reduction, and regular addition, subtraction, multiplication, and division.

[0045] In addition, the computation unit also includes a register file (not shown in the figure), shared memory, and a tensor core memory unit for storing source and destination data related to computation tasks. Shared memory within a computation unit is used to share data between cores within that unit. The tensor core memory unit is a storage resource closely related to the tensor core, used to store intermediate data when the tensor core performs tensor operations (such as matrix multiplication). Furthermore, it can perform data format processing on the tensor data to be used in tensor operations so that data loaded from external sources conforms to the tensor core's data format requirements.

[0046] In parallel computing, computational tasks are typically executed by multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block distribution module. Figure 1 (Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0047] In each computing unit, the thread beam scheduling / distribution module ( Figure 1(Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple computing cores within the computing unit can run thread bundles. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can be executed concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory execution instructions are issued to shared memory within the computing unit or further issued to intermediate-level caches, global caches, or global memory for read and write operations, etc.

[0048] Figure 2 This is a schematic diagram of the data flow in a processor for a scaling factor.

[0049] like Figure 2 As shown, currently, the scaling factor of each tensor needs to be calculated in the vector computation kernel.

[0050] For example, such as Figure 2 As shown, high-precision (e.g., BF16, FP16, etc.) tensors A, B, and C, loaded into the computing unit from memory or other locations, first enter the vector computation kernel. In the vector computation kernel, the scaling factor α of tensor A, the scaling factor β of tensor B, and the scaling factor σ of tensor C are calculated. Then, tensors A, B, and C are quantized into floating-point quantized tensors A', B', and C' with low precision (e.g., FP4, FP8, etc.). Afterwards, the scaling factors α, β, and σ, along with the quantized tensors A', B', and C', are... Figure 2 The path shown by the solid line passes through the tensor data storage unit and is transmitted to shared memory, then enters the tensor kernel. In the tensor kernel, a general matrix multiplication operation using a scaling factor is performed to obtain a high-precision calculation result, namely tensor D.

[0051] After that, tensor D along Figure 2 The path shown by the dashed line enters the vector computation kernel, where the scaling factor γ of D is calculated, and D is quantized into a floating-point tensor D'. Finally, the vector computation kernel outputs the scaling factor γ of tensor D and the quantized tensor D'.

[0052] like Figure 2 As shown, currently, a vector computation kernel is needed to calculate the scaling factor and quantize the floating-point tensor. However, the vector computation kernel also needs to execute other vector-related operators, and calculating the scaling factor will preempt the computational resources of these other vector-related operators, leading to performance degradation. Furthermore, the path for transmitting the scaling factor to the tensor kernel will block the transmission of other tensor data, for example, by occupying shared memory caches and competing with other data for transmission bandwidth, resulting in decreased overall throughput and increased latency.

[0053] At least one embodiment of this disclosure provides a scaling factor processing module for determining and caching scaling factors of tensors related to matrix multiplication operations in a processor, and quantizing each tensor from a first floating-point format to a second floating-point format. The scaling factor indicates the scaled data representation range when the corresponding tensor is converted from the first floating-point format to the second floating-point format. The floating-point precision of the first floating-point format is higher than that of the second floating-point format. The processor includes a computing unit and memory. The computing unit includes a tensor core configured to perform matrix multiplication operations using scaling factors. At least one storage module is provided on the data path between the tensor core and memory. The at least one storage module is exclusively used by the tensor core when the tensor core performs tensor operations. The scaling factor processing module is disposed on at least one storage module.

[0054] In at least one embodiment, by setting an additional hardware module, namely a scaling factor processing module, in the storage module of the computing unit, the transmission efficiency of the scaling factor is improved, the latency caused by insufficient cache capacity is reduced, the computing pressure of the vector computing core is alleviated, the efficiency of other vector operators is improved, and the computing power of the vector computing core is reduced to save hardware area, thereby improving the overall execution efficiency of low-precision matrix multiplication using scaling factors.

[0055] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, but this disclosure is not limited to these specific embodiments. Figure 3 A schematic structural diagram of a processor provided for at least one embodiment of this disclosure.

[0056] For example, such as Figure 3 As shown, processor 100 includes a computing unit 110 and memory 120. Computation unit 110 includes a tensor core 102 configured to perform matrix multiplication operations using a scaling factor. As previously described, this matrix multiplication operation using a scaling factor can be represented as... ,or .

[0057] For a description of computing unit 110, memory 120, and tensor kernel 102, please refer to the aforementioned documentation. Figure 1 The details of the introduction will not be repeated here.

[0058] The computing unit also includes a scaling factor processing module 101, which is used to determine the scaling factor of each tensor related to the matrix multiplication operation in the processor 100, and to cache the scaling factor of each tensor. In addition, it performs floating-point quantization on each tensor, quantizing each tensor from a first floating-point format to a second floating-point format.

[0059] The scaling factor is used to scale the range of data representation when the corresponding tensor is converted from the first floating-point format to the second floating-point format. The scaling factor can extend the expressive power of low-precision data.

[0060] For example, the floating-point precision of the first floating-point format is higher than that of the second floating-point format. For example, the bit width of the first floating-point format is greater than that of the second floating-point format; for example, the first floating-point format is BF16 and the second floating-point format is FP4.

[0061] During dequantization, A'×α ≈ A, meaning that A' multiplied by the scaling factor α is approximately equal to the original high-precision tensor A.

[0062] like Figure 3 As shown, at least one storage module 103 is provided on the data path between the tensor kernel 102 and the memory 120, and the at least one storage module 103 is located inside the computing unit 110. It should be noted that... Figure 3 The diagram shows one storage module, but as those skilled in the art will know, multiple storage modules can be configured on the data path, which will not be shown again here.

[0063] At least one storage module 103 is exclusively used by the tensor core 102 when performing tensor operations. The scaling factor processing module 101 is disposed on the at least one storage module 103, and the at least one storage module 103 is exclusively used by the tensor core 102 when performing tensor-related operations. For example, tensor-related operations can be any tensor-related operation, such as matrix multiplication.

[0064] For example, storage module 103 includes storage module 103 associated with tensor core 102, such as storage module that tensor core 102 can directly use.

[0065] For example, storage module 103 may include a storage module dedicated to tensor kernel 102, such as a tensor data storage unit dedicated to the preprocessing of data related to tensor kernel 102.

[0066] For example, storage module 103 may not be dedicated solely to tensor core 102, but it is exclusively used by tensor core 102 when it performs tensor operations. That is, other computation modules (such as vector computation cores) cannot use it when tensor core 102 is performing tensor-related operations. For example, storage module 103 includes shared memory, which is a storage area shared by all threads in the computation unit. When the tensor core is performing tensor-related operations, the shared memory is dedicated to the tensor core, and the vector computation core cannot use the shared memory at this time.

[0067] For a description of tensor data storage units and shared memory, please refer to the preceding text. Figure 1 The relevant information will not be repeated here.

[0068] Figure 4This is a schematic diagram of the internal structure of a scaling factor processing module provided in at least one embodiment of the present disclosure.

[0069] like Figure 4 As shown, the scaling factor processing module includes an arithmetic logic unit (ALU) 1011 and a buffer 1012.

[0070] Arithmetic logic unit 1011 is configured to receive tensors in a first floating-point format, perform the calculation of the scaling factor of each tensor, and perform floating-point quantization on each tensor to obtain the quantized tensor corresponding to each tensor after floating-point quantization, wherein the quantized tensor is in a second floating-point format.

[0071] Buffer 1012 is configured to cache the scaling factor of each tensor.

[0072] like Figure 4 As shown, the arithmetic logic unit 1011 receives high-precision tensors in a first floating-point format, namely the first tensor A, the second tensor B, and the third tensor C, performs the scaling factor calculation operation of the first tensor A, the second tensor B, and the third tensor C to obtain the scaling factor α of the first tensor A, the scaling factor β of the second tensor B, and the scaling factor σ of the third tensor C, and caches the scaling factor α, the scaling factor β, and the scaling factor σ in the buffer area 1012.

[0073] In addition, the arithmetic logic unit 1011 performs floating-point quantization on each tensor according to the scaling factor to obtain the first quantized tensor A', the second quantized tensor B', and the third quantized tensor C'.

[0074] Scaling factors α, β, and σ, along with the first quantized tensor A', the second quantized tensor B', and the third quantized tensor C', are transmitted to tensor core 102. Tensor core 102 then performs matrix multiplication operations using the scaling factors, for example, described as follows: ,or .

[0075] After performing the matrix multiplication operation described above, tensor kernel 102 will transmit the result of the matrix multiplication operation with scaling factor, i.e., the fourth tensor D, to the arithmetic logic unit. The fourth tensor D is a high-precision tensor.

[0076] For example, the precision of the fourth tensor D can be the same as that of the first tensor A, the second tensor B, and the third tensor C, or it can be different from that of the first tensor A, the second tensor B, and the third tensor C. That is, for the fourth tensor D, it can correspond to a set of first floating-point number formats and second floating-point number formats that are different from those of the first tensors. For example, in one embodiment, the floating-point number format of the first tensor A, etc. is FP16, the floating-point number format of the first quantized tensor A', etc. is FP4, the floating-point number format of the fourth tensor D is BF16, and the floating-point number format of the fourth quantized tensor D' is FP8; for example, in another embodiment, the floating-point number formats of the first tensor A, the fourth tensor D, etc. are all FP16, and the floating-point number formats of the first quantized tensor A', the fourth quantized tensor D', etc. are all FP4. Those skilled in the art can set these as needed, and this disclosure does not impose any specific limitations on them.

[0077] Arithmetic logic unit 1011 performs floating-point quantization on the fourth tensor D to obtain the fourth quantized tensor D', which is a low-precision tensor. Furthermore, it calculates the scaling factor γ of the fourth tensor D and stores it in buffer 1012.

[0078] Of course, in other embodiments, the fourth tensor D calculated by the tensor kernel can also be output to other subsequent modules without using the scaling factor processing module for related operations. That is, the floating-point quantization and scaling factor calculation of the fourth tensor D are optional. When it is not necessary to perform floating-point quantization and scaling factor calculation on the fourth tensor D, the fourth tensor is directly transmitted to other computing units or memory through the transmission path between the tensor kernel and memory.

[0079] In at least one embodiment of this disclosure, by setting up an arithmetic logic unit dedicated to performing scaling factor calculation operations and floating-point quantization, the relevant calculation operations in the vector calculation core are transferred to the arithmetic logic unit set on the storage module in the calculation unit. This avoids the occupation of the vector calculation core's computing resources due to scaling factor calculation, improves the efficiency of other vector operators, and reduces the computing power of the vector calculation unit to save hardware area.

[0080] In addition, by setting up an additional buffer to cache the scaling factors of each tensor, such as a buffer dedicated to caching scaling factors, the original limited storage resources of the storage module are avoided in the transmission path, the latency caused by insufficient cache capacity is reduced, the transmission efficiency of scaling factors is improved, and the overall throughput and the overall execution efficiency of tensor kernels performing low-precision matrix multiplication operations using scaling factors are improved.

[0081] For example, for any tensor in the first floating-point format received by the scaling factor processing module 101, the tensor is divided into multiple tensor blocks, each tensor block includes multiple tensor elements, each tensor block corresponds to a scaling factor parameter, and multiple tensor elements share a scaling factor parameter.

[0082] For example, taking a tensor A in the first floating-point format as an example, the shape size of A is [m, s×k], and the shape size of the scaling factor of tensor A is [m, s]. For example, the s×k elements in each row of A are divided into s groups, and every k consecutive elements form a tensor block, that is, each tensor block includes k tensor elements, and these k tensor elements correspond to a scaling factor parameter.

[0083] For example, when the arithmetic logic unit 1011 receives each tensor and performs the calculation of the scaling factor of each tensor, it includes performing the following operations: for each tensor block: determining the sum of the maximum value among the absolute values ​​of multiple tensor elements and a preset floating-point number; determining the product of the sum and a preset constant, wherein the preset constant is the quotient of 1 and the maximum value of the precision expression range of the second floating-point number format; converting the product result into the precision specified by the scaling factor of any tensor to obtain the scaling factor parameter corresponding to the tensor block.

[0084] For example, let's take a tensor block from tensor A as an example to illustrate the above operation. For instance, this tensor block X includes the 1st to kth tensor elements from a certain row of tensor A, such as X = [X0, X1, ..., Xk]. k-1 ],X0,X1,...,X k-1 These represent the k tensor elements respectively.

[0085] First, refer to the following formula to determine the sum:

[0086] X_max=max(abs(X))+t

[0087] When X_max equals 0, X_max() += t

[0088] X_max represents the summation result, abs() represents the absolute value function, max() represents the maximum value function, and t is a very small floating-point number (e.g., t=1e). -12 This is used to prevent scenarios involving division by zero.

[0089] Then, refer to the following formula to determine the scaling factor parameter S corresponding to the tensor block X:

[0090]

[0091] Where q_max is known in advance, 1 / q_max is a pre-calculated preset constant, q_max represents the maximum value of the precision range of the second floating-point number format. For example, if the second floating-point number format is FP4, then q_max can be 6; sType indicates that X_max×(1 / q_max) is converted into the expected precision specified by the scaling factor. For example, if the precision of the scaling factor is specified as FP8.

[0092] It should be noted that the precision of the scaling factor does not need to be the same as the second floating-point number format. For example, the precision of the result of X_max×(1 / q_max) can be higher than the precision of the scaling factor.

[0093] For example, by referring to the above process, the scaling factor parameters corresponding to each tensor block are obtained, thereby obtaining the scaling factor of the first tensor A. Repeated steps will not be repeated.

[0094] The process of determining the scaling factor for the second tensor B, the second tensor C, and the second tensor D is similar to the process of determining the scaling factor for the first tensor A, and will not be repeated here.

[0095] After the arithmetic logic unit obtains the scaling factor, when the arithmetic logic unit performs floating-point quantization on each tensor to obtain the quantized tensor corresponding to each tensor, it includes performing the following operations: For tensor blocks: Based on the scaling factor parameter, determine the division lookup table; Based on the division lookup table, determine the quotient of multiple tensor elements with the scaling factor parameter, and quantize the quotient into a second floating-point number format to obtain the quantized tensor elements in the second floating-point number format corresponding to the multiple tensor elements respectively.

[0096] For example, the following formula can be used for floating-point quantization:

[0097]

[0098] Where Xq represents the quantized tensor block corresponding to tensor block X, which includes k quantized tensor elements in second floating-point format that correspond one-to-one with the k tensor elements in tensor block X. This represents a division lookup table determined based on the scaling factor parameter. qType() indicates that the parameter is quantized to the target precision of the matrix multiplication operation, i.e., the second floating-point format.

[0099] For example, by referring to the above process, each tensor block in the first tensor A is quantized into floating-point quantization, thereby obtaining the first quantized tensor A. Repeated steps will not be repeated.

[0100] The floating-point quantization process for the second tensor B, the second tensor C, and the second tensor D is similar to that for the first tensor A, and will not be repeated here.

[0101] For example, in some embodiments, the arithmetic logic unit 1011 and the cache 1012 are located on the same storage module 103. For example, the arithmetic logic unit 1011 and the cache 1012 are both located on shared memory, or the arithmetic logic unit 1011 and the cache 1012 are both located on a tensor data storage unit.

[0102] For example, in other embodiments, the arithmetic logic unit and the cache are located on different storage modules.

[0103] For example, a matrix multiplication operation using scaling factors involves combining the scaling factor α of the first tensor A with the scaling factor β of the second tensor B, performing a matrix multiplication of the first tensor A and the second tensor B, and combining the scaling factor γ of the third tensor C to determine the sum of the result of the matrix multiplication operation with the result of the third tensor C, resulting in a fourth tensor D as the result of the matrix multiplication operation.

[0104] For example, in some embodiments, the cache includes a first cache and a second cache, the first cache being configured to cache the scaling factor α of a first tensor A, the scaling factor β of a second tensor B, and the scaling factor σ of a third tensor C, and the second cache being configured to cache the scaling factor γ of a fourth tensor D.

[0105] For example, at least one storage module includes a first storage module and a second storage module, wherein the first storage module is closer to the tensor kernel than the second storage module is closer to the tensor kernel.

[0106] For example, in some embodiments, the first storage module is shared memory and the second storage module is a tensor data storage unit. For example, in other embodiments, the first storage module is a tensor data storage unit and the second storage module is shared memory. The first and second storage modules can be determined according to the hardware architecture, and this disclosure does not impose specific limitations on them.

[0107] For example, the second cache and the first cache can be located on different storage modules, while the second cache and the arithmetic logic unit (ALU) can be located on the same storage module. Alternatively, the ALU and the second cache can be located on the second storage module, while the first cache can be located on the first storage module.

[0108] Figure 5 This is a schematic structural diagram of a computing unit provided in an embodiment of the present disclosure.

[0109] like Figure 5As shown, multiple storage modules are arranged on the data path between memory 120 and tensor core 102, including a first storage module and a second storage module. The first and second storage modules are located inside the computing unit and are exclusively used by the tensor core when it performs tensor operations. The first storage module is closer to the tensor core 102. For details about the first and second storage modules, please refer to the foregoing content; they will not be repeated here.

[0110] like Figure 5 As shown, the second cache and arithmetic logic unit are located on the second storage module, and the first cache is located on the first storage module.

[0111] like Figure 5 As shown, the first tensor A, the second tensor B, and the third tensor C are high-precision tensors, such as BF16, FP16, etc., which can come from memory or other computing units or the previous output of the current computing unit, etc. This disclosure does not impose specific restrictions on them.

[0112] Tensors A, B, and C are input into the arithmetic logic unit (ALU). The ALU determines the scaling factor α for tensor A, β for tensor B, and σ for tensor C. Then, tensors A, B, and C are quantized to obtain quantized tensors A', B', and C'. The specific process for determining the scaling factors and floating-point quantization can be found in the preceding text and will not be repeated here.

[0113] Subsequently, the scaling factor α of the first tensor A, the scaling factor β of the second tensor B, and the scaling factor σ of the third tensor C calculated by the arithmetic logic unit are transmitted and cached in the first cache area via the data path between the second and first storage modules. The arithmetic logic unit processes the first tensor A, the second tensor B, and the third tensor C to obtain the first quantized tensor A', the second quantized tensor B', and the third quantized tensor C' respectively after floating-point quantization, which are then transmitted and cached in the first storage module via the data path between the second and first storage modules. The first quantized tensor A', the second quantized tensor B', and the third quantized tensor C' after floating-point quantization are low-precision tensors, such as FP4 or FP8 format.

[0114] Therefore, the first quantized tensor A', the second quantized tensor B', and the third quantized tensor C' are still stored in shared memory, while the scaling factor α of the first tensor A, the scaling factor β of the second tensor B, and the scaling factor σ of the third tensor C are cached in the first cache area, reducing the capacity of the increased cache area and the resulting hardware area, and improving data transmission efficiency and reducing latency caused by insufficient cache capacity.

[0115] Subsequently, the scaling factor α of the first tensor A, the scaling factor β of the second tensor B, and the scaling factor σ of the third tensor C, as well as the first quantized tensor A', the second quantized tensor B', and the third quantized tensor C', are transferred to the tensor kernel via the data path between the first storage module and the tensor kernel for matrix multiplication operations using the scaling factors, such as performing... or However, this disclosure does not restrict the specific execution process of matrix multiplication operations using scaling factors.

[0116] After performing matrix multiplication with scaling factors, the tensor kernel yields a fourth tensor D. For example... Figure 5 As shown, the tensor kernel transmits the fourth tensor D to the first storage module via the data path between the first storage module and the tensor kernel, and then transmits it to the arithmetic logic unit of the second storage module via the data path between the first storage module and the second storage module.

[0117] The arithmetic logic unit is also configured to determine the scaling factor γ of the fourth tensor D and cache it in the second buffer, and to perform floating-point quantization on the fourth tensor D to obtain the fourth quantized tensor D' in the second floating-point format corresponding to the fourth tensor. The fourth quantized tensor D' is also in a low-precision format, such as FP4 or FP8 format.

[0118] Subsequently, the fourth quantized tensor D' and the scaling factor γ of the fourth tensor are transferred to other relevant modules. For example, they are transferred to other computing units, memory, or re-enter the current computing unit for the next round of operations.

[0119] exist Figure 5 In one embodiment not shown, after determining the fourth tensor D, the tensor kernel directly outputs the fourth tensor through the first storage module and the second storage module without calculating the scaling factor and quantizing the floating point of the fourth tensor D. At this time, the second buffer is still set in the second storage module.

[0120] For example, in some other embodiments, there is no need to quantize the fourth tensor. In this case, the second buffer can be omitted, and the fourth tensor D can be directly transmitted to other relevant modules via the first storage module and the second storage module. This will not be elaborated here.

[0121] For example, in some other embodiments, the matrix multiplication operation may not involve adding to the third tensor C, but only involves calculating A×B. The specific process is similar to the matrix multiplication operation described above, that is, there is no need to perform floating-point quantization and scaling factor calculations for the third tensor C. The specific process will not be described in detail here.

[0122] In at least one embodiment of this disclosure, by setting an additional hardware module, namely a scaling factor processing module, in the computing unit, the transmission efficiency of the scaling factor is improved, the latency caused by insufficient cache capacity is reduced, the computing pressure of the vector computing core is alleviated, the efficiency of other vector operators is improved, and the computing power of the vector computing core is reduced to save hardware area, thereby improving the overall execution efficiency of low-precision matrix multiplication using scaling factors.

[0123] In at least one embodiment of this disclosure, the processor can be a processor of any architecture, such as a graphics processor, a tensor processor, or a data processor. The following description uses a graphics processor as an example to illustrate the illustrative structure of a graphics processor provided in at least one embodiment of this disclosure.

[0124] Figure 6 This is a schematic structural diagram of a graphics processor provided in at least one embodiment of the present disclosure. Figure 6 As shown, the graphics processor 200 includes multiple streaming processor clusters and memory, with each streaming processor cluster comprising multiple computing units. For a more detailed description of the streaming processor clusters, computing units, and memory, please refer to the foregoing. Figure 1 The relevant records will not be repeated here.

[0125] like Figure 6 As shown, each computational unit includes a tensor kernel and a storage module. For example, the storage module may include at least one of shared memory and a tensor data processing unit.

[0126] like Figure 6 As shown, each computing unit's storage module is also equipped with a scaling factor processing module 101. The scaling factor processing module 101 is used to determine and cache the scaling factors of each tensor related to matrix multiplication operations in the processor, and to quantize each tensor from a first floating-point format to a second floating-point format. The scaling factor is used to scale the data representation range when the corresponding tensor is converted from the first floating-point format to the second floating-point format, where the floating-point precision of the first floating-point format is higher than that of the second floating-point format. For a more detailed description of the scaling factor processing module 101, please refer to the previously described documentation of the scaling factor processing module 101; repeated details will not be elaborated upon here.

[0127] For example, the arithmetic logic unit and the cache are located on the same storage module, such as both on shared memory, or both on the tensor data processing unit.

[0128] For example, the arithmetic logic unit and the cache are located on different storage modules.

[0129] For example, the cache includes a first cache and a second cache. The first cache is configured to cache the scaling factors of the first tensor, the second tensor, and the third tensor, and the second cache is configured to cache the scaling factor of the fourth tensor.

[0130] At least one storage module includes a first storage module and a second storage module, wherein the first storage module is closer to the tensor kernel than the second storage module. An arithmetic logic unit and a second cache are disposed on the second storage module, and the first cache is disposed on the first storage module.

[0131] For example, in one embodiment, the first storage module is shared memory, and the second storage module is a tensor data processing unit. In this case, the first cache is located on the shared memory, and the second cache and the arithmetic logic unit are located on the tensor data processing unit. For example, in another embodiment, the first storage module is a tensor data processing unit, and the second storage module is shared memory. In this case, the first cache is located on the tensor data processing unit, and the second cache and the arithmetic logic unit are located on the shared memory.

[0132] For more information about the scaling factor processing module and its interaction with other units in the graphics processor, please refer to the aforementioned description of the scaling factor processing module. Repeated descriptions will not be repeated here.

[0133] In at least one embodiment, by setting an additional hardware module, namely a scaling factor processing module, in the computing unit of the graphics processor, the transmission efficiency of the scaling factor is improved, the latency caused by insufficient cache capacity is reduced, the computing pressure of the vector computing core is alleviated, the efficiency of other vector operators is improved, and the computing power of the vector computing core is reduced to save hardware area, thereby improving the overall execution efficiency of low-precision matrix multiplication using scaling factors.

[0134] In addition, it should be noted that Figure 6 The components of the graphics processor 200 shown are merely exemplary and not limiting. The graphics processor 200 may have other components as needed for the actual application.

[0135] Figure 7 A schematic diagram of an electronic device provided in at least one embodiment of the present disclosure is shown.

[0136] For example, such as Figure 7 As shown, the electronic device 300 includes a processor 200. For example, the processor 200 may employ... Figure 6 The structure shown is implemented as described. For example, the electronic device 300 can be any electronic device including those with computing capabilities, such as a laptop, tablet, desktop computer, web server, etc., and the embodiments disclosed herein are not limited thereto.

[0137] For example, the electronic device may also include other forms of processing units with data processing capabilities and / or instruction execution capabilities, such as a central processing unit (CPU) or a digital signal processor (DSP), as well as storage units. The electronic device may also have an operating system and application programming interfaces (e.g., OpenGL (Open Graphics Library), Metal, etc.). For example, the electronic device may also include output components such as display components, such as liquid crystal displays (LCDs), organic light-emitting diode (OLED) displays, quantum dot light-emitting diode (QLED) displays, etc. The embodiments disclosed herein are not limited in this regard.

[0138] It should be noted that, for clarity and brevity, this disclosure does not show all the constituent units of the electronic device 300. To achieve the necessary functions of the electronic device 300, those skilled in the art can provide and set other constituent units (not shown) according to specific needs, and this disclosure does not limit this.

[0139] The following is for reference. Figure 8 It illustrates a specific structural diagram of an electronic device (e.g., a terminal device or a server) 300 suitable for implementing a processor including embodiments of the present disclosure.

[0140] The terminal devices in this disclosure can include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. For example, the electronic device can be in the form of a server, used for various application scenarios such as deep learning and artificial intelligence, scientific computing, graphics rendering and video editing, virtual reality and game development, and cloud services. For example, the electronic device can be a dedicated server such as a data center or cloud computing center that is deployed with tasks such as deep learning training, large-scale data analysis, and high-performance computing.

[0141] Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0142] like Figure 8As shown, the electronic device 300 may include a processing unit 301, such as the aforementioned processor 200, which can execute various appropriate actions and processes according to non-transitory computer-readable instructions stored in memory to achieve various functions. The processing unit 301 may also include devices with instruction optimization capabilities and / or program execution capabilities, such as a central processing unit (CPU) or a tensor processor (TPU). The CPU can be based on x86, ARM, or RISC-V architectures. The GPU can be directly integrated into the SOC, directly integrated onto the motherboard, or built into the northbridge chip of the motherboard.

[0143] like Figure 8 As shown, for example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 303 and / or cache memory, etc., for example, computer-readable instructions may be loaded from storage device 308 into RAM 303 to execute computer-readable instructions. Non-volatile memory may include, for example, read-only memory (ROM) 302, hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. Various applications and various data, such as style images, and various data used and / or generated by the applications, may also be stored in the computer-readable storage medium.

[0144] For example, the processing device 301, the read-only memory (ROM) 302, and the random access memory (RAM) 303 are interconnected via a bus 304. The input / output (I / O) interface 305 is also connected to the bus 304.

[0145] Typically, the following devices can be connected to the input / output (I / O) interface 305: input devices 306 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 308 including, for example, magnetic tape, hard disk, flash memory, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 8 An electronic device 300 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 300 may alternatively implement or have more or fewer devices. For example, the processing device 301 may control other components in the electronic device 300 to perform desired functions.

[0146] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0147] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0148] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0149] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire at least two Internet Protocol (IP) addresses; send a node evaluation request including the at least two IP addresses to a node evaluation device, wherein the node evaluation device selects an IP address from the at least two IP addresses and returns it; and receive the IP address returned by the node evaluation device; wherein the acquired IP address indicates an edge node in a content delivery network.

[0150] Alternatively, the aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: receive a node evaluation request including at least two Internet Protocol (IP) addresses; select an IP address from the at least two IP addresses; and return the selected IP address; wherein the received IP address indicates an edge node in the content delivery network.

[0151] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0152] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0153] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0154] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0155] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0156] The following points should be noted regarding this disclosure:

[0157] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0158] (2) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0159] The above description is only a specific embodiment of this disclosure, but the protection scope of this disclosure is not limited thereto. The protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A processor, characterized in that, The processor includes a computing unit and memory. The computing unit includes a tensor kernel configured to perform matrix multiplication operations using a scaling factor. The computation unit further includes a scaling factor processing module, configured to determine and cache the scaling factors of each tensor related to the matrix multiplication operation, and to quantize each tensor from a first floating-point format to a second floating-point format. The scaling factor is used to scale the data representation range when the corresponding tensor is converted from the first floating-point format to the second floating-point format, wherein the floating-point precision of the first floating-point format is higher than that of the second floating-point format. The computing unit further includes at least one storage module disposed on the data path between the tensor kernel and the memory, wherein the at least one storage module is exclusively used by the tensor kernel when the tensor kernel performs tensor-related operations. The scaling factor processing module is located on the at least one storage module.

2. The processor according to claim 1, characterized in that, The scaling factor processing module includes an arithmetic logic unit and a cache. The arithmetic logic unit is configured to receive each tensor in the first floating-point format, perform a scaling factor calculation operation on each tensor, and perform floating-point quantization on each tensor to obtain a quantized tensor corresponding to each tensor after floating-point quantization, wherein the quantized tensor is in the second floating-point format. The cache is configured to cache the scaling factors of each tensor.

3. The processor according to claim 2, characterized in that, For any tensor in the first floating-point format received by the scaling factor processing module, the tensor is divided into multiple tensor blocks, each tensor block includes multiple tensor elements, each tensor block corresponds to a scaling factor parameter, and the multiple tensor elements share the same scaling factor parameter. When the arithmetic logic unit performs the operation of receiving the tensors and calculating the scaling factors of the tensors, it includes the following operations: For each tensor block: Determine the sum of the maximum absolute value among the plurality of tensor elements and a preset floating-point number; Determine the product of the summation result and a preset constant, wherein the preset constant is the quotient of 1 and the maximum value of the precision expression range of the second floating-point number format; The product result is converted to the precision specified by the scaling factor of any tensor to obtain the scaling factor parameter corresponding to the tensor block.

4. The processor according to claim 3, characterized in that, When the arithmetic logic unit performs floating-point quantization on each tensor to obtain the quantized tensor corresponding to each tensor after floating-point quantization, it includes performing the following operations: For the tensor block: Based on the scaling factor parameter, determine the division lookup table; Based on the division lookup table, the quotients of the plurality of tensor elements and the scaling factor parameter are determined respectively, and the quotients are quantized into the second floating-point number format to obtain the quantized tensor elements corresponding to the plurality of tensor elements respectively.

5. The processor according to claim 2, characterized in that, The arithmetic logic unit and the cache are located on the same storage module.

6. The processor according to claim 2, characterized in that, The matrix multiplication operation using the scaling factor includes combining the scaling factors of the first tensor and the second tensor to perform matrix multiplication of the first tensor and the second tensor, and combining the scaling factor of the third tensor to determine the sum of the result of the matrix multiplication operation and the result of the third tensor, thus obtaining a fourth tensor as the result of the matrix multiplication operation. The cache includes a first cache and a second cache. The first cache is configured to cache the scaling factors of the first tensor, the second tensor, and the third tensor. The second cache is configured to cache the scaling factor of the fourth tensor. The at least one storage module includes a first storage module and a second storage module, wherein the first storage module is closer to the tensor kernel than the second storage module is closer to the tensor kernel. The arithmetic logic unit and the second cache are disposed on the second storage module, and the first cache is disposed on the first storage module.

7. The processor according to claim 6, characterized in that, The scaling factors of the first tensor, the second tensor, and the third tensor calculated by the arithmetic logic unit are transmitted and cached in the first cache area via the data path between the second storage module and the first storage module. The arithmetic logic unit processes the first tensor, the second tensor, and the third tensor to obtain the first quantized tensor, the second quantized tensor, and the third quantized tensor respectively after floating-point quantization. These are then transmitted and cached in the first storage module via the data path between the second storage module and the first storage module. The scaling factors of the first tensor, the second tensor, and the third tensor, as well as the first quantized tensor, the second quantized tensor, and the third quantized tensor, are transmitted to the tensor kernel via the data path between the first storage module and the tensor kernel to perform the matrix multiplication operation using the scaling factors.

8. The processor according to claim 6, characterized in that, The fourth tensor is transmitted to the first storage module via the data path between the first storage module and the tensor kernel, and then transmitted to the arithmetic logic unit of the second storage module via the data path between the first storage module and the second storage module. The arithmetic logic unit is further configured to determine the scaling factor of the fourth tensor and cache it in the second buffer, and perform floating-point quantization on the fourth tensor to obtain the fourth quantized tensor in the second floating-point format corresponding to the fourth tensor after floating-point quantization.

9. The processor according to claim 2, characterized in that, The matrix multiplication operation using the scaling factor includes combining the scaling factors of the first tensor and the second tensor to perform matrix multiplication of the first tensor and the second tensor, and combining the scaling factor of the third tensor to determine the sum of the result of the matrix multiplication operation and the result of the third tensor, thus obtaining a fourth tensor as the result of the matrix multiplication operation. The fourth tensor is directly transmitted to other computing units or the memory via the transmission path between the tensor core and the memory.

10. The processor according to any one of claims 2-9, characterized in that, The at least one storage module includes at least one tensor data storage unit in the computing unit and at least one shared memory. The tensor data storage unit is dedicated to the preprocessing of data related to the tensor kernel. The shared memory is a storage area shared by all threads in the computing unit.

11. An electronic device, characterized in that, Includes the processor as described in any one of claims 1-10.

Images