Processors, electronic devices

Abstract

A processor and electronic device applied to the field of data processing. The processor comprises a scaling factor processing module, the scaling factor processing module comprises a smoothing preprocessing unit and a processing unit, the smoothing preprocessing unit is configured to receive a first tensor, a second tensor and a smoothing control parameter, perform a smoothing processing operation on the first tensor and the second tensor through a plurality of arithmetic logic units and special function units in the processor, and obtain a third tensor and a fourth tensor; the processing unit is configured to determine a scaling factor of the third tensor and the fourth tensor, and perform a matrix multiplication accumulation operation using the scaling factor based on the scaling factor of the third tensor and the fourth tensor to obtain a matrix multiplication operation result. The processor provides an efficient smoothing preprocessing hardware, without performing the processing in a dedicated hardware device such as a vector core, realizing hardware-level efficient execution of smoothing preprocessing, effectively reducing hardware resource scheduling overhead and calculation delay, and improving hardware execution efficiency.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of data processing, and more specifically to a processor and an electronic device. Background Technology

[0002] Floating-point quantization refers to converting high-precision floating-point formats (such as FP16, BF16, FP32, etc.) into low-precision floating-point formats (such as FP4, FP8, etc.). Low-precision floating-point formats can significantly improve computational efficiency, especially for General Matrix Multiplication (GEMM) operations required by Artificial Intelligence (AI) models. However, low-precision matrix multiplication introduces precision loss, thus reducing the prediction and generation performance of AI models. Therefore, in the low-precision computation process of AI models, a scaling factor is typically calculated from the current data during floating-point quantization to scale the data's representation range, thereby reducing the precision loss in the computational results.

[0003] In the distribution of activation functions in large models, the numerical differences in the matrix mainly come from the accumulated dimension. Current matrix multiplication operations using scaling factors can greatly affect the overall scaling of data blocks, thus impacting quantization accuracy. Summary of the Invention

[0004] This invention application provides at least one embodiment of a processor, including a scaling factor processing module for performing a matrix multiplication and accumulation operation using a scaling factor on a first tensor and a second tensor. The scaling factor processing module includes a smoothing preprocessing unit and a processing unit. The smoothing preprocessing unit is configured to receive the first tensor, the second tensor, and smoothing control parameters, and perform smoothing operations on the first tensor and the second tensor using multiple arithmetic logic units and special function units in the processor to obtain a smoothed third tensor corresponding to the first tensor and a smoothed fourth tensor corresponding to the second tensor. The smoothing operation includes, in response to the existence of at least one outlier in a first dimension of the first tensor, scaling the values ​​of all elements of the first tensor to a uniform range according to the smoothing control parameters, wherein the smoothing control parameters control the smoothing intensity on the first tensor, and the first dimension is the accumulation dimension when performing the matrix multiplication and accumulation operation. The processing unit is configured to determine the scaling factors of the third tensor and the fourth tensor, and perform the matrix multiplication and accumulation operation using the scaling factors based on the scaling factors of the third tensor and the fourth tensor to obtain the matrix multiplication result.

[0005] For example, in a processor provided in at least one embodiment of this application, when the smoothing preprocessing unit performs the smoothing operation, it includes performing the following operations: determining a smoothing vector for scaling the first tensor and the second tensor in a granularity of 1 in the first dimension, and using the smoothing vector to simultaneously scale the first tensor and the second tensor to obtain the third tensor and the fourth tensor, wherein the smoothing operation does not change the matrix multiplication result of the first tensor and the second tensor, and the length of the smoothing vector is the same as the size of the first tensor and the second tensor in the first dimension.

[0006] For example, in a processor provided in at least one embodiment of this application, the smoothing preprocessing unit includes a smoothing vector determination unit and a first multiplication unit. The smoothing vector determination unit is configured to determine the smoothing vector and the smoothing inverse vector, wherein each element in the smoothing inverse vector is the inverse of each element in the smoothing vector. The first multiplication unit is configured to perform a multiplication operation between the smoothing inverse vector and the first tensor to obtain the third tensor, and to perform a multiplication operation between the smoothing vector and the second tensor to obtain the fourth tensor.

[0007] For example, in a processor provided in at least one embodiment of this invention, the smoothing vector determination unit includes a first unit, a second unit, and a third unit. The first unit is configured to determine the maximum column value corresponding to each column in the first tensor, obtain din maximum column values ​​and form a first vector, and determine the maximum row value corresponding to each row in the second tensor, obtain din maximum row values ​​and form a second vector, wherein din represents the size of the first tensor and the second tensor in the first dimension, the maximum column value is the maximum absolute value among all elements in the corresponding column, and the maximum row value is the maximum absolute value among all elements in the corresponding row; the second unit is configured to determine a first power result and a second power result, wherein the first power result is the result obtained by raising the first vector to the power of α, and the second power result is the result obtained by raising the second vector to the power of 1-α, where α represents the smoothing control parameter, α is less than 1 and greater than 0; the third unit is configured to determine the ratio of the first power result and the second power result as the smoothing vector, and determine the ratio of the second power result and the first power result as the smoothing reciprocal vector.

[0008] For example, in a processor provided in at least one embodiment of this application, the second unit includes a logarithmic unit, a second multiplication unit, and an exponentiation unit. The logarithmic unit is configured to perform a base-2 logarithmic operation on each element of the first vector and the second vector to obtain a first intermediate vector and a second intermediate vector. The second multiplication unit is configured to perform a multiplication operation between the first intermediate vector and α to obtain a third intermediate vector, and to perform a multiplication operation between the second intermediate vector and 1-α to obtain a fourth intermediate vector. The exponentiation unit is configured to perform a base-2 exponentiation operation on each element of the third intermediate vector to obtain a first power result, and to perform a base-2 exponentiation operation on each element of the fourth intermediate vector to obtain a second power result.

[0009] For example, in a processor provided in at least one embodiment of this application, the third unit includes a third multiplication unit, a first reciprocal unit, and a second reciprocal unit. The first reciprocal unit is configured to determine the reciprocal of the first power result to obtain a first reciprocal result. The third multiplication unit is configured to perform a multiplication operation between the second power result and the first reciprocal result, and the obtained multiplication result is used as the smooth reciprocal vector. The second reciprocal unit is configured to determine the reciprocal of the multiplication result obtained by the third multiplication unit to obtain the smooth vector.

[0010] For example, in a processor provided in at least one embodiment of this application, the processing unit includes a quantization unit and a matrix multiplication unit. The quantization unit is configured to determine the scaling factors of the third tensor and the fourth tensor, and to quantize the third tensor and the fourth tensor from a first floating-point format to a second floating-point format based on the scaling factors of the third tensor and the fourth tensor. The scaling factors are used to scale the data representation range when the corresponding tensor is converted from the first floating-point format to the second floating-point format, wherein the floating-point precision of the first floating-point format is higher than that of the second floating-point format. The matrix multiplication unit is configured to perform the matrix multiplication accumulation operation using the scaling factors of the third tensor and the fourth tensor, and the quantized third tensor and the fourth tensor, to obtain the matrix multiplication operation result.

[0011] For example, in a processor provided in at least one embodiment of this application, the processor is configured to perform inference or training of a natural language processing model, the matrix multiplication and accumulation operation is an operation performed in the linear layer of the natural language processing model, the first tensor is an activation tensor, the second tensor is a weight tensor, the first tensor is a numerical semantic feature obtained by processing the text data input to the natural language processing model, and the values ​​of each element in the second tensor are uniformly distributed.

[0012] For example, in the processor provided in at least one embodiment of this application, the processor is a parallel processor, which includes a plurality of computing units for performing computing tasks in parallel, and the scaling factor processing module is provided on each computing unit.

[0013] This application provides at least one embodiment of an electronic device, including a processor as described in at least one embodiment of this application.

[0014] The processor provided in this invention application introduces a new hardware unit, namely a smoothing preprocessing unit. Before performing matrix multiplication using a scaling factor, this smoothing preprocessing unit smoothly scales the element values ​​of the first tensor, which has outliers in the first dimension, to a uniform range. This avoids individual outliers affecting the overall scaling of the entire data block when determining the scaling factor and using it for tensor scaling, thereby improving quantization accuracy.

[0015] Furthermore, the smoothing preprocessing unit in this invention application is implemented by multiple arithmetic logic units (ALUs) and special function units (SFUs). By setting these ALUs and SFUs in the processor, an efficient smoothing preprocessing method can be provided, eliminating the need to perform the processing in dedicated hardware devices such as vector cores. This frees up the hardware computing resources of the vector core, avoids resource competition between different computations on the vector core, and significantly improves the computing power utilization and main task execution efficiency of the vector core. Relying on the customized design of the hardware unit, efficient hardware-level execution of smoothing preprocessing is achieved, effectively reducing hardware resource scheduling overhead and computational latency, and improving the overall hardware execution efficiency of the processor. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0017] Figure 1 This is a schematic diagram of the architecture of a general-purpose graphics processing unit (GPGPU).

[0018] Figure 2 This is a schematic diagram of an activation tensor A and a weight tensor W.

[0019] Figure 3 This is a schematic diagram showing the numerical distribution of the activation function A and weight function W in a certain layer of the large language model LLaMA.

[0020] Figure 4 A schematic structural diagram of a processor provided for at least one embodiment of this disclosure;

[0021] Figure 5 The numerical distributions of the third and fourth tensors obtained by smoothing operations are provided in at least one embodiment of this disclosure;

[0022] Figure 6 A schematic structural diagram of a smoothing preprocessing unit provided in at least one embodiment of this disclosure;

[0023] Figure 7 A schematic structural diagram of the second unit provided for at least one embodiment of this disclosure;

[0024] Figure 8 A schematic structural diagram of the third unit provided in at least one embodiment of this disclosure;

[0025] Figure 9 A schematic structural diagram of a processor provided for at least one embodiment of this disclosure;

[0026] Figure 10 A schematic block diagram of an electronic device provided in one embodiment of this disclosure;

[0027] Figure 11 This is a schematic diagram of the specific structure of another electronic device provided in at least one embodiment of the present disclosure. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0029] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described object changes. To keep the following description of the embodiments of this disclosure clear and concise, detailed descriptions of some known functions and components are omitted.

[0030] Floating-point numbers (FP) are primarily used to represent decimals and typically consist of three parts: a sign bit, an exponent, and a mantissa. The exponent part can also be called the exponent code. For example, a floating-point number V can usually be represented in the following form:

[0031]

[0032] In this context, the sign bit s can be 1 bit, determining whether the floating-point number V is negative or positive; M represents the mantissa, which can include multiple bits and is a binary fractional form, defining the precision of the floating-point number; E represents the exponent (also called the exponent value), used to weight the floating-point number, reflecting the position of the decimal point in the floating-point number V, and defining the range of values ​​for the floating-point number.

[0033] Traditional floating-point numbers typically include three formats: half-precision floating-point (FP16), single-precision floating-point (FP32), and double-precision floating-point (FP64), with different numbers of bits in their exponent and mantissa parts.

[0034] AI accelerators are widely used for training deep learning models. For convolution operations, common in deep learning models, both hardware and software designs have been specifically optimized to accelerate computation. For example, various floating-point data formats have been developed and optimized for fields such as artificial intelligence and deep learning, including BF16 (brain floating point 16, 16-bit width), BF24 (brain floating point 24, 24-bit width), and TF32 (Tensor Float 32, 19-bit width). These data formats can significantly reduce computational processing, especially the computational resources and power consumption required for matrix multiplication or convolution multiplication operations. In addition, the processor also supports some common floating-point types, such as half-precision floating-point numbers (FP16, 16-bit width) or single-precision floating-point numbers (FP32, 32-bit width).

[0035] Figure 1 This is a schematic diagram of a general-purpose graphics processing unit (GPGPU).

[0036] like Figure 1As shown, a general-purpose graphics processing unit (GPU) is actually an array of programmable multiprocessors, such as a streaming processor cluster (SPC). In a general-purpose GPU, one streaming processor cluster handles one computational task, or multiple streaming processor clusters handle one computational task. Multiple streaming processor clusters share data through a global cache or global memory.

[0037] like Figure 1 As shown, a streaming processor cluster includes multiple computing units.

[0038] A computing unit comprises multiple cores (also called computing kernels or computing cores). Each computing core includes an arithmetic logic unit (ALU), a floating-point unit, etc., and is used to execute specific computational tasks. In addition, the computing unit also includes registers (e.g., ... Figure 1 The register file and shared memory in a computing unit are used to store source and destination data related to computing tasks in a hierarchical manner. The shared memory in a computing unit is used to share data between the cores of that computing unit.

[0039] like Figure 1 As shown, each computational unit also provides a tensor core for performing tensor-related computations, such as matrix multiplication operations using the GEMM operator. Tensors are a crucial data structure in deep learning; they are high-dimensional generalizations of scalars, vectors, and matrices. Tensor operations are commonly used in the training and inference of deep learning models, and tensor cores can accelerate matrix multiplication. Tensor cores across multiple computational units can be uniformly scheduled and controlled.

[0040] For example, each computing unit can also provide a Vector Core Engine (not shown in the figure) for performing vector-related computations, such as vector-related arithmetic and logical operations, such as accumulation, reduction, and regular addition, subtraction, multiplication, and division.

[0041] In parallel computing, computational tasks are typically executed by multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block distribution module. Figure 1(Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0042] In each computing unit, the thread beam scheduling / distribution module ( Figure 1 (Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple computing cores within the computing unit can run thread bundles. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can be executed concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory execution instructions are issued to shared memory within the computing unit or further issued to intermediate-level caches (such as vector operation unit caches), global caches, or global memory for read and write operations, etc.

[0043] Low-precision matrix multiplication is increasingly being used in the training and inference of large AI models due to its significant performance gains with acceptable precision loss. In GPUs (Graphics Processing Units) or GPGPUs, matrix multiplication is typically performed in hardware by tensor kernels. Low-precision tensor kernels are several times more computationally powerful than high-precision tensor kernels, resulting in higher computational efficiency. Furthermore, the data volume of a low-precision tensor is also only a few times that of a high-precision tensor, leading to higher data transfer efficiency. Therefore, the end-to-end efficiency improvement brought by low-precision tensor computation is almost several times greater. For example, the computational power of an FP4 tensor kernel can be 2 to 8 times that of FP8, 4 to 16 times that of FP16 / BF16, or even higher. Moreover, the data volume is half that of FP8 and one-quarter that of FP16 / BF16.

[0044] To reduce the loss of precision, low-precision matrix multiplication introduces a scaling factor to maximize the numerical expressive power of low-precision tensors. For example, low-precision matrix multiplication includes matrix multiplication for low-precision tensors (such as FP4, FP8, etc.).

[0045] For example, you can refer to Formula 1 below for floating-point quantization:

[0046] (Formula 1)

[0047] Where Xq represents the quantized tensor block corresponding to tensor block X, which includes k quantized tensor elements in second floating-point format that correspond one-to-one with the k tensor elements in tensor block X. This represents a division lookup table determined based on the scaling factor parameter. qType() indicates that the parameter is quantized to the target precision of the matrix multiplication operation, such as FP4.

[0048] The aforementioned quantization methods do indeed mitigate the negative impact of low precision on model performance to a large extent. However, research on the activation function distribution of current large models reveals that the numerical differences in the activation tensors, particularly in linear layers, are primarily differentiated along the first dimension (cumulative dimension), rather than the second dimension. Here, the matrix multiplication calculation in a linear layer can be represented as A×W, where A represents the activation tensor, specifically L×din, where din is the size of the first dimension and L is the size of the second dimension; W represents the weight tensor, specifically din×dout, where dout is the size of the third dimension, and L, din, and dout are all positive integers. In a specific scenario, L can represent the number of input tokens, and din can represent the length of a single token, such as 2048 or 4096.

[0049] Figure 2 This is a schematic diagram of an activation tensor A and a weight tensor W.

[0050] like Figure 2 As shown, the activation tensor A is divided into T first tensor blocks in the column direction, for example... Figure 2 The first block A0, first block A1, ..., first block AT-1 in the array have a shape and size of L×k. The weight tensor W is divided into T second blocks in the row direction, for example... Figure 2 The second block B0, the second block B1, ..., the second block BT-1 in the array have a shape and size of k×dout for each second block.

[0051] When scaling, the scaling factor is determined on a tensor block basis and the scaling is performed using that scaling factor.

[0052] For example, tensor block X includes the 1st to kth tensor elements in a certain row of tensor A, such as X=[X0,X1,...,X]. k-1 ],X0,X1,...,X k-1 These represent the k tensor elements respectively.

[0053] First, refer to the following formula to determine the sum:

[0054] X_max=max(abs(X))+t

[0055] When X_max equals 0, X_max() += t

[0056] X_max represents the summation result, abs() represents the absolute value function, max() represents the maximum value function, and t is a very small floating-point number (e.g., t=1e). -12 This is used to prevent scenarios involving division by zero.

[0057] Then, refer to Formula 2 below to determine the scaling factor parameter S corresponding to the tensor block X:

[0058] (Formula 2)

[0059] Where q_max is known in advance, 1 / q_max is a pre-calculated preset constant, q_max represents the maximum value of the precision range of the second floating-point number format. For example, if the second floating-point number format is FP4, then q_max can be 6; sType indicates that X_max×(1 / q_max) is converted into the expected precision specified by the scaling factor. For example, if the precision of the scaling factor is specified as FP8.

[0060] For example, by referring to the above process, the scaling factor parameters corresponding to each tensor block are obtained, thereby obtaining the scaling factor of the activated tensor A. Repeated steps will not be repeated.

[0061] This presents a significant challenge when the numerical differences in the activation function matrix of large models, such as linear layers, lie primarily in the first dimension rather than the second. This is because matrix multiplication involves accumulation in the first dimension. As mentioned earlier, even with fine-grained quantization using scaling factors, the hardware tensor kernel needs to perform a large number of multiplications and accumulations (dot products) simultaneously to ensure efficient matrix multiplication. Therefore, quantization with a granularity of 1 in the accumulation dimension is not feasible. This means that if a tensor block (e.g., k=128) contains even a few outliers, their presence can drastically affect the overall scaling of that data block, thus impacting quantization accuracy.

[0062] Specifically, during low-precision training, especially in extremely low-precision formats like FP4, these outliers can cause significant precision loss during low-precision quantization. Typically, the activation tensor A contains a large range of outliers in its first dimension (accumulation dimension). According to the quantization method described above, when quantizing to the target precision (e.g., FP4), these large outliers cause many ordinary values ​​to be scaled too small, making them unrepresentable by FP4 and thus scaled to 0. This results in a significant loss of information and negatively impacts subsequent low-precision matrix multiplication outputs.

[0063] Figure 3 This is a schematic diagram illustrating the numerical distribution of the activation tensor A and weight tensor W in a certain layer of the Large Language Model (LLaMA).

[0064] exist Figure 3 In the image, normal values ​​are represented by white, while abnormal values ​​are represented by gray, dark gray, or black. According to... Figure 3 It can be clearly seen that the overall distribution of the weight tensor W is relatively uniform, with values ​​almost all close to 0. For the activation tensor A, two columns show significant outliers (manifested as...). Figure 3 The data in the two columns containing multiple dark points is relatively evenly distributed, while the data values ​​in the other parts are mostly close to 0. When performing matrix multiplication using scaling factors, such as the scaling process described above, and processing is done in the form of tensor blocks, the other values ​​in the tensor blocks containing the extreme values ​​in these two columns may be directly scaled to 0, resulting in a large amount of information loss and directly and negatively impacting the subsequent low-precision matrix multiplication output.

[0065] This disclosure provides at least one embodiment of a processor, a data processing method, an electronic device, and a non-transitory computer-readable storage medium. The processor includes a scaling factor processing module for performing a matrix multiplication-accumulation operation using a scaling factor on a first tensor and a second tensor, wherein the first tensor has at least one outlier in a first dimension, the first dimension being the accumulation dimension when performing the matrix multiplication-accumulation operation. The scaling factor processing module includes a smoothing preprocessing unit and a processing unit. The smoothing preprocessing unit is configured to receive a first tensor, a second tensor, and smoothing control parameters. Through multiple arithmetic logic units and special function units in the processor, it performs smoothing operations on the first and second tensors to obtain a smoothed third tensor corresponding to the first tensor and a smoothed fourth tensor corresponding to the second tensor. The multiple hardware units include arithmetic logic units and special function units. The smoothing operation includes scaling the values ​​of all elements of the first tensor to a uniform range according to the smoothing control parameters. The smoothing control parameters are used to control the smoothing intensity of the first tensor. The processing unit is configured to determine the scaling factors of the third and fourth tensors and, based on the scaling factors of the third and fourth tensors, perform a matrix multiplication accumulation operation using the scaling factors to obtain the matrix multiplication result.

[0066] In the processor provided in at least one embodiment of this disclosure, a new hardware unit, namely a smoothing preprocessing unit, is introduced. Before performing matrix multiplication using a scaling factor, the smoothing preprocessing unit smoothly scales the element values ​​of the first tensor, which has outliers in the first dimension, to a uniform range. This avoids individual outliers affecting the overall scaling of the entire data block when determining the scaling factor and using the scaling factor for tensor scaling, thereby improving quantization accuracy.

[0067] Furthermore, the smoothing preprocessing unit in this disclosure is implemented by multiple arithmetic logic units (ALUs) and special function units (SFUs). By setting these ALUs and SFUs in the processor, an efficient smoothing preprocessing method can be provided, eliminating the need to perform the processing in dedicated hardware devices such as vector cores. This frees up the hardware computing resources of the vector core, avoids resource competition between different computations on the vector core, and significantly improves the computing power utilization and main task execution efficiency of the vector core. Relying on the customized design of the hardware unit, efficient hardware-level execution of smoothing preprocessing is achieved, effectively reducing hardware resource scheduling overhead and computational latency, and improving the overall hardware execution efficiency of the processor.

[0068] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, but this disclosure is not limited to these specific embodiments.

[0069] Figure 4 A schematic structural diagram of a processor provided for at least one embodiment of this disclosure.

[0070] like Figure 4 As shown, the processor 10 includes a scaling factor processing module 100, which is used to perform matrix multiplication and accumulation operations using scaling factors on the first and second tensors. Of course, the scaling factor processing module can also be used to perform matrix multiplication and accumulation operations using scaling factors, including offset terms; this disclosure does not specifically limit this. For details regarding matrix multiplication and accumulation operations using scaling factors, please refer to the following text; they will not be repeated here.

[0071] For example, the first tensor may have at least one outlier in the first dimension, which is the accumulation dimension when performing matrix multiplication and accumulation operations.

[0072] For example, the matrix multiplication and accumulation operation of the first tensor and the second tensor can be represented as: A×W=(L×din)×(din×dout), where the first dimension is the column dimension of the first tensor and also the row dimension of the second tensor. The size of the first tensor in the first dimension is din, and the size of the second tensor in the first dimension is also din.

[0073] For a first tensor containing at least one outlier in its first dimension, this could mean at least one outlier in each row. For example, the first tensor might have at least one outlier column containing outliers that are higher, lower, or significantly higher or lower than the data in other columns. See, for example, [reference needed]. Figure 3 Example, Figure 3 The two black columns of data in the active tensor A can be called two outlier columns.

[0074] like Figure 4As shown, the scaling factor processing module 100 includes a smoothing preprocessing unit 101 and a processing unit 102.

[0075] The smoothing preprocessing unit 101 is configured to receive a first tensor, a second tensor, and smoothing control parameters, and perform smoothing operations on the first tensor and the second tensor through multiple arithmetic logic units and special function units in the processor to obtain a smoothed third tensor corresponding to the first tensor and a smoothed fourth tensor corresponding to the second tensor.

[0076] The Arithmetic Logic Unit (ALU) is the basic computing core in the processor, responsible for high-frequency simple operations. The Special Function Unit (SFU) is a dedicated acceleration unit in the processor, responsible for complex special function operations. The two do not compete for resources and can be scheduled in parallel.

[0077] The arithmetic logic unit is the dedicated execution carrier for basic arithmetic / logic operations, covering arithmetic operations such as multiplication and addition, logical operations such as comparison and selection, and simple vector operations, such as basic vector addition and subtraction, dot product, and element-wise multiplication.

[0078] The special function unit is a dedicated hardware computing unit independent of the ALU. It is designed specifically for transcendental functions or complex special functions. It replaces software iteration with hardware-based algorithms, solving the problems of low efficiency and high latency when the ALU processes complex functions. It covers operations such as reciprocals, logarithms, and exponentiations.

[0079] In this disclosure, a smoothing preprocessing unit can be additionally provided in the processor. In this case, corresponding arithmetic logic units and special function units can be additionally provided to form a smoothing preprocessing unit to perform smoothing operations; or, some arithmetic logic units and special function units originally provided in the processor can be reused or partially reused to form a smoothing preprocessing unit and perform smoothing operations.

[0080] The smoothing operation includes scaling the values ​​of all elements of the first tensor to a uniform range in response to the presence of at least one outlier in the first dimension, according to smoothing control parameters. The smoothing control parameters control the degree of smoothing applied to the first tensor. By smoothing outliers in the first tensor, outlier smoothing shifts the difficulty of quantization from the activation tensor to the model weight tensor. However, the numerical distribution of the model weight tensor is usually relatively uniform and less susceptible to the influence of outliers.

[0081] After obtaining the smoothed third and fourth tensors, the processing unit 102 is configured to determine the scaling factors of the third and fourth tensors, and perform a matrix multiplication operation using the scaling factors based on the scaling factors of the third and fourth tensors to obtain the matrix multiplication operation result.

[0082] In other words, this disclosure performs matrix multiplication on the smoothed third and fourth tensors using a scaling factor. Since the outliers in the first tensor have been smoothed and the values ​​of all elements in the first tensor have been scaled to a uniform range, the loss of information during the scaling process caused by outliers can be avoided when scaling the third and fourth tensors using the scaling factor in the subsequent process. This improves the quantization accuracy and further ensures the accuracy of the matrix multiplication accumulation result of the subsequent low-precision matrix multiplication.

[0083] For example, when the smoothing preprocessing unit performs a smoothing operation, it includes performing the following operations: determining a smoothing vector for scaling the first tensor and the second tensor in a first dimension with a granularity of 1, and using the smoothing vector to simultaneously scale the first tensor and the second tensor to obtain a third tensor and a fourth tensor.

[0084] In this disclosure, the smoothing operation does not change the matrix multiplication result of the first tensor and the second tensor. The first dimension is the accumulated dimension when performing the matrix multiplication operation, and the length of the smoothing vector is the same as the size of the first tensor and the second tensor in the first dimension.

[0085] The smoothing operation used in this disclosure is based on the core idea of ​​finding a smoothing vector v to simultaneously scale the first tensor and the second tensor. The smoothing vector can achieve scaling with a granularity of 1 in the first dimension of the first tensor and the second tensor without changing the matrix multiplication result of the first tensor and the second tensor (which means that "inverse quantization" is not required).

[0086] For example, this disclosure uses the following formula 3 for smoothing operations:

[0087] (Formula 3)

[0088] Where Y represents the result of the matrix multiplication operation, A represents the first tensor, W represents the second tensor, v represents the smoothing vector, Asmooth represents the third tensor, and Wsmooth represents the fourth tensor.

[0089] A smooth vector v can be understood as a special "scaling factor" that scales the vector in the first dimension with a granularity of 1, and its definition is as follows:

[0090]

[0091] in, This represents the first vector composed of the maximum values ​​of each column in the first tensor. The maximum value of a column is the maximum absolute value of all elements in that column. α represents the smoothing control parameter. Let the second vector be a sequence of the maximum values ​​of each row in the second tensor. The maximum value of a row is the largest of the absolute values ​​of all elements in that row. The length of the smooth vector is the same as the dimensions of the first and second tensors in the first dimension, which is din.

[0092] Here, the smoothing control parameter 𝛼 controls the smoothing strength of the first tensor X. Its default value is 0.5, but it can be different for each linear layer, depending on the post-training calibration (PTQ). The smoothing operation isn't free; it ultimately shifts the quantization difficulty from the first tensor to the second tensor. However, analysis shows that the numerical distribution of the second tensor is usually more uniform and less susceptible to outliers. For example... Figure 3 As shown, the numerical distribution of the weighting function is relatively even, and is basically close to 0.

[0093] Figure 5 The numerical distribution of the third and fourth tensors obtained by smoothing operations is provided for at least one embodiment of this disclosure.

[0094] like Figure 5 As shown, after the smoothing operation, the outlier amplitude of the third tensor in the first dimension is greatly reduced, and the overall numerical distribution of the third tensor is smoother. The numerical distribution of the fourth tensor remains uniform and close to 0.

[0095] The smoothing control parameter can be used to control the smoothing intensity of the first tensor. The smoothing control parameter can be any value between 0 and 1 and can be set according to the needs of the smoothing intensity. When the smoothing control parameter is set to a small value, the smoothing intensity is small and the value of the first tensor changes little. When the smoothing control parameter is set to a large value, the smoothing intensity is large and the value of the first tensor changes significantly.

[0096] The specific hardware structure of the smoothing preprocessing unit provided in at least one embodiment of this disclosure is described below.

[0097] Figure 6 This is a schematic structural diagram of a smoothing preprocessing unit provided in at least one embodiment of the present disclosure.

[0098] like Figure 6 As shown, the smoothing preprocessing unit 101 includes a smoothing vector determination unit 1011 and a first multiplication unit 1012.

[0099] The smoothing vector determination unit 1011 is configured to determine the smoothing vector v and the smoothing inverse vector 1 / v.

[0100] The first multiplication unit 1012 is configured to perform a multiplication operation between the smoothed reciprocal vector and the first tensor (i.e., A×(1 / v)) to obtain the third tensor Asmooth, and to perform a multiplication operation between the smoothed vector v and the second tensor (i.e., W×v) to obtain the fourth tensor Wsmooth. Each element in the smoothed reciprocal vector is the reciprocal of each element in the smoothed vector.

[0101] The smoothed vector *v* and its inverse smoothed vector *1 / v* each contain *din* elements. The *din* columns in the first tensor *A* correspond to the *din* elements in the inverse smoothed vector. When performing v×W, each column of elements in the first tensor A is multiplied by the corresponding element in the smoothing inverse vector. The din rows in the second tensor W correspond to the din elements in the smoothing vector. When performing v×W, each row of elements in the second tensor W is multiplied by the corresponding element in the smoothing vector.

[0102] For example, the first multiplication unit could be an arithmetic logic unit (MUL ALU) that performs multiplication operations.

[0103] like Figure 6 As shown, the smoothing vector determination unit 1011 includes a first unit 1013, a second unit 1014, and a third unit 1015.

[0104] The first unit 1013 is configured to determine the maximum value of each column in the first tensor A, thereby obtaining din maximum values ​​of each column to form the first vector Aamax, and to determine the maximum value of each row in the second tensor W, thereby obtaining din maximum values ​​of each row to form the second vector Wamax.

[0105] din represents the size of the first and second tensors in the first dimension. The column maximum is the maximum absolute value of all elements in a column, and the row maximum is the maximum absolute value of all elements in a row.

[0106] For example, the first unit 1013 can be implemented using the ABSMAX (Absolute Maximum) module.

[0107] For example, the first unit 1013 is configured to determine the maximum value of each column in the din columns of the first tensor, obtain din column maximum values ​​and form a first vector Aamax, and determine the maximum value of each row in the din rows of the second tensor, obtain din row maximum values ​​and form a second vector Wamax.

[0108] The second unit 1014 is configured to determine the first power result and the second power result.

[0109] The first power result is obtained by raising the first vector to the power of α, denoted as Aamax. αThe second power result is obtained by raising the second vector to the power of 1-α, denoted as Wamax. 1-α α represents the smoothing control parameter, which is less than 1 and greater than 0.

[0110] The third unit is configured to determine the ratio of the first power result to the second power result as a smoothing vector, denoted as v=Aamax. α / Wamax 1-α And determine the ratio of the second power result to the first power result as the smoothed reciprocal vector, expressed as 1 / v=Wamax 1-α / Aamax α The slash " / " represents division.

[0111] For Aamax α and Wamax 1-α Direct calculation is quite complex, and current arithmetic logic units and special function units do not have hardware modules that directly support this calculation. In this disclosure, the second unit is implemented by combining the base-2 logarithmic unit and the base-2 exponential unit, which are provided and directly supported in the hardware, with the multiplication unit.

[0112] Specifically, based on the functions supported by the arithmetic logic unit and the special function unit, Aamax can be... α Convert to:

[0113] Aamax α =2 (α×log2(Aamax))

[0114] Similarly, Wamax can be used 1-α Convert to:

[0115] Wamax 1-α =2 ((1-α)×log2(Wamax))

[0116] Figure 7 A schematic structural diagram of the second unit provided for at least one embodiment of this disclosure.

[0117] like Figure 7 As shown, the second unit 1014 includes a logarithmic unit, a second multiplication unit, and an exponential unit.

[0118] The logarithmic unit is configured to perform a base-2 logarithmic operation on each element of the first and second vectors to obtain a first intermediate vector and a second intermediate vector. The first intermediate vector is represented as log2(Aamax), and the second intermediate vector is represented as log2(Wamax). The logarithmic unit can also be a special function unit.

[0119] The second multiplication unit is configured to perform a multiplication operation between the first intermediate vector and α to obtain the third intermediate vector, i.e., α × log2(Aamax), and to perform a multiplication operation between the second intermediate vector and 1-α to obtain the fourth intermediate vector, i.e., (1-α) × log2(Aamax). The second multiplication unit can be an arithmetic logic unit that performs element-wise multiplication.

[0120] The exponent unit is configured to perform a base-2 exponentiation operation on each element of the third intermediate vector to obtain the first power result, Aamax. α And by performing a base-2 exponentiation operation on each element of the fourth intermediate vector, we obtain the second power result, Wamax. 1-α The exponential unit can be a special function unit.

[0121] Of course, it should be noted that this disclosure is not limited thereto. Those skilled in the art can make reasonable modifications based on the arithmetic logic units and special function units supported by the hardware, and are able to implement Aamax. α and Wamax 1-α The calculation can be performed.

[0122] Figure 8 A schematic structural diagram of the third unit provided in at least one embodiment of this disclosure.

[0123] like Figure 8 As shown, the third unit 1015 includes a third multiplication unit, a first reciprocal unit, and a second reciprocal unit.

[0124] The first reciprocal unit is configured to determine the first power result (Aamax) α The reciprocal of the product is used to obtain the first reciprocal result ry. Specifically, the first power result Aamax is first determined. α If any element in the array contains a zero, then add t (a very small floating-point number, e.g., t = 1e). -12 To avoid division by zero, the first reciprocal unit performs the reciprocal operation on each element in the preprocessed first exponentiation result to obtain the first reciprocal result ry.

[0125] The third multiplication unit is configured to execute the second power result, Wamax. 1-α The multiplication operation with the first reciprocal result ry yields a multiplication result that serves as the smoothed reciprocal vector 1 / v.

[0126] The second reciprocal unit is configured to determine the reciprocal of the multiplication result of the third multiplication unit, thus obtaining the smooth vector v. Specifically, it first checks if any element in the multiplication result is 0. If it is, it adds t (a very small floating-point number, e.g., t=1e). -12( ), avoiding division by 0; then the second reciprocal unit performs the reciprocal operation on each element in the preprocessed multiplication result to obtain the smooth vector v.

[0127] For example, the first and second reciprocal units can be special function units, and the third multiplication unit can be an arithmetic logic unit that performs vector multiplication.

[0128] Of course, it should be noted that this disclosure is not limited thereto. Those skilled in the art can make reasonable modifications based on the arithmetic logic units and special function units supported by the hardware, and are able to implement Aamax. α and Wamax 1-α The ratio can be calculated. For example, those skilled in the art can also determine Wamax using the first reciprocal unit. 1-α The reciprocal of the result, then with Aamax α This disclosure does not impose specific restrictions on operations such as multiplication.

[0129] In the above embodiments, this disclosure provides a specific hardware structure for a smoothing preprocessing unit. This smoothing preprocessing unit is implemented based on the arithmetic logic unit and special function unit provided by the processor. It can efficiently perform smoothing preprocessing operations without performing the processing in dedicated hardware devices such as vector cores, thus freeing up the hardware computing resources of the vector core and avoiding resource competition between different computations on the vector core. This significantly improves the computing power utilization and main task execution efficiency of the vector core. Relying on the customized design of the hardware unit, it achieves efficient hardware-level execution of smoothing preprocessing, effectively reducing hardware resource scheduling overhead and computation latency, and improving the overall hardware execution efficiency of the processor.

[0130] For example, in some embodiments, the processing unit includes a quantization unit and a matrix multiplication unit.

[0131] The quantization unit is configured to determine the scaling factors of the third and fourth tensors, and to quantize the third and fourth tensors from a first floating-point format to a second floating-point format based on the scaling factors of the third and fourth tensors.

[0132] For example, a scaling factor is used to scale the range of data representation when the corresponding tensor is converted from a first floating-point format to a second floating-point format, where the floating-point precision of the first floating-point format is higher than that of the second floating-point format.

[0133] For example, the bit width of the first floating-point format is greater than that of the second floating-point format; for instance, the first floating-point format is BF16 and the second floating-point format is FP4. Of course, this disclosure is not limited to this.

[0134] The scaling factor can be determined by referring to Formula 2 mentioned above, and will not be elaborated further.

[0135] For example, in one embodiment, the quantization unit may include an arithmetic logic unit and a buffer, wherein the arithmetic logic unit is used to determine the scaling factor of each tensor and to quantize each tensor into floating-point form, and the buffer is used to cache the scaling factor of each tensor.

[0136] For example, in one specific embodiment, taking the third tensor as an example, the third tensor is divided into multiple tensor blocks, each tensor block includes multiple tensor elements, each tensor block corresponds to a scaling factor parameter, and multiple tensor elements share a scaling factor parameter.

[0137] The arithmetic logic unit is configured for each tensor block to: determine the sum of the maximum absolute value among multiple tensor elements and a preset floating-point number; determine the product of the sum and a preset constant, where the preset constant is the quotient of 1 and the maximum value of the precision range of the second floating-point number format; and convert the product into the precision specified by the scaling factor of any tensor to obtain the scaling factor parameter corresponding to the tensor block.

[0138] Of course, this disclosure does not specifically limit the structure and execution process of the quantization unit. The quantization unit can use any feasible technical solution to determine the scaling factors of the input third and fourth tensors and quantize them.

[0139] The matrix multiplication unit is configured to perform matrix multiplication operations using scaling factors based on the third and fourth tensors, as well as the quantized third and fourth tensors, to obtain the matrix multiplication result.

[0140] For matrix multiplication such as D = A' × B' + C', where A', B', C', and D are all high-precision tensors, the general matrix multiplication operation using scaling factors can be described as follows:

[0141] D = (α⊙A) × (β⊙B) + σ⊙C or D = (α⊙A) × (β⊙B) + σ⊙C β)⊙(A×B)+σ⊙C

[0142] Where ⊙ represents element-wise multiplication. σ represents the outer product, × represents matrix multiplication, α is the scaling factor of tensor A', β is the scaling factor of tensor B', σ is the scaling factor of tensor C', and γ is the scaling factor of tensor D. Quantized tensors A, B, C, and D are the low-precision tensors obtained by quantizing tensors A', B', C', and D' respectively after floating-point quantization. For example, the data dimensions of each parameter are shown below:

[0143] A: [m,T×k], B: [T×k,n], C: [m,n], D: [m,n], α: [m,T], β: [T,n], σ: [m,r] or [r,n], γ: [m,r] or [r,n].

[0144] Where m, n, s, k, and r are all positive integers. For any row in tensor A', every k tensor elements in that row share a single scaling factor parameter, and for any column in tensor B', every k tensor elements in that column share a single scaling factor parameter.

[0145] Matrix multiplication using scaling factors can be implemented using dot product units, adders, etc. A dot product unit is a hardware structure within a tensor kernel used to perform dot product operations. It efficiently calculates the dot product of elements in two vectors and sums the results to obtain the final dot product. Tensor kernels achieve large-scale matrix multiplication through the collaborative work of multiple dot product units. For example, dot product units can be organized into an array for efficient matrix multiplication.

[0146] This disclosure does not impose specific limitations on the specific implementation of matrix multiplication using scaling factors, and matrix multiplication using scaling factors can be performed in any feasible manner.

[0147] For example, the processor is configured to perform inference or training of a natural language processing model (e.g., a large language model). Matrix multiplication is an operation performed in the linear layers of the natural language processing model. The first tensor is the activation function matrix (or activation tensor), and the second tensor is the weight function matrix (or weight tensor). The first tensor represents the numerical semantic features obtained by processing the text data input to the natural language processing model, and the values ​​of the elements in the second tensor are uniformly distributed.

[0148] For example, matrix multiplication in a linear layer can be represented as X×W, where X represents the activation tensor, specifically L×din, where din is the size value of the first dimension, L is the size value of the second dimension, and W represents the weight tensor, specifically din×dout, where dout is the size value of the third dimension.

[0149] Of course, the processor can also be configured to perform related processing for other large models. It should be noted that, depending on the application domain, the first tensor and the second tensor can have different physical meanings.

[0150] For example, in the field of speech processing, the first and second tensors can be any parameters that require matrix multiplication operations and are used, input, or generated in tasks such as feature extraction, speech enhancement, and speech recognition, such as speech feature vectors and filtering parameters.

[0151] For example, in the field of image processing, the first and second tensors can be any parameters that require matrix multiplication operations and are used, input, or generated in tasks such as image preprocessing, feature extraction, image segmentation, and object detection. These parameters include image feature vectors, various edge detection operators (such as Sobel, Canny, and Prewitt operators), image filtering operators (such as Gaussian filtering, median filtering, and bilateral filtering), and morphological operators (such as erosion, dilation, opening, and closing operations).

[0152] For example, in the field of text processing, the first and second tensors can be any parameters that require matrix multiplication operations and are used, input, or generated in tasks such as text classification, sentiment analysis, and text generation, such as the semantic feature vector of the text.

[0153] For example, in the field of video processing, the first tensor and the second tensor can be parameters used in the field of image processing as described above, or parameters used, input, or generated by video processing-specific operators such as optical flow operators (used to estimate motion between video frames) and target tracking operators (used to track specific targets in video).

[0154] Of course, this disclosure is not limited to this. For other application scenarios or fields, such as computer graphics, machine learning, data mining, signal processing, natural language processing, geographic information systems, oceanography, environmental science and other technical fields, as long as matrix multiplication operations are required, the processor described in at least one embodiment of this disclosure can be used, and will not be described in detail here.

[0155] For example, the processor can be a parallel processor, including but not limited to a graphics processor, a general-purpose graphics processor, a data processor, a neural network processor, etc.

[0156] For example, a parallel processor includes multiple computing units for performing computational tasks in parallel. For instance, when the parallel processor is a graphics processing unit (GPU) or a general-purpose GPU, the computing units can be... Figure 1 The computing unit (CU) shown is provided in the diagram. For example, a scaling factor processing module is provided on each computing unit.

[0157] Figure 9 A schematic structural diagram of a processor provided for at least one embodiment of this disclosure.

[0158] like Figure 9 As shown, processor 10 includes multiple streaming processor clusters and memory, with each streaming processor cluster comprising multiple computing units. For a more detailed description of the streaming processor clusters, computing units, and memory, please refer to the foregoing. Figure 1 The relevant records will not be repeated here.

[0159] like Figure 9As shown, each computational unit includes a tensor kernel and a scaling factor processing module 100. The scaling factor processing module 100 can be referred to in the detailed description of the foregoing embodiments, and will not be repeated here.

[0160] In addition, it should be noted that Figure 9 The components of the processor 10 shown are merely exemplary and not limiting. The processor 10 may have other components as needed for the actual application, and this disclosure does not impose any specific limitations on this.

[0161] Figure 10 A schematic diagram of an electronic device provided in at least one embodiment of the present disclosure is shown.

[0162] For example, such as Figure 10 As shown, the electronic device 200 includes a processor 10. For example, the processor 10 may employ... Figure 4 The structure shown is implemented as described. For example, the electronic device 200 can be any electronic device including those with computing capabilities, such as a laptop, tablet, desktop computer, web server, etc., and the embodiments disclosed herein are not limited thereto.

[0163] For example, the electronic device may also include other forms of processing units with data processing capabilities and / or instruction execution capabilities, such as a central processing unit (CPU) or a digital signal processor (DSP), as well as storage units. The electronic device may also have an operating system and application programming interfaces (e.g., OpenGL (Open Graphics Library), Metal, etc.). For example, the electronic device may also include output components such as display components, such as liquid crystal displays (LCDs), organic light-emitting diode (OLED) displays, quantum dot light-emitting diode (QLED) displays, etc. The embodiments disclosed herein are not limited in this regard.

[0164] It should be noted that, for clarity and brevity, this disclosure does not show all the constituent units of the electronic device 200. To achieve the necessary functions of the electronic device 200, those skilled in the art can provide and set other constituent units (not shown) according to specific needs, and this disclosure does not limit this.

[0165] The following is for reference. Figure 11 It illustrates a specific structural diagram of an electronic device (e.g., a terminal device or a server) 300 suitable for implementing a processor including embodiments of the present disclosure.

[0166] The terminal devices in this disclosure can include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. For example, the electronic device can be in the form of a server, used for various application scenarios such as deep learning and artificial intelligence, scientific computing, graphics rendering and video editing, virtual reality and game development, and cloud services. For example, the electronic device can be a dedicated server such as a data center or cloud computing center that is deployed with tasks such as deep learning training, large-scale data analysis, and high-performance computing.

[0167] Figure 11 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0168] like Figure 11 As shown, the electronic device 300 may include a processing unit 301, such as the aforementioned processor 10, which can execute various appropriate actions and processes according to non-transitory computer-readable instructions stored in memory to achieve various functions. The processing unit 301 may also include devices with instruction optimization capabilities and / or program execution capabilities, such as a central processing unit (CPU) or a tensor processor (TPU). The CPU can be based on x86, ARM, or RISC-V architectures. The GPU can be directly integrated into the SOC, directly integrated onto the motherboard, or built into the northbridge chip of the motherboard.

[0169] like Figure 11 As shown, for example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 303 and / or cache memory, etc., for example, computer-readable instructions may be loaded from storage device 308 into RAM 303 to execute computer-readable instructions. Non-volatile memory may include, for example, read-only memory (ROM) 302, hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. Various applications and various data, such as various data used and / or generated by applications, may also be stored in the computer-readable storage medium.

[0170] For example, the processing device 301, the read-only memory (ROM) 302, and the random access memory (RAM) 303 are interconnected via a bus 304. The input / output (I / O) interface 305 is also connected to the bus 304.

[0171] Typically, the following devices can be connected to the input / output (I / O) interface 305: input devices 306 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 308 including, for example, magnetic tape, hard disk, flash memory, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 11 An electronic device 300 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 300 may alternatively implement or have more or fewer devices. For example, the processing device 301 may control other components in the electronic device 300 to perform desired functions.

[0172] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0173] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0174] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0175] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire at least two Internet Protocol (IP) addresses; send a node evaluation request including the at least two IP addresses to a node evaluation device, wherein the node evaluation device selects an IP address from the at least two IP addresses and returns it; and receive the IP address returned by the node evaluation device; wherein the acquired IP address indicates an edge node in a content delivery network.

[0176] Alternatively, the aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: receive a node evaluation request including at least two Internet Protocol (IP) addresses; select an IP address from the at least two IP addresses; and return the selected IP address; wherein the received IP address indicates an edge node in the content delivery network.

[0177] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0178] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0179] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0180] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0181] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0182] The following points should be noted regarding this disclosure:

[0183] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0184] (2) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0185] The above description is only a specific embodiment of this disclosure, but the protection scope of this disclosure is not limited thereto. The protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A processor, characterized in that, Includes a scaling factor processing module for performing matrix multiplication and accumulation operations using scaling factors on the first and second tensors. The scaling factor processing module includes a smoothing preprocessing unit and a processing unit. The smoothing preprocessing unit is configured to receive the first tensor, the second tensor, and smoothing control parameters, and to perform smoothing operations on the first tensor and the second tensor through multiple arithmetic logic units and special function units in the processor to obtain a smoothed third tensor corresponding to the first tensor and a smoothed fourth tensor corresponding to the second tensor. The smoothing operation includes scaling the values ​​of all elements of the first tensor to a uniform range according to the smoothing control parameters in response to the existence of at least one outlier in the first dimension of the first tensor. The smoothing control parameters are used to control the smoothing intensity of the first tensor, and the first dimension is the accumulation dimension when performing the matrix multiplication accumulation operation. The processing unit is configured to determine the scaling factor of each tensor block in the third tensor and the fourth tensor, and to perform the matrix multiplication and accumulation operation using the scaling factor based on the scaling factor of each tensor block in the third tensor and the fourth tensor to obtain the matrix multiplication operation result, wherein the third tensor and the fourth tensor are divided into multiple tensor blocks in the first dimension, and the floating-point precision in the matrix multiplication and accumulation operation is FP4; When the smoothing preprocessing unit performs the smoothing operation, it includes the following operations: A smoothing vector is determined for scaling the first dimension of the first tensor and the second tensor with a granularity of 1. The first tensor and the second tensor are scaled simultaneously using the smoothing vector to obtain the third tensor and the fourth tensor. The smoothing operation does not change the matrix multiplication result of the first tensor and the second tensor, and the length of the smoothing vector is the same as the size of the first tensor and the second tensor in the first dimension. The smoothing preprocessing unit includes a smoothing vector determination unit and a first multiplication unit. The smoothing vector determination unit is configured to determine the smoothing vector and the smoothing inverse vector, wherein each element in the smoothing inverse vector is the reciprocal of each element in the smoothing vector; The first multiplication unit is configured to perform a multiplication operation between the smoothed inverse vector and the first tensor to obtain the third tensor, and to perform a multiplication operation between the smoothed vector and the second tensor to obtain the fourth tensor; The smoothing vector determination unit includes a first unit, a second unit, and a third unit, wherein the first unit is the absolute value maximum module. The first unit is configured to determine the maximum value of each column in the first tensor, obtain din maximum values ​​of each column and form a first vector, and to determine the maximum value of each row in the second tensor, obtain din maximum values ​​of each row and form a second vector, wherein din represents the size of the first tensor and the second tensor in the first dimension, the maximum value of each column is the maximum value of the absolute values ​​of all elements in the corresponding column, and the maximum value of each row is the maximum value of the absolute values ​​of all elements in the corresponding row. The second unit is configured to determine a first power result and a second power result, wherein the first power result is the result obtained by raising the first vector to the power of α, and the second power result is the result obtained by raising the second vector to the power of 1-α, where α represents the smoothing control parameter, and α is less than 1 and greater than 0; The third unit is configured to determine the ratio of the first power result to the second power result as the smoothing vector, and to determine the ratio of the second power result to the first power result as the smoothing reciprocal vector; The second unit includes a logarithmic unit, a second multiplication unit, and an exponential unit. The logarithmic unit is configured to perform a base-2 logarithmic operation on each element of the first vector and the second vector to obtain a first intermediate vector and a second intermediate vector; The second multiplication unit is configured to perform a multiplication operation between the first intermediate vector and α to obtain a third intermediate vector, and to perform a multiplication operation between the second intermediate vector and 1-α to obtain a fourth intermediate vector; The exponent unit is configured to perform a base-2 exponentiation operation on each element of the third intermediate vector to obtain the first power result, and to perform a base-2 exponentiation operation on each element of the fourth intermediate vector to obtain the second power result.

2. The processor according to claim 1, characterized in that, The third unit includes a third multiplication unit, a first reciprocal unit, and a second reciprocal unit. The first reciprocal unit is configured to determine the reciprocal of the first exponentiation result to obtain the first reciprocal result; The third multiplication unit is configured to perform a multiplication operation between the second power result and the first reciprocal result, and the resulting multiplication result is used as the smooth reciprocal vector; The second reciprocal unit is configured to determine the reciprocal of the multiplication result obtained by the third multiplication unit to obtain the smooth vector.

3. The processor according to claim 1, characterized in that, The processing unit includes a quantization unit and a matrix multiplication unit. The quantization unit is configured to determine the scaling factors of the third tensor and the fourth tensor, and to quantize the third tensor and the fourth tensor from a first floating-point format to a second floating-point format based on the scaling factors of the third tensor and the fourth tensor. The scaling factors are used to scale the data representation range when the corresponding tensor is converted from the first floating-point format to the second floating-point format. The floating-point precision of the first floating-point format is higher than that of the second floating-point format. The matrix multiplication unit is configured to perform the matrix multiplication accumulation operation using the scaling factor based on the scaling factor of the third tensor and the fourth tensor, as well as the quantized third tensor and the fourth tensor, to obtain the matrix multiplication operation result.

4. The processor according to claim 1, characterized in that, The processor is configured to perform inference or training of a natural language processing model. The matrix multiplication and accumulation operation is an operation performed in the linear layer of the natural language processing model. The first tensor is the activation tensor, and the second tensor is the weight tensor. The first tensor is a numerical semantic feature obtained by processing the text data input to the natural language processing model. The values ​​of each element in the second tensor are uniformly distributed.

5. The processor according to any one of claims 1-4, characterized in that, The processor is a parallel processor, which includes multiple computing units for executing computing tasks in parallel. The scaling factor processing module is provided on each computing unit.

6. An electronic device, characterized in that, Includes the processor as described in any one of claims 1-5.

Images