Processor, chip product, computer equipment and tensor calculation method

Abstract

The invention discloses a processor, a chip product, computer equipment and a tensor calculation method, and relates to the technical field of processors. A first tensor is stored in a first storage unit of the processor; a plurality of scaling factors corresponding to the first tensor are stored in a second storage unit of the processor, the scaling factor set comprises a plurality of scaling factors, each scaling factor corresponds to one sub-block in the first tensor, each sub-block comprises at least one element in the first tensor, and each sub-block comprises at least one element in the first tensor; the scaling factor is used for adjusting the dynamic range of each element in the same sub-block; a tensor calculation engine of the processor is used for reading a first tensor from the first storage unit and reading a scaling factor set from the second storage unit; and for any sub-block in the first tensor, executing logical operation on elements in the sub-block according to the scale factor corresponding to the sub-block to obtain an operation result. According to the method and the device, the tensor calculation engine can realize accurate operation of the low-precision tensor through the scale factor of the sub-block granularity.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of processor technology, and in particular to a processor, a chip product, a computer device, and a tensor calculation method. Background Art

[0002] A large model is a machine learning model with a large number of parameters and complex structure. As the scale of the large model increases, the requirements for the computing power and storage capacity of the processor are also getting higher and higher.

[0003] In related technologies, data is constructed by replacing high-precision data formats with low-precision data formats to reduce the amount of calculation and storage required for large models, thereby reducing the requirements for the computing power and storage capacity of the processor. For example, tensors in the FP32 (Floating-Point 32-bit, 32-bit floating point) data format are replaced with tensors in the FP16 (Floating-Point 16-bit, 16-bit floating point) data format to perform tensor calculations to achieve the functions of large models.

[0004] However, it is currently difficult to implement tensor computations on low-precision data formats using existing processors. Summary of the invention

[0005] The embodiments of the present application provide a processor, a chip product, a computer device and a tensor calculation method. The technical solution provided by the embodiments of the present application includes the following contents.

[0006] According to one aspect of an embodiment of the present application, a processor is provided, the processor comprising: a tensor computing engine, a first storage unit and a second storage unit;

[0007] The first storage unit stores a first tensor;

[0008] The second storage unit stores a scale factor set corresponding to the first tensor, the scale factor set includes a plurality of scale factors, each scale factor corresponds to a sub-block in the first tensor, each sub-block includes at least one element in the first tensor, and the scale factor is used to adjust the dynamic range of each element in the same sub-block;

[0009] The tensor computing engine is used to read the first tensor from the first storage unit and read the scale factor set from the second storage unit; for any sub-block in the first tensor, perform logical operations on the elements in the sub-block according to the scale factor corresponding to the sub-block to obtain an operation result.

[0010] According to one aspect of an embodiment of the present application, a chip product is provided, wherein the chip product includes the above-mentioned processor.

[0011] According to one aspect of an embodiment of the present application, a computer device is provided, wherein the computer device includes the above-mentioned processor.

[0012] According to one aspect of an embodiment of the present application, there is provided a tensor computing method applied to a processor, the processor comprising: a tensor computing engine, a first storage unit and a second storage unit;

[0013] The first storage unit stores a first tensor;

[0014] The second storage unit stores a scale factor set corresponding to the first tensor, the scale factor set includes a plurality of scale factors, each scale factor corresponds to a sub-block in the first tensor, each sub-block includes at least one element in the first tensor, and the scale factor is used to adjust the dynamic range of each element in the same sub-block;

[0015] The method comprises:

[0016] The tensor computation engine reads the first tensor from the first storage unit, and reads the set of scaling factors from the second storage unit;

[0017] For any sub-block in the first tensor, the tensor calculation engine performs a logical operation on the elements in the sub-block according to a scale factor corresponding to the sub-block to obtain an operation result.

[0018] The technical solution provided by the embodiments of the present application may include the following beneficial effects.

[0019] By supporting the tensor computing engine to perform logical operations on the elements included in any sub-block in the tensor according to the scaling factor of the sub-block granularity (that is, each sub-block in the tensor corresponds to a scaling factor, rather than the entire tensor corresponds to a scaling factor), the problem of data precision loss caused by using the scaling factor of the tensor granularity is avoided. Even when processing low-precision tensors (that is, the encoded data group of each element in the tensor is constructed in a low-precision data format, such as the total number of bits corresponding to the low-precision data format is less than 8), the present application implements the scaling factor based on the sub-block granularity to avoid data precision loss, thereby accurately performing low-precision tensor calculations, which is beneficial to expand the scope of application of the processor to the data format and improve the accuracy of the processor's data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a schematic diagram of a processor provided in a possible implementation of the present application;

[0022] Figure 2 is a schematic diagram of a tensor computing engine provided in a possible implementation of the present application;

[0023] Figure 3 is a schematic diagram of the arrangement of scaling factors provided in a possible implementation of the present application;

[0024] Figure 4 is a schematic diagram of the arrangement of scaling factors provided in another possible implementation of the present application;

[0025] Figure 5 is a schematic diagram of the arrangement of scaling factors provided in another possible implementation of the present application;

[0026] Figure 6 is a schematic diagram of a processing unit performing a multiplication-accumulation operation provided in a possible implementation of the present application;

[0027] Figure 7 is a schematic diagram of a tensor storage engine provided in a possible implementation of the present application;

[0028] Figure 8 is a schematic diagram of tensor transmission and tensor storage provided in a possible implementation of the present application;

[0029] Fig. 9 is a schematic diagram of a graphics processor provided in a possible implementation of the present application;

[0030] Fig.10 is a schematic diagram of a processor in a graphics processor provided in a possible implementation of the present application;

[0031] Fig.11 is a flowchart of a tensor calculation method applied to a processor provided in a possible implementation of the present application;

[0032] Fig.12 It is a simplified structural block diagram of a computer device provided in a possible implementation of the present application. DETAILED DESCRIPTION

[0033] Before introducing the embodiments of the present application, the relevant terms involved in the present application are first explained.

[0034] Tensor: It is a general term and extension of concepts such as scalar, vector, matrix, etc. It is a basic data structure in the field of machine learning. It can be understood as a multidimensional array, which includes at least one element. For example, a 0-dimensional tensor is 1 number, a 1-dimensional tensor is equivalent to a vector, a 2-dimensional tensor corresponds to a matrix, and a 3-dimensional tensor is a cube. The embodiment of the present application does not limit the dimension of the tensor, such as the dimension of the tensor can also be 4 dimensions, and more than 4 dimensions.

[0035] Tensors can store more information about objects, such as the height, color, depth, width, etc. of images, the number of frames, color, height, width, etc. of videos, and the time domain, frequency domain, and fundamental frequency of audio.

[0036] Tensor computing mainly refers to the calculation and processing of tensors in the fields of deep learning and machine learning. These calculations usually involve operations such as matrix multiplication and convolution, and are key steps in machine learning model training and reasoning.

[0037] Floating point type: A data type used to represent real numbers. It allows the decimal point to float, so that fractions, decimals, and some irrational numbers can be represented. Real numbers represented by floating point types are called floating point numbers.

[0038] The floating-point number type includes at least one of the following: single-precision floating-point type (float), double-precision floating-point type (double), and floating-point types such as FP16, BF16 (Brain Floating-Point 16-bit, 16-bit floating point number), FP8 (Floating-Point 8-bit, 8-bit floating point number), FP6 (Floating-Point 6-bit, 6-bit floating point number), and FP4 (Floating-Point 4-bit, 4-bit floating point number).

[0039] Floating point numbers are usually represented in scientific notation. For example, FP16: half-precision floating point number, 16-bit floating point representation, that is, it uses a 16-bit value to represent real numbers, including 1-bit sign, 5-bit exponent and 10-bit mantissa. FP8: uses an 8-bit value to represent real numbers, including 4-bit exponent and 3-bit exponent (E4M3), or 5-bit exponent and 2-bit mantissa (E5M2), aiming to improve computing efficiency by reducing the number of bits of data representation.

[0040] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0041] Please refer to Figure 1 , which shows a schematic diagram of a processor provided in a possible implementation of the present application. The processor 100 may include: a tensor compute engine (TCE) 101, a first storage unit 103 and a second storage unit 104.

[0042] The processor 100 may be a processor supporting tensor processing, such as the processor 100 supporting at least one of the following operations for tensors: transmission, storage, and calculation. For example, the tensor storage engine 102 in the processor 100 supports tensor transmission, each storage unit in the processor 100 supports tensor storage, and the tensor calculation engine 101 in the processor 100 supports tensor calculation. Among them, tensor transmission may refer to the transmission of tensor data corresponding to a tensor, tensor storage may refer to the storage of tensor data corresponding to a tensor, and tensor calculation may refer to the calculation of tensor data corresponding to a tensor.

[0043] The embodiment of the present application does not limit the processor 100, and the processor 100 may be a newly designed one or an improved one of an existing processor. Exemplarily, the processor 100 may be implemented as at least one of the following: a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a tensor processor (TPU), a field programmable gate array (FPGA), a tensor core (Tensor Core), and an NPU (Neural Processing Unit).

[0044] In one example, the processor 100 supports the processing of low-precision tensors. A low-precision tensor refers to a tensor in which the encoded data of an element is represented in a low-precision data format. In an embodiment of the present application, a low-precision data format may refer to a data format in which the total number of bits (or total number of bits) is less than or equal to 8, such as FP8, FP6, or FP4. For example, if the encoded data of each element in a tensor is represented in a data format such as FP4, FP6, or FP8, the tensor at this time may be referred to as a low-precision tensor. In some embodiments, a data format in which the total number of bits (or total number of bits) is less than or equal to 8 may be referred to as a low-precision data format.

[0045] Optionally, the processor 100 also supports the processing of non-low-precision tensors, such as tensors in data formats such as FP8, INT8 (8-bit integer), and FP16, which is not limited in the present embodiment of the application. For example, if the encoded data of each element in the tensor is represented by a data format such as FP32, FP16, or BF16, the tensor at this time can be called a non-low-precision tensor.

[0046] The tensor computing engine 101 refers to a computing engine in the processor 100 for performing tensor computing. The operation logic supported by the tensor computing engine 101 includes at least one of the following: matrix multiplication operation, convolution operation, multiplication-accumulation operation, and addition operation. For example, the tensor computing engine 101 can be used to implement multiplication-accumulation operations for tensors to obtain multiplication-accumulation operation results of tensors. The embodiment of the present application does not limit the number of tensor computing engines 101 included in the processor 100, and it can be set and adjusted according to actual usage requirements.

[0047] In one example, the tensor computing engine 101 supports the computation of low-precision tensors, so that the processor 100 supports the computation of low-precision tensors. For example, the tensor computing engine 101 can be used to implement logical operations on tensors in FP6 data format and tensors in FP4 data format.

[0048] The tensor computing engine 101 can be used to read tensors from the first storage unit 103 to perform logical operations on the tensors to achieve tensor computing. In the processor 100, the tensor can be represented by tensor data. Among them, tensor data refers to data used to describe the tensor, which can be composed of encoded data generated for each element in the tensor. In an embodiment of the present application, processing tensors is essentially processing tensor data, such as performing logical operations on the tensor data of the tensor to achieve tensor computing.

[0049] In an embodiment of the present application, a plurality of encoded data groups of tensors are stored in the first storage unit 103, and the encoded data groups are generated based on the encoded data of the elements, and each encoded data is used to characterize an element in the tensor, such as each encoded data is used to indicate the value of an element in the tensor. Optionally, each encoded data group may include at least one encoded data. For example, for the FP6 data format, each element may be represented by a set of 6-bit data, and the 6-bit data respectively indicates the sign, exponent and mantissa of the floating point number (ie, the element). In the case where the data amount of the encoded data group is 8 bits, each encoded data group may be generated based on one encoded data. For the FP4 data format, each element may be represented by a set of 4-bit data, and the 4-bit data respectively indicates the sign, exponent and mantissa of the floating point number (ie, the element). In the case where the data amount of the encoded data group is 8 bits, each encoded data group may be generated based on 2 consecutive encoded data.

[0050] In one example, the storage and transmission of tensor data can be aligned with 8 bits (i.e., the minimum access granularity below). For example, for the FP6 data format, tensor data can be stored in a continuous manner of 128 consecutive elements to form 96Bytes of data, which can be aligned to 32Bytes (i.e., 32 8bits). The specific number of elements to be accessed continuously can be determined based on the bandwidth of memory access. For example, an integer multiple of 32Bytes can be used to access tensor data. For the FP6 data format, 128 elements (corresponding to 96Bytes) can be accessed continuously, and for the FP4 data format, 64 or 128 elements (corresponding to 32Bytes or 64Bytes) can be accessed continuously. The embodiment of the present application does not limit the memory access method of tensor data.

[0051] A data output channel (such as a bus) is provided between the first storage unit 103 and the tensor computing engine 101 to realize the transmission of tensor data. The first storage unit 103 can be used to temporarily store tensor data to be calculated. Optionally, the first storage unit 103 can be used to store tensor data of low-precision tensors. For example, the first storage unit 103 can be used to store tensor data of tensors in FP4 data format and tensor data of tensors in FP6 data format.

[0052] Exemplarily, the first storage unit 103 can be implemented as an on-chip cache, recorded as local memory, to achieve temporary storage of tensor data, thereby improving the speed at which the tensor computing engine 101 acquires tensor data. The first storage unit 103 can also be implemented as a register to achieve temporary storage of tensor data, thereby further improving the speed at which the tensor computing engine 101 acquires tensor data (the data reading speed of the register is much faster than the data reading speed of the memory). In some embodiments, the first storage unit 103 can be implemented as a first-level cache in the processor 100, i.e., an L1 cache, which is the cache layer closest to the tensor computing engine 101, and usually has a relatively small storage space, but a very fast data reading speed.

[0053] The tensor computing engine 101 can be used to read the scaling factor set corresponding to the tensor from the second storage unit 104 to assist the logical operation of the tensor. The scaling factor set includes multiple scaling factors (Scaling Factor), each scaling factor corresponds to a sub-block in the tensor, and each sub-block includes at least one element in the tensor, that is, each scaling factor corresponds to the encoded data corresponding to a sub-block. The scaling factor is used to adjust the dynamic range of each element in the same sub-block, and a value in the dynamic range can be determined as the value of the element.

[0054] A data output channel is provided between the second storage unit 104 and the tensor computing engine 101 to realize the transmission of the scale factor. The second storage unit 104 can be used to temporarily store the scale factor set to be used. Exemplarily, the second storage unit 104 can be implemented as an on-chip cache, recorded as a local memory (Local Memory), to realize the temporary storage of the scale factor set, thereby improving the speed at which the tensor computing engine 101 obtains the scale factor set. The second storage unit 104 can also be implemented as a register, such as a vector register (Vector Register) / vector register file (Vector Register File), to realize the temporary storage of the scale factor set, thereby further improving the speed at which the tensor computing engine 101 obtains the scale factor set. The second storage unit 104 can also be implemented as an L1 cache in the processor 100, that is, the L1 cache includes at least one of the following: the first storage unit 103, the second storage unit 104. Optionally, the first storage unit 103 and the second storage unit 104 can be implemented as the same storage unit.

[0055] Optionally, the processor 100 may also include: a tensor memory engine (TME) 102 and a third storage unit 105. The tensor storage engine 102 refers to a computing engine in the processor 100 for performing tensor transmission and tensor storage. Exemplarily, the tensor storage engine 102 can be used to read tensor data and a scale factor set from the third storage unit 105 to achieve the transmission of tensors and scale factor sets, and to store the tensor data and the scale factor set in the first storage unit 103 and the second storage unit 104, respectively, to achieve the storage of tensors and scale factor sets. Optionally, the tensor storage engine 102 can be used for the transmission of tensor data of low-precision tensors. For example, the tensor storage engine 102 can be used to transmit and store tensor data of tensors in FP4 data format, and can also be used to transmit and store tensor data of tensors in FP6 data format.

[0056] Data output channels are respectively provided between the tensor storage engine 102 and the first storage unit 103 , the second storage unit 104 and the third storage unit 105 to implement transmission of tensor data and scale factor sets in the processor 100 .

[0057] The third storage unit 105 can be implemented as a shared cache of multiple tensor computing engines 101, recorded as global memory, which can support multiple tensor computing engines 101 to access data thereto, such as by supporting multiple tensor computing engines 101 to access tensor data, so as to realize the sharing of tensor data among multiple tensor computing engines 101.

[0058] In some embodiments, the third storage unit 105 can be implemented as a secondary cache in the processor 100, namely, an L2 cache. In this case, the third storage unit 105 is located between the L1 cache and the main memory of the processor 100 (such as the L3 cache) and is used to store more data so as to provide more backup in case of an L1 cache miss. The storage space of the L2 cache is larger than the storage space of the L1 cache, but the data reading speed of the L2 cache is lower than the data reading speed of the L1 cache.

[0059] To summarize, by supporting the transmission of low-precision tensors by the tensor storage engine, the storage of low-precision tensors by the first storage unit, and the calculation of low-precision tensors by the tensor computing engine, the processor supports the processing of low-precision tensors, thereby expanding the range of data formats that the processor can support, so that the processor supports the processing of tensor data in more data formats.

[0060] In some embodiments, taking the first tensor as an example, the process of implementing tensor calculation by the tensor calculation engine 101 is described.

[0061] The first tensor mentioned above may refer to any tensor to be processed by the processor 100. Optionally, the first tensor may be a tensor composed of some elements in a tensor, such as when an M×N tensor (i.e., a matrix) is evenly divided into k (M / k)×N blocks, the first tensor may refer to a tensor composed of elements corresponding to any block in the k blocks. The first tensor may also be an undivided tensor, such as the first tensor is an M×N tensor. The embodiment of the present application does not limit the source of the first tensor.

[0062] The embodiments of the present application do not limit the elements in the first tensor. Exemplarily, the first tensor can be used to store model parameters of a neural network model, such as model parameters (such as weight parameters, bias parameters, etc.) of each model layer (such as convolutional layer, fully connected layer, attention mechanism layer, etc.) in a large model. The first tensor can also be used to store feature data in deep learning tasks, such as image feature data, speech feature data, text feature data, video feature data, etc. The first tensor can also be used to store coefficients in functions or formulas.

[0063] Please refer to Figure 2 , the process of implementing tensor calculation by the tensor calculation engine 101 may include the following contents:

[0064] 1. The tensor computing engine 101 is used to read the first tensor from the first storage unit 103 and read the scale factor set from the second storage unit 104.

[0065] Reading the first tensor may refer to reading the tensor data of the first tensor. The tensor data of the first tensor is used to represent the first tensor. The above-mentioned scale factor set refers to the scale factor set corresponding to the first tensor. Optionally, the first storage unit 103 stores the first tensor, that is, the first storage unit 103 includes multiple encoded data groups for representing the first tensor, and each encoded data group includes encoded data generated according to the second data format for at least one element in the first tensor.

[0066] Each encoded data group may include at least one encoded data, and each encoded data is used to represent an element of the first tensor. Each encoded data may be a value of the element generated according to the second data format, that is, each encoded data group may be used to indicate the value of one or more elements. At least one encoded data in each encoded data group corresponding to the first tensor may constitute tensor data.

[0067] For example, when each coding data group corresponds to one element, that is, each coding data group includes coding data of one element, each coding data group can be used to determine the value of one element; when each coding data group corresponds to two elements, that is, each coding data group includes coding data of two elements, each coding data group can be used to determine the values ​​of the two elements respectively.

[0068] The total number of bits of the encoded data of each element may be determined based on the second data format. For example, the total number of bits of the encoded data of each element is the same as the total number of bits (i.e., the total number of bits) corresponding to the first data format. The embodiment of the present application does not limit the format type of the second data format, such as the format type of the second data format may include at least one of the following: floating point type, integer type.

[0069] In one example, the total number of bits corresponding to the second data format is less than 8, that is, the first tensor belongs to a low-precision tensor. Exemplarily, the second data format may include at least one of the following: FP4, FP6, FP8, FP16, FP32. For example, when the second data format is FP4, each element can be represented by a set of 4-bit data; when the second data format is FP6, each element can be represented by a set of 6-bit data. Optionally, each encoded data can be represented by a set of binary numbers, such as 0, 1. The sign of the element may correspond to a binary number, and the remaining binary numbers correspond to the exponent and mantissa of the element. The number of binary numbers corresponding to the exponent and the number of binary numbers corresponding to the mantissa are not limited. The embodiment of the present application improves the scope of application of the processor for the data format by supporting the calculation of tensor data in the FP4 data format and the FP6 data format, and is conducive to further improving the computing efficiency of the processor for tensors.

[0070] The total number of bits of the above-mentioned coded data groups can be set and adjusted according to actual use requirements, such as 8bit, 16bit, 32bit, etc. For ease of explanation, the embodiment of the present application is described by taking the total number of bits of each coded data group as 8bit as an example. For example, when the total number of bits of the coded data group is 8bit, if the second data format is FP4, each coded data group may be composed of 4-bit coded data of two consecutive elements, that is, the first 4 bits of coded data correspond to the first element, and the latter 4 bits of coded data correspond to the second element. If the second data format is FP6, each coded data group can be generated based on the 6-bit coded data of a single element, such as the latter 6 bits of coded data correspond to one element.

[0071] In one example, the tensor computing engine 101 may be used to sequentially read multiple encoded data groups corresponding to the tensor data from the first storage unit 103 according to an operation instruction for the first tensor, so as to implement reading of each element in the first tensor.

[0072] The operation instruction of the first tensor is used to instruct the tensor computing engine 101 to perform a logical operation on the first tensor. Optionally, the tensor computing engine 101 can determine the data structure information and address information of the tensor data based on the operation instruction of the first tensor, and the data structure information includes at least one of the following: the dimension of the tensor data, the location of the tensor data, and the data format of the elements in the tensor data. The address information includes at least one of the following: the base address (Base Address) of the tensor data in the first storage unit 103 and the addressing space.

[0073] The tensor computing engine 101 can determine the coordinate addresses of multiple encoded data groups in the first storage unit 103 according to the data structure information and address information of the tensor data, and then read the multiple encoded data groups corresponding to the tensor data from the first storage unit 103 in sequence according to the coordinate addresses of the multiple encoded data groups to complete the reading of the first tensor.

[0074] For example, refer to Figure 2 , the processor 100 also includes a constant memory 106, which is used to store data that does not change during the tensor calculation process, in order to improve performance and reduce the bandwidth occupation of the third storage unit 105. The tensor calculation engine 101 parses the operation instructions from the instruction unit (not shown in the figure) to obtain the tensor descriptor of the tensor data from the constant memory 106. The tensor descriptor is used to indicate the data format, arrangement mode, etc. of the elements in the tensor data, and obtain the dimension and position of the tensor data from the register stack. The tensor calculation engine 101 is used to determine the coordinate address of each encoded data group in the first storage unit 103 based on the information such as the dimension, position, data format, arrangement mode, etc. of the tensor data, and then read multiple encoded data groups from the first storage unit 103 in sequence based on the coordinate address.

[0075] Optionally, the second storage unit 104 stores a scale factor set corresponding to the first tensor, where the scale factor set includes multiple scale factors, each scale factor corresponds to a sub-block in the first tensor, each sub-block includes at least one element in the first tensor, and the scale factor is used to adjust the dynamic range of each element in the same sub-block.

[0076] A scale factor set may refer to a set or sequence of scale factors. A scale factor is a parameter used to adjust the dynamic range of an element, which can be used to quantize an element to map the element from one dynamic range to another. For example, for a floating point number xi , based on the scale factor, INT8 symmetric quantization can be performed on it, then the quantized x i The dynamic range of x can be [-128, 127]. i It can be expressed as follows:

[0077] x int8 =round(S*x i );

[0078] in, is the scale factor, round() means rounding to a given number of digits, max() means taking the maximum value, and abs() means taking the absolute value.

[0079] The above sub-blocks may also be referred to as sub-tensors of the first tensor, which may be obtained by dividing the first tensor. For example, the first tensor may be evenly divided into a plurality of sub-blocks, and the embodiment of the present application does not limit the division method of the sub-blocks, such as dividing each consecutive set number of elements in the first tensor into a sub-block, and the set number may include at least one of the following: 32, 64, 16, 8. For example, each consecutive 32 elements in the first tensor may be divided into a sub-block.

[0080] Quantization refers to a technique for performing calculations and storing tensor data at a bit width lower than floating-point precision. It is one of the effective ways to reduce the computing time and energy consumption of neural networks. For example, the quantization of a tensor is a process of converting floating-point numbers in a tensor into integers, usually from FP32 floating-point numbers to INT8 integers. The basic principle of quantization is linear quantization based on dynamic range.

[0081] In one example, the number of elements included in each sub-block is related to the inner product size of the tensor computing engine 101 (i.e., the number of elements corresponding to a tensor of a specified size that satisfies the inner product operation condition). Exemplarily, the inner product size of the tensor computing engine 101 can be determined as the number of elements included in each sub-block. Assuming that the inner product size of the computing engine 101 is 32, 32 can be determined as the number of elements included in each sub-block.

[0082] After the first tensor is divided into multiple sub-blocks, a scale factor can be determined for each sub-block, thereby obtaining a set of scale factors corresponding to the first tensor. In one example, the data format of the above-mentioned scale factor includes FP8, that is, an unsigned floating point number represented by 8-bit encoded data can be used as a scale factor. The meaning of the scale factor is E8M0, the exponent bias (offset value) is 127, and the dynamic range is -127 to 127. The embodiment of the present application supports the scale factor of the FP8 data format, which is beneficial for the processor 100 to reuse the tensor computing engine 101 that supports the FP8 data format, thereby helping to reduce the development cost of the processor 100. The embodiment of the present application does not limit the data format of the scale factor, and it can be set and adjusted according to actual usage requirements.

[0083] For example, refer to Figure 3 For the 8*128 matrix A, the corresponding scale factors 301 can be arranged in the second storage unit 104 according to the arrangement order of the sub-blocks in the matrix A. Every 32 consecutive elements in the matrix A form a sub-block, each sub-block corresponds to a scale factor 301, and each scale factor 301 can be represented by 8-bit coded data. M0 to M7 represent 8 rows of the matrix respectively, and k0 to k127 represent 128 columns of the matrix respectively.

[0084] The 32 elements corresponding to each sub-block share the same scale factor 301 corresponding to the sub-block. For example, during the quantization process, the 32 elements corresponding to the sub-block are quantized based on the scale factor 301 corresponding to the sub-block.

[0085] In one example, for any sub-block, the process of determining the scale factor of the sub-block may include the following:

[0086] (1) Obtain the exponent e_max of the maximum normal number in the second data format.

[0087] For example, for the FP4 data format (E2M1), e_max is 2; for the FP6 data format (E2M3), e_max is 2; for the FP8 data format, if it is in the (E4M3) format, e_max is 8; if it is in the (E5M2) format, e_max is 15.

[0088] (2) Obtain the maximum absolute value v_max of the elements in the sub-block.

[0089] (3) Based on e_max and v_max, the scaling factor of the sub-block is calculated.

[0090] For example, the scale factor of a sub-block can be expressed as: S = log 2 (v_max)-e_max.

[0091] Optionally, the following formula can be used to calculate the element v in the sub-block: i Quantify: v i / (2 S ).

[0092] In some embodiments, each element in the first tensor may also share a scaling factor. Figure 3 Each element in the matrix A shares a scaling factor, which is not limited in this embodiment of the present application.

[0093] In one example, the tensor computing engine 101 may be used to read the scale factor from the second storage unit 104 according to the index of the scale factor. The index of the scale factor may be used to indicate the position of the scale factor in the second storage unit 104 to read the scale factor. For example, when the second storage unit 104 is a register, the index of the scale factor is the index of the scale factor in the register, and the tensor computing engine 101 may read the scale factor from the register according to the index. Optionally, the second storage unit 104 may be implemented as at least one of the following: Vector Register File, Scale register. The Scale register is dedicated to storing a set of scale factors.

[0094] Optionally, the index of the scale factor may also be stored in the second storage unit 104 (such as Vector RegisterFile). The embodiment of the present application does not limit the arrangement of the index of the scale factor. For example, the arrangement of the index of the scale factor may be the same as the arrangement of the scale factor, or may be different from the arrangement of the scale factor, and it may be set and adjusted according to actual use requirements.

[0095] In one example, the scaling factors in the second storage unit 104 may be stored by the tensor storage engine 102 or a load store unit (LSU), such as the tensor storage engine 102 or the load store unit may store the scaling factors in sequence in the direction of any dimension of the first tensor. Exemplarily, the tensor storage engine 102 or the load store unit is used to store the scaling factors corresponding to the sub-blocks of the 1st row to the mth row in each column in the order of the 1st column to the nth column in the second storage unit 104, where m and n are positive integers. In this case, the index of the scaling factor is the register index corresponding to the scaling factor in the second storage unit, and the index of the scaling factor is used to read the scaling factor.

[0096] For example, refer to Figure 3 and Figure 4, register 401 (i.e., Ra) is used to store the scale factor corresponding to matrix A, and register 402 (i.e., Rb) is used to store the scale factor corresponding to matrix B. Then the register index of register 401 is the index of the scale factor corresponding to matrix A, and the register index of register 402 is the index of the scale factor corresponding to matrix B. Figure 3 For example, when the storage granularity of register 401 is 8 bits (i.e., 1 byte), each storage area of ​​register 401 is used to store 4 scale factors, and the index of each storage area is the index of the 4 scale factors. For example, storage area T0 is used to store the first 4 scale factors in the first column corresponding to matrix A, i.e., m=0 to 3 correspond to M0 to M3 in the first column, respectively, and the index of storage area T0 is the index of the first 4 scale factors. Storage area T1 is used to store the last 4 scale factors in the first column corresponding to matrix A, i.e., m=4 to 7 correspond to M4 to M7 in the first column, respectively, and the index of storage area T1 is the index of the last 4 scale factors. By analogy, register 401 implements the storage of the scale factors of matrix A through 8 consecutive storage areas, and register 402 implements the storage of the scale factors of matrix B through 8 consecutive storage areas.

[0097] For example, refer to Figure 5 For matrix A, after the proportional factors corresponding to the first column sub-blocks corresponding to matrix A are stored through the first 8 continuous storage areas (i.e., T0 to T7) in register 501, the proportional factors corresponding to the second column sub-blocks corresponding to matrix A are stored through the last 8 continuous storage areas (i.e., T8 to T15) in register 501, and so on. For matrix B, after the proportional factors corresponding to the first column sub-blocks corresponding to matrix B are stored through the first 8 continuous storage areas in register 502, the proportional factors corresponding to the second column sub-blocks corresponding to matrix B are stored through the last 8 continuous storage areas in register 502, and so on.

[0098] The embodiment of the present application can realize the rapid acquisition of the scale factor of the sub-block granularity through the index of the scale factor, which is conducive to improving the acquisition accuracy of the scale factor.

[0099] In some embodiments, when the elements in the first tensor can share the same scale factor, the tensor storage engine 102 can be used to read the same scale factor from the second storage unit 104 according to the index of the second storage unit 104 (such as the index of the VectorRegister where the scale factor is located), which is not limited to this embodiment of the present application.

[0100] In one example, reference Figure 2The tensor computing engine 101 also includes a control logic unit 1011, which can be used to read data from the first storage unit 103 and the second storage unit 104 to move the tensor data and the scale factor set to the tensor computing engine 101.

[0101] The tensor computing engine 101 also includes a processing unit array 1012, which includes a plurality of processing units (DP for short) for performing logical operations. The embodiment of the present application does not limit the number of processing units in the processing unit array 1012, such as 8×8=64, 4×6=32, 8×16=128. The embodiment of the present application does not limit the type of logical operations that can be performed by the processing unit, and the logical operations supported by the processing unit include at least one of the following: matrix multiplication, convolution, multiplication-accumulation, and addition. The embodiment of the present application does not limit the structure of the processing unit. Exemplarily, when the processing unit is used to perform multiplication-accumulation operations, the processing unit may include a multiplier, an adder, and the like. The structures of the processing units may be the same, or the structures of the processing units may be different.

[0102] Optionally, the control logic unit 1011 may be connected to each processing unit in the processing unit array 1012 to distribute multiple elements (ie, encoded data groups) and scaling factors corresponding to the tensor data to the processing units.

[0103] Exemplarily, the control logic unit 1011 is used to move, for any sub-block in the first tensor, the elements in the sub-block from the first storage unit 103 to a target processing unit corresponding to the sub-block among multiple processing units, and to move the scaling factor corresponding to the sub-block from the second storage unit 104 to the target processing unit.

[0104] Among them, carrying the elements of the sub-block to the target processing unit may refer to carrying the encoded data group corresponding to the sub-block to the target processing unit. The target processing unit may refer to any processing unit of a plurality of processing units, such as a processing unit for processing the above-mentioned sub-block. Optionally, the control logic unit 1011 may be used to assign the sub-blocks of the first tensor (i.e., the encoded data groups corresponding to the sub-blocks) to different processing units in sequence. Exemplarily, for a third sub-block located before the first sub-block and adjacent to the first sub-block, the second processing unit corresponding to the third sub-block is located before the first processing unit corresponding to the first sub-block and adjacent to the first processing unit. The first sub-block may refer to any sub-block of the plurality of sub-blocks corresponding to the first tensor. The first processing unit is a processing unit corresponding to the first sub-block.

[0105] For example, the control logic unit 1101 is used to read the encoded data group corresponding to the first sub-block from the Local Memory to transfer the encoded data group corresponding to the first sub-block to the first processing unit, and to read the first scale factor corresponding to the first sub-block from the Vector RegisterFile to transfer the first scale factor corresponding to the first sub-block to the first processing unit.

[0106] In one example, the control logic unit 1011 is used to transfer the encoded data group corresponding to the sub-block from the first storage unit 103 to the target processing unit according to the second access granularity, so as to realize the transfer of the elements corresponding to the sub-block from the first storage unit 103 to the target processing unit. The second access granularity can be used to indicate the amount of data read from the first storage unit 103 by the control logic unit 1011 each time. The embodiment of the present application does not limit the second access granularity, such as the second access granularity includes at least one of the following: 32B (Byte), 64B, 96B. Optionally, the second access granularity can be an integer multiple of 32B.

[0107] For example, when the second access granularity is 32B, if each coded data group is 8 bits, the control logic unit 1011 is used to read 32 coded data groups from the first storage unit 103 each time until the tensor data of the first tensor is read. That is, the control logic unit 1011 can be used to assign each continuous 32 coded data groups to a processing unit (including the target processing unit), and when the continuous 32 coded data groups correspond to the same sub-block, the continuous 32 coded data groups can be determined as the coded data groups corresponding to the sub-block.

[0108] In the embodiment of the present application, the coded data group corresponding to the sub-block may only include the coded data corresponding to each element in the sub-block.

[0109] In one example, when the first tensor is represented by multiple encoded data groups, the processing unit is used to, for any encoded data group in the multiple encoded data groups, in order from low to high bits, treat every s bits in the encoded data group as an element, where s is the second bit. The second bit is the total number of bits corresponding to the second data format.

[0110] Exemplarily, the processing unit is used to, when the first data format is FP8, if the second data format is FP6, take the lower 6 bits in each coded data group as an element, that is, the last 6 bits of data in each coded data group constitute one element of coded data, then the 6 bits of data included in each coded data group corresponding to the sub-block can constitute the coded data corresponding to the sub-block. The data format supported by the processing unit in the tensor computing engine 101 is the first data format.

[0111] For example, when the first sub-block corresponds to 32 consecutive coded data groups, if one of the 32 consecutive coded data groups is: 00111011, the processing unit can determine 111011 as the coded data corresponding to an element, and then combine the coded data of the 32 elements to obtain the coded data corresponding to the first sub-block.

[0112] The coded data group corresponding to the sub-block may include the coded data corresponding to the sub-block as well as the coded data of the sub-block adjacent to the sub-block. Exemplarily, the processing unit is used to treat the upper 4 digits in each coded data group as one element and the lower 4 digits in each coded data group as another element when the first data format is FP8 and the second data format is FP4, that is, the first 4 bits of data in each coded data group constitute the coded data of one element, and the last 4 bits of data in each coded data group constitute the coded data of another element, then the 4 bits of data corresponding to the first half of the number of coded data groups can constitute the coded data corresponding to the sub-block, and the 4 bits of data corresponding to the second half of the number of coded data groups can constitute the coded data of the sub-block adjacent to the sub-block.

[0113] For example, when the first sub-block corresponds to 32 consecutive coded data groups, if one of the 32 consecutive coded data groups is: 11111011, the processing unit can determine 1111 (the first 4 bits of data) as the coded data corresponding to one element, and determine 1011 (the last 4 bits of data) as the coded data corresponding to another element, and then determine the coded data of the first 16 elements as the coded data corresponding to the first sub-block, and determine the coded data of the last 16 elements as the coded data corresponding to the third sub-block.

[0114] The embodiment of the present application supports identifying the encoded data of the element from the encoded data group to realize tensor calculation in the second data format, which is conducive to expanding the application scope of the processor. In addition, when the second data format is FP4, under the same second access granularity, the control logic unit 1101 can carry more elements each time, thereby effectively reducing the occupancy of transmission resources.

[0115] Optionally, when the coded data group corresponding to the first sub-block includes only the coded data corresponding to the first sub-block, the control logic unit 1101 is used to transfer the first scale factor corresponding to the first sub-block from the second storage unit 104 to the first processing unit. When the coded data group corresponding to the first sub-block includes both the coded data corresponding to the first sub-block and the coded data corresponding to the third sub-block, the control logic unit 1101 is used to transfer both the first scale factor corresponding to the first sub-block and the third scale factor corresponding to the third sub-block from the second storage unit 104 to the first processing unit.

[0116] 2. The tensor computing engine 101 is used to perform a logical operation on the elements in any sub-block in the first tensor according to the scale factor corresponding to the sub-block to obtain an operation result.

[0117] Performing a logical operation on the elements in the sub-block may refer to performing a logical operation on the coded data group corresponding to the sub-block. Optionally, before performing a logical operation on the coded data group corresponding to the sub-block, the tensor calculation engine 101 is further configured to identify the coded data of the sub-block from the coded data group corresponding to the sub-block, and then perform a logical operation on the coded data of the sub-block to obtain a calculation result.

[0118] Exemplarily, after the first processing unit obtains the coded data group corresponding to the first sub-block, the first processing unit is used to perform a logical operation on the coded data group corresponding to the first sub-block according to the first scale factor to obtain an operation result corresponding to the first sub-block.

[0119] For example, when the coded data group corresponding to the first sub-block only includes the coded data corresponding to the first sub-block, the first processing unit is used to perform a logical operation on the coded data corresponding to the first sub-block according to the first scaling factor to obtain an operation result corresponding to the first sub-block.

[0120] In the case where the coded data group corresponding to the first sub-block includes both the coded data corresponding to the first sub-block and the coded data corresponding to the third sub-block, the first processing unit is used to perform a logical operation on the coded data corresponding to the first sub-block and the coded data corresponding to the third sub-block according to the first proportional factor corresponding to the first sub-block and the third proportional factor corresponding to the third sub-block, so as to obtain the operation results corresponding to the first sub-block and the third sub-block, that is, the first processing unit takes the coded data corresponding to the first sub-block and the coded data corresponding to the third sub-block as a whole as input, performs a logical operation, but when the proportional factors are involved, the processing is performed based on the respective proportional factors.

[0121] Optionally, in the process of performing the calculation of the first tensor, the operation involving the scale factor includes at least one of the following: quantization and index alignment. Exemplarily, in the process of quantizing the encoded data corresponding to the sub-block, for any sub-block, the encoded data corresponding to the sub-block may be quantized according to the scale factor corresponding to the sub-block; in the process of index alignment, the index alignment may be performed on the operation result according to the scale factors corresponding to the sub-blocks involved.

[0122] Index alignment refers to making tensors of different dimensions consistent by expanding their dimensions so that corresponding mathematical operations can be performed. For example, when the dimensions of two tensors are inconsistent, in order to perform operations such as addition and multiplication, the tensor with lower dimension needs to be aligned with the tensor with higher dimension by expanding its dimensions.

[0123] In one example, before performing logical operations on the encoded data group corresponding to the sub-block, the tensor calculation engine 101 is used to quantize the elements in the sub-block according to the scale factor corresponding to the sub-block to obtain the quantized elements in the sub-block; and perform logical operations on the quantized elements in the sub-block to obtain the operation results.

[0124] The elements in the quantized sub-block may be a coded data group corresponding to the quantized sub-block, and the quantized coded data group corresponding to the sub-block is used to represent the quantized elements in the sub-block. For each coded data group, the dynamic range of the coded data group is different from the dynamic range of the quantized coded data group corresponding to the coded data group. The logical operation is performed on the quantized elements in the sub-block, and the logical operation is performed on the quantized coded data group corresponding to the sub-block.

[0125] Among them, the coded data group corresponding to the quantized sub-block is actually the coded data corresponding to the quantized sub-block, so as to convert the coded data corresponding to the sub-block from the second data format to the first data format supported by the processing unit, and adjust the dynamic range of the coded data corresponding to the sub-block to the dynamic range supported by the processing unit, to obtain the quantized coded data to perform logical operations. Optionally, the processing unit may support quantization of different data formats such as FP6 (E2M3 and E3M2), FP8 (E4M3 and E5M2), FP4, etc., and the embodiments of the present application are not limited to this. For example, when the first data format is FP6, to realize the multiplexing of the processing unit of the FP8 data format, it is necessary to convert the coded data in the FP6 data format to the coded data in the FP8 data format.

[0126] Exemplarily, when the encoded data group corresponding to the first sub-block only includes the encoded data corresponding to the first sub-block, the first processing unit is used to quantize the encoded data corresponding to the first sub-block according to the first scaling factor to obtain the quantized encoded data corresponding to the first sub-block, and perform a logical operation on the quantized encoded data corresponding to the first sub-block to obtain the operation result corresponding to the first sub-block.

[0127] In the case where the coded data group corresponding to the first sub-block includes both the coded data corresponding to the first sub-block and the coded data corresponding to the third sub-block, the first processing unit is used to quantize the coded data corresponding to the first sub-block according to the first scale factor corresponding to the first sub-block to obtain the quantized coded data corresponding to the first sub-block, and to quantize the coded data corresponding to the third sub-block according to the third scale factor corresponding to the third sub-block to obtain the quantized coded data corresponding to the third sub-block; perform logical operations on the quantized coded data corresponding to the first sub-block and the quantized coded data corresponding to the third sub-block to obtain the operation results corresponding to the first sub-block and the third sub-block respectively.

[0128] When quantizing each element in a sub-block, the corresponding scaling factor of the sub-block is shared. In this way, quantization is performed by the scaling factor of the sub-block granularity, which is beneficial to reduce the precision loss of data, thereby improving the accuracy of the operation result, and further beneficial to improving the calculation accuracy of the processor.

[0129] In one example, the tensor computing engine 101 is used to perform logical operations on the quantized elements in the sub-block to obtain a preliminary operation result; based on the scale factor corresponding to the sub-block, the exponent of the preliminary operation result is adjusted to obtain the operation result.

[0130] Among them, performing logical operations on the quantized elements in the sub-block can be performing logical operations on the quantized encoded data group corresponding to the sub-block. The preliminary calculation result refers to the calculation result whose exponent is not adjusted to the target data format. The target data format can be set and adjusted according to actual needs, such as the target data format includes at least one of the following: FP32, FP16, FP8. For example, the tensor calculation engine 101 is used to adjust the exponent of the preliminary calculation result to the exponent corresponding to the FP32 data format based on the scale factor, and the calculation result can be obtained, that is, the calculation result is represented by a 32-bit floating point number.

[0131] Exemplarily, in the case where the coded data group corresponding to the first sub-block only includes the coded data corresponding to the first sub-block, the first processing unit is used to adjust the index of the preliminary operation result corresponding to the first sub-block according to the first proportional factor to obtain the operation result corresponding to the first sub-block. In the case where the coded data group corresponding to the first sub-block includes both the coded data corresponding to the first sub-block and the coded data corresponding to the third sub-block, the first processing unit is used to adjust the index of the preliminary operation results corresponding to the first sub-block and the third sub-block respectively according to the first proportional factor corresponding to the first sub-block and the third proportional factor corresponding to the third sub-block to obtain the operation results corresponding to the first sub-block and the third sub-block respectively.

[0132] In one example, the tensor computing engine 101 is used to sum the scale factor corresponding to the sub-block and the exponent of the preliminary operation result to obtain a first sum value; determine the exponent of the operation result based on the first sum value; and obtain the operation result based on the exponent of the operation result.

[0133] Exemplarily, when the encoded data group corresponding to the first sub-block only includes the encoded data corresponding to the first sub-block, the first processing unit is used to sum the proportional factor corresponding to the first sub-block and the exponent of the preliminary operation result corresponding to the first sub-block to obtain a first sum value corresponding to the first sub-block; determine the first sum value corresponding to the first sub-block as the exponent of the operation result of the first sub-block; and obtain the operation result of the first sub-block based on the exponent of the operation result of the first sub-block.

[0134] The exponent of the preliminary operation result is adjusted based on the scale factor of the sub-block granularity to obtain the operation result, which is conducive to further reducing the precision loss of data, thereby improving the accuracy of the operation result.

[0135] In one example, when the data format supported by the processing unit in the tensor computing engine 101 is the first data format and the input dimension of the processing unit is the first dimension, if the data format of the first tensor is the second data format, the processing unit is related to the quotient of the first digit corresponding to the first data format and the second digit corresponding to the second data format and the first dimension for the input dimension of the first tensor.

[0136] The input dimension of the processing unit is used to indicate the maximum amount of data that the processing unit can process simultaneously, that is, the number of encoded data. Optionally, the product between the quotient of the first digit corresponding to the first data format and the second digit corresponding to the second data format and the first dimension is the input dimension of the processing unit for the first tensor.

[0137] Exemplarily, taking the first dimension as 32, when the first data format is FP8, if the second data format is FP4, the input dimension of the processing unit for the first tensor is 8 / 4*32=64, that is, the processing unit can process 64 consecutive elements in the first tensor at a time.

[0138] After determining that the second data format is FP4, since the 32B coded data group includes 64 elements of coded data, the control logic unit 1011 can send the 32B coded data group to the processing unit with an input dimension of 32, so that the processing unit can perform logical operations on the 64 elements of the coded data to obtain the operation result. When the second data format is FP4, the embodiment of the present application can process 64 elements simultaneously through a single processing unit with an input dimension of 32, effectively improving the computing efficiency of the processor for data.

[0139] When the first data format is FP8, if the second data format is FP6, the input dimension of the processing unit for the first tensor is 8 / 6*32=32, that is, the processing unit can process 32 consecutive elements in the first tensor each time.

[0140] After determining that the second data format is FP6, since the 32B coded data group includes 32 elements of coded data, the control logic unit 1011 can send the 32B coded data group to the processing unit with an input dimension of 32, so that the processing unit can perform a logical operation on the 32 elements of the coded data to obtain an operation result. In the case where the second data format is FP6, the embodiment of the present application can avoid the architectural modification of the processing unit by reusing the processing unit that supports the FP8 data format (input dimension is 32), thereby helping to reduce the development cost of the processor.

[0141] In one example, reference Figure 2 The tensor computing engine 101 also includes: a reuse buffer, which can be used to store data that may be reused in the tensor computing process (hereinafter referred to as duplicate data), so that the tensor computing engine 101 can directly obtain the duplicate data from the reuse buffer without repeatedly reading the duplicate data from the first storage unit 103 or the second storage unit 104, which is conducive to reducing the reading of duplicate data, thereby reducing processor power consumption.

[0142] For example, reference Figure 2 , the tensor computing engine 101 may include at least one of the following reuse caches: a first reuse cache for storing repeated data of matrix A, a second reuse cache for storing repeated data of matrix B, a third reuse cache for storing repeated data of matrix C, and a fourth reuse cache for storing scaling factors.

[0143] The matrix A, matrix B and matrix C are used to refer to three matrices in implementing the A*B+C operation. The fourth reuse cache can be used to temporarily store each scale factor corresponding to the first tensor without the tensor calculation engine 101 repeatedly reading the scale factor from the second storage unit 104.

[0144] Optionally, the tensor computing engine 101 may also be used to store the calculation results output by each processing unit in the second storage unit 104 .

[0145] To summarize, the technical solution provided by the embodiments of the present application supports the tensor computing engine to perform logical operations on the elements included in any sub-block in the tensor according to the scaling factor of the sub-block granularity (that is, each sub-block in the tensor corresponds to a scaling factor, rather than the entire tensor corresponds to a scaling factor), thereby avoiding the problem of data precision loss caused by using the scaling factor of the tensor granularity. Even when processing low-precision tensors (that is, the encoded data group of each element in the tensor is constructed in a low-precision data format, such as the total number of bits corresponding to the low-precision data format is less than 8), the present application implements the scaling factor based on the sub-block granularity to avoid data precision loss, thereby accurately performing low-precision tensor calculations, which is beneficial to expanding the scope of application of the processor to the data format and improving the accuracy of the processor's data processing.

[0146] In some embodiments, when the processing unit is used to perform a multiplication-accumulation operation, the processing unit is used to perform a multiplication-accumulation operation on the quantized elements in the first sub-block of the first tensor and the quantized elements in the first sub-block of the second tensor to obtain a preliminary operation result. The second tensor and the first tensor are two different tensors, the first sub-block of the first tensor corresponds to the first sub-block of the second tensor, the quantized elements in the first sub-block of the first tensor are quantized based on the scale factor corresponding to the first sub-block of the first tensor, and the quantized elements in the first sub-block of the second tensor are quantized based on the scale factor corresponding to the first sub-block of the second tensor.

[0147] For example, reference Figure 6 In the case where the first processing unit 601 is used to perform multiplication and accumulation operations, the process of obtaining the above preliminary operation results may include the following content.

[0148] 1. The first processing unit 601 is used to quantize the encoded data group corresponding to the first sub-block according to the first scale factor of the first sub-block in the first tensor to obtain the quantized encoded data group corresponding to the first sub-block, and quantize the encoded data group corresponding to the second sub-block according to the second scale factor of the first sub-block in the second tensor to obtain the quantized encoded data group corresponding to the second sub-block.

[0149] The second tensor and the first tensor are two different tensors. For example, the first tensor may refer to a matrix A in a matrix multiplication operation, where the matrix A includes elements A0, A1, ..., and the second tensor may refer to a matrix B in a matrix multiplication operation, where the matrix B includes elements B0, B1, .... Optionally, the data format of the matrix A and the data format of the matrix B are the second data format described above, such as FP6 or FP4.

[0150] For example, the first processing unit 601 identifies 32 consecutive elements of coded data from the coded data group corresponding to the first sub-block in the first tensor, recorded as A0 to A31, and then quantizes A0 to A31 based on the first scale factor in the 0th stage pipeline to obtain quantized A0 to A31. The first processing unit 601 identifies 32 consecutive elements of coded data from the coded data group corresponding to the first sub-block in the second tensor, recorded as B0 to B31, and then quantizes B0 to B31 based on the second scale factor in the 0th stage pipeline to obtain quantized B0 to B31.

[0151] For ease of explanation, the embodiment of the present application is described by taking an example in which the coded data group corresponding to the first sub-block only includes the coded data corresponding to the first sub-block.

[0152] 2. The first processing unit 601 is used to perform a multiplication-accumulation operation on the quantized coded data group corresponding to the first sub-block in the first tensor and the quantized coded data group corresponding to the first sub-block in the second tensor to obtain a preliminary multiplication-accumulation operation result.

[0153] refer to Figure 6 In the pipeline from the 1st stage to the 6th stage, the first processing unit 601 is used to perform a multiplication-accumulation operation on the quantized A0 to A31 and the quantized B0 to B31 to obtain a preliminary multiplication-accumulation operation result. The preliminary multiplication-accumulation operation result is the preliminary operation result.

[0154] In the first stage pipeline, the first processing unit 601 is used to perform multiplication operations to obtain multiple multiplication results: quantized A0*quantized B0, quantized A1*quantized B1, ..., quantized A31*quantized B31.

[0155] In the second stage of the pipeline, the first processing unit 601 is used to perform addition operations to obtain multiple addition results; in the third stage of the pipeline, the first processing unit 601 is used to obtain the largest exponent max_exp from the exponents of multiple multiplication results; in the fourth stage of the pipeline, the first processing unit 601 is used to align the exponents of multiple multiplication results based on the largest exponent max_exp, such as right-shifting the mantissas of other multiplication results based on the largest exponent max_exp; in the fifth and sixth stages of the pipeline, the first processing unit 601 is used to continue to perform addition operations to obtain preliminary multiplication-accumulation operation results. The preliminary multiplication-accumulation operation result is obtained by accumulating multiple multiplication results, and the preliminary multiplication-accumulation operation result can be expressed as follows:

[0156] Psum(A0*B0+A1*B1+…+A31*B31).

[0157] In some embodiments, when the processing unit is used to perform multiplication and accumulation operations, the above-mentioned preliminary operation result is obtained by accumulating multiple multiplication results; the processing unit is used to obtain the largest exponent from the exponents of the multiple multiplication results; the largest exponent, the proportional factor corresponding to the first sub-block of the first tensor, and the proportional factor corresponding to the first sub-block of the second tensor are added to obtain a first sum.

[0158] For example, reference Figure 6 The first processing unit 601 is used to adjust the exponent of the preliminary multiplication-accumulation operation result based on the first scale factor corresponding to the first sub-block of the first tensor and the second scale factor corresponding to the first sub-block of the second tensor to obtain the multiplication-accumulation operation result.

[0159] In the seventh stage of the pipeline, the first processing unit 601 is used to add the maximum exponent max_exp, the first scale factor, and the second scale factor to obtain a first sum; determine the exponent of the multiplication-accumulation operation result based on the first sum; and obtain the multiplication-accumulation operation result based on the exponent of the multiplication-accumulation operation result. The multiplication-accumulation operation result is the above operation result.

[0160] For example, the exponential of the result of a multiply-accumulate operation can be expressed as follows:

[0161] max_exp+(scaling_factor_a1-127)+(scaling_factor_b1-127);

[0162] Among them, scaling_factor_a1 is the first scaling factor, scaling_factor_b1 is the second scaling factor, and the sum of max_exp, scaling_factor_a1 and scaling_factor_b1 is the first sum.

[0163] In one example, when there is a matrix C, before adding the multiplication-accumulation operation result to the matrix C, the exponent of the multiplication-accumulation operation result is aligned with the exponent of the matrix C to obtain the multiplication-accumulation operation result after the index alignment, and then the multiplication-accumulation operation result after the index alignment is added to the matrix C to obtain the operation result of A*B+C. Optionally, the data format of the matrix C includes FP32.

[0164] Figure 6 The "cov" in the table indicates format conversion (such as converting data formats such as FP8, FP6, or FP4 into internal data formats), "mul" indicates multiplication, "add" indicates addition, "align" indicates exponent alignment, and "Norm" indicates normalization. Optionally, the data format of the result of the operation of A*B+C can also be FP16 data format or FP32 data format.

[0165] To sum up, the technical solution provided in the embodiment of the present application, when obtaining a low-precision matrix, supports the processing unit to perform multiplication and accumulation operations of the low-precision matrix according to the scaling factor of the sub-block granularity, so that the multiplication and accumulation operations of the low-precision matrix can be performed accurately, which is beneficial to expand the application scope of the processor for data formats and improve the processor's processing accuracy of data.

[0166] In some embodiments, the process of implementing tensor transmission and tensor storage by the tensor storage engine 102 is described by taking the first tensor as an example. Figure 7 , the embodiments of the present application may also include the following contents.

[0167] 1. The tensor storage engine 102 is used to move the scale factor set from the first storage unit 103 to the second storage unit 104 after moving the scale factor set from the third storage unit 105 to the first storage unit 103 .

[0168] The third storage unit 105 stores a set of scaling factors for each tensor to be processed (including the first tensor). Optionally, the scaling factors corresponding to each tensor are arranged in the third storage unit 105 in the same manner as Figure 3 The arrangement shown is the same, such as the arrangement of the scale factors corresponding to the first tensor in the third storage unit 105 is the same as the arrangement of the sub-blocks corresponding to the first tensor in the third storage unit 105.

[0169] In one example, when it is determined that there is a need to obtain a scale factor set of the first tensor, the tensor storage engine 102 is used to read the scale factor set from the third storage unit 105 to store the scale factor set in the first storage unit 103, and then move the scale factor set from the first storage unit 103 to the second storage unit 104 according to the LMA (Load Memory Address). The LMA may be used to indicate at least one of the following: a storage location of the scale factor set in the first storage unit 103, and a storage location of the scale factor set in the second storage unit 104.

[0170] Exemplarily, the tensor storage engine 102 is used to read the scale factor set from the Global Memory, store the scale factor set in the Local Memory, and then move the scale factor set from the Local Memory to the Vector Register through the LMA.

[0171] In an example, the processor further includes a load store unit (LSU), which is used to move the scale factor set from the third storage unit 105 to the second storage unit 104 .

[0172] When it is determined that there is a need to obtain the scale factor set of the first tensor, the tensor storage engine 102 is used to directly move the scale factor set from the third storage unit 105 to the second storage unit 104 through the LSU (Load Store Unit).

[0173] Exemplarily, the tensor storage engine 102 is used to directly move the scale factor set from the Global Memory to the Vector Register File or the Scale register through the LSU, which is not limited in the embodiment of the present application.

[0174] 2. The tensor storage engine 102 is used to read the first tensor from the third storage unit 105 and store the first tensor in the first storage unit 103.

[0175] Reading the first tensor may be reading the tensor data of the first tensor. Optionally, the third storage unit 105 stores the encoded data of each element included in the first tensor, and the encoded data of each element is used to represent the value of the element. For example, when the second data format is FP4, the encoded data of each element consists of 4 bits of data to indicate the value of the element. When the second data format is FP6, the encoded data of each element consists of 6 bits of data to indicate the value of the element.

[0176] In one example, when the third storage unit 105 includes multiple encoded data for representing a first tensor, the tensor storage engine 102 is used to pack multiple groups of encoded data in sequence according to the lowest access granularity of the third storage unit 105 to obtain multiple transmission data groups, and the total number of bits of the transmission data groups is greater than the total number of bits of the encoded data; and read the transmission data groups in sequence according to the first access granularity of the tensor storage engine 102.

[0177] The embodiment of the present application does not limit the minimum access granularity of the third storage unit 105, which can be set and adjusted according to actual usage requirements. The minimum access granularity is used to align tensor data.

[0178] Optionally, the tensor storage engine 102 is used to obtain the lowest access granularity of the third storage unit 105 and the least common divisor between the second digit of the second data format corresponding to the encoded data; divide the least common divisor by the second digit of the second data format to obtain a packing dimension; and pack multiple encoded data in sequence according to the packing dimension to obtain multiple transmission data groups.

[0179] Exemplarily, the tensor storage engine 102 is used to pack every two consecutive encoded data into one transmission data group when the second data format is FP4 and the minimum access granularity of the third storage unit 105 is 8 bits (ie, 1 byte).

[0180] Among them, the lowest common divisor of 4 and 8 is 8, so the packing dimension is 8 / 4=2. By packing each continuous 8 bits of data into 1 byte, a transmission data group is obtained, so that each transmission data group can carry 2 elements of encoded data. Under the same computing power of FP8 (that is, each transmission data group can carry 2 elements of encoded data), 1 / 2 of the bandwidth can be saved.

[0181] Optionally, the tensor storage engine 102 is used to pack every 128 consecutive encoded data into 96 transmission data groups when the second data format is FP6 and the minimum access granularity of the third storage unit 105 is 8 bits (ie, 1 byte).

[0182] Among them, the lowest common divisor of 6 and 8 is 24, so the packing dimension is 24 / 6=4, that is, every 4 consecutive encoded data are packed into 3 transmission data groups, so that every 128 consecutive encoded data can be packed into 96 transmission data groups. By packing every 128 consecutive encoded data into 96 transmission data groups, that is, packing every consecutive 128*6bit data into 96 bytes, 96 transmission data groups are obtained. In this way, every 3 consecutive transmission data groups can carry 4 elements of encoded data, which can save 1 / 4 of the bandwidth under the same computing power of FP8.

[0183] The embodiment of the present application does not limit the above-mentioned first access granularity, which can be set and adjusted according to actual use requirements. For example, the first access granularity can be an integer multiple of 32B, such as including 32B. Exemplarily, when the first access granularity is 32B, the tensor storage engine 102 reads 32B of data from the third storage unit 105 each time, that is, 32 transmission data groups.

[0184] The embodiments of the present application can reduce the bandwidth required for transmission of high-precision tensors by compactly transmitting the encoded data corresponding to low-precision tensors, thereby saving transmission overhead.

[0185] In one example, the tensor storage engine 102 is used to restore multiple encoded data according to the second bit number of the second data format after the transmission data group is read; fill the multiple encoded data respectively according to the minimum access granularity of the first storage unit 103 to obtain multiple encoded data groups; and store the multiple encoded data groups in the first storage unit 103 in turn.

[0186] The second number of the second data format is the total number of bits (or total number of bits) of the second data format.

[0187] Exemplarily, when the second data format is FP4, the tensor storage engine 102 is used to sequentially split each transmission data group into two encoded data, thereby obtaining a plurality of encoded data corresponding to the first tensor. When the second data format is FP6, the tensor storage engine 102 is used to sequentially split each three consecutive transmission data groups into four encoded data, thereby obtaining a plurality of encoded data corresponding to the first tensor.

[0188] The embodiment of the present application does not limit the minimum access granularity of the first storage unit 103, which can be set and adjusted according to actual use requirements. Exemplarily, the minimum access granularity of the first storage unit 103 is greater than or equal to 8 bits.

[0189] Optionally, when the total number of bits corresponding to the lowest access granularity of the first storage unit 103 is greater than or equal to the total number of bits of the coded data, the target number of setting values ​​is filled before the coded data to obtain the coded data group. The target number may be the difference between the total number of bits corresponding to the lowest access granularity of the first storage unit 103 and the total number of bits of the coded data. The setting value may be set and adjusted according to actual usage requirements, such as 1, 0, etc. The total number of bits of the coded data is the second digit of the second data format.

[0190] Exemplarily, when the minimum access granularity of the first storage unit 103 is 8 bits, the tensor storage engine 102 is used to fill 2 bits before each encoded data when the second data format is FP6, so as to obtain multiple encoded data groups; when the second data format is FP4, for every two consecutive encoded data, the first encoded data is filled before the second encoded data, so as to obtain multiple encoded data groups.

[0191] For example, refer to Figure 8 , taking the second data format as FP6, the minimum access granularity of the first storage unit 103 as 8 bits, and the first access granularity as 32 bytes as an example. For the encoded data of 128 elements in the FP6 data format, the tensor storage engine 102 packs the encoded data (4×6 bits) of each consecutive 4 elements into 3 transmission data groups (3×8 bits), and each consecutive 24 transmission data groups (24 bytes) correspond to 32 encoded data (32×6 bits), and each consecutive 96 transmission data groups (96×8 bytes) correspond to 128 encoded data (128×6 bytes). The tensor storage engine 102 reads 32 bytes of data (i.e., 32 transmission data groups) from the Global Memory each time. After 3 reads, the tensor storage engine 102 can move all the encoded data corresponding to the 128 elements to the Local Memory.

[0192] For every 24 consecutive transmission data groups, the tensor storage engine 102 can obtain 32 encoded data 801 according to the total number of bits corresponding to the FP6 data format. The total number of bits of each encoded data 801 is 6 bits. For each encoded data 801, the tensor storage engine 102 fills 2 bits before the encoded data 801 to obtain the encoded data group 802. The total number of bits of each encoded data group 802 is 8 bits.

[0193] In a feasible example, the above tensor transmission and tensor storage can also be implemented by an ALU (Arithmetic Logic Unit), such as a BIT ALU that supports bit as a basic component unit.

[0194] Optionally, refer to Figure 7 The processor also includes an asynchronous barrier unit (Async Barrier). After all encoded data groups corresponding to the first tensor are read, the tensor storage engine 102 will update the asynchronous barrier unit to trigger the interruption of the tensor storage engine 102 to read data from the third storage unit 105.

[0195] Optionally, refer to Figure 7 , the tensor storage engine 102 can read information such as the dimension (Block_dim) and the position (Block_pos) of the tensor data from the register stack based on the operation instruction for the first tensor, and read the tensor descriptor (Tensor_info) of the tensor data from the constant memory 106, and the tensor descriptor is used to indicate the data format, arrangement mode, etc. of the elements in the tensor data. The tensor storage engine 102 determines the coordinate address of the tensor data in the third storage unit 105 based on the dimension, position, data format, arrangement mode, etc. of the tensor data, and the tensor storage engine 102 reads the tensor data from the third storage unit 105 based on the coordinate address of the tensor data, thereby realizing the transmission of the tensor data. Wherein, in the case where the first tensor belongs to a certain tensor, the position of the tensor data can be represented by coordinates to indicate the position of the first tensor in the tensor.

[0196] Optionally, after the tensor data is stored in the first storage unit 103, the data structure information and address information corresponding to the tensor data are also updated accordingly. For example, the data structure information and address information corresponding to the tensor data are updated from the encoded data for indicating the element to the encoded data group for indicating the encoded data group. That is, based on the data structure information and address information before the update, the coordinate address of the encoded data of the element in the third storage unit 105 can be determined, and the updated data structure information and address information can determine the coordinate address of the encoded data group in the first storage unit 103.

[0197] In summary, the technical solution provided by the embodiment of the present application can effectively reduce the bandwidth required for tensor transmission by tightly compressing and transmitting low-precision tensors, thereby saving transmission overhead. By tightly compressing and storing low-precision tensors, the space required for tensor storage can be effectively reduced, thereby saving storage overhead, especially for tensor data in FP4 data format.

[0198] In some embodiments, reference Fig. 9, taking the processor implemented as a graphics processor as an example. The graphics processor 900 may include multiple processor clusters (PCs) 901 (e.g., 4), each processor cluster 901 may include multiple processor execution engines (PXs) 902 (e.g., 4), and each processor execution engine 902 may include multiple processors (Ps) 903 (e.g., 2).

[0199] The multiple processor execution engines 902 in each processor cluster 901 share an L2 cache 904 (ie, the third storage unit 105 ). The main memory of the graphics processor 900 is implemented as an L3 cache 905 .

[0200] refer to Fig.10 Each processor 903 may include: a tensor computing engine 9031, a tensor storage engine 9032, an L1 cache 9033 (ie, the above-mentioned first storage unit 103) and a register 9034 (ie, the above-mentioned second storage unit 104).

[0201] The L2 cache 904 stores a plurality of encoded data corresponding to the first tensor, each encoded data is generated by an element in the first tensor according to the FP6 data format, that is, each encoded data includes 6 bits of data.

[0202] The L2 cache 904 also stores multiple scaling factors corresponding to the first tensor, each scaling factor corresponds to a sub-block in the first tensor, each sub-block includes 32 consecutive elements in the first tensor, and the scaling factor is used to adjust the dynamic range of the 32 elements in the same sub-block.

[0203] The tensor storage engine 9032 is used to pack every 128 consecutive encoded data into 96 transmission data groups (i.e., forming 3 groups of 32B transmission data) according to the minimum access granularity of 8 bits of the L2 cache 904, and obtain multiple transmission data groups corresponding to the first tensor; according to the first access granularity of 32B, the multiple transmission data groups corresponding to the first tensor are read from the L2 cache 904 to the L1 cache 9033.

[0204] The tensor storage engine 9032 is used to restore multiple transmission data groups corresponding to the first tensor into multiple encoded data corresponding to the first tensor according to the total number of bits corresponding to the FP6 data format; fill 2 bits before each encoded data to obtain multiple encoded data groups corresponding to the first tensor, each encoded data group includes 8 bits of data.

[0205] The tensor storage engine 9032 is used to read multiple scale factors corresponding to the first tensor from the L2 cache 904 to the register 9034.

[0206] The tensor computing engine 9031 is used to read multiple encoded data groups corresponding to the first tensor from the L1 cache 9033, and read multiple proportional factors corresponding to the first tensor from the register 9034; for any sub-block in the first tensor, according to the proportional factor corresponding to the sub-block, a logical operation is performed on the encoded data group corresponding to the sub-block to obtain an operation result.

[0207] For example, the tensor computing engine 9031 is used to read 32 encoded data groups from the L1 cache 9033 each time according to the second access granularity of 32B. The tensor computing engine 9031 can identify the encoded data corresponding to 32 elements (i.e., one sub-block) from the 32 encoded data groups, and read the proportional factors corresponding to the 32 elements from the register 9034. Based on the proportional factors corresponding to the 32 elements, logical operations are performed on the encoded data corresponding to the 32 elements to obtain operation results.

[0208] In some embodiments, taking a matrix as an example, the instructions involved in the tensor computing engine are shown in Table 1 below.

[0209] Table 1

[0210]

[0211]

[0212] For matrix a, the tensor calculation engine can read from register a_reg or from Local Memory: a_desc. For matrix b, it can read from Local Memory: b_desc.

[0213] In the embodiment of the present application, a_dtype and b_dtype increase support for FP4 and FP8, and a new bit field of the scaling factor is added (corresponding to scaling_a / scaling_b), and scaling_a / scaling_b increases the reading of the scaling factor for the sub-block granularity.

[0214] To summarize, the technical solution provided by the embodiment of the present application, when obtaining low-precision tensors in the FP6 data format, supports the tensor computing engine to perform logical operations on the encoded data group corresponding to any sub-block according to the scaling factor of the sub-block granularity, so that the tensor computing engine can realize low-precision tensor calculation while avoiding the loss of data precision caused by using the scaling factor of the tensor granularity, thereby being able to accurately perform low-precision tensor calculations, which is beneficial to expanding the scope of application of the graphics processor to the data format and improving the accuracy of the graphics processor in processing data.

[0215] The following is an embodiment of the method of the present application. For details not disclosed in the embodiment of the method of the present application, please refer to the above embodiment.

[0216] Please refer to Fig.11 , which shows a flowchart of a tensor calculation method applied to a processor provided in a possible implementation of the present application. The components of the processor can be found in the description of the above embodiment. Fig.11 As shown, the method may include the following steps:

[0217] Step 1101 , the tensor computing engine reads a first tensor from a first storage unit, and reads a scale factor set from a second storage unit.

[0218] In one example, a first tensor is stored in a first storage unit.

[0219] In one example, a second storage unit stores a set of scale factors corresponding to the first tensor, the scale factor set includes multiple scale factors, each scale factor corresponds to a sub-block in the first tensor, each sub-block includes at least one element in the first tensor, and the scale factor is used to adjust the dynamic range of each element in the same sub-block.

[0220] Step 1102: For any sub-block in the first tensor, the tensor calculation engine performs a logical operation on the elements in the sub-block according to the scale factor corresponding to the sub-block to obtain a calculation result.

[0221] In one example, the tensor computing engine quantizes the elements in the sub-block according to the scaling factors corresponding to the sub-block to obtain the quantized elements in the sub-block; and performs logical operations on the quantized elements in the sub-block to obtain operation results.

[0222] In one example, the tensor computing engine performs logical operations on the quantized elements in the sub-block to obtain a preliminary operation result; based on the scaling factor corresponding to the sub-block, the exponent of the preliminary operation result is adjusted to obtain the operation result.

[0223] In one example, the tensor computing engine sums the scale factor corresponding to the sub-block and the exponent of the preliminary operation result to obtain a first sum value; determines the exponent of the operation result based on the first sum value; and obtains the operation result based on the exponent of the operation result.

[0224] In one example, a tensor computing engine includes: a processing unit array, the processing unit array includes multiple processing units for performing logical operations; when the processing unit is used to perform multiplication and accumulation operations, the processing unit performs multiplication and accumulation operations on the quantized elements in the first sub-block of the first tensor and the quantized elements in the first sub-block of the second tensor to obtain preliminary operation results; wherein the second tensor and the first tensor are two different tensors, the first sub-block of the first tensor corresponds to the first sub-block of the second tensor, the quantized elements in the first sub-block of the first tensor are quantized based on the proportional factor corresponding to the first sub-block of the first tensor, and the quantized elements in the first sub-block of the second tensor are quantized based on the proportional factor corresponding to the first sub-block of the second tensor.

[0225] In one example, a preliminary operation result is obtained by accumulating multiple multiplication results; a processing unit is used to obtain a maximum exponent from the exponents of the multiple multiplication results; the maximum exponent, a scaling factor corresponding to a first sub-block of a first tensor, and a scaling factor corresponding to a first sub-block of a second tensor are added to obtain a first sum.

[0226] In one example, a tensor computing engine includes: a control logic unit and a processing unit array, the processing unit array including a plurality of processing units for performing logic operations;

[0227] The control logic unit moves, for any sub-block in the first tensor, elements in the sub-block from the first storage unit to a target processing unit corresponding to the sub-block among the multiple processing units, and moves a scaling factor corresponding to the sub-block from the second storage unit to the target processing unit.

[0228] In one example, the processor also includes: a third storage unit, in which a scale factor set is stored; the processor also includes a tensor storage engine, which moves the scale factor set from the first storage unit to the second storage unit after moving the scale factor set from the third storage unit to the first storage unit; or, the processor also includes a loading storage unit, which moves the scale factor set from the third storage unit to the second storage unit.

[0229] In one example, when a first tensor includes m×n sub-blocks, a tensor storage engine or a load storage unit stores the scaling factors corresponding to the sub-blocks in the 1st to the mth rows in each column in the order of the 1st to the nth columns in a second storage unit, where the index of the scaling factor is the register index corresponding to the scaling factor in the second storage unit, and the index of the scaling factor is used to read the scaling factor, and m and n are positive integers.

[0230] In one example, when the data format supported by the processing unit in the tensor computing engine is the first data format and the input dimension of the processing unit is the first dimension, if the data format of the first tensor is the second data format, the processing unit is related to the quotient of the first digit corresponding to the first data format and the second digit corresponding to the second data format and the first dimension for the input dimension of the first tensor.

[0231] In one example, the second data format includes at least one of the following: 4-bit floating point number FP4, 6-bit floating point number FP6; when the first data format is FP8, if the second data format is FP4, the input dimension of the processing unit for the first tensor is 64; if the second data format is FP6, the input dimension of the processing unit for the first tensor is 32.

[0232] In one example, a first storage unit includes multiple encoded data groups for representing a first tensor, each encoded data group includes encoded data generated according to a second data format for at least one element in the first tensor; for any encoded data group among the multiple encoded data groups, the processing unit treats every s bits in the encoded data group as an element in order from low to high, where s is the second bit.

[0233] In one example, when the first data format is FP8, if the second data format is FP4, the processing unit takes the upper 4 digits of each encoded data group as an element, and takes the lower 4 digits of each encoded data group as another element; if the second data format is FP6, the processing unit takes the lower 6 digits of each encoded data group as an element.

[0234] In one example, a third storage unit includes multiple encoded data for representing a first tensor, each encoded data is used to represent an element in the first tensor; the tensor storage engine packs the multiple encoded data in sequence according to the lowest access granularity of the third storage unit to obtain multiple transmission data groups, and the total number of bits of the transmission data groups is greater than the number of bits of the encoded data; the tensor storage engine reads the transmission data groups in sequence according to the first access granularity of the tensor storage engine.

[0235] In one example, a first storage unit includes multiple encoded data groups for representing a first tensor, each encoded data group includes encoded data generated according to a second data format for at least one element in the first tensor; after the transmission data group is read, the tensor storage engine restores multiple encoded data according to a second bit of the second data format; fills the multiple encoded data respectively according to the lowest access granularity of the first storage unit to obtain multiple encoded data groups; and stores the multiple encoded data groups in the first storage unit in sequence.

[0236] In one example, when the second data format is FP6, the tensor storage engine fills 2 bits before each encoded data to obtain multiple encoded data groups; or, when the second data format is FP4, for every two consecutive encoded data, the first encoded data is filled before the second encoded data to obtain multiple encoded data groups.

[0237] In one example, a tensor storage engine obtains the lowest access granularity of a third storage unit and the lowest common divisor between a second digit of a second data format corresponding to the encoded data; divides the lowest common divisor by the second digit of the second data format to obtain a packing dimension; and packs multiple encoded data in sequence according to the packing dimension to obtain multiple transmission data groups.

[0238] In one example, the tensor storage engine packs every 2 consecutive encoded data into 1 transmission data group when the second data format is FP4 and the minimum access granularity is 8; or, when the second data format is FP6 and the minimum access granularity is 8, packs every 128 consecutive encoded data into 96 transmission data groups.

[0239] To summarize, the technical solution provided by the embodiments of the present application supports the tensor computing engine to perform logical operations on the elements included in any sub-block in the tensor according to the scaling factor of the sub-block granularity (that is, each sub-block in the tensor corresponds to a scaling factor, rather than the entire tensor corresponds to a scaling factor), thereby avoiding the problem of data precision loss caused by using the scaling factor of the tensor granularity. Even when processing low-precision tensors (that is, the encoded data group of each element in the tensor is constructed in a low-precision data format, such as the total number of bits corresponding to the low-precision data format is less than 8), the present application implements the scaling factor based on the sub-block granularity to avoid data precision loss, thereby accurately performing low-precision tensor calculations, which is beneficial to expanding the scope of application of the processor to the data format and improving the accuracy of the processor's data processing.

[0240] Please refer to Fig.12 , which shows a simplified structural block diagram of a computer device provided by an embodiment of the present application. The computer device 1200 can be any electronic device with data calculation, processing and storage functions.

[0241] The computer device 1200 includes a processor 1201 and a memory 1202 .

[0242] The processor 1201 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1201 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1201 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1201 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1201 may also include an AI processor for processing computing operations related to machine learning.

[0243] The memory 1202 may include one or more computer-readable storage media, which may be non-transitory. The memory 1202 may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1202 is used to store a computer program, which is configured to be executed by one or more processors.

[0244] Those skilled in the art will understand that Fig.12 The structure shown in the figure does not constitute a limitation on the computer device 1200, and the computer device 1200 may include more or less components than those shown in the figure, or combine some components, or adopt a different arrangement of components.

[0245] In some embodiments, a chip product is also provided, the chip product includes the processor described above. Optionally, the chip product may be a GPU chip product, the processor described above is a GPU, and the GPU chip product includes the GPU described above. Optionally, the above chip product may be implemented as a graphics card, and the graphics card includes the processor described above, such as a GPU.

[0246] In some embodiments, a computer device is also provided, the computer device including the processor described above. Optionally, the computer device may be a personal computer, a workstation, a game console, and some mobile devices (such as a tablet computer, a smart phone, etc.), or a vehicle-mounted terminal device, a smart home device, a smart TV, an intelligent robot, etc., or a server, a server cluster, an artificial intelligence computing cluster, a cloud computing cluster, etc., wherein the artificial intelligence computing cluster may also be referred to as an intelligent computing cluster or a smart computing cluster, which is not limited in this application.

[0247] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0248] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A processor, characterized in that: The processor includes: a tensor computing engine, a first storage unit and a second storage unit; The first storage unit stores a first tensor; The second storage unit stores a scale factor set corresponding to the first tensor, the scale factor set includes a plurality of scale factors, each scale factor corresponds to a sub-block in the first tensor, each sub-block includes at least one element in the first tensor, and the scale factor is used to adjust the dynamic range of each element in the same sub-block; The tensor computing engine is used to read the first tensor from the first storage unit and read the scale factor set from the second storage unit; for any sub-block in the first tensor, perform logical operations on the elements in the sub-block according to the scale factor corresponding to the sub-block to obtain an operation result.

2. The processor according to claim 1, characterized in that The tensor computation engine is used to: quantizing the elements in the sub-block according to the scale factor corresponding to the sub-block to obtain the quantized elements in the sub-block; A logical operation is performed on the quantized elements in the sub-block to obtain the operation result.

3. The processor according to claim 2, characterized in that The tensor computation engine is used to: Performing logic operations on the quantized elements in the sub-blocks to obtain preliminary operation results; Based on the scale factor corresponding to the sub-block, the exponent of the preliminary operation result is adjusted to obtain the operation result.

4. The processor according to claim 3, characterized in that The tensor computation engine is used to: Summing the scale factor corresponding to the sub-block and the exponent of the preliminary operation result to obtain a first sum value; Determining an exponent of the operation result based on the first sum value; Based on the exponent of the operation result, the operation result is obtained.

5. The processor according to claim 4, characterized in that The tensor computing engine includes: a processing unit array, the processing unit array including a plurality of processing units for performing logic operations; In the case where the processing unit is used to perform a multiplication-accumulation operation, the processing unit is used to: Performing a multiplication-accumulation operation on the quantized elements in the first sub-block of the first tensor and the quantized elements in the first sub-block of the second tensor to obtain the preliminary operation result; Among them, the second tensor and the first tensor are two different tensors, the first sub-block of the first tensor corresponds to the first sub-block of the second tensor, the quantized elements in the first sub-block of the first tensor are quantized based on the scale factor corresponding to the first sub-block of the first tensor, and the quantized elements in the first sub-block of the second tensor are quantized based on the scale factor corresponding to the first sub-block of the second tensor.

6. The processor according to claim 5, characterized in that The preliminary operation result is obtained by accumulating multiple multiplication results; the processing unit is used for: Obtaining a maximum exponent from the exponents of the plurality of multiplication results; The maximum exponent, the scaling factor corresponding to the first sub-block of the first tensor, and the scaling factor corresponding to the first sub-block of the second tensor are added to obtain the first sum value.

7. The processor according to any one of claims 1 to 6, characterized in that: The tensor computing engine includes: a control logic unit and a processing unit array, wherein the processing unit array includes a plurality of processing units for performing logic operations; The control logic unit is used to transfer, for any sub-block in the first tensor, the elements in the sub-block from the first storage unit to a target processing unit corresponding to the sub-block among the multiple processing units, and to transfer the scaling factor corresponding to the sub-block from the second storage unit to the target processing unit.

8. The processor according to any one of claims 1 to 6, characterized in that: The processor further includes: a third storage unit, wherein the scale factor set is stored in the third storage unit; The processor further includes a tensor storage engine, and the tensor storage engine is used to move the scale factor set from the first storage unit to the second storage unit after moving the scale factor set from the third storage unit to the first storage unit; or, The processor further includes a load storage unit, and the load storage unit is used to move the scale factor set from the third storage unit to the second storage unit.

9. The processor according to claim 8, characterized in that The tensor storage engine or the load storage unit is used to: In the case where the first tensor includes m×n sub-blocks, the scaling factors corresponding to the sub-blocks in the 1st to the mth rows in each column are stored in the second storage unit in sequence from the 1st column to the nth column, the index of the scaling factor is the register index corresponding to the scaling factor in the second storage unit, the index of the scaling factor is used to read the scaling factor, and m and n are positive integers.

10. The processor according to any one of claims 1 to 6, characterized in that: In the case where the data format supported by the processing unit in the tensor computing engine is the first data format, and the input dimension of the processing unit is the first dimension, If the data format of the first tensor is the second data format, the processing unit is related to the input dimension of the first tensor, the quotient of the first digit corresponding to the first data format and the second digit corresponding to the second data format, and the first dimension.

11. The processor according to claim 10, characterized in that The second data format includes at least one of the following: 4-bit floating point number FP4, 6-bit floating point number FP6; When the first data format is FP8, if the second data format is FP4, the input dimension of the processing unit for the first tensor is 64; if the second data format is FP6, the input dimension of the processing unit for the first tensor is 32.

12. The processor according to claim 10, characterized in that The first storage unit includes a plurality of encoded data groups for representing the first tensor, each encoded data group includes encoded data generated according to the second data format for at least one element in the first tensor; and the processing unit is used to: For any coded data group among the multiple coded data groups, every s bits in the coded data group are taken as an element in order from low bits to high bits, where s is the second bit.

13. The processor according to claim 12, characterized in that The processing unit is used for: In the case where the first data format is FP8, If the second data format is FP4, the upper 4 digits in each coded data group are taken as one element, and the lower 4 digits in each coded data group are taken as another element; If the second data format is FP6, the lower 6 digits in each encoded data group are taken as an element.

14. The processor according to claim 8, characterized in that The third storage unit includes a plurality of encoded data for representing the first tensor, each encoded data is used to represent an element in the first tensor; and the tensor storage engine is used to: According to the lowest access granularity of the third storage unit, the plurality of coded data are sequentially packaged to obtain a plurality of transmission data groups, wherein the total number of bits of the transmission data groups is greater than the number of bits of the coded data; The transmission data groups are read sequentially according to a first access granularity of the tensor storage engine.

15. The processor according to claim 14, characterized in that The first storage unit includes a plurality of encoded data groups for representing the first tensor, each encoded data group includes encoded data generated according to a second data format for at least one element in the first tensor; and the tensor storage engine is used to: After the transmission data group is read, recovering the plurality of encoded data according to the second bit number of the second data format; Filling the plurality of coded data respectively according to the lowest access granularity of the first storage unit to obtain the plurality of coded data groups; The plurality of encoded data groups are stored in the first storage unit in sequence.

16. The processor according to claim 15, characterized in that The tensor storage engine is used to: When the second data format is FP6, 2 bits are filled before each coded data to obtain the multiple coded data groups; or, When the second data format is FP4, for every two consecutive encoded data, the first encoded data is padded before the second encoded data to obtain the multiple encoded data groups.

17. The processor according to claim 15, characterized in that The tensor storage engine is used to: Obtaining a lowest common divisor between a minimum access granularity of the third storage unit and a second digit of a second data format corresponding to the encoded data; Dividing the least common divisor by the second digit of the second data format to obtain a packing dimension; According to the packing dimension, the multiple encoded data are packed in sequence to obtain the multiple transmission data groups.

18. The processor according to claim 17, characterized in that The tensor storage engine is used to: When the second data format is FP4 and the minimum access granularity is 8, every two consecutive encoded data are packaged into one transmission data group; or, When the second data format is FP6 and the minimum access granularity is 8, every 128 consecutive encoded data are packaged into 96 transmission data groups.

19. A chip product, characterized in that: The chip product comprises the processor according to any one of claims 1 to 18.

20. A computer device, characterized in that: The computer device comprises a processor as claimed in any one of claims 1 to 18.

21. A tensor calculation method applied to a processor, characterized in that: The processor includes: a tensor computing engine, a first storage unit and a second storage unit; The first storage unit stores a first tensor; The second storage unit stores a scale factor set corresponding to the first tensor, the scale factor set includes a plurality of scale factors, each scale factor corresponds to a sub-block in the first tensor, each sub-block includes at least one element in the first tensor, and the scale factor is used to adjust the dynamic range of each element in the same sub-block; The method comprises: The tensor computation engine reads the first tensor from the first storage unit, and reads the set of scaling factors from the second storage unit; For any sub-block in the first tensor, the tensor calculation engine performs a logical operation on the elements in the sub-block according to a scale factor corresponding to the sub-block to obtain an operation result.

Images