Processor, chip product, computer equipment and tensor processing method

Abstract

The invention discloses a processor, a chip product, computer equipment and a tensor processing method, and relates to the technical field of processors. The processor comprises a tensor calculation engine and a target storage unit, wherein the tensor calculation engine comprises a tensor calculation unit and a first buffer area; the tensor calculation unit executes logical operation on the tensor data to obtain an operation result of a first sequence, and the operation result of the first sequence is in a first data format; the tensor calculation unit is also used for caching the operation result of the first sequence to the first buffer area; the tensor calculation unit is also used for converting the operation result of the first sequence cached in the first buffer area into an operation result of a second sequence according to a second data format; and storing the operation result of the second sorting to the target storage unit. In this way, inverse conversion does not need to be carried out after the tensor storage engine executes data format conversion, and the influence of the storage mode adopted by the operation result of the first sequence in the storage unit does not need to be considered.

Description

Technical Field

[0001] This application relates to the field of processor technology, and in particular to a processor, chip product, computer device, and tensor processing method. Background Technology

[0002] Currently, common processors generally include TME (Tensor Memory Engine) and TCE (Tensor Compute Engine). TME is responsible for accessing memory units and accelerating the storage and retrieval of tensor data, while TCE is responsible for tensor computation.

[0003] TCE (Tencent Processing Unit) performs logical operations on tensor data to obtain the first operation result. This first result includes the operation results of each element within the tensor, distributed across different registers. When the first operation result needs to be used in subsequent operations, the processor's SIMD (Single Instruction Multiple Data) unit performs data format conversion on the operation results of each element and packages the converted operation results from different registers to obtain a packaged first operation result. This packaged first operation result is then used in subsequent operations. However, because the order of the converted operation results is adjusted during the packaging process, the order of the operation results corresponding to each element in the packaged first operation result becomes inconsistent with the order of the operation results corresponding to each element in the first operation result itself. This is detrimental to subsequent operations (i.e., adjustment is required before subsequent operations).

[0004] To address the issue of inconsistent arrangement order, related technologies employ column transformation to restore the order of the results corresponding to each element in the first packaged operation result back to their original order. Column transformation is essentially the inverse of data format conversion. However, this approach becomes highly complex when the first operation result is stored row-wise. Summary of the Invention

[0005] This application provides a processor, a chip product, a computer device, and a tensor processing method. The technical solution is as follows: According to one aspect of the embodiments of this application, a processor is provided, the processor comprising: a tensor computation engine and a target storage unit, the tensor computation engine comprising a tensor computation unit and a first buffer; The tensor computation unit is used to perform logical operations on tensor data to obtain a first sorted operation result, wherein the first sorted operation result is in a first data format. The tensor computation unit is also used to store the operation result of the first sorting into the first buffer; The tensor calculation unit is further configured to convert the first sorting operation result cached in the first buffer into a second sorting according to the second data format to obtain the second sorting operation result; and store the second sorting operation result into the target storage unit.

[0006] According to one aspect of the embodiments of this application, a chip product is provided, the chip product including the processor described above.

[0007] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including the processor described above.

[0008] According to one aspect of the embodiments of this application, a tensor processing method applied to a processor is provided, the processor comprising: a tensor computation engine and a target storage unit, the tensor computation engine comprising a tensor computation unit and a first buffer; the method comprising: The tensor computation unit performs logical operations on the tensor data to obtain a first sorted operation result, which is in a first data format. The tensor computation unit stores the result of the first sorting operation into the first buffer; The tensor calculation unit converts the first sorting operation result cached in the first buffer into a second sorting operation result according to the second data format. The tensor computation unit stores the results of the second sorting operation into the target storage unit.

[0009] The technical solutions provided in this application embodiment may have the following beneficial effects: By adding a first buffer to the tensor computation engine, the tensor computation unit reorders the computation results, ensuring that the data arrangement after format conversion conforms to the input of the next matrix multiplication, or meets the requirement of larger length for external storage. The computation results converted to the second sort can be directly stored in the target storage unit without needing to perform a reverse conversion after the tensor storage engine performs the data format conversion. Since the tensor computation unit directly reorders the computation results of the first sort, there is no need to consider the impact of the storage method used for the computation results of the first sort in the storage unit on the complexity of the data format conversion. Attached Figure Description

[0010] Figure 1This is a schematic diagram of a processor provided in one possible implementation of this application; Figure 2 This is a schematic diagram of a matrix operation method provided in one possible implementation of this application; Figure 3 This is a schematic diagram of a tensor computation unit provided in one possible implementation of this application; Figure 4 This is a schematic diagram of matrix multiplication decomposition provided in one possible implementation of this application; Figure 5 This is a schematic diagram of a graphics processor provided in one possible implementation of this application; Figure 6 This is a schematic diagram of the processor in a graphics processor provided in one possible implementation of this application; Figure 7 This is a schematic diagram of the TCE data source and connection provided in one possible implementation of this application; Figure 8 This is a schematic diagram of the TCE data source and connection provided in another possible implementation of this application; Figure 9 This is a schematic diagram of the arrangement of matrix D in the register provided in one possible implementation of this application; Figure 10 This is a schematic diagram of the arrangement of FP8 matrix A in the register provided in one possible implementation of this application; Figure 11 This is a schematic diagram of data exclusion in the post-processing of the D matrix provided in one possible implementation of this application; Figure 12 This is a schematic diagram of TME column transformation provided in one possible implementation of this application; Figure 13 This is a schematic diagram of a possible implementation of the data arrangement transformation provided in this application; Figure 14 This is a flowchart of a tensor processing method applied to a processor provided in one possible implementation of this application; Figure 15 This is a schematic diagram of a data format conversion method for the computation result provided in one possible implementation of this application; Figure 16 This is a schematic diagram of a data format conversion method for the computation result provided in another possible implementation of this application; Figure 17 This is a simplified structural block diagram of a computer device provided in one possible implementation of this application. Detailed Implementation

[0011] Before describing the embodiments of this application, the relevant terms involved in this application will be explained.

[0012] Tensor: A general term and extension of concepts such as scalar, vector, and matrix. It is a fundamental data structure in the field of machine learning. It can be understood as a multidimensional array that includes at least one element. For example, a 0-dimensional tensor is a single number, a 1-dimensional tensor is equivalent to a vector, a 2-dimensional tensor corresponds to a matrix, and a 3-dimensional tensor is a cube. This application does not limit the dimension of the tensor; for example, the dimension of a tensor can also be 4 dimensions or higher.

[0013] Tensors can store more information about objects, such as the height, color, depth, and width of an image, the frame rate, color, height, and width of a video, and the time domain, frequency domain, and fundamental frequency of audio.

[0014] Tensor computation mainly refers to the computations and processing performed on tensors in the fields of deep learning and machine learning. These computations typically involve operations such as matrix multiplication and convolution, and are key steps in the training and inference of machine learning models.

[0015] Floating-point number type: This is a data type used to represent real numbers, allowing the decimal point to float, thus enabling the representation of fractions, decimals, and certain irrational numbers. Real numbers represented using the floating-point number type are called floating-point numbers.

[0016] Floating-point types include at least one of the following: single-precision floating-point (float), double-precision floating-point (double), high-precision floating-point (decimal), and floating-point types such as FP16, BF16 (Brain Floating-Point 16-bit), FP8 (Floating-Point 8-bit), FP6 (Floating-Point 6-bit), and FP4 (Floating-Point 4-bit).

[0017] Floating-point numbers are typically represented using scientific notation. For example, FP16: half-precision floating-point, a 16-bit floating-point representation, meaning it uses a 16-bit value to represent a real number, including 1 sign bit, 5 exponent bits, and 10 mantissa bits. FP8: uses an 8-bit value to represent a real number, including a 4-bit exponent and a 3-bit exponent (E4M3), or a 5-bit exponent and a 2-bit mantissa (E5M2), designed to improve computational efficiency by reducing the number of bits used to represent the data.

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0019] Please refer to Figure 1 The diagram illustrates a processor provided in one possible implementation of this application. The processor 100 may include a Tensor Compute Engine (TCE) 110 and a Tensor Memory Engine (TME) 120. The Tensor Compute Engine 110 includes a tensor computation unit 111 and a first buffer 112, and the Tensor Memory Engine 120 includes a first storage unit 121 and a second storage unit 122.

[0020] The processor 100 mentioned above can refer to a processor that supports tensor processing, such as processor 100 supporting at least one of the following operations for tensors: transfer, storage, and computation. For example, the tensor storage engine 120 in processor 100 supports tensor transfer, each storage unit in processor 100 supports tensor storage, and the tensor computation engine 110 in processor 100 supports tensor computation. Tensor transfer can refer to the transfer of tensor data corresponding to a tensor, tensor storage can refer to the storage of tensor data corresponding to a tensor, and tensor computation can refer to the computation of tensor data corresponding to a tensor. Tensor data is used to represent tensors. Optionally, processor 100 can be implemented as a physical circuit for processing tensor data.

[0021] In some embodiments, processor 100 may be implemented as a processing unit (PU) supporting tensor processing. This application embodiment does not limit the processor 100; processor 100 may be a completely new design or an improvement upon an existing processor. Exemplarily, processor 100 may be implemented as at least one of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), General-Purpose Computing on Graphics Processing Units (GPGPU), Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Tensor Processing Unit (TPU), Field Programmable Gate Array (FPGA), Tensor Core, or Neural Processing Unit (NPU).

[0022] In one example, the processor 100 supports processing data formats including at least one of the following: floating-point numbers in data formats such as FP8, FP16, BF16, FP6, FP4, FP32, and TF32 (Tensor-Floating 32-bit, 32-bit mixed floating-point numbers); fixed-point numbers in data formats such as U8 and S8; and integers in data formats such as INT8 (8-bit integer) and INT16 (16-bit integer).

[0023] For example, processor 100 supports the processing of tensor data in FP8 data format. Tensor data refers to data used to represent a tensor, which can consist of encoded data generated for each element in the tensor, with each element represented by a set of encoded data. For example, tensor data in FP8 data format includes: encoded data generated for each element in the tensor according to the FP8 data format, where each set of encoded data can include 8 bits of data, and each 8-bit data can determine the value of a floating-point number (i.e., an element). The order of the encoded data corresponding to each element is consistent with the order of the elements in the tensor.

[0024] Optionally, each set of encoded data may consist of a set of binary numbers, such as 0s and 1s. For example, in the FP6 data format, each set of encoded data may include 6 bits of data, such as 6 binary numbers, which indicate the sign, exponent, and mantissa of a floating-point number (i.e., an element). In the FP4 data format, each set of encoded data may include 4 bits of data, such as 4 binary numbers, which indicate the sign, exponent, and mantissa of a floating-point number (i.e., an element).

[0025] Tensor computation engine 110 refers to the computation engine in processor 100 used to perform tensor computations. For example, tensor computation engine 110 can be implemented as a physical circuit for performing tensor computations. Optionally, the arithmetic logic supported by tensor computation engine 110 includes at least one of the following: matrix multiplication, convolution, multiply-accumulate, and addition. For example, tensor computation engine 110 can be used to perform multiply-accumulate operations on tensor data to achieve tensor multiply-accumulate operations and obtain the tensor multiply-accumulate operation result. This application embodiment does not limit the number of tensor computation engines 110 included in processor 100; they can be set and adjusted according to actual usage requirements. Tensor computation engine 110 is used to read tensor data to be computed from first storage unit 121 and store the computed tensor data in second storage unit 122.

[0026] Tensor storage engine 120 refers to the engine in processor 100 used to perform tensor transfer and tensor storage. The engine can be implemented as a hardware system for processing data, such as physical circuitry. For example, tensor storage engine 120 can be implemented as physical circuitry used to perform tensor transfer and tensor storage.

[0027] In one example, the first storage unit 121 and the second storage unit 122 are implemented as different storage units within the tensor storage engine 120. For instance, the first storage unit 121 is implemented as the LMS (Least Mean Squares) within the tensor storage engine 120, and the second storage unit 122 is implemented as a register (REG) within the tensor storage engine. In this case, the tensor storage engine 120 also needs to transfer the computed tensor data stored in the second storage unit 122 to the first storage unit 121, so that the tensor computation engine 110 can read this portion of tensor data from the first storage unit 121 to perform subsequent logical operations.

[0028] In another example, the first storage unit 121 and the second storage unit 122 are implemented as the same storage unit in the tensor storage engine 120. Exemplarily, both the first storage unit 121 and the second storage unit 122 are implemented as register REG in the tensor storage engine 120. In this case, the tensor computation engine 110 reads the tensor data to be computed from the first storage unit 121, performs logical operations on it, and then stores the computed tensor data back into the first storage unit 121, overwriting the tensor data previously stored in the first storage unit 121. At this time, the tensor storage engine 120 does not need to transfer the storage address of the computed tensor data; the tensor computation engine 110 can subsequently continue to call that portion of the tensor data from the first storage unit 121.

[0029] Optionally, a data transmission channel (such as a data bus) is provided between the first storage unit 121 and the second storage unit 122 in the tensor storage engine 120, and the tensor storage engine 120 can realize the transmission of tensor data through the data transmission channel. For example, the tensor storage engine 120 can be used to move tensor data in FP8 data format stored in the first storage unit 121 to the second storage unit 122 through the data transmission channel.

[0030] In some embodiments, the first storage unit 121 can be implemented as an on-chip cache, referred to as local memory, to temporarily store tensor data, thereby improving the speed at which the tensor calculation engine 110 acquires tensor data. Optionally, the second storage unit 122 can also be implemented as a register, such as a vector register / vector register file, to temporarily store tensor data, thereby further improving the speed at which the tensor calculation engine 110 acquires tensor data (the data read speed of a register is much faster than the data read speed of memory).

[0031] In some embodiments, the processor 100 further includes a third storage unit 130. The third storage unit 130 may be implemented as a shared cache of at least one tensor computation engine 110, referred to as global memory, which may support at least one tensor computation engine 1102 to access its data, such as by supporting multiple tensor computation engines 110 to access tensor data, so as to realize the sharing of tensor data among multiple tensor computation engines 110.

[0032] For example, the second storage unit 122 can be implemented as a Level 1 cache in the processor 100, i.e., an L1 cache, which is the cache layer closest to the tensor calculation engine 110. It typically has a small storage space but a very fast data read speed. The first storage unit 121 can be implemented as a Level 2 cache in the processor 100, i.e., an L2 cache. In this case, the first storage unit 121 is located between the L1 cache (i.e., the second storage unit 122) and the main memory of the processor 100 (such as an L3 cache). It is used to store more tensor data so as to provide more backups in case of L1 cache misses. The storage space of the L2 cache is larger than that of the L1 cache, but the data read speed of the L2 cache is smaller than that of the L1 cache.

[0033] Optionally, the processor 100 may further include a fourth storage unit 140. The tensor computation engine 110 may also be used to store the computation results of tensor data into the fourth storage unit 140. Optionally, the fourth storage unit 140 may be implemented as a register file to store computation results corresponding to different threads, computation results corresponding to different tensor computation engines, or computation results of different processing units in the tensor computation engine. This application embodiment does not limit this.

[0034] In some embodiments, the result of tensor operations can also be used as input for subsequent operations. For example, in the field of deep learning technology, for two connected model layers in a neural network model, the output data of the first model layer (i.e., the result of tensor operations) will be used as the input data of the second model layer. Therefore, it is necessary to convert the data format of the output data into the data format supported by the second model layer, and to package the converted output data into tensor computation engine 110 for subsequent operations.

[0035] Based on the above, the structure of the processor provided in the embodiments of this application has been understood. The technical problems solved by the embodiments of this application will be described next.

[0036] Tensor computation unit 111 is used to perform calculations on tensor data according to operational logic. For example, the tensor computation unit calculates D = A*B + C, where A, B, C, and D are four matrices. Matrix A has a dimension of M*K, matrix B has a dimension of K*N, matrix C has a dimension of M*N, and matrix D also has a dimension of M*N. Figure 2 The diagram shows a matrix where K equals 6. The value of each element in D is equal to the sum of the product of the corresponding row element of matrix A and the corresponding column element of matrix B, plus the element at the corresponding position in matrix C. For example, the element in the first row and first column of matrix D is equal to the sum of the product of the first row element of matrix A and the first column element of matrix B, plus the element in the first row and first column of matrix C.

[0037] The structure of the tensor computation unit 111 is as follows: Figure 3 As shown, the tensor computation unit 111 is a 2D array. Each DP in the array consists of multiple multiply-accumulators. The array dimension and the dimension in the DP can be determined according to various factors such as chip usage scenario, power consumption, area, bandwidth, storage and result accuracy.

[0038] like Figure 4As shown, the matrix multiplication to be calculated is typically large. To fully utilize each processing unit (MP) in the GPU, the matrix C / D needs to be decomposed into tiles, with each tile size represented as tile_M * tile_N. Each MP calculates one or more tiles. For each tile calculation, the MP needs to obtain the complete tile_M * K A matrix and tile_N * K B matrix. When K is large, it is impossible to completely move the corresponding A and / or B matrix data into the MP. Therefore, K needs to be split, and only tile_K size needs to be obtained and calculated at a time. Because the A and / or B matrix data are stored in GDDR, MP access will have a significant latency. To mask this latency and increase the utilization of the tensor computation engine 110, a multi-level buffering scheme can be adopted. Specifically, the data to be calculated is prefetched before the calculation begins. After obtaining the data, the calculation begins, and the data to be calculated in the next round is prefetched simultaneously. In this way, the calculation time and memory access time can be masked.

[0039] With processor as Figure 5 Taking the GPU shown as an example, a GPU 5000 consists of multiple compute cores (MPCs) 5100, each compute core 5100 consists of multiple compute clusters (MPXs) 5110, and two compute units (MPs) 5111 form an MPX 5110. Figure 6 As shown, a computation unit (MP) 5111 contains one or more complete sets of instruction execution and computation logic 610, and the figure shows four sets. Tensor computation engine (TCE) 110 is located within MP5111, and each TCE 110 contains four tensor computation units (ALU0~ALU3), and the four ALUs share one control logic (CTRL LOGIC).

[0040] In the above calculation process, data from matrix A and / or matrix B are continuously retrieved from GDDR and added to the MP, accumulating with the results of the previous round of calculation. Each tensor calculation engine's four ALUs need to work together to complete the calculation of one TILE. A feasible data source and ALU arrangement for a tensor calculation engine is as follows: Figure 7 As shown, the data of matrix A and / or matrix B comes from LMS (the internal storage unit of MP), and the data of matrix C and / or matrix D comes from the registers inside MPE. At the same time, in order to avoid the complexity of the connection between MPE registers and tensor calculation engine ALU, a one-to-one connection method is adopted.

[0041] In one use case, the output of one matrix multiplication serves as the input for another. If both operators need to write their results to GDDR while the other operator needs to read its result from GDDR as input, these memory access operations can be avoided by fusing the two operators, performing two consecutive matrix calculations within the same operator. For example, in Flashat attention, a QK matrix multiplication is performed, followed by a softmax operation, and then a matrix multiplication with matrix V to produce the final result.

[0042] Where Q, K, and V are three matrices. This is the transpose of the K matrix. It is the greatest common divisor.

[0043] If adopted Figure 7 The method requires writing the result of the QK matrix multiplication back to the LMS. During this process, the tensor computation engine 110 outputs the result of the QK matrix multiplication to a register within the MPE, and then transfers it to the LMS based on the LMA instruction (a special execution unit that accesses the LMS unit), which introduces additional operations. Therefore, a method is proposed... Figure 8 The connection method shown allows the result of QK matrix multiplication to also come from a register. Similarly, a one-to-one connection method is used. In addition to avoiding connection complexity, this also ensures that the output result is consistent with the connection of the K matrix, which facilitates operator fusion.

[0044] Returning to the above calculations of matrix D, Figure 8 The connection method shown means that the data of matrix A can be obtained directly from the register or from the LMS, the data of matrix B can be obtained from the LMS, and the data of matrix C can be obtained from the register. Then the calculation result is output to the register (that is, the data of matrix D can be written back to the register), overwriting the data originally stored in the register. In this way, the tensor storage engine does not need to transfer the data (it can be understood that the data of matrix D can be directly used as the input of the next matrix multiplication, that is, as the data of matrix A for the next matrix multiplication).

[0045] like Figure 3 As shown, the array size of the tensor computation engine 110 is 8x8, therefore the size of the output D matrix is ​​also 8x8. The GPU uses a SIMT or SIMD architecture, with each wave containing 32 threads, and the registers are vector-width structures of 32. Therefore, it is necessary to map the D matrix and the A matrix to the vector registers. One arrangement of the D matrix is ​​as follows: Figure 9As shown, each register is 32 bits in size. The 32 registers (Ra) for each wave are arranged in an 8x4 format. An 8x8 output requires two registers, Ra and Ra+1. For larger matrix multiplications, the number of registers can be increased accordingly. The tensor computation engine supports A / B matrix inputs of fp4, fp8, fp16, bf16, and tf32, and outputs a D matrix of fp32. The arrangement of the A matrix in the registers is as follows... Figure 10 As shown, the registers of each wave are still arranged in an 8x4 manner, and each thread's register can hold four fp8 elements, so each wave's register can hold 8x16 elements.

[0046] To generate a D matrix from matrix operations, post-processing of the elements in the D matrix is ​​required, such as format conversion or activation function operations (ReLU / GeLU / Softmax, etc.). The final output may be fp16, bf16, or fp8, depending on the requirements. Taking fp8 as an example, an fp32 element is converted into an fp8 data point. Each fp8 data point occupies only 8 bits of a 32-bit register. Therefore, four consecutive fp32 elements can be converted into four consecutive fp8 elements and merged into a single 32-bit register.

[0047] For example, refer to Figure 11 Matrix 1101 has a size of M*K, matrix 1102 has a size of K*N, and matrix 1103 is the result of the operation between matrices 1101 and 1102. For example, performing a matrix multiplication operation on matrices 1101 and 1102 yields matrix 1103. Matrix 1103 can be used to represent the result of the matrix multiplication operation between matrices 1101 and 1102, and its size is M*N. Each element in matrix 1103 (i.e., ...) represents the result of the matrix multiplication operation between matrices 1101 and 1102. Figure 11 The cell corresponding to matrix 1103 corresponds to a multiply-accumulate operation result, and the data format of the multiply-accumulate operation result is FP32. For example, the first element in the top left of matrix 1103 is the result of multiplying and accumulating the element in the first row of matrix 1101 (denoted as M0) and the element in the first column of matrix 1102 (denoted as N0).

[0048] The elements in matrix 1103 are stored in different registers. For example, the first four columns of elements in matrix 1103 are stored in register R0, the second four columns are stored in register R1, the third four columns are stored in register R2, and the last four columns are stored in register R3.

[0049] T0 to T31 are 32 threads started by the format conversion unit. These 32 threads are used to perform data format conversion and packing on the encoded data of each element in matrix 1103. Data format conversion refers to the process of changing the data format from one data format to another, and packing refers to the process of packing multiple data together to form continuous data. The format conversion unit can be implemented as at least one of the following: SIMD, SIMT (Single Instruction Multiple Thread), or a vector unit on the NPU.

[0050] Taking thread T0 as an example, thread T0 needs to perform data format conversion and packing on the first element in R0, the first element in R1, the first element in R2, and the first element in R3. For example, thread T0 can convert the data format of each element corresponding to T0 from FP32 to FP8, obtaining four elements in FP8 data format. Then, it packs the four elements in FP8 data format into a single register in sequence. That is, the four elements in FP32 data format form 32-bit encoded data, which includes four 8-bit encoded data, with each 8-bit encoded data corresponding to one element.

[0051] When sorted according to the columns of matrix 1103, the order of the elements in matrix 1103 is: N0 to N15. However, after processing by the format conversion unit (i.e., after data format conversion and packaging), the order of the elements changes to: N0, N4, N8, N12, N1, N5, N9, N13, N2, N6, N10, N14, N3, N7, N11, and N15. The order of the elements after processing is inconsistent with the order before processing. If the processed elements need to be used in subsequent calculations, the order of the elements after processing needs to be restored to be consistent with the order before processing before subsequent calculations can be performed. This will increase the complexity of subsequent calculations (e.g., if the original order needs to be readjusted for calculation).

[0052] The post-processed data has two possibilities: either it becomes the input for the next matrix multiplication, or it is directly output. However, after... Figure 11 After the format conversion shown, the data format is the same as... Figure 10 The formats shown are quite different, and the positions of different elements have been swapped, such as... Figure 11As shown, the data in column 4 is converted to the data in column 2 after format conversion. To better utilize memory bandwidth, the data length accessed by each thread should be as long as possible, typically greater than or equal to 4 bytes. For data lengths less than 4 bytes, memory utilization is relatively poor. If each thread uses one register and four threads write to GDDR, the data in columns 0, 4, 8, and 12 will be written to contiguous addresses in global memory, effectively transforming the original data arrangement. Therefore, the format-converted arrangement cannot be directly output to global memory.

[0053] To address the issue of inconsistent arrangement order, related technologies propose performing column transformations on the B matrix. This column transformation is performed via TME during the transport of the B matrix, essentially the reverse process of the format conversion command's arrangement change. After matrix multiplication, the format conversion restores the original arrangement. For example... Figure 12 As shown, taking fp16 as an example, for the first row of result elements in matrix 1201, T0 can pack the result elements of column N0 and column N1 into a 32-bit compressed data, T1 can pack the result elements of column N2 and column N3 into a 32-bit compressed data, T2 can pack the result elements of column N4 and column N5 into a 32-bit compressed data, T3 can pack the result elements of column N6 and column N7 into a 32-bit compressed data, and so on. The order of the first row of result elements after being packed is: the result elements of column N0, column N1, column N2, ..., column N15. This is the same as the order of the encoded data of column N0 to column N15 in the first storage unit 121.

[0054] However, this approach has a drawback: when the B matrix is ​​stored row-wise, the column transformation method cannot be implemented (or its implementation complexity is too high). Furthermore, since the LMS stores the column-transformed B matrix, in some scenarios, such as during backpropagation of Flashattention, it is necessary to calculate the matrix row-wise and column-wise once each. Therefore, an additional LMS buffer is required to store the original data.

[0055] In addition, a method has been proposed in related technologies to modify the output of TCE. For example... Figure 13As shown, this data arrangement modifies the layout of fp16. Two adjacent elements (e.g., t0 and t1) are stored in two adjacent registers of each thread. This allows them to be directly used as input to the next matrix after formatting, while also maintaining a 4-byte data length when written out. However, this method does not work with input formats such as fp4 / fp8 / tf32.

[0056] To address the aforementioned issues, this application proposes a tensor processing method. A first buffer 112 is set in the TCE to cache the calculation result of the AB matrix multiplication in the first buffer 112. Then, the tensor calculation unit 111 converts the calculation result of the AB matrix multiplication into a second sort and stores it in the target storage unit.

[0057] Please refer to Figure 14 The diagram illustrates a flowchart of a tensor processing method provided in one embodiment of this application. This method can be applied to... Figure 1 In the processor shown, the method includes at least one of the following steps 1410 to 1440.

[0058] Step 1410: The tensor computation unit performs logical operations on the tensor data to obtain the first sorted operation result, which is in the first data format.

[0059] To facilitate understanding, the following will use the calculation process of D=A*B+C as an example to describe the technical solution of the embodiments of this application in detail.

[0060] In some embodiments, the tensor computation unit acquires tensor data in a third data format, which is the tensor data to be computed. The tensor data to be computed refers to the matrix to be computed, or the matrix data corresponding to one or more TILEs after being divided into multiple TILEs.

[0061] For example, the tensor data in the third data format is matrix A and matrix B. In some embodiments, the tensor data in the third data format can be the complete data of matrix A and / or matrix B, or it can be the data of matrix A and / or matrix B corresponding to one or more TILEs after D is divided into different TILEs.

[0062] In some embodiments, the third data format refers to the storage format of tensor data. The third data format can reflect the data type, bit depth, etc., of the tensor data. The tensor data to be computed is stored in the tensor storage engine in a third data format. For example, the third data format may be FP16, FP32, or BF16, etc.

[0063] In some embodiments, the third data format is the data format of the tensor data when performing logical operations on the tensor data. In some embodiments, the storage format of the tensor data to be calculated can be the same or different. For example, the data format of matrix A and the storage format of matrix B can be the same or different. For example, the storage format of matrix A and matrix B can both be FP16, or the storage format of matrix A can be FP16 and the storage format of matrix B can be FP32. In some embodiments, if the storage formats of the tensor data to be calculated are different, the tensor data to be calculated can be unified into tensor data of the third data format. For example, quantization or dequantization can be used to unify the tensor data to be calculated into tensor data of the third data format.

[0064] In some embodiments, the tensor computation unit obtains tensor data in a third data format from the tensor storage engine. In some embodiments, the tensor data in the third data format may be tensor data stored in the LMS (first storage unit) of the tensor storage engine, or tensor data stored in the registers (second storage unit) of the tensor storage engine.

[0065] In some embodiments, the tensor computation unit performs logical operations on tensor data in a third data format, such as performing cumulative multiplication and addition on matrix A and matrix B. In some embodiments, after performing logical operations on the tensor data in the third data format, the result obtained is the result of the first sorted operation. For example, after performing cumulative multiplication and addition on two FP16 tensor data, the result is obviously not 16 bits long enough to represent it. In this case, the FP32 format is needed to represent the result to ensure the accuracy of the result.

[0066] In some embodiments, after obtaining the result of the first sorting operation, the result of the first sorting operation can be converted into a third data format and then stored in the target storage unit for use in the next stage of operation. For example, after obtaining the FP32 operation result, it is converted into a combination of FP16 tensor data and a divisor. Although this will result in a loss of some data precision, it can reduce the data dimensionality and reduce the memory used to store the operation result.

[0067] Step 1420: The tensor computation unit caches the results of the first sorting operation into the first buffer.

[0068] In some embodiments, after the tensor computation unit calculates the result of the first sorting operation, it needs to store the result at an address before performing further processing. However, if the result is stored in the tensor storage engine, the tensor computation unit needs to read it from the tensor storage engine again. Therefore, a first buffer can be added to the tensor computation engine to cache the result of the first sorting operation. Then, the tensor computation unit can perform a format conversion on the result in the first buffer.

[0069] In some embodiments, the first buffer is a 32-column write-back buffer, for example, the first buffer is a 32-column buffer. In some embodiments, the result of the first sorting operation is stored column-wise in the first buffer.

[0070] Step 1430: The tensor computation unit converts the operation result of the first sorting cached in the first buffer into the second sorting to obtain the operation result of the second sorting.

[0071] In some embodiments, the result of the first sorting operation is reordered to obtain the result of the second sorting operation, which is also in the first data format. That is, the process of converting the result of the first sorting operation to the result of the second sorting operation does not involve format conversion. For example, as... Figure 15 As shown, T0-T3 in register R0 of the first sorting operation result are located in the first position of registers R1-R3 respectively after being converted to the second sorting operation result. Although the order of the elements has changed, the number of registers occupied by the elements has not changed, which shows that the format conversion of the first sorting operation result has not occurred.

[0072] Step 1440: The tensor computation unit stores the results of the second sorting operation into the target storage unit.

[0073] In some embodiments, after the format conversion is completed, the tensor computation unit stores the result of the second sorting operation into the target storage unit.

[0074] The technical solution provided in this application adds a first buffer to the tensor computation engine, allowing the tensor computation unit to reorder the computation results. This ensures that the data arrangement after format conversion conforms to the input of the next matrix multiplication, or meets the requirement for larger storage lengths. The computation results converted to the second sort can be directly stored in the target storage unit without needing to perform a reverse conversion after the tensor storage engine performs the data format conversion. Since the tensor computation unit directly reorders the computation results from the first sort, there is no need to consider the impact of the storage method used for the first sorted computation results in the storage unit on the complexity of the data format conversion.

[0075] Next, we will combine Figure 15 The examples shown illustrate the solutions of the embodiments of this application in detail. Figure 15 The diagram illustrates how, using FP8 as an example, the result of the first sorting operation is used as the input for the next matrix multiplication.

[0076] In some embodiments, the step of the tensor computation unit converting the operation result of the first sorting into the operation result of the second sorting may include the following step a.

[0077] Step a: The tensor computation unit rearranges the elements in the first sorting operation result based on the first parameter to obtain the second sorting operation result; wherein, the first parameter is used to indicate the conversion granularity between the first data format and the second data format.

[0078] In some embodiments, the first parameter is related to the target storage unit. In some embodiments, the target storage unit can be divided into a temporary storage unit and a global storage unit according to the different operation results. The temporary storage unit is used to store temporary tensor data generated during the operation; for example, the temporary storage unit can be implemented as described above. Figure 1 The first or second storage unit is shown. The global storage unit is used to store the tensor data obtained after the operation is completed, and is shared with all processors. For example, the global storage unit can be implemented as follows: Figure 1 The third storage unit is shown.

[0079] In some embodiments, the result of the first sorting operation may be as follows: Figure 9 The calculation result is shown. In some embodiments, the first parameter is related to the target storage unit. For example, if the target storage unit is a temporary storage unit, then the first parameter can be a smaller value. For example, if the target storage unit is a global storage unit, then the first parameter can be a larger value.

[0080] In some embodiments, the first parameter is related to the target storage unit, or in other words, the first parameter is related to the second data format, which is the data format after format conversion. It should be noted that the technical solution of this application does not involve format conversion, but merely reorders the results of the first sorting operation, and the resulting operation result is still in the first data format. In some embodiments, the first parameter is defined in the instructions of the tensor computation engine. For example, a first field is added to the TCE instruction, which is used to indicate the first parameter. For example, a bit field DST_INNER_STRIDE is added to the TCE instruction, which is used to indicate the first parameter.

[0081] In some embodiments, the first parameter is configured by the developer when designing the TCE.

[0082] In some embodiments, the first parameter is used to indicate the relationship between the number of bits in the first data format and the number of bits in the second data format. For example, the first parameter is used to indicate the multiple relationship between the number of bits in the first data format and the number of bits in the second data format. In some embodiments, the value range of the first parameter is {0, 2, 4, 8}. In some embodiments, if the value of the first parameter is 0, it indicates that the result of the first sorting operation is output according to the default sorting, or in other words, the result of the first sorting operation does not need to be reordered and can be directly output according to the result of the first sorting operation.

[0083] For example, the first data format is a 32-bit floating-point number FP32; When the first parameter is 8, the second data format is a 4-bit floating-point number FP4; or, When the first parameter is 4, the second data format is an 8-bit floating-point number FP8; or, When the first parameter is 2, the second data format is a 16-bit floating-point number FP16; or, When the first parameter is 0, the second data format is a 32-bit floating-point number FP32.

[0084] For example, Figure 15 In this case, the second data format is FP8, and the first parameter is 4. In some embodiments, the first parameter can also represent the rearrangement step size corresponding to the conversion of the first sorting result to the second sorting result. For example, if the rearrangement step size is 4, it means that after four consecutive elements in the first sorting result are converted to the second sorting result, they will each occupy the corresponding number of bits in four register addresses. Figure 15 In the first sorting operation result, T0-T3 in register R0 are respectively located in the first position of registers R1-R3 after being converted to the second sorting operation result.

[0085] In some embodiments, step a above can be implemented as at least one of steps a1 to a3.

[0086] Step a1: The tensor computation unit determines the number of bits in the second data format based on the number of bits in the first data format and the first parameter.

[0087] For example, the first parameter is the quotient of the number of bits in the first data format and the number of bits in the second data format. After determining the number of bits in the first data format and the first parameter, the number of bits in the second data format can be determined.

[0088] In some embodiments, the number of bits in the first data format, the first parameter, and the number of bits in the second data format are all known, in which case step a1 does not need to be performed. Step a2: The tensor computation unit determines the coordinate address of each column element in the second sorting operation result based on the number of bits in the second data format for each column element in the first sorting operation result.

[0089] In some embodiments, for the nth column element in the first sorting operation result, the coordinates corresponding to the nth column element in the second sorting operation result are determined based on the number of bits in the second data format, where n is an integer less than the number of the first column, and the number of the first column is the number of columns occupied by the first sorting operation result in the first buffer.

[0090] In some embodiments, the number of the first column is predefined or preconfigured. For example, the number of the first column may be preconfigured by the developer or defined according to a standard. For example, the number of the first column may be 32.

[0091] In some embodiments, the tensor computation unit determines a first intermediate parameter and a second intermediate parameter based on the value of n and the number of bits in the second data format. The first intermediate parameter is used to determine the register address corresponding to the element in the nth column in the second sorting operation result, and the second intermediate parameter is used to determine the bit position of the element in the nth column in the register address corresponding to the element in the second sorting operation result. Based on the value of n and the number of bits in the first data format, a third intermediate parameter is determined. The third intermediate parameter is used to determine the offset bit position corresponding to the rearrangement step size of the element in the nth column. Based on the first intermediate parameter, the second intermediate parameter, and the third intermediate parameter, the coordinate address corresponding to the element in the nth column in the second sorting operation result is determined.

[0092] For example, such as Figure 15 As shown, the result of the first sorting operation is stored in registers R0-R7. After reordering, it is still stored in registers R0-R7, but the positions of the elements in the result have changed. Therefore, it is necessary to determine the register address of each element in the result of the first sorting operation after it has been converted to the result of the second sorting operation. After determining the register address, it is also necessary to determine the bit position of the element in the current register address before the coordinate address of the element after the conversion to the second sorting can be determined.

[0093] In some embodiments, the tensor computation unit determines a third intermediate parameter based on the value of n and the number of bits in the second data format. The third intermediate parameter is used to determine the position of the nth column element in a single rearrangement step. The unit also determines a first intermediate parameter and a second intermediate parameter based on a fourth intermediate parameter, the number of bits in the second data format, and the number of bits in the first data format.

[0094] In some embodiments, the register address of an element is determined by rearranging the first parameter as the step size. For example, with... Figure 15For example, element T3 in register R0 of the first data format is the 4th bit of the first rearrangement step, therefore it is in the 4th register of the second data format, i.e., register R3. For example, element T0 in register R3 of the first data format is the 1st bit of the 4th rearrangement step, therefore it is in the 1st register of the second data format, i.e., register R0.

[0095] In some embodiments, the examples above mention the position of an element within a specific rearrangement step, where the rearrangement step refers to the bit position of the element in the corresponding register address. For example, if the first parameter is 4, the rearrangement step is 4. Elements in the result of the first sorting operation are assigned column-wise, with columns 0-3 representing the first rearrangement step, columns 4-7 representing the second rearrangement step, and so on. If an element is in the i-th rearrangement step, then in the second data format, the element will be in the i-th position in the corresponding register address. Figure 15 As shown, the elements in columns 0, 4, 8, and 16 of the first data format are arranged in the second data format.

[0096] In some embodiments, tensor computation units are reordered according to the following logic: uint32_t temp_block_dim = 32; uint32_t outer_stride = inner_stride * 4; for (uint32_t n = 0; n<32; ++n) { uint32_t inner = n % outer_stride; uint32_t outer = n / outer_stride * outer_stride; uint32_t target_n = (inner * inner_stride) / outer_stride + (inner *inner_stride) % outer_stride + outer; } Where `temp_block_dim` is the first column number, i.e., the number of columns in the first buffer. `outer_stride` is the number of bits in the first data format, `inner_stride` is the number of bits in the second data format, and 4 is the first parameter. `n` is the column number of the element in the first buffer, and the value of `n` is less than the number of columns in the first buffer. `inner` is the fourth intermediate parameter, and `outer` is the third intermediate parameter. `(inner * inner_stride) / outer_stride` is the first intermediate parameter, and `(inner * inner_stride) % outer_stride` is the second intermediate parameter. `target_n` is the column number of the nth column element in the first sorting result in the second data format.

[0097] by Figure 15 Taking register R5 in the first sorting operation as an example, T3 corresponds to n with a value of 23, outer_stride = 32, and inner_stride = 4. Therefore, we can calculate inner = n%outer_stride = 23%32 = 23, outer = n / outer_stride*outer_stride = 23 / 32*32 = 0, and target_n = (inner*inner_stride) / outer_stride + (inner*inner_stride)%outer_stride + outer = (23*8) / 32 + (23*8)%32 + 0 = 5 + 24 = 29. In other words, the element in column 23 of the first sorting result, after resorting, is located in column 29 of the second sorting result. At this point, the arrangement is done with a step size of 4. That is, after placing registers R0-R3, registers R4-R7 are placed next. Therefore, the element of column 23 is now located in the first bit of register R7, the result of the second sorting operation. In both the first and second sorting operations, the column number is calculated starting from 0.

[0098] by Figure 15Taking register R7 in the first sorting operation as an example, T3 corresponds to n with a value of 31, outer_stride = 32, and inner_stride = 8. Therefore, we can calculate inner = n%outer_stride = 31%32 = 31, outer = n / outer_stride*outer_stride = 31 / 32*32 = 0, and target_n = (inner*inner_stride) / outer_stride + (inner*inner_stride)%outer_stride + outer = (31*8) / 32 + (31*8)%32 + 0 = 7 + 24 = 31. In other words, the element in column 31 of the first sorting result, after resorting, is located in column 31 of the second sorting result. At this point, the arrangement is done with a step size of 4. That is, after placing registers R0-R3, registers R4-R7 are placed next. Therefore, the element in column 31 is located in the third bit of register R7, which represents the result of the second sorting operation. In both the first and second sorting operations, the column number is calculated starting from 0.

[0099] If the output needs to be stored in global memory, a larger data length is required. `DST_INNER_STRIDE` (the first parameter) needs to be configured to 8, then the post-processed layout will be as follows: Figure 16 As shown, when eight consecutive elements are arranged together and written out via LSU, the length is set to 8, which improves memory bandwidth utilization. Although the inner_stride value calculated based on the number of bits in the second data format and the first parameter differs from the target sorting bit count, the result of the reordering operation is identical to the register layout requirements, therefore no adjustment to the reordering scheme is necessary.

[0100] by Figure 16Taking register R1's T2 as an example, its corresponding value for n is 6, outer_stride = 32, and inner_stride = 4. Then, we can calculate inner = n%outer_stride = 6%32 = 6, outer = n / outer_stride*outer_stride = 6 / 32*32 = 0, target_n = (inner*inner_stride) / outer_stride + (inner*inner_stride)%outer_stride + outer = (6*4) / 32 + (6*4)%32 + 0 = 0 + 24 = 24. That is, the element in column 6 of the first sorting result, after reordering, is in column 24 of the second sorting result; that is, the element in column 6 is now in the first position of register R6 of the second sorting result. In both the first and second sorting, the column number is calculated starting from 0.

[0101] by Figure 16 Taking register R4 in the first sorting operation as an example, T3 corresponds to n with a value of 19, outer_stride = 32, and inner_stride = 4. Therefore, we can calculate inner = n%outer_stride = 19%32 = 19, outer = n / outer_stride*outer_stride = 19 / 32*32 = 0, and target_n = (inner*inner_stride) / outer_stride + (inner*inner_stride)%outer_stride + outer = (19*4) / 32 + (19*4)%32 + 0 = 2 + 12 = 14. In other words, the 18th element in the first sorting result, after reordering, is located in the 14th element of the second data format. That is, the 18th element is now located in the second bit of register R3 in the second sorting result.

[0102] Step a3: The tensor computation unit rearranges the elements in the first sorting result based on the coordinate addresses of each column element in the second sorting operation result to obtain the second sorting operation result.

[0103] In some embodiments, the tensor computation unit maps the first sorted computation result to the second sorted computation result according to the coordinate addresses corresponding to each column in the first sorted computation result in the second sorted computation result.

[0104] For example, such as Figure 15As shown, after rearranging, the tensor computation unit maps the tensor data of FP32 to the temporary storage unit after rearranging it according to the FP8 format.

[0105] In some embodiments, if the result of the second sorting operation needs to be used as tensor data in other tensor operations, then the result of the second sorting operation also needs to be formatted and packaged.

[0106] In some embodiments, the processor further includes a format conversion unit. The format conversion unit is configured to convert the operation result of the second sorting from the first data format to the second data format, and package it to obtain the output operation result, wherein each element in the output operation result is arranged according to the first sorting.

[0107] For example, such as Figure 11 As shown, T0 to T31 are 32 threads started by the format conversion unit. These 32 threads are used to perform data format conversion and packing on the encoded data of each element in matrix 1103. Data format conversion refers to the process of converting the data format from one data format to another data format. Packing refers to the process of packing multiple data together to form continuous data.

[0108] In some embodiments, the format conversion unit may be implemented as at least one of the following: a vector unit on SIMD, SIMT, or NPU.

[0109] Taking thread T0 as an example, thread T0 needs to perform data format conversion and packing on the first element in R0, the first element in R1, the first element in R2, and the first element in R3. For example, thread T0 can convert the data format of each element corresponding to T0 from FP32 to FP8, obtaining four elements in FP8 data format. Then, it packs the four elements in FP8 data format into a single register in sequence. That is, the four elements in FP32 data format form 32-bit encoded data, which includes four 8-bit encoded data, with each 8-bit encoded data corresponding to one element.

[0110] After understanding the data format conversion and packaging process, refer to... Figure 16 The result of the second sorting operation shown in the figure can be converted and packaged to restore the result of the second sorting operation to the result of the first sorting operation.

[0111] The technical solution adopted in this application embodiment realizes the arrangement of tensor computation unit outputs in registers by adding a bit field and a rearrangement algorithm, which meets various application scenarios and supports various input data formats. Hardware implementation of tensor data rearrangement avoids offline data processing and the complexity of software implementation.

[0112] In some embodiments, reference Figure 5 Taking a processor implemented as a graphics processing unit (GPU) as an example, a GPU5000 consists of multiple compute cores (MPCs) 5100, each compute core 5100 consists of multiple compute clusters (MPXs) 5110, and two compute units (MPs) 5111 form an MPX 5110. For example... Figure 6 As shown, a computation unit (MP) 5111 contains one or more complete sets of instruction execution and computation logic 610; the figure shows four sets. A tensor computation engine (TCE) 110 is located within the MP 5111, and each TCE 110 contains four sets of tensor computation units (ALU0~ALU3), with the four ALUs sharing a single control logic (CTRL LOGIC). The registers in each computation unit 5111 are implemented as the aforementioned target storage units. The tensor computation engine 110 includes tensor computation units and a first buffer.

[0113] In some embodiments, the tensor computation unit is used to perform logical operations on tensor data to obtain a first sorted operation result, wherein the first sorted operation result is in a first data format.

[0114] In some embodiments, the tensor computation unit is further configured to cache the results of the first sorting operation to a first buffer.

[0115] In some embodiments, the tensor computation unit is further configured to convert the operation result of the first sorting cached in the first buffer into a second sorting to obtain the operation result of the second sorting; and store the operation result of the second sorting into the target storage unit.

[0116] In some embodiments, the tensor computation unit is configured to: determine a first parameter, the first parameter being used to indicate the conversion granularity between a first data format and a second data format; and, based on the first parameter, perform a rearrangement of the elements in the result of the first sorting operation to obtain the result of the second sorting operation.

[0117] In some embodiments, the first parameter is used to indicate the relationship between the number of bits in the first data format and the number of bits in the second data format; The first data format is a 32-bit floating-point number FP32; When the first parameter is 8, the second data format is a 4-bit floating-point number FP4; or, When the first parameter is 4, the second data format is an 8-bit floating-point number FP8; or, When the first parameter is 2, the second data format is a 16-bit floating-point number FP16; or, When the first parameter is 0, the second data format is a 32-bit floating-point number FP32.

[0118] In some embodiments, the tensor computation unit is configured to: determine the number of bits in a second data format based on the number of bits in a first data format and a first parameter; determine the coordinate address corresponding to each column element in the second sorting operation result based on the number of bits in the second data format for each column element in the first sorting operation result; and rearrange the elements in the first sorting operation result based on the coordinate address corresponding to each column element in the second sorting operation result to obtain the second sorting operation result.

[0119] In some embodiments, the tensor computation unit is configured to: for the nth column element in the first sorting operation result, determine the coordinates corresponding to the nth column element in the second sorting operation result based on the number of bits in the second data format, where n is an integer less than the number of the first column, and the number of the first column is the number of columns occupied by the first sorting operation result in the first buffer.

[0120] In some embodiments, the tensor computation unit is configured to: determine a first intermediate parameter and a second intermediate parameter based on the value of n and the number of bits in the second data format, wherein the first intermediate parameter is used to determine the register address corresponding to the element in the nth column in the second sorting operation result, and the second intermediate parameter is used to determine the bit position of the element in the register address corresponding to the element in the nth column in the second sorting operation result; and determine the coordinate address corresponding to the element in the nth column in the second sorting operation result based on the first intermediate parameter and the second intermediate parameter.

[0121] In some embodiments, the tensor computation unit is configured to: determine a fourth intermediate parameter based on the value of n and the number of bits in the first data format, the fourth intermediate parameter being used to determine the position of the nth column element in a single rearrangement step; and determine a first intermediate parameter and a second intermediate parameter based on the fourth intermediate parameter, the number of bits in the second data format, and the number of bits in the first data format.

[0122] In some embodiments, the tensor calculation unit is configured to: determine a third intermediate parameter based on the value of n and the number of bits in the first data format, wherein the third intermediate parameter is used to determine the offset number of bits corresponding to the rearrangement step size of the nth column element; and determine the coordinate address corresponding to the nth column element in the second sorting operation result based on the first intermediate parameter, the second intermediate parameter and the third intermediate parameter.

[0123] In some embodiments, the processor further includes a format conversion unit; the format conversion unit converts the second sorted operation result from the first data format to the second data format, and packages it to obtain the output operation result, wherein each element in the output operation result is arranged according to the first sort.

[0124] The technical solution provided in this application adds a first buffer to the tensor computation engine, allowing the tensor computation unit to reorder the computation results. This ensures that the data arrangement after format conversion conforms to the input of the next matrix multiplication, or meets the requirement for larger storage lengths. The computation results converted to the second sort can be directly stored in the target storage unit without needing to perform a reverse conversion after the tensor storage engine performs the data format conversion. Since the tensor computation unit directly reorders the computation results from the first sort, there is no need to consider the impact of the storage method used for the first sorted computation results in the storage unit on the complexity of the data format conversion.

[0125] Please refer to Figure 17 This diagram illustrates a simplified structural block diagram of a computer device provided in one embodiment of this application. The computer device 1700 can be any electronic device capable of data calculation, processing, and storage.

[0126] Computer device 1700 includes: processor 1701 and memory 1702.

[0127] Processor 1701 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1701 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1701 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1701 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1701 may also include an AI processor for handling computational operations related to machine learning.

[0128] Memory 1702 may include one or more computer-readable storage media, which may be non-transitory. Memory 1702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in memory 1702 is used to store a computer program configured to be executed by one or more processors.

[0129] Those skilled in the art will understand that Figure 17 The structure shown does not constitute a limitation on the computer device 1700, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0130] In some embodiments, a chip product is also provided, which includes the processor described above. Optionally, the chip product may be a GPU chip product, where the processor described above is a GPU, and the GPU chip product includes the GPU described above. Optionally, the above chip product may be implemented as a graphics card, which includes the processor described above, such as a GPU.

[0131] In some embodiments, a computer device is also provided, which includes the processor described above. Optionally, the computer device may be a personal computer, workstation, game console, and some mobile devices (such as tablet computers, smartphones, etc.), or it may be an in-vehicle terminal device, smart home device, smart TV, smart robot, etc., or it may be a server, server cluster, artificial intelligence computing cluster, cloud computing cluster, etc., wherein the artificial intelligence computing cluster may also be simply referred to as an intelligent computing cluster or smart computing cluster, and this application does not limit it in this way.

[0132] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0133] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A processor, characterized in that, The processor includes a tensor computation engine and a target storage unit, wherein the tensor computation engine includes a tensor computation unit and a first buffer; The tensor computation unit is used to perform logical operations on tensor data to obtain a first sorted operation result, wherein the first sorted operation result is in a first data format. The tensor computation unit is also used to store the operation result of the first sorting into the first buffer; The tensor calculation unit is further configured to convert the first sorting operation result cached in the first buffer into a second sorting according to the second data format to obtain the second sorting operation result; and store the second sorting operation result into the target storage unit.

2. The processor according to claim 1, characterized in that, The tensor computation unit is used for: Based on the first parameter, the elements in the result of the first sorting operation are rearranged to obtain the result of the second sorting operation; wherein, the first parameter is used to indicate the conversion granularity between the first data format and the second data format.

3. The processor according to claim 2, characterized in that, The first parameter is used to indicate the relationship between the number of bits in the first data format and the number of bits in the second data format; The first data format is a 32-bit floating-point number FP32; When the first parameter is 8, the second data format is a 4-bit floating-point number FP4; or, When the first parameter is 4, the second data format is an 8-bit floating-point number FP8; or, When the first parameter is 2, the second data format is a 16-bit floating-point number FP16; or, When the first parameter is 0, the second data format is a 32-bit floating-point number FP32.

4. The processor according to claim 2, characterized in that, The tensor computation unit is used for: The number of bits in the second data format is determined based on the number of bits in the first data format and the first parameter. For each column element in the first sorting operation result, the coordinate address corresponding to each column element in the second sorting operation result is determined based on the number of bits in the second data format; Based on the coordinate addresses of each column element in the second sorting result, the elements in the first sorting result are rearranged to obtain the second sorting result.

5. The processor according to claim 4, characterized in that, The tensor computation unit is used for: For the element in the nth column of the first sorting operation result, the coordinates of the element in the nth column in the second sorting operation result are determined based on the number of bits in the second data format, where n is an integer less than the number of the first column, and the number of the first column is the number of columns occupied by the first sorting operation result in the first buffer.

6. The processor according to claim 5, characterized in that, The tensor computation unit is used for: Based on the value of n and the number of bits in the second data format, a first intermediate parameter and a second intermediate parameter are determined. The first intermediate parameter is used to determine the register address corresponding to the element in the nth column in the second sorting operation result, and the second intermediate parameter is used to determine the bit position of the element in the nth column in the register address corresponding to the element in the second sorting operation result. Based on the value of n and the number of bits in the first data format, a third intermediate parameter is determined. The third intermediate parameter is used to determine the offset number of bits corresponding to the rearrangement step size of the nth column element. Based on the first intermediate parameter, the second intermediate parameter, and the third intermediate parameter, the coordinate address corresponding to the element in the nth column in the second sorting operation result is determined.

7. The processor according to claim 6, characterized in that, The tensor computation unit is used for: Based on the value of n and the number of bits in the first data format, a fourth intermediate parameter is determined, which is used to determine the position of the nth column element in a single rearrangement step. The first intermediate parameter and the second intermediate parameter are determined based on the fourth intermediate parameter, the number of bits in the second data format, and the number of bits in the first data format.

8. The processor according to any one of claims 1 to 7, characterized in that, The processor also includes a format conversion unit; The format conversion unit is used to convert the calculation result of the second sorting from the first data format to the second data format, and package it to obtain the output calculation result, wherein each element in the output calculation result is arranged according to the first sorting.

9. A chip product, characterized in that, The chip product includes the processor as described in any one of claims 1 to 8.

10. A computer device, characterized in that, The computer device includes the processor as described in any one of claims 1 to 8.

11. A tensor processing method applied to a processor, characterized in that, The processor includes: a tensor computation engine and a target storage unit, the tensor computation engine including a tensor computation unit and a first buffer; the method includes: The tensor computation unit performs logical operations on the tensor data to obtain a first sorted operation result, which is in a first data format. The tensor computation unit stores the result of the first sorting operation into the first buffer; The tensor calculation unit converts the first sorting operation result cached in the first buffer into a second sorting operation result according to the second data format. The tensor computation unit stores the results of the second sorting operation into the target storage unit.

Images