Processor, chip product, computer equipment and tensor processing method

Abstract

The invention discloses a processor, a chip product, computer equipment and a tensor processing method, and relates to the technical field of processors. A first tensor having a plurality of elements arranged according to a first arrangement order is stored in the first storage unit; the tensor storage engine is used for reading the plurality of elements from the first storage unit to the second storage unit according to a second arrangement sequence to obtain a rearranged tensor; the tensor calculation engine is used for executing logical operation on the rearranged tensor to obtain an operation result; the format conversion unit is used for converting the data format of each element in the operation result into a target data format to obtain an operation result after format conversion, and storing the operation result after format conversion into a third storage unit, the operation result after format conversion comprises the operation results of the multiple elements, and the third storage unit is used for storing the operation results after format conversion. A third arrangement sequence of the operation results of the plurality of elements in the third storage unit is the same as the first arrangement sequence. The development cost of the processor can be reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of processor technology, and in particular to a processor, a chip product, a computer device, and a tensor processing method. Background Art

[0002] At present, common processors generally include TME (Tensor Memory Engine) and TCE (Tensor Compute Engine). TME is responsible for accessing storage units and accelerating the storage and access of tensor data, while TCE is responsible for tensor computing.

[0003] TCE is used to perform logical operations on tensor data to obtain a first operation result of the tensor data. The first operation result includes the operation results of each element in the tensor corresponding to the tensor data. The operation results of each element are distributed in different registers. When the first operation result needs to be used for subsequent operations, the SIMD (Single Instruction Multiple Data) unit of the processor will perform data format conversion on the operation results of each element, and pack the operation results after data format conversion in different registers to obtain the packed first operation result, and then put the packed first operation result into the subsequent operation. However, since the arrangement order of the operation results after data format conversion will be adjusted during the packing process, the arrangement order of the operation results corresponding to each element in the packed first operation result will become inconsistent with the arrangement order of the operation results corresponding to each element in the first operation result, which is not conducive to subsequent operations (i.e., adjustment is required before subsequent operations).

[0004] To address the problem of inconsistent arrangement order, the relevant technology adopts an offline data rearrangement solution, that is, before TCE performs logical operations on tensor data, a process for offline rearrangement of tensor data is added, such as adjusting the arrangement order of each encoded data in the tensor data (that is, the encoded data corresponding to the elements in the tensor) through software design to obtain the adjusted tensor data, and then performing logical operations on the adjusted tensor data through TCE to obtain a second operation result. After the second operation result is converted into a data format and packaged, a packaged second operation result is obtained. Since the arrangement order of each element in the second operation result will be adjusted again during the packaging process, the arrangement order of the operation results corresponding to each element in the packaged second operation result can be made consistent with the arrangement order of the operation results corresponding to each element in the above-mentioned first operation result, which is facilitating subsequent calculations (that is, no adjustment is required before subsequent calculations).

[0005] However, since the related art implements an additional offline re-arrangement process by increasing software design, the development cost of the processor is increased. Summary of the invention

[0006] The embodiments of the present application provide a processor, a chip product, a computer device and a tensor processing method. The technical solution provided by the embodiments of the present application includes the following contents.

[0007] According to one aspect of an embodiment of the present application, a processor is provided, the processor comprising: a tensor storage engine, a tensor computing engine, a format conversion unit, a first storage unit, a second storage unit, and a third storage unit; a first tensor is stored in the first storage unit, and the first tensor comprises a plurality of elements arranged in a first arrangement order;

[0008] The tensor storage engine is used to read the plurality of elements from the first storage unit to the second storage unit in a second arrangement order to obtain a rearranged tensor, wherein the second arrangement order is different from the first arrangement order;

[0009] The tensor computing engine is used to perform a logical operation on the rearranged tensor to obtain an operation result;

[0010] The format conversion unit is used to convert the data format of each element in the operation result into a target data format, obtain the operation result after format conversion, and store the operation result after format conversion into the third storage unit;

[0011] The calculation result after the format conversion includes the calculation results of each of the multiple elements, and the third arrangement order of the calculation results of each of the multiple elements in the third storage unit is the same as the first arrangement order.

[0012] According to one aspect of an embodiment of the present application, a chip product is provided, wherein the chip product includes the above-mentioned processor.

[0013] According to one aspect of an embodiment of the present application, a computer device is provided, wherein the computer device includes the above-mentioned processor.

[0014] According to one aspect of an embodiment of the present application, a tensor processing method applied to a processor is provided, the processor comprising: a tensor storage engine, a tensor computing engine, a format conversion unit, a first storage unit, a second storage unit, and a third storage unit; a first tensor is stored in the first storage unit, and the first tensor comprises a plurality of elements arranged in a first arrangement order;

[0015] The method comprises:

[0016] The tensor storage engine reads the plurality of elements from the first storage unit to the second storage unit in a second arrangement order to obtain a rearranged tensor, wherein the second arrangement order is different from the first arrangement order;

[0017] The tensor calculation engine performs a logical operation on the rearranged tensor to obtain an operation result;

[0018] The format conversion unit converts the data format of each element in the operation result into a target data format to obtain the operation result after format conversion, and stores the operation result after format conversion into the third storage unit;

[0019] The calculation result after the format conversion includes the calculation results of each of the multiple elements, and the third arrangement order of the calculation results of each of the multiple elements in the third storage unit is the same as the first arrangement order.

[0020] The technical solution provided by the embodiments of the present application may include the following beneficial effects.

[0021] In the process of the tensor storage engine reading the first tensor from the first storage unit to the second storage unit, the tensor storage engine is supported to automatically rearrange multiple elements in the first tensor so that the third arrangement order of the operation results of the multiple elements in the third storage unit is the same as the first arrangement order of the multiple elements, without the need to develop additional software design to support additional rearrangement processes, thereby reducing the complexity of the tensor processing process in the processor, avoiding the development cost caused by software design, and further reducing the development cost of the processor.

[0022] In addition, since the present application implements the reordering of tensors at the existing hardware level, it can avoid the impact of adding a reordering process on the end-to-end performance of the processor, thereby helping to improve the end-to-end performance of the processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a schematic diagram of a processor provided in a possible implementation of the present application;

[0025] Figure 2 is a schematic diagram of a method for packaging calculation results provided in a possible implementation of the present application;

[0026] Figure 3 is a schematic diagram of a tensor storage engine provided in a possible implementation of the present application;

[0027] Figure 4 It is a schematic diagram of a staggered arrangement provided in a possible implementation of the present application;

[0028] Figure 5 is a schematic diagram of rearranging the first tensor in a staggered arrangement provided in a possible implementation of the present application;

[0029] Figure 6 It is a schematic diagram of filling preset elements provided in a possible implementation of the present application;

[0030] Figure 7 It is a schematic diagram of rearranging, converting data formats, and packaging a first tensor without interleaving arrangement provided in a possible implementation of the present application;

[0031] Figure 8 is a schematic diagram of a graphics processor provided in a possible implementation of the present application;

[0032] Fig. 9 is a schematic diagram of a processor in a graphics processor provided in a possible implementation of the present application;

[0033] Fig.10 is a flowchart of a tensor processing method applied to a processor provided in a possible implementation of the present application;

[0034] Fig.11 It is a simplified structural block diagram of a computer device provided in a possible implementation of the present application. DETAILED DESCRIPTION

[0035] Before introducing the embodiments of the present application, the relevant terms involved in the present application are first explained.

[0036] Before introducing the embodiments of the present application, the relevant terms involved in the present application are first explained.

[0037] Tensor: It is a general term and extension of concepts such as scalar, vector, matrix, etc. It is a basic data structure in the field of machine learning. It can be understood as a multidimensional array, which includes at least one element. For example, a 0-dimensional tensor is 1 number, a 1-dimensional tensor is equivalent to a vector, a 2-dimensional tensor corresponds to a matrix, and a 3-dimensional tensor is a cube. The embodiment of the present application does not limit the dimension of the tensor, such as the dimension of the tensor can also be 4 dimensions, and more than 4 dimensions.

[0038] Tensors can store more information about objects, such as the height, color, depth, width, etc. of images, the number of frames, color, height, width, etc. of videos, and the time domain, frequency domain, and fundamental frequency of audio.

[0039] Tensor computing mainly refers to the calculation and processing of tensors in the fields of deep learning and machine learning. These calculations usually involve operations such as matrix multiplication and convolution, and are key steps in machine learning model training and reasoning.

[0040] Floating point type: A data type used to represent real numbers. It allows the decimal point to float, so that fractions, decimals, and some irrational numbers can be represented. Real numbers represented by floating point types are called floating point numbers.

[0041] The floating-point number type includes at least one of the following: single-precision floating-point type (float), double-precision floating-point type (double), high-precision floating-point type (decimal), and floating-point types such as FP16, BF16 (Brain Floating-Point 16-bit, 16-bit brain floating point number), FP8 (Floating-Point 8-bit, 8-bit floating point number), FP6 (Floating-Point 6-bit, 6-bit floating point number), FP4 (Floating-Point 4-bit, 4-bit floating point number).

[0042] Floating point numbers are usually represented in scientific notation. For example, FP16: half-precision floating point number, 16-bit floating point representation, that is, it uses a 16-bit value to represent real numbers, including 1-bit sign, 5-bit exponent and 10-bit mantissa. FP8: uses an 8-bit value to represent real numbers, including 4-bit exponent and 3-bit exponent (E4M3), or 5-bit exponent and 2-bit mantissa (E5M2), aiming to improve computing efficiency by reducing the number of bits of data representation.

[0043] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0044] Please refer to Figure 1 , which shows a schematic diagram of a processor provided in a possible implementation of the present application. The processor 100 may include: a tensor memory engine (TME) 101, a first storage unit 103 and a second storage unit 104.

[0045] The above-mentioned processor 100 may refer to a processor that supports tensor processing, such as the processor 100 supports at least one of the following operations for tensors: transmission, storage, and calculation. For example, the tensor storage engine 101 in the processor 100 supports tensor transmission, each storage unit in the processor 100 supports tensor storage, and the tensor calculation engine 102 in the processor 100 supports tensor calculation. Among them, tensor transmission may refer to the transmission of tensor data corresponding to a tensor, tensor storage may refer to the storage of tensor data corresponding to a tensor, and tensor calculation may refer to the calculation of tensor data corresponding to a tensor. Tensor data is used to characterize a tensor. Optionally, the processor 100 may be implemented as a physical circuit for processing tensor data.

[0046] In some embodiments, the processor 100 may be implemented as a processing unit (Processing Unit, referred to as PU) that supports tensor processing. The embodiment of the present application does not limit the processor 100. The processor 100 may be a newly designed one or an improved one of an existing processor. Exemplarily, the processor 100 may be implemented as at least one of the following: a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a tensor processor (TPU), a field programmable gate array (FPGA), a tensor core (Tensor Core), and an NPU (Neural Processing Unit).

[0047] In one example, the data formats supported by the processor 100 include at least one of the following: floating-point numbers in data formats such as FP8, FP16, BF16, FP6, FP4, FP32, and TF32 (Tensor-Floating 32-bit, 32-bit mixed floating point numbers), fixed-point numbers in data formats such as U8 and S8, and integers in data formats such as INT8 (8-bit integer) and INT16 (16-bit integer).

[0048] Exemplarily, the processor 100 supports the processing of tensor data of tensors in the FP8 data format. Tensor data refers to data used to characterize a tensor, which may be composed of encoded data generated for each element in the tensor, and each element may be represented by one encoded data. For example, the tensor data of a tensor in the FP8 data format includes: encoded data generated according to the FP8 data format for each element in the tensor, each encoded data may include 8-bit data, and 8-bit data may determine the value of a floating point number (ie, an element). The order of arrangement of the encoded data corresponding to each element is consistent with the order of arrangement of each element in the tensor.

[0049] Optionally, each encoded data may consist of a set of binary numbers, such as 0 and 1. For example, for the FP6 data format, each encoded data may include 6 bits of data, such as 6 binary numbers, which respectively indicate the sign, exponent, and mantissa of the floating point number (i.e., element). For the FP4 data format, each encoded data may include 4 bits of data, such as 4 binary numbers, which respectively indicate the sign, exponent, and mantissa of the floating point number (i.e., element).

[0050] The tensor storage engine 101 refers to an engine for performing tensor transmission and tensor storage in the processor 100, and the engine can be implemented as a hardware system for processing data, such as a physical circuit. For example, the tensor storage engine 101 can be implemented as a physical circuit for performing tensor transmission and tensor storage.

[0051] Optionally, the processor 100 may further include: a tensor compute engine (TCE) 102. Exemplarily, the tensor storage engine 101 is used to read tensors (i.e., tensor data of tensors) from the first storage unit 103 to store the tensors (i.e., tensor data of tensors) in the second storage unit 104, so that the tensor computing engine 102 can read tensors (i.e., tensor data of tensors) from the second storage unit 104, and perform logical operations on tensors (i.e., tensor data of tensors). Optionally, a data transmission channel (such as a data bus BUS) is provided between the tensor storage engine 101 and the first storage unit 103 and the second storage unit 104, respectively, and the tensor storage engine 101 can realize the transmission of tensor data through the data transmission channel. For example, the tensor storage engine 101 can be used to transfer the tensor data of the tensor in the FP8 data format in the first storage unit 103 to the second storage unit 104 through the data transmission channel.

[0052] The first storage unit 103 is used to store tensor data to be processed by the processor 100, and the second storage unit 104 is used to temporarily store tensor data to be calculated. The tensor calculation engine 102 can directly read tensor data from the second storage unit 104 to perform tensor calculation.

[0053] In one example, the second storage unit 104 can be implemented as an on-chip cache, recorded as a local memory (LocalMemory), to achieve temporary storage of tensor data, thereby improving the speed at which the tensor computing engine 102 obtains tensor data. Optionally, the second storage unit 104 can also be implemented as a register (Register), such as a vector register (VectorRegister) / vector register file (VectorRegister File), to achieve temporary storage of tensor data, thereby further improving the speed at which the tensor computing engine 102 obtains tensor data (the data reading speed of the register is much faster than the data reading speed of the memory).

[0054] The first storage unit 103 can be implemented as a shared cache of at least one tensor computing engine 102, recorded as global memory, which can support at least one tensor computing engine 102 to access data thereto, such as by supporting multiple tensor computing engines 102 to access tensor data thereto, so as to realize the sharing of tensor data among multiple tensor computing engines 102.

[0055] Exemplarily, the second storage unit 104 can be implemented as a first-level cache in the processor 100, i.e., an L1 cache, which is the cache layer closest to the tensor computing engine 102, and usually has a relatively small storage space, but a very fast data reading speed. The first storage unit 103 can be implemented as a second-level cache in the processor 100, i.e., an L2 cache. In this case, the first storage unit 104 is located between the L1 cache (i.e., the second storage unit 104) and the main memory of the processor 100 (such as an L3 cache), and is used to store more tensor data, so as to provide more backups in the event of a L1 cache miss. The storage space of the L2 cache is larger than the storage space of the L1 cache, but the data reading speed of the L2 cache is lower than the data reading speed of the L1 cache.

[0056] The tensor computing engine 102 refers to a computing engine in the processor 100 for performing tensor computing, such as the tensor computing engine 102 can be implemented as a physical circuit for performing tensor computing. Optionally, the operation logic supported by the tensor computing engine 102 includes at least one of the following: matrix multiplication operation, convolution operation, multiplication-accumulation operation, and addition operation. For example, the tensor computing engine 102 can be used to perform multiplication-accumulation operations on tensor data to implement multiplication-accumulation operations of tensors and obtain multiplication-accumulation operation results of tensors. The embodiment of the present application does not limit the number of tensor computing engines 102 included in the processor 100, which can be set and adjusted according to actual usage requirements.

[0057] Exemplarily, a data transmission channel is provided between the tensor computing engine 102 and the second storage unit 104, so that the tensor computing engine 102 can read data from the second storage unit 104. For example, the tensor computing engine 102 can be used to read tensor data in the FP8 data format from the second storage unit 104, and perform operation logic on the tensor data in the FP8 data format to obtain the operation result of the tensor, so as to realize tensor calculation.

[0058] Optionally, the processor 100 may further include a third storage unit 105. The tensor computing engine 102 may also be used to store the operation results of the tensor data in the third storage unit 105. Optionally, the third storage unit 105 may be implemented as a register file to store the operation results corresponding to different threads or the operation results corresponding to different tensor computing engines or the operation results of different processing units in the tensor computing engine, which is not limited in the embodiment of the present application.

[0059] In some embodiments, the operation result of the tensor will also be used as the input of subsequent operations. For example, in the field of deep learning technology, for two model layers connected in a neural network model, the output data of the first model layer (i.e., the operation result of the tensor data) will be used as the input data of the second model layer, so it is necessary to convert the data format of the output data into the data format supported by the second model layer, and package the output data after the data format conversion to the tensor calculation engine 102 for subsequent operations.

[0060] For example, refer to Figure 2 , the size of matrix 201 is M*K, the size of matrix 202 is K*N, and matrix 203 is the result of the operation between matrix 201 and matrix 202. For example, by performing a matrix multiplication operation on matrix 201 and matrix 202, matrix 203 can be obtained. Matrix 203 can be used to represent the result of the matrix multiplication operation between matrix 201 and matrix 202, and its size is M*N. Each element in matrix 203 (i.e. Figure 2 The cell corresponding to the matrix 203 in the matrix 201 corresponds to a multiplication-accumulation operation result, and the data format of the multiplication-accumulation operation result is FP32. For example, for the first element in the upper left corner of the matrix 203, it is the multiplication-accumulation operation result obtained by performing a multiplication-accumulation operation on the first row element in the matrix 201 (denoted as M0) and the first column element in the matrix 202 (denoted as N0).

[0061] Each element in the matrix 203 is stored in a different register. For example, the first 4 columns of elements in the matrix 203 are stored in register R0, the second 4 columns of elements are stored in register R1, the third 4 columns of elements are stored in register R2, and the last 4 columns of elements are stored in register R3.

[0062] T0 to T31 are format conversion units ( Figure 1 The 32 threads opened by the matrix 203 are used to perform data format conversion and packaging on the encoded data of each element in the matrix 203. Data format conversion refers to the process of converting the data format of data from one data format to another data format, and packaging refers to the process of packaging multiple data together to form continuous data. The format conversion unit can be implemented as at least one of the following: SIMD, SIMT (Single Instruction Multiple Thread, single instruction multiple thread architecture), and vector unit on NPU.

[0063] Taking thread T0 as an example, thread T0 needs to convert and pack the data format of the first element in R0, the first element in R1, the first element in R2, and the first element in R3. For example, thread T0 can convert the data format of each element corresponding to T0 from FP32 to FP8, obtain 4 elements in FP8 data format, and then pack the 4 elements in FP8 data format into a single register in sequence, that is, the 4 elements in FP32 data format form 32-bit encoded data, and the 32-bit encoded data includes 4 8-bit encoded data, and each 8-bit encoded data corresponds to one element (that is, the above operation result).

[0064] When the matrix 203 is sorted according to its columns, the arrangement order of the element columns in the matrix 203 is: N0 column elements to N15 column elements. After being processed by the format conversion unit (i.e., after data format conversion and packaging), the arrangement order of the processed element columns is changed to: N0 column elements, N4 column elements, N8 column elements, N12 column elements, N1 column elements, N5 column elements, N9 column elements, N13 column elements, N2 column elements, N6 column elements, N10 column elements, N14 column elements, N3 column elements, N7 column elements, N11 column elements, and N15 column elements. The arrangement order of the processed element columns is inconsistent with the arrangement order of the element columns before processing. When the processed element columns need to be put into subsequent calculations, the arrangement order of the processed element columns needs to be restored to be consistent with the arrangement order of the element columns before processing before they can be put into subsequent calculations. This will increase the complexity of subsequent calculations (e.g., the arrangement order needs to be readjusted to the original order for calculation).

[0065] In response to the above-mentioned problem of inconsistent arrangement order, in the process of the tensor storage engine 101 reading tensors from the first storage unit 103, the embodiment of the present application supports the tensor storage engine 101 to automatically rearrange the arrangement order of tensors in the second storage unit 104, so that the arrangement order of the calculation results after processing by the format conversion unit is consistent with the arrangement order of the calculation results obtained by normal execution of tensor calculation (i.e., without going through the rearrangement process), thereby reducing the complexity of subsequent calculations (i.e., there is no need to readjust the original arrangement order for calculation).

[0066] For example, the arrangement order of the encoded data of each element column in the matrix 202 in the first storage unit 103 is: the encoded data of the N0 column element to the encoded data of the N15 column element, that is, the arrangement order of the coordinate address of the encoded data of each element column is: the coordinate address of the encoded data of the N0 column element to the coordinate address of the encoded data of the N15 column element, forming the coordinate address information of the tensor data. The tensor storage engine 101 can adjust the arrangement order of the coordinate address of the encoded data of each element column to: the coordinate address of the encoded data of the N0 column element, the encoded data of the N4 column element, and the encoded data of the N15 column element. The tensor storage engine 101 obtains the adjusted coordinate address information according to the coordinate address of the encoded data of the N0 column elements, the coordinate address of the encoded data of the N8 column elements, ..., the coordinate address of the encoded data of the N15 column elements. After that, the tensor storage engine 101 reads the encoded data of each element column from the first storage unit 103 to the second storage unit 104 in turn according to the adjusted coordinate address information. The encoded data of each element column in the second storage unit 104 is arranged in the following order: the encoded data of the N0 column elements, the encoded data of the N4 column elements, the encoded data of the N8 column elements, ..., the encoded data of the N15 column elements.

[0067] After the tensor computing engine 102 performs logical operations on the encoded data of each element column, the operation results corresponding to each element column are arranged in the following order: the operation results of the encoded data of the N0 column elements, the operation results of the encoded data of the N4 column elements, the operation results of the encoded data of the N8 column elements, ..., the operation results of the encoded data of the N15 column elements (i.e., matrix 203).

[0068] The format conversion unit performs data format conversion and packaging on the calculation results corresponding to each element column. Taking thread T0 as an example, thread T0 needs to perform data format conversion and packaging on the first element in R0 (corresponding to column N0), the first element in R1 (corresponding to column N1), the first element in R2 (corresponding to column N2), and the first element in R3 (corresponding to column N3). For example, thread T0 can convert the data format of each element corresponding to T0 from FP32 to FP8, obtain 4 elements in FP8 data format, and then pack the 4 elements in FP8 data format into a single register in sequence, so that the first element in column N0, the first element in column N1, the first element in column N2, and the first element in column N3 are restored to a continuous arrangement.

[0069] By analogy, after data format conversion and packaging, the arrangement order of the calculation results corresponding to each element column is changed to: the calculation results of the elements in the N0 column to the calculation results of the elements in the N15 column. In this way, the arrangement order of the calculation results after being processed by the format conversion unit can be consistent with the arrangement order of the calculation results obtained by normal tensor calculation (that is, without the rearrangement process).

[0070] The embodiment of the present application supports the tensor storage engine 101 to automatically reorder tensors while transmitting tensors, without the need to develop additional software design to support additional reordering processes, thereby reducing the complexity of the tensor processing process in the processor and avoiding the development costs caused by software design, thereby reducing the development costs of the processor.

[0071] In some embodiments, taking the first tensor as an example, the process of implementing tensor reordering by the tensor storage engine 101 is described.

[0072] The first tensor mentioned above may refer to any tensor to be processed by the processor 100. Optionally, the first tensor may be a tensor composed of some elements in a tensor, such as when an M×N tensor (i.e., a matrix) is evenly divided into k (M / k)×N blocks, the first tensor may refer to a tensor composed of elements corresponding to any block in the k blocks. The first tensor may also be an undivided tensor, such as the first tensor is an M×N tensor. The embodiment of the present application does not limit the source of the first tensor.

[0073] The embodiments of the present application do not limit the elements in the first tensor. Exemplarily, the first tensor can be used to store model parameters of a neural network model, such as model parameters (such as weight parameters, bias parameters, etc.) of each model layer (such as convolutional layer, fully connected layer, attention mechanism layer, etc.) in a large model. The first tensor can also be used to store feature data in deep learning tasks, such as image feature data, speech feature data, text feature data, video feature data, etc. The first tensor can also be used to store coefficients in functions or formulas.

[0074] Optionally, the first storage unit 103 stores a first tensor, which includes multiple elements arranged in a first arrangement order, that is, the first storage unit 103 stores tensor data of the first tensor, which includes multiple encoded data, and each encoded data is used to represent an element in the first tensor.

[0075] Optionally, the first arrangement order of the multiple elements in the first storage unit 103 is the same as the arrangement order of the multiple elements in the first tensor (i.e., there is no staggered arrangement as described below), and the first arrangement order of the multiple elements in the first storage unit 103 may also be different from the arrangement order of the multiple elements in the first tensor, such as there is a staggered arrangement as described below. The embodiment of the present application does not limit the first arrangement order, which can be set and adjusted according to actual usage requirements.

[0076] The tensor data of the first tensor is used to represent the first tensor, and the tensor data of the first tensor may be composed of a coded data corresponding to each element in the first tensor, and each coded data may be used to determine the value of an element.

[0077] Optionally, the total number of bits of the encoded data of each element may be determined based on the first data format. For example, the total number of bits of the encoded data of each element is the same as the total number of bits (i.e., the total number of bits) corresponding to the first data format. The embodiment of the present application does not limit the format type of the first data format, such as the format type of the first data format may include at least one of the following: floating point type, integer type. Exemplarily, the first data format may include at least one of the following: FP8, FP16, BF16, FP6, FP4, FP32, TF32, U8, S8, INT8, INT16. For example, when the first data format is FP4, each element can be represented by 4-bit data to form a coded data; when the first data format is FP6, each element can be represented by 6-bit data to form a coded data.

[0078] Please refer to Figure 3The process of implementing tensor rearrangement by the tensor storage engine 101 may include the following contents: the tensor storage engine 101 is used to read multiple elements from the first storage unit 103 to the second storage unit 104 according to the second arrangement order to obtain a rearranged tensor.

[0079] The second arrangement order is different from the first arrangement order. The second arrangement order can be used to indicate the reading order of the multiple elements in the first storage unit 103, that is, the arrangement order of the multiple elements in the second storage unit 104. The rearranged tensor is obtained by rearranging the multiple elements in the first tensor according to the second arrangement order.

[0080] In an example, the tensor storage engine 101 can rearrange the elements in the first tensor by rearranging the arrangement order of the coordinate address information corresponding to each element in the first tensor in the first storage unit 103, and the embodiment of the present application may also include the following content:

[0081] 1. The tensor storage engine 101 is used to obtain coordinate address information of a first tensor, where the coordinate address information includes coordinate addresses of multiple elements in the first storage unit 103.

[0082] The coordinate addresses of the multiple elements in the first storage unit 103 are actually the coordinate addresses of the encoded data of the multiple elements in the first storage unit 103.

[0083] The embodiment of the present application does not limit the reading granularity of the first tensor, which can be set and adjusted according to actual usage requirements. The reading granularity of the first tensor may refer to the number of elements read by the tensor storage engine 101 each time.

[0084] Exemplarily, for the tensor data of the first tensor, the data to be read corresponding to each time of the first tensor can be determined according to the reading granularity of the first tensor. The data to be read can be the entire tensor data, that is, the data to be read includes the encoded data corresponding to all elements in the first tensor. The data to be read can also refer to part of the data in the tensor data, that is, the data to be read includes the encoded data corresponding to part of the elements in the first tensor.

[0085] For example, refer to Figure 2 For matrix 202, the encoded data corresponding to the 16 columns of elements can be determined as the data to be read, or the encoded data corresponding to every 8 columns of elements can be determined as the data to be read, or the encoded data corresponding to each row of elements can be determined as the data to be read, which is not limited in this embodiment of the present application.

[0086] The coordinate address information of the first tensor may be used to indicate the storage location of the tensor data of the first tensor in the first storage unit 103, and the coordinate address information of the data to be read may be used to indicate the storage location of the data to be read in the first storage unit 103. The coordinate address information of the data to be read may be composed of the coordinate addresses of multiple encoded data in the data to be read. The coordinate address of the encoded data of each element may be used to indicate the storage location of the encoded data of the element in the first storage unit 103.

[0087] The coordinate address may be an identifier of a physical address to point to a specific storage area in the first storage unit 103. For example, the size of each storage area is usually 1 byte, which can store 1 byte (ie 8 bits) of data. The physical address is an absolute fixed address.

[0088] The tensor storage engine 101 can access the data in the first storage unit 103 through the coordinate address. For example, the coordinate address can be used to indicate the storage area of ​​the encoded data in the first storage unit 103, and the tensor storage engine 101 can read the encoded data from the storage area based on the coordinate address.

[0089] In one example, the tensor storage engine 101 is used to obtain control instructions for a first tensor; according to the control instructions, basic information of the first tensor is obtained from the fourth storage unit 106, and the basic information includes at least one of the following: dimension information, coordinate information, arrangement method, and data format; based on the basic information of the first tensor, the coordinate address information of the first tensor is obtained.

[0090] Exemplarily, taking the above-mentioned data to be read as an example, the tensor storage engine 101 is used to obtain control instructions for the data to be read; according to the control instructions, basic information of the data to be read is obtained from the fourth storage unit 106, and the basic information includes at least one of the following: dimension information, coordinate information, arrangement method, data format; based on the basic information of the data to be read, the coordinate address information of the data to be read is obtained.

[0091] The control instruction for the first tensor is used to instruct the tensor storage engine 101 to read the tensor data from the first storage unit 103, and the control instruction for the data to be read is used to instruct the tensor storage engine 101 to read the data to be read from the first storage unit 103. Exemplarily, the tensor storage engine 101 can be used to parse the control instruction, obtain access information of the data to be read, obtain basic information of the data to be read according to the access information of the data to be read, and determine the coordinate address information of the data to be read in the first storage unit 103 based on the basic information of the data to be read.

[0092] The above-mentioned access information is used to indicate the storage unit that the tensor storage engine 101 needs to access, and the data that needs to be accessed in the storage unit that needs to be accessed. For example, the access information of the data to be read can be used to indicate the basic information of the data to be read by the tensor storage engine 101, and the basic information includes at least one of the following: dimension information, coordinate information, arrangement method, and data format. Among them, the dimension information of the data to be read is used to indicate the dimension of the data to be read, the coordinate information of the data to be read is used to indicate the position of the data to be read in the tensor data, such as the coordinates of the first element of the data to be read in the tensor data, the arrangement method of the data to be read is used to indicate the arrangement method of each encoded data in the data to be read, and the data format of the data to be read is used to refer to the data format corresponding to each encoded data in the data to be read.

[0093] The storage unit indicated by the access information of the data to be read is the storage unit where the basic information of the data to be read is stored. In the case where the basic information of the data to be read is distributed in different storage units, the access information of the data to be read may instruct the tensor storage engine 101 to access different storage units to read the basic information of the data to be read.

[0094] The fourth storage unit 106 is used to store basic information of the tensor (including the data to be read). Optionally, the fourth storage unit 106 can be implemented as at least one of the following: a register, a register file, and a constant memory. Figure 3 Since the basic information of the data to be read can be stored in different storage units, the fourth storage unit 106 can include a register file and a constant memory at the same time. The register file can be used to store dimension information (i.e., block_dim), coordinate information (i.e., block_pos), etc., and the constant memory can be used to store tensor descriptors (i.e., tensor_info). Tensor descriptors refer to objects used to describe information such as arrangement mode and data format.

[0095] The tensor storage engine 101 can read the dimension information and coordinate information from the register file and the tensor description from the constant memory according to the access information corresponding to the data to be read, so as to obtain the arrangement mode and data format. Then, based on the dimension information, coordinate information, arrangement mode and data format, the coordinate address information of the data to be read is calculated.

[0096] Exemplarily, the dimension information, arrangement method and data format may be used to determine a calculation method for the coordinate address information, and the coordinate information may be used to calculate the coordinate address information according to the calculation method. For example, the calculation formula for the coordinate address information of the data to be read may be as follows:

[0097] block_global_address=tensor_desc.global_address+block_pos[0]

[0098] +block_pos[1]*tensor_desc.stride[0]

[0099] +block_pos[2]*tensor_desc.stride[1]

[0100] +block_pos[3]*tensor_desc.stride[2]

[0101] +block_pos[4]*tensor_desc.stride[3];

[0102] Among them, block_global_address is the coordinate address information of the data to be read in the first storage unit 103, tensor_desc.global_address is the coordinate address information of the tensor data in the first storage unit 103, block_pos[] refers to the coordinate information of the data to be read (such as the coordinates of the data to be read on a certain dimension), and tensor_desc.stride[] refers to the tensor descriptor of the first tensor (such as the stride indicated by the data format, that is, the number of elements between adjacent indexes on the same dimension).

[0103] In an embodiment of the present application, the tensor storage engine 101 can determine the coordinate address information of the tensor in the first storage unit 103 based on the basic information of the tensor, thereby avoiding a large amount of address calculation and boundary checking, thereby achieving fast access to the tensor.

[0104] 2. The tensor storage engine 101 is also used to adjust the arrangement order of the coordinate addresses included in the coordinate address information to obtain the adjusted coordinate address information.

[0105] The coordinate addresses in the coordinate address information are arranged in sequence according to the arrangement order of the coded data corresponding to the coordinate addresses in the first storage unit 103. Figure 2 , when the encoded data of the matrix 202 is the data to be read (or tensor data), the coordinate addresses of the encoded data of each element column (N0 to N15) are sequentially arranged as follows: the coordinate addresses of the encoded data of the elements of the N0 column, the coordinate addresses of the encoded data of the elements of the N1 column, ..., the coordinate addresses of the encoded data of the elements of the N15 column. The coordinate addresses of the encoded data of each column of elements are composed of the coordinate addresses of the encoded data of each element in the column of elements.

[0106] Optionally, the sorting order of each coordinate address in the adjusted coordinate address information is different from the arrangement order of each coordinate address in the coordinate address information. Figure 2 The adjusted coordinate address information may be: the coordinate address of the encoded data of the N0 column element, the coordinate address of the encoded data of the N4 column element, the coordinate address of the encoded data of the N8 column element, ..., the coordinate address of the encoded data of the N15 column element.

[0107] Optionally, the tensor storage engine 101 may use the coordinate addresses in sequence according to the arrangement order of the coordinate addresses to read the encoded data (i.e., elements) corresponding to each coordinate address from the first storage unit 103, so as to reorder the first tensor during the reading process of the first tensor.

[0108] For example, for the first encoded data and the second encoded data in the data to be read, when the coordinate address information is the first coordinate address and the second coordinate address, the tensor storage engine 101 may first read the first encoded data from the first storage unit 103 according to the first coordinate address, and then read the second encoded data from the first storage unit 103 according to the second coordinate address, and the first coordinate address is used earlier than the second coordinate address.

[0109] When the adjusted coordinate address information is the second coordinate address and the first coordinate address, the tensor storage engine 101 may first read the second encoded data from the first storage unit 103 according to the second coordinate address, and then read the first encoded data from the first storage unit 103 according to the first coordinate address. The second coordinate address is used earlier than the first coordinate address.

[0110] Optionally, the tensor storage engine 101 may store the encoded data groups in the second storage unit 104 in sequence according to the order in which the encoded data are read, so as to adjust the arrangement order of each encoded data in the data to be read in the second storage unit 104. After at least one data to be read is read, the arrangement order of each encoded data in the tensor data in the second storage unit 104 can be adjusted.

[0111] For example, based on the above embodiment, for the coordinate address information, the position of the first encoded data in the second storage unit 104 is before the position of the second encoded data in the second storage unit 104; and for the adjusted coordinate address information, the position of the second encoded data in the second storage unit 104 is before the position of the first encoded data in the second storage unit 104.

[0112] In one example, in order to support the tensor storage engine 101 to reorder the first tensor (i.e., tensor data), the embodiment of the present application newly stores reorder parameters corresponding to the tensor data in the fourth storage unit 106, such as one reorder parameter for each data to be read. The reorder parameters are used to adjust the arrangement order of the coordinate addresses, such as the reorder parameters can be used to adjust the arrangement order of the coordinate addresses of each encoded data in the coordinate address information to obtain the adjusted coordinate address information.

[0113] Exemplarily, the tensor storage engine 101 is also used to obtain a reordering parameter from the fourth storage unit 106, and the reordering parameter is used to adjust the arrangement order of the coordinate address; when the reordering parameter indicates the reordering of the coordinate address, the arrangement order of the coordinate address included in the coordinate address information is adjusted according to the reordering parameter to obtain the adjusted coordinate address information.

[0114] Optionally, the tensor storage engine 101 is further used to obtain the reordering parameters of the first tensor from the fourth storage unit 106 when receiving the control instruction for the first tensor, and the reordering parameters of the first tensor include at least one reordering parameter corresponding to the data to be read. Alternatively, the tensor storage engine 101 is further used to obtain the reordering parameters of the first tensor from the fourth storage unit 106 according to the access information corresponding to the control instruction, that is, the access information is also used to indicate which storage unit the tensor storage engine 101 obtains the reordering parameters from, and this embodiment of the application is not limited to this.

[0115] Optionally, the reordering parameter includes a third parameter, which is used to indicate whether the tensor storage engine 101 reorders the multiple coordinate addresses corresponding to the first tensor. Exemplarily, when the third parameter is a first numerical value, the tensor storage engine 101 reorders the multiple coordinate addresses corresponding to the first tensor; when the third parameter is a second numerical value, the tensor storage engine 101 does not reorder the multiple coordinate addresses corresponding to the first tensor. The embodiment of the present application does not limit the first numerical value and the second numerical value, such as the first numerical value is 1 and the second numerical value is 0. In some embodiments, when the operation result of the first tensor needs to be used for subsequent calculations, the third parameter can be set to the first numerical value. For example, when the operation result of the matrix multiplication operation A*B is used for subsequent calculations, the third parameter corresponding to the matrix B can be set to 1, and the third parameter corresponding to the matrix A can be set to 0. Rearranging coordinate addresses refers to the process of adjusting the arrangement order of coordinate addresses.

[0116] For example, refer to Figure 3, the register stack stores a reorder parameter (reorder_stride) of the first tensor, and the tensor storage engine 101 is used to read the reorder parameter of the first tensor from the register stack when receiving a control instruction for the first tensor, and when the third parameter in the reorder parameter is a first numerical value, adjust the arrangement order of multiple coordinate addresses corresponding to the first tensor according to the reorder parameter to obtain the adjusted coordinate address information of the first tensor, such as adjusting the arrangement order of multiple coordinate addresses corresponding to the data to be read according to the reorder parameter of the data to be read to obtain the adjusted coordinate address information of the data to be read, and after obtaining the adjusted coordinate address information of all the data to be read, the adjusted coordinate address information of the first tensor is obtained.

[0117] The embodiment of the present application can enable the tensor storage engine 101 to rearrange the coordinate addresses based on the reordering parameters, thereby causing multiple elements (i.e., encoded data) corresponding to the first tensor to be rearranged, thereby simplifying the tensor reordering process without any performance loss, which is beneficial to reducing the development complexity of the processor, and further beneficial to reducing the development cost of the processor.

[0118] In one example, the reordering parameters of the first tensor are related to the target data format to which the operation result corresponding to the first tensor needs to be converted, that is, the correspondence between the reordering parameters and the target data format can be set, and the correspondence can be stored in the fourth storage unit 106. The tensor storage engine 101 can also be used to obtain the reordering parameters from the fourth storage unit 106 based on the target data format to which the operation result corresponding to the first tensor needs to be converted, such as determining the reordering parameters corresponding to the target data format as the reordering parameters of the first tensor, thereby reading the reordering parameters corresponding to the first tensor from the fourth storage unit 106.

[0119] The embodiment of the present application does not limit the target data format, such as the target data format includes at least one of the following: FP16, FP8, FP4. For example, when the target data format is FP16, the tensor storage engine 101 is used to determine the reordering parameters corresponding to the FP16 data format in the fourth storage unit 106 as the reordering parameters of the first tensor. In this way, according to the target data format, the reordering parameters can be accurately obtained to avoid the coordinate address from being incorrectly reordered. The target data format corresponding to the first tensor can be set and adjusted based on actual usage requirements, and the embodiment of the present application does not limit this.

[0120] In the embodiment of the present application, the operation result corresponding to the first tensor may refer to the operation result obtained by the tensor computing engine 102 performing a logical operation on the rearranged first tensor (i.e., the above-mentioned rearranged tensor). Performing a logical operation on the rearranged tensor is essentially performing a logical operation on the tensor data of the rearranged tensor, and the tensor data of the rearranged tensor is obtained by combining the encoded data of each element in the rearranged tensor according to the second arrangement order.

[0121] In an example, the reordering parameters may also include a first parameter and a second parameter, wherein the first parameter is used to indicate the relationship between the total number of bits in the data format of the operation result and the total number of bits in the target data format, and the second parameter is used to indicate the reordering granularity of the first tensor.

[0122] Optionally, the quotient between the total number of bits in the data format of the operation result and the total number of bits in the target data format may be used as the first parameter. For example, when the data format of the operation result is FP32 and the target data format is FP16, 32 / 16=2 may be used as the first parameter.

[0123] The reordering granularity is used to indicate the span of the operation result in a certain dimension and the span involved in a reordering, such as the number of elements, rows, or columns in a certain dimension of the first tensor involved in each reordering. Figure 2 For matrix 203, when the data format of the operation result is FP32 and the target data format is FP8, under the first dimension of matrix 203 (i.e., dim0, the data change direction indicated by M), 16 columns of elements are packed at one time, that is, the 16 columns of elements under the first dimension of matrix 202 need to be rearranged so that after packing, the arrangement order of the operation results remains: the operation result of the encoded data of the N0 column elements, the operation result of the encoded data of the N4 column elements, the operation result of the encoded data of the N8 column elements, ..., the operation result of the encoded data of the N15 column elements, and the span is 16, so 16 can be determined as the second parameter.

[0124] Exemplarily, taking the data format of the operation result as FP32 as an example, the setting of the rearrangement parameter may include at least one of the following:

[0125] 1. When the target data format is FP4, the first parameter is 8 and the second parameter is 32.

[0126] 2. When the target data format is FP8, the first parameter is 4 and the second parameter is 16.

[0127] 3. When the target data format is FP16, the first parameter is 2 and the second parameter is 8.

[0128] In one example, after obtaining the rearrangement parameters, the process of obtaining the adjusted coordinate address information may include the following:

[0129] (1) The tensor storage engine 101 is used to obtain a first intermediate parameter and a second intermediate parameter for each coordinate address in the coordinate address information according to an arrangement position of the coordinate address in the coordinate address information and a second parameter.

[0130] Optionally, based on the packing method of the operation results, in the process of rearranging the first tensor, the first tensor may be rearranged as a single or entire row or entire column element in a certain dimension. Figure 2 For the matrix 202, since the calculation result is packed in the first dimension, the entire column elements (i.e., N0 to N15) of the matrix 202 in the second dimension can be rearranged as units, or the single row elements of the matrix 202 in the first dimension can be rearranged, which is not limited in this embodiment of the present application.

[0131] Therefore, when obtaining the adjusted coordinate address information, if the above-mentioned data to be read is all the encoded data of the matrix 202 (i.e., the first tensor), the tensor storage engine 101 can be used to rearrange the coordinate addresses within the rearrangement granularity (such as N0 to N15) with the coordinate addresses corresponding to the entire column of elements as units. If the above-mentioned data to be read is the encoded data corresponding to a row of elements of the matrix 202 (i.e., the first tensor), the tensor storage engine 101 can be used to rearrange the coordinate addresses corresponding to each element in a row of elements each time within the rearrangement granularity (such as N0 to N15).

[0132] For example, refer to Figure 2 , the coordinate address information of the matrix 202 is: the coordinate address of the N0 column element, the coordinate address of the N1 column element, ..., the coordinate address of the N15 column element, and the arrangement positions of each coordinate address in the coordinate address information are: 0, 1, 2, ..., 15.

[0133] The arrangement position of the coordinate address can be used to indicate the order in which the corresponding encoded data is read using the coordinate address. The order in which the encoded data is obtained can be adjusted by adjusting the arrangement position of the coordinate address in the coordinate address information, thereby adjusting the storage order of the encoded data and further rearranging the encoded data. Therefore, the rearrangement of the tensor can be achieved by re-arranging the arrangement order of the coordinate address.

[0134] The first intermediate parameter and the second intermediate parameter can be used to adjust the arrangement position of the coordinate address to adjust the arrangement order of each coordinate address in the coordinate address information. For example, the calculation formula of the first intermediate parameter can be expressed as follows:

[0135] uint32_t outer=source_pos / outer_stride*outer_stride;

[0136] Among them, outer is the first intermediate parameter, source_pos is the arrangement position of the coordinate address (such as 0 to 15 corresponding to the matrix 202), outer_stride is the second parameter (such as 16 corresponding to the matrix 202), and uint32_t represents an unsigned 32-bit integer.

[0137] For example, refer to Figure 2 , the arrangement position of the coordinate address corresponding to the N0 column element is 0, then outer=0 / 16*16=0, that is, the arrangement position corresponding to the N0 column element belongs to the 0th rearrangement granularity (i.e., N0 to N15); the arrangement position of the coordinate address corresponding to the N1 column element is 1, then outer=1 / 16*16=0, that is, the coordinate address corresponding to the N1 column element also belongs to the 0th rearrangement granularity (i.e., N0 to N15), and so on. It can be determined that the coordinate addresses corresponding to the N0 column elements to the N15 column elements all belong to the 0th rearrangement granularity, that is, the tensor storage engine 101 needs to arrange the coordinate addresses corresponding to the N0 column elements to the N15 column elements in the same rearrangement process.

[0138] Optionally, the calculation formula of the second intermediate parameter can be expressed as follows:

[0139] uint32_t inner=source_pos%outer_stride;

[0140] Among them, inner is the second intermediate parameter, source_pos is the arrangement position of the coordinate address (such as 0 to 15 corresponding to the matrix 202), and outer_stride is the second intermediate parameter (such as 16 corresponding to the matrix 202).

[0141] For example, refer to Figure 2 , the arrangement position of the coordinate address corresponding to the element in the N0 column is 0, then inner=0%16=0, that is, the coordinate address corresponding to the element in the N0 column ranks 0th in the 0th rearrangement granularity; the arrangement position of the coordinate address corresponding to the element in the N1 column is 1, then inner=1%16=1, that is, the coordinate address corresponding to the element in the N1 column ranks 1st in the 0th rearrangement granularity, the arrangement position of the coordinate address corresponding to the element in the N2 column is 2, then inner=2%16=2, that is, the coordinate address corresponding to the element in the N2 column ranks 2nd in the 0th rearrangement granularity, and so on. The coordinate addresses corresponding to the elements in the N0 column to the N15 column can be determined, and their respective numbers of bits in the 0th rearrangement granularity.

[0142] (2) The tensor storage engine 101 is used to obtain a third intermediate parameter based on the first parameter, the first intermediate parameter, the second intermediate parameter and the second parameter, where the third intermediate parameter is used to indicate the arrangement position of the coordinate address in the adjusted coordinate address information; and obtain the adjusted coordinate address information based on the third intermediate parameters corresponding to each coordinate address contained in the coordinate address information.

[0143] Optionally, for the first coordinate address in the coordinate address information, the third intermediate parameter corresponding to the first coordinate address can be used to indicate the arrangement position of the second coordinate address to which the first coordinate address needs to be replaced, that is, based on the third intermediate parameter, the arrangement position of the second coordinate address can be replaced with the arrangement position of the first coordinate address. The first coordinate address can refer to any coordinate address in the coordinate address information.

[0144] Exemplarily, the calculation formula of the third intermediate parameter can be expressed as follows:

[0145] uint32_t target=(inner*inner_stride) / outer_stride+(inner*inner_stride)%outer_stride+outer; wherein target is the third intermediate parameter.

[0146] For example, refer to Figure 2 , the arrangement position of the coordinate address corresponding to the N0 column element is 0, then the third intermediate parameter corresponding to the coordinate address of the N0 column element is: target = (0*4) / 16 + (0*4)% 16 + 0 = 0, that is, the arrangement position of the coordinate address corresponding to the N0 column element remains unchanged (that is, the reading order is ranked first). The arrangement position of the coordinate address corresponding to the N1 column element is 1, then the third intermediate parameter of the coordinate address of the N1 column element is: target = (1*4) / 16 + (1*4)% 16 + 0 = 4, the arrangement position of the coordinate address of the N4 column element is 4, that is, the arrangement position of the coordinate address of the N4 column element is updated to 1 (that is, the reading order is ranked second). The arrangement position of the coordinate address corresponding to the N2 column element is 2, then the third intermediate parameter corresponding to the coordinate address of the N2 column element is: target = (2 * 4) / 16 + (2 * 4)% 16 + 0 = 8, the arrangement position of the coordinate address of the N8 column element is 8, that is, the arrangement position of the coordinate address of the N8 column element is updated to 3 (that is, the reading order is ranked third), and so on, the third parameters are as follows: 0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15, then the adjusted coordinate address information is: the coordinate address of the N0 column element, the coordinate address of the N4 column element, the coordinate address of the N8 column element, the coordinate address of the N12 column element, ..., the coordinate address of the N15 column element. It should be noted that the third intermediate parameter is calculated based on the arrangement position of the coordinate address in the coordinate address information, rather than the arrangement position of the coordinate address in the adjusted coordinate address information.

[0147] The embodiment of the present application can avoid affecting the calculation result by rearranging the coordinate addresses according to single, entire row or entire column elements of the tensor in a certain dimension.

[0148] In one example, the re-arrangement of the coordinate addresses is related to the arrangement of the first tensor. The arrangement of the first tensor refers to the arrangement of each element in the first tensor, such as the arrangement of the first tensor includes at least one of the following: none-interleave and interleave.

[0149] The difference between non-interleaved and interleaved arrangements lies mainly in the way elements are stored and accessed. When the arrangement of the first tensor is non-interleaved, the encoded data of the elements are stored in their natural order in memory, usually from left to right and from top to bottom. When the arrangement of the first tensor is interleaved, the encoded data of the elements are interleaved according to a specific pattern, rather than sorted in natural order, to optimize computing efficiency or adapt to specific hardware architectures.

[0150] Exemplarily, the tensor storage engine 101 is used to adjust the arrangement order of the coordinate addresses included in the coordinate address information according to the first dimension corresponding to the first tensor based on the rearrangement parameter when the arrangement manner of the first tensor is an interlaced arrangement, so as to obtain the adjusted coordinate address information.

[0151] Optionally, the first dimension may be a dimension in which the data changes in the horizontal direction, denoted as dim0. Figure 4 , for matrix 401 (such as the first tensor), the continuous change direction of K elements in each element row is the data change direction of the first dimension. According to the first dimension, matrix 401 can be divided into multiple element rows. If the arrangement mode of matrix 401 in the first storage unit 103 is non-interlaced arrangement, dim0 is M (i.e. corresponding to M element rows), dim1 is K (K element columns), if its arrangement mode in the first storage unit 103 is interlaced arrangement, and the size of the interlaced arrangement X is K / 2, then dim0 is M, dim1 is (k0, k1), and matrix 401 is divided into two sub-matrices (2 M*X) for storage, dim1 as a whole includes two parts, such as storing k0 first and then storing k1.

[0152] For example, refer to Figure 5, the arrangement of the matrix 501 is staggered. According to the first dimension dim0, the matrix 501 can be divided into 9 element rows, and the arrangement positions of the coordinate addresses of the 9 element rows are 0 to 8. When the first parameter of the matrix 501 is 2 and the second parameter is 8 (i.e., the rearrangement granularity is 8), according to 2 and 8, the arrangement positions of the coordinate addresses of the first 8 element rows (i.e., the 0th rearrangement granularity) can be adjusted to: 0, 2, 4, 6, 1..., 7. Since the element row 8 is the first bit in the first rearrangement granularity, the arrangement position of the coordinate address of the element row 8 remains unchanged. That is, the adjusted coordinate address information of the matrix 501 is: the coordinate address of the element row 0, the coordinate address of the element row 2, the coordinate address of the element row 4,..., the coordinate address of the element row 7, and the coordinate address of the element row 8.

[0153] Optionally, the tensor storage engine 101 is also used to adjust the arrangement order of the coordinate addresses included in the coordinate address information according to the second dimension corresponding to the first tensor based on the rearrangement parameter when the arrangement manner of the first tensor is non-interleaved, so as to obtain the adjusted coordinate address information.

[0154] The data change direction corresponding to the first dimension is different from the data change direction corresponding to the second dimension. Exemplarily, the data change direction corresponding to the first dimension is perpendicular to the data change direction corresponding to the second dimension. Optionally, the second dimension may refer to a dimension in which the data change direction is a vertical direction, denoted as dim1.

[0155] For example, refer to Figure 2 For the matrix 202 (such as the data to be read), if the arrangement mode of the matrix 202 in the first storage unit 103 is non-interlaced arrangement, dim0 is K (K element rows) and dim1 is N (N element columns). The elements in the vertical direction (i.e., N direction) are continuous as the second dimension. According to the second dimension, the matrix 202 can be divided into a plurality of element columns (i.e., N0 column elements to N15 column elements). According to the rearrangement parameter, the coordinate address of each group of coded data in the matrix 202 can be adjusted with the element column as a unit to obtain the adjusted coordinate address information of the matrix 202. The adjusted coordinate address information is: the coordinate address of the N0 column element, the coordinate address of the N4 column element, the coordinate address of the N8 column element, the coordinate address of the N12 column element, ..., the coordinate address of the N15 column element.

[0156] The embodiment of the present application can avoid affecting the calculation results and improve the efficiency of coordinate address re-arrangement by re-arranging the coordinate addresses according to the entire row or column elements of the tensor in a certain dimension.

[0157] 3. The tensor storage engine 101 is further used to read multiple elements from the first storage unit 103 to the second storage unit 104 in sequence according to the arrangement order of each coordinate address contained in the adjusted coordinate address information, so as to obtain a rearranged tensor.

[0158] Optionally, the encoded data contained in the tensor data of the first tensor is stored in the second storage unit 104 according to the adjusted coordinate address information to form reordered data corresponding to the reordered tensor, and the arrangement order of each encoded data in the reordered data is consistent with the arrangement sequence of the coordinate address of each encoded data in the adjusted coordinate address information, that is, the above-mentioned second arrangement order.

[0159] For example, refer to Figure 2 and Figure 7 For the matrix 202, the tensor storage engine 101 can read the following from the first storage unit 103 in sequence based on the adjusted coordinate address information: the encoded data of the N0 column elements, the encoded data of the N4 column elements, the encoded data of the N8 column elements, the encoded data of the N12 column elements, ..., the encoded data of the N15 column elements. The tensor storage engine 101 is used to store the encoded data of each column element in the second storage unit 104 in sequence based on the adjusted coordinate address information, and the rearrangement matrix 701 (i.e., the rearrangement tensor) corresponding to the matrix 202 can be obtained. The rearrangement data corresponding to the rearrangement matrix 701 are: the encoded data of the N0 column elements, the encoded data of the N4 column elements, the encoded data of the N8 column elements, the encoded data of the N12 column elements, ..., the encoded data of the N15 column elements.

[0160] In one example, the tensor storage engine 101 is also used to fill L preset elements after multiple elements in the second storage unit 104 when the number of groups of multiple elements is less than a second parameter; wherein L is the difference between the second parameter and the number of groups of multiple elements, and L is a positive integer.

[0161] That is, when the number of groups of coded data contained in the tensor data of the first tensor is less than the second parameter, L preset coded data (ie, coded data of preset elements) are filled after the rearranged data in the second storage unit 104 .

[0162] Optionally, the embodiment of the present application does not limit the division method of the number of groups of multiple elements (ie, multiple coded data). Figure 2 , when the first tensor includes multiple coded data corresponding to element row 0 in the matrix 202, the number of elements in element row 0 can be determined as the number of groups. When the first tensor includes multiple coded data corresponding to element rows 0 to element rows K in the matrix 202, the number of element rows can be determined as the number of groups in units of element rows.

[0163] For example, refer to Figure 6 , the arrangement of matrix 601 is staggered arrangement, dim0 of matrix 601 is M, dim1 is (k0, k1, k2). Matrix 601 can be divided into multiple element rows according to the first dimension. When dim1=0 or 1 or 2, the tensor storage engine 101 needs to rearrange the arrangement order of the coordinate addresses of element rows 0 to element rows 6 (that is, the number of groups is 7). If the first parameter is 2 and the second parameter is 8, the adjusted coordinate address information can be obtained based on 2 and 8: the coordinate address of element row 0, the coordinate address of element row 2, ..., the coordinate address of element row 5. However, the second parameter indicates that the rearrangement granularity of matrix 601 is 8, then after the encoded data of element row 0 to element row 6 are moved to the second storage unit 104, 8-7=1 preset encoded data can be filled after the encoded data of element row 5 to form element row 7. This ensures that there is enough data for packaging each time in the subsequent process.

[0164] The embodiment of the present application does not limit the preset elements, which can be set and adjusted according to actual usage requirements, such as 0.

[0165] In one example, the tensor storage engine 101 is further configured to update the asynchronous barrier unit after completing the reading of the first tensor, so as to end the reading of the first tensor by triggering an interrupt through the asynchronous barrier unit.

[0166] Among them, the asynchronous barrier unit is a unit constructed based on the asynchronous barrier mechanism (Async Barrier). Optionally, after completing the reading of the last encoded data corresponding to the first tensor, the tensor storage engine 101 sends an interrupt signal to the asynchronous barrier unit, and the asynchronous barrier unit interrupts the tensor storage engine 101 from reading the first tensor from the first storage unit 103 based on the interrupt signal. Reading the first tensor is essentially reading the tensor data of the first tensor.

[0167] To summarize, the technical solution provided by the embodiments of the present application, in the process of the tensor storage engine reading the first tensor from the first storage unit to the second storage unit, supports the tensor storage engine to automatically rearrange multiple elements in the first tensor, so that the third arrangement order of the calculation results of the multiple elements in the third storage unit is the same as the first arrangement order of the multiple elements, without the need to develop additional software design to support additional rearrangement processes, thereby reducing the complexity of the tensor processing process in the processor, avoiding the development cost caused by software design, and thereby reducing the development cost of the processor.

[0168] In addition, since the present application implements the reordering of tensors at the existing hardware level, it can avoid the impact of adding a reordering process on the end-to-end performance of the processor, thereby helping to improve the end-to-end performance of the processor.

[0169] In some embodiments, after obtaining the rearranged tensor of the first tensor (ie, the rearranged data), the embodiments of the present application may further include the following contents.

[0170] 1. The tensor calculation unit 102 is used to perform logical operations on the rearranged tensor to obtain operation results.

[0171] Performing logical operations on the shuffled tensor actually performs logical operations on the shuffled data of the shuffled tensor.

[0172] For example, refer to Figure 7 The tensor calculation unit 102 is used to perform a matrix multiplication operation on the encoded data of the matrix 201 and the rearranged data of the rearranged matrix 701 to obtain an operation result 702 corresponding to the rearranged matrix 701, and the operation result 702 includes M*N multiplication and accumulation operation results.

[0173] The tensor calculation unit 102 may store the operation result 702 in the third storage unit 105. Optionally, when the third storage unit 105 is implemented as a register file, the third storage unit 105 may include 4 registers: register R0 to register R3.

[0174] The M*N multiplication-accumulation operation results are stored in registers R0 to R3 respectively, and each register stores 8*4 multiplication-accumulation operation results. For example, the 8 multiplication-accumulation operation results corresponding to the elements of column N0, the 8 multiplication-accumulation operation results corresponding to the elements of column N1, the 8 multiplication-accumulation operation results corresponding to the elements of column N2, and the 8 multiplication-accumulation operation results corresponding to the elements of column N3 are stored in register R0 in sequence.

[0175] Optionally, the data format of each multiplication-accumulation operation result is FP32, that is, each multiplication-accumulation operation result corresponds to 32 bits of data.

[0176] 2. The format conversion unit is used to convert the data format of each element in the calculation result into the target data format to obtain the calculation result after format conversion.

[0177] For example, refer to Figure 7 The SIMT unit is used to convert the data format of M*N multiplication and accumulation operation results from FP32 to FP16, and obtain M*N multiplication and accumulation operation results after format conversion.

[0178] In an example, the format conversion unit is used to convert the data format of the result elements corresponding to each thread in four registers into FP8 through multiple threads when the data format of the operation result is FP32.

[0179] Optionally, each element in the result of an operation can be referred to as a result element. Figure 7, the M*N multiplication-accumulation operation results are the result elements corresponding to the operation result 702. In the case where the fourth storage unit 105 is implemented as a register file, the third storage unit 105 may include 4 registers.

[0180] For example, reference Figure 7 The SIMT unit can simultaneously start 32 threads: T0 to T31, and convert the data format of M*N multiplication and accumulation operation results from FP32 to FP8 through 32 threads in parallel. 32 threads can perform the data format conversion of 32 multiplication and accumulation operation results in parallel each time, that is, it can complete the data format conversion of M*N multiplication and accumulation operation results by executing it once for each register.

[0181] In one example, the format conversion unit is further used to convert the data format of the result elements corresponding to each thread in the four registers into FP16 through multiple threads when the data format of the operation result is FP32.

[0182] For example, reference Figure 7 The SIMT unit can simultaneously start 32 threads: T0 to T31, and convert the data format of M*N multiplication and accumulation operation results from FP32 to FP16 through 32 threads in parallel. 32 threads can perform the data format conversion of 32 multiplication and accumulation operation results in parallel each time, that is, it can complete the data format conversion of M*N multiplication and accumulation operation results by executing it once for each register.

[0183] 3. The format conversion unit is further used to store the calculation result after format conversion into the third storage unit 105.

[0184] Optionally, the format conversion unit stores the operation result after format conversion in the third storage unit 105 according to the storage granularity of the third storage unit 105 .

[0185] The embodiment of the present application does not limit the storage granularity of the third storage unit 105. Optionally, the storage granularity of the third storage unit 105 includes 32 bits. For example, when the third storage unit 105 is implemented as a register file, the storage granularity of each register is 32 bits.

[0186] In an example, when the storage granularity of the third storage unit 105 is 32 bits and the target data format is FP8, the format conversion unit is used to pack the result elements corresponding to each thread in 4 registers into compressed data according to the storage granularity of the registers through multiple threads in parallel, so as to be stored in any of the 4 registers in sequence.

[0187] For example, reference Figure 2Since the data size of the multiplication and accumulation operation result after the data format conversion is 8 bits, the 4 multiplication and accumulation operation results after the data format conversion can be packed together to obtain 32-bit compressed data. The 32-bit compressed data can be stored at the storage granularity of the register, so the 8*16 multiplication and accumulation operation results after the data format conversion can be packed into 32*32-bit compressed data. The 32*32-bit compressed data can be packed into the same register through 32 threads at the same time.

[0188] In an example, when the storage granularity of the third storage unit 105 is 32 bits and the target data format is FP16, the format conversion unit is used to pack the result elements corresponding to each thread in two registers into compressed data according to the storage granularity of the registers through multiple threads in parallel, so as to be stored in any one of the two registers in sequence.

[0189] For example, reference Figure 7 Since the data size of the multiplication and accumulation result after data format conversion is 16 bits, the two multiplication and accumulation results after data format conversion can be packed together to obtain 32-bit compressed data. The 32-bit compressed data can be stored at the storage granularity of the register. Therefore, the 8*16 multiplication and accumulation results after data format conversion can be packed into 64*32-bit compressed data. The 64*32-bit compressed data can be divided into two 32*32-bit compressed data. 32 threads can pack the two 32*32-bit compressed data into two registers in two times.

[0190] Optionally, the operation result after format conversion includes the operation results of each of the multiple elements in the first tensor, and the third arrangement order of the operation results of each of the multiple elements in the third storage unit 105 is the same as the first arrangement order. The operation result of the element can be referred to as the result element in the operation result after format conversion. Optionally, the operation result corresponding to each element column in the first tensor is the operation result corresponding to each element in the element column.

[0191] For example, refer to Figure 7, when the storage granularity of the third storage unit 105 is 32 bits and the target data format is FP16, for the first row of result elements in the rearrangement matrix 701, T0 can pack the result elements of the N0 column elements and the result elements of the N1 column elements into a 32-bit compressed data, T1 can pack the result elements of the N2 column elements and the result elements of the N3 column elements into a 32-bit compressed data, T2 can pack the result elements of the N4 column elements and the result elements of the N5 column elements into a 32-bit compressed data, T3 can pack the result elements of the N6 column elements and the result elements of the N7 column elements into a 32-bit compressed data, and so on. The third arrangement order of the first row of result elements after being packed is: result elements of the N0 column elements, result elements of the N1 column elements, result elements of the N2 column elements, …, result elements of the N15 column elements, which is the same as the first arrangement order of the encoded data of the N0 column elements to the N15 column elements in the first storage unit 103.

[0192] To summarize, the technical solution provided in the embodiment of the present application, by supporting the tensor storage engine 101 to rearrange tensors according to the rearrangement parameters when reading tensors, can make the third arrangement order of the packed result elements corresponding to the tensor consistent with the first arrangement order of the tensor in the first storage unit 103, without the need to develop additional software design to support additional rearrangement processes, thereby reducing the complexity of the tensor processing process in the processor, avoiding the development cost caused by software design, and further reducing the development cost of the processor.

[0193] In some embodiments, reference Figure 8 , taking the processor implemented as a graphics processor as an example. The graphics processor 800 may include multiple processor clusters (PCs) 801 (e.g., 4), each processor cluster 801 may include multiple processor execution engines (PXs) 802 (e.g., 4), and each processor execution engine 802 may include multiple processors (Ps) 803 (e.g., 2).

[0194] The multiple processor execution engines 802 in each processor cluster 801 share an L2 cache 804 (ie, the first storage unit 103 ). The main memory of the graphics processor 800 is implemented as an L3 cache 805 .

[0195] refer to Fig. 9 Each processor 803 may include: a tensor computing engine 8031, a tensor storage engine 8032, an L1 cache 8033 (ie, the above-mentioned second storage unit 104) and a register 8034 (ie, the above-mentioned third storage unit 105).

[0196] The L2 cache 804 stores a plurality of encoded data (ie, tensor data) corresponding to the first tensor, each encoded data is generated by an element in the first tensor according to the FP8 data format, that is, each encoded data includes 8 bits of data.

[0197] The tensor storage engine 8032 is used to parse the control instruction after receiving the control instruction for the first tensor, obtain access information of the first tensor, and determine the coordinate address information of the first tensor in the L2 cache 804 according to the access information.

[0198] The tensor storage engine 8032 is further configured to read the reordering parameters of the first tensor from the register 8035 (the fourth storage unit 106 ) after receiving a control instruction for the first tensor.

[0199] The tensor storage engine 8032 is further used to rearrange the arrangement positions of the coordinate addresses in the coordinate address information corresponding to the first tensor according to the rearrangement parameter of the first tensor, so as to obtain the adjusted coordinate address information corresponding to the first tensor.

[0200] The tensor storage engine 8032 is also used to read each encoded data corresponding to the first tensor from the L2 cache 804 in sequence according to the arrangement order of each coordinate address contained in the adjusted coordinate address information, and to store each encoded data in the L1 cache 8033 in sequence to obtain the rearranged data corresponding to the first tensor.

[0201] The tensor computing engine 8031 ​​is used to read the rearranged data corresponding to the first tensor from the L1 cache 8033 , and perform a logical operation on the rearranged data corresponding to the first tensor to obtain an operation result; and store the operation result in the register 8034 .

[0202] The SIMT unit in the processor 803 is used to perform data format conversion on the operation result to obtain the operation result after format conversion, and repack the operation result after format conversion into the register 8034 according to the storage granularity of 32 bits to obtain compressed data corresponding to the operation result.

[0203] The arrangement order of the operation results corresponding to each element in the compressed data in the register 8034 is consistent with the arrangement order of the encoded data of each element in the L2 cache 804. The compressed data can be directly used in subsequent calculation processes without adjustment.

[0204] In some embodiments, taking a matrix as an example, the instructions involved in the tensor reordering process of the tensor storage engine are shown in Table 1 below.

[0205] Table 1

[0206]

[0207] In an embodiment of the present application, the tensor storage engine can determine the coordinate address of the tensor data of the first tensor in the L2 cache 804 based on block_pos, block_dim and tensor_desc, and reorder the arrangement position of the coordinate address corresponding to the first tensor based on reorder_stride to achieve the reordering of the first tensor in the L1 cache 8033.

[0208] To summarize, the technical solution provided in the embodiments of the present application, by supporting the tensor storage engine 101 to rearrange tensors according to the rearrangement parameters when reading tensors, can make the arrangement order of the packed result elements corresponding to the tensors consistent with the arrangement order of the tensors in the first storage unit 103, without the need to develop additional software designs to support additional rearrangement processes, thereby reducing the complexity of the tensor processing process in the processor, avoiding the development costs caused by software design, and further reducing the development costs of the processor.

[0209] The following is an embodiment of the method of the present application. For details not disclosed in the embodiment of the method of the present application, please refer to the above embodiment.

[0210] Please refer to Fig.10 , which shows a flowchart of a tensor processing method applied to a processor provided in a possible implementation of the present application. The components of the processor can be found in the description of the above embodiment. Fig.10 As shown, the method may include the following steps:

[0211] Step 1001: A tensor storage engine reads a plurality of elements from a first storage unit to a second storage unit in a second arrangement order to obtain a rearranged tensor, wherein the second arrangement order is different from the first arrangement order.

[0212] In one example, a tensor storage engine obtains coordinate address information of a first tensor, the coordinate address information including coordinate addresses of multiple elements in a first storage unit; adjusts an arrangement order of the coordinate addresses included in the coordinate address information to obtain adjusted coordinate address information; and reads multiple elements from the first storage unit to the second storage unit in sequence according to the arrangement order of each coordinate address included in the adjusted coordinate address information to obtain a rearranged tensor.

[0213] In one example, the processor also includes a fourth storage unit; the tensor storage engine obtains a reordering parameter from the fourth storage unit, and the reordering parameter is used to adjust the arrangement order of the coordinate address; when the reordering parameter indicates the reordering of the coordinate address, the arrangement order of the coordinate address included in the coordinate address information is adjusted according to the reordering parameter to obtain the adjusted coordinate address information.

[0214] In one example, the reordering parameter includes a first parameter and a second parameter, the first parameter is used to indicate the relationship between the total number of bits of the data format of the operation result and the total number of bits of the target data format, and the second parameter is used to indicate the reordering granularity of the first tensor;

[0215] For each coordinate address in the coordinate address information, the tensor storage engine obtains a first intermediate parameter and a second intermediate parameter according to the arrangement position of the coordinate address in the coordinate address information and the second parameter, wherein the first intermediate parameter is used to indicate the reordering granularity to which the coordinate address belongs, and the second intermediate parameter is used to indicate the number of bits of the coordinate address in the reordering granularity to which the coordinate address belongs; obtains a third intermediate parameter according to the first parameter, the first intermediate parameter, the second intermediate parameter, and the second parameter, wherein the third intermediate parameter is used to indicate the arrangement position of the coordinate address in the adjusted coordinate address information; obtains the adjusted coordinate address information based on the third intermediate parameters corresponding to each coordinate address contained in the coordinate address information.

[0216] In one example, when the target data format is FP4, the first parameter is 8 and the second parameter is 32; or, when the target data format is FP8, the first parameter is 4 and the second parameter is 16; or, when the target data format is FP16, the first parameter is 2 and the second parameter is 8.

[0217] In one example, the tensor storage engine obtains the reordering parameter from the fourth storage unit based on the target data format to which the operation result needs to be converted.

[0218] In one example, a tensor storage engine is used to adjust the arrangement order of coordinate addresses included in the coordinate address information according to a first dimension corresponding to the first tensor, when the arrangement mode of the first tensor is an interlaced arrangement, to obtain adjusted coordinate address information according to a rearrangement parameter; or, when the arrangement mode of the first tensor is a non-interlaced arrangement, adjust the arrangement order of coordinate addresses included in the coordinate address information according to a second dimension corresponding to the first tensor, according to a rearrangement parameter, to obtain adjusted coordinate address information; wherein the data change direction corresponding to the first dimension is different from the data change direction corresponding to the second dimension.

[0219] In one example, when the number of groups of multiple elements is less than a second parameter, the tensor storage engine fills L preset elements after the multiple elements in the second storage unit; wherein the second parameter is used to indicate the reordering granularity of the first tensor, L is the difference between the second parameter and the number of groups of multiple elements, and L is a positive integer.

[0220] In one example, the processor includes a fourth storage unit; the tensor storage engine obtains a control instruction for a first tensor; according to the control instruction, basic information of the first tensor is obtained from the fourth storage unit, and the basic information includes at least one of the following: dimension information, coordinate information, arrangement method, and data format; based on the basic information of the first tensor, the coordinate address information of the first tensor is obtained.

[0221] In one example, after completing the reading of the first tensor, the tensor storage engine updates the asynchronous barrier unit to end the reading of the first tensor by triggering an interrupt through the asynchronous barrier unit.

[0222] Step 1002: The tensor computing engine performs a logical operation on the rearranged tensor to obtain a calculation result.

[0223] Step 1003, the format conversion unit converts the data format of each element in the calculation result into the target data format, obtains the calculation result after format conversion, and stores the calculation result after format conversion into the third storage unit; wherein, the calculation result after format conversion includes the calculation results of each of the multiple elements, and the third arrangement order of the calculation results of each of the multiple elements in the third storage unit is the same as the first arrangement order.

[0224] To summarize, the technical solution provided by the embodiments of the present application, in the process of the tensor storage engine reading the first tensor from the first storage unit to the second storage unit, supports the tensor storage engine to automatically rearrange multiple elements in the first tensor, so that the third arrangement order of the calculation results of the multiple elements in the third storage unit is the same as the first arrangement order of the multiple elements, without the need to develop additional software design to support additional rearrangement processes, thereby reducing the complexity of the tensor processing process in the processor, avoiding the development cost caused by software design, and thereby reducing the development cost of the processor.

[0225] In addition, since the present application implements the reordering of tensors at the existing hardware level, it can avoid the impact of adding a reordering process on the end-to-end performance of the processor, thereby helping to improve the end-to-end performance of the processor.

[0226] Please refer to Fig.11 , which shows a simplified structural block diagram of a computer device provided by an embodiment of the present application. The computer device 1100 can be any electronic device with data calculation, processing and storage functions.

[0227] The computer device 1100 includes a processor 1101 and a memory 1102 .

[0228] The processor 1101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1101 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1101 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1101 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1101 may also include an AI processor for processing computing operations related to machine learning.

[0229] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1102 is used to store a computer program, which is configured to be executed by one or more processors.

[0230] Those skilled in the art will understand that Fig.11 The structure shown in the figure does not constitute a limitation on the computer device 1100, and the computer device 1100 may include more or less components than shown in the figure, or combine some components, or adopt a different component arrangement.

[0231] In some embodiments, a chip product is also provided, the chip product includes the processor described above. Optionally, the chip product may be a GPU chip product, the processor described above is a GPU, and the GPU chip product includes the GPU described above. Optionally, the above chip product may be implemented as a graphics card, and the graphics card includes the processor described above, such as a GPU.

[0232] In some embodiments, a computer device is also provided, the computer device including the processor described above. Optionally, the computer device may be a personal computer, a workstation, a game console, and some mobile devices (such as a tablet computer, a smart phone, etc.), or a vehicle-mounted terminal device, a smart home device, a smart TV, an intelligent robot, etc., or a server, a server cluster, an artificial intelligence computing cluster, a cloud computing cluster, etc., wherein the artificial intelligence computing cluster may also be referred to as an intelligent computing cluster or a smart computing cluster, which is not limited in this application.

[0233] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0234] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A processor, characterized in that: The processor includes: a tensor storage engine, a tensor calculation engine, a format conversion unit, a first storage unit, a second storage unit, and a third storage unit; a first tensor is stored in the first storage unit, and the first tensor includes a plurality of elements arranged in a first arrangement order; The tensor storage engine is used to read the plurality of elements from the first storage unit to the second storage unit in a second arrangement order to obtain a rearranged tensor, wherein the second arrangement order is different from the first arrangement order; The tensor computing engine is used to perform a logical operation on the rearranged tensor to obtain an operation result; The format conversion unit is used to convert the data format of each element in the operation result into a target data format, obtain the operation result after format conversion, and store the operation result after format conversion into the third storage unit; The calculation result after the format conversion includes the calculation results of each of the multiple elements, and the third arrangement order of the calculation results of each of the multiple elements in the third storage unit is the same as the first arrangement order.

2. The processor according to claim 1, characterized in that The tensor storage engine is used to: Acquire coordinate address information of the first tensor, where the coordinate address information includes coordinate addresses of the plurality of elements in the first storage unit; Adjusting the arrangement order of the coordinate addresses included in the coordinate address information to obtain adjusted coordinate address information; According to the arrangement order of each coordinate address included in the adjusted coordinate address information, the multiple elements are read from the first storage unit to the second storage unit in sequence to obtain the rearranged tensor.

3. The processor according to claim 2, characterized in that The processor further includes a fourth storage unit; the tensor storage engine is used for: Acquire a rearrangement parameter from the fourth storage unit, wherein the rearrangement parameter is used to adjust the arrangement order of the coordinate addresses; In the case where the rearrangement parameter indicates to rearrange the coordinate addresses, the arrangement order of the coordinate addresses included in the coordinate address information is adjusted according to the rearrangement parameter to obtain the adjusted coordinate address information.

4. The processor according to claim 3, characterized in that The reordering parameters include a first parameter and a second parameter, wherein the first parameter is used to indicate the relationship between the total number of bits in the data format of the operation result and the total number of bits in the target data format, and the second parameter is used to indicate the reordering granularity of the first tensor; the tensor storage engine is used to: For each coordinate address in the coordinate address information, according to the arrangement position of the coordinate address in the coordinate address information and the second parameter, a first intermediate parameter and a second intermediate parameter are obtained, wherein the first intermediate parameter is used to indicate the reordering granularity to which the coordinate address belongs, and the second intermediate parameter is used to indicate the number of bits of the coordinate address in the reordering granularity to which the coordinate address belongs; Obtaining a third intermediate parameter according to the first parameter, the first intermediate parameter, the second intermediate parameter and the second parameter, wherein the third intermediate parameter is used to indicate an arrangement position of the coordinate address in the adjusted coordinate address information; The adjusted coordinate address information is obtained based on the third intermediate parameters respectively corresponding to the coordinate addresses contained in the coordinate address information.

5. The processor according to claim 4, characterized in that When the target data format is a 4-bit floating point number FP4, the first parameter is 8 and the second parameter is 32; or When the target data format is an 8-bit floating point number FP8, the first parameter is 4 and the second parameter is 16; or When the target data format is a 16-bit floating point number FP16, the first parameter is 2 and the second parameter is 8.

6. The processor according to claim 3, characterized in that The tensor storage engine is used to: The rearrangement parameter is obtained from the fourth storage unit based on the target data format to which the operation result needs to be converted.

7. The processor according to claim 3, characterized in that: The tensor storage engine is used to: In the case where the arrangement of the first tensor is staggered arrangement, adjusting the arrangement order of the coordinate addresses included in the coordinate address information according to the rearrangement parameter and the first dimension corresponding to the first tensor to obtain the adjusted coordinate address information; or When the arrangement mode of the first tensor is non-interlaced arrangement, adjusting the arrangement order of the coordinate addresses included in the coordinate address information according to the rearrangement parameter and the second dimension corresponding to the first tensor to obtain the adjusted coordinate address information; The data change direction corresponding to the first dimension is different from the data change direction corresponding to the second dimension.

8. The processor according to any one of claims 1 to 7, characterized in that: The tensor storage engine is used to: When the number of groups of the plurality of elements is less than a second parameter, filling L preset elements after the plurality of elements in the second storage unit; The second parameter is used to indicate the reordering granularity of the first tensor, L is the difference between the second parameter and the number of groups of the multiple elements, and L is a positive integer.

9. The processor according to any one of claims 1 to 7, characterized in that: The processor includes a fourth storage unit; the tensor storage engine is used for: Obtaining a control instruction for the first tensor; Obtaining basic information of the first tensor from the fourth storage unit according to the control instruction, the basic information including at least one of the following: dimension information, coordinate information, arrangement mode, and data format; Based on the basic information of the first tensor, coordinate address information of the first tensor is obtained.

10. The processor according to any one of claims 1 to 7, characterized in that: The tensor storage engine is used to: After the reading of the first tensor is completed, the asynchronous barrier unit is updated to end the reading of the first tensor by triggering an interrupt through the asynchronous barrier unit.

11. A chip product, characterized in that: The chip product comprises the processor according to any one of claims 1 to 10.

12. A computer device, characterized in that: The computer device comprises a processor as claimed in any one of claims 1 to 10.

13. A tensor processing method applied to a processor, characterized in that: The processor includes: a tensor storage engine, a tensor calculation engine, a format conversion unit, a first storage unit, a second storage unit, and a third storage unit; a first tensor is stored in the first storage unit, and the first tensor includes a plurality of elements arranged in a first arrangement order; The method comprises: The tensor storage engine reads the plurality of elements from the first storage unit to the second storage unit in a second arrangement order to obtain a rearranged tensor, wherein the second arrangement order is different from the first arrangement order; The tensor calculation engine performs a logical operation on the rearranged tensor to obtain an operation result; The format conversion unit converts the data format of each element in the operation result into a target data format to obtain the operation result after format conversion, and stores the operation result after format conversion into the third storage unit; The calculation result after the format conversion includes the calculation results of each of the multiple elements, and the third arrangement order of the calculation results of each of the multiple elements in the third storage unit is the same as the first arrangement order.

Images