Tensor calculation method and device, storage medium and computer equipment

Abstract

The invention provides a tensor calculation method and device, a storage medium and computer equipment, and the method comprises the steps: obtaining task parameters of a tensor calculation task corresponding to a tensor calculation coprocessor and matrix data of a plurality of to-be-processed matrixes; the plurality of matrixes to be processed comprise two matrixes to be multiplied and one matrix to be added; if a first target matrix of which the bit width corresponding to the initial data type is greater than the target bit width exists in the matrix to be multiplied, converting the first target matrix into a second target matrix of the target data type by using a data type conversion unit, and determining a scaling factor corresponding to the second target matrix; and scheduling the task parameter, the matrix data and the scaling factor of the second target matrix, the matrix data of a third target matrix except the first target matrix in the to-be-multiplied matrix, and the matrix data of the to-be-added matrix to a calculation array of a tensor coprocessor for tensor calculation to obtain a tensor calculation result.

Description

Technical Field

[0001] This application relates to the field of integrated circuit technology, and more specifically, to a tensor computation method, apparatus, storage medium, and computer device. Background Technology

[0002] With the rapid development of artificial intelligence technology, various deep learning neural networks are being widely applied in various scenarios. Tensor computation, as a typical application scenario of neural networks, focuses on matrix multiplication and addition. To improve the computational efficiency of neural networks when performing tensor computations and reduce storage and bandwidth requirements, neural network quantization technology has gradually become an important technical means. Among various neural network quantization techniques, the application of low-bit floating-point quantization technology is becoming increasingly common. This quantization technology uses low-bit data (such as 4-bit or 8-bit data) and a scaling factor for joint encoding and decoding, thereby improving numerical stability and computational accuracy.

[0003] However, existing hardware that supports low-bit floating-point quantization technology often only supports quantization processing for specific data types and cannot be applied to various data types, resulting in poor versatility and flexibility. Summary of the Invention

[0004] In view of this, this application provides a tensor computation method, apparatus, storage medium, and computer device, aiming to improve the flexibility and applicability of low-bit floating-point quantization under various data types.

[0005] Specifically, this application is implemented through the following technical solution: In a first aspect, embodiments of this disclosure provide a tensor computation method, including: Obtain the task parameters of the tensor computation task corresponding to the tensor computation coprocessor and the matrix data of multiple matrices to be processed; the multiple matrices to be processed include two matrices to be multiplied and one matrix to be added; If, in the matrices to be multiplied, there exists a first target matrix whose bit width corresponding to the initial data type is greater than the target bit width, then the data type conversion unit is used to convert the first target matrix into a second target matrix of the target data type, and the scaling factor corresponding to the second target matrix is ​​determined; the bit width corresponding to the target data type is not greater than the target bit width. The task parameters, the matrix data and scaling factor of the second target matrix, the matrix data of the third target matrix other than the first target matrix in the matrix to be multiplied, and the matrix data of the matrix to be added are scheduled to the computing array of the tensor coprocessor for tensor computation to obtain the tensor computation result.

[0006] Secondly, embodiments of this disclosure also provide a tensor computation apparatus, the apparatus comprising: The acquisition module is used to acquire the task parameters of the tensor computation task corresponding to the tensor computation coprocessor and the matrix data of multiple matrices to be processed; the multiple matrices to be processed include two matrices to be multiplied and one matrix to be added; The conversion module is used to convert the first target matrix into a second target matrix of the target data type if there is a first target matrix in the matrix to be multiplied that has a bit width corresponding to the initial data type that is greater than the target bit width, and to determine the scaling factor corresponding to the second target matrix; the bit width corresponding to the target data type is not greater than the target bit width; The calculation module is used to schedule the task parameters, the matrix data and scaling factor of the second target matrix, the matrix data of the third target matrix other than the first target matrix in the matrix to be multiplied, and the matrix data of the matrix to be added to the tensor coprocessor's calculation array for tensor calculation to obtain the tensor calculation result.

[0007] Thirdly, an optional implementation of this disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when run, implements the steps of the first aspect above, or any possible implementation of the first aspect.

[0008] Fourthly, an optional implementation of this disclosure also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, performs the steps of the first aspect above, or any possible implementation of the first aspect.

[0009] The tensor computation method, apparatus, storage medium, and computer device provided in this disclosure, when performing tensor computation tasks, for a first target matrix whose initial data type does not meet the target data type, can first convert the data type of the first target matrix, and then use the converted second target matrix to perform tensor computation. This can achieve low-bit floating-point quantization processing for multiple data types, thereby enabling matrix data under multiple data types to be used for tensor computation, improving the versatility and flexibility of data quantization across multiple data types.

[0010] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating a tensor computation method according to an exemplary embodiment of this application; Figure 2This is a schematic diagram illustrating the structure of a data type conversion unit according to an exemplary embodiment of this application; Figure 3 This is a schematic diagram illustrating a matrix data scheduling method according to an exemplary embodiment of this application; Figure 4 This is a schematic diagram illustrating the data transpose process of a B matrix of FP6 / FP8 data type according to an exemplary embodiment of this application; Figure 5 This is a schematic diagram illustrating the data transpose process of a B matrix of FP4 data type according to an exemplary embodiment of this application; Figure 6 This is a schematic diagram of the overall structure of a tensor computation process shown in an exemplary embodiment of this application; Figure 7 This is a schematic diagram of a tensor computing device shown in an exemplary embodiment of this application; Figure 8 This is a schematic diagram of the structure of a computer device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0012] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0013] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0014] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0015] Research has shown that low-bit floating-point quantization, as a common choice in low-bit quantization technology, can effectively improve the computational efficiency of neural networks and save network power consumption. Low-bit floating-point quantization often needs to adhere to the low-bit-width data representation standard specified by Microscaling Formats (MX). The core of this standard lies in achieving adjustable data precision and storage compression by jointly encoding a scaling factor (Scale) and a set of private elements (P) in the matrix that share the scaling factor. Specifically, the MX format mainly includes three components: 1. Scaling factor (Scale): This factor can be a single scalar value used to uniformly scale each private data element P in a scaling block of the matrix; 2. Private data element P: Represents multiple data elements sharing the same Scale. All P elements can be encoded using the same bit width and data type, and the actual value is restored through the decoding process in conjunction with the corresponding Scale; 3. Scaling Block: This specifies the number N of private data elements P contained in each scaling block. The number N in a scaling block can be configured with different values ​​in different application scenarios. Structurally, the MX format implements a "vectorized" data organization mechanism, that is, combining N scalar data into a logical vector through a shared scaling factor. Therefore, the total bit width of an MX unit containing K private elements can be expressed by the following formula: (Formula 1) in, Indicates the total encoding bit width. Indicates the bit width of the scale. N represents the bit width of each private data element P, and N represents the number of private data elements P contained in a scaling block.

[0016] Based on the MX format, storage bandwidth and memory usage can be significantly reduced, and efficient matrix multiply-add (GEMM) operations are supported, making it particularly suitable for AI accelerators and low-power neural network inference scenarios.

[0017] For low-bit floating-point quantization, existing technology, as described in "DR Lutz, A. Saini, M. Kroes, T. Elmer, and H. Valsaraju, “Fused FP8 4-Way Dot Product With Scaling and FP32 Accumulation,” in 2024 IEEE 31st Symposium on Computer Arithmetic (ARITH), Malaga, Spain: IEEE, June 2024, pp. 40–47. doi: The corresponding paper, 10.1109 / ARITH61463.2024.00016, proposes a 4-way dot product unit design, focusing on FP8 precision fused multiply-accumulate operations. It improves numerical stability through FP32 accumulation and a specific scaling mechanism. While this scheme optimizes the precision and performance of FP8 operations, its design is limited to 4-way dot product operations, supporting only FP8 precision. It lacks support for other precisions (such as FP16, BF16, FP32, INT8, INT4) or emerging floating-point formats (such as FP6, FP4, or MX formats), resulting in poor flexibility and adaptability, and an inability to handle a wider range of neural network tasks. In other words, existing low-bit floating-point quantization techniques often provide reliable quantization results when directly receiving low-bit data of specific data types, but their quantization results for other data types are poor. Therefore, improving the applicability and flexibility of low-bit floating-point quantization across various data types has become a noteworthy technical problem.

[0018] Based on the above research, this disclosure provides a tensor computation method, apparatus, storage medium, and computer device. When performing tensor computation tasks, for a first target matrix whose initial data type does not meet the target data type, the data type of the first target matrix is ​​first converted, and then the second target matrix obtained after the conversion is used for tensor computation. This can realize low-bit floating-point quantization processing for multiple data types, thereby enabling matrix data under multiple data types to be used for tensor computation, improving the versatility and flexibility of data quantization across multiple data types.

[0019] The shortcomings of the above solutions are the result of the inventor's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below are all contributions made by the inventor to this disclosure.

[0020] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0021] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0022] To facilitate understanding of this embodiment, a tensor computation method disclosed in this disclosure will first be described in detail. The execution subject of the tensor computation method provided in this disclosure is generally a terminal device or other processing device with certain computing capabilities. The terminal device may be a user equipment (UE), mobile device, terminal, personal digital assistant (PDA), handheld device, computer device, tensor computation coprocessor, etc. In some possible implementations, the tensor computation method can be implemented by the processor calling computer-readable instructions stored in the memory.

[0023] The tensor calculation method provided in this disclosure is illustrated below using a computer device as the execution subject as an example.

[0024] like Figure 1 The diagram shown is a flowchart of a tensor calculation method provided in an embodiment of this disclosure, which may include the following steps: S101: Obtain the task parameters of the tensor computation task corresponding to the tensor computation coprocessor and the matrix data of multiple matrices to be processed; the multiple matrices to be processed include two matrices to be multiplied and one matrix to be added.

[0025] Here, the tensor computation task can specifically be a matrix multiplication and addition task. This task can correspond to three matrices: two matrices to be multiplied and one matrix to be added to the result of the multiplication. For example, the matrix multiplication and addition task can be: A*B+C, where matrices A and B are the matrices to be multiplied, and matrix C is the matrix to be added.

[0026] When performing matrix multiplication and addition, tensor computation coprocessors typically use the three parameters MNK to represent the matrix sizes of the three matrices (matrices A, B, and C) involved in the tensor computation. Matrix A has a size of M×K, matrix B has a size of K×N, and matrix C has a size of M×N. The matrix multiplication and addition result in a single matrix (matrice D). For example, matrix D can be obtained using the following formula: (Formula 2) in, This represents the data in the i-th row and j-th column of matrix D. This represents the kth data point in the i-th row of matrix A. This represents the kth data point in the j-th column of matrix B. This represents the data in the i-th row and j-th column of matrix C.

[0027] Task parameters may include at least the matrix size and data type of each matrix to be processed, the rounding method of the computation array results, and the storage address of external memory. Various task parameters can be transferred by reading and writing configuration parameter registers through a preset bus interface (such as the AXI4 bus interface, AXI-Lite bus interface, etc.). One type of task parameter corresponds to one register, as shown in Table 1 below, which illustrates the configuration for various task parameters: (Table 1) In Table 1, matrices A and B are the two matrices to be multiplied, and matrix C is the matrix to be added. The A / B / C matrix data type indicates the data type that matrices A / B / C can be input into the computation array described later. For the FP8 data type, there are two specific selectable data formats: E5M2 and E4M3, where E and M represent the number of bits in the exponent and mantissa, respectively. Similarly, for the FP6 data type, there are two specific selectable data formats: E3M2 and E2M3, where E and M represent the number of bits in the exponent and mantissa, respectively. This application embodiment supports, under the same hardware computing array size, scheduling to support the mapping of computations in micro-scaling formats of MXINT8, MXFP8 (E5M2 E4M3), MXFP6 (E3M2, E2M3), and MXFP4 (E2M1) to the computing array for computation processing. It also supports multiple matrix sizes, including but not limited to: M32N8K32, M16N16K32, M8N32K32, M32N8K64, M16N16K64, and M8N32K64. Therefore, the values ​​of M / N in the aforementioned registers can include, but are not limited to, 8, 16, and 32, and the values ​​of K can include, but are not limited to, 8, 16, 32, and 64.

[0028] The matrix data of each matrix to be processed can be stored in external memory, and the matrix data of each matrix can be stored in row-major format. That is, the matrix data of each matrix is ​​arranged closely in rows in external memory.

[0029] In practice, when there is a tensor computation task, the various task parameters corresponding to the tensor computation task can be configured through registers, and the matrix data of each matrix to be processed in the tensor computation task can be transferred from external memory row by row into the tensor computation coprocessor.

[0030] S102: If there exists a first target matrix in the matrices to be multiplied, whose bit width is the target bit width and whose initial data type is the target data type, then the data type conversion unit is used to convert the first target matrix into a second target matrix of the target data type, and the scaling factor corresponding to the second target matrix is ​​determined; the bit width of the target data type is less than the target bit width.

[0031] Here, the initial data type can be the original data type of the matrix data to be multiplied, and the matrix data can be stored in external memory according to the original data type. Since the tensor calculation method proposed in this application supports low-bit precision quantization, the target bit width can be the maximum bit width corresponding to the data type under low-bit precision, for example, the target bit width can be 8 bits. It should be noted that for BF16 type matrix data, each data element can be padded with zeros to 32 bits for storage in external memory, which means that the bit width of BF16 type matrix data in external memory is also 32 bits.

[0032] The data type conversion unit is used to convert the data type of matrix data. Specifically, it can be used to convert high-precision data types (such as FP32 / FP16 / BF16 data types) to low-precision data types (such as FP4 / FP6 / FP8 / INT8 data types). The target data type can be any pre-specified low-precision data type, and can be any of MXINT8, MXFP8 (E5M2, E4M3), MXFP6 (E3M2, E2M3), and MXFP4 (E2M1). The bit width of the target data type is smaller than the bit width of the initial data type.

[0033] The first target matrix can be a matrix to be multiplied with a bit width of the target bit width corresponding to the initial data type. For example, the first target matrix can be matrix A and / or matrix B.

[0034] The scaling factor is used to uniformly scale a set of matrix data in the matrices to be multiplied. For example, it can uniformly scale at least one row of matrix A and at least one column of matrix B.

[0035] In practice, for each matrix to be multiplied, the initial data type of the matrix data can be determined, and it can be checked whether the bit width of the data elements in the matrix data of the initial data type in external memory is greater than 8 bits. If so, the matrix to be multiplied can be used as the first target matrix; otherwise, the matrix to be multiplied can be determined to be a low-bit precision matrix, and no data type conversion is required.

[0036] Furthermore, after determining the first target matrix, for each target matrix, the data type of each data element in that target matrix can be converted from the initial data type to the target data type. For example, if the initial data type is FP32 / BF16 / FP16 and the target data type is FP4 / FP6 / FP8 / INT8 in micro-scaling format (MX), each data element of type FP32 / BF16 / FP16 can be converted to data elements of type FP4 / FP6 / FP8 / INT8, thus obtaining a second target matrix with the target data type. Moreover, when converting the data type of the first target matrix, the scaling factor of each group of matrix data in the converted second target matrix in MX format can be determined. Thus, utilizing a hardware data type conversion unit within the tensor coprocessor for data type conversion improves conversion efficiency.

[0037] In one embodiment, S102 described above can be implemented according to the following steps: S102-1: For any set of initial matrix data in the first target matrix, use the data type conversion unit to determine the scaling factor corresponding to the initial matrix data based on the absolute value of each data element in the initial matrix data; a set of initial matrix data is a row of data or a column of data in the first target matrix.

[0038] Here, the data elements included in each set of initial matrix data can be determined according to the matrix type of the first target matrix. Specifically, each set of initial matrix data can include K matrix elements. When the first target matrix is ​​matrix A, since K is the column value of matrix A, a set of initial matrix data can be K matrix elements of a row of matrix A. When the first target matrix is ​​matrix B, since K is the row value of matrix B, a set of initial matrix data can be K matrix elements of a column of matrix B.

[0039] Data elements are the individual data included in matrix data. For example, for matrix A, if K=32 and the data type of matrix A is FP32, then a row of matrix A data can include 32 data elements, each of which is 32 bits of data.

[0040] In practical implementation, for each set of initial matrix data in the first target matrix, it can be input to the data type conversion unit in the form of a data vector e[K]. Here, e[K] includes K data elements of the first target matrix. For example, if the data type of the first target matrix is ​​FP32, e[K] includes K FP32 data elements in IEEE 754 standard format; if the data type of the first target matrix is ​​FP16 / BF16, e[K] includes K floating-point data elements of FP32 / FP16 / BF16. The value of K can be, but is not limited to, 16, 32, 64, etc.

[0041] Furthermore, the absolute value of each data element included in the input e[K] can be determined using the data type conversion unit. Then, based on the maximum value among the absolute values, the scaling factor corresponding to e[K] can be constructed according to the mantissa of the scaling factor of the target data type in MX format. This scaling factor can then be used as the scaling factor of the input set of matrix data.

[0042] In one embodiment, S102-1 described above can be implemented according to the following steps: S102-1-1: Using the data type conversion unit, determine the maximum value from the absolute value of each data element in the initial matrix data.

[0043] In practical implementation, the scaling factor can be extracted using the scaling factor extraction module in the data type conversion unit. Specifically, the scaling factor extraction module can be based on a parallel comparison tree and utilize the maximum absolute value (MaxAbs) for scaling factor extraction.

[0044] In practice, for any initial matrix data set, K data elements from that set can be input into the scaling factor extraction module. The scaling factor extraction module processes the data based on a parallel comparison tree to determine the absolute value of each of the K data elements. Then, the maximum value among the K absolute values ​​is used as the candidate scaling factor (range_candidate).

[0045] S102-1-2: Extract the exponent value of the maximum value and determine whether the exponent value is a preset outlier.

[0046] S102-1-3: If not, determine the scaling factor corresponding to the initial matrix data based on the value of the exponent.

[0047] Here, since the initial data type of the first target matrix is ​​FP32, FP16, or BF16, each data element in the first target matrix includes a sign bit, an exponent bit, and a mantissa bit. Preset outliers can be all zeros in the exponent bit or illegal values ​​specified in the MX format. The exponent bit values ​​are the individual values ​​of the exponent portion of the data element.

[0048] In practice, since the scaling factor is typically 8 bits in MX format and the data format of the scaling factor in MX format is... This means an 8-bit exponent and a 0-bit mantissa. Therefore, the exponent portion (exponent field in offset binary format) of `range_candidate` can be extracted according to the scaling factor data format of MX format and the exponent bit value in the initial data type to construct the scaling factor. Furthermore, when constructing the scaling factor, the sign bit can always be positive, and the sign bit defaults to 0.

[0049] The following will provide examples of scaling factor extraction methods for three initial data types: FP32, FP16, and BF16. FP32: ; BF16: ; FP16: .

[0050] In FP32 type, the exponent bits from bits 23 to 30 of the range_candidate can be used as the scaling factor. In BF16 type, the exponent bits from bits 7 to 14 of the range_candidate can be used as the scaling factor. In FP16 type, the exponent bits from bits 10 to 14 of the range_candidate can be used as the scaling factor. For scaling factors less than 8 bits, 8-bit scaling factors can be obtained by padding with 0s in the high-order bits.

[0051] Furthermore, after selecting the exponent values ​​for the scaling factor based on the exponent values ​​of the candidate factors, it's possible to first determine whether each selected exponent value is all 0 or an invalid value. If it's neither all 0 nor an invalid value, the scaling factor can be constructed based on the selected exponent values. If all 0 or an invalid value is present, an error code, such as an underflow error code, can be output. In this way, after the error code is output, it can be propagated continuously, directly affecting the calculation of the corresponding set of matrix data during the computation of the computation array.

[0052] For example, after receiving the K input data elements, the scaling factor extraction module can process the K data elements using a parallel comparison tree to obtain the maximum positive number (max_val) and the absolute value of the minimum value (min_val_abs) among the K data elements. Then, the maximum value between max_val and min_val_abs is used as the range_candidate. Next, based on the initial data type and the mantissa and exponent bits of the initial data type in MX format, the exponent part (i.e., the exp field in offset binary format) of the range_candidate is extracted to obtain the scaling factor.

[0053] In one embodiment, the data type conversion unit may further include an abnormal input checking module. This module is used to perform anomaly checks on a set of input initial matrix data to determine whether the initial matrix data is abnormal. Specifically, before converting the first target matrix into a second target matrix of the target data type, anomaly checks can be performed through the following steps 1 and 2: Step 1: For any set of initial matrix data in the first target matrix, use the data type conversion unit to check for outliers in each data element of the initial matrix data to determine whether there are any abnormal data elements.

[0054] Here, outliers can typically include the following types: 1. NaN (i.e., the exponent is all 1s and the mantissa is non-zero); 2. ±∞ (i.e., the exponent is all 1s and the mantissa is all 0s); 3. ±0 (i.e., the exponent and mantissa are both 0s).

[0055] In practice, the abnormal input checking module can examine each data element in a set of K initial matrix data of type FP32 / BF16 / FP16 to determine whether each data element is an abnormal value. If none of the data elements are abnormal, the K data elements can be input to the scaling factor extraction module for scaling factor extraction. If they are abnormal, then step 2 below can be executed.

[0056] Step 2: If so, modify the abnormal data element to a preset element.

[0057] Here, the preset elements can be set based on experience, and this embodiment of the disclosure does not impose specific limitations. For example, the preset element can be 0, or a preset exception code.

[0058] In practice, if there are abnormal data elements among the K data elements, each abnormal data element can be modified to a preset element. Then, all non-abnormal data elements and the modified abnormal data elements are input into the scaling factor extraction module to extract the scaling factor. Here, after the abnormal data element is modified to a preset element, the abnormal information of the abnormal data element can be continuously transmitted, which facilitates the identification of abnormal data elements when the computation array performs tensor calculations.

[0059] S102-2: Based on the exponent value and scaling factor of each data element in the initial matrix data, convert each data element into a data element of the target data type to obtain a set of target matrix data corresponding to the initial matrix data in the second target matrix.

[0060] Here, both the target data type and the initial data type can be specified through configuration information when inputting each set of matrix data. For example, the configuration information can include the input data type (IN_datatype) and the output data type (OUT_datatype). The input data type is the initial data type of the initial matrix data, with an address of 0x10 and a bit width that can be set, such as 3 bits. The initial data type is specified according to the data type of the first target matrix; for example, a configuration signal with BF16 / FP32 / FP16 precision can be selected. The output data type is the target data type, with an address of 0x0C and a bit width that can be set, such as 3 bits. The target data type is used to determine the target quantization precision of the input data and can be a low-bit data type in MX format. For example, the data formats supported by the target data type and the maximum representation value in each data format are shown in Table 2 below: (Table 2) In practical implementation, the scaling factor extraction module can be used to input the extracted scaling factor and K matrix data together into the exponential difference precision mapping module in the data type conversion unit. The exponential difference precision mapping module replaces floating-point division with exponential subtraction, thereby achieving data type conversion and saving computational overhead during conversion. Specifically, the exponential difference precision mapping module can subtract the exponent value of each data element using the scaling factor to obtain the exponent and mantissa values ​​of each data element in the target data type. Then, the exponential difference precision mapping module can input the data elements of each initial data type, the exponent and mantissa values ​​in the target data type, into the output assembly and packaging module in the data type conversion unit. The output assembly and packaging module concatenates the exponent and mantissa values ​​of each data element of the initial data type in the target data type, along with the sign bit of the initial data type data element, to obtain the data element corresponding to the target data type for each data element. Thus, each data element corresponding to the target data type can form a set of target matrix data corresponding to the input initial matrix data in the second target matrix.

[0061] In one embodiment, regarding S102-2 above, the initial matrix data can be type-converted using different methods by the exponential difference precision mapping module, depending on the target data type. Specifically, when the target data type includes floating-point types and the bit width is less than or equal to the preset bit width, data type conversion can be performed using steps A1-A4 in method A as follows: A1: When the target data type includes floating-point type and the bit width is less than or equal to the preset bit width, determine the initial conversion data of each data element based on the exponent value, scaling factor and preset conversion value of the target data type of each data element in the initial matrix data.

[0062] Here, the preset bit width can be the bit width corresponding to the low-bit floating-point type in MX format, for example, the preset bit width can be 8 bits. The preset conversion values ​​for various target data types can be determined based on the maximum representation value of the target data type in Table 2 above. For example, they can be determined using the following Formula 3 based on the maximum representation value of various target data types in Table 2 above: Exp_fpmax = ceil(log2(fp_max)); (Formula 3) Where Exp_fpmax represents the target transformation value, ceil function represents the rounding up function, and fp_max represents the maximum representation value.

[0063] For example, for any data format of the target data type of floating-point type, the preset conversion value corresponding to the data format can be determined using Formula 3 above, based on the maximum representation value of that data format in Table 2 above. Then, the preset conversion values ​​for each target data type of floating-point type under each data format can be stored in Table 3 below: (Table 3) When converting the initial matrix data of any floating-point type target data type, the corresponding preset conversion value can be obtained by looking up Table 3 above.

[0064] The initial transformation data can be the data elements in the initial data matrix, corresponding to the exponential quantum (exp_quant) obtained after conversion to the target data type.

[0065] In specific implementation, when the target data type includes floating-point types and the bit width is less than or equal to the preset bit width, the target data type can be one of MXFP8_E5M2, MXFP8_E4M3, MXFP6_E3M2, MXFP6_E2M3, and MXFP4_E2M1. Using the exponential difference precision mapping module, the preset conversion value (Exp_fpmax) corresponding to the target data type is determined according to Table 3 above, and the scaling factor corresponding to the initial matrix data output by the scaling factor extraction module is obtained. Then, the initial matrix data can be split according to the initial matrix data type into the sign bit (sign_i), exponent bit (exp_i), and mant bit (mant_i) of each data element xi in the initial data matrix. For example, when the initial data type is FP32, each xi can be split into the highest-order sign bit (sign_i), the exponent bits (exp_i) from the 23rd to the 30th bits of the FP32 type data element, and the mant bit bits (mant_i) from the 0th to the 22nd bits of the FP32 type data element. The value of each exponent bit is the same as the exponent value mentioned above. That is, the FP32 data element is split into: sign_i = the most significant bit, exp_i = bits [30:23] of the FP32 data element, and mant_i = bits [22:0] of the FP32 data element. With the initial data type being BF16, each xi can be split into: the 1-bit sign bit (sign_i), the exponent bits (bits 7 to 14) of the FP32 data element (exp_i), and the mant bits (bits 0 to 6) of the FP32 data element. That is, the BF16 data element is split into: sign_i = the most significant bit, exp_i = bits [14:7] of the BF16 data element, and mant_i = bits [6:0] of the BF16 data element. With the initial data type being FP16, each xi can be split into: the highest-order sign bit (sign_i), the exponent bits (10th to 14th bits) of the FP16 data element (exp_i), and the mant bits (0th to 9th bits) of the FP16 data element (mant_i). That is, the FP16 data element is split into: sign_i = highest-order bit, exp_i = bits [14:10] of the FP16 data element, and mant_i = bits [9:0] of the FP16 data element.

[0066] At the same time, the scaling factor is preset (i.e., exp_scale= That is, directly using the scaling factor extraction module output. The scaling factor for the format. The default conversion value is Exp_fpmax=ceil(log2(fp_max)), which is determined by looking up Table 3 above.

[0067] Then, for each data element in the initial matrix data, the exponential difference precision mapping module performs subtraction on the values ​​of each exponent bit, the scaling factor, and Exp_fpmax under the target data type for that data element. Based on the magnitude of the subtraction result, the result is truncated to obtain the initial transformed data exp_quant for that data element. For example, exp_quant = exp_i - exp_scale - Exp_fpmax. Simultaneously, if exp_quant is less than 0, it is truncated to the minimum representable value, such as 0; if exp_quant is greater than the preset maximum supported value, it is truncated to the maximum representable value, such as positive or negative infinity or any maximum value that can be represented.

[0068] A2: Based on the mantissa width of the target data type, truncate the mantissa value of each data element to obtain the truncated mantissa value.

[0069] A3: Adjust the initial conversion data of each data element according to the exponent offset corresponding to the target data type to obtain the target conversion data.

[0070] Here, the mantissa bit width is the bit width required by the target data type. Different target data types correspond to different exponent offsets. By subtracting the corresponding exponent offset from the initial conversion data, the exponent bits that conform to the target data type can be obtained.

[0071] As shown in Table 4 below, the mantissa bit width, exponent bit width, and exponent offset of various target data types provided in the embodiments of this application are displayed under various data formats: (Table 4) In practice, for each data element in the initial matrix data, the exponent difference precision mapping module can be used to obtain the mantissa bit width and exponent offset corresponding to the target data type from Table 4 above. Then, the mantissa bit value of each data element is truncated according to the mantissa bit width to obtain the truncated mantissa bit values ​​(Mantissa_t). Simultaneously, the exponent offset can be subtracted from the initial converted data of each data element to obtain the target converted data. The exponent bit width of the target converted data is the same as the exponent bit width corresponding to the target data type indicated in Table 4 above.

[0072] A4: Based on the sign bit value of each data element, the target transformed data, and the truncated mantissa bit value, determine the data element of the target data type corresponding to each data element, and obtain a set of target matrix data corresponding to the initial matrix data in the second target matrix.

[0073] In practice, after obtaining the target transformation data and the truncated mantissa value for each data element, these can be input into the output assembly and packaging module for data packaging and output. Specifically, the output assembly and packaging module can be used to concatenate the sign bit value, target transformation data, and truncated mantissa value for each data element xi to obtain the quantization result corresponding to each data element xi. .For example, ={sign_i, exp_quant, Mantissa_t}. Quantization result. This refers to the data element of the target data type corresponding to data element xi. After concatenating the quantization results corresponding to each data element, the output assembly and packaging module can encapsulate them into a set of target matrix data corresponding to the initial matrix data and the second target matrix.

[0074] The output assembly and packaging module supports fixed-length floating-point compressed packaging (such as FP4, FP6, and FP8 format packaging) and fixed-point integer packaging (such as INT packaging). In specific implementation, the packaging format required by the output assembly and packaging module can be determined according to the target data type.

[0075] The output assembly and packaging module outputs a set of target matrix data, which can be quantized into a vector. Output. Then, quantize the resulting vector. It can be processed into a continuous bit stream through bit-packing, which facilitates reading from the input buffer or computation array as described later. Here, the continuous bit stream refers to the vectors of continuously output quantized results. For the first target matrix, if the number of data type conversion units is the same as the value of K for the first target matrix, then each data type conversion unit can be used to synchronously output the quantization result vectors corresponding to each set of initial vector data included in the first target matrix. These quantization result vectors can then be directly input into the computation array for tensor computation. If the number of data type conversion units is less than the value of K for the first target matrix, then each data type conversion unit can be used to synchronously output the quantization result vectors corresponding to a portion of the initial vector data included in the first target matrix. Then, each quantization result vector is written to the computation array periodically, ensuring that the quantization result vectors written to the computation array are consistent with the size of the block scheduling described later.

[0076] In another embodiment, when the target data type includes a floating-point type and the bit width is less than or equal to a preset bit width, data type conversion can be performed using steps B1 to B3 in method B as follows: B1: When the target data type includes integer type and the bit width is less than or equal to the preset bit width, perform a padding operation on the mantissa of each data element to obtain the initial mantissa value.

[0077] Here, if the target data type includes integers and the bit width is less than or equal to the preset bit width, it can be stated that the target data type is MXINT8.

[0078] For example, when the target data type is MXINT8, for each data element xi in the initial matrix data, the sign bit (sign_i), exponent bit (exp_i), and mant bit (mant_i) of xi can be separated according to the data type of xi. Then, the mant bit value of xi is padded to obtain the initial mant bit value corresponding to the mant bit. Specifically, during the padding, if the mant bit value is a normalized mantissa that conforms to the preset data format requirements, it is padded with 1; otherwise, if it is a non-normalized mantissa that does not conform to the preset data format requirements, it is padded with 0.

[0079] B2: Based on the difference between the exponent value and the scaling factor of each data element, shift the initial mantissa value of each data element to obtain the target mantissa value.

[0080] In practice, for each xi, the exponent value of xi can be calculated, along with the scaling factor (exp_scale=) extracted by the scaling factor extraction module. The difference is calculated. Then, based on the difference for each xi, the initial mantissa value of each xi is right-shifted by the number of bits corresponding to the difference, to obtain the target mantissa value for each xi.

[0081] B3: Based on the sign bit value and target mantissa value of each data element, determine the data element of the target data type corresponding to each data element, and obtain a set of target matrix data corresponding to the initial matrix data in the second target matrix.

[0082] In practice, the sign bit value and the target mantissa value of each data element xi can be input to the output assembly and packaging module for data packaging and output, thereby obtaining the quantization result of each data element in MXINT8 type. Quantification results This refers to the data element of type MXINT8 corresponding to data element xi. Therefore, in this embodiment, the quantization result under MXINT8 is the result of [sign bit + mantissa truncation (i.e., the target mantissa value)].

[0083] Furthermore, after the output assembly and packaging module concatenates the quantization results corresponding to each data element, it can encapsulate them into a set of target matrix data corresponding to the initial matrix data and the second target matrix. Meanwhile, under the MXINT8 type, the scaling factor of the target data matrix is ​​the scaling factor extracted by the scaling factor extraction module (exp_scale= ).

[0084] S102-3: Use the scaling factor corresponding to the initial matrix data as the scaling factor of the target matrix data.

[0085] In practice, the scaling factor extracted by the scaling factor extraction module for the initial matrix data can be used as the scaling factor for a set of target matrix data corresponding to the initial matrix data.

[0086] Thus, after inputting a set of initial matrix data e[K] from the first target matrix into the data type conversion unit, a set of quantization result vectors can be obtained. and scaling factor in exponential format .in, It includes K quantized values, each of which is a target data type (e.g., FP8 / FP6 / FP4 / INT8). A quantized value is a data element in a set of target matrix data corresponding to the initial matrix data in the second target matrix. Scaling factor This is the scaling factor in E8M0 format, which includes an 8-bit exponent and a 0-bit mantissa.

[0087] It should be noted that if multiple data type conversion units exist, they can be used to simultaneously convert the data types of multiple sets of initial matrix data in the first target matrix, thereby obtaining a set of target matrix data corresponding to the multiple sets of initial matrix data in the second target matrix.

[0088] Thus, the data type conversion unit provided in this application supports quantization of various low-bit floating-point formats (including MXINT8, MXFP8, MXFP6, and MXFP4). This unit is implemented in hardware and is designed for low-bit representation requirements in tensor computation. It is suitable for channel vector input scenarios with K=16 / 32 / 64 in structured matrix multiplication such as M16N16K16 / M8N32K16 / M32N8K16 / M32N8K32 / M16N16K32 / M8N32K32 / M32N8K64 / M16N16K64 / M8N32K64. This unit features high throughput, low power consumption, and strong compatibility. It supports hardware and software co-configuration and can be deployed in the internal quantization path of artificial intelligence inference chips or system-on-chips (SoCs).

[0089] like Figure 2 The diagram shown is a structural schematic of a data type conversion unit provided in an embodiment of this application. After the data vector e[K] corresponding to K data elements is input into the data type conversion unit, it sequentially passes through various abnormal input checking modules, scaling factor extraction modules, exponential difference method precision mapping modules, and output assembly and packaging modules to output the quantization result. .

[0090] S103: Schedule the task parameters, matrix data and scaling factor of the second target matrix, matrix data of the third target matrix other than the first target matrix in the matrix to be multiplied, and matrix data of the matrix to be added to the computing array of the tensor coprocessor for tensor computation to obtain the tensor computation result.

[0091] Here, the computation array is pre-configured within the tensor coprocessor and can be used for tensor computation processing. The computation array supports tensor computations on various low-bit-precision matrix data in the MX format, for example, at least supporting tensor computations of MXINT8, MXFP8 (E5M2 / E4M3), MXFP6 (E3M2 / E2M3), and MXFP4 (E2M1) types. The size of the computation array can be defined according to preset hardware parameters. These preset hardware parameters can be the hardware parameters related to the tensor computation coprocessor defined in the Chisel language. In this embodiment, various configuration parameters of the tensor computation coprocessor can be pre-configured using Chisel parameters. These configuration parameters may include, as described below, the number of input buffers, the size of the computation array, the number of output buffers, and the data transfer bandwidth. After the Chisel parameters are configured, the architecture of the tensor computation coprocessor is determined accordingly.

[0092] Whether a third objective matrix exists depends on the number of first objective matrices in the matrices to be multiplied. If both matrices to be multiplied are first objective matrices, then the third objective matrix is ​​empty; if one of the two matrices to be multiplied is a first objective matrix, then the third objective matrix is ​​the one that is not a first objective matrix.

[0093] If the third target matrix is ​​not empty, the scaling factors corresponding to the third target matrix can be obtained synchronously with the task parameters and matrix data. In this case, the scaling factors of the third target matrix can be specified in advance. The scaling factors can be set based on experience, and this embodiment does not impose specific limitations. For example, they can be 0 or other values.

[0094] In practice, after obtaining the second target matrix, the task parameters can be scheduled to the computation array of the tensor coprocessor, along with the matrix data and scaling factors of each second target matrix, the third target matrix, and the matrix data of the matrices to be added. Then, using the computation array, matrix multiplication and addition are performed with the scheduled matrix data and scaling factors, and the results are rounded according to the rounding method indicated by the task parameters, thus obtaining the tensor computation result of the tensor computation task. The tensor computation result can be the result matrix obtained after matrix multiplication and addition.

[0095] In one embodiment, when the first target matrix does not exist in the matrices to be multiplied, that is, when the initial data types of both matrices to be multiplied are low-bit types in MX format, tensor computation can be performed according to the following P1~P4: P1: If the first target matrix does not exist in the matrices to be multiplied, the matrix data in the multiple matrices to be processed are mapped to the input buffers corresponding to each matrix to be processed, and the scaling factor set for each group of matrix data of each matrix to be multiplied is mapped to the scaling factor buffer corresponding to each matrix to be multiplied.

[0096] Here, if the initial data types of the two matrices to be multiplied are both FP4 / FP6 / FP8 / IN8, a scaling factor can be pre-set for each set of matrix data for each matrix to be multiplied. Each matrix to be processed has at least one input buffer, and the task parameters can also have at least one task parameter buffer. The number of task parameter buffers can be the same as the number of input buffers for each matrix to be processed, and the number of input buffers for each matrix to be processed can also be the same. The two matrices to be multiplied can also each have a scaling factor buffer, with the number of scaling factor buffers for each matrix to be multiplied being the same as the number of input buffers for the matrices to be multiplied. Optionally, the scaling factor buffer can be the same as the input buffer, meaning that the input buffer for each matrix to be multiplied can be used to cache both the scaling factor and each set of matrix data.

[0097] In practice, if the first target matrix does not exist in the two matrices to be multiplied, each set of matrix data of each matrix to be processed can be mapped to the input cache corresponding to the matrix to be processed. At the same time, the task parameters can be mapped to the task parameter cache for caching, and the scaling factor of each set of matrix data of each matrix to be multiplied can be mapped to the scaling factor cache corresponding to the matrix to be multiplied.

[0098] P2: Schedule task parameters to the computation array, and schedule matrix data in each input buffer and scaling factors in each scaling factor buffer to the computation array in blocks.

[0099] Here, the size and order of block scheduling for each matrix to be processed can be determined based on the scale of the computing array and the matrix dimensions (i.e., the values ​​of M, N, and K). A state machine is used for block scheduling; the number of states in the state machine can be determined based on the scale of the computing array and the matrix dimensions of each matrix to be processed. Each time a state changes, block scheduling is performed. The computing array can be a stack of several dot product units (DPs) or a stack of several multipliers and accumulators; this embodiment does not impose specific limitations.

[0100] In practice, after the corresponding data has been cached in each input buffer, task parameter buffer, and scaling factor buffer, the data in each buffer can be scheduled to perform tensor computation. For example, task parameters can be scheduled from the task parameter buffer to the computation array. Then, according to the state machine's state transitions, the matrix data of each matrix to be processed is scheduled to the scheduling array in blocks from each input buffer. Since matrix multiplication of low-bit data types containing scaling factors requires additional scheduling of scaling factors compared to traditional quantized low-bit matrix multiplication, the scaling factors stored in the scaling factor buffer after the above mapping can also be scheduled according to the blocks of data of the two matrices to be multiplied, and the scaling factors corresponding to the blocks of data of the two matrices to be multiplied are scheduled to the corresponding computation arrays. Specifically, while scheduling the matrix data of each matrix to be processed in blocks each time, the scaling factors corresponding to the data of each matrix to be multiplied in the blocks can be scheduled from the scaling factor buffer of the matrix to be multiplied to the computation array.

[0101] In the matrix scheduling design of this application, all scheduling-related parameters can be defined in software through preset hardware parameters. This allows the scheduling process to flexibly adapt to the computational needs of matrices of different sizes and shapes without requiring hardware redesign. The preset hardware parameters can be chisel parameters. Scheduling-related parameters may include, for example, the size of the column being calculated and the value of k corresponding to each scaling factor. Here, k is not strongly bound to K in the matrix size; k can be the same as or different from K, and can be flexibly configured through chisel parameters.

[0102] For the computing array, the input data consists of matrix data of matrices to be multiplied in types INT8 / FP8 / FP6 / FP4, the scaling factor corresponding to the matrix data of the matrices to be multiplied, and matrix data of matrices to be added in types FP16 / FP32 / INT32. The output is the matrix data of the accumulated high-precision FP32 / FP16 / INT32 matrices. By pre-designing the computing array, the hardware array control signal broadcasting can be completed. Specifically, it can support multiplication of two precisions in FP8: E5M2*E5M2, E5M2*E4M3, E4M3*E5M2, E4M3*E4M3; multiplication of two precisions in FP6: E3M2*E3M2, E3M2*E2M3, E2M3*E3M2, E2M3*E2M3; matrix multiplication in INT8 precision; and matrix multiplication in FP4 precision. The matrix data to be multiplied, which is of type INT8 / FP8 / FP6 / FP4, can be directly input from external memory or obtained by converting the matrix data of type FP16 / FP32 / BF16 using a data type conversion unit.

[0103] P3: Using a computation array, determine the computation result for each block scheduling based on task parameters, matrix data for each block scheduling, and scaling factor.

[0104] In practice, a computing array can be used to perform multiplication and addition calculations on the matrix data and scaling factors of each matrix to be processed in each block scheduling, obtaining the multiplication and addition calculation result. The multiplication and addition calculation result is then rounded using the rounding method indicated by the task parameters to obtain the calculation result for each block scheduling. The calculation result for each block scheduling can be high-precision data, such as FP32 / FP16 / INT32 data.

[0105] P4: Determine the tensor calculation result based on the calculation result of each block scheduling.

[0106] In practice, after the computing array completes the calculation results of each block scheduling, the results of each block scheduling can be combined into a result matrix according to the order of block scheduling and transmitted to external memory for storage.

[0107] In one embodiment, S103 described above can be implemented according to the following steps: S103-1: Map the matrix data of the second target matrix, the matrix data of the third target matrix, and the matrix data of the matrix to be added to the input buffers corresponding to each matrix to be processed, and map the scaling factors of the second target matrix and the scaling factors of the third target matrix to the scaling factor buffers corresponding to each matrix to be multiplied.

[0108] Here, if a third target matrix exists, the scaling factor for each set of matrix data in the third target matrix can be pre-specified and input to the tensor coprocessor. The scaling factor for each set of matrix data in the second target matrix can be output by the data type conversion unit.

[0109] In practical implementation, after obtaining the second target matrix and its scaling factor, each set of matrix data (i.e., each set of target matrix data) of the second target matrix can be mapped to the input buffer corresponding to the second target matrix. Similarly, each set of matrix data of the third target matrix can be mapped to the input buffer corresponding to the third target matrix, and the matrix data of the matrices to be added can be mapped to the input buffer corresponding to the matrices to be added. Simultaneously, when mapping each set of matrix data of the second target matrix, the scaling factor corresponding to each set of matrix data can be mapped to the scaling factor buffer corresponding to the second target matrix; similarly, when mapping each set of matrix data of the third target matrix, the scaling factor corresponding to each set of matrix data can be mapped to the scaling factor buffer of the third target matrix. The set of matrix data of the second and third target matrices can be determined based on the matrix type of the second and third target matrices. For example, if the second / third target matrix is ​​matrix A, a set of matrix data can be a row of matrix A; if the second / third target matrix is ​​matrix B, a set of matrix data can be a column of matrix B. Furthermore, task parameters can also be mapped to the task parameter buffer.

[0110] S103-2: Schedule task parameters to the computing array, and schedule matrix data in each input buffer and scaling factors in each scaling factor buffer to the computing array in blocks.

[0111] In practice, after the corresponding data has been cached in each input buffer, task parameter buffer, and scaling factor buffer, the data in each buffer can be scheduled to perform tensor computation. For example, task parameters can be scheduled from the task parameter buffer to the computation array. Then, according to the state machine's state transitions, the matrix data of each matrix to be processed can be scheduled from each input buffer to the scheduling array in blocks. Additionally, the scaling factors corresponding to the block scheduling data of two matrices to be multiplied can be scheduled to the corresponding computation arrays.

[0112] Optionally, after converting the second target matrix using the data type conversion unit, instead of mapping each set of matrix data and scaling factor of the second target matrix to the corresponding input buffer and scaling factor buffer, only mapping each set of matrix data and scaling factor of the third target matrix to the corresponding input buffer and scaling factor buffer, mapping the task parameters to the task parameter buffer, and mapping the matrix data of the matrices to be added to the corresponding input buffer. Then, during scheduling, the task parameters are scheduled to the computation array from the task parameter buffer, the matrix data and scaling factor of the third target matrix are scheduled to the computation array in blocks from the input buffer and scaling factor buffer corresponding to the third target matrix, the matrix data of the matrices to be added are scheduled to the computation array from the input buffer corresponding to the matrices to be added, and the block matrix data and scaling factors corresponding to the block matrix data in the second target matrix input by the data type conversion unit are directly scheduled to the computation array.

[0113] Typically, the initial data types of the two matrices to be added can be the same. Therefore, if a first target matrix exists, both matrices to be added belong to the first target matrix. The data type conversion unit can then be used to convert the data types of the two matrices to obtain the second target matrix and scaling factor corresponding to each matrix. At this point, the third target matrix is ​​empty. Further, during scheduling, the block scheduling data and scaling factors corresponding to the two second target matrices can be scheduled to the computation array, and the block scheduling data of the matrices to be added can be scheduled to the computation array from the input buffer corresponding to the matrices to be added, as well as the task parameters can be scheduled to the computation array from the task parameter buffer. Alternatively, each set of matrix data and scaling factor of the two second target matrices can be mapped to the input buffer and scaling factor buffer corresponding to the two second target matrices, respectively. Then, during scheduling, data scheduling and tensor computation are implemented according to steps P2 to P4.

[0114] S103-3: Using a computing array, determine the computation result of each block scheduling based on task parameters, matrix data for each block scheduling, and scaling factor.

[0115] S103-4: Determine the tensor calculation result based on the calculation result of each block scheduling.

[0116] For the specific implementation process of S103-3 to S103-4 above, please refer to the introduction of P3 to P4 above, which will not be repeated here.

[0117] To facilitate understanding of the matrix data scheduling process in the embodiments of this application, as follows: Figure 3 The diagram shown is a schematic representation of matrix data scheduling according to an embodiment of this application. Wherein, in Figure 3In this diagram, the matrix size is M16N16K16 (i.e., matrices A, B, and C are all 16×16 matrices), and the state machine has 8 states. Sets 1 to 8 represent the block matrix data scheduled in matrices A, B, and C respectively under each of the 8 states, along with the scaling factors of the scheduled A and B matrices. For any set, 0 to 3 in matrix A correspond to 4 rows and 16 columns of data, and 0 to 3 in the scaling factors of matrix A correspond to the scaling factors of each of the 4 rows of data. Similarly, 0 to 3 in matrix B correspond to 16 rows and 4 columns of data, and 0 to 3 in the scaling factors of matrix B correspond to the scaling factors of each of the 4 columns of data. 0 to 7 in matrix C correspond to 8 rows and 4 columns of data, and matrix C has no scaling factor. The filled portion in each set represents the matrix data or scaling factor required for that set to be scheduled. For example, in set1, matrix A needs to schedule 8 rows and 16 columns of matrix data along with the scaling factors corresponding to the 8 rows of matrix data; matrix B needs to schedule 16 rows and 4 columns of matrix data along with the scaling factors corresponding to the 4 columns of matrix data; and matrix C needs to schedule 8 rows and 4 columns of matrix data. For the data required for scheduling in other sets, please refer to the description of set1, which will not be repeated here. Thus, through 8 state transitions of the state machine, the scheduling process of sets1 to 8 is implemented, resulting in 8 block scheduling calculations. Finally, based on the integration of the 8 block scheduling calculation results, the resulting matrix D corresponding to A*B+C is obtained.

[0118] Understandably, regarding the matrix data scheduling process under other matrix shapes, and... Figure 3 The scheduling process under M16N16K16 is similar, the difference being the size of the block matrix and the number of scaling factors during each scheduling. Therefore, scheduling for various other matrix sizes can be referenced. Figure 3 The scheduling process is adaptively adjusted, and will not be described in detail in the embodiments of this application.

[0119] In one embodiment, when the matrix to be processed is the first matrix among two matrices to be multiplied, the input buffer may include a first buffer, and the scaling factor buffer may include a first scaling factor buffer. Here, when neither of the two matrices to be multiplied is a first target matrix, it is explained that the two matrices to be multiplied are the third target matrices. The two matrices to be multiplied can be the multiplying matrix and the multiplied matrix, respectively. For example, in the case of A*B, matrix A can be the multiplying matrix, and matrix B can be the multiplied matrix. The first matrix is ​​the multiplying matrix, and the second matrix is ​​the multiplied matrix. For the first matrix among the two third target matrices that serves as the multiplying matrix, the tensor computation coprocessor may include a first buffer, a first storage unit, and a first scaling factor buffer corresponding to the first matrix. For ease of description, the following description will use matrix A as the first matrix and matrix B as the second matrix as an example. The first buffer, first storage unit, and first scaling factor buffer corresponding to matrix A in the tensor computation coprocessor can be referred to as the A matrix data buffer, A data store, and A scaling factor buffer, respectively. Since the A*B operation requires multiplying the row data of matrix A with the column data of matrix B, a set of matrix data of matrix A is equivalent to a row of matrix data of matrix A.

[0120] Specifically, the step of "mapping the matrix data in the first matrix to the input buffer corresponding to the first matrix, and mapping the scaling factor set for each group of matrix data in the first matrix to the scaling factor buffer corresponding to the first matrix" can be implemented as follows: Steps 11-12 Step 11: For any row of matrix data in the first matrix, map the scaling factor set for that row of matrix data to the first scaling factor cache.

[0121] In practical implementation, scaling factors can be set for each row of matrix A data, row by row. Then, for any row of matrix A data, the tensor computation coprocessor can receive the scaling factor set for that row of matrix A data and store the scaling factor of that row in the A scaling factor cache. The bit width of each scaling factor in matrix A can be a set value, such as 8 bits. Understandably, the bit width of the scaling factor can be set according to computational needs, and this embodiment does not impose a specific limitation.

[0122] Step 12: If the bit width corresponding to the initial data type of the first matrix is ​​the first preset bit width, then each data element in the row of matrix data is padded with zeros to the second preset bit width, and the zero-padded row of matrix data is mapped to the corresponding row in the first cache.

[0123] Here, the first preset bit width can be set empirically, and this application embodiment does not impose a specific limitation. The following description uses a first preset bit width of 6 bits as an example. When the first preset bit width is 6 bits, the initial data type can be FP6 data type. When the initial data type of matrix A is FP6, each data element of matrix A is 6 bits. The second preset bit width can be 8 bits.

[0124] In practice, the tensor computation coprocessor receives each row of matrix A data in row-major order and stores it in a data store. Then, it checks if the initial data type of matrix A has a bit width of 6 bits (i.e., if the data type is FP6). If not, each row of matrix A data can be directly mapped to a data buffer. If so, each row of matrix A data is padded with zeros to 8 bits. Finally, the padded rows are mapped to the corresponding rows in the data buffer.

[0125] For example, after receiving row-matrix A data in row-major order, the tensor computation coprocessor can store it in a row-major format in an A data store. Then, if the initial data type of the A matrix is ​​FP6, each FP6 data element in each row of the A matrix data needs to be padded with zeros to 8 bits within the A data store. This padded A matrix data is then mapped to the corresponding row in the A data buffer. If the initial data type of the A matrix is ​​FP4 / FP8 / INT8, no zero-padding is needed; the A data store directly maps each row of the A matrix data to the corresponding row in the A data buffer, thus storing the A matrix data in a row-major format in the A data buffer. Simultaneously, the tensor computation coprocessor can also receive scaling factors generated row-by-row on the A matrix and pass them into the A scaling factor buffer in row-major order.

[0126] Optionally, after obtaining the second target matrix, if the second target matrix is ​​matrix A, each row of the second target matrix can be passed to data store A row by row. Based on the target data type of the second target matrix, it can be determined whether to pad each row with zeros. If so, zeros can be padded to 8 bits before mapping to the corresponding row in data buffer A. If not, the data can be directly mapped from data store A to the corresponding row in data buffer A. Simultaneously, the tensor computation coprocessor can also receive the scaling factor generated for each row of the second target matrix and pass it to scaling factorbuffer A in row-major format.

[0127] In one embodiment, when the matrix to be processed is the second matrix among two matrices to be multiplied, the input buffer may include a second buffer, and the scaling factor buffer may include a second scaling factor buffer. Here, when neither of the two matrices to be multiplied is a first target matrix, it is explained that the two matrices to be multiplied are the third target matrices. For the second matrix among the two third target matrices that serves as the matrix to be multiplied, the tensor computation coprocessor may include a second buffer, a second storage unit, and a second scaling factor buffer corresponding to the second matrix. The second buffer, second storage unit, and second scaling factor buffer corresponding to the B matrix in the tensor computation coprocessor can be referred to as the B matrix data buffer, B data store, and B scaling factor buffer, respectively. Since the A*B operation requires multiplying the row data of matrix A with the column data of matrix B, a set of matrix data of matrix B is equivalent to a column of matrix data of matrix B.

[0128] Specifically, the step of "mapping the matrix data in the second matrix to the input buffer corresponding to the second matrix, and mapping the scaling factor set for each group of matrix data in the second matrix to the scaling factor buffer corresponding to the second matrix" can be implemented as follows: Steps 21-22 Step 21: Map the scaling factor set for each column of matrix data in the second matrix to the second scaling factor cache.

[0129] In practice, scaling factors can be set for each column of B matrix data, column by column. Then, for any column of B matrix data, the tensor computation coprocessor can receive the scaling factor set for that column of B matrix data and store it in the B scaling factor cache. The bit width of each scaling factor in the B matrix can be a set number of bits, such as 8 bits.

[0130] Step 22: If the bit width corresponding to the initial data type of the second matrix is ​​the first preset bit width, then for any row of matrix data in the second matrix, each data element in the row of matrix data is padded with zeros to the second preset bit width, and the zero-padded row of matrix data is mapped to the corresponding column in the second cache according to the second preset bit width.

[0131] Here, the first preset bit width can be 6 bits. When the initial data type of matrix B is FP6, each data element of matrix B is 6 bits. The second preset bit width can be 8 bits.

[0132] In practice, the tensor computation coprocessor receives row-matrix data for each row in row-major order and stores it in the B data store. Then, it checks if the initial data type of the B matrix has a bit width of 6 bits (i.e., whether the data type is FP6). If not, it transposes each row of the B matrix data into 8-bit arrays and maps it to the corresponding column in the B data buffer. If so, it pads each data element of each row of the B matrix with zeros to 8 bits. Then, it transposes each row of the zero-padded data into 8-bit arrays and maps it to the corresponding column in the B data buffer.

[0133] For example, after receiving row-matrix data in B-matrix format, the tensor computation coprocessor can store it in the B data store in row-matrix format. Then, if the initial data type of the B matrix is ​​FP6, each FP6 data element in each row of the B matrix data needs to be padded with zeros to 8 bits within the B data store. Then, each row of the B matrix data after zero-padding is transposed in 8-bit format to obtain column matrix data, which is then mapped to the corresponding column in the B data buffer. If the initial data type of the B matrix is ​​FP4 / FP8 / INT8, zero-padding is not required; each row of the B matrix is ​​directly transposed in 8-bit format, and the resulting column data is mapped to the corresponding column in the B data buffer, thus storing the B matrix data in column-matrix format in the B data buffer. Simultaneously, the tensor computation coprocessor can also receive scaling factors generated column-by-column for the B matrix and pass them into the B scaling factor buffer in column-matrix format.

[0134] Optionally, after obtaining the second target matrix, if the second target matrix is ​​a B matrix, each row of the second target matrix can be passed to the B data store row by row. Based on the target data type of the second target matrix, it can be determined whether to pad each row with zeros. If so, zeros can be added to 8 bits, then transposed according to the 8-bit format and mapped to the corresponding column in the B data buffer. If not, each row of the B matrix stored in the B data store can be directly transposed according to the 8-bit format to obtain a column-major format, and the transposed column matrix data can be mapped to the corresponding column in the B data buffer. Simultaneously, the tensor computation coprocessor can also receive the scaling factor generated for each column of the second target matrix and pass it to the B scaling factor buffer in the tensor computation coprocessor in column-major format.

[0135] In one embodiment, since the matrix data of the B matrix needs to be transposed according to a second preset bit width when cached in the B data buffer, different transposition situations may occur depending on the initial data type of the B matrix data. Therefore, different transposition situations need to be handled separately. Specifically, when transposing according to the second preset bit width, since the data bit width of FP6 / FP8 / INT8 type B matrices is the second preset bit width, each transposition can transpose the entire row of matrix data of the B matrix into an entire column of matrix data. Thus, when subsequently scheduled to the computing array, scheduling can be achieved directly by the entire column. Therefore, for FP6 / FP8 / INT8 type B matrices, after transposing according to 8 bits and caching in the B data buffer, scheduling can be directly performed according to each column of data cached in the B data buffer. When the data type of the B matrix is ​​FP4 type, since the data bit width of this type is 4 bits, transposing according to 8 bits transposes two data elements. In order to separate the two transposed data elements during scheduling, this application embodiment also proposes a double transposition method.

[0136] Specifically, if the bit width corresponding to the initial data type of the second matrix is ​​the third preset bit width, then for any row of matrix data in the second matrix, the row of matrix data is mapped to the corresponding column in the second cache according to the second preset bit width; wherein, the second preset bit width is twice the third preset bit width.

[0137] Here, the second preset bit width is twice the third preset bit width. Specifically, when the second preset bit width is 4 bits, the third preset bit width can be 4 bits. When the third preset bit width is 4 bits, the initial data type of the B matrix can be FP4.

[0138] In practice, when the B matrix is ​​of type FP4, for each row of B matrix data in the B datastore input to the tensor computation coprocessor, the matrix data in that row can be transposed according to the second preset bit width. This transposes the B matrix data based on two data elements, which is also the first transpose in the aforementioned double transpose. Then, each transposed B matrix data can be mapped to the corresponding column in the B data buffer.

[0139] Furthermore, regarding the step of "scheduling the matrix data in the input buffer corresponding to the second matrix to the computing array in blocks", each column of matrix data in the second buffer can be divided into two columns of sub-matrix data according to the third preset bit width, and the matrix data of the second matrix can be scheduled to the computing array in blocks according to the relative column positions of the two columns of sub-matrix data corresponding to each column of matrix data; wherein, the relative column positions are used to indicate the order of the data elements of the sub-matrix data in the two data elements included in the corresponding column of matrix data.

[0140] Here, since each row of the transposed B matrix data contains two data elements during the first transpose according to the second preset bit width, the column corresponding to the preceding data element in each row of each B matrix data can be called a sub-matrix data column, and the column corresponding to the following data element in each row of each B matrix data column can be called another sub-matrix data column. The relative column positions of the two sub-matrix data columns are the preceding and following positions of the data elements in the same row of the two sub-matrix data columns in the corresponding column of matrix data in the B data buffe, or they can also be called the high and low positions.

[0141] In practice, when scheduling the B matrix data from each column in the B data buffer to the computation array, each column of matrix data in the B data buffer can be divided into a sub-matrix data column corresponding to the first 4 bits and a sub-matrix data column corresponding to the last 4 bits, using 8 bits each. Then, based on the relative column position (i.e., high and low bit position) of each sub-matrix data column within its corresponding column of B matrix data in the B data buffer, the two sub-matrix data columns are transposed a second time, thereby assigning the first 4 bits and the last 4 bits to different columns. Finally, based on the result of the second transpose, the B matrix data in the B data buffer is scheduled to the computation array.

[0142] Optionally, if the bit width corresponding to the initial data type of the second matrix is ​​a third preset bit width, then for any row of matrix data in the second matrix, the row of matrix data is transposed for the first time according to the second preset bit width to obtain the initial column matrix data corresponding to the first transpose. Then, according to the third preset bit width, each initial column matrix data is divided into two sub-matrix data, and according to the relative column positions of the two sub-matrix data corresponding to each initial column matrix data, a second transpose is performed on each initial column matrix data to obtain the transposed target column matrix data. Then, each target column matrix data is mapped to the corresponding column in the B data buffer.

[0143] For example, regarding the mapping of B matrix data, since the tensor computation coprocessor internally uses inner product arrays, it is necessary to convert the B matrix data to column-major format. To facilitate computation, a double transpose method is used during the transpose mapping of the B matrix data to transpose the 8-bit / 4-bit wide B matrix data. For 8-bit wide B matrix data (wherein, FP6 type B matrix data has been padded with zeros to 8 bits in the B data store), this application can support input matrix sizes of at least M32N8K32 / M16N16K32 / M8N32K32. For 4-bit wide B matrix data matrix computation, it supports input matrix size calculations of M32N8K64, M16N16K64, and M8N32K64.

[0144] When transposing a B matrix, because the FP4 data type undergoes K extension (i.e., K is 64), the FP4 and FP8 data types occupy the same amount of data storage space. During the first transpose, block transpose units can be used to transpose each row of the B matrix data in 8-bit units, converting the row-matrix input to column-matrix format. After the first transpose, if the initial data type of the B matrix is ​​FP6 or FP8, each row of the B matrix has undergone a complete transpose operation, accurately converting the row-matrix input to column-matrix format. Therefore, each column of the transposed B matrix data can be directly mapped to the corresponding column in the B databuffer for storage.

[0145] If the initial data type of matrix B is FP4, the first transpose is performed by transposing two data elements of each row of matrix B. Therefore, the FP4 row matrix data of matrix B at this point is not completely transposed, but only partially transposed. In this case, a fine-grained transpose module can be used to perform a second transpose on the result of the first transpose. Specifically, the fine-grained transpose module can extract the first 4 bits and the last 4 bits of each 8-bit data obtained after the first transpose of each row of matrix B data. The first 4 bits are mapped to the high-order bits, and the last 4 bits are mapped to the low-order bits, completing the complete transpose. Afterwards, each column of matrix B data obtained after the second transpose can be mapped to the corresponding column in the B data buffer for storage.

[0146] Meanwhile, the tensor computation coprocessor can receive scaling factors generated column-wise by the B matrix and map them into the B scaling factor buffe in row-major format.

[0147] like Figure 4 The diagram shown illustrates the data transpose process of a B matrix of FP6 / FP8 data type according to an embodiment of this application. The B matrix has a size of 32×8, each data element is 8 bits wide, and the B matrix contains 256 8-bit data elements. Figure 4 The data in the matrix ranges from 0 to 256. After transposing the data in the matrix according to an 8-bit size, the row data of matrix B is transposed into the column data.

[0148] like Figure 5 The diagram shown illustrates the data transpose process of a B matrix of FP4 data type according to an embodiment of this application. The B matrix has a size of 64×8, each data element is 4 bits wide, and there are a total of 512 4-bit data elements in the B matrix (for simplicity). Figure 5 (Data elements in some rows are represented by ellipses), that is... Figure 4 The data ranges from 0 to 511. After the first transpose using 8 bits, a 4x64 matrix is ​​obtained. A second transpose yields an 8x64 matrix. Each column of this 8x64 matrix can then be stored in its corresponding column in the B data buffer. Figure 4 and Figure 5 In the image, the purple section illustrates the transpose of a set of data before and after column transposition.

[0149] In one embodiment, when the matrix to be processed is a matrix to be added, the input buffer may include a third buffer. Here, the matrix to be added may be the C matrix mentioned above. The third buffer and third storage unit corresponding to the tensor computation coprocessor for the C matrix can be referred to as the C matrix data buffer and C data store, respectively. Since when performing the A*B+C operation, the result of multiplying the row data of matrix A with the column data of matrix B needs to be added row by row to the matrix data of matrix C, a set of matrix data of matrix C is equivalent to a row of matrix data of matrix C.

[0150] The step of "matrix data in the matrices to be added to the corresponding input buffers of the matrices to be added" can be implemented as follows: If the data type of the matrices to be added is a preset data type, then for each row of matrix data in the matrix to be added, each data element in that row of matrix data is padded with zeros to the fourth preset bit width, and the zero-padded row of matrix data is mapped to the corresponding row in the third cache.

[0151] Here, since the data type of the C matrix in this embodiment supports FP16 and FP32, the preset data type can be FP16. The fourth preset bit width can be 32 bits.

[0152] In practice, the tensor computation coprocessor can receive each row of C matrix data in row-major order and first store the received C matrix data in the C data store. Then, it determines whether the initial data type of the C matrix is ​​FP16 (e.g., it can be determined by the C datatype in the task parameters). If not, it can directly map each row of the B matrix data from the C data store to the corresponding row in the C data buffer. If so, it can then, for each row of the C matrix data, pad each data element in the C data store with zeros to 32 bits, and then map the zero-padded row of matrix data to the corresponding row in the C data buffer for storage.

[0153] In one embodiment, the input cache corresponding to each matrix to be processed may include multiple caches, and the number of scaling factor caches corresponding to each matrix to be multiplied is the same as the number of input caches corresponding to each matrix to be processed. Simultaneously, the number of task parameter units, the number of storage units, and the number of scaling factor caches can also be the same as the number of input caches corresponding to each matrix to be processed. For example, if there are X input caches for each matrix to be processed, the number of storage units corresponding to each matrix to be processed can be X, the number of scaling factor caches for each matrix to be multiplied can also be X, and the number of task parameter caches can also be X. Here, X is greater than or equal to 2, and the number of X can be adaptively set according to the computing resources of the computing platform on which the tensor computing coprocessor is deployed.

[0154] After mapping the matrix data from multiple matrices to be processed to the input buffers corresponding to each matrix, and mapping the scaling factor set for each set of matrix data for each matrix to be multiplied to the scaling factor buffer corresponding to each matrix to be multiplied, idle input buffers and scaling factor buffers can be selected from the multiple input buffers corresponding to each matrix to be processed; idle storage units can be selected from the storage units corresponding to each matrix to be multiplied; and idle task parameter buffers can be selected from the task parameter buffers. The idle task parameter buffers are used to cache the task parameters for new tensor computation tasks. The idle storage units are used to store the matrix data from each new matrix to be processed. The idle input buffers are used to map the matrix data from each new matrix to be processed (that is, to map the matrix data from each matrix to be processed stored in the idle storage units to the idle input buffers), and the idle scaling factor buffers are used to map the scaling factors set for the new matrices to be multiplied in each new matrix to be processed.

[0155] For example, if the number of task parameter caches, the number of input caches for each matrix to be processed, and the number of scaling factor caches for the matrices to be added are all two, it can be determined that the tensor computation coprocessor uses a dual-buffer mode (i.e., two caches are stored alternately). For the mapping of task parameters, the writing and output of task parameters can be updated according to a ping-pong read-write mechanism. In dual-buffer mode, the writing and reading of task parameters are isolated, allowing tensor computation and input data processing to be executed in parallel. When the computation array completes the reading of input data (including task parameters, matrix data, and scaling factors) and completes the corresponding tensor computation task, it will raise the finish signal, thereby instructing the control to switch to a new output buffer (i.e., a new task parameter cache). This allows the task parameters corresponding to the new tensor computation task to be obtained from the switched output buffer and scheduled to the computation array for tensor computation.

[0156] Similarly, in dual-buffer mode, the matrix data mapping corresponding to the matrix to be processed can also be updated by updating the matrix data write and output according to the ping-pong read-write mechanism. In dual-buffer mode, the writing and reading of new matrix data are isolated, allowing tensor computation and input data processing to be executed in parallel. Once the computation array has finished reading the input data (including task parameters, matrix data, and scaling factors) and completed the corresponding tensor computation task, it will raise the finish signal, indicating the selection of a new output buffer (i.e., a new input buffer). This allows the matrix data corresponding to the new tensor computation task to be obtained from the switched output buffer and scheduled in blocks to the computation array for tensor computation. In dual-buffer mode, the scaling factor mapping corresponding to the matrix to be multiplied can also be updated by updating the scaling factor write and output according to the ping-pong read-write mechanism. In dual-buffer mode, the writing and reading of new scaling factors are isolated, allowing tensor computation and input data processing to be executed in parallel. Once the computing array has finished reading the input data (including task parameters, matrix data, and scaling factors) and completed the corresponding tensor computation task, it will raise the finish signal to indicate the selection of a new output buffer (i.e., a new scaling factor buffer) so that the scaling factor corresponding to the new tensor computation task can be obtained from the switched output buffer and scheduled to the computing array in blocks for tensor computation.

[0157] It should be noted that when integrating the tensor computing coprocessor of this application into the required computing platform, the number of caches can be flexibly configured according to the processor configuration requirements and computing resources. The number of caches does not affect the correctness of the calculation, but is only related to the computing throughput, whether data writing and calculation are isolated, and whether they can be executed in parallel.

[0158] Understandably, when both the number of task parameter caches and the number of matrix data caches are greater than two, it can be determined that a multi-buffer caching mode (i.e., multiple caches storing data alternately) should be enabled. The writing and reading of task parameters, matrix data, and scaling factors in multi-buffer caching mode are similar to those in dual-buffer caching mode and will not be elaborated here. For example, multiple task parameter caches may include task parameter buffer0, task parameter buffer1, and task parameter buffer2; the input cache for each matrix to be processed may include input buffer0, input buffer1, and input buffer2; and the scaling factor cache for each matrix to be multiplied may include scaling factor buffer0, scaling factor buffer1, and scaling factor buffer2. After the computation array finishes reading the input data from task parameter buffer0, input buffer0, and scaling factor buffer0 and completes the tensor computation task, it will pull the finish signal high, indicating a switch to task parameter buffer1, input buffer1, and scaling factor buffer1. At this point, task parameter buffer0, input buffer0, and scaling factor buffer0 can be released and used to receive the task parameters, matrix data, and scaling factor for a new tensor computation task. Similarly, after the computation array finishes reading the input data from task parameter buffer1, input buffer1, and scaling factor buffer1 and completes the tensor computation task, it will pull the finish signal high, indicating a switch to task parameter buffer2, input buffer2, and scaling factor buffer2. At this point, task parameter buffer1, input buffer1, and scaling factor buffer1 can be released and used to receive the task parameters, matrix data, and scaling factor for a new tensor computation task. After the computing array reads the input data for a new tensor computation task from task parameter buffer2, input buffer2, and scaling factor buffer2 and completes the tensor computation task, it will pull the finish signal high, thereby indicating that the control will switch to task parameter buffer0, input buffer0, and scaling factor buffer0. At this time, task parameter buffer2, input buffer2, and scaling factor buffer2 can be released and can be used to receive the task parameters, matrix data, and scaling factor corresponding to the new tensor computation task.

[0159] For example, in double-buffer mode, when mapping matrix A data, state machine switching can be used. The initial state is write buffer0. After all rows of matrix A data are written to buffer0, the write is complete, and then the state automatically switches to buffer1, writing alternately. When the computation array feedback buffer0 is finished (i.e., all data in buffer0 is written to the computation array), the finish signal goes high, indicating that the currently used buffer0 will be marked as free; otherwise, it remains in use. New matrix A data will only be accepted for writing if and only if there is at least one free buffer in the A data buffer that can be written to; otherwise, it will wait for the buffer to become free before allowing writing. The mapping of the scaling factor of matrix A is also stored in a double-buffer buffer, and the state machine switching is the same as the double-buffer write switching in the A matrix data mapping part. For the mapping of matrix B data, the transposed B matrix data can be written to the B data double buffer. The double-buffer switching and output logic for B matrix data are the same as the double-buffer write switching in the A matrix data mapping part, and will not be described again here. The mapping of scaling factors for matrix B is also handled using a double-buffered cache, with state machine switching identical to the double-buffered write switching in the matrix A data mapping section. For the mapping of matrix C data, the row-major input C matrix data can be mapped to a C data doubleBuffer. The double-buffered switching and output logic for C matrix data are the same as the double-buffered write switching in the matrix A data mapping section, and will not be elaborated further here. Regarding the caching of task parameters, because pipelined processing is required, to ensure separation of computation and data writing, not only matrix data and scaling factors are double-buffered, but task parameters also require corresponding double-buffered caching. That is, each time matrix A data is written, the task parameters are updated, and internally, a double-buffered cache is used to map the task parameters. The double-buffered mode of the task parameters, along with the A / B / C matrix data, uses a double-buffered switching mechanism to transmit them to the subsequent computation array. After the A, B, and C matrix data and their scaling factors have all been stored and mapped, the input data (i.e., task parameters, matrix data, and scaling factors) in each buffer are transmitted to the computation array. After the computing array has finished using the input data, it will clear the used buffer storage according to the dual-buffer caching mode, switch the buffer selection, and update the data output.

[0160] Thus, the tensor computation method provided in this application embodiment enables fast and smooth computation of matrix data, allowing the process of inputting matrix data from an external bus or other interface into an input buffer and scheduling it from the input buffer to the computation array for computation. Compared to the traditional method of directly inputting matrix data from external memory to the computation array for computation, this method has a higher computation-to-memory ratio and enables more data reuse. Meanwhile, utilizing a hardware data type conversion unit for data type conversion offers the following advantages: 1. No division logic: Data type conversion is performed using the exponential difference method, effectively avoiding the high costs associated with deploying a divider; 2. Low latency and high throughput: Concurrent conversion of K=16 / 3264 channels per cycle is possible; 3. Flexible precision configuration: Supports five MX floating-point formats and INT8, adapting to various AI inference scenarios; 4. Standard format output: The scaling factor of the E8M0 format is compatible with the MX specification of the Open Compute Project (OCP); 5. Compact encoded output: Bit-level packaging improves bandwidth utilization. Bit-level packaging means that the matrix data converted by the data type conversion unit and the scaled data are packaged together for output, i.e., the converted data... Outputting the scaling factor along with the data; 6. Comprehensive anomaly management, with NaN / ±∞ / ±0 handling to prevent the propagation of computational anomalies. Furthermore, this application employs a Chisel-based design, where the k-dimensionality, computation array size, and external input data bandwidth are all configurable in software, and the hardware computation scale can be directly generated from hardware configuration parameters.

[0161] like Figure 6The diagram shown illustrates the overall structure of a tensor computation process according to an embodiment of this application. When the initial data types of the two matrices to be multiplied are FP4 / FP6 / FP8 / INT8, external data can be transferred via a preset bus (such as an Advanced eXtensible Interface (AXI) bus). This involves transmitting the matrix data of matrix A (A data), matrix B (B data), and matrix C (C data), as well as the scaling factors (A scaling factor and B scaling factor) corresponding to matrices A and B, stored in external memory, to the tensor computation coprocessor. Simultaneously, the task parameters (parameterset) of the tensor computation task can be transmitted to the tensor computation coprocessor via external configuration signals. Then, A data can be mapped to an idle buffer in the A data double buffer, which includes A data buffer0 and A data buffer1. For each row of matrix A, the corresponding scaling factor can be mapped to an idle buffer in the A scaling factor double buffer, which includes A scaling factor buffer0 and A scaling factor buffer1. Similarly, B data is transposed and mapped to an idle buffer in the B matrix data double buffer, which also includes B data buffer0 and B data buffer1. If the initial data type of matrix B is FP4, this is the first transpose; a second transpose (fine-grained transpose) is required when scheduling the B matrix data to the computation array. If the initial data type of matrix B is not FP4, a single transpose is sufficient, and a second transpose is not needed before scheduling to the computation array.For each column of matrix B, the corresponding B scaling factor can be mapped to an idle buffer in the B scaling factor double buffer, which includes B scaling factor buffer0 and B scaling factor buffer1. Similarly, C data can be mapped to an idle buffer in the C matrix data double buffer, which includes C data buffer0 and C data buffer1. Task parameters can be mapped to an idle buffer in the parameter data double buffer, which includes parameter data buffer0 and parameter data buffer1. Then, the data buffered in each input buffer can be scheduled to the computation array, and the FP4 type B matrix data is transposed a second time before scheduling. Finally, tensor computation is performed using the computation array, and the tensor computation result is output to external memory.

[0162] Thus, this application embodiment supports data scheduling (i.e., scheduling from the scale of computational requirements to the actual hardware array) of micro-scaling precision formats under the OCP specification MXINT8 / MXFP8(E5M2, E4M3) / MXFP6(E3M2, E2M3) / MXFP4(E2M1) standards; supports data mapping of low-bit floating-point data types; supports dual hardware transposition for B-matrix data within the tensor computation coprocessor to achieve transpose mapping and recombination of matrix data; and supports data mapping for fine-grained quantization micro-scaling range factors (i.e., scaling factors). In the data type conversion unit, the conversion of FP32 K-dimensional vectors to MXINT8 / FP8 / FP6 / FP4 precision can be completed through this hardware. This data type conversion not only reduces memory access but also effectively reduces divider resource consumption by replacing exponential division with exponential subtraction. Furthermore, the tensor computation coprocessor's support is flexibly configurable through the chisel parameter, eliminating the need for hardware redesign.

[0163] Based on the same inventive concept, this disclosure also provides a tensor calculation device corresponding to the tensor calculation method. Since the principle of the device in this disclosure for solving the problem is similar to the tensor calculation method described above in this disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0164] like Figure 7 The diagram shown is a schematic of a tensor computing device provided in an embodiment of this disclosure, comprising: The acquisition module 701 is used to acquire the task parameters of the tensor computation task corresponding to the tensor computation coprocessor and the matrix data of multiple matrices to be processed; the multiple matrices to be processed include two matrices to be multiplied and one matrix to be added; The conversion module 702 is used to convert the first target matrix into a second target matrix of the target data type if there is a first target matrix in the matrix to be multiplied that has a bit width corresponding to the initial data type that is greater than the target bit width, and to determine the scaling factor corresponding to the second target matrix; the bit width corresponding to the target data type is not greater than the target bit width; The calculation module 703 is used to schedule the task parameters, the matrix data and scaling factor of the second target matrix, the matrix data of the third target matrix other than the first target matrix in the matrix to be multiplied, and the matrix data of the matrix to be added to the calculation array of the tensor coprocessor for tensor calculation to obtain the tensor calculation result.

[0165] In one possible implementation, the conversion module 702, when using the data type conversion unit to convert the first target matrix into a second target matrix of the target data type and determining the scaling factor corresponding to the second target matrix, is used to: For any set of initial matrix data in the first target matrix, the data type conversion unit is used to determine the scaling factor corresponding to the initial matrix data based on the absolute value of each data element in the initial matrix data; a set of initial matrix data is a row of data or a column of data in the first target matrix; Based on the exponent value of each data element in the initial matrix data and the scaling factor, each data element is converted into a data element of the target data type to obtain a set of target matrix data corresponding to the initial matrix data in the second target matrix; The scaling factor corresponding to the initial matrix data is used as the scaling factor for the target matrix data.

[0166] In one possible implementation, the conversion module 702, when using the data type conversion unit to determine the scaling factor corresponding to the initial matrix data based on the absolute value of each data element in the initial matrix data, is configured to: Using the data type conversion unit, the maximum value is determined from the absolute value of each data element in the initial matrix data; Extract the exponent value of the maximum value and determine whether the exponent value is a preset outlier. If not, then the scaling factor corresponding to the initial matrix data is determined based on the value of the exponent.

[0167] In one possible implementation, the conversion module 702, when converting each data element into a data element of the target data type according to the exponent value of each data element in the initial matrix data and the scaling factor, to obtain a set of target matrix data corresponding to the second target matrix, is used to: When the target data type includes floating-point type and the bit width is less than or equal to the preset bit width, the initial conversion data of each data element is determined according to the exponent value of each data element in the initial matrix data, the scaling factor and the preset conversion value under the target data type. Based on the mantissa bit width of the target data type, the mantissa value of each data element is truncated to obtain the truncated mantissa value. Based on the exponent offset corresponding to the target data type, the initial conversion data of each data element is adjusted to obtain the target conversion data; Based on the sign bit value, target transformed data, and truncated mantissa value of each data element, determine the target data type data element corresponding to each data element, and obtain a set of target matrix data corresponding to the initial matrix data in the second target matrix.

[0168] In one possible implementation, the conversion module 702, when converting each data element into a data element of the target data type according to the exponent value of each data element in the initial matrix data and the scaling factor, to obtain a set of target matrix data corresponding to the second target matrix, is used to: When the target data type includes an integer type and the bit width is less than or equal to the preset bit width, the mantissa value of each data element is padded to obtain the initial mantissa value. Based on the difference between the exponent value of each data element and the scaling factor, the initial mantissa value of each data element is shifted to obtain the target mantissa value. Based on the sign bit value and the target mantissa value of each data element, the data elements of the target data type corresponding to each data element are determined, and a set of target matrix data corresponding to the initial matrix data in the second target matrix is ​​obtained.

[0169] In one possible implementation, the conversion module 702, before converting the first target matrix into a second target matrix of the target data type, is further configured to: For any set of initial matrix data in the first target matrix, an outlier check is performed on each data element in the initial matrix data using a data type conversion unit to determine whether there are any abnormal data elements. If so, the abnormal data element will be modified to a preset element.

[0170] In one possible implementation, the device further includes a mapping module 704, for: If the first target matrix is ​​not present in the matrix to be multiplied, the matrix data in the plurality of matrices to be processed are mapped to the input buffer corresponding to each matrix to be processed, and the scaling factor set for each group of matrix data of each matrix to be multiplied is mapped to the scaling factor buffer corresponding to each matrix to be multiplied. The calculation module 703 is further configured to: schedule the task parameters to the calculation array, and schedule the matrix data in each input cache and the scaling factors in each scaling factor cache to the calculation array in blocks; use the calculation array to determine the calculation result of each block scheduling based on the task parameters and the matrix data and scaling factors of each block scheduling; and determine the tensor calculation result based on the calculation result of each block scheduling.

[0171] In one possible implementation, the computing module 703, when scheduling the task parameters, the matrix data and scaling factor of the second target matrix, the matrix data of the third target matrix other than the first target matrix in the matrix to be multiplied, and the matrix data of the matrix to be added to the computing array of the tensor coprocessor for tensor computation to obtain the tensor computation result, is used to: The matrix data of the second target matrix, the matrix data of the third target matrix, and the matrix data of the matrix to be added are respectively mapped to the input buffer corresponding to each matrix to be processed, and the scaling factor of the second target matrix and the scaling factor of the third target matrix are mapped to the scaling factor buffer corresponding to each matrix to be multiplied. The task parameters are scheduled to the computing array, and the matrix data in each input cache and each scaling factor in each scaling factor cache are scheduled to the computing array in blocks; Using a computing array, the calculation result for each block scheduling is determined based on the task parameters, the matrix data for each block scheduling, and the scaling factor. The tensor calculation result is determined based on the calculation result of each block scheduling.

[0172] In one possible implementation, when the matrix to be processed is the first matrix of two matrices to be multiplied, the input buffer includes a first buffer, and the scaling factor buffer includes a first scaling factor buffer; a set of matrix data of the first matrix is ​​a row of matrix data in the first matrix; The mapping module 704, when mapping matrix data in the first matrix to the input buffer corresponding to the first matrix, and mapping the scaling factor set for each group of matrix data in the first matrix to the scaling factor buffer corresponding to the first matrix, is used for: For any row of matrix data in the first matrix, the scaling factor set for that row of matrix data is mapped to the first scaling factor cache; If the bit width corresponding to the initial data type of the first matrix is ​​the first preset bit width, then each data element in the row of matrix data is padded with zeros to the second preset bit width, and the zero-padded row of matrix data is mapped to the corresponding row in the first cache.

[0173] In one possible implementation, when the matrix to be processed is the second matrix of two matrices to be multiplied, the input buffer includes a second buffer, and the scaling factor buffer includes a second scaling factor buffer; a set of matrix data of the second matrix is ​​a column of matrix data in the second matrix; The mapping module 704, when mapping matrix data in the second matrix to the input buffer corresponding to the second matrix, and mapping the scaling factor set for each group of matrix data in the second matrix to the scaling factor buffer corresponding to the second matrix, is used for: The scaling factor set for each column of matrix data in the second matrix is ​​mapped to the second scaling factor cache; If the bit width corresponding to the initial data type of the second matrix is ​​the first preset bit width, then for any row of matrix data in the second matrix, each data element in the row of matrix data is padded with zeros to the second preset bit width, and the zero-padded row of matrix data is mapped to the corresponding column in the second cache according to the second preset bit width.

[0174] In one possible implementation, the mapping module 704 is further configured to: If the bit width corresponding to the initial data type of the second matrix is ​​the third preset bit width, then for any row of matrix data in the second matrix, the row of matrix data is mapped to the corresponding column in the second cache according to the second preset bit width; wherein, the second preset bit width is twice the third preset bit width; The computing module 703, when scheduling the matrix data in the input buffer corresponding to the second matrix to the computing array in blocks, is used for: According to the third preset bit width, each column of matrix data in the second cache is divided into two columns of sub-matrix data, and the matrix data of the second matrix is ​​scheduled to the computing array in blocks according to the relative column positions of the two columns of sub-matrix data corresponding to each column of matrix data; wherein, the relative column positions are used to indicate the order of the data elements of the sub-matrix data in the two data elements included in the corresponding column of matrix data.

[0175] In one possible implementation, when the matrix to be processed is the matrix to be added, the input buffer includes a third buffer; The mapping module 704, when mapping matrix data in the matrix to be added to the input buffer corresponding to the matrix to be added, is used for: If the data type of the matrix to be added is a preset data type, then for each row of matrix data of the matrix to be added, each data element in the row of matrix data is padded with zeros to the fourth preset bit width, and the zero-padded row of matrix data is mapped to the corresponding row in the third cache.

[0176] In one possible implementation, the input buffer corresponding to each matrix to be processed includes multiple buffers, and the number of scaling factor buffers corresponding to each matrix to be multiplied is the same as the number of input buffers corresponding to each matrix to be processed. The mapping module 704, after mapping the matrix data in the plurality of matrices to be processed to the input buffer corresponding to each matrix to be processed, and mapping the scaling factor set for each group of matrix data of each matrix to be multiplied to the scaling factor buffer corresponding to each matrix to be multiplied, is further configured to: Select an idle input buffer from the multiple input buffers corresponding to each matrix to be processed, and select an idle scaling factor buffer from the scaling factor buffers corresponding to each matrix to be multiplied. Utilize the idle input buffer to map the matrix data in the new individual matrices to be processed, and utilize the idle scaling factor buffer to map the scaling factor set for the new multiplication matrix in the new individual matrices to be processed.

[0177] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0178] Based on the same technical concept, embodiments of this application also provide a computer device. (Refer to...) Figure 8 The diagram shown is a structural schematic of a computer device provided in an embodiment of this application, comprising: The system comprises a processor 801, a memory 802, and a bus 803. The memory 802 stores machine-readable instructions executable by the processor 801. The processor 801 executes these machine-readable instructions, performing the following steps: S101: Obtaining task parameters and matrix data of multiple matrices to be processed from the tensor computation coprocessor; the multiple matrices to be processed include two matrices to be multiplied and one matrix to be added; S102: If, among the matrices to be multiplied, there exists a first target matrix with a bit width corresponding to an initial data type equal to the target bit width, then using a data type conversion unit, converting the first target matrix into a second target matrix of the target data type, and determining the scaling factor corresponding to the second target matrix; the bit width corresponding to the target data type is less than the target bit width; and S103: Scheduling the task parameters, the matrix data and scaling factor of the second target matrix, the matrix data of the third target matrix (excluding the first target matrix) among the matrices to be multiplied, and the matrix data of the matrix to be added to the computation array of the tensor coprocessor for tensor computation, obtaining the tensor computation result.

[0179] The aforementioned memory 802 includes a main memory 8021 and an external memory 8022. The main memory 8021, also known as internal memory, is used to temporarily store the computational data in the processor 801, as well as the data exchanged with external memory such as a hard disk 8022. The processor 801 exchanges data with the external memory 8022 through the main memory 8021. When the computer device is running, the processor 801 and the memory 802 communicate through the bus 803, so that the processor 801 executes the execution instructions mentioned in the above method embodiments.

[0180] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the tensor computation method described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0181] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the tensor calculation method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0182] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0183] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it can be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0184] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0185] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0186] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0187] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A tensor computation method, characterized in that, The method includes: Obtain the task parameters of the tensor computation task corresponding to the tensor computation coprocessor and the matrix data of multiple matrices to be processed; the multiple matrices to be processed include two matrices to be multiplied and one matrix to be added; If, in the matrices to be multiplied, there exists a first target matrix whose bit width corresponding to the initial data type is greater than the target bit width, then the data type conversion unit is used to convert the first target matrix into a second target matrix of the target data type, and the scaling factor corresponding to the second target matrix is ​​determined; the bit width corresponding to the target data type is not greater than the target bit width. The task parameters, the matrix data and scaling factor of the second target matrix, the matrix data of the third target matrix other than the first target matrix in the matrix to be multiplied, and the matrix data of the matrix to be added are scheduled to the computing array of the tensor coprocessor for tensor computation to obtain the tensor computation result.

2. The method according to claim 1, characterized in that, The step of using a data type conversion unit to convert the first target matrix into a second target matrix with the target data type and determining the scaling factor corresponding to the second target matrix includes: For any set of initial matrix data in the first target matrix, the data type conversion unit is used to determine the scaling factor corresponding to the initial matrix data based on the absolute value of each data element in the initial matrix data; a set of initial matrix data is a row of data or a column of data in the first target matrix; Based on the exponent value of each data element in the initial matrix data and the scaling factor, each data element is converted into a data element of the target data type to obtain a set of target matrix data corresponding to the initial matrix data in the second target matrix; The scaling factor corresponding to the initial matrix data is used as the scaling factor for the target matrix data.

3. The method according to claim 2, characterized in that, The step of using the data type conversion unit to determine the scaling factor corresponding to the initial matrix data based on the absolute value of each data element in the initial matrix data includes: Using the data type conversion unit, the maximum value is determined from the absolute value of each data element in the initial matrix data; Extract the exponent value of the maximum value and determine whether the exponent value is a preset outlier. If not, then the scaling factor corresponding to the initial matrix data is determined based on the value of the exponent.

4. The method according to claim 2, characterized in that, The step of converting each data element into a data element of the target data type based on the exponent value of each data element in the initial matrix data and the scaling factor, to obtain a set of target matrix data corresponding to the initial matrix data in the second target matrix, includes: When the target data type includes floating-point type and the bit width is less than or equal to the preset bit width, the initial conversion data of each data element is determined according to the exponent value of each data element in the initial matrix data, the scaling factor and the preset conversion value under the target data type. Based on the mantissa bit width of the target data type, the mantissa value of each data element is truncated to obtain the truncated mantissa value. Based on the exponent offset corresponding to the target data type, the initial conversion data of each data element is adjusted to obtain the target conversion data; Based on the sign bit value, target transformed data, and truncated mantissa value of each data element, determine the target data type data element corresponding to each data element, and obtain a set of target matrix data corresponding to the initial matrix data in the second target matrix.

5. The method according to claim 2, characterized in that, The step of converting each data element into a data element of the target data type based on the exponent value of each data element in the initial matrix data and the scaling factor, to obtain a set of target matrix data corresponding to the initial matrix data in the second target matrix, includes: When the target data type includes an integer type and the bit width is less than or equal to the preset bit width, the mantissa value of each data element is padded to obtain the initial mantissa value. Based on the difference between the exponent value of each data element and the scaling factor, the initial mantissa value of each data element is shifted to obtain the target mantissa value. Based on the sign bit value and the target mantissa value of each data element, the data elements of the target data type corresponding to each data element are determined, and a set of target matrix data corresponding to the initial matrix data in the second target matrix is ​​obtained.

6. The method according to claim 1, characterized in that, Before converting the first target matrix into a second target matrix of the target data type, the method further includes: For any set of initial matrix data in the first target matrix, an outlier check is performed on each data element in the initial matrix data using a data type conversion unit to determine whether there are any abnormal data elements. If so, the abnormal data element will be modified to a preset element.

7. The method according to claim 1, characterized in that, The method further includes: If the first target matrix is ​​not present in the matrix to be multiplied, the matrix data in the plurality of matrices to be processed are mapped to the input buffer corresponding to each matrix to be processed, and the scaling factor set for each group of matrix data of each matrix to be multiplied is mapped to the scaling factor buffer corresponding to each matrix to be multiplied. The task parameters are scheduled to the computing array, and the matrix data in each input cache and each scaling factor in each scaling factor cache are scheduled to the computing array in blocks; Using a computing array, the calculation result for each block scheduling is determined based on the task parameters, the matrix data for each block scheduling, and the scaling factor. The tensor calculation result is determined based on the calculation result of each block scheduling.

8. The method according to claim 1, characterized in that, The process of scheduling the task parameters, the matrix data and scaling factor of the second target matrix, the matrix data of the third target matrix (excluding the first target matrix) in the matrix to be multiplied, and the matrix data of the matrix to be added to the tensor coprocessor's computing array for tensor computation to obtain the tensor computation result includes: The matrix data of the second target matrix, the matrix data of the third target matrix, and the matrix data of the matrix to be added are respectively mapped to the input buffer corresponding to each matrix to be processed, and the scaling factor of the second target matrix and the scaling factor of the third target matrix are mapped to the scaling factor buffer corresponding to each matrix to be multiplied. The task parameters are scheduled to the computing array, and the matrix data in each input cache and each scaling factor in each scaling factor cache are scheduled to the computing array in blocks; Using a computing array, the calculation result for each block scheduling is determined based on the task parameters, the matrix data for each block scheduling, and the scaling factor. The tensor calculation result is determined based on the calculation result of each block scheduling.

9. The method according to claim 7, characterized in that, When the matrix to be processed is the first matrix of two matrices to be multiplied, the input buffer includes a first buffer, and the scaling factor buffer includes a first scaling factor buffer; A set of matrix data in the first matrix is ​​a row of matrix data in the first matrix; Map the matrix data in the first matrix to the input buffer corresponding to the first matrix, and map the scaling factor set for each group of matrix data in the first matrix to the scaling factor buffer corresponding to the first matrix, including: For any row of matrix data in the first matrix, the scaling factor set for that row of matrix data is mapped to the first scaling factor cache; If the bit width corresponding to the initial data type of the first matrix is ​​the first preset bit width, then each data element in the row of matrix data is padded with zeros to the second preset bit width, and the zero-padded row of matrix data is mapped to the corresponding row in the first cache.

10. The method according to claim 7, characterized in that, When the matrix to be processed is the second matrix of two matrices to be multiplied, the input buffer includes a second buffer, and the scaling factor buffer includes a second scaling factor buffer; The set of matrix data in the second matrix is ​​a column of matrix data in the second matrix; Map the matrix data in the second matrix to the corresponding input buffer of the second matrix, and map the scaling factor set for each group of matrix data in the second matrix to the corresponding scaling factor buffer of the second matrix, including: The scaling factor set for each column of matrix data in the second matrix is ​​mapped to the second scaling factor cache; If the bit width corresponding to the initial data type of the second matrix is ​​the first preset bit width, then for any row of matrix data in the second matrix, each data element in the row of matrix data is padded with zeros to the second preset bit width, and the zero-padded row of matrix data is mapped to the corresponding column in the second cache according to the second preset bit width.

11. The method according to claim 10, characterized in that, The method further includes: If the bit width corresponding to the initial data type of the second matrix is ​​the third preset bit width, then for any row of matrix data in the second matrix, the row of matrix data is mapped to the corresponding column in the second cache according to the second preset bit width; wherein, the second preset bit width is twice the third preset bit width; The matrix data in the input buffer corresponding to the second matrix is ​​divided into blocks and scheduled to the computing array, including: According to the third preset bit width, each column of matrix data in the second cache is divided into two columns of sub-matrix data, and the matrix data of the second matrix is ​​scheduled to the computing array in blocks according to the relative column positions of the two columns of sub-matrix data corresponding to each column of matrix data; wherein, the relative column positions are used to indicate the order of the data elements of the sub-matrix data in the two data elements included in the corresponding column of matrix data.

12. The method according to claim 7, characterized in that, When the matrix to be processed is the matrix to be added, the input buffer includes a third buffer; Mapping matrix data in the matrices to be added to the corresponding input buffers of the matrices to be added includes: If the data type of the matrix to be added is a preset data type, then for each row of matrix data of the matrix to be added, each data element in the row of matrix data is padded with zeros to the fourth preset bit width, and the zero-padded row of matrix data is mapped to the corresponding row in the third cache.

13. The method according to claim 7, characterized in that, Each matrix to be processed has multiple input buffers, and the number of scaling factor buffers for each matrix to be multiplied is the same as the number of input buffers for each matrix to be processed. After mapping the matrix data in the plurality of matrices to be processed to the input buffer corresponding to each matrix to be processed, and mapping the scaling factor set for each group of matrix data of each matrix to be multiplied to the scaling factor buffer corresponding to each matrix to be multiplied, the method further includes: Select an idle input buffer from the multiple input buffers corresponding to each matrix to be processed, and select an idle scaling factor buffer from the scaling factor buffers corresponding to each matrix to be multiplied. Utilize the idle input buffer to map the matrix data in the new individual matrices to be processed, and utilize the idle scaling factor buffer to map the scaling factor set for the new multiplication matrix in the new individual matrices to be processed.

14. A tensor computing device, characterized in that, The device includes: The acquisition module is used to acquire the task parameters of the tensor computation task corresponding to the tensor computation coprocessor and the matrix data of multiple matrices to be processed; the multiple matrices to be processed include two matrices to be multiplied and one matrix to be added; The conversion module is used to convert the first target matrix into a second target matrix of the target data type if there is a first target matrix in the matrix to be multiplied that has a bit width corresponding to the initial data type that is greater than the target bit width, and to determine the scaling factor corresponding to the second target matrix; the bit width corresponding to the target data type is not greater than the target bit width; The calculation module is used to schedule the task parameters, the matrix data and scaling factor of the second target matrix, the matrix data of the third target matrix other than the first target matrix in the matrix to be multiplied, and the matrix data of the matrix to be added to the tensor coprocessor's calculation array for tensor calculation to obtain the tensor calculation result.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 13.

16. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 13.

Images