Tensor kernel operation component

Abstract

The invention provides a tensor kernel operation component, and relates to the technical field of artificial intelligence chips, and the tensor kernel operation component comprises a decompression layer which is used for carrying out decompression processing on floating-point number elements in a first matrix of a preset floating-point number system to obtain first floating-point number elements in a preset bit format, or carrying out decompression processing on the first floating-point number elements in the first matrix of the preset floating-point number system to obtain second floating-point number elements in a preset bit format; the decompression module is used for decompressing floating-point number elements in a second matrix of a preset floating-point number system to obtain second floating-point number elements of a preset bit format; the calculation layer is used for multiplying and adding all first floating-point number elements corresponding to the first matrix and all second floating-point number elements corresponding to the second matrix to obtain a floating-point number in a target bit format; and the compression layer is used for receiving the floating-point number in the target bit format and carrying out compression conversion processing on the floating-point number in the target bit format to obtain a floating-point number operation result of a number system of the target floating-point number. The tensor kernel operation component provided by the invention is more in line with the matrix multiplication calculation dimension requirement of a large language model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence chip technology, and in particular to a tensor core computing component. Background Art

[0002] In a large language model, the activation matrix data is usually in FP16, BF16, or FP8 format. If the weight matrix data is not quantized, the weight matrix and the activation matrix have the same format. If the weight matrix is ​​quantized, it is usually stored in UINT8 or UINT4 format.

[0003] To efficiently complete large language model training and inference calculations, Tensor Cores need to support both the activation matrix and the weight matrix in FP16 or BF16 format, or the activation matrix in FP16 or BF16 format, while the weight matrix is ​​in the above fixed-point format.

[0004] However, the computing components in existing hardware acceleration chips are not customized for large language models, resulting in a waste of chip power consumption and area. Therefore, a tensor core computing component is urgently needed to solve the above problems. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a tensor core computing component.

[0006] The present invention provides a tensor core computing component, comprising a decompression layer, a computing layer and a compression layer, wherein: The decompression layer is used to decompress the floating-point elements in the first matrix of the preset floating-point number system to obtain the first floating-point elements in the preset bit format, or to decompress the floating-point elements in the second matrix of the preset floating-point number system to obtain the second floating-point elements in the preset bit format, wherein the first matrix and the second matrix are two matrices in a matrix multiplication operation; the preset floating-point number system includes a 16-bit floating-point number system and an 8-bit floating-point number system; The computing layer is used to perform multiplication and addition operations on all the first floating-point number elements corresponding to the first matrix and all the second floating-point number elements corresponding to the second matrix to obtain a floating-point number in a target bit format; The compression layer is used to receive the target bit format floating point number, and perform compression conversion processing on the target bit format floating point number to obtain a floating point operation result of the target floating point number system.

[0007] According to a tensor core computing component provided by the present invention, the decompression layer includes 512 identical decompression units; the preset bit format is a 20-bit format with bits from the 0th to the 19th, including a first preset bit format and a second preset bit format, wherein: In the case where the preset floating-point number system is the 16-bit floating-point number system, the decompression unit is used to decompress a floating-point number element in the first matrix to obtain the first floating-point number element in the first preset bit format; or to decompress a floating-point number element in the second matrix to obtain the second floating-point number element in the first preset bit format; Wherein, the 19th to 12th bits in the first preset bit format are 8-bit exponents, and the 11th to 0th bits in the first preset bit format are 12-bit signed number mantissas; In the case where the preset floating-point number system is the 8-bit floating-point number system, the decompression unit is used to decompress the floating-point number elements in the two first matrices to obtain the first floating-point number elements in the second preset bit format; or to decompress the floating-point number elements in the two second matrices to obtain the second floating-point number elements in the second preset bit format; Among them, the 19th to 15th bits in the second preset bit format are 5-bit exponent bits, the 14th to 10th bits in the second preset bit format are 5-bit signed number mantissa bits, the 9th to 5th bits in the second preset bit format are 5-bit exponent bits, and the 4th to 0th bits in the second preset bit format are 5-bit signed number mantissa bits.

[0008] According to a tensor core computing component provided by the present invention, the computing layer is composed of 256 identical computing units, and the computing units include a first exponent addition module, a second exponent addition module, a first maximum exponent obtaining module, a second maximum exponent obtaining module, a first mantissa right shift bit number calculation module, a second mantissa right shift bit number calculation module, a first mixed number system mantissa multiplication module, a second mixed number system mantissa multiplication module, a mantissa alignment accumulation module and a normalization module, wherein: The first exponential addition module includes 16 groups of 8-bit adders, and the second exponential addition module includes 16 groups of 5-bit adders; The first mantissa right shift bit number calculation module includes 16 groups of 9-bit subtractors, and the second mantissa right shift bit number calculation module includes 16 groups of 6-bit subtractors; The first mixed number system mantissa multiplication module includes 16 groups of 12-bit by 12-bit signed number multipliers, and the second mixed number system mantissa multiplication module includes 16 groups of 5-bit by 5-bit signed number multipliers.

[0009] According to a tensor core operation component provided by the present invention, the first exponential addition module is used for: In a case where the preset floating-point number system is the 16-bit floating-point number system, adding an 8-bit exponent in the first floating-point number element in the first preset bit format and an 8-bit exponent in the second floating-point number element in the first preset bit format, and subtracting a corresponding floating-point number exponent offset value from the addition result to obtain a first exponent addition calculation result; In a case where the preset floating-point number system is the 8-bit floating-point number system, adding a 5-bit exponent in the first floating-point number element in the second preset bit format and a 5-bit exponent in the second floating-point number element in the second preset bit format, and subtracting a corresponding floating-point number exponent offset value from the addition result to obtain a second exponent addition calculation result; The second exponent addition module is used for: When the preset floating point number system is the 16-bit floating point number system, performing operand isolation and register clock gating; When the preset floating-point number system is the 8-bit floating-point number system, the 5-bit exponent in the first floating-point number element in the second preset bit format and the 5-bit exponent in the second floating-point number element in the second preset bit format are added, and the corresponding floating-point number exponent offset value in the addition result is subtracted to obtain a third exponent addition calculation result.

[0010] According to a tensor core computing component provided by the present invention, when the preset floating point number system is the 16-bit floating point number system, the first maximum exponent obtaining module outputs a first maximum exponent, wherein the first maximum exponent is a maximum bit exponent in the first exponent addition calculation result output by the first exponent addition module; When the preset floating-point number system is the 8-bit floating-point number system, the first maximum exponent obtaining module or the second maximum exponent obtaining module outputs a second maximum exponent, wherein the second maximum exponent is the larger maximum exponent of the first branch maximum exponent and the second branch maximum exponent, the first branch maximum exponent is the maximum bit exponent in the second exponent addition calculation result output by the first maximum exponent obtaining module, and the second branch maximum exponent is the maximum bit exponent in the third exponent addition calculation result output by the second maximum exponent obtaining module.

[0011] According to a tensor core computing component provided by the present invention, when the preset floating point number system is the 16-bit floating point number system, the first mantissa right shift bit number calculation module is used to output a first exponent difference, and the output result of the second mantissa right shift bit number calculation module is 0, wherein the first exponent difference is the exponent difference between the first exponent addition calculation result and the first maximum exponent; When the preset floating point number system is the 8-bit floating point number system, the first mantissa right shift bit number calculation module is used to output a second exponent difference, and the second mantissa right shift bit number calculation module is used to output a third exponent difference, wherein the second exponent difference is the exponent difference between the second exponent addition calculation result and the second maximum exponent; and the third exponent difference is the exponent difference between the third exponent addition calculation result and the second maximum exponent.

[0012] According to a tensor core operation component provided by the present invention, the first mixed number system mantissa multiplication module is used for: In a case where the preset floating-point number system is the 16-bit floating-point number system, multiplying a 12-bit signed number mantissa in the first floating-point number element in the first preset bit format and a 12-bit signed number mantissa in the second floating-point number element in the first preset bit format to obtain a first mantissa multiplication calculation result; When the preset floating point number system is the 8-bit floating point number system, multiplying a 5-bit signed number mantissa in the first floating point number element in the second preset bit format and a 5-bit signed number mantissa in the second floating point number element in the second preset bit format to obtain a second mantissa multiplication calculation result; The second mixed number system mantissa multiplication module is used for: When the preset floating point number system is the 16-bit floating point number system, performing operand isolation and register clock gating; When the preset floating-point number system is the 8-bit floating-point number system, the 5-bit signed mantissa in the first floating-point number element in the second preset bit format and the 5-bit signed mantissa in the second floating-point number element in the second preset bit format are multiplied to obtain a third mantissa multiplication calculation result.

[0013] According to a tensor core computing component provided by the present invention, the mantissa alignment accumulation module is used for: In a case where the preset floating-point number system is the 16-bit floating-point number system, according to the first exponent difference, the mantissa bits in the multiple groups of the first mantissa multiplication calculation results are arithmetically shifted right, and the first mantissa multiplication calculation results after the multiple groups of the mantissa bits are arithmetically shifted right are aligned starting from the high bit, and the alignment results are accumulated to obtain a first mantissa alignment accumulation result; When the preset floating-point number system is the 8-bit floating-point number system, the mantissa bits in the multiple groups of second mantissa multiplication calculation results are arithmetically shifted right according to the second exponent difference; the mantissa bits in the multiple groups of third mantissa multiplication calculation results are arithmetically shifted right according to the third exponent difference; and the second mantissa multiplication calculation results after the multiple groups of mantissa bits are arithmetically shifted right and the third mantissa multiplication calculation results after the multiple groups of mantissa bits are arithmetically shifted right are aligned starting from the high bit, and the aligned results are accumulated to obtain the second mantissa aligned accumulated result.

[0014] According to a tensor core computing component provided by the present invention, the normalization module is used for: Determine a target sign bit value according to the highest sign bit in the first mantissa alignment accumulation result or the second mantissa alignment accumulation result; Convert the mantissa in the first mantissa alignment accumulation result or the second mantissa alignment accumulation result into an unsigned number, and obtain the index value corresponding to the position of the first 1 from the unsigned number in the direction from the highest bit to the lowest bit; Based on the index value, shift the unsigned number to the left to obtain a target mantissa; constructing a target index bit according to the first maximum index and the index value; or constructing the target index bit according to the second maximum index and the index value; Obtaining the target bit format floating point number according to the target sign bit value, the target exponent bit and the target mantissa bit; The target bit format floating point number is a 20-bit bit format floating point number.

[0015] According to a tensor core computing component provided by the present invention, the compression layer includes 256 identical floating-point number compression units, and the floating-point number compression units are used to: When the target floating-point number system is the FP16 number system, based on the IEEE754 standard, compressing the target bit format floating-point number into the floating-point operation result of the FP16 number system; When the target floating-point number system is a BF16 number system, based on the IEEE754 standard, compressing the target bit format floating-point number into the floating-point operation result of the BF16 number system; Among them, during the compression process, if the target exponent bit in the target bit format floating-point number is greater than or equal to the preset floating-point bit, the floating-point number compression unit outputs an infinite number; if the target exponent bit in the target bit format floating-point number is less than or equal to 0, the floating-point number compression unit outputs a non-normal number; if the target bit format floating-point number is a non-number, the floating-point number compression unit outputs a silent non-number, wherein the preset floating-point bit is the maximum value corresponding to the exponent bit of the target floating-point number system.

[0016] The tensor core computing component provided by the present invention constructs a tensor core computing component architecture with a decompression layer, a computing layer and a compression layer. The decompression layer uniformly decompresses 16-bit floating-point numbers or 8-bit floating-point numbers, so that the decompression layer specifically decompresses data, avoiding resource waste caused by repeated data decompression and saving chip area and power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic diagram of the structure of the tensor core computing component provided by the present invention; Figure 2 A schematic diagram of the architecture of a computing unit provided by the present invention; Figure 3 A schematic diagram of mantissa alignment accumulation based on the mantissa alignment accumulation module provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] As a general-purpose artificial intelligence chip, the graphics processing unit (GPU) is a general-purpose architecture that supports image processing, scientific computing, artificial intelligence model training and reasoning. The existing GPUs mainly include three series: V100, A100 and H100. However, the three dimensions of matrix multiplication and addition supported by the GPU tensor cores of these series are not symmetrical, and all contain the dimension "4". Therefore, the existing GPU tensor cores are relatively "flat". It may be to make the GPU more universal. Considering the application of tensor cores in image computing scenarios, AI computing cards have adopted the dimensional design of graphics cards containing the dimension "4". In addition, since the GPU is a general-purpose artificial intelligence chip, its architectural design needs to support multiple applications. Therefore, in order to avoid the waste of GPU performance due to the small dimension of matrix multiplication in some application scenarios and the large dimension of matrix multiplication and addition supported by the tensor core, the existing GPU tensor core cannot support a large dimension of matrix multiplication and addition.

[0021] In the process of training and reasoning large language models, in addition to some attention algorithms, more than 90% of the chip computing power needs to be used to support large-dimensional (dimensions exceeding 1024) matrix multiplication calculations. Therefore, when the GPU is used for the calculation of large language models, the calculation of tensor cores with such a small dimension as "4" will bring about problems such as complex data arrangement and inconvenient software calls. It is known that the larger the cumulative dimension of the tensor core, the more simplified operations can be performed in the intermediate accumulation process of floating-point operations, and thus the larger the computing power area ratio and computing power consumption ratio, and the better the performance of the tensor core. For general-purpose artificial intelligence chips such as GPUs, their architectural design needs to take into account supporting multiple applications, so that the tensor core dimensions of the existing GPU designs are not large, resulting in limited simplified operations in the accumulation process, and relatively low computing power area ratio and computing power consumption.

[0022] Ascend 910 is oriented to AI application fields, and is mainly suitable for application scenarios such as AI training and autonomous driving. The main computing power of Ascend 910 is provided by a Cube Core that supports 16×16×16 (in FP16 number system) and 16×32×16 (in INT8 number system). Cube Core does not support the BF16 number system commonly used in large language models, and the activation matrix and weight matrix supported by Cube Core are both matrix operation methods in the INT8 number system, which also does not match the operations in large language models.

[0023] The tensor cores inside the Tensor Processing Unit (TPU) chip use a systolic array to arrange data flow. The advantage of the systolic array solution is that the data flow is simple and the structure is simple, so the front-end design of the TPU and the layout and wiring of the back-end are relatively simple and easy to do. In addition, all operations within the systolic array group are multiplication and addition operations, and one data is multiplied and added each time, so the calculation can be performed in full accordance with the IEEE 754 specification, with high calculation accuracy. However, in the systolic array solution, the data needs to be moved from one end of the systolic array to the other end in sequence to complete the calculation, resulting in a large delay, and the data in the systolic array is added sequentially, and the data path cannot be efficiently used to simplify the calculation, that is, parallel calculation is not possible. Moreover, the existing TPU has poor compatibility with some mainstream model frameworks in the field of artificial intelligence, and developers need to spend extra costs to adapt to the architecture of the TPU when using the TPU.

[0024] Although the existing TPU systolic array solution has high accuracy, it has a large latency, and the data in the systolic array is added sequentially, with an accumulated dimension of only 1, so the computing power to area ratio and computing power to power ratio are very low. TPU is not suitable for the current large language model calculations that require high computing power and certain scenarios that are insensitive to errors. In addition, the existing mainstream large model algorithms and firmware are all developed by GPUs. Using the systolic array tensor core TPU, which is completely different from the GPU tensor core, will lead to complex software migration, complex firmware development, and high learning costs.

[0025] In view of the problems existing in the above-mentioned prior art, the large language model architecture based on the Transformer Decoder-Only technical route has become mature, and dedicated chips for this architecture are urgently needed to serve the training and reasoning of large language models with higher efficiency and lower cost. The present invention is a tensor core computing component customized for large language models, and the supported number system fully matches the number system required by large language models. The supported input floating-point number systems include FP16 (sign-exponent-mantissa is 1-5-10) number system, BF16 (1-7-8) number system, FP8_e5m2 (1-5-2) number system and FP8_e4m3 (1-4-3) number system that does not comply with IEEE 754. The supported output floating-point number systems include FP16 number system and BF16 number system. In addition, the tensor core dimensions of 16×16×16 in the FP16 number system and 16×16×32 in the FP8 number system adopted by the present invention meet the large-dimensional matrix multiplication requirements of large language models, which can not only reduce the difficulty of data arrangement and software calling, but also provide a larger optimization space for the intermediate accumulation process due to the larger accumulation dimension, thereby improving the computing power-to-area ratio and computing power-to-power ratio of the chip.

[0026] Compared with the existing floating-point number decompression and multiplication and addition scheme in the computing unit, the present invention saves 15 / 16 of the decompression area and corresponding power consumption by designing a tensor core computing component architecture of decompression layer-computation layer-compression layer to avoid repeated decompression of data. In addition, the clear layered architecture is also beneficial for back-end design and development personnel to optimize timing and devices and improve chip computing power utilization. In the computing layer, the present invention realizes native support for FP16 number system, BF16 number system, FP8_e5m2 number system and FP8_e4m3 (1-4-3) number system mixed number system input premise by multiplexing modules such as exponential addition, maximum exponent, right shift bit number calculation of mantissa, and mixed number system mantissa multiplication, saving chip area and power consumption. In addition, in order to solve the problem of fuzzy data form in the intermediate process between the decompression layer, the calculation layer, and the compression layer, the present invention also designs a 20-bit custom floating-point number format and corresponding processing rules. The introduction of this format improves the data processing efficiency in the tensor core computing component and improves the performance of the tensor core.

[0027] Figure 1 A schematic diagram of the structure of the tensor core computing unit provided by the present invention, such as Figure 1 As shown, the present invention provides a tensor core computing component, including a decompression layer 101, a computing layer 102 and a compression layer 103, wherein: The decompression layer 101 is used to decompress the floating-point elements in a first matrix of a preset floating-point number system to obtain the first floating-point elements in a preset bit format, or to decompress the floating-point elements in a second matrix of the preset floating-point number system to obtain the second floating-point elements in the preset bit format, wherein the first matrix and the second matrix are two matrices in a matrix multiplication operation; the preset floating-point number system includes a 16-bit floating-point number system and an 8-bit floating-point number system.

[0028] In the present invention, the floating point number systems supported for input include 16-bit floating point number systems and 8-bit floating point number systems, specifically: FP16 (sign-exponent-mantissa is 1-5-10) number system, BF16 (1-7-8) number system, FP8_e5m2 (1-5-2) number system that comply with IEEE 754, and FP8_e4m3 (1-4-3) number system that does not comply with IEEE 754.

[0029] For the tensor core matrix multiplication operation implemented by the present invention, if the matrix multiplication is implemented in a 16-bit floating point number system (FP16 number system or BF16 number system), its mathematical expression is: Let A, B and C all be 16×16 floating point number matrices, C is the product of matrix A and matrix B, denoted as C=AB, where the elements of the i-th row and j-th column of matrix C can be expressed as: ; If the matrix multiplication is implemented in 8-bit floating point system (FP8_e4m3 or FP8_e5m2), the mathematical expression is: Let A be a 16×32 floating point matrix, B be a 32×16 floating point matrix, and C be a 16×16 floating point matrix. C is the product of matrix A and matrix B, denoted as C=AB, where the elements of the i-th row and j-th column of matrix C can be expressed as: ; In the present invention, the decompression layer 101 is composed of 512 identical decompression units, which can decompress 512 16-bit floating-point numbers (i.e., the preset floating-point number system is a 16-bit floating-point number system) or 1024 8-bit floating-point numbers (i.e., the preset floating-point number system is an 8-bit floating-point number system) according to the floating-point number format into a form that is easy to process by the floating-point number calculation unit, wherein the input of the decompression unit is 16 bits and the output is 20 bits.

[0030] When the input is a 16-bit floating point number, each decompression unit can decompress the 16-bit floating point number elements of one matrix A (i.e., the first matrix) or matrix B (i.e., the second matrix), thereby obtaining the first floating point number element or the second floating point number element in a preset bit format. When the input is an 8-bit floating point number, each decompression unit can simultaneously decompress the 8-bit floating point number elements of two matrices A or two matrices B, thereby obtaining the first floating point number element or the second floating point number element in a preset bit format.

[0031] The computing layer 102 is used to perform multiplication and addition operations on all the first floating-point elements corresponding to the first matrix and all the second floating-point elements corresponding to the second matrix to obtain floating-point numbers in a target bit format.

[0032] In the present invention, the computing layer 102 is composed of 256 identical computing units, each computing unit performs multiplication operations on 16 groups of floating-point numbers respectively, and then adds the 16 groups of multiplication results to obtain a floating-point number, thereby completing a group of floating-point multiplication and addition operations. When calculating 16-bit or 8-bit floating-point multiplication, the input of the computing unit is 16 decompressed 20-bit custom format (i.e., preset bit format) floating-point numbers as elements of matrix A, and 16 decompressed 20-bit custom format floating-point numbers as elements of matrix B, and the computing unit outputs a custom form floating-point number (i.e., target bit format floating-point number) containing a sign bit, an exponent bit, and a mantissa bit.

[0033] In the calculation layer 102, the floating point calculation units are recorded as calc(i, j), i, j∈[0, 15]. The input connection relationship between the decompression layer 101 and the calculation layer 102 is: ; The decompression units are recorded as unpack(i, j) in sequence.

[0034] The compression layer 103 is used to receive the target bit format floating point number, and perform compression conversion processing on the target bit format floating point number to obtain a floating point operation result of the target floating point number system.

[0035] In the present invention, the compression layer 103 is composed of 256 identical floating-point compression units, each of which corresponds to a floating-point calculation unit. The floating-point compression unit reversely converts (i.e., compresses and converts) the 20-bit custom-format floating-point number output by the corresponding calculation unit into a 16-bit floating-point number in the FP16 number system or the BF16 number system (i.e., the floating-point operation result of the target floating-point number system) in accordance with IEEE 754.

[0036] On the basis of the above embodiment, the decompression layer includes 512 identical decompression units; the preset bit format is a 20-bit format with bits from the 0th to the 19th, including a first preset bit format and a second preset bit format, wherein: In the case where the preset floating-point number system is the 16-bit floating-point number system, the decompression unit is used to decompress a floating-point number element in the first matrix to obtain the first floating-point number element in the first preset bit format; or to decompress a floating-point number element in the second matrix to obtain the second floating-point number element in the first preset bit format; Wherein, the 19th to 12th bits in the first preset bit format are 8-bit exponents, and the 11th to 0th bits in the first preset bit format are 12-bit signed number mantissas; In the case where the preset floating-point number system is the 8-bit floating-point number system, the decompression unit is used to decompress the floating-point number elements in the two first matrices to obtain the first floating-point number elements in the second preset bit format; or to decompress the floating-point number elements in the two second matrices to obtain the second floating-point number elements in the second preset bit format; Among them, the 19th to 15th bits in the second preset bit format are 5-bit exponent bits, the 14th to 10th bits in the second preset bit format are 5-bit signed number mantissa bits, the 9th to 5th bits in the second preset bit format are 5-bit exponent bits, and the 4th to 0th bits in the second preset bit format are 5-bit signed number mantissa bits.

[0037] In the present invention, multiple decompression units are sequentially recorded as unpack (i, j), where i, j ∈ [0, 15]. When the floating-point matrix of the input tensor core computing component is a 16-bit floating-point number system of the FP16 number system or the BF16 number system, the decompression unit outputs a 20-bit custom format floating-point number (i.e., the first preset bit format), recorded as unpack_out [19: 0], where unpack_out [19: 12] (i.e., the 19th to 12th bits in the first preset bit format) is an 8-bit exponent, corresponding to the floating-point exponent bits of the original 16-bit floating-point number, and unpack_out [11: 0] (i.e., the 11th to 0th bits in the first preset bit format) is the original floating-point number mantissa, after adding hidden bits according to the IEEE 754 regulations, converted into a 12-bit signed number mantissa represented by the complement according to the sign of the sign bit.

[0038] When the floating-point matrix input to the tensor core computing component is two 8-bit floating-point numbers in the FP8_e4m3 system or the FP8_e5m2 system, the decompression unit outputs a 20-bit custom-format floating-point number (i.e., the second preset bit format), recorded as unpack_out[19:0], where unpack_out[19:15] (i.e., the 19th to 15th bits in the second preset bit format) is a 5-bit exponent, i.e., the 8-bit floating-point exponent bits in the original first matrix (i.e., the first of the two first matrices or the two second matrices), and unpack_out[14:10] is the 8-bit floating-point number in the first matrix (i.e., the first of the two first matrices or the two second matrices). (i.e., the 14th to 10th bits in the second preset bit format) are the floating-point mantissa bits in the original first matrix (i.e., the first of the two first matrices or the two second matrices) after adding hidden bits according to the IEEE754 regulations, and converted into a 5-bit signed mantissa represented by the complement according to the sign bit mark; similarly, unpack_out[9:5] (i.e., the 9th to 5th bits in the second preset bit format) are the 5-bit exponent, i.e., the 8-bit floating-point exponent bits in the original second matrix (i.e., the second of the two first matrices or the two second matrices), and unpack_out[4:0] (i.e., the 4th to 0th bits in the second preset bit format) are the floating-point mantissa bits in the original second matrix (i.e., the second of the two second matrices or the two second matrices) after adding hidden bits according to the IEEE 754 regulations, and converted into a 5-bit signed mantissa represented by the complement according to the sign bit mark.

[0039] The decompression layer in the present invention uniformly decompresses 256 16-bit floating-point numbers or 512 8-bit floating-point numbers. For existing tensor core computing units, the data needs to be decompressed before calculation, so each data will be decompressed 16 times in the computing unit, and repeated decompression of data causes a waste of resources. Therefore, the present invention constructs a decompression layer to specifically decompress data, and a data is only decompressed once, which can save 15 / 16 of the chip area and corresponding power consumption of the decompression part. At the same time, a clear layered architecture also means that it is easier to identify the calculation process, which is beneficial for back-end design developers to optimize timing and devices.

[0040] The tensor core computing component provided by the present invention constructs a tensor core computing component architecture with a decompression layer, a computing layer and a compression layer. The decompression layer uniformly decompresses 16-bit floating-point numbers or 8-bit floating-point numbers, so that the decompression layer specifically decompresses data, avoiding resource waste caused by repeated data decompression and saving chip area and power consumption.

[0041] On the basis of the above embodiment, the computing layer is composed of 256 identical computing units, and the computing units include a first exponent addition module, a second exponent addition module, a first maximum exponent obtaining module, a second maximum exponent obtaining module, a first mantissa right shift bit number calculation module, a second mantissa right shift bit number calculation module, a first mixed number system mantissa multiplication module, a second mixed number system mantissa multiplication module, a mantissa alignment accumulation module and a normalization module, wherein: The first exponential addition module includes 16 groups of 8-bit adders, and the second exponential addition module includes 16 groups of 5-bit adders; The first mantissa right shift bit number calculation module includes 16 groups of 9-bit subtractors, and the second mantissa right shift bit number calculation module includes 16 groups of 6-bit subtractors; The first mixed number system mantissa multiplication module includes 16 groups of 12-bit by 12-bit signed number multipliers, and the second mixed number system mantissa multiplication module includes 16 groups of 5-bit by 5-bit signed number multipliers.

[0042] Figure 2 A schematic diagram of the architecture of the computing unit provided by the present invention, such as Figure 2 As shown, in the present invention, each computing unit includes 2 exponent addition modules, 2 maximum exponent obtaining modules, 2 mantissa right shift bit number calculation modules, 2 mixed number system mantissa multiplication modules, 1 mantissa alignment accumulation module and 1 normalization module.

[0043] Based on the above embodiment, the first exponential addition module is used for: In a case where the preset floating-point number system is the 16-bit floating-point number system, adding an 8-bit exponent in the first floating-point number element in the first preset bit format and an 8-bit exponent in the second floating-point number element in the first preset bit format, and subtracting a corresponding floating-point number exponent offset value from the addition result to obtain a first exponent addition calculation result; In a case where the preset floating-point number system is the 8-bit floating-point number system, adding a 5-bit exponent in the first floating-point number element in the second preset bit format and a 5-bit exponent in the second floating-point number element in the second preset bit format, and subtracting a corresponding floating-point number exponent offset value from the addition result to obtain a second exponent addition calculation result; The second exponent addition module is used for: When the preset floating point number system is the 16-bit floating point number system, performing operand isolation and register clock gating; When the preset floating-point number system is the 8-bit floating-point number system, the 5-bit exponent in the first floating-point number element in the second preset bit format and the 5-bit exponent in the second floating-point number element in the second preset bit format are added, and the corresponding floating-point number exponent offset value in the addition result is subtracted to obtain a third exponent addition calculation result.

[0044] In the present invention, the first exponential addition module includes 16 groups of 8-bit adders, and the second exponential addition module includes 16 groups of 5-bit adders.

[0045] When the floating-point matrix input to the tensor core operation unit is a 16-bit floating-point number system, the first exponential addition module can complete the 8-bit exponent addition of 16 groups of 16-bit floating-point numbers, and subtract the corresponding floating-point exponent offset value to obtain the first exponential addition calculation result. At this time, the second exponential addition module performs operand isolation and register clock gating to save power consumption.

[0046] When the floating-point matrix of the input tensor core operation component is an 8-bit floating-point number, the 8-bit adder of the first exponential addition module needs to be reused to complete the addition of 5-bit exponents in 16 groups of 8-bit floating-point numbers, and subtract the corresponding floating-point exponent offset value to obtain the second exponential addition calculation result. The second exponential addition module completes the addition of 5-bit exponents in 16 groups of 8-bit floating-point numbers, and subtracts the corresponding floating-point exponent offset value to obtain the third exponential addition calculation result.

[0047] On the basis of the above embodiment, when the preset floating point number system is the 16-bit floating point number system, the first maximum exponent obtaining module outputs a first maximum exponent, wherein the first maximum exponent is a maximum bit exponent in the first exponent addition calculation result output by the first exponent addition module; When the preset floating-point number system is the 8-bit floating-point number system, the first maximum exponent obtaining module or the second maximum exponent obtaining module outputs a second maximum exponent, wherein the second maximum exponent is the larger maximum exponent of the first branch maximum exponent and the second branch maximum exponent, the first branch maximum exponent is the maximum bit exponent in the second exponent addition calculation result output by the first maximum exponent obtaining module, and the second branch maximum exponent is the maximum bit exponent in the third exponent addition calculation result output by the second maximum exponent obtaining module.

[0048] In the present invention, when the preset floating point number system is the 16-bit floating point number system, the first maximum exponent obtaining module obtains the maximum value of the 16 groups of 8-bit exponent addition results calculated by the first exponent addition module, that is, the first maximum exponent.

[0049] When the preset floating-point number system is the 8-bit floating-point number system, the first maximum exponent obtaining module obtains the maximum value from the 16 groups of 8-bit / 5-bit addition results calculated by the first exponent addition module, and obtains the maximum value from the 16 groups of 5-bit exponent addition results calculated by the second exponent addition module, and then, the larger of the two maximum values ​​(at this time, the two maximum values ​​can be defined as corresponding branch maximum values) is determined as the second maximum exponent, and the corresponding maximum exponent obtaining module outputs this maximum value.

[0050] Specifically, when the floating-point number matrix input to the tensor core operation component is a 16-bit floating-point number, only the first maximum exponent obtaining module outputs the first maximum exponent; when the floating-point number matrix input to the tensor core operation component is an 8-bit floating-point number, the first maximum exponent obtaining module and the second maximum exponent obtaining module calculate the corresponding branch maximum values, and then compare the two branch maximum values, and the maximum exponent obtaining module corresponding to the larger branch maximum value outputs the second maximum exponent. In the present invention, the result output by the maximum exponent obtaining module is recorded as the maximum exponent exp_max.

[0051] On the basis of the above embodiment, when the preset floating point number system is the 16-bit floating point number system, the first mantissa right shift bit number calculation module is used to output a first exponent difference, and the output result of the second mantissa right shift bit number calculation module is 0, wherein the first exponent difference is the exponent difference between the first exponent addition calculation result and the first maximum exponent; When the preset floating point number system is the 8-bit floating point number system, the first mantissa right shift bit number calculation module is used to output a second exponent difference, and the second mantissa right shift bit number calculation module is used to output a third exponent difference, wherein the second exponent difference is the exponent difference between the second exponent addition calculation result and the second maximum exponent; and the third exponent difference is the exponent difference between the third exponent addition calculation result and the second maximum exponent.

[0052] In the present invention, when the preset floating point number system is a 16-bit floating point number system, the first mantissa right shift bit number calculation module uses 16 groups of 9-bit subtractor units to output the difference between the 16 groups of 9-bit exponents and the first maximum exponent. At this time, the output result of the 16 groups of 6-bit subtractor units of the second mantissa right shift bit number calculation module is 0.

[0053] When the preset floating-point number system is an 8-bit floating-point number system, 16 groups of 9-bit subtractors of the first mantissa right-shift bit number calculation module are reused to output the differences between 16 groups of 6-bit exponents and the second maximum exponent; at the same time, 16 groups of 6-bit subtractors of the second mantissa right-shift bit number calculation module are used to output the differences between 16 groups of 6-bit exponents and the second maximum exponent, thereby outputting a total of 32 groups of exponent differences.

[0054] Based on the above embodiment, the first mixed number system mantissa multiplication module is used for: In a case where the preset floating-point number system is the 16-bit floating-point number system, multiplying a 12-bit signed number mantissa in the first floating-point number element in the first preset bit format and a 12-bit signed number mantissa in the second floating-point number element in the first preset bit format to obtain a first mantissa multiplication calculation result; When the preset floating point number system is the 8-bit floating point number system, multiplying a 5-bit signed number mantissa in the first floating point number element in the second preset bit format and a 5-bit signed number mantissa in the second floating point number element in the second preset bit format to obtain a second mantissa multiplication calculation result; The second mixed number system mantissa multiplication module is used for: When the preset floating point number system is the 16-bit floating point number system, performing operand isolation and register clock gating; When the preset floating-point number system is the 8-bit floating-point number system, the 5-bit signed mantissa in the first floating-point number element in the second preset bit format and the 5-bit signed mantissa in the second floating-point number element in the second preset bit format are multiplied to obtain a third mantissa multiplication calculation result.

[0055] In the present invention, the first mixed number system mantissa multiplication module includes 16 groups of 12-bit×12-bit signed number multipliers, outputting a 23-bit result, and the second mixed number system mantissa multiplication module includes 16 groups of 5-bit×5-bit signed number multipliers, outputting a 9-bit result.

[0056] When the floating point number system is preset to be a 16-bit floating point number system, the present invention only uses the first mixed number system mantissa multiplication module to perform 16 groups of 12-bit × 12-bit mantissa multiplications to obtain a 23-bit signed number mantissa multiplication result, that is, the first mantissa multiplication calculation result. At this time, the second mixed number system mantissa multiplication module performs operand isolation and register clock gating to save power consumption.

[0057] When the preset floating-point number system is an 8-bit floating-point number system, 16 groups of 12-bit × 12-bit signed number multipliers of the first mixed number system mantissa multiplication module are reused to output a 23-bit signed number mantissa multiplication result, i.e., the second mantissa multiplication calculation result; at the same time, 16 groups of 5-bit × 5-bit signed number multipliers of the second mixed number system mantissa multiplication module are used to output a 9-bit signed number mantissa multiplication result, i.e., the third mantissa multiplication calculation result, thereby outputting a total of 32 groups of mantissa multiplication results.

[0058] In the present invention, the exponent addition module, the maximum exponent obtaining module and the mantissa right shift bit number calculation module in the calculation unit only involve exponent data, and the mixed number system mantissa multiplication module only involves mantissa data. This is because the large bit width multiplication operation time is long, so the above three exponent calculation related modules are designed to be calculated in parallel with the mantissa multiplication module, thereby realizing full data flow, saving registers, and reducing chip area and power consumption.

[0059] Based on the above embodiment, the mantissa alignment accumulation module is used for: In a case where the preset floating-point number system is the 16-bit floating-point number system, according to the first exponent difference, the mantissa bits in the multiple groups of the first mantissa multiplication calculation results are arithmetically shifted right, and the first mantissa multiplication calculation results after the multiple groups of the mantissa bits are arithmetically shifted right are aligned starting from the high bit, and the alignment results are accumulated to obtain a first mantissa alignment accumulation result; When the preset floating-point number system is the 8-bit floating-point number system, the mantissa bits in the multiple groups of second mantissa multiplication calculation results are arithmetically shifted right according to the second exponent difference; the mantissa bits in the multiple groups of third mantissa multiplication calculation results are arithmetically shifted right according to the third exponent difference; and the second mantissa multiplication calculation results after the multiple groups of mantissa bits are arithmetically shifted right and the third mantissa multiplication calculation results after the multiple groups of mantissa bits are arithmetically shifted right are aligned starting from the high bit, and the aligned results are accumulated to obtain the second mantissa aligned accumulated result.

[0060] In floating-point multiplication, due to the difference in exponents, the mantissa after multiplication may need to be moved to the right to align the exponent of the result. When the floating-point number system is preset to be a 16-bit floating-point number system, the mantissa alignment accumulation module moves the mantissa bits to the right according to the first exponent difference obtained in the above embodiment. After completing the movement of the mantissa bits, since the floating-point numbers may have different decimal point positions when stored, the mantissa alignment accumulation module needs to align the mantissa parts of all floating-point numbers involved in the operation in order to perform accumulation, and obtain the first mantissa alignment accumulation result, and the output result can be recorded as mantissa[27:0].

[0061] When the floating-point number system is preset to an 8-bit floating-point number system, the mantissa alignment accumulation module can realize the right shift of 16 groups of 23-bit signed numbers and the right shift of 16 groups of 9-bit signed numbers, and accumulate the 16 groups of 23-bit signed numbers after the right shift and the 16 groups of 9-bit signed numbers after the right shift after aligning the high bits, outputting a 28-bit signed number result, and obtaining a second mantissa alignment accumulation result, and the output result is recorded as mantissa[27:0]. Figure 3 The schematic diagram of the mantissa alignment accumulation based on the mantissa alignment accumulation module provided by the present invention, and the data structure for realizing the mantissa alignment accumulation can be referred to Figure 3 shown.

[0062] Based on the above embodiment, the normalization module is used for: Determine a target sign bit value according to the highest sign bit in the first mantissa alignment accumulation result or the second mantissa alignment accumulation result; Convert the mantissa in the first mantissa alignment accumulation result or the second mantissa alignment accumulation result into an unsigned number, and obtain the index value corresponding to the position of the first 1 from the unsigned number in the direction from the highest bit to the lowest bit; Based on the index value, shift the unsigned number to the left to obtain a target mantissa; constructing a target index bit according to the first maximum index and the index value; or constructing the target index bit according to the second maximum index and the index value; Obtaining the target bit format floating point number according to the target sign bit value, the target exponent bit and the target mantissa bit; The target bit format floating point number is a 20-bit bit format floating point number.

[0063] In the present invention, the main function of the normalization module is to perform normalization processing according to the mantissa obtained by the mantissa alignment accumulation module and the maximum exponent obtained by the maximum exponent obtaining module, thereby outputting a 20-bit custom format floating point number, that is, a target bit format floating point number.

[0064] Specifically, the normalization module processes the mantissa as follows: obtain the highest sign bit of the mantissa [27:0], which is also the sign bit value of the final floating-point number output by the normalization module, which can be recorded as s (where the sign bit is 1, indicating that the floating-point number is negative, and the sign bit is 0, indicating that the floating-point number is positive); restore the mantissa in the first mantissa alignment accumulation result or the second mantissa alignment accumulation result to an unsigned number, recorded as mantissa_out [26:0]. Further, from the highest bit to the lowest bit of mantissa_out [26:0], find the position of the first 1, and set the nth bit to be the position of the first 1 (i.e., the index value); then, shift mantissa_out [26:0] left by n bits, recorded as mantissa_sft [26:0]. Therefore, starting from the highest bit, the target bit format floating point number of the 20-bit bit format floating point number finally output is self_def[19:0], which is then input to the floating point compression unit for compression, wherein the sign bit self_def

[19] =s, the exponent bit self_def[18:11]=exp_max-n+6, and the mantissa bit self_def[10:0]=mantissa_sft[26:17], wherein the 32 groups of mantissa accumulation results have a total of 7 digits before the decimal point, which is the standard form of 1 digit before the decimal point. Therefore, the mantissa decimal point is shifted left by 6 digits, and the exponent should be added by 6, that is, the number "6" in "exp_max-n+6". In the present invention, the method for obtaining the mantissa bit is not a simple truncation, but a rounding method (round to nearest even) toward the nearest even number is adopted.

[0065] Based on the above embodiment, the compression layer includes 256 identical floating point number compression units, and the floating point number compression units are used to: When the target floating-point number system is the FP16 number system, based on the IEEE754 standard, compressing the target bit format floating-point number into the floating-point operation result of the FP16 number system; When the target floating-point number system is a BF16 number system, based on the IEEE754 standard, compressing the target bit format floating-point number into the floating-point operation result of the BF16 number system; Among them, during the compression process, if the target exponent bit in the target bit format floating-point number is greater than or equal to the preset floating-point bit, the floating-point number compression unit outputs an infinite number; if the target exponent bit in the target bit format floating-point number is less than or equal to 0, the floating-point number compression unit outputs a non-normal number; if the target bit format floating-point number is a non-number, the floating-point number compression unit outputs a silent non-number, wherein the preset floating-point bit is the maximum value corresponding to the exponent bit of the target floating-point number system.

[0066] In the present invention, the 20-bit custom floating point number to be compressed (i.e., the target bit format floating point number) is self_def[19:0], where the sign bit is self_def

[19] =s, the exponent bit is self_def[18:11], and the unsigned mantissa is self_def[10:0]. If the final output number system is the FP16 number system, the floating point number compression result is FP16_out[15:0], FP16_out

[15] =s, FP16_out [14:10]=self_def[15:11], FP16_out [9:0]=self_def[9:0].

[0067] Specifically, during the compression process, when the final output number system is the FP16 number system, if the target exponent bit (i.e., exp_max–n+6) in the target bit format floating-point number is greater than or equal to the preset floating-point bit (0x1F), i.e., exp_max–n+6≥0x1F, then self_def[18:11]=0x1F, and the infinite number specified by IEEE 754 is output. The compression result is FP16_out

[15] =s, FP16_out[14:10]=5'h1F, FP16_out[9:0]=10'h0, where 0x1F is the maximum value that the exponent bit of the target floating-point number system can represent; if exp_max–n+6≤0, the subnormal specified by IEEE 754 is output; if there is a non-number (NAN) specified by IEEE 754, the output is quiet NAN, and the compression result is FP16_out

[15] =s, FP16_out[14:10]=5'h1F, FP16_out[9:0]=10'h201.

[0068] When the final output number system is BF16, the floating point number compression result is BF16_out[15:0], where BF16_out

[15] =s, BF16_out[14:7]=self_def[18:11], and BF16_out [6:0]=self_def[9:3]. If exp_max–n+6≥0xFF, self_def[18:11]=0xFF, and the infinite number specified by IEEE 754 is output, the compression result is BF16_out

[15] =s, BF16_out [14:7]=8'hFF, and BF16_out [6:0]=7'h3F; if exp_max–n+6<0, the subnormal number specified by IEEE 754 is output. If there is a non-number specified by IEEE754, the output is quiet NAN, and the compression result is BF16_out

[15] =s, BF16_out [14:7]=8'hFF, BF16_out [6:0]=7'h81.

[0069] The tensor core computing unit provided by the present invention supports the input of FP16, BF16, FP8_e5m2 (1-5-2) and FP8_e4m3 (1-4-3) number systems, and outputs FP16 and BF16 number systems. The number systems it supports fully match the number systems required by large language models. The tensor core multiplication and addition dimensions of 16×16×16 under the FP16 number system and 16×16×32 under the FP8 number system adopted by the present invention meet the matrix multiplication calculation dimension requirements of large language models. Through customized design, it can cover functional requirements without additional waste, thereby improving chip performance and reducing chip costs.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A tensor core computing component, characterized in that: It includes decompression layer, calculation layer and compression layer, where: The decompression layer is used to decompress the floating-point elements in the first matrix of the preset floating-point number system to obtain the first floating-point elements in the preset bit format, or to decompress the floating-point elements in the second matrix of the preset floating-point number system to obtain the second floating-point elements in the preset bit format, wherein the first matrix and the second matrix are two matrices in a matrix multiplication operation; the preset floating-point number system includes a 16-bit floating-point number system and an 8-bit floating-point number system; The computing layer is used to perform multiplication and addition operations on all the first floating-point number elements corresponding to the first matrix and all the second floating-point number elements corresponding to the second matrix to obtain a floating-point number in a target bit format; The compression layer is used to receive the target bit format floating point number, and perform compression conversion processing on the target bit format floating point number to obtain a floating point operation result of the target floating point number system.

2. The tensor core computing unit according to claim 1, characterized in that: The decompression layer includes 512 identical decompression units; the preset bit format is a 20-bit format with bits from the 0th to the 19th, including a first preset bit format and a second preset bit format, wherein: In the case where the preset floating-point number system is the 16-bit floating-point number system, the decompression unit is used to decompress a floating-point number element in the first matrix to obtain the first floating-point number element in the first preset bit format; or to decompress a floating-point number element in the second matrix to obtain the second floating-point number element in the first preset bit format; Wherein, the 19th to 12th bits in the first preset bit format are 8-bit exponents, and the 11th to 0th bits in the first preset bit format are 12-bit signed number mantissas; In the case where the preset floating-point number system is the 8-bit floating-point number system, the decompression unit is used to decompress the floating-point number elements in the two first matrices to obtain the first floating-point number elements in the second preset bit format; or to decompress the floating-point number elements in the two second matrices to obtain the second floating-point number elements in the second preset bit format; Among them, the 19th to 15th bits in the second preset bit format are 5-bit exponent bits, the 14th to 10th bits in the second preset bit format are 5-bit signed number mantissa bits, the 9th to 5th bits in the second preset bit format are 5-bit exponent bits, and the 4th to 0th bits in the second preset bit format are 5-bit signed number mantissa bits.

3. The tensor core computing unit according to claim 2, characterized in that: The computing layer is composed of 256 identical computing units, which include a first exponent addition module, a second exponent addition module, a first maximum exponent obtaining module, a second maximum exponent obtaining module, a first mantissa right shift bit number calculation module, a second mantissa right shift bit number calculation module, a first mixed number system mantissa multiplication module, a second mixed number system mantissa multiplication module, a mantissa alignment accumulation module and a normalization module, wherein: The first exponential addition module includes 16 groups of 8-bit adders, and the second exponential addition module includes 16 groups of 5-bit adders; The first mantissa right shift bit number calculation module includes 16 groups of 9-bit subtractors, and the second mantissa right shift bit number calculation module includes 16 groups of 6-bit subtractors; The first mixed number system mantissa multiplication module includes 16 groups of 12-bit by 12-bit signed number multipliers, and the second mixed number system mantissa multiplication module includes 16 groups of 5-bit by 5-bit signed number multipliers.

4. The tensor core computing unit according to claim 3, characterized in that: The first exponential addition module is used for: In a case where the preset floating-point number system is the 16-bit floating-point number system, adding an 8-bit exponent in the first floating-point number element in the first preset bit format and an 8-bit exponent in the second floating-point number element in the first preset bit format, and subtracting a corresponding floating-point number exponent offset value from the addition result to obtain a first exponent addition calculation result; In a case where the preset floating-point number system is the 8-bit floating-point number system, adding a 5-bit exponent in the first floating-point number element in the second preset bit format and a 5-bit exponent in the second floating-point number element in the second preset bit format, and subtracting a corresponding floating-point number exponent offset value from the addition result to obtain a second exponent addition calculation result; The second exponent addition module is used for: When the preset floating point number system is the 16-bit floating point number system, performing operand isolation and register clock gating; When the preset floating-point number system is the 8-bit floating-point number system, the 5-bit exponent in the first floating-point number element in the second preset bit format and the 5-bit exponent in the second floating-point number element in the second preset bit format are added, and the corresponding floating-point number exponent offset value in the addition result is subtracted to obtain a third exponent addition calculation result.

5. The tensor core computing unit according to claim 4, characterized in that: When the preset floating point number system is the 16-bit floating point number system, the first maximum exponent obtaining module outputs a first maximum exponent, wherein the first maximum exponent is a maximum bit exponent in the first exponent addition calculation result output by the first exponent addition module; When the preset floating-point number system is the 8-bit floating-point number system, the first maximum exponent obtaining module or the second maximum exponent obtaining module outputs a second maximum exponent, wherein the second maximum exponent is the larger maximum exponent of the first branch maximum exponent and the second branch maximum exponent, the first branch maximum exponent is the maximum bit exponent in the second exponent addition calculation result output by the first maximum exponent obtaining module, and the second branch maximum exponent is the maximum bit exponent in the third exponent addition calculation result output by the second maximum exponent obtaining module.

6. The tensor core computing unit according to claim 5, characterized in that: When the preset floating point number system is the 16-bit floating point number system, the first mantissa right shift bit number calculation module is used to output a first exponent difference, and the output result of the second mantissa right shift bit number calculation module is 0, wherein the first exponent difference is the exponent difference between the first exponent addition calculation result and the first maximum exponent; When the preset floating point number system is the 8-bit floating point number system, the first mantissa right shift bit number calculation module is used to output a second exponent difference, and the second mantissa right shift bit number calculation module is used to output a third exponent difference, wherein the second exponent difference is the exponent difference between the second exponent addition calculation result and the second maximum exponent; and the third exponent difference is the exponent difference between the third exponent addition calculation result and the second maximum exponent.

7. The tensor core computing unit according to claim 6, characterized in that: The first mixed number system mantissa multiplication module is used for: In a case where the preset floating-point number system is the 16-bit floating-point number system, multiplying a 12-bit signed number mantissa in the first floating-point number element in the first preset bit format and a 12-bit signed number mantissa in the second floating-point number element in the first preset bit format to obtain a first mantissa multiplication calculation result; When the preset floating point number system is the 8-bit floating point number system, multiplying a 5-bit signed number mantissa in the first floating point number element in the second preset bit format and a 5-bit signed number mantissa in the second floating point number element in the second preset bit format to obtain a second mantissa multiplication calculation result; The second mixed number system mantissa multiplication module is used for: When the preset floating point number system is the 16-bit floating point number system, performing operand isolation and register clock gating; When the preset floating-point number system is the 8-bit floating-point number system, the 5-bit signed mantissa in the first floating-point number element in the second preset bit format and the 5-bit signed mantissa in the second floating-point number element in the second preset bit format are multiplied to obtain a third mantissa multiplication calculation result.

8. The tensor core computing unit according to claim 7, characterized in that: The mantissa alignment accumulation module is used for: In a case where the preset floating-point number system is the 16-bit floating-point number system, according to the first exponent difference, the mantissa bits in the multiple groups of the first mantissa multiplication calculation results are arithmetically shifted right, and the first mantissa multiplication calculation results after the multiple groups of the mantissa bits are arithmetically shifted right are aligned starting from the high bit, and the alignment results are accumulated to obtain a first mantissa alignment accumulation result; In a case where the preset floating point number system is the 8-bit floating point number system, according to the second exponent difference, the mantissa bits in the multiplication calculation results of the plurality of groups of the second mantissas are arithmetically shifted right; according to the third exponent difference, the mantissa bits in the multiplication calculation results of the plurality of groups of the third mantissas are arithmetically shifted right; The second mantissa multiplication calculation result after the arithmetic right shift of the multiple groups of mantissa bits and the third mantissa multiplication calculation result after the arithmetic right shift of the multiple groups of mantissa bits are aligned starting from the high bit, and the alignment results are accumulated to obtain the second mantissa alignment accumulation result.

9. The tensor core computing unit according to claim 8, characterized in that: The normalization module is used to: Determine a target sign bit value according to the highest sign bit in the first mantissa alignment accumulation result or the second mantissa alignment accumulation result; Convert the mantissa in the first mantissa alignment accumulation result or the second mantissa alignment accumulation result into an unsigned number, and obtain the index value corresponding to the position of the first 1 from the unsigned number in the direction from the highest bit to the lowest bit; Based on the index value, shift the unsigned number to the left to obtain a target mantissa; constructing a target index bit according to the first maximum index and the index value; or, constructing the target index bit according to the second maximum index and the index value; Obtaining the target bit format floating point number according to the target sign bit value, the target exponent bit and the target mantissa bit; The target bit format floating point number is a 20-bit bit format floating point number.

10. The tensor core computing unit according to claim 9, characterized in that: The compression layer includes 256 identical floating point compression units, and the floating point compression units are used to: When the target floating-point number system is the FP16 number system, based on the IEEE754 standard, compressing the target bit format floating-point number into the floating-point operation result of the FP16 number system; When the target floating-point number system is a BF16 number system, based on the IEEE754 standard, compressing the target bit format floating-point number into the floating-point operation result of the BF16 number system; Among them, during the compression process, if the target exponent bit in the target bit format floating-point number is greater than or equal to the preset floating-point bit, the floating-point number compression unit outputs an infinite number; if the target exponent bit in the target bit format floating-point number is less than or equal to 0, the floating-point number compression unit outputs a non-normal number; if the target bit format floating-point number is a non-number, the floating-point number compression unit outputs a silent non-number, wherein the preset floating-point bit is the maximum value corresponding to the exponent bit of the target floating-point number system.

Images