Matrix multiplication and addition operation unit, method, system, chip, board and electronic equipment

Abstract

This invention proposes a matrix multiplication and addition operation unit, method, system, chip, board, and electronic device, relating to the field of data processing technology. The matrix multiplication and addition operation unit includes: a dot product module, used to map matrix input data according to data type, determine the mantissa and exponent data corresponding to the matrix input data based on the mapping result, and obtain the dot product calculation result; and an accumulation module, used to load intermediate data to be operated into an intermediate sum register group, read the corresponding intermediate data from the intermediate sum register group according to the matrix multiplication and addition operation identifier signal, accumulate it with the dot product calculation result, write the accumulation result into the intermediate sum register group, and output the accumulation result at the end of the operation. This invention can uniformly support operations on multiple data types, optimizing hardware deployment complexity and reducing power consumption while ensuring data processing efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more particularly to a matrix multiplication and addition operation unit, method, system, chip, board, and electronic device. Background Technology

[0002] With the rapid development of artificial intelligence and deep learning technologies, matrix multiply-accumulate (MAC) operations have become the core computational operations in neural network inference and training. To meet the computational accuracy and performance requirements of different application scenarios, modern AI chips need to support matrix operations in multiple data formats, such as floating-point formats like FP4, FP6, FP8, FP16, BF16, and TF32, as well as integer formats (e.g., INT8).

[0003] However, existing matrix multiplication and addition units suffer from numerous technical problems when handling multiple data types. For example, current units only support a limited number of floating-point and integer data formats, resulting in poor flexibility. On the same hardware architecture, processing multiple data types involves a long computational pipeline, leading to low efficiency and increased hardware design complexity. Logical resources cannot be shared, multiplier and adder resources are underutilized, and different data types often require independent computational units, significantly increasing hardware area and power consumption. Therefore, a technical solution that overcomes these shortcomings is urgently needed. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention proposes a matrix multiplication and addition operation unit, method, system, chip, board, and electronic device. This invention improves the matrix multiplication and addition operation unit to achieve unified support for multiple data types, and can improve data processing efficiency, optimize hardware deployment complexity, and reduce power consumption.

[0005] In a first aspect of the present invention, a matrix multiplication and addition unit is provided, comprising: The dot product module is used to map the matrix input data according to the data type, determine the mantissa data and exponent data corresponding to the matrix input data based on the mapping result, and obtain the dot product calculation result after dot product calculation. The accumulation module is used to load the intermediate data to be calculated into the intermediate sum register group, read the corresponding intermediate data from the intermediate sum register group according to the matrix multiplication and addition operation identifier signal, accumulate it with the dot product calculation result, write the accumulation result into the intermediate sum register group, and output the accumulation result when the operation ends.

[0006] In a second aspect of the present invention, a matrix multiplication and addition operation method is proposed, which is executed based on the aforementioned matrix multiplication and addition operation unit, including: The matrix input data is mapped according to the data type. Based on the mapping result, the mantissa data and exponent data corresponding to the matrix input data are determined. The dot product calculation result is obtained by dot product calculation. The intermediate data to be calculated is loaded into the intermediate sum register group. The corresponding intermediate data is read from the intermediate sum register group according to the matrix multiplication and addition operation identifier signal. The intermediate data is accumulated with the dot product calculation result and the accumulated result is written into the intermediate sum register group. The accumulated result is output when the operation ends.

[0007] In a third aspect of the present invention, a matrix multiplication and addition operation system is proposed, the system comprising a plurality of the aforementioned matrix multiplication and addition operation units.

[0008] In a fourth aspect of the present invention, a chip is proposed, comprising a plurality of computing cores, wherein some or all of the computing cores include the aforementioned matrix multiplication and addition unit.

[0009] In a fifth aspect of the present invention, a board is provided, including the aforementioned chip.

[0010] In a sixth aspect of the present invention, an electronic device is provided, including the aforementioned board.

[0011] The matrix multiplication and addition unit, method, system, chip, board, and electronic device proposed in this invention can support matrix multiplication and addition operations with multiple data types, efficiently reuse hardware resources, and significantly reduce hardware area and power consumption without sacrificing computational performance by combining multiple control mechanisms. The accumulation process combines the switching mechanism between intermediate and register groups, the loading storage system, and the precise synchronization of matrix multiplication and addition operation identification signals to ensure the accuracy and continuity of intermediate value accumulation during matrix multiplication and addition. It has high parallel scalability and can flexibly adapt to high-performance computing scenarios. In large-scale integration scenarios such as wafer-level chips, it can achieve a significant improvement in computing power density and energy efficiency ratio, meeting the high precision, high computing power, and low power consumption requirements of different scenarios. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of the architecture of a matrix multiplication and addition operation unit according to an embodiment of the present invention.

[0014] Figure 2 This is a schematic diagram of the architecture of a matrix multiplication and addition system according to an embodiment of the present invention.

[0015] Figure 3 This is a schematic diagram of the architecture of a matrix multiplication and addition operation unit according to a specific embodiment of the present invention.

[0016] Figure 4 This is a schematic diagram of the zero-value skipping architecture of a specific embodiment of the present invention.

[0017] Figure 5 This is a schematic flowchart of a matrix multiplication and addition operation method according to an embodiment of the present invention. Detailed Implementation

[0018] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0019] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments.

[0020] To address the problems of existing technologies failing to provide unified support for multiple floating-point and integer data formats, leading to increased hardware design complexity, low resource utilization of multipliers and adders, and significant increases in hardware area and power consumption due to the need for separate computing units for different data types, this invention proposes a matrix multiplication and addition operation unit, method, system, chip, board, and electronic device. This invention can achieve unified support for multiple data types, improve data processing efficiency, optimize hardware deployment complexity, and reduce power consumption.

[0021] Figure 1 This is a schematic diagram of the architecture of a matrix multiplication and addition operation unit according to an embodiment of the present invention. Figure 1 As shown, the matrix multiplication and addition unit includes: The dot product module 100 is used to map the matrix input data according to the data type, determine the mantissa data and exponent data corresponding to the matrix input data based on the mapping result, and obtain the dot product calculation result after dot product calculation. The accumulation module 200 is used to load the intermediate data to be calculated into the intermediate sum register group, read the corresponding intermediate data from the intermediate sum register group according to the matrix multiplication and addition operation identifier signal, accumulate it with the dot product calculation result, write the accumulation result into the intermediate sum register group, and output the accumulation result when the operation ends.

[0022] In one embodiment, the dot product module 100 supports input data types including at least: FP4 (S1E2M1), FP6E2 (S1E2M3), FP6E3 (S1E3M2), FP8E4 (S1E4M3), FP8E5 (S1E5M2), FP16 (S1E5M10), BF16 (S1E8M7), TF32 (S1E8M10), and INT8.

[0023] refer to Figure 2 This is a schematic diagram of the architecture of a matrix multiplication and addition system according to an embodiment of the present invention. Figure 2 As shown, the matrix multiplication-addition system includes multiple matrix multiplication-addition units (MACs), forming a computation array arranged in a regular pattern. This computation array loads intermediate data for computation through an input channel (Load input), inputs the first set of matrix input data through the left matrix input channel, and inputs the second set of matrix input data through the right matrix input channel. The computation result is output through an output channel (Store output). The input control signals include a mode signal and a PC signal. The mode signal controls the operation mode of each sub-module in the dot product module of the MAC, informing the computation array which operation is currently being performed and controlling the data type being processed, such as Zero Skipping, Clock Gating, etc. The PC signal (matrix multiplication-addition operation identifier signal) identifies the position of the currently computed MAC in the computation array, thereby determining which address of intermediate data should be loaded and which memory address the newly calculated result should be stored in, ensuring that data is correctly accumulated to the corresponding final location during block computation.

[0024] Based on the same inventive concept, the present invention also proposes a chip including multiple computing cores, wherein some or all of the computing cores include the aforementioned matrix multiplication and addition operation unit.

[0025] Specifically, this invention is applicable to wafer-level chips, wherein the wafer-level chip can be configured with multiple computing cores, some or all of which include the matrix multiplication and addition operation unit of this invention; the chip can be used in scenarios such as AI large model training and high-performance scientific computing, and by integrating a high-density MAC array at the wafer-level scale, the advantages of this invention, such as multi-data type fusion, low power consumption, and high parallelism, are realized.

[0026] Based on the same inventive concept, the present invention also proposes a board card, including the aforementioned chip.

[0027] Based on the same inventive concept, the present invention also proposes an electronic device, including the aforementioned board.

[0028] According to the above embodiments, the matrix multiplication and addition operation supports the fusion of multiple data types through mapping processing, which efficiently reuses hardware resources. At the same time, combined with multiple control mechanisms, it significantly reduces hardware area overhead and power consumption without sacrificing computing performance. The accumulation process combines the switching mechanism of intermediate and register groups, the loading storage system and the matrix multiplication and addition operation identification signal for precise synchronization, which ensures the accuracy and continuity of intermediate value accumulation during matrix multiplication and addition. It has high parallel expansion capability and can flexibly adapt to high-performance computing scenarios. In large-scale integration scenarios such as wafer-level chips, it can achieve a significant improvement in computing power density and energy efficiency ratio, meeting the high precision, high computing power and low power consumption requirements of different scenarios.

[0029] refer to Figure 3 This is a schematic diagram illustrating the architectural relationship of a matrix multiplication and addition operation unit according to a specific embodiment of the present invention. Figure 3 As shown, the dot product module 100 includes a data preprocessing submodule 110, an exponent comparison submodule 120, an exponent difference submodule 130, a mantissa operation submodule 140, a shift alignment submodule 150, a mantissa accumulation submodule 160, a normalization submodule 170, and a correction submodule 180. The accumulator module 200 includes: an intermediate register group 210, a data distribution submodule 220, a data selection submodule 230, and an accumulator 240.

[0030] The following provides a detailed description of each submodule.

[0031] Data preprocessing submodule 110: A unified data bit format is used to map the matrix input data. The input data consists of two sets of matrix input data (e.g., left matrix input, right matrix input), corresponding to a0, a1, ..., a1. n-1 b0, b1, ..., b n-1 Each group contains n data points.

[0032] The input data (a, b) has a bit width of 19 bits, and the mapping method for different data types is as follows: FP4 (S1E2M1): 4 channels, arranged in [15:12], [11:8], [7:4], [3:0] respectively; FP6 (S1E2M3 / S1E3M2): 2 channels, placed in [14:9] and [5:0] respectively; FP8 (S1E4M3 / S1E5M2): 2 channels, placed in [15:8] and [7:0] respectively; FP16(S1E5M10): Channel 1, placed in [15:0]; BF16(S1E8M7): Channel 1, placed in [15:0]; TF32 (S1E8M10): Channel 1, placed in [18:0]; INT8: Channel 2, placed in [15:9], [6:0].

[0033] Multiplexing input data can reduce the number of pipeline registers corresponding to input data channels.

[0034] With two sets of input data (vectors) of the same length , For example, with a length of L, each element in the vector is 19 bits. The mapping relationship of vector elements of different input data types is shown in Table 1.

[0035] Table 1

[0036] In the table, S, S0, and S1 represent the sign bit, E, E0, and E1 represent the exponent bit, and M, M0, and M1 represent the mantissa bit.

[0037] In one embodiment, the matrix input data is mapped to data bits according to a preset bit as an element; in order to reduce power consumption and unnecessary flips, invalid bits of each data type are filled with 0.

[0038] The number of supported channels is determined based on the data type, where each element contains valid data for the corresponding number of channels.

[0039] Referring to Table 1, FP4 supports four channels; FP6E2, FP6E3, FP8E4, FP8E5, and INT8 support two channels; FP16, BF16, and TF32 support one channel. That is, each 19-bit element contains four FP4 data, or two FP6E2 / FP6E3 / FP8E4 / FP8E5 / INT8 data, or one FP16 / BF16 / TF32 data.

[0040] Based on the mapping result, valid data is extracted. If the data type of the matrix input data is floating point, the hidden bits of the matrix input data are determined and the hidden bits are added to the highest bit of the mantissa to extract valid data. If the data type of the matrix input data is INT8, the valid data of the matrix input data is extracted directly. The valid data is rearranged according to the preset multiplier structure, and the mantissa data adapted to the multiplier bit width is output. For matrix input data of floating-point type, the exponent data is obtained by exponential summation operation; for matrix input data of data type INT8, the exponent summation operation is not performed.

[0041] Specifically, if the input data is of floating-point type (FP4, FP6E2, FP6E3, FP8E4, FP8E5, FP16, BF16, TF32), determine its hidden bit (H) and add it to the highest bit of the mantissa; if the input data is of INT8 type, directly extract the valid data.

[0042] The data is rearranged and output according to the multiplier structure of the mantissa operation submodule 140; the output mantissa is A. m44 Am 480 A m481 A m88 B m44 B m480 B m481 B m88 ; Where A and B are the two multipliers of the multiplier, m44 is a 4x4 multiplier, m480 is the first 4x8 multiplier, m481 is the second 4x8 multiplier, and m88 is an 8x8 multiplier.

[0043] A m44 A m480 A m481 B m44 A is 4-bit wide. m88 B m480 B m481 B m88 The output is 8-bit wide; the mantissa is composed of the following bits: 1'b0 indicates that the corresponding bit is filled with 0, a H0 a H1 a H2 a H3 Represents the input vector The hidden bits of each floating-point number in the array, b H0 b H1 b H2 b H3 Represents the input vector The hidden bits of each floating-point number in the array, a M0 a M1 a M2 a M3 Represents the input vector The mantissas of each floating-point number in b M0 b M1 b M2 b M3 Represents the input vector The mantissa of each floating-point number in the array; the numbers following the subscripts H and M represent the channel number in the 19-bit element.

[0044] For FP4, the mantissa is 1 bit, and after adding the hidden bit, it becomes 2 bits: A m44 = [1'b0, 1'b0, a H3 , a M3 ] B m44 = [1'b0, 1'b0, b H3 , b M3 ] A m480 = [1'b0, 1'b0, a H2 , a M2 ] B m480 = [1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, b H2 , b M2 ] A m481 = [1'b0, 1'b0, a H1 , a M1 ] B m481 = [1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, b H1 , b M1 ] A m88 = [1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, a H0 , a M0 ] B m88 = [1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, b H0 , b M0 ] FP6E2 / FP8E4, number of tails 3 bits, number of blocks 4 bits: A m44 = [a H1 , a M1 ] B m44 = [b H1 , b M1 ] A m480 = [a H0 , a M0 ] B m480 = [1'b0, 1'b0, 1'b0, 1'b0, b H0 , bM0 ] A m481 = [1'b0, 1'b0, 1'b0, 1'b0] B m481 = [1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0] A m88 = [1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0] B m88 = [1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0] For FP6E3 / FP8E5, the mantissa is 2 bits, which becomes 3 bits after adding the hidden bits: A m44 = [1'b0, a H1 , a M1 ] B m44 = [1'b0, b H1 , b M1 ] A m480 = [1'b0, a H0 , a M0 ] B m480 = [1'b0, 1'b0, 1'b0, 1'b0, 1'b0, b H0 , b M0 ] A m481 = [1'b0, 1'b0, 1'b0, 1'b0] B m481 = [1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0] A m88 = [1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0] B m88 = [1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0] For FP16 / TF32, the mantissa is 10 bits, A m / B mZ represents the 11-bit mantissa after padding with hidden bits in FP16 / TF32. m A represents m and B m The product of: A m = A m高3bit 256 + A m低8bit

[0045] B m = B m高3bit 256 + B m低8bit

[0046] Z m = A m B m

[0047] = (A m高3bit 256 + A m低8bit ) (B m高3bit 256 + B m低8bit )

[0048] = (A m高3bit B m高3bit 256 256) +

[0049] (A m高3bit B m低8bit 256) +

[0050] (A m低8bit B m高3bit 256) +

[0051] (A m低8bit B m低8bit )

[0052] Multiplying by 256 can be achieved by shifting the product left by 8 bits, therefore: A m44 = [1'b0, A m高3bit ] B m44 = [1'b0, B m高3bit ] Am480 = [1'b0, A m高3bit ] B m480 = [B m低8bit ] A m481 = [1'b0, B m高3bit ] B m481 = [A m低8bit ] A m88 = [A m低8bit ] B m88 = [B m低8bit ] For BF16, the mantissa is 7 bits, which becomes 8 bits after adding the hidden bits: A m44 = [1'b0, 1'b0, 1'b0, 1'b0] B m44 = [1'b0, 1'b0, 1'b0, 1'b0] A m480 = [1'b0, 1'b0, 1'b0, 1'b0] B m480 = [1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0] A m481 = [1'b0, 1'b0, 1'b0, 1'b0] B m481 = [1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0, 1'b0] A m88 = [a H , a M ] B m88 = [b H , b M ] For INT8, all significant numbers are 8 bits. Similarly, the data in the first channel is split according to the FP16 mantissa, and the data in the second channel does not need to be split, or the data in the second channel is split but the data in the first channel is not split: A m0 = A m0高4bit 16 + A m低4bit

[0053] Zm0 = A m0 B m0

[0054] = (A m0高4bit 16 + A m低4bit ) B m0

[0055] = (A m高4bit B m0 16) +

[0056] (A m低4bit B m0 )

[0057] A m44 = [1'b0, 1'b0, 1'b0, 1'b0]

[0058] B m44 = [1'b0, 1'b0, 1'b0, 1'b0]

[0059] A m480 = [A m高4bit ]

[0060] B m480 = [B m0 ]

[0061] A m481 = [A m低4bit ]

[0062] B m481 = [B m0 ]

[0063] A m88 = [a M1 ]

[0064] B m88 = [b M1 ]

[0065] Different data types are selected based on the input mode, and then output after selection by a multiplexer (MUX); The floating-point elements in two vectors are summed pairwise using their sign-magnitude representations, for example, E a0 +E b0 E a1 +E b1 , ..., E an +E bn ; For FP4, each element has four exponents of valid floating-point data, each 2 bits; For FP6E2, each element has an exponent of two valid floating-point data, each 2 bits; For FP6E3, each element has two valid floating-point exponents, each 3 bits; For FP8E4, each element has an exponent of two valid floating-point data, each 4 bits; For FP8E5, each element has two valid floating-point exponents, each 5 bits; For FP16, each element has a valid floating-point exponent of 5 bits; For BF16, each element has an exponent of valid floating-point data, which is 8 bits; For TF32, each element has an exponent of a valid floating-point data, which is 8 bits; For INT8, it does not participate in exponentiation summation operations.

[0066] To prevent overflow, 0 needs to be padded to the high bits of each exponent. Therefore, a 12-bit adder can be used to complete the exponent summation operation. The bit mapping of different data types in the adder is shown in Table 2.

[0067] Table 2

[0068] To reduce power consumption and unnecessary flips, invalid bits for each data type are filled with 0.

[0069] The data preprocessing submodule 110 can also be configured with a ZeroSkipping mechanism. Before calling the multiplier to calculate the mantissa product corresponding to the mantissa data, the register is used to tick the matrix input data to check whether the matrix input data is zero. If it is zero, the register keeps the matrix input data of the previous tick unchanged; otherwise, it is updated to the current matrix input data.

[0070] Specifically, before the input data is input to the multiplier (in the mantissa operation submodule 140), in order to meet the frequency requirements, the input data can be clocked using the register R_MUL_IN. This clocked register is used to implement the ZeroSkipping mechanism to reduce power consumption. Check if the input data is 0. If it is 0, the register R_MUL_IN remains unchanged from the previous input data. Otherwise, update the register R_MUL_IN to the current input data. The indicator signal indicating whether the input data is 0 is delayed by one cycle and used as the selection signal for the two-to-one multiplexer of the mantissa multiplier output data. If the indicator signal is 0, the multiplexer output is the product of the multiplier; otherwise, it outputs 0.

[0071] In practical applications, a certain proportion of the input data to AI networks is zero data, which does not need to participate in the calculation. This mechanism can reduce unnecessary logic flips and lower power consumption.

[0072] refer to Figure 4 This is a schematic diagram of a zero-value skipping architecture according to a specific embodiment of the present invention. This architecture detects whether the input data of the multiplier is zero and skips invalid operations with zero values ​​to reduce module power consumption. Figure 4 As shown, the processing procedure includes: The system receives two input data streams, A and B, which are then fed into corresponding 0 detection modules (one for data A and one for data B). Each 0 detection module determines if the current input data is zero and outputs a corresponding 0 flag signal (this signal is set when the data is zero). Input data selection and temporary storage are also included.

[0073] The input data selector selects the input source based on the 0 flag signal. If the current data is zero, it selects the data from the previous clock cycle of the delay register (skipping the operation on zero values); if the data is non-zero, it selects the current input data.

[0074] The selected result is stored in the multiplier input data registers (multiplier input data register A and multiplier input data register B) as the multiplier's input for operation.

[0075] The 0 flag signal is delayed by one clock cycle through a delay register to keep it synchronized with the operation timing of the multiplier (multiplication requires one clock cycle). The multiplier performs multiplication operations on the contents of the two multiplier input data registers.

[0076] The input data selector selects the output based on the delayed 0 flag signal; if the 0 flag signal is set (corresponding to zero input data), it directly selects 0 output; if the signal is not set (input is non-zero), it selects the result of the multiplier operation to output; finally, it outputs the multiplication output data.

[0077] The above logic allows skipping invalid multiplication operations when the input data is zero, reducing signal switching of the multiplier and thus lowering power consumption.

[0078] Index Comparison Submodule 120: The maximum value of the exponent is determined by exponential comparison. Data of data type INT8 is not included in the exponential comparison.

[0079] Specifically, a comparator tree consisting of m levels and n comparators per level is used to compare the exponents output by the data preprocessing submodule 110. Registers are inserted between each level to meet performance requirements. m and n can be considered comprehensively based on the current process, timing, and area conditions.

[0080] Reuse a 12-bit unsigned adder; exponentiation operations on all data types use the original code. For FP4, with 4 channels, the exponent width is 2 bits, and it is extended to 3 bits by padding with 0s in the high bits, resulting in a total of 4L 3-bit exponent sums; For FP6E2, which has 2 channels, the exponent is 2 bits wide and is padded with 0s in the high bits to extend it to 3 bits, resulting in a total of 2L 3-bit exponent sums. For FP6E3, which has 2 channels, the exponent width is 3 bits, and it is extended to 4 bits by padding with 0s in the high bits, resulting in a total of 2L 4-bit exponent sums; For the FP8E4, which has 2 channels, the exponent width is 4 bits, and it is extended to 5 bits by padding with 0s in the high bits, resulting in a total of 2L 5-bit exponent sums; For the FP8E5, which has 2 channels, the exponent width is 5 bits, and it is extended to 6 bits by padding with 0s in the high bits, resulting in a total of 2L 6-bit exponent sums; For FP16, channel 1, the exponent width is 5 bits, and it is extended to 6 bits by padding with 0s in the high bits, and there are a total of L 6-bit exponents; For BF16, channel 1, the exponent width is 8 bits, and it is extended to 9 bits by padding with 0s in the high bits, and there are a total of L 9-bit exponents; For TF32, channel 1, the exponent width is 8 bits, and it is extended to 9 bits by padding with 0s in the high bits, and there are a total of L 9-bit exponents; For INT8, it is not involved in the calculation of this module.

[0081] A single comparator is designed as a comparator with 2 or more inputs and a bit width of 12 bits. The comparator needs to have a mode selection input, and the output is the maximum value of the input data. The adder used for exponent addition is reused to reduce the resource consumption of the adder.

[0082] The bit mappings of different data types at the comparator input and output are shown in Table 3.

[0083] Table 3

[0084] To reduce power consumption and unnecessary flips, invalid bits for each data type are filled with 0.

[0085] Exponential subtraction submodule 130: The difference between the maximum index value and each index data is calculated to determine the index difference; however, data corresponding to the INT8 data type is not included in the difference calculation.

[0086] Specifically, the maximum value of the index output by the index comparison submodule 120 is compared with the various index data output by the data preprocessing submodule 110 to calculate the index difference.

[0087] Mantissa operation submodule 140: The multiplier is invoked to calculate the mantissa product corresponding to the mantissa data. For the mantissa product corresponding to data types FP16, TF32 or INT8, the output results of multiple multipliers are summed to obtain the mantissa product.

[0088] It consists of an 8x8, two 4x8, and one 4x4 integer multiplier and a 24-bit adder tree, used to calculate the mantissa product of the data preprocessing submodule 110.

[0089] Z m44 Z is the 8-bit product result of a 4x4 multiplier. m480 Z is the 12-bit product result of the first 4x8 multiplier. m481 Z is the 12-bit product result of the second 4x8 multiplier. m88 This is the 16-bit product result of an 8x8 multiplier. m0 This refers to the product result of the mantissa operation submodule 140 when there is only one channel of data, or the product result of the first channel of the mantissa operation submodule 140 when there are two or four channels. m1 Z is the product result of the second channel of the mantissa operation submodule 140 when dealing with two / four-channel data. m2 Z m3 This is the product of the third and fourth channels of the mantissa operation submodule 140 in the four-channel operation.

[0090] For FP16 / TF32, the outputs of the four integer multipliers need to be summed together: Z m0 = Z m44 256 256 + Z m480 256 + Z m481 256 + Z m88

[0091] For INT8, the outputs of the four integer multipliers need to be summed together: Z m0 = Z m44 + Zm480 16 + Z m481 16 + Z m88

[0092] Multiplying by 16 can be achieved by shifting the product left by 4 bits, multiplying by 256 can be achieved by shifting the product left by 8 bits, and multiplying by two 256s requires shifting by 16 bits.

[0093] For other data types, the adder is not used.

[0094] All mantissa product terms are represented in original code at this time. After converting the mantissa to two's complement based on the sign bit, the data is output to the shift alignment submodule 150 for shift alignment.

[0095] Shift Alignment Submodule 150: The mantissa product is shifted based on the exponent difference, wherein the mantissa product corresponding to the data type INT8 is not involved in the shift operation.

[0096] Specifically, the product terms output by the mantissa operation submodule 140 are shifted based on the exponent difference of each floating-point data output by the exponent difference submodule 130. The shift is implemented using a barrel shifter, and INT8 data does not participate in the shift operation.

[0097] For FP4, the product term consists of four 4-bit data items; For FP6E2 / FP8E4, the product term consists of two 8-bit data; For FP6E3 / FP8E5, the product term consists of two 6-bit data items; For FP16, the product term is a 24-bit data; For BF16 / TF32, the product term is a 16-bit data.

[0098] Accumulation submodule 160: For the mantissa product after the shift operation or the mantissa product corresponding to the data type INT8, the adder is called to perform the accumulation calculation to obtain the accumulated mantissa result.

[0099] Specifically, the shifted mantissas output by the shift alignment submodule 150, or the original INT8 integer product data, are accumulated using an adder implemented with a Wallace tree. In practical applications, depending on the process and timing requirements, the Wallace tree addition implementation can be divided into single-level or multi-level implementations.

[0100] Normalization submodule 170: The result of the mantissa accumulation is normalized; however, the result of the mantissa accumulation corresponding to the data type INT8 is not included in the normalization process. Specifically, the output after accumulation by the accumulator submodule 160 includes the maximum exponent and the sum of the mantissas. The mantissas are normalized according to the IEEE 754 standard. For INT8 input data types, they do not participate in the calculation of this module.

[0101] In this embodiment, the normalization method of the IEEE 754 standard is used to normalize the output of the adder.

[0102] The IEEE 754 standard rounding method is used to round the normalized data.

[0103] The need to add a pipeline register before and after normalization and rounding depends on the operating frequency and process. After rounding, the output is standard FP format data.

[0104] Whether or not to support all rounding methods depends on the accuracy requirements, which can reduce the hardware area.

[0105] Correction Submodule 180: Corrects the maximum value of the exponent.

[0106] The exponent is processed based on the input mode information and unified to the output data type FP32. The exponent is calculated using the original code throughout the processing path, therefore a correction is required. The correction formula is as follows: This represents the exponent of the final output data; + This represents the maximum exponent corresponding to the two sets of matrix input data; This represents the exponent bias of the input floating-point number, and is a constant. This represents the exponential bias of the FP32, which is a constant, such as 127.

[0107] Correction submodule 180 needs to add to the maximum exponent output by accumulation submodule 160. - - In particular, special handling is performed for special numbers (inf / nan) or subnormal numbers that occur during the calculation process.

[0108] The final output data type of the dot product module 100 is as follows: for matrix input data of type floating point, exponential processing is performed according to the mode information, and the output data type is FP32; for matrix input data of type INT8, the output data type is INT32.

[0109] Intermediate and register set 210: Load the intermediate data to be calculated, store the current accumulated result, and output the accumulated result when the calculation ends.

[0110] The intermediate and register groups employ two sets of ping-pong switching control to store the intermediate values ​​of the accumulated calculation results from the MAC module. Let the left matrix be M. α K γ The right matrix is ​​N β K γ Based on the characteristics of matrix multiplication, each group requires the first dimension α of the input matrix. It has β 32-bit registers to store the final FP32 or INT32 result; it also has a Load interface and a Store interface; at the start of the calculation, the Host loads the data to be accumulated into the registers through the Load interface; at the end of the calculation, the Host obtains the final calculation result through the Store interface.

[0111] The Load interface includes a 1-bit load enable signal, a 1-bit ping-pong flag signal, and log2(α) β)bit register id signal, load data signal; The Store interface includes a 1-bit storage enable signal, a 1-bit ping-pong flag signal, and log2(α) β)bit register id signal, stored data signal.

[0112] Data distribution submodule 220: Based on the matrix multiplication and addition operation identifier signal (PC signal) corresponding to the current calculation result, the accumulated result is written into the corresponding intermediate sum register group 210; Data selection submodule 230: Based on the matrix multiplication and addition operation identifier signal (PC signal) corresponding to the current input data, the intermediate data of the intermediate and register group 210 is loaded into one of the addends of the accumulator 240; The matrix multiplication and addition operation identifier signal (PC signal) includes the first dimension information of the first matrix and the first dimension information of the second matrix in the two sets of matrix input data; for the matrix input data input in each clock cycle, a matrix multiplication and addition operation identifier signal is matched as the identifier of the current dot product calculation result.

[0113] The PC signal mechanism is explained below.

[0114] For each clock cycle, the input matrix data is matched with a PC signal to identify the current matrix multiplication result. The PC signal consists of the first dimension information of the first matrix and the first dimension information of the second matrix input for matrix multiplication.

[0115] If there is only one MAC unit in the matrix multiplication and addition operation system, the calculation output of the MAC unit is shown in Table 4.

[0116] Table 4

[0117] In the table, M x K x N x K y M x N y Both represent x rows and y columns of a matrix.

[0118] If a matrix multiplication and addition system has 4x4 MAC units, the system can compute M4K in one clock cycle. γ N4, the (0,0)th MAC unit calculates M1K γ N1, the (0,1)th MAC unit calculates M2K γ N1, ..., the (1,0)th MAC unit calculates M1K γ N2, the (1,1)th MAC unit calculates M2K γ N2, ..., the (3,3)th MAC unit calculates M4K γ N4; For the (0,0)th MAC unit, the calculated output is shown in Table 5.

[0119] Table 5

[0120] For the (0,1)th MAC unit, the calculated output is shown in Table 6.

[0121] Table 6

[0122] The same applies to other MAC units, so they will not be elaborated on here.

[0123] Accumulator 240: The intermediate data is summed with the dot product calculation result output by the dot product module to obtain the summed result.

[0124] In this embodiment, a clock and data gating module (not shown) is also included to implement the Clock / Data Gating mechanism. This module is used to shut down the registers corresponding to inactive data types and the logical resources on the logical data path during the current matrix multiplication and addition operation.

[0125] Specifically, in the current instruction, disabling the register clock associated with data types that do not require operation reduces register power consumption; by using AND gates to disable logic resources on logic data paths associated with data types that do not require operation, unnecessary signal toggling is reduced, thereby reducing power consumption.

[0126] In practical applications, based on the Clock / Data Gating mechanism, multiple data types will not appear simultaneously in the same instruction, and the related logic of data types that do not work in the current instruction does not need to be toggled, thus reducing power consumption.

[0127] This invention achieves unified support for multiple data types through the collaborative operation of the dot product module and the accumulation module. A single MAC unit can handle nine data formats, including FP4 / FP6E2 / FP6E3 / FP8E3 / FP8E4 / FP16 / BF16 / TF32 / INT8, significantly improving hardware resource utilization. The reused design of multipliers and adders significantly reduces hardware area and power consumption, resulting in a substantial reduction in hardware cost compared to traditional solutions requiring multiple independent computing units. The zero-skipping mechanism effectively identifies and skips zero-value operations, avoiding unnecessary power consumption and demonstrating significant power optimization in sparse matrix operations. The clock / data gating mechanism dynamically shuts down unnecessary hardware modules based on the actual data type being processed, further reducing system power consumption. Support for unified output in FP32 and INT32 formats ensures computational accuracy while simplifying subsequent data processing.

[0128] Based on the same inventive concept, this invention also proposes a matrix multiplication and addition operation method. This method is applied to the aforementioned matrix multiplication and addition operation unit and is executed based on the aforementioned matrix multiplication and addition operation unit. This matrix multiplication and addition operation method and the matrix multiplication and addition operation unit in the embodiments of this invention described above are based on the same inventive concept and have similar problem-solving principles. Therefore, the implementation of the matrix multiplication and addition operation method is the same as the implementation of the matrix multiplication and addition operation unit in the embodiments of this application described above, and repeated details will not be repeated. Figure 5 As shown, the specific methods include: S101, the matrix input data is mapped according to the data type, and the mantissa data and exponent data corresponding to the matrix input data are determined based on the mapping result. The dot product calculation result is obtained by dot product calculation. S102, load the intermediate data to be calculated into the intermediate sum register group, read the corresponding intermediate data from the intermediate sum register group according to the matrix multiplication and addition operation identifier signal, accumulate it with the dot product calculation result, write the accumulation result into the intermediate sum register group, and output the accumulation result when the operation ends.

[0129] The matrix multiplication and addition unit, method, system, chip, board, and electronic device proposed in this invention can support matrix multiplication and addition operations with multiple data types, efficiently reuse hardware resources, and significantly reduce hardware area and power consumption without sacrificing computational performance by combining multiple control mechanisms. The accumulation process combines the switching mechanism between intermediate and register groups, the loading storage system, and the precise synchronization of matrix multiplication and addition operation identification signals to ensure the accuracy and continuity of intermediate value accumulation during matrix multiplication and addition. It has high parallel scalability and can flexibly adapt to high-performance computing scenarios. In large-scale integration scenarios such as wafer-level chips, it can achieve a significant improvement in computing power density and energy efficiency ratio, meeting the high precision, high computing power, and low power consumption requirements of different scenarios.

[0130] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.

[0131] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0132] This invention is described with reference to flowchart illustrations and / or block diagrams of methods and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0133] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0134] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0135] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A matrix multiplication and addition operation unit, characterized in that, include: The dot product module is used to map the matrix input data according to the data type, determine the mantissa data and exponent data corresponding to the matrix input data based on the mapping result, and obtain the dot product calculation result after dot product calculation. The accumulation module is used to load the intermediate data to be calculated into the intermediate sum register group, read the corresponding intermediate data from the intermediate sum register group according to the matrix multiplication and addition operation identifier signal, accumulate it with the dot product calculation result, write the accumulation result into the intermediate sum register group, and output the accumulation result when the operation ends.

2. The matrix multiplication and addition unit according to claim 1, characterized in that, The dot product module supports input data types including at least: FP4, FP6E2, FP6E3, FP8E4, FP8E5, FP16, BF16, TF32 and INT8.

3. The matrix multiplication and addition unit according to claim 1, characterized in that, The dot product module includes at least a data preprocessing submodule; wherein... The data preprocessing submodule is used to: perform mapping processing on the matrix input data using a unified data bit format; Based on the mapping result, valid data is extracted. If the data type of the matrix input data is floating point, the hidden bits of the matrix input data are determined and the hidden bits are added to the highest bit of the mantissa to extract valid data. If the data type of the matrix input data is INT8, the valid data of the matrix input data is extracted directly. The valid data is rearranged according to the preset multiplier structure, and the mantissa data adapted to the multiplier bit width is output. For matrix input data of floating-point type, the exponent data is obtained by exponential summation operation; for matrix input data of data type INT8, the exponent summation operation is not performed.

4. The matrix multiplication and addition unit according to claim 3, characterized in that, The data preprocessing submodule is also used for: The matrix input data is mapped according to a preset bit as an element, and invalid bits are filled with 0; The number of supported channels is determined based on the data type, where each element contains valid data for the corresponding number of channels.

5. The matrix multiplication and addition unit according to claim 3, characterized in that, The dot product module further includes: an exponent comparison submodule, an exponent difference submodule, a mantissa operation submodule, a shift alignment submodule, and a mantissa accumulation submodule; wherein... The index comparison submodule is used to: determine the maximum index value in the index data through index comparison; The index difference calculation submodule is used to: calculate the difference between the maximum index value and each index data to determine the index difference; wherein, the data corresponding to the data type INT8 does not participate in the index comparison and difference calculation; The mantissa operation submodule is used to: call the multiplier to calculate the mantissa product corresponding to the mantissa data. For the mantissa product corresponding to data types FP16, TF32 or INT8, the output results of multiple multipliers are accumulated to obtain the mantissa product. The shift alignment submodule is used to: perform a shift operation on the mantissa product according to the exponent difference, wherein the mantissa product corresponding to the data type INT8 does not participate in the shift operation; The accumulation submodule is used to: call the adder to perform accumulation calculation for the mantissa product after the shift operation or the mantissa product corresponding to the data type INT8 to obtain the mantissa accumulation result.

6. The matrix multiplication and addition unit according to claim 5, characterized in that, The data preprocessing submodule is also used to: before calling the multiplier to calculate the mantissa product corresponding to the mantissa data, use the register to clock the matrix input data and detect whether the matrix input data is zero; if it is zero, the register keeps the matrix input data of the previous clock unchanged, otherwise it is updated to the current matrix input data.

7. The matrix multiplication and addition unit according to claim 5, characterized in that, The dot product module further includes: a normalization submodule and a correction submodule; wherein... The normalization submodule is used to: normalize the mantissa accumulation result; wherein, the mantissa accumulation result corresponding to data type INT8 does not participate in the normalization process; The correction submodule is used to: correct the maximum value of the exponent, and the calculation formula is: This represents the exponent of the final output data; + This represents the maximum exponent corresponding to the two sets of matrix input data; Indicates the exponent bias of the input floating-point number; This indicates the exponential bias of FP32.

8. The matrix multiplication and addition unit according to claim 4, characterized in that, The dot product module is specifically used for: for matrix input data of type floating point, performing exponential processing according to mode information, and outputting data of type FP32; for matrix input data of type INT8, outputting data of type INT32.

9. The matrix multiplication and addition unit according to claim 1, characterized in that, The accumulation module includes: an intermediate and register group, a data distribution submodule, a data selection submodule, and an accumulator; The intermediate and register groups are used to: load intermediate data that needs to be processed, store the current accumulated result, and output the accumulated result when the operation ends; The data distribution submodule is used to: write the accumulated result into the corresponding intermediate and register groups according to the matrix multiplication and addition operation identifier signal corresponding to the current calculation result; The data selection submodule is used to: load intermediate data from the intermediate and register groups into one of the addends of the accumulator according to the matrix multiplication and addition operation identifier signal corresponding to the current input data; The accumulator is used to: perform an accumulation operation on the intermediate data and the dot product calculation result output by the dot product module to obtain the accumulated result.

10. The matrix multiplication and addition unit according to claim 1 or 9, characterized in that, The matrix multiplication and addition operation identifier signal includes the first dimension information of the first matrix and the first dimension information of the second matrix in the two sets of matrix input data; for each clock cycle of input matrix input data, a matrix multiplication and addition operation identifier signal is matched as the identifier of the current dot product calculation result.

11. The matrix multiplication and addition unit according to claim 1, characterized in that, Also includes: The clock and data gating module is used to shut down registers and logical resources on logical data paths corresponding to inactive data types during the current matrix multiplication and addition operation.

12. A matrix multiplication and addition operation method, characterized in that, The method is performed based on the matrix multiplication and addition unit according to any one of claims 1 to 11, comprising: The matrix input data is mapped according to the data type. Based on the mapping result, the mantissa data and exponent data corresponding to the matrix input data are determined. The dot product calculation result is obtained by dot product calculation. The intermediate data to be calculated is loaded into the intermediate sum register group. The corresponding intermediate data is read from the intermediate sum register group according to the matrix multiplication and addition operation identifier signal. The intermediate data is accumulated with the dot product calculation result and the accumulated result is written into the intermediate sum register group. The accumulated result is output when the operation ends.

13. A matrix multiplication and addition operation system, characterized in that, The system includes a plurality of matrix multiplication and addition units as described in any one of claims 1 to 11.

14. A chip, characterized in that, It includes multiple computing cores, wherein some or all of the computing cores include the matrix multiplication and addition operation unit as described in any one of claims 1 to 11.

15. A circuit board, characterized in that, Includes the chip described in claim 14.

16. An electronic device, characterized in that, Includes the board as described in claim 15.

Images