Mixed data precision matrix multiplication and addition unit and calculation method

Abstract

The invention provides a mixed data precision matrix multiplication and addition unit and a calculation method, the matrix multiplication and addition unit comprises a calculation unit, and the calculation unit comprises a format division module, a multiplication array module, an addition tree module, an accumulator module, a normalization module and a shift register module. The calculation unit converts the first input matrix and the second input matrix into input data in a middle floating point format; executing parallel multiplication operation on the input data to generate an intermediate product result; performing index alignment and accumulation on the intermediate product result to generate an intermediate accumulated value; accumulating the intermediate product result and the value of the third input matrix in a form of accumulating an intermediate accumulated value, and outputting an accumulated result; and converting an accumulation result into a normalized result and outputting the normalized result. The format division module supports various precisions and converts data with different widths into an intermediate floating point format, so that other hardware units can be reused, and the problems that hardware resources are complex and different model reasoning scenes are difficult to meet are solved.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a mixed data precision matrix multiplication and addition unit and a calculation method. Background Art

[0002] The matrix multiplication and addition unit (MMU) is a hardware component used in neural network computing. It is used to perform matrix multiplication and accumulation operations. It can execute intelligent computing tasks with exponentially growing parameter sizes, such as convolutional neural networks and large language models. Taking large language models as an example, their inference and training processes require processing billions to trillions of mixed-precision matrix operations, and different computing stages have different requirements for accuracy, throughput, and energy efficiency. These tasks require hardware components to achieve dynamic compatibility with multiple precisions and efficient reuse of computing resources within a limited chip area, while also meeting high computing density and low power consumption.

[0003] To meet the above requirements, the matrix multiplication acceleration solution is based on a fixed-precision computing architecture. For example, a high-precision floating-point arithmetic unit, such as FP32, is used to complete all calculations, and high numerical precision is achieved through dedicated floating-point multipliers and accumulators. In addition, some accelerators use fixed-point arithmetic units, such as INT8, to optimize the energy efficiency of inference scenarios, but their hardware architecture is completely decoupled from the floating-point arithmetic unit to form an independent data path.

[0004] However, the fixed-precision architecture is not compatible with mixed-precision computing requirements, resulting in complex hardware resources and difficulty in meeting different model inference scenarios. Summary of the Invention

[0005] This application provides a mixed data precision matrix multiplication and addition unit and calculation method to solve the problem of complex hardware resources and difficulty in meeting different model inference scenarios.

[0006] In a first aspect, the present application provides a mixed data precision matrix multiply-add unit, comprising: a computing unit, a control register, a cache unit, and a cache management unit;

[0007] The calculation unit includes:

[0008] a format division module, configured to convert the first input matrix and the second input matrix into input data in an intermediate floating point format;

[0009] a multiplication array module, configured to perform parallel multiplication operations on the input data in the intermediate floating point format to generate intermediate product results;

[0010] an addition tree module, configured to perform exponent alignment and accumulation on the intermediate product results to generate an intermediate accumulated value;

[0011] an accumulator module, configured to accumulate the intermediate product result and the value of the third input matrix using the intermediate accumulated value to output an accumulated result;

[0012] A normalization module is used to convert the accumulated result into an output matrix.

[0013] In some feasible embodiments, the control register is used to receive a control signal from a central processing unit, wherein the control signal includes a matrix data type signal, a quantization type signal, a low-precision mode signal, and a set matrix dimension signal;

[0014] The buffer unit includes a first preset number of buffers for caching the first input matrix, the second input matrix, the third input matrix and the normalized result;

[0015] The buffer management unit is configured to output the first input matrix and the second input matrix to the calculation unit in sequence according to the control signal.

[0016] In some feasible embodiments, the method further includes a shift register module, wherein the shift register module is configured to serially load the third input matrix; and receive the normalized results in parallel and convert the normalized results into serial data;

[0017] The control register is also used to send the matrix data type signal and the low-precision mode signal to the format division module, send the quantization type signal to the shift register module, and send the set matrix dimension signal to the buffer management unit and the shift register module.

[0018] In some feasible embodiments, the accumulator module includes an accumulation register;

[0019] The accumulation register is further configured to read the third input matrix, accumulate the intermediate product result and the value of the third input matrix in the form of the intermediate accumulated value, and generate an accumulation completion signal;

[0020] The normalization module is used to receive the accumulation completion signal and convert the intermediate accumulation value into a normalized result in a target format.

[0021] In some feasible embodiments, the format division module is further configured to receive a matrix data type signal sent by the control register, and segment the mantissas of the elements of the first input matrix and the second input matrix according to a preset number of bits based on the input precision of the first input matrix and the second input matrix;

[0022] Generate a first intermediate floating-point format data block and a second intermediate floating-point format data block, wherein the first intermediate floating-point format data block is obtained by appending a sign bit and an exponent offset value of the first input matrix to the segmented data, and the second intermediate floating-point format data block is obtained by appending a sign bit and an exponent offset value of the second input matrix to the segmented data;

[0023] Grouping is performed on the first intermediate floating-point format data block and the second intermediate floating-point format data block to output input data in an intermediate floating-point format.

[0024] In some feasible embodiments, the shift register module further includes a shift register array, a first buffer manager, and a second buffer manager;

[0025] The first buffer manager is used to receive a third input matrix and send the third input matrix to the shift register array;

[0026] The shift register array is further configured to receive the normalized result and write the result into the second cache manager;

[0027] The second cache manager is configured to read the normalized result from and generate an output matrix, and output the output matrix to an external storage unit.

[0028] In some feasible embodiments, the format division module is further configured to:

[0029] Obtaining a first sequence number and a second sequence number based on an order in which the first intermediate floating-point format data block and the second intermediate floating-point format data block are generated, wherein the first sequence number is the sequence number of the first intermediate floating-point format data block, and the second sequence number is the sequence number of the second intermediate floating-point format data block;

[0030] combining the first intermediate floating point format data block and the second intermediate floating point format data block to generate a product term, wherein the product term is input into the multiplication array module;

[0031] The first intermediate floating point format data block and the second intermediate floating point format data block are grouped according to the product terms to output input data in an intermediate floating point format, wherein the product terms of the input data in the same group are less than a preset threshold.

[0032] In some feasible embodiments, the format division module is also used to ignore the target mantissa segments in the first input matrix and the second input matrix according to the low-precision mode signal sent by the control register, and convert the non-target mantissa segments into intermediate floating-point format data blocks, wherein the target mantissa segment is the smallest mantissa segment in the target group.

[0033] In some feasible embodiments, the multiplication array module includes a second preset number of multiplication array groups multiplied by a second preset number, the format division module is connected to a second preset number of multipliers, and the number of the format division modules is the first preset number;

[0034] The multiplication array group is used to perform a multiplication operation on the input data in the intermediate floating-point format within the same group to generate a product result;

[0035] The product result is stored in an extended precision format to generate an intermediate product result, wherein exponent bits and mantissa bits of the intermediate product result are larger than exponent bits and mantissa bits of the first input matrix and the second input matrix.

[0036] In a second aspect, the present application provides a calculation method for a mixed data precision matrix multiplication and addition unit, comprising:

[0037] Converting the first input matrix and the second input matrix into input data in an intermediate floating point format;

[0038] performing a parallel multiplication operation on the input data in the intermediate floating point format to generate an intermediate product result;

[0039] performing exponent alignment and accumulation on the intermediate product results to generate an intermediate accumulation value;

[0040] Accumulating the intermediate product result and the value of the third input matrix in the form of the intermediate accumulated value to output an accumulated result;

[0041] The accumulated result is converted into an output matrix.

[0042] As can be seen from the above technical solution, the present application provides a mixed data precision matrix multiplication and addition unit and calculation method, the matrix multiplication and addition unit includes: a calculation unit, a control register, a buffer unit, a buffer management unit and a normalization module, the calculation unit includes a format division module, a multiplication array module, an addition tree module and an accumulator module. The format division module is used to convert the first input matrix and the second input matrix into input data in an intermediate floating-point format, the multiplication array module is used to perform parallel multiplication operations on the input data in the intermediate floating-point format to generate an intermediate product result, the addition tree module is used to perform exponent alignment and accumulation on the intermediate product result to generate an intermediate accumulated value, the accumulator module is used to accumulate the intermediate product result and the value of the third input matrix in the form of an intermediate accumulated value to output the accumulated result, the normalization module converts the accumulated result into a normalized result, the shift register module receives the third input matrix value and inputs it into the accumulation module, accepts the normalized result and outputs the matrix. The format division module supports multiple precisions and converts data of different widths into an intermediate floating-point format, so that other hardware units can be reused without additional overhead, thereby solving the problem of complex hardware resources and difficulty in meeting different model inference scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 A schematic diagram of the structure of a mixed data precision matrix multiplication and addition unit provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram showing the connection between the format division modules A and B provided in an embodiment of the present application;

[0046] Figure 3 Schematic diagram of the FP32 IFFs0 partitioning order provided in an embodiment of the present application;

[0047] Figure 4 Schematic diagram of the FP16 IFFs0 partitioning sequence provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of the BF16 IFFs0 partitioning sequence provided in an embodiment of the present application;

[0049] Figure 6 Schematic diagram of the INT16 IFFs0 division order provided in an embodiment of the present application;

[0050] Figure 7Schematic diagram of the INT8 IFFs0 partitioning sequence provided in an embodiment of the present application;

[0051] Figure 8 Schematic diagram of the INT4 IFFs0 division order provided in an embodiment of the present application;

[0052] Figure 9 A schematic diagram of the structure of the accumulator module provided in an embodiment of the present application;

[0053] Figure 10 This is a schematic diagram of the shift register module structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following embodiments are described in detail, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numbers in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following embodiments are not intended to represent all possible implementations consistent with the present application. They are merely examples of systems and methods consistent with certain aspects of the present application, as detailed in the claims.

[0055] The matrix multiplication acceleration solution is based on a fixed-precision computing architecture. For example, it uses high-precision floating-point arithmetic units, such as FP32, to complete all calculations, and achieves high numerical accuracy through dedicated floating-point multipliers and accumulators. In addition, some accelerators use fixed-point arithmetic units, such as INT8, to optimize the energy efficiency of inference scenarios, but their hardware architecture is completely decoupled from the floating-point arithmetic units to form an independent data path.

[0056] To solve the above problems, in some embodiments, a data path that decouples floating-point and fixed-point calculations is adopted, that is, in floating-point mode, floating-point data path calculations are used, and in integer mode, integer data path calculations are used. That is, when one mode is used, the computing resources of the other mode are idle, which increases the chip area requirement but reduces the reuse rate of computing resources.

[0057] In other embodiments, a multiply-add unit combining fixed-point and floating-point operations is used to improve hardware resource reuse. For example, a reconfigurable computing architecture supports multi-precision computing through a flexible hardware structure. This architecture offers high flexibility, enabling runtime adjustments to computing resources and precision configurations. It boasts high resource utilization and adaptability, supporting multiple computing modes and precision requirements simultaneously. However, this architecture carries high hardware complexity and design challenges, and the reconfigurable logic introduces additional overhead, impacting performance and energy efficiency.

[0058] To solve the above problems, some embodiments of this application provide a mixed data precision matrix multiplication and addition unit, such as Figure 1As shown, it includes: a computing unit, a control register, a cache unit and a cache management unit. The computing unit is used to convert input matrices of different precisions into an intermediate floating-point format, such as IFFs0, and complete parallel multiplication and addition operations. The control register is used to receive control signals from the CPU (central processing unit) and send them to other units. The cache unit includes multiple caches to cache input data, intermediate results and accumulated results. The cache management unit is used to dynamically allocate caches according to control signals.

[0059] This embodiment communicates with an external memory or CPU through the AXI4-full bus protocol and a DMA unit (Direct Memory Access Unit) to obtain data required for matrix calculation and output the result of the calculation.

[0060] In some embodiments, the control register is used to receive control signals from a central processing unit, wherein the control signals include a matrix data type signal, a quantization type signal, a low-precision mode signal, and a set matrix dimension signal.

[0061] Among them, for example, there are three input matrices, namely A, B, and C, and the matrix data type signals correspond to csr_atype, csr_btype, and csr_ctype signals; the low-precision mode signal is the csr_is_low signal that controls whether it is a high-precision calculation; the quantization type signal is csr_dtype, which sets the target precision of the output matrix D (such as FP32, INT32); the matrix dimension signals are set to csr_isize, csr_jsize, and csr_ksize to define the matrix dimension, for example, A is M×K and B is K×N.

[0062] The low-precision mode signal is sent to the format division module, and the format division module controls the order of multiplication and addition of the two multiplier calculation parts and the number of groups; the quantization type signal is sent to the shift register module to define the precision type of the output matrix D, such as FP32, INT32, and the matrix dimension setting signal is sent to the cache management unit, the first cache manager and the second cache manager to define the input matrix dimension.

[0063] In some embodiments, the cache unit includes a first preset number of cache BANKs for caching the first input matrix, the second input matrix, the intermediate accumulated values, and the accumulated results. In this embodiment, the first preset number is 16, and each BANK has a bit width of 32 bits and a depth of 512.

[0064] The buffer management unit is configured to write the first input matrix, the second input matrix, the third input matrix, and the normalized result into the buffer unit according to a control signal. To facilitate storage of data of different input matrices, in some embodiments, two buffer management units are included, one for acquiring the first input matrix and the other for acquiring the second input matrix, and sending the matrixes to the calculation unit for calculation in the order of matrix multiplication. The input matrices can be arranged in row-major or column-major storage.

[0065] After the data is stored, it is calculated by a calculation unit, wherein the calculation unit includes:

[0066] A format division module is used to convert the first input matrix A and the second input matrix B into input data in an intermediate floating-point format. The precision of the first input matrix and the second input matrix can be different. For example, the first input matrix A is INT8 and the second input matrix B is FP16. A and B are converted to IFFs0. The first input matrix and the second input matrix can also have the same precision.

[0067] Exemplarily, FP32 is split into 6 groups of IFFs0, each group contains a sign bit (1 bit), an exponent bit (9 bits), and a mantissa bit (4 bits).

[0068] The following table defines the original precision of the input data and the rules for converting it to the intermediate floating-point format. It also divides the number of IFFs0 and BIAS for different input data types. The number of IFFs0 is calculated by dividing the number of mantissa bits of the data / the number of mantissa bits of IFFs0 by the integer.

[0069] symbol index mantissa BIAS Number of IFFs0 Number of inputs each time (32 bits) FP32 1 8 23(+1) 127 6 1 FP16 1 5 10(+1) 15 3 2 BF16 1 8 7(+1) 127 2 2 INT16 1 0 15 0 4 2 INT8 1 0 7 0 2 4 INT4 1 0 3 0 1 8

[0070] Among them, sign is the data sign bit (1 bit indicates positive or negative), for example, 1 bit; exponent is the number of exponent bits of the original data type; mantissa is the number of mantissa bits of the original data type, and the hidden bit description is in brackets; BIAS is the exponent offset value used to adjust the actual value of the exponent; the number of IFFs0s is the quotient of the number of original mantissa bits and the number of IFFs0 mantissa bits after the original mantissa is segmented into blocks according to the IFFs0 format; the number of inputs each time is the number of original data that can be input in parallel on the 32-bit bus, which is determined by the total number of data bits.

[0071] For example, the exponent of FP16 is 5 bits, the mantissa of FP16 is 10 bits, +1 indicates a hidden bit, the BIAS of FP16 is 15, the mantissa of FP16 is 10 bits, the quotient is 2.5, rounded up to 3 blocks, the single data of FP16 is 16 bits, and the 32-bit bus can input 2.

[0072] For the calculation of IFFs0, the mantissa of FP32 is 23 bits, and the mantissa of IFFs0 is defined as 4 bits, so the number of segments is 6. The mantissa of INT4 is 3 bits, the number of segments is 1, and zeros are directly padded to 4 bits.

[0073] The actual exponent value is the difference between the stored value and the BIAS. For example, if the BIAS of FP32 is 127 and the stored exponent value is 130, the actual exponent value is 3.

[0074] The number of inputs per time is determined by the total number of data bits and the number of bus bits, which is 32 bits. For example, INT8 single data is 8 bits, and a 32-bit bus can input 4.

[0075] See the table below, which defines three intermediate floating-point formats used in the calculation process: IFFs0, IFFs1, and IFFs2, which are used to unify input data of different precisions.

[0076] symbol index mantissa IFFs0 1 9 4 IFFs1 (complement) x 10 10 IFFs2 (complement) x 10 20

[0077] The index of IFFs0 is calculated by the following formula:

[0078] IFFs0 exp =data exp -bias-offset;

[0079] Among them, data exp is the exponential value of the data itself, bias is the bias of the exponent, and offset is determined by the sequence number of the divided IFFs0.

[0080] For FP32 or BF16 mantissa processing, the most significant bit of the mantissa is padded with 1, and the remaining bits are padded with zeros to a maximum of 4 bits. For FP16 mantissa processing, the most significant bit of the mantissa is padded with 1, and the least significant bit is padded with zeros to a maximum of 4 bits. For fixed-point numbers (such as INT8), the mantissa is padded with zeros in 4-bit segments. IFFs1 and IFFs2 use two's complement to represent the exponent and mantissa, eliminating the need for separate processing of the sign bit during multiplication and accumulation, simplifying hardware design.

[0081] Taking the conversion of FP16 to IFFs0 as an example, the FP16 data is split into a 1-bit sign, a 5-bit exponent, and a 10-bit mantissa. The 10-bit mantissa is split into three IFFs0 blocks (4 bits per block, and zero is added if the value is less than 0). The original exponent value is the quotient of the stored value and the BIAS. The BIAS of FP16 is 15. The offset of each IFFs0 segment is determined by the segment number. For example, the offset of the first segment is 0, the offset of the second segment is 1×4, and the offset of the third segment is 2×4. The final index IFFs0 exp According to the above formula, the format of each IFFs0 block is 1-bit sign, 9-bit exponent (after adjustment) and 4-bit mantissa.

[0082] All input data types are converted to IFFs0 format, enabling reuse of multiplication arrays and adder tree modules, reducing hardware complexity. Extended mantissas in IFFs1 and IFFs2 preserve precision and prevent overflow during computation. Two's complement representation and piecewise zero-padding simplify data alignment logic, adapting to parallel computing architectures.

[0083] In some embodiments, the control register is also used to send the matrix data type signal to the format division module. The csr_atype and csr_btype signals define the original precision of the input matrix, such as FP32 and INT8. The format division module receives the matrix data type signal sent by the control register.

[0084] In some embodiments, based on the input precision of the first input matrix and the second input matrix, the mantissas of the elements of the first input matrix and the second input matrix are segmented by a preset number of bits;

[0085] Generate a first intermediate floating-point format data block and a second intermediate floating-point format data block, wherein the first intermediate floating-point format data block is obtained by appending a sign bit and an exponent offset value of the first input matrix to the segmented data, and the second intermediate floating-point format data block is obtained by appending a sign bit and an exponent offset value of the second input matrix to the segmented data;

[0086] Specifically, taking the first input matrix as INT8 as an example, the mantissa is segmented according to the preset number of bits based on the input precision. For example, the 8-bit mantissa of INT8 is split into two 4-bit segments, and a sign bit and an exponent offset value are added. For example, the sign bit and exponent offset value of INT8 are 0, and the sign bit and exponent offset value of FP16 are 15 to generate a data block.

[0087] Then, grouping is performed on the first intermediate floating-point format data block and the second intermediate floating-point format data block to output input data in an intermediate floating-point format.

[0088] like Figure 2 As shown, there are two types of format division modules, A and B, which serve to divide the input data of the two matrices A and B respectively. The computing unit consists of eight A, B, and format division modules. Each bank is connected to a format division module, and each format division module is connected to a group of eight IFFs0 multipliers.

[0089] Taking FP32 calculations as an example, a single FP32 is divided into six parts. Based on the allocation ratio, 36 multiplication and addition calculations are required to obtain the product of two numbers. Each group of eight multipliers can be divided into five groups. This embodiment groups the numbers a and b based on the closest difference in their sums, with the difference within 2. This ensures that the exponents of the calculation results of each group of IFFs0 multipliers are as close as possible, reducing the phenomenon of partial sums being ignored due to large exponent differences.

[0090] like Figure 3 As shown, the 23-bit FP32 mantissa is split into 6 groups of 4 bits each. a5-a0 corresponds to matrix A, and b5-b0 corresponds to matrix B. 36 multiplication terms (a5b0, a4b0, ..., a0b5) are generated in the order of outer products. The grouping strategy is to use similar sequence numbers. For example, a5b0 and a4b1 are in the same group. Group 1 example: a5b0, a4b1, a3b2; Group 2 example: a5b1, a4b2, a3b3... (a total of 5 groups, the first and fifth groups have 6 terms, and the remaining groups have 8 terms).

[0091] like Figure 4 As shown, the 10-bit mantissa of FP16 is split into 3 groups, a2-a0 corresponds to matrix A, b2-b0 corresponds to matrix B, and 9 multiplication terms (a2b0, a1b0, ..., a0b2) are generated. Example of group 1: a2b0, a1b1, a0b2, example of group 2: a2b1, a1b2, example of group 3: a2b2.

[0092] like Figure 5 As shown, the 7-bit mantissa of BF16 is split into two groups, a1-a0 corresponds to matrix A, and b1-b0 corresponds to matrix B, generating four multiplication terms (a1b0, a0b0, a1b1, a0b1). Group 1 example: a1b0, a0b1, group 2 example: a1b1, a0b0. Adjacent groups are dynamically merged according to the input data to reduce the calculation cycle.

[0093] like Figure 6 As shown, the 16-bit integer of INT16 is split into 4 groups, a3-a0 corresponds to matrix A, b3-b0 corresponds to matrix B, and 16 multiplication terms (a3b0, a2b0, ..., a0b3) are generated. Example of group 1: a3b0, a2b1, a1b2, a0b3, example of group 2: a3b1, a2b2, a1b3, example of group 3: a3b2, a2b3, example of group 4: a3b3.

[0094] like Figure 7 、 8 As shown in the figure, the IFFs0 partitioning order for INT8 or INT4 is shown. Taking INT8 as an example, the 8-bit integer is split into two groups, a1-a0 corresponding to matrix A, and b1-b0 corresponding to matrix B. This generates four multiplication terms: a1b0, a0b0, a1b1, and a0b1. Group 1: a1b0, a0b1; Group 2: a1b1, a0b0.

[0095] INT4: A 4-bit integer is directly grouped as a single unit to generate one multiplication term (a0b0). INT4 does not require grouping and can be calculated in a single cycle, achieving peak throughput.

[0096] The bit width of each input number is 32 bits, so the number of input data is different, and the format division module can be divided according to different types and different numbers of data, such as Figure 3-Figure 8 As shown, for multiplication calculations, the input bandwidth can be maximized. Figure 3-8 This is the division method of floating-point numbers. FP32 inputs one number at a time, eight in a group. FP16 and BF16 input two numbers at a time, four in a group. Fixed-point numbers use a similar scheme. FP16 and BF16 input two numbers at a time, four in a group, corresponding to the input numbers, for a total of eight.

[0097] The multiplication array module performs parallel multiplication operations on intermediate floating-point input data to generate intermediate product results. It also performs parallel multiplication operations on IFFs0 data from the format partitioning module to generate intermediate results in IFFs1 format. The multiplication array module takes as input the IFFs0 data pair of the A and B matrices output by the BPU and outputs the product results in IFFs1 format, including an exponent and a 10-bit mantissa.

[0098] In some embodiments, the multiplication array module includes a second preset number of multiplication array groups multiplied by a second preset number of multiplication array groups, the format division module is connected to a second preset number of multipliers, and the number of the format division modules is a first preset number;

[0099] The multiplication array group is used to perform a multiplication operation on the input data in the intermediate floating-point format within the same group to generate a product result;

[0100] The product result is stored in an extended precision format to generate an intermediate product result, wherein exponent bits and mantissa bits of the intermediate product result are larger than exponent bits and mantissa bits of the first input matrix and the second input matrix.

[0101] The multiplication array module includes 512 parallel multipliers, wherein the second preset number is 8, i.e., 8×8 multiplication array groups. Each group of multipliers receives the IFFs0 data blocks of the same group and stores the intermediate product results with extended precision, such as a 10-bit mantissa and a 10-bit exponent, after performing the multiplication operation.

[0102] Each format division module is connected to 8 multipliers to form an 8×8 multiplication array group; the full computing unit has a total of 16 format division modules, that is, 8 A and B matrices each, 64 in a single layer, and a total of 8 layers, forming a 512 parallel multiplier group.

[0103] The product result is stored in extended precision format (IFFs1), which has wider exponent and mantissa bits than the original IFFs0.

[0104] Before the calculation starts, the external C matrix data is preloaded into the shift register, and the parallel calculation results output by the accumulator module are converted into a serial data stream to adapt to the bandwidth limitation of the storage interface. The loading of the next batch of data is executed synchronously during the multiplication and addition calculation process to avoid pipeline stalls.

[0105] The adder tree module is used to perform exponent alignment and accumulation on the intermediate product results to generate intermediate accumulated values. Specifically, it performs exponent alignment and accumulation on the IFFs1 results output by the multiplication array. For example, it uses a shifter to adjust the mantissa bits to generate intermediate accumulated values ​​in IFFs2 format. The adder tree module inputs the product results in IFFs1 format and outputs the partial sums. The partial sums are in IFFs2 format, with a 10-bit exponent and a 20-bit mantissa.

[0106] like Figure 9 As shown, the accumulator module is used to accumulate intermediate accumulated values, intermediate product results, and the value of the third input matrix to output the accumulated result. Specifically, the accumulator module continuously accumulates the partial sums output by the adder tree module and the value of the third input matrix in multi-cycle calculations until all multiplication and addition operations are completed. The accumulator module inputs the IFFs2 format partial sums of the adder tree, and outputs the final accumulated result.

[0107] The normalization module converts the accumulated result into a normalized result. The normalization module converts the high-precision intermediate result output by the accumulator module into the target output format, such as FP32 or INT32. The input of the normalization module is the final result of the accumulator, and the output is the normalized result of the target output format.

[0108] In some embodiments, the computing unit further includes a shift register module, the shift register module being configured to serially load the third input matrix and convert the parallel received normalized results and the output matrix into serial data.

[0109] The shift register module is a C / D shift register. During the input phase, the external input C matrix, i.e., the third input matrix, is serially loaded into the C / D shift register. During the accumulation phase, it is added to the accumulated result. During the output phase, the normalized D matrix is ​​converted from parallel data to a serial stream to adapt to limited bandwidth transmission.

[0110] The buffer port width of the shift register module is limited and can only output 8x32b data (8 banks) at a time, but the accumulator requires 8x8x32 bits of data, so the serial-to-parallel method is used. Figure 10 As shown, in some embodiments, the shift register module further includes a shift register array, a first buffer manager, and a second buffer manager.

[0111] The first cache manager is used to receive a third input matrix C and send the third input matrix to the shift register array, wherein the shift register array is composed of registers in multiple rows and columns, that is, an 8×8 array, and each register stores 32 bits of data.

[0112] The shift register array is used to group the third input matrix and store the grouped third input matrix in an input data storage area. The input data storage area is used to store the grouped data of the third input matrix and occupies the first half of the columns of the shift register array, for example, columns 0 to 7.

[0113] The third input matrix is ​​serially input into the shift register. The shift register contains 8×8 values, corresponding to a single-layer multiplication array. For example, one layer has 64 multiplier nodes, and eight layers have 8×64 nodes. Each of the 8 nodes corresponds to an accumulator, for a total of 64 accumulators. The 64 numbers in the C matrix correspond to the 64 accumulator nodes. After accumulation, the values ​​are normalized and replaced with the shift register values, i.e., the values ​​in the D matrix, before being output.

[0114] The shift register array is further configured to receive the C matrix and the D matrix obtained by normalizing and accumulating the normalized results by the normalization module, and output the D matrix to a corresponding address in the memory under the control of the second cache manager.

[0115] The second cache manager receives the normalized results, converts them into an output matrix, and outputs them serially to external storage through a parallel interface in a time-sharing manner, converting the internal parallel data (8×32b) into a serial stream (1×32b / cycle) to match the external bus bandwidth limitation.

[0116] During the calculation process, the input of the C matrix and the output of the D matrix are completed. That is, 8 cycles are consumed to read the C matrix during the format division, multiplication calculation, and addition tree accumulation process. After the final accumulation and normalization are completed, the D matrix data will replace the C matrix. During the serial output of the D matrix, the C matrix is ​​simultaneously serially input.

[0117] Moreover, there are a total of 8 rows and 8 columns, and each serial input and output is 1 column.

[0118] Exemplarily, the first cache manager receives C matrix data from the external storage interface and loads it into the shift register array in parallel bit width of 8×32b. The input is Creg[31:0] (32-bit data bus), with a total of 8 parallel channels (corresponding to 8 BANKs). The second cache manager receives the serial data stream Results[31:0] output by the shift register and stores it into the external memory in the target format.

[0119] The format division module is responsible for converting the input matrix into an intermediate floating-point format and grouping it according to a dynamic strategy. For the grouping rule, the format division module obtains a first sequence number and a second sequence number based on the order of generating the first intermediate floating-point format data block and the second intermediate floating-point format data block. The first sequence number is the sequence number of the first intermediate floating-point format data block, and the second sequence number is the sequence number of the second intermediate floating-point format data block.

[0120] combining the first intermediate floating-point format data block and the second intermediate floating-point format data block to generate a product term, wherein the product term is an arithmetic sum of the first sequence number and the second sequence number;

[0121] The first intermediate floating point format data block and the second intermediate floating point format data block are grouped according to the product terms to output input data in an intermediate floating point format, wherein the product terms of the input data in the same group are less than a preset threshold.

[0122] Each IFFs0 data block of the first input matrix A, i.e., the first intermediate floating-point format data block, is assigned a first serial number, such as i = 0, 1, ..., 5 corresponding to the 6 groups of FP32. Each IFFs0 data block of the second input matrix B, i.e., the second intermediate floating-point format data block, is assigned a second serial number, such as j = 0, 1, ..., 5.

[0123] When combining data blocks A and B, the arithmetic sum (i + j) of their sequence numbers is calculated, and the absolute value of the sequence number difference (Δ = |ij|) is calculated to determine whether they are in the same group. A preset threshold, such as 2, Δ ≤ 2, is used to allow only data blocks with sequence number differences within the threshold to be combined into the same group. Data blocks with similar sequence numbers have small exponential differences, and the mantissa shift during accumulation is controllable, reducing precision loss, limiting cross-group calculations, and reducing the complexity of the addition tree.

[0124] For example, if A's sequence number i=3 and B's sequence number j=4, then Δ=1, and the same group calculation is allowed. If A's sequence number i=5 and B's sequence number j=2, then Δ=3, exceeding the threshold, and is excluded.

[0125] Based on the principle of sequence number and closest difference, the exponent difference within the same group is minimized, that is, ≤2, to reduce the mantissa alignment error.

[0126] In low-precision mode, the format division module optimizes the computational efficiency by ignoring the last grouping. The last grouping is the segmentation corresponding to the least significant bit of the mantissa in each group. The control register sends csr_is_low=1 to activate the low-precision mode. In low-precision mode, the format division module will ignore the last grouping and perform the division of the next pair of data.

[0127] The format division module is also used to ignore the target mantissa segments in the first input matrix and the second input matrix according to the low-precision mode signal sent by the control register, and convert the non-target mantissa segments into intermediate floating-point format data blocks, wherein the target mantissa segment is the smallest mantissa segment in the target group, wherein the target group is the last group.

[0128] The multiplier input of the last group is skipped and does not participate in the current calculation cycle. The freed hardware resources are dynamically allocated to the valid groups. The discard of the last group results in an error of ≤ 1e-11, which is corrected by the compensation logic of the normalization module.

[0129] For example, 5 groups are originally required in FP32 mode, each group can calculate up to 8, and 4 groups are required in low-precision mode, and the 5th group is skipped.

[0130] Taking FP32 as an example, retain groups 1 to 4 and complete the calculation of the first four groups, covering most of the significant mantissas. Discard group 5 and directly skip low-weight items such as a0b5, a1b4, and a2b3.

[0131] This application converts data of different widths into a unified format through IFFs0 division, and then all subsequent hardware units can be reused to perform multiplication and accumulation calculations. The format division module consumes very little additional resources and only occupies 1.2% of the entire design area under TSMC 28nm process. In addition, this application also maximizes the use of input bandwidth, maximizes the utilization rate of computing resources and throughput. In particular, in low-precision computing mode, all computing resources can be utilized, and one level of pipeline can be reduced to improve throughput, and the precision loss is very small, only on the order of 1e-11.

[0132] In the accumulation phase, in some embodiments, the accumulator module includes an accumulation register psum-reg, and the accumulator module is further configured to cyclically accumulate the intermediate accumulation values ​​into the accumulation register and generate an accumulation completion signal;

[0133] The normalization module is used to receive the accumulation completion signal and convert the intermediate accumulation value into a normalized result in a target format.

[0134] The results of the addition tree module are input into the accumulator and continuously accumulated in psum-reg. When the input data is traversed sequentially and the calculation is completed, it is normalized and output. Before the first calculation, it is passed to the shift register array for storage. After this, psum-reg will be added in a loop, and after calculation, it will be output to the normalization module for IEEE754 standard conversion, and then the result will be stored in the shift register module. At the same time, the normalization module will generate a signal called acc finish to activate the D cache to store the results from the shift register. The normalization module can meet different precision conversions. The accumulation precision and output precision are set by the control register to meet the precision requirements in different scenarios.

[0135] In the parallel loading phase, the C matrix data is loaded from the first cache manager to the register array in parallel, with each BANK corresponding to a row. Figure 10 , outputting one column at a time. Except for the first calculation, which directly loads the C matrix, the rest of the time, the process of serially outputting the D matrix also synchronously inputs the C matrix serially. That is, there is no process without additional sequential input.

[0136] The accumulator module accumulates the result of the multiplication array and the addition tree (A×B) with the C matrix data to generate the final result D. While outputting the D matrix, the next batch of C matrix data is loaded into the idle accumulation registers through the first cache manager, achieving pipeline parallelism of calculation and data transfer.

[0137] The multiply-add unit provided in this embodiment supports input matrices of the same or different precision, and the output matrix also supports multiple precisions, such as Float32, Float16, BFloat16, and INT32. This highly versatile unit is suitable for multi-precision calculations in various scenarios. Furthermore, this embodiment offers high hardware resource utilization and throughput, and can further improve throughput by increasing tolerance to precision loss. This significantly accelerates both training and inference of neural network models that rely on matrix multiplication and addition operations.

[0138] Based on the above-mentioned mixed data precision matrix multiply-add unit, some embodiments of the present application provide a calculation method of the mixed data precision matrix multiply-add unit, including:

[0139] Converting the first input matrix and the second input matrix into input data in an intermediate floating point format;

[0140] performing a parallel multiplication operation on the input data in the intermediate floating point format to generate an intermediate product result;

[0141] performing exponent alignment and accumulation on the intermediate product results to generate an intermediate accumulation value;

[0142] Accumulating the intermediate product results and the value of the third input matrix in the form of accumulating the intermediate accumulated values ​​to output an accumulated result;

[0143] The accumulated result is converted into an output matrix.

[0144] The effects of the above method embodiment during operation can be found in the effects of the above multiplication and addition unit embodiment, which will not be described in detail here.

[0145] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without expending creative work shall fall within the scope of protection of this application.

Claims

1. A mixed data precision matrix multiply-add unit, characterized in that: include: Computing unit, control register, cache unit and cache management unit; The calculation unit includes: a format division module, configured to convert the first input matrix and the second input matrix into input data in an intermediate floating point format; a multiplication array module, configured to perform parallel multiplication operations on the input data in the intermediate floating point format to generate intermediate product results; an addition tree module, configured to perform exponent alignment and accumulation on the intermediate product results to generate an intermediate accumulated value; an accumulator module, configured to accumulate the intermediate product result and the value of the third input matrix in the form of the intermediate accumulated value, so as to output an accumulated result; The normalization module is used to convert the accumulated result into a normalized result.

2. The mixed data precision matrix multiply-add unit according to claim 1, characterized in that: The control register is used to receive control signals from the central processing unit, wherein the control signals include a matrix data type signal, a quantization type signal, a low-precision mode signal, and a signal for setting matrix dimension; The buffer unit includes a first preset number of buffers for caching the first input matrix, the second input matrix, the third input matrix and the normalized result; The buffer management unit is configured to output the first input matrix and the second input matrix to the calculation unit in sequence according to the control signal.

3. The mixed data precision matrix multiply-add unit according to claim 2, characterized in that: Also included is a shift register module; The shift register module is used to serially load the third input matrix; and receive the normalized results in parallel and convert the normalized results into serial data; The control register is also used to send the matrix data type signal and the low-precision mode signal to the format division module, send the quantization type signal to the shift register module, and send the set matrix dimension signal to the buffer management unit and the shift register module.

4. The mixed data precision matrix multiply-add unit according to claim 1, characterized in that: The accumulator module includes an accumulation register; The accumulation register is further configured to read the third input matrix, accumulate the intermediate product result and the value of the third input matrix in the form of the intermediate accumulated value, and generate an accumulation completion signal; The normalization module is used to receive the accumulation completion signal and convert the intermediate accumulation value into a normalized result in a target format.

5. The mixed data precision matrix multiply-add unit according to claim 1, characterized in that: The format division module is further configured to receive a matrix data type signal sent by the control register, and segment the mantissas of the elements of the first input matrix and the second input matrix according to a preset number of bits based on the input precision of the first input matrix and the second input matrix; Generate a first intermediate floating-point format data block and a second intermediate floating-point format data block, wherein the first intermediate floating-point format data block is obtained by appending a sign bit and an exponent offset value of the first input matrix to the segmented data, and the second intermediate floating-point format data block is obtained by appending a sign bit and an exponent offset value of the second input matrix to the segmented data; Grouping is performed on the first intermediate floating-point format data block and the second intermediate floating-point format data block to output input data in an intermediate floating-point format.

6. The mixed data precision matrix multiply-add unit according to claim 5, characterized in that: Also included is a shift register module, wherein the shift register module further includes a shift register array, a first buffer manager, and a second buffer manager; The first buffer manager is used to receive a third input matrix and send the third input matrix to the shift register array; The shift register array is further configured to receive the normalized result and write the result into the second cache manager; The second cache manager is configured to read the normalized result from and generate an output matrix, and output the output matrix to an external storage unit.

7. The mixed data precision matrix multiply-add unit according to claim 5, characterized in that: The format division module is also used for: Obtaining a first sequence number and a second sequence number based on an order in which the first intermediate floating-point format data block and the second intermediate floating-point format data block are generated, wherein the first sequence number is the sequence number of the first intermediate floating-point format data block, and the second sequence number is the sequence number of the second intermediate floating-point format data block; combining the first intermediate floating point format data block and the second intermediate floating point format data block to generate a product term, wherein the product term is input into the multiplication array module; The first intermediate floating point format data block and the second intermediate floating point format data block are grouped according to the product terms to output input data in an intermediate floating point format, wherein the product terms of the input data in the same group are less than a preset threshold.

8. The mixed data precision matrix multiply-add unit according to claim 1, characterized in that: The format division module is also used to ignore the target mantissa segments in the first input matrix and the second input matrix according to the low-precision mode signal sent by the control register, and convert the non-target mantissa segments into intermediate floating-point format data blocks, wherein the target mantissa segment is the smallest mantissa segment in the target group.

9. The mixed data precision matrix multiply-add unit according to claim 1, characterized in that: The multiplication array module includes a second preset number of multiplication array groups multiplied by a second preset number, the format division module is connected to a second preset number of multipliers, and the number of the format division modules is the first preset number; The multiplication array group is used to perform a multiplication operation on the input data in the intermediate floating-point format within the same group to generate a product result; The product result is stored in an extended precision format to generate an intermediate product result, wherein exponent bits and mantissa bits of the intermediate product result are larger than exponent bits and mantissa bits of the first input matrix and the second input matrix.

10. A method for calculating a mixed data precision matrix multiplication and addition unit, characterized in that: include: Converting the first input matrix and the second input matrix into input data in an intermediate floating point format; performing a parallel multiplication operation on the input data in the intermediate floating point format to generate an intermediate product result; performing exponent alignment and accumulation on the intermediate product results to generate an intermediate accumulation value; Accumulating the intermediate product result and the value of the third input matrix in the form of the intermediate accumulated value to output an accumulated result; The accumulated result is converted into a normalized result.

Images