Configurable BF16 format matrix multiplication data path implementation method, components and device

By configuring the data path and selecting the appropriate calculation granularity, splitting the matrix multiplication operation, and optimizing the pipeline interval using the 16-bit floating point multiplication module and the 32/64-bit floating point addition module, the problem of decreasing the calculation bandwidth of the BF16 format matrix multiplication data path is solved. , improves hardware design efficiency and system performance, and supports multiple execution granularities.

CN113961878BActive Publication Date: 2024-12-06NAT UNIV OF DEFENSE TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111241122.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-25
Publication Date
2024-12-06
Estimated Expiration
2041-10-25

AI Technical Summary

Technical Problem

When designing the BF16 format matrix multiplication data path, the calculation bandwidth drops to half of the ideal design, which cannot effectively improve hardware design efficiency and system performance, and cannot support the execution granularity of multiple data.

Method used

By configuring the data path, selecting the appropriate calculation granularity and module combination, the slicing matrix multiplication operation is a micro operation, and the multiplication and accumulation or point multiplication operation is completed through the 16-bit floating point multiplication module and the 32/64-bit floating point addition module to optimize the pipeline intervals to improve computational efficiency.

Benefits of technology

It improves the hardware design efficiency of BF16 design, provides higher system performance, supports the execution granularity of multiple data, reduces the cost of transplanting time between design solutions, and achieves a balance between front-end instructions and back-end data processing capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113961878B_ABST
    Figure CN113961878B_ABST
Patent Text Reader

Abstract

The present invention discloses a configurable BF16 format matrix multiplication data path implementation method, component and device. The configurable BF16 format matrix multiplication data path implementation method of the present invention includes: performing data path configuration, including the number of data paths C, the configuration of the data width V supported by a single data path; selecting a data path, selecting the calculation granularity supported by the calculation component in the processor in the data path, dividing the original matrix multiplication operation into one or more micro-operations, and executing them according to the set pipeline interval of continuous micro-operations. The present invention provides a flexible BF16 format matrix multiplication data path implementation method based on the existing typical data execution path, which can improve the hardware design efficiency of the BF16 design, provide higher system performance, and support the execution granularity of multiple data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a microprocessor architecture and an execution unit design during microprocessor design, and in particular to a configurable BF16 format matrix multiplication data path implementation method, unit and device. Background Art

[0002] Artificial intelligence applications have developed rapidly in recent years and are widely used in fields such as image recognition, target recognition, and natural language processing. Taking the image recognition field as an example, the image recognition error rate of the champion of the 2016 IMAGENET competition is lower than that of humans. The field of artificial intelligence has attracted the attention of both academia and industry and is rapidly moving towards the application stage. Artificial intelligence applications are becoming one of the representatives of emerging applications. Its applications also show a strong demand for computing and storage computing capabilities. Considering that such applications have certain fault tolerance capabilities and the data types used in their operations are different from traditional applications, Google proposed the BF16 format to replace fp16 or fp32. The BF16 format can maintain the same data length as fp16 while increasing the representation range of the data itself and improving the accuracy of the application.

[0003] The data length of the BF16 format is 16 bits, of which the sign bit occupies 1 bit, the exponent occupies 8 bits, and the mantissa occupies 7 bits, which has the same exponent length as fp32. Instructions based on BF16 mainly serve artificial intelligence applications, and computing instructions mainly include two operations: dot multiplication and matrix multiplication. The source operands of the matrix multiplication instruction are three matrices, and their matrix sizes are 2x4, 4x2, and 2x2 respectively. The matrix data is saved in the processor in the form of vectors, and the data length of the basic vector is 128 bits. Considering the overlap between the data encoding method of BF16 and FP32, the data path design of the BF16 matrix multiplication operation can naturally rely on the data path of the original fp32 data format, but the effective computing bandwidth of its data execution path depends on fp32. Considering that the width of BF16 is half of that of fp32, the computing bandwidth of the above scheme will drop to half of the ideal design. Summary of the invention

[0004] The technical problem to be solved by the present invention is as follows: In view of the above-mentioned problems in the prior art, a configurable BF16 format matrix multiplication data path implementation method, components and devices are provided. The present invention provides a flexible BF16 format matrix multiplication data path implementation method based on the existing typical data execution path, which can improve the hardware design efficiency of the BF16 design, provide higher system performance, and support the execution granularity of multiple data.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A configurable BF16 format matrix multiplication data path implementation method, comprising:

[0007] 1) Perform data path configuration, including the number of data paths C and the data width V supported by a single data path;

[0008] 2) Select a data path, select the calculation granularity supported by the calculation unit in the processor in the data path, divide the original matrix multiplication operation into one or more micro-operations, and execute them according to the set pipeline interval of continuous micro-operations.

[0009] Optionally, step 2) includes:

[0010] A2.1) Select a data path PATH i ;

[0011] A2.2) Activate the data path PATH i 16-bit floating-point multiplication module fmul16 and 32-bit floating-point addition module fadd32;

[0012] A2.3) In this data path, the multiplication-accumulation operation is used as the calculation granularity, and the original matrix multiplication operation is divided into four micro-operations based on the multiplication-accumulation operation. Each micro-operation is passed through the data path PATH i The 16-bit floating-point multiplication module fmul16 and the 32-bit floating-point addition module fadd32 complete the multiplication and accumulation operations, and the pipeline interval of continuous micro-operations is 3 beats.

[0013] Optionally, the input first source operand opa and the second source operand opb of the original matrix multiplication operation are two sets of matrix data, one set of matrix data has a size of 2×4, and the other set of matrix data has a size of 4×2, and the output destination operand res is a matrix data with a size of 2×2; in step A2.3), each micro-operation is passed through the data path PATH i The 16-bit floating-point multiplication module fmul16 and the 32-bit floating-point addition module fadd32 complete the multiplication and accumulation operation, including: performing three levels of nested loop traversal for n, j, and k in sequence, and calculating the element with index j in the subvector with index n of the destination operand according to the following formula in the third level of loop traversal:

[0014] res[n][j]+=opa[n][k]*opb[j][k]

[0015] In the above formula, res[n][j] represents the element with index j in the subvector with index n of the destination operand res with an element length of 32 bits, "+=" represents an accumulation operation, opa[n][k] represents the element with index k in the subvector with index n of the first source operand opa with an element length of 16 bits, opb[j][k] represents the element with index k in the subvector with index j of the second source operand opb with an element length of 16 bits, where n and j are in the range of [0,2), k is in the range of [0,4), n is the row number of the first source operand opa corresponding to the operation element, j is the row number of the second source operand opb corresponding to the operation element, and k is the matrix column number corresponding to the operation element.

[0016] Optionally, step 2) includes:

[0017] B2.1) Select a data path PATH i ;

[0018] B2.2) Activate the data path PATH i The 16-bit floating-point multiplication module fmul16, the 32-bit floating-point addition module fadd32 and the 64-bit floating-point addition module fadd64, wherein the 64-bit floating-point addition module fadd64 supports two-way parallel 32-bit floating-point addition operations;

[0019] B2.3) In this data path, the multiplication and accumulation operation is used as the calculation granularity, and the original matrix multiplication operation is divided into two micro-operations based on the dot multiplication operation. Each micro-operation is passed through the data path PATH i The 16-bit floating-point multiplication module fmul16, the 32-bit floating-point addition module fadd32 and the 64-bit floating-point addition module fadd64 complete the point multiplication operation, and the pipeline interval of continuous micro-operations is 3 beats.

[0020] Optionally, the input first source operand opa and the second source operand opb of the original matrix multiplication operation are two sets of matrix data, one set of matrix data has a size of 2×4, and the other set of matrix data has a size of 4×2, and the output destination operand res is a matrix data with a size of 2×2; in step B2.3), each micro-operation is passed through the data path PATH i The 16-bit floating-point multiplication module fmul16, the 32-bit floating-point addition module fadd32 and the 64-bit floating-point addition module fadd64 complete the dot multiplication operation including: performing three levels of nested loop traversal for n, j and k in sequence, and calculating the element with index j in the subvector with index n of the destination operand according to the following formula in the third level of loop traversal:

[0021] res[n][j]+=opa[n][2k]*opb[j][2k]+opa[n][2k+1]*opb[j][2k+1]

[0022] In the above formula, res[n][j] represents the element with index j in the subvector with index n of the destination operand res with element length of 32 bits, "+=" represents an accumulation operation, opa[n][2k] represents the element with index 2k in the subvector with index n of the first source operand opa with element length of 16 bits, opa[n][2k+1] represents the element with index 2k+1 in the subvector with index n of the first source operand opa with element length of 16 bits, opb[j][2 k] represents the element with index 2k in the subvector with index j of the second source operand opb with an element length of 16 bits, and opb[j][2k+1] represents the element with index 2k+1 in the subvector with index j of the second source operand opb with an element length of 16 bits, where n, j, k are in the range of [0,2), n is the row number of the first source operand opa corresponding to the operation element, j is the row number of the second source operand opb corresponding to the operation element, and k is the matrix column number corresponding to the operation element.

[0023] Optionally, step 2) includes:

[0024] C2.1) Select two data paths;

[0025] C2.2) activating the 16-bit floating-point multiplication module fmul16, the 32-bit floating-point addition module fadd32 and the 64-bit floating-point addition module fadd64 of the two data paths, wherein the 64-bit floating-point addition module fadd64 supports two-way parallel 32-bit floating-point addition operations;

[0026] C2.3) In the data path, matrix multiplication operation is used as the calculation granularity, and the original matrix multiplication operation is converted into one micro-operation based on matrix multiplication, and the pipeline interval of continuous micro-operations is 1 beat, and the input includes 8 source registers and destination registers; for each data path, the 8 input data are input into the 16-bit floating-point multiplication module fmul16 of the data path in pairs, and the multiplication result is first multiplied with the destination register in the 32-bit floating-point addition module fadd32 to obtain the multiplication result, and then the multiplication result is sent to the 64-bit floating-point addition module fadd64 of the data path or another data path, and is accumulated with the multiplication result of another data path in the 64-bit floating-point addition module fadd64 to obtain the final result.

[0027] Optionally, the input first source operand opa and the second source operand opb of the original matrix multiplication operation are two groups of matrix data, one group of matrix data has a size of 2×4, and the other group of matrix data has a size of 4×2, and the output destination operand res is a matrix data of size 2×2; in step C2.3), the 8 input data are input two by two into the 16-bit floating-point multiplication module fmul16 of the data path, the multiplication result is first multiplied with the destination register in the 32-bit floating-point addition module fadd32 to obtain the multiplication result, and then the multiplication result is sent to the 64-bit floating-point addition module fadd64 of the data path or another data path, and the multiplication result is accumulated with the multiplication result of another data path in the 64-bit floating-point addition module fadd64 to obtain the final result, including: performing two levels of nested loop traversal for n and j in turn, and calculating the element with index j in the subvector with index n of the destination operand according to the following formula in the second level of loop traversal:

[0028] res[n][j]+=opa[n][0]*opb[j][0]+opa[n][1]*opb[j][1]+opa[n][2]*opb[j][2]+opa[n][3]*opb[j][3]

[0029] In the above formula, res[n][j] represents the element with index j in the subvector with index n of the destination operand res with an element length of 32 bits, "+=" represents an accumulation operation, opa[n][0] represents the element with index 0 in the subvector with index n of the first source operand opa with an element length of 16 bits, opb[j][0] represents the element with index 0 in the subvector with index j of the second source operand opb with an element length of 16 bits, opa[n][1] represents the element with index 1 in the subvector with index n of the first source operand opa with an element length of 16 bits, and opb[j][1] represents the element length of 1. The 6-bit second source operand opb has an index of element 1 in the subvector indexed by j, opa[n][2] represents the element indexed by 2 in the subvector indexed by n of the first source operand opa with an element length of 16 bits, opb[j][2] represents the element indexed by 2 in the subvector indexed by j of the second source operand opb with an element length of 16 bits, opa[n][3] represents the element indexed by 3 in the subvector indexed by n of the first source operand opa with an element length of 16 bits, and opb[j][3] represents the element indexed by 3 in the subvector indexed by j of the second source operand opb with an element length of 16 bits.

[0030] The present invention also provides a configurable BF16 format matrix multiplication data path implementation component, comprising a control unit, multiple data paths and a configuration register for configuring a selected data path and its data width V, wherein the control unit is respectively connected to the multiple data paths and the configuration register, each data path comprises a 16-bit floating-point multiplication module fmul16, a 32-bit floating-point addition module fadd32 and a 64-bit floating-point addition module fadd64, and the number of the 64-bit floating-point addition modules fadd64 is twice the number of the 32-bit floating-point addition modules fadd32, and the number of the 32-bit floating-point addition modules fadd32 is twice the number of the 16-bit floating-point multiplication modules fmul16, and the control unit is programmed or configured to execute the steps of the configurable BF16 format matrix multiplication data path implementation method.

[0031] In addition, the present invention also provides a computer device, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the steps of the configurable BF16 format matrix multiplication data path implementation method.

[0032] In addition, the present invention also provides a computer-readable storage medium, in which a computer program is stored. The computer program is a computer program programmed or configured to execute the configurable BF16 format matrix multiplication data path implementation method.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] 1. The configurable BF16 format matrix multiplication data path implementation method of the present invention can be designed for the BF16 format matrix multiplication data path, and according to the characteristics of the existing functional module execution path, a variety of design schemes with different data execution widths are provided, and a hardware upgrade scheme for the corresponding data path is provided;

[0035] 2. The configurable BF16 format matrix multiplication data path implementation method of the present invention clarifies the hardware performance corresponding to different design schemes, can serve the design of microprocessors with different front-end instruction emission capabilities, and reduce the time cost of transplantation between spectrum design schemes.

[0036] 3. The BF16 format is a storage format that cannot be used independently and must rely on the FP32 format. The characteristics of the BF16 format itself determine that its corresponding hardware design is an incremental design. On the other hand, a hidden requirement of the microprocessor architecture is the balance between the front-end instruction dispatch capability and the back-end data processing capability. In addition to providing a fully functional design solution for the BF16 data format, the configurable BF16 format matrix multiplication data path implementation method provides a corresponding back-end data path design solution for microprocessor architectures with different instruction dispatch capabilities, indirectly decoupling the front-end and back-end limitations of the original design. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The figure is a basic flow chart of a method according to a first embodiment of the present invention.

[0038] Figure 2 It is a schematic diagram of activating the computing components in the data path in the first embodiment of the present invention.

[0039] Figure 3 It is a schematic diagram of activating the computing components in the data path in the second embodiment of the present invention.

[0040] Figure 4 It is a schematic diagram of activating the computing components in the data path in the third embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention is further described in detail below through specific implementation cases and drawings. The specific implementation cases described here are only used to explain the present invention but not to limit the present invention.

[0042] Embodiment 1:

[0043] like Figure 1 As shown, the configurable BF16 format matrix multiplication data path implementation method of this embodiment includes:

[0044] 1) Perform data path configuration, including the number of data paths C and the data width V supported by a single data path;

[0045] 2) Select a data path, select the calculation granularity supported by the calculation unit in the processor in the data path, divide the original matrix multiplication operation into one or more micro-operations, and execute them according to the set pipeline interval of continuous micro-operations.

[0046] In this embodiment, when the data path is configured in step 1), based on the configuration of the number of data paths C and the data width V supported by a single data path, the total data width supported can be C*V. The restrictions on the configuration parameters C and V are as follows: the number of data paths C is greater than or equal to 2 and is an even number, and the data width V supported by a single data path is greater than or equal to 128 bits. For the convenience of description, it is agreed that the data width of the data execution path is 128 bits, the number of data paths is 2, the floating-point multiplication operation is 3 beats, and the floating-point addition operation is 2 beats.

[0047] According to the matrix multiplication operation, the elements of any output matrix are the result of the accumulation of 5 elements, four of which are multiplication results. In the pipeline, it is assumed that the timing of the floating-point multiplication operation is 3 beats and the timing of the floating-point addition operation is 3 beats. Taking into account the current design ideas of processor instructions, instructions can be subdivided into multiple micro-operations in the processor, and their specific implementations vary depending on the calculation granularity of the data. This embodiment and subsequent implementations will propose 3 examples of data path implementations that support BF16 format matrix multiplication, and give the corresponding hardware execution efficiency and pipeline gap.

[0048] In this embodiment, step 2) includes:

[0049] A2.1) Select a data path PATH i ;

[0050] A2.2) Activate the data path PATH i The 16-bit floating-point multiplication module fmul16 and the 32-bit floating-point addition module fadd32, such as Figure 2 As shown in the dashed box, 16 in the figure represents a 16-bit floating-point multiplication module fmul1, and 32 represents a 32-bit floating-point addition module fadd32;

[0051] A2.3) In this data path, the multiplication-accumulation operation is used as the calculation granularity, and the original matrix multiplication operation is divided into four micro-operations based on the multiplication-accumulation operation. Each micro-operation is passed through the data path PATH i The 16-bit floating-point multiplication module fmul16 and the 32-bit floating-point addition module fadd32 complete the multiplication and accumulation operations, and the pipeline interval of continuous micro-operations is 3 beats.

[0052] Referring to steps A2.1) to A2.3), it can be seen that step 2) of this embodiment divides the original matrix multiplication operation into calculation granularity based on the multiplication-accumulation operation (FMUL-FADD). Each matrix multiplication operation can be divided into 4 micro-operations, and the pipeline interval of consecutive micro-operations is 3 beats (FMUL-FADD data bypass), so as to ensure the validity of the source operands of the multiplication-accumulation operation.

[0053] The BF16 matrix multiplication has three source operands, namely: two source operands with an element length of 16 bits (opa and opb) and one source operand with an element length of 32 bits (acc); the destination operand is a destination operand with an element length of 32 bits (res). The source operands Opa and Opb are 64 bits wide, and each bit is divided into two sub-vectors, each sub-vector contains four elements. For the convenience of description, the sub-vector label corresponding to the source operand is the source operand label followed by a relative serial number, such as the first element of the first sub-vector of Opa is Opa[0][0]. The data execution path in this embodiment is as follows Figure 2 As shown in the figure, the data path execution width is 128 bits, and the number of data paths is two. Considering that the modules in the data path have two attributes, namely, computing function and supported data bit width, Figure 2 The identifier only lists the supported data bit widths. For example, the module marked as 16 in PATH0 represents a functional module that supports 16-bit data width, including addition and multiplication components.

[0054] In this embodiment, the input first source operand opa and the second source operand opb of the original matrix multiplication operation are two sets of matrix data, one set of matrix data has a size of 2×4, and the other set of matrix data has a size of 4×2, and the output destination operand res is a matrix data with a size of 2×2; in step A2.3), each micro-operation is passed through the data path PATH i The 16-bit floating-point multiplication module fmul16 and the 32-bit floating-point addition module fadd32 complete the multiplication and accumulation operation, including: performing three levels of nested loop traversal for n, j, and k in sequence, and calculating the element with index j in the subvector with index n of the destination operand according to the following formula in the third level of loop traversal:

[0055] res[n][j]+=opa[n][k]*opb[j][k]

[0056] In the above formula, res[n][j] represents the element with index j in the sub-vector with index n of the destination operand res with element length of 32 bits, "+=" represents the accumulation operation, opa[n][k] represents the element with index k in the sub-vector with index n of the first source operand opa with element length of 16 bits, opb[j][k] represents the element with index k in the sub-vector with index j of the second source operand opb with element length of 16 bits, wherein n, j ranges from [0,2), k ranges from [0,4), n is the row number of the first source operand opa corresponding to the operation element, j is the row number of the second source operand opb corresponding to the operation element, and k is the matrix column number corresponding to the operation element. In this embodiment, three levels of nested loop traversal are performed for n, j, and k in sequence, and the pseudo code for calculating the element with index j in the sub-vector with index n of the destination operand according to the following formula in the third level of loop traversal is as follows:

[0057]

[0058] It should be noted that the order of the three-level nesting is not limited to the above example.

[0059] Taking the matrix element res[0][0] as an example, the granularity in this embodiment is the "opa[0][0]*opb[0][0]" level in the matrix element, and the matrix multiplication operation is split into 4 micro-operations, each of which completes the multiplication and accumulation operation. The BF16 multiplication operation generates the corresponding 32-bit floating point number fp32. Considering that the effective digits after the BF16 mantissa multiplication are still smaller than the mantissa of the 32-bit floating point number fp32, the result of the BF16 multiplication can be directly given to the normal data channel (non-fused instruction channel) of the 32-bit floating point number fp32.

[0060] The pipeline of the data channel corresponding to this embodiment is shown in Table 1.

[0061] Table 1: Pipeline of data channel corresponding to Example 1.

[0062] I1 I2 I3 M1 M2 M3 A1 A2 A3 W1 I1 I2 I3 M1 M2 M3 A1 A2 A3 W1

[0063] In the above table, I1~I3 represent the instruction issuance stage, M1~M3 represent the floating-point multiplication unit execution stage, A1~A3 represent the floating-point addition unit execution stage, and W1 represents the data write-back stage. Due to the data dependency of continuous accumulation operations in micro-operations, the micro-operation pipeline formed by this design has bubbles, and there are at least two beats between continuous micro-operations.

[0064] The hardware resources of the execution path naturally meet the requirements for the implementation of the method of this embodiment. During the matrix multiplication operation, each 64-bit data width includes a 64-bit wide execution unit, two 32-bit wide execution units and four 16-bit wide execution units. The calculation efficiency of the hardware resources according to the effective calculation data width is (32+16)*V / 32 / V / 3=50%, and the matrix multiplication calculation throughput of the corresponding structure (the average ratio of effective calculation data to ideal calculation data per beat) is C*V / 4 / 3 / (C*V)=1 / 12.

[0065] In addition, this embodiment also provides a configurable BF16 format matrix multiplication data path implementation component, including a control unit, multiple data paths, and a configuration register for configuring a selected data path and its data width V, the control unit is respectively connected to the multiple data paths and the configuration register, each data path includes a 16-bit floating-point multiplication module fmul16, a 32-bit floating-point addition module fadd32, and a 64-bit floating-point addition module fadd64, and the number of 64-bit floating-point addition modules fadd64 is twice the number of 32-bit floating-point addition modules fadd32, and the number of 32-bit floating-point addition modules fadd32 is twice the number of 16-bit floating-point multiplication modules fmul16, the control unit is programmed or configured to execute the steps of the aforementioned configurable BF16 format matrix multiplication data path implementation method.

[0066] In addition, this embodiment also provides a computer device, including a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the steps of the aforementioned configurable BF16 format matrix multiplication data path implementation method.

[0067] In addition, this embodiment also provides a computer-readable storage medium, in which a computer program is stored. The computer program is a computer program programmed or configured to execute the aforementioned configurable BF16 format matrix multiplication data path implementation method.

[0068] Embodiment 2:

[0069] This embodiment is basically the same as the first embodiment, and the main difference is that the calculation granularity supported by the calculation unit in the processor is different. In this embodiment, the original matrix multiplication operation is divided into two micro-operations with the point multiplication operation as the calculation granularity. Each matrix multiplication operation can be divided into two micro-operations, and the pipeline interval of continuous micro-operations is three beats (data dependency is completed through the data bypass from the floating-point addition unit to the floating-point addition unit), so as to ensure the validity of the source operand of the multiplication and accumulation operation and eliminate the risk of physical resource competition. In this embodiment, step 2) includes:

[0070] B2.1) Select a data path PATH i ;

[0071] B2.2) Activate the data path PATH i The 16-bit floating-point multiplication module fmul16, the 32-bit floating-point addition module fadd32 and the 64-bit floating-point addition module fadd64, wherein the 64-bit floating-point addition module fadd64 supports two parallel 32-bit floating-point addition operations, such as Figure 3 As shown in the dashed box, 16 in the figure represents a 16-bit floating-point multiplication module fmul1, 32 represents a 32-bit floating-point addition module fadd32, and 64 represents a 64-bit floating-point addition module fadd64;

[0072] B2.3) In this data path, the multiplication and accumulation operation is used as the calculation granularity, and the original matrix multiplication operation is divided into two micro-operations based on the dot multiplication operation. Each micro-operation is passed through the data path PATH i The 16-bit floating-point multiplication module fmul16, the 32-bit floating-point addition module fadd32 and the 64-bit floating-point addition module fadd64 complete the point multiplication operation, and the pipeline interval of continuous micro-operations is 3 beats.

[0073] In this embodiment, the input first source operand opa and the second source operand opb of the original matrix multiplication operation are two sets of matrix data, one set of matrix data has a size of 2×4, and the other set of matrix data has a size of 4×2, and the output destination operand res is a matrix data with a size of 2×2; in step B2.3), each micro-operation is passed through the data path PATH i The 16-bit floating-point multiplication module fmul16, the 32-bit floating-point addition module fadd32 and the 64-bit floating-point addition module fadd64 complete the dot multiplication operation including: performing three levels of nested loop traversal for n, j and k in sequence, and calculating the element with index j in the subvector with index n of the destination operand according to the following formula in the third level of loop traversal:

[0074] res[n][j]+=opa[n][2k]*opb[j][2k]+opa[n][2k+1]*opb[j][2k+1]

[0075] In the above formula, res[n][j] represents the element with index j in the subvector with index n of the destination operand res with element length of 32 bits, "+=" represents an accumulation operation, opa[n][2k] represents the element with index 2k in the subvector with index n of the first source operand opa with element length of 16 bits, opa[n][2k+1] represents the element with index 2k+1 in the subvector with index n of the first source operand opa with element length of 16 bits, opb[j][2 k] represents the element with index 2k in the sub-vector with index j of the second source operand opb with element length of 16 bits, opb[j][2k+1] represents the element with index 2k+1 in the sub-vector with index j of the second source operand opb with element length of 16 bits, where n, j, k range in [0,2), n is the row number of the first source operand opa corresponding to the operation element, j is the row number of the second source operand opb corresponding to the operation element, and k is the matrix column number corresponding to the operation element. In this embodiment, three levels of nested loop traversal are performed for n, j, and k in sequence, and in the third level of loop traversal, the element with index j in the sub-vector with index n of the destination operand is calculated according to the following formula, which can be expressed in pseudo code as follows:

[0076]

[0077] It should be noted that the order of the three-level nesting is not limited to the above example.

[0078] Taking the matrix element res[0][0] as an example, the granularity in this embodiment is the "opa[0][0]*opb[0][0]+opa[0][1]*opb[0][1]" level in the matrix element, and the matrix multiplication operation is split into 2 micro-operations, each of which completes the dot multiplication operation. The pipeline of the data channel corresponding to this embodiment is shown in Table 2.

[0079] Table 2: Pipeline of data channel corresponding to Example 2.

[0080] I1 I2 I3 M1 M2 M3 A1 A2 A3 A1 A2 A3 W1 I1 I2 I3 M1 M2 M3 A1 A2 A3 A1 A2 A3 W1

[0081] In the above table, I1~I3 represent the instruction issuance stage, M1~M3 represent the floating-point multiplication unit execution stage, A1~A3 represent the floating-point addition unit execution stage, and W1 represents the data write-back stage. Due to the data dependency of continuous accumulation operations in micro-operations, the micro-operation pipeline formed by this design has bubbles, and there are at least two beats between continuous micro-operations.

[0082] The dot multiplication operation process requires three steps, namely floating-point multiplication FMUL, floating-point addition FADD and floating-point addition FADD. According to the given data execution path configuration, this operation cannot be supported because the data path lacks the 32-bit floating-point addition module fadd32. In this embodiment, it is further proposed to design the FADD64 module as a 64-bit floating-point addition module fadd64 compatible with two 32-bit floating points to support the dot multiplication operation. The micro-operation pipeline interval of this method is 3 beats, and the calculation efficiency of the hardware resources according to the effective calculation data width is (32+16+16+32)*V / 32 / V / 3=100%, and the matrix multiplication calculation throughput of the corresponding structure is C*V / 2 / 3 / (C*V)=1 / 6.

[0083] In addition, this embodiment also provides a configurable BF16 format matrix multiplication data path implementation component, including a control unit, multiple data paths, and a configuration register for configuring a selected data path and its data width V, the control unit is respectively connected to the multiple data paths and the configuration register, each data path includes a 16-bit floating-point multiplication module fmul16, a 32-bit floating-point addition module fadd32, and a 64-bit floating-point addition module fadd64, and the number of 64-bit floating-point addition modules fadd64 is twice the number of 32-bit floating-point addition modules fadd32, and the number of 32-bit floating-point addition modules fadd32 is twice the number of 16-bit floating-point multiplication modules fmul16, the control unit is programmed or configured to execute the steps of the aforementioned configurable BF16 format matrix multiplication data path implementation method.

[0084] In addition, this embodiment also provides a computer device, including a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the steps of the aforementioned configurable BF16 format matrix multiplication data path implementation method.

[0085] In addition, this embodiment also provides a computer-readable storage medium, in which a computer program is stored. The computer program is a computer program programmed or configured to execute the aforementioned configurable BF16 format matrix multiplication data path implementation method.

[0086] Embodiment three:

[0087] This embodiment is basically the same as the first embodiment, and the main difference is that the calculation granularity supported by the calculation unit in the selection processor is different. In this embodiment, each matrix multiplication operation can be divided into a unique micro-operation, and the pipeline interval of continuous micro-operations is 1 beat, and there is no competition and risk for data and resources. For a data channel with V greater than 256 bits, it can support a 128-bit wide matrix multiplication operation. In this embodiment, step 2) includes:

[0088] C2.1) Select two data paths, such as PATH i and PATH j , preferably two adjacent data paths to reduce the distance of data transmission;

[0089] C2.2) Activate the 16-bit floating-point multiplication module fmul16, the 32-bit floating-point addition module fadd32, and the 64-bit floating-point addition module fadd64 of the two data paths, wherein the 64-bit floating-point addition module fadd64 supports two parallel 32-bit floating-point addition operations, such as Figure 4 As shown in the dashed box, 16 in the figure represents a 16-bit floating-point multiplication module fmul1, 32 represents a 32-bit floating-point addition module fadd32, and 64 represents a 64-bit floating-point addition module fadd64;

[0090] C2.3) In the data path, matrix multiplication operation is used as the calculation granularity, and the original matrix multiplication operation is converted into one micro-operation based on matrix multiplication, and the pipeline interval of continuous micro-operations is 1 beat, and the input includes 8 source registers and destination registers; for each data path, the 8 input data are input into the 16-bit floating-point multiplication module fmul16 of the data path in pairs, and the multiplication result is first multiplied with the destination register in the 32-bit floating-point addition module fadd32 to obtain the multiplication result, and then the multiplication result is sent to the 64-bit floating-point addition module fadd64 of the data path or another data path, and is accumulated with the multiplication result of another data path in the 64-bit floating-point addition module fadd64 to obtain the final result.

[0091] In this embodiment, the input first source operand opa and the second source operand opb of the original matrix multiplication operation are two groups of matrix data, one group of matrix data has a size of 2×4, and the other group of matrix data has a size of 4×2, and the output destination operand res is a matrix data of size 2×2; in step C2.3), the 8 input data are input two by two into the 16-bit floating-point multiplication module fmul16 of the data path, the multiplication result is first multiplied with the destination register in the 32-bit floating-point addition module fadd32 to obtain the multiplication result, and then the multiplication result is sent to the 64-bit floating-point addition module fadd64 of the data path or another data path, and the multiplication result is accumulated with the multiplication result of another data path in the 64-bit floating-point addition module fadd64 to obtain the final result, including: performing two levels of nested loop traversal for n and j in turn, and calculating the element with index j in the subvector with index n of the destination operand according to the following formula in the second level of loop traversal:

[0092] res[n][j]+=opa[n][0]*opb[j][0]+opa[n][1]*opb[j][1]+opa[n][2]*opb[j][2]+opa[n][3]*opb[j][3]

[0093] In the above formula, res[n][j] represents the element with index j in the subvector with index n of the destination operand res with an element length of 32 bits, "+=" represents an accumulation operation, opa[n][0] represents the element with index 0 in the subvector with index n of the first source operand opa with an element length of 16 bits, opb[j][0] represents the element with index 0 in the subvector with index j of the second source operand opb with an element length of 16 bits, opa[n][1] represents the element with index 1 in the subvector with index n of the first source operand opa with an element length of 16 bits, and opb[j][1] represents the element length of 1. The 6-bit second source operand opb is the element with index 1 in the sub-vector with index j, opa[n][2] represents the element with index 2 in the sub-vector with index n of the first source operand opa with element length of 16 bits, opb[j][2] represents the element with index 2 in the sub-vector with index j of the second source operand opb with element length of 16 bits, opa[n][3] represents the element with index 3 in the sub-vector with index n of the first source operand opa with element length of 16 bits, and opb[j][3] represents the element with index 3 in the sub-vector with index j of the second source operand opb with element length of 16 bits. In this embodiment, two levels of nested loop traversal are performed for n and j in turn, and the element with index j in the sub-vector with index n of the destination operand is calculated in the second level of loop traversal according to the following formula, which can be expressed in pseudo code as follows:

[0094]

[0095] It should be noted that the order of the two-level nesting is not limited to the above example.

[0096] Taking the matrix element res[0][0] as an example, the granularity in this embodiment is the "opa[0][0]*opb[0][0]+opa[0][1]*opb[0][1]+opa[0][2]*opb[0][2]+opa[0][3]*opb[0][3]" level in the matrix element, and the matrix multiplication operation is converted into 1 micro-operation. The pipeline of the data channel corresponding to this embodiment is shown in Table 3.

[0097] Table 3: Pipeline of data channel corresponding to Example 3.

[0098] I1 I2 I3 M1 M2 M3 A1 A2 A3 A1 A2 A3 A1 A2 A3 W1

[0099] In the above table, I1~I3 represent the instruction issuance stage, M1~M3 represent the floating-point multiplication component execution stage, A1~A3 represent the floating-point addition component execution stage, and W1 represents the data write-back stage. Referring to Table 3, in this embodiment, since each matrix multiplication operation can be divided into a unique micro-operation, the pipeline interval of continuous micro-operations is 1 beat, and there is no competition and risk for data and resources. In this embodiment, the calculation efficiency of the hardware resources according to the effective calculation data width is (32+16+16+32)*V / 32 / V / 3=100%, and the matrix multiplication calculation throughput of the corresponding structure is C*V / 1 / 1 / (C*V)=1. This design may activate multiple data channels, so additional transmission queue modifications will be introduced.

[0100] In addition, this embodiment also provides a configurable BF16 format matrix multiplication data path implementation component, including a control unit, multiple data paths, and a configuration register for configuring a selected data path and its data width V, the control unit is respectively connected to the multiple data paths and the configuration register, each data path includes a 16-bit floating-point multiplication module fmul16, a 32-bit floating-point addition module fadd32, and a 64-bit floating-point addition module fadd64, and the number of 64-bit floating-point addition modules fadd64 is twice the number of 32-bit floating-point addition modules fadd32, and the number of 32-bit floating-point addition modules fadd32 is twice the number of 16-bit floating-point multiplication modules fmul16, the control unit is programmed or configured to execute the steps of the aforementioned configurable BF16 format matrix multiplication data path implementation method.

[0101] In addition, this embodiment also provides a computer device, including a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the steps of the aforementioned configurable BF16 format matrix multiplication data path implementation method.

[0102] In addition, this embodiment also provides a computer-readable storage medium, in which a computer program is stored. The computer program is a computer program programmed or configured to execute the aforementioned configurable BF16 format matrix multiplication data path implementation method.

[0103] In summary, the configurable BF16 format matrix multiplication data path implementation method of the present invention includes: performing data path configuration, including the number of data paths C, and the configuration of the data width V supported by a single data path; selecting a data path, selecting the calculation granularity supported by the calculation component in the processor in the data path, dividing the original matrix multiplication operation into one or more micro-operations, and executing according to the set pipeline interval of continuous micro-operations. The present invention provides a flexible BF16 format matrix multiplication data path implementation method based on the existing typical data execution path, which can improve the hardware design efficiency of the BF16 design, provide higher system performance, and support the execution granularity of multiple data. For the three designs of the above embodiments one to three, the latter two need to increase computing resources in the data pipeline to meet the calculation requirements. Because the matrix multiplication operation pipeline requires twice as many addition resources as multiplication resources. Therefore, the proposed invention defaults to incorporating a 64-bit floating-point addition module fadd64 into the latter two designs, and the corresponding 64-bit floating-point addition module fadd64 is modified to support a 64-bit floating-point addition module fadd64 of parallel 32bit. Because the BF16 operation does not consider exceptions and specifies a fixed rounding mode, we believe that the hardware overhead of this modification is acceptable. The advantage of the first embodiment is that there is no need to make major modifications to the original execution path, and the design of the BF16 matrix multiplication can be completed by fully reusing the multiplication and accumulation data path of the original path. The advantages of the second and third embodiments are that the data throughput rate is relatively high, but a customized design is required. The microprocessor can adopt a suitable computing component design according to the transmission capability of the front-end data, so as to achieve a balanced design. The configurable BF16 format matrix multiplication data path implementation method of the present invention can serve the design of microprocessors with different processor front-end capabilities, and reduce the time cost of spectrum design transplantation.

[0104] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0105] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A configurable BF16 format matrix multiplication data path implementation method, characterized in that: include: 1) Perform data path configuration, including the number of data paths C and the data width V supported by a single data path; 2) Select a data path, select the calculation granularity supported by the calculation unit in the processor in the data path, divide the original matrix multiplication operation into one or more micro-operations, and execute them according to the set pipeline interval of continuous micro-operations; The input first source operand opa and the second source operand opb of the original matrix multiplication operation are two sets of matrix data, one set of matrix data has a size of 2×4, and the other set of matrix data has a size of 4×2, and the output destination operand res is a matrix data of a size of 2×2; step 2) includes: A2.1) Select a data path PATH i ; A2.2) Activate the data path PATH i 16-bit floating-point multiplication module fmul16 and 32-bit floating-point addition module fadd32; A2.3) In this data path, the multiplication and accumulation operation is used as the calculation granularity, and the original matrix multiplication operation is divided into four micro-operations based on the multiplication and accumulation operation. Each micro-operation is passed through the data path PATH i The 16-bit floating-point multiplication module fmul16 and the 32-bit floating-point addition module fadd32 complete the multiplication and accumulation operations, and the pipeline interval of continuous micro-operations is 3 beats; Alternatively, the input first source operand opa and the second source operand opb of the original matrix multiplication operation are two sets of matrix data, one set of matrix data has a size of 2×4, and the other set of matrix data has a size of 4×2, and the output destination operand res is a matrix data with a size of 2×2; step 2) includes: B2.1) Select a data path PATH i ; B2.2) Activate the data path PATH i The 16-bit floating-point multiplication module fmul16, the 32-bit floating-point addition module fadd32 and the 64-bit floating-point addition module fadd64, wherein the 64-bit floating-point addition module fadd64 supports two-way parallel 32-bit floating-point addition operations; B2.3) In this data path, the multiplication and accumulation operation is used as the calculation granularity, and the original matrix multiplication operation is divided into two micro-operations based on the dot multiplication operation. Each micro-operation is passed through the data path PATH i The 16-bit floating-point multiplication module fmul16, the 32-bit floating-point addition module fadd32 and the 64-bit floating-point addition module fadd64 complete the dot multiplication operation, and the pipeline interval of continuous micro-operations is 3 beats; Alternatively, the input first source operand opa and the second source operand opb of the original matrix multiplication operation are two sets of matrix data, one set of matrix data has a size of 2×4, and the other set of matrix data has a size of 4×2, and the output destination operand res is a matrix data with a size of 2×2; step 2) includes: C2.1) Select two data paths; C2.2) Activate the 16-bit floating-point multiplication module fmul16, the 32-bit floating-point addition module fadd32, and the 64-bit floating-point addition module fadd64 of the two data paths, where the 64-bit floating-point addition module fadd64 supports two-way parallel 32-bit floating-point addition operations; C2.3) In this data path, matrix multiplication operation is used as the calculation granularity, and the original matrix multiplication operation is converted into one micro-operation based on matrix multiplication, and the pipeline interval of continuous micro-operations is 1 beat, and the input includes 8 source registers and destination registers; for each data path, the 8 input data are input into the 16-bit floating-point multiplication module fmul16 of the data path in pairs, and the multiplication result is first multiplied with the destination register in the 32-bit floating-point addition module fadd32 to obtain a new multiplication result, and then the new multiplication result is sent to the 64-bit floating-point addition module fadd64 of this data path or another data path, and is accumulated with the multiplication result of another data path in the 64-bit floating-point addition module fadd64 to obtain the final result.

2. The method for implementing a configurable BF16 format matrix multiplication data path according to claim 1, characterized in that: In step A2.3), each micro-operation is passed through the data path PATH i The 16-bit floating-point multiplication module fmul16 and the 32-bit floating-point addition module fadd32 complete the multiplication and accumulation operation, including: performing three levels of nested loop traversal for n, j, and k in sequence, and calculating the element with index j in the subvector with index n of the destination operand according to the following formula in the third level of loop traversal: res[n][j] += opa[n][k]*opb[j][k] In the above formula, res[n][j] represents the element with index j in the subvector with index n of the destination operand res with an element length of 32 bits, "+=" represents an accumulation operation, opa[n][k] represents the element with index k in the subvector with index n of the first source operand opa with an element length of 16 bits, opb[j][k] represents the element with index k in the subvector with index j of the second source operand opb with an element length of 16 bits, where n and j are in the range of [0,2), k is in the range of [0,4), n is the row number of the first source operand opa corresponding to the operation element, j is the row number of the second source operand opb corresponding to the operation element, and k is the matrix column number corresponding to the operation element.

3. The method for implementing a configurable BF16 format matrix multiplication data path according to claim 1, characterized in that: In step B2.3), each micro-operation is passed through the data path PATH i The 16-bit floating-point multiplication module fmul16, the 32-bit floating-point addition module fadd32 and the 64-bit floating-point addition module fadd64 complete the dot multiplication operation including: performing three levels of nested loop traversal for n, j and k in sequence, and calculating the element with index j in the subvector with index n of the destination operand according to the following formula in the third level of loop traversal: res[n][j] += opa[n][2k]*opb[j][2k]+ opa[n][2k+1]*opb[j][2k+1] In the above formula, res[n][j] represents the element with index j in the subvector with index n of the destination operand res with element length of 32 bits, "+=" represents the accumulation operation, opa[n][2k] represents the element with index 2k in the subvector with index n of the first source operand opa with element length of 16 bits, opa[n][2k+1] represents the element with index 2k+1 in the subvector with index n of the first source operand opa with element length of 16 bits, opb[j][2k] represents the element with index 2k in the subvector with index j of the second source operand opb with element length of 16 bits, and opb[j][2k+1] represents the element with index 2k+1 in the subvector with index j of the second source operand opb with element length of 16 bits, where n, j, The value range of k is [0,2), n is the row number of the first source operand opa corresponding to the operation element, j is the row number of the second source operand opb corresponding to the operation element, and k is the matrix column number corresponding to the operation element.

4. The method for implementing a configurable BF16 format matrix multiplication data path according to claim 1, characterized in that: In step C2.3), the eight input data are input two by two into the 16-bit floating-point multiplication module fmul16 of the data path, the multiplication result is first multiplied with the destination register in the 32-bit floating-point addition module fadd32 to obtain the multiplication result, and then the multiplication result is sent to the 64-bit floating-point addition module fadd64 of the data path or another data path, and the multiplication result is accumulated with the multiplication result of another data path in the 64-bit floating-point addition module fadd64 to obtain the final result, including: performing two-level nested loop traversals for n and j in turn, and calculating the element with index j in the subvector with index n of the destination operand according to the following formula in the second-level loop traversal: res[n][j]+=opa[n][0]*opb[j][0]+opa[n][1]*opb[j][1]+opa[n][2]*opb[j][2]+opa[n][3]*opb[j][3] In the above formula, res[n][j] represents the element with index j in the subvector with index n of the destination operand res with an element length of 32 bits, "+=" represents an accumulation operation, opa[n][0] represents the element with index 0 in the subvector with index n of the first source operand opa with an element length of 16 bits, opb[j][0] represents the element with index 0 in the subvector with index j of the second source operand opb with an element length of 16 bits, opa[n][1] represents the element with index 1 in the subvector with index n of the first source operand opa with an element length of 16 bits, and opb[j][1] represents the element length of 1 The 6-bit second source operand opb has an index of element 1 in the subvector indexed by j, opa[n][2] represents the element indexed by 2 in the subvector indexed by n of the first source operand opa with an element length of 16 bits, opb[j][2] represents the element indexed by 2 in the subvector indexed by j of the second source operand opb with an element length of 16 bits, opa[n][3] represents the element indexed by 3 in the subvector indexed by n of the first source operand opa with an element length of 16 bits, and opb[j][3] represents the element indexed by 3 in the subvector indexed by j of the second source operand opb with an element length of 16 bits.

5. A configurable BF16 format matrix multiplication data path implementation component, characterized in that: The invention comprises a control unit, a plurality of data paths and a configuration register for configuring a selected data path and a data width V thereof, wherein the control unit is respectively connected to the plurality of data paths and the configuration register, each data path comprises a 16-bit floating-point multiplication module fmul16, a 32-bit floating-point addition module fadd32 and a 64-bit floating-point addition module fadd64, and the number of the 64-bit floating-point addition modules fadd64 is twice the number of the 32-bit floating-point addition modules fadd32, and the number of the 32-bit floating-point addition modules fadd32 is twice the number of the 16-bit floating-point multiplication modules fmul16, and the control unit is programmed or configured to execute the steps of the configurable BF16 format matrix multiplication data path implementation method as described in any one of claims 1 to 4.

6. A computer device comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the steps of the configurable BF16 format matrix multiplication data path implementation method as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored therein, characterized in that: The computer program is a computer program programmed or configured to execute the configurable BF16 format matrix multiplication data path implementation method as described in any one of claims 1 to 4.