Computing engine, data processing method thereof, processing chip and computer equipment

Abstract

The invention provides a calculation engine and a data processing method thereof, a processing chip and computer equipment, the calculation engine comprises a dot multiplication unit array and a control logic unit, the dot multiplication unit array comprises a plurality of dot multiplication units, and each dot multiplication unit comprises a first calculation unit group and a second calculation unit group, the point multiplication unit supports point multiplication of data equal to or lower than the first precision; a second calculation unit group supporting point multiplication of data equal to or lower than a second precision, the second precision being lower than the first precision; the third calculation unit group supports point multiplication operation of data with third precision, the third precision is lower than the second precision, the control logic unit reads data to be processed according to an obtained control instruction and transmits the data to the point multiplication unit array, and when the data to be processed is the data with the third precision, the point multiplication operation of the data to be processed is carried out. And the control logic unit transmits the data with the third precision to the dot multiplication unit according to a data processing mode.

Description

Technical Field

[0001] This application relates to the field of high-performance computing technology, and in particular to a computing engine and its data processing method, processing chip, and computer device. Background Technology

[0002] The rapid iteration and continuous growth of large-scale artificial intelligence models are not only increasing the demand for computing power and storage capacity, but also leading to greater power consumption. To reduce the computational and storage requirements of large models, one solution is to replace high-precision data formats with low-precision data formats.

[0003] In artificial intelligence applications, the value distribution range of data within tensors is relatively large, while low-precision data formats represent a smaller range of data with lower precision. To resolve the conflict between the precision of low-precision data and tensor data in practical applications, quantization methods are typically used to convert tensor data in practical applications from high precision to low precision. This reduces the computational load, storage load, and power consumption of the model while preserving as much key information as possible.

[0004] Commonly used quantization methods are categorized by computational granularity from coarse to fine: channel-wise quantization, tile-wise quantization, group quantization, and element-wise quantization. Among these, group quantization groups multiple consecutive elements and quantizes each group separately. Furthermore, the group size can be adjusted as needed, such as vector-granular quantization or block-granular quantization. The scaling factor used in quantization is typically stored in a high-precision data format.

[0005] It should be noted that the above description of the technical background is only for the purpose of providing a clear and complete explanation of the technical solutions of the present invention and facilitating understanding by those skilled in the art. It should not be assumed that the above technical solutions are known to those skilled in the art simply because they have been described in the background section of this invention. Summary of the Invention

[0006] Existing computing devices that support large-scale artificial intelligence models, such as graphics processing units (GPUs), tensor processing units (TPUs), and edge computing devices, integrate tensor computing engines (TCEs) to improve the efficiency of tensor computation.

[0007] Figure 1This is a schematic diagram of the components of a tensor computation engine. For example... Figure 1 As shown, the tensor computation engine 100 includes a dot product (DP) array 101, a control logic (CTRL_LOGIC) unit 102, a data reuse buffer 103, an address swizzle 104, a local memory unit 105, and a vector register file 106, etc., for data computation, logic control, address allocation, and data temporary storage. The DP array contains 64 dot product units 107 arranged in 8 rows and 8 columns (8x8), and the number of multipliers within each DP can be adjusted according to requirements.

[0008] The data processing flow of the tensor computation engine 100 includes: the control logic unit 102 triggers the reading of data to be processed from the tensor memory engine (TME) 108 and stores it in the local storage unit 105 or the vector register file 106; the address allocator 104 allocates the data in the local storage unit 105 and the vector register file 106 to the corresponding DP according to the address rules of the computation requirements; and the control logic unit 102 triggers the DP to complete the dot product (multiplicative accumulation) operation of the data to be processed according to predetermined computation parameters (e.g., data read from the constant memory unit 109). The tensor storage engine 108 includes an address allocator 110 for address allocation of the stored data to facilitate its transfer to the tensor computation engine 100.

[0009] The tensor calculation process includes: taking the TCE receiving two sets of data A0~A31 and B0~B31 as an example, the address allocator 104 allocates these two sets of data to the dot product unit 107, and the dot product unit 107 performs the dot product operation: A0*B0+A1*B1+…+A31*B31.

[0010] To adapt to different artificial intelligence application scenarios, tensor computing engines typically support multiple floating-point data types, such as 32-bit tensor floating-point format (TF32), 16-bit brain floating-point format (BF16), and 16-bit half-precision floating-point format (FP16). For example, the data types of two sets of data A0~A31 and B0~B31 are TF32, BF16, or FP16.

[0011] Figure 2 This is a diagram illustrating floating-point data format. For example... Figure 2 As shown, the maximum exponent bits for TF32, BF16, and FP16 are 8 bits, and the maximum mantissa bits are 10 bits. Therefore, these floating-point data formats can all adopt the same implementation method.

[0012] For input data in different formats, it needs to be converted to a unified predefined format (or internal format (internalfmt)). The internal format includes 1 floating-point bit (or sign bit), more than 9 bits of exponent bits, and more than 23 bits of mantissa bits. Since the maximum exponent of the input data is 8 bits, and multiplication of floating-point numbers involves addition of exponents, the exponent bits of the output data need to be greater than 9 bits to avoid overflow after adding exponents. The output data format uses FP32, which has 23 bits of mantissa bits, ensuring sufficient mantissa bits to avoid precision loss during calculation.

[0013] Figure 3 This is another illustration of floating-point data format. For example... Figure 3 As shown, the FP8 format includes a 4-bit exponent and 3-bit mantissa floating-point format (E4M3) and a 5-bit exponent and 2-bit mantissa floating-point format (E5M2). Compared to E5M2, data represented using E4M3 has more mantissa bits and higher precision, but relatively fewer exponent bits, resulting in a smaller data range. Therefore, weights and activation tensors in tensor computation are typically stored using E4M3, while gradient tensors are typically stored using E5M2. With the scale factor configured, the data range represented by E4M3 increases, making it a viable primary implementation for FP8.

[0014] The inventors of this application have discovered that currently, in mixed-precision training models for artificial intelligence, tensor computation engines typically employ high-precision data formats (e.g., FP16, BF16, and FP32), and sometimes lower-precision data formats (e.g., FP8). As the computational load increases and the demands for computational speed rise, using lower-precision data formats for computation has become an optional technical path. However, how to support computations involving lower-precision data formats (e.g., tensor computation) has become a problem that needs to be solved.

[0015] To address at least one of the aforementioned problems or other similar issues, embodiments of this application provide a computing engine and its data processing method, processing chip, and computer device. In this computing engine, each dot product unit includes a first computing unit group, a second computing unit group, and a third computing unit group supporting dot product operations of data with different precisions. A control logic unit can transmit third-precision data to the dot product unit according to a data processing mode. This allows for flexible computation of low-precision data to meet computational needs. For example, the first and second computing unit groups can participate in processing at least a portion of the third-precision data, thereby supporting third-precision data operations (e.g., dot product operations) at a lower cost.

[0016] According to one aspect of the embodiments of this application, a computing engine is provided, the system including a dot product unit array and a control logic unit, wherein...

[0017] The array of dot product units includes multiple dot product units, each dot product unit comprising:

[0018] The first set of computing units supports dot multiplication of data with a precision equal to or lower than the first precision.

[0019] A second set of computing units supports dot product operations on data with a precision equal to or lower than a second precision, where the second precision is lower than the first precision; and

[0020] The third computational unit group supports dot product operations on data with a third precision, which is lower than the second precision.

[0021] The control logic unit reads the data to be processed and transmits it to the dot product unit array according to the acquired control instructions. When the data to be processed is the third precision data, the control logic unit transmits the third precision data to the dot product unit according to the data processing mode.

[0022] In some embodiments, each dot product unit further includes a processing unit that processes at least two of the operation results of the first calculation unit group, the operation results of the second calculation unit group, and the operation results of the third calculation unit group.

[0023] In some embodiments, when the data processing mode is a first mode, the control logic unit inputs the first, second, and third portions of the third-precision data to the first calculation unit group, the second calculation unit group, and the third calculation unit group, respectively.

[0024] The processing unit integrates the calculation results of the first calculation unit group, the second calculation unit group, and the third calculation unit group.

[0025] In some embodiments, when the data processing mode is the second mode, the control logic unit inputs the data of the third precision into the third calculation unit group.

[0026] The calculation result of the third calculation unit group is input as input data to the first calculation unit group and the second calculation unit group.

[0027] The processing unit processes the calculation results of the first calculation unit group and the calculation results of the second calculation unit group.

[0028] In some embodiments, the operation results of the two multipliers in the third computing unit group are summed by an adder, and the sum is input as a multiplier into one of the multipliers in the first computing unit group or the second computing unit group, wherein the other multiplier of the multiplier is 1.

[0029] In some embodiments, when the data processing mode is the third mode, the control logic unit inputs all the data of the third precision into the third calculation unit group.

[0030] The processing unit integrates the calculation results of the third calculation unit group and the first calculation unit group and / or the calculation results of the second calculation unit group and / or the calculation results of the constant calculation unit.

[0031] In some embodiments, the multiplication operation in the dot product operation performed by the third computing unit group includes:

[0032] For multiplication of floating-point numbers containing 1 bit of decimal place, perform integer multiplication followed by division by 2 or 4;

[0033] Determine the result of the multiplication operation based on the correspondence between the two multipliers and their product in the lookup table; or

[0034] Perform multiplication on the mantissas of the two multipliers, add the exponents of the two multipliers, and align the exponents after the addition to 0.

[0035] In some embodiments, the processing unit:

[0036] The number of leading zeros in the operation result of the third calculation unit group is calculated, and the result is converted into a predetermined format through shifting. An exponent value is then calculated, and the exponent value is exponentially aligned with the operation results of at least one of the first calculation unit group, the second calculation unit group, and the constant calculation unit; or...

[0037] While keeping the format of the exponent bits of the operation result in the third calculation unit group unchanged, the integer bits of the operation result are increased.

[0038] In some embodiments, the control logic unit is further configured to perform the following operations:

[0039] Based on the supported matrix size and scaling factor in the control instruction, calculate the register address of the scaling factor that the dot product unit array needs to read;

[0040] Read the scaling factor and store it in the storage module of the computing engine; and

[0041] The stored scaling factor is sent to the first calculation unit group, the second calculation unit group, and the third calculation unit group of each of the dot product units.

[0042] In some embodiments, the computing engine further includes:

[0043] A format conversion unit is used to convert the data in the dot product unit array into a predetermined format.

[0044] In this configuration, a predetermined number of dot product units in the dot product unit array share one format conversion unit.

[0045] According to another aspect of the embodiments of this application, a processing chip is provided, the processing chip including one or more processor clusters, each of the processor clusters including a computing engine as described in any of the foregoing embodiments.

[0046] According to another aspect of the embodiments of this application, a data processing method for a computing engine is provided, the computing engine including a dot product unit array and a control logic unit, wherein...

[0047] The array of dot product units includes multiple dot product units, each dot product unit comprising:

[0048] The first set of computing units supports dot multiplication of data with a precision equal to or lower than the first precision.

[0049] A second set of computing units supports dot product operations on data with a precision equal to or lower than a second precision, where the second precision is lower than the first precision; and

[0050] The third computational unit group supports dot product operations on data with a third precision, which is lower than the second precision.

[0051] The data processing method includes:

[0052] The control logic unit sequentially reads and transmits the data to be processed to the dot product unit array according to the acquired control instructions. When the data to be processed is data of the third precision, the control logic unit transmits the data of the third precision to the dot product unit according to the data processing mode.

[0053] The first calculation unit group, the second calculation unit group, and the third calculation unit group each perform dot multiplication on the input data.

[0054] According to another aspect of the embodiments of this application, a computer device is provided, including a memory and at least one processing chip, wherein the memory stores a computer program that can run on the processing chip, and the processing chip executes the computer program to implement the data processing method as described in the above embodiments.

[0055] One of the beneficial effects of the embodiments of this application includes: in the calculation engine, each dot product unit includes a first calculation unit group, a second calculation unit group and a third calculation unit group that support dot product operations of data with different precisions. The control logic unit can transmit the data of the third precision to the dot product unit according to the data processing mode. Thus, low-precision data can be operated in a flexible manner to meet the calculation requirements.

[0056] Specific embodiments of the invention are disclosed in detail with reference to the following description and accompanying drawings, indicating how the principles of the invention can be employed. It should be understood that the embodiments of the invention are not therefore limited in scope. Within the spirit and scope of the appended claims, embodiments of the invention include many changes, modifications, and equivalents.

[0057] Features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.

[0058] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, whole, step, or component, but does not exclude the presence or addition of one or more other features, wholes, steps, or components. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. The drawings, which provide a further understanding of the embodiments of the present invention, constitute a part of the specification and are used to illustrate the implementation of the present invention, and together with the text description, explain the principles of the present invention. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0060] Figure 1 This is a schematic diagram of the tensor computation engine structure;

[0061] Figure 2 This is a schematic diagram of floating-point data format;

[0062] Figure 3 This is another schematic diagram of floating-point data format;

[0063] Figure 4 This is a schematic diagram of a computing engine according to an embodiment of this application;

[0064] Figure 5 This is a schematic diagram of the dot product unit in an embodiment of this application;

[0065] Figure 6 This is a schematic diagram of a scaling factor allocation method according to an embodiment of this application;

[0066] Figure 7 This is a schematic diagram illustrating the connection relationship between the computing engine and the multiplication processing unit in an embodiment of this application;

[0067] Figure 8 This is a schematic diagram of the scaling factor storage method according to an embodiment of this application;

[0068] Figure 9 This is a schematic diagram of the format conversion unit of the dot product unit array in an embodiment of this application;

[0069] Figure 10 This is another schematic diagram of floating-point data format;

[0070] Figure 11 This is another schematic diagram of the computing engine in an embodiment of this application;

[0071] Figure 12 This is another schematic diagram of the dot product unit in an embodiment of this application;

[0072] Figure 13 This is a partial schematic diagram of a lookup table according to an embodiment of this application;

[0073] Figure 14 This is another schematic diagram of the dot product unit in an embodiment of this application;

[0074] Figure 15 This is another schematic diagram of the dot product unit in an embodiment of this application;

[0075] Figure 16 This is a schematic diagram of a data processing method for a computing engine according to an embodiment of this application;

[0076] Figure 17 This is another schematic diagram of a data processing method for a computing engine according to an embodiment of this application;

[0077] Figure 18 This is a schematic diagram of a processing chip according to an embodiment of this application;

[0078] Figure 19 This is another schematic diagram of the processing chip in an embodiment of this application;

[0079] Figure 20 This is a schematic diagram of a computer device;

[0080] Figure 21 This is another schematic diagram of a computer device. Detailed Implementation

[0081] Referring to the accompanying drawings, the foregoing and other features of this application will become apparent from the following description. Specific embodiments of this application are specifically disclosed in the description and drawings, illustrating partial implementations in which the principles of this application can be adopted. It should be understood that this application is not limited to the described embodiments; rather, it includes all modifications, variations, and equivalents falling within the scope of the appended notes. Various embodiments of this application are described below with reference to the accompanying drawings. These embodiments are merely exemplary and not intended to limit the scope of this application.

[0082] In the embodiments of this application, the terms "first," "second," etc., are used to distinguish different elements by name, but do not indicate the spatial arrangement or chronological order of these elements, and these elements should not be limited by these terms. The term "and / or" includes any one or more of the associated listed terms and all combinations thereof. The terms "comprising," "having," etc., refer to the presence of the stated features, elements, components, or assemblies, but do not exclude the presence or addition of one or more other features, elements, components, or assemblies.

[0083] In the embodiments of this application, the singular forms "a," "the," etc., including the plural forms, should be broadly understood as "a kind" or "a class" rather than limited to the meaning of "an"; furthermore, the term "the" should be understood to include both the singular and plural forms, unless the context clearly indicates otherwise. Additionally, the term "according to" should be understood as "at least in part according to…," unless the context clearly indicates otherwise.

[0084] First aspect of the embodiments

[0085] An embodiment of the first aspect of this application provides a computing engine, which may be a computing engine 400 that supports dot product operations on data with FP8 or higher precision, or a computing engine 1100 that supports dot product operations on data with FP4 or higher precision.

[0086] The following describes the computing engine 400 of an embodiment of this application. Figure 4 This is a schematic diagram of the computing engine in an embodiment of this application, as shown below. Figure 4 As shown, the computing engine 400 includes a dot product unit array 401 and a control logic unit 402.

[0087] The dot product unit array 401 includes multiple dot product units 403, each dot product unit 403 including a first calculation unit group 404, a second calculation unit group 405, a constant calculation unit 406, and a processing unit 407. The first calculation unit group 404 supports dot product operations on data with a precision equal to or lower than a first precision; the second calculation unit group 405 supports dot product operations on data with a second precision lower than the first precision; the constant calculation unit 406 supports constant operations; and the processing unit 407 processes at least two of the operation results from the first calculation unit group 404, the operation results from the second calculation unit group 405, and the constant calculation unit 406.

[0088] The control logic unit 402 reads the data to be processed and transmits it to the dot product unit array 401 in sequence according to the acquired control instructions.

[0089] In this application, since the number of second-precision data (elements) that can be transmitted under the same bandwidth exceeds the number of first-precision elements, when processing second-precision data, each dot product unit 403 in the computing engine 400 uses a first computing unit group 404 supporting the first precision to process a portion of the second-precision data to be processed. On this basis, a second computing unit group 405 supporting the second precision is added to process the remaining data in the second-precision data to be processed, thereby realizing the dot product operation function supporting second-precision data. The first computing unit group 404 participates in processing at least a portion of the second-precision data, thereby supporting the operation of second-precision data (e.g., dot product operation) at a lower cost and reducing the development cost of the computing engine 400.

[0090] In some examples, the first precision includes formats such as FP16, BF16, or TF32, and the second precision is, for example, FP8. Under the same bandwidth, the number of FP8 format data (elements) that can be transmitted is twice the number of FP16 format elements. Therefore, each dot product unit 403 in the computing engine 400 can reuse the first computing unit group 404 in each dot product unit of the computing engine supporting the FP16 format to process half of the data to be processed in the FP8 format (e.g., A0~A15, B0~B15). On this basis, a second computing unit group 405 containing 16 dot product units (DP16) is used to process the other half of the data to be processed (e.g., A16~A31, B16~31), thereby supporting data operations in the FP8 format.

[0091] The following describes the implementation of the dot product unit 403 of the calculation engine 400.

[0092] Figure 5 This is a schematic diagram of a dot product unit according to an embodiment of this application. The dot product unit includes a 32-bit dot product unit (DP32) that supports data operations in FP8 and higher formats. Figure 5 As shown, the dot product unit 403 includes a first calculation unit group 404, a second calculation unit group 405 and a constant calculation unit 406 arranged in parallel, as well as a processing unit 407.

[0093] The first computing unit group 404 contains 16 point multiplication units (DP16), that is, 16 multipliers 501, which can support multiplication operations of data in FP16, BF16 and TF32 formats. Half of the data in FP8 format to be processed (e.g., A0~A15, B0~B15) can be assigned to the first computing unit group 404 for multiplication operations.

[0094] The second computing unit group 405 contains 16 dot multiplication units, that is, 16 multipliers 502, which can only support data operations in FP8 format. The other half of the FP8 format data to be processed (e.g., A16~A31, B16~31) can be assigned to the second computing unit group 405 for multiplication operations.

[0095] Since the maximum mantissa of the FP8 format is 4 bits, and its mantissa contains 3 bits of mantissa and 1 bit of implicit value (e.g., 1), the specifications (e.g., precision, complexity, etc.) of the multiplier 502 and adder a2 in the second computing unit group 405 are smaller than those of the multiplier 501 and adder a1 used to process mantissa multiplication operations in the first computing unit group 404, which is beneficial to saving chip area.

[0096] like Figure 5 As shown, the processing unit 407 includes an exponent maximum value calculation module 503, an alignment module 504 for aligning elements within the first calculation unit group 404 or the second calculation unit group 405, an addition module 505 for summing elements within the first calculation unit group 404 or the second calculation unit group 405, a quantization module 506 for quantizing or dequantizing elements within the first calculation unit group 404 or the second calculation unit group 405, an alignment module 507 for aligning elements of the first calculation unit group 404, the second calculation unit group 405, and the constant calculation unit 406, an addition module 508 for summing elements between the first calculation unit group 404, the second calculation unit group 405, and the constant calculation unit 406, a normalization module 509, and a conversion unit 510 for data format conversion.

[0097] In the description of this application: quantization, quantization processing, and quantization operation have the same meaning, that is, the process of converting floating-point numbers into integers; dequantization processing means the process of converting integers into floating-point numbers.

[0098] The following describes the floating-point alignment method for the FP8 format of the processing unit 407, the allocation method of the scaling factor in quantization operations, and the format conversion unit multiplexing method. The scaling factor can also be used for inverse quantization processing.

[0099] (1) Floating-point alignment method in FP8 format

[0100] When performing dot product operations in FP8 format, exponent alignment of DP16 in the first computation unit group 404, DP16 in the second computation unit group 405, and constant computation unit 406 includes the following two methods:

[0101] Method 1: When performing FP8 format calculations, the data of DP16 in the first calculation unit group 404 and the data of DP16 in the second calculation unit group 405 can be simultaneously aligned by exponent. That is, when calculating the maximum exponent, the maximum value of the exponent of the 32 elements is taken. The exponent alignment of the constant calculation unit 406 can be performed together with these 32 elements. After the exponent alignment is completed, the addition module 508 can be used directly for addition operations, followed by normalization processing using the normalization module 509.

[0102] Method 2: When performing FP8 format calculations, the DP16 of the first calculation unit group 404 and the DP16 of the second calculation unit group 405 can be exponentially aligned respectively. That is, when calculating the maximum value of the exponent, the maximum value is taken for the 16 elements of the two groups respectively. Then, the elements in each group are added to generate the sum of the two groups, and then the exponent is aligned with the constant calculation unit 406. After that, the addition operation and normalization processing are performed.

[0103] Compared to Method 1, in Method 2, after the summation of the data in DP16 of the first calculation unit group 404 and the data in DP16 of the second calculation unit group 405, the exponent of the constant calculation unit 406 is uniformly aligned, which facilitates the forwarding of the data matrix of the constant calculation unit 406. Furthermore, since the tensor operation dimension K in a single processing step of the TCE instruction is usually greater than the number of dot product units contained in the TCE calculation unit, the output of each calculation by the calculation unit needs to be used as the input for the next calculation, i.e., the data of the constant calculation unit 406. Moving the exponent alignment step of the data in the constant calculation unit 406 after the addition avoids the timing problem caused by having to perform exponent alignment for each summation as in Method 1.

[0104] (2) Method for allocating scaling factors

[0105] Traditional AI hybrid training models configure only a single global scaling factor. For mixed-precision training models that include low-precision formats (e.g., FP8), one approach is to directly reuse existing tensor computation units that support high-precision data formats, i.e., using tensor-level quantization. However, this coarse-grained quantization can easily lead to significant precision loss for low-precision data operations. Another approach is that some low-precision hybrid training models, even if they achieve block-level quantization, still need to calculate the results of the tensor computation engine and the scaling factor in the vector computation unit. This increases the data transfer process and reduces the throughput of tensor operations.

[0106] Figure 6 This is a schematic diagram of a scaling factor allocation method according to an embodiment of this application. For example... Figure 6 As shown, the control logic unit 402 is configured to perform the following operations:

[0107] Operation 601: Calculate the register address of the scaling factor that the dot product cell array 401 needs to read, based on the supported matrix (K) size and the group size of the scaling factor in the control instruction.

[0108] Operation 602: Read the scaling factor and store it in the storage module of the computing engine 400 (not shown in the figure); and

[0109] Operation 603: Send (copy or broadcast) the stored scaling factor to the first calculation unit group 404, the second calculation unit group 405 and the third calculation unit group 406 of each dot product unit 403.

[0110] In this application, for the first computing unit group 404 and the second computing unit group 405, scaling factors that match the local data features of the tensor can be set respectively, which refines the granularity of the scaling factor and improves the accuracy of low-precision data operation compared with the scaling factor at the granularity of the entire tensor.

[0111] For example, the TCE control instructions from the control logic unit 402 support matrix shapes K1 of size 32, 64, or 128 in FP8 format, and matrix shapes K2 of size 64, 128, or 256 in FP4 format. If the scale factor group size is 16, then in the case of K2=256, 16 consecutive registers are needed to store the scale factor. Users need to choose an appropriate group size according to actual needs to meet the accuracy requirements of model training and inference while minimizing the storage capacity of the scale factor.

[0112] In Operation 601, the TCE control instruction adds a scale factor bit field to specify the starting address of the scale factor in the register, and a group size bit field to specify the group size.

[0113] In operation 603, taking data format FP8, K=128, group size=128 as an example, the scale factor only needs to occupy one register. The TCE control unit reads the scale factor at the beginning of the calculation and stores it in the internal storage module of the TCE. For the FP8 format, the TCE's DP calculates the K=32 element each time. Therefore, the TCE needs to loop four times to complete the calculation of an K=128 element. In each loop, the TCE does not need to continue to read the scale factor from the register. The scale factor stored in the internal storage of the TCE can be reused in all four loops.

[0114] Because the dot product unit in the FP8 format uses a group size of 16 as its internal unit, the TCE control unit needs to internally copy or broadcast the scale factor to the two DP16 units. For the FP4 format, it needs to be broadcast to the four DP16 units of DP64.

[0115] Figure 7 This is a schematic diagram illustrating the connection relationship between the computation engine and the multiplication processing unit in an embodiment of this application. For example... Figure 7 As shown, a computing engine 400 contains four arithmetic logic units (ALUs) 701, and the input of each ALU is connected to four multiplier processing elements (MPEs) 703a-703d of the processing module 702.

[0116] The computing engine 400 reads the B matrix data from the local memory storage (LMS) unit 704 and broadcasts it to the four ALUs of the TCE. The four arithmetic logic units 701a-701d of the TCE all need to read the scaling factor A matrix of the B matrix. That is, the four multiplication processing units 703a-703d can all store the complete scaling factor A matrix of the B matrix.

[0117] The outputs of the four arithmetic logic units 701a-701d of the TCE are connected to the four multiplication processing units 706a-706d of the processing module 705, respectively.

[0118] Therefore, TCE and MPE use a one-to-one connection method, avoiding the complex wiring caused by many-to-many connections.

[0119] Figure 8 This is a schematic diagram of the scaling factor storage method according to an embodiment of this application. For example... Figure 8 As shown, scaling factors 802a, 802b, 802c and 802d are stored contiguously in the M direction in global memory unit 801. The size of scaling factors 802a, 802b, 802c and 802d is M1*N1, for example, M1=128 and N1=1.

[0120] A group of collaborative computational units (waves) within each MPE (not shown in the diagram) execute load instructions (e.g., LSU.LD.BURST4 instructions) to load 128 elements in the M direction into the registers of multiplication processing units 703a-703d. This ensures that each MPE's registers contain 128 scaling factors, covering the range of the B matrix required for the current operation. Multiple scaling factors are arranged in contiguous address pairs within the MPE's registers.

[0121] For example, the register of the multiplication processing unit 703a stores scaling factors 802a, 802b, 802c and 802d. The size of scaling factors 802a, 802b, 802c and 802d is M2*N2, for example, M2=32 and N1=4. The arrangement of scaling factors 802a, 802b, 802c and 802d in the multiplication processing units 703b-703d is the same as that in the multiplication processing unit 703a.

[0122] Thus, each MPE can obtain the complete and identical scaling factors 802a, 802b, 802c, and 802d of the B matrix from its internal registers, satisfying the data synchronization requirements of parallel computing. Furthermore, the throughput of the hardware is improved through continuous storage and batch transfer.

[0123] (3) Format conversion unit reuse method

[0124] Figure 9 This is a schematic diagram of the format conversion unit of the dot product unit array according to an embodiment of this application. For example... Figure 9 As shown, the format conversion unit 510 is used to convert the data in the dot product unit array 401 into a predetermined format, wherein a predetermined number of dot product units in the dot product unit array 401 share one format conversion unit.

[0125] Since each row and column in the DP array reads the same input data, the step of converting the input data into a predetermined format can be extracted outside the DP. That is, multiple DPs in each row share a format conversion unit to convert the multiplier matrix A in the input data. For example, dot product units 403a, 403b, ..., 403c share conversion unit 510a; multiple DPs in each column share a format conversion unit to convert the multiplier matrix B in the input data. For example, dot product units 403a, 403d, ..., 403e share conversion unit 510b.

[0126] Therefore, by extracting the two format conversion units of each DP to the outside of the row or column of the DP array 401, the number of conversion units 510 in the DP array 401 is reduced, saving chip area.

[0127] The following combination Figures 4 to 9 The computational engine 400 in the document describes the computational process for data in FP8 format:

[0128] 1. The computing engine 400 receives matrix A and matrix B data containing 32 elements respectively, and the control logic unit 402 distributes the matrix A and matrix B data to each DP unit 403 of the DP array 401 respectively;

[0129] 2. Adopting, for example Figure 9 The conversion unit 510 shown converts the A matrix and B matrix data of each row and each column into a predetermined format;

[0130] 3. The first calculation unit group 404 and the second calculation unit group 405 perform multiplication calculations on the data after the format conversion, and the calculation method refers to the existing technology of floating-point calculation;

[0131] 4. The processing unit 407 aligns the data after multiplication within the first computing unit group 404 and the second computing unit group 405 respectively, and then performs floating-point addition, referring to the prior art of floating-point alignment and addition;

[0132] 5. According to, Figures 6 to 8 The scaling factor allocation method is to allocate corresponding scaling factors to the first computing unit group 404 and the second computing unit group 405, and the processing unit 407 performs quantization or dequantization processing on the data in the first computing unit group 404 and the second computing unit group 405 respectively.

[0133] 6. The processing unit 407 uses the above-mentioned floating-point alignment method of FP8 format to align the data of the first calculation unit group 404, the second calculation unit group 405 and the constant calculation unit 406, and then performs floating-point addition operations.

[0134] 7. The processing unit 407 normalizes the summation result to obtain the calculation result.

[0135] The following describes the computing engine 1100 of an embodiment of this application.

[0136] Figure 11 This is another schematic diagram of the computing engine in an embodiment of this application, such as... Figure 11 As shown, the computing engine 1100 includes a dot product unit array 1101 and a control logic unit 1102.

[0137] The dot product unit array 1101 includes multiple dot product units 1103. Each dot product unit 1103 includes a first calculation unit group 1104, a second calculation unit group 1105, a third calculation unit group 1106, a constant calculation unit 1107, and a processing unit 1108. The first calculation unit group 1104 supports dot product operations on data with a precision equal to or lower than a first precision; the second calculation unit group 1105 supports dot product operations on data with a precision equal to or lower than a second precision, where the second precision is lower than the first precision; the third calculation unit group 1106 supports dot product operations on data with a third precision, where the third precision is lower than the second precision; the constant calculation unit 1107 supports constant operations; and the processing unit 1108 processes at least two of the operation results from the first calculation unit group 1104, the second calculation unit group 1105, the third calculation unit group 1106, and the constant calculation unit 1107.

[0138] The control logic unit 1102 reads the data to be processed and transmits it to the dot product unit array 1101 in sequence according to the acquired control instructions.

[0139] exist Figure 11 In the example shown, the dot product unit 1103 includes a constant calculation unit 1107. This application is not limited to this; in other examples, the dot product unit 1103 may not have a constant calculation unit 1107. The following description will use the example of the dot product unit 1103 having a constant calculation unit 1107.

[0140] The first calculation unit group 1104, the second calculation unit group 1105, and the constant calculation unit 1107 in each dot product unit 1103 in the calculation engine 1100 are the same as the first calculation unit group 404, the second calculation unit group 405, and the constant calculation unit 406 in each dot product unit 403 in the calculation engine 400.

[0141] In some embodiments, when the data to be processed is data of third precision, the control logic unit 1102 can transmit the data of third precision to the dot product unit according to the data processing mode. The data processing mode can be a first mode, a second mode, or a third mode. The control logic unit 1102 can obtain indication information for indicating the data processing mode from control instructions and determine the data processing mode based on the indication information; alternatively, the control logic unit 1102 can be pre-configured with information related to the data processing mode and determine the data processing mode based on the pre-configured information.

[0142] In different data processing modes, the dot product unit 1103 can have different structures (i.e., implementations) to perform operations on third-precision data (e.g., dot product operations).

[0143] In different data processing modes, third-precision data can be transmitted to the third computing unit group 1106. In addition, it can also be transmitted to the first computing unit group 1104 and / or the second computing unit group 1105, so that the corresponding computing unit group can participate in the calculation of third-precision data, thereby enabling the calculation of low-precision data in a flexible manner to meet the calculation requirements.

[0144] For example, under the same bandwidth, the number of third-precision data (elements) that can be transmitted exceeds the number of second-precision elements. Therefore, when processing third-precision data, in some data processing modes, each dot product unit 1103 in the computing engine 1100 can use the first computing unit group 1104 and the second computing unit group 1105 to process a portion of the data in the third-precision data, and the remaining data in the third-precision data is processed by the third computing unit group 1106, thereby supporting the dot product operation function of the third-precision data. The first computing unit group 1104 and the second computing unit group 1105 participate in processing at least a portion of the data in the third-precision data, thereby supporting the operation of the third-precision data (e.g., dot product operation) at a lower cost and reducing the development cost of the computing engine 1100.

[0145] In some examples, the first precision may include FP16, BF16, or TF32 formats, and the second precision may be FP8 or FP4 formats. Under the same bandwidth, the number of FP4 format data (elements) that can be transmitted is twice the number of FP8 format elements. Therefore, each dot product unit 1103 in the computing engine 1100 can reuse the first computing unit group 1104 and the second computing unit group 1105 in each dot product unit of the computing engine supporting the FP8 format to process half of the data to be processed in the FP4 format (e.g., A0~A31, B0~B31), and use a third computing unit group 1106 containing 32 dot product units (DP32) to process the other half of the data to be processed (e.g., A32~A63, B32~63), thereby supporting data operations in the FP4 format.

[0146] The computing engine 1100 of this application embodiment includes a 64-point multiplication unit (DP64) that supports data operations in FP4 and above formats, and has three implementation methods. The first implementation method of the multiplication unit 1103, namely the multiplication unit 1103A, is described below.

[0147] Figure 12 This is another schematic diagram of the dot product unit in an embodiment of this application, corresponding to the first mode of data processing mode. For example... Figure 12As shown, the dot product unit 1103A includes a first calculation unit group 1104, a second calculation unit group 1105, a third calculation unit group 1106, and a constant calculation unit 1107 arranged in parallel, as well as a processing unit 1108 and a conversion unit 1201 (e.g., with the same function as the conversion unit 510), and a control logic unit 1102 (e.g., ...). Figure 11 As shown, the first, second, and third parts of the data with the third precision are input into the first calculation unit group 1104, the second calculation unit group 1105, and the third calculation unit group 1106, respectively. The processing unit 1108 integrates the calculation results of the first calculation unit group 1104, the second calculation unit group 1105, and the third calculation unit group 1106.

[0148] The first calculation unit group 1104, the second calculation unit group 1105, and the constant calculation unit 1107 have the same structure as the first calculation unit group 404, the second calculation unit group 405, and the constant calculation unit 406 in the dot product unit 403 of the calculation engine 400.

[0149] The third computational unit group 1106 is used to perform dot product operations in FP4 format.

[0150] The processing unit 1108 includes the functions of the processing unit 407 of the computing engine 400. For example, the processing unit 1108 includes an addition module 1203 and a normalization module 1204. The function of the addition module 1203 can be the same as that of the addition module 505, and the function of the normalization module 1204 can be the same as that of the normalization module 509. In addition, it also includes an alignment module 1202 that supports floating-point alignment in FP4 format.

[0151] The following describes the dot product operation method and the floating-point alignment method in FP4 format.

[0152] (1) The dot product operation method in FP4 format, which is also applicable to Figure 14 The dot product unit 1103B and Figure 15 The dot product unit 1103C:

[0153] Figure 10 This is another illustration of floating-point data format. For example... Figure 10 As shown, the FP4 format includes a 2-digit exponent and 1-digit mantissa floating-point format (E2M1), which can represent 15 values ​​with a range of -6 to 6. If negative numbers are removed, only 8 values ​​can be represented.

[0154] Because the FP4 format represents a relatively small number of values, the dot product operation (DOT32) of the 32 elements in the FP4 format can be performed using the following three methods:

[0155] Method 1: For example Figure 10 As shown, since the maximum value of data represented by the FP4 floating-point format is 6 and the minimum value is 0.5, the FP4 floating-point format is then... Figure 6 The mapping conversion is to a 3-bit integer + 1-bit fractional format, then a 4-bit multiplication operation is performed. After that, the multiplication operation of the floating-point number containing 1 bit of fractional part is performed by integer multiplication and then divided by 2 or 4.

[0156] For example, when one multiplier is a decimal and the other is an integer, the result is divided by 2; when both multipliers are decimals, the result is divided by 4.

[0157] Method 2: Use a lookup table to find the product based on the combination of the multiplier and multiplicand.

[0158] Figure 13 This is a partial schematic diagram of a lookup table according to an embodiment of this application. For example... Figure 13 As shown, since the multiplier and multiplicand each contain 8 values ​​after removing the sign bit, the product contains 64 values. Multiplication is converted into a table lookup operation. Then, the signs of the multiplier and multiplicand are XORed to obtain the sign bit of the product.

[0159] Since the product of 0 and any number is 0, we can check whether the multiplier and multiplicand are 0 before looking up the table. If they are, we don't need to look up the table and can directly output the product result as 0. Otherwise, we can look up the table, which can further reduce the size of the lookup table.

[0160] Method 3: Use a 2-bit multiplier to multiply the mantissas of the FP4 format data, and then align the result of the exponent addition to the exponent 0. If the exponent is less than 0, shift right; if the exponent is greater than 0, shift left. Thus, compared with traditional floating-point multiplication, the process of finding the maximum exponent is eliminated.

[0161] The above three FP4 data format point multiplication methods of this application replace the traditional floating-point multiplication and addition methods, reducing the number of multipliers and adders, thereby saving chip area of ​​the third computing unit group 1106.

[0162] (2) FP4 format floating-point alignment method, which is also applicable to Figure 15 The dot product unit 1103C:

[0163] After obtaining the integer multiplication and addition result or the fixed-point multiplication and addition result in the third calculation unit group 1106, it needs to be added to the multiplication and addition result of the first calculation unit group 1104 and the second calculation unit group 1105, which support FP8 format calculation. Therefore, the calculation result of the third calculation unit group 1106 needs to be converted into a predetermined format, which can also be called the internal format (internal fmt). The conversion methods include the following:

[0164] Method 1: Calculate the number of leading zeros in the accumulated result obtained from the third calculation unit group 1106. Shift the data left based on the number of leading zeros until it is converted to an internal format where the integer bits are 1. Then, subtract the number of leading zeros from the exponent bias of the internal format to obtain the final exponent value. Then, align the exponent with the results from the constant calculation unit 1107, the first calculation unit group 1104, and the second calculation unit group 1105.

[0165] Method 2: Set the exponent of the cumulative result of the third calculation unit group 1106 to the internal format exponent bias.

[0166] Compared to Method 1, Method 2 does not require calculating the number of leading zeros in the accumulated result of the third calculation unit group 1106, simplifying the calculation process. However, in Method 2, in order to maintain mantissa alignment when summing with the results of the first calculation unit group 1104 and the second calculation unit group 1105, the integer part of the accumulated result of the third calculation unit group 1106 needs to be expanded based on Method 1. For example, the result after conversion in Method 1 contains 1 integer bit, while the result after conversion in Method 2 contains 2 or 3 integer bits.

[0167] (3) Scaling factor configuration method

[0168] In the first computing unit group 1104, the dot product operation in FP16 format adopts the DOT16 hardware structure. The first half of the data in FP8 format to be processed is calculated using the first computing unit group 1104, and the second half of the data in FP8 format to be processed is calculated using the second computing unit group 1105. That is, with a single DOT16 hardware structure, the minimum supported group size can be set to 16.

[0169] exist Figure 12 Before the alignment module 1202, the results of the first calculation unit group 1104 and the second calculation unit group 1105 are multiplied by the scale factor, that is, floating-point multiplication. The scale factor is in the format of 8-bit exponent and 0-bit mantissa (E8M0) and FP8 (E4M3).

[0170] For the third computing unit group 1106, DP32 is decomposed into two DP16, so that the first computing unit group 1104, the second computing unit group 1105 and the third computing unit group 1106 can all support a scaling factor of group size 16.

[0171] In this application, scaling factors matching local data features of tensors can be set for the first computing unit group 1104, the second computing unit group 1105, and the third computing unit group 1106, respectively, which refines the granularity of the scaling factors and improves the accuracy of low-precision data operations compared with the scaling factor at the granularity of the entire tensor.

[0172] For example, for low-precision (e.g., FP4 format) data A0~A63 and B0~B63 to be processed, data segments A0~A15 and B0~B15 are assigned to the first computing unit group 1104, data segments A16~A31 and B16~B31 are assigned to the second computing unit group 1105, and data segments A32~A63 and B32~A63 are assigned to the third computing unit group 1106. Based on the numerical distribution of data segments A0~A15 and B0~B15, a corresponding scaling factor S1 can be assigned to the first computing unit group 1104. Based on the numerical distribution of segments A16~A31 and B16~B31, a corresponding scaling factor S2 is assigned to the second calculation unit group 1105. Based on the numerical distribution of data segments A32~A63 and B32~A63, a corresponding scaling factor S3 is assigned to the third calculation unit group 1106. For example, a larger scaling factor is assigned to data segments with a larger numerical distribution range, and a smaller scaling factor is assigned to data segments with a smaller numerical distribution range. Thus, compared with assigning the same scaling factor to data segments A0~A63 and B0~B63, the data accuracy of each data segment can be improved.

[0173] The following describes the computation flow in FP4 format for the dot product unit 1103A of the computation engine 1100:

[0174] 1. The computing engine 1100 receives matrix A and matrix B data containing 64 elements respectively. The control logic unit 1102 distributes the matrix A and matrix B data to each DP unit 1103 of the DP array 1101 (including the first computing unit group 1104, the second computing unit group 1105 and the third computing unit group 1106).

[0175] 2. Adopting, for example Figure 9 The conversion unit 510 shown converts the A matrix and B matrix data of each row and each column into a predetermined format;

[0176] 3. The first calculation unit group 1104, the second calculation unit group 1105 and the third calculation unit group 1106 perform multiplication calculations on the data after the format conversion, and the calculation method refers to the dot multiplication operation method of the FP4 format mentioned above;

[0177] 4. The processing unit 1108 aligns the data after multiplication operations within the first computing unit group 1104, the second computing unit group 1105, and the third computing unit group 1106, respectively, and then performs floating-point addition operations, referring to the prior art of floating-point alignment and addition.

[0178] 5. According to, Figures 6 to 8 The scaling factor allocation method is to allocate corresponding scaling factors to the first computing unit group 1104, the second computing unit group 1105 and the third computing unit group 1106 respectively, and the processing unit 1108 performs quantization or dequantization processing on the data in the first computing unit group 1104, the second computing unit group 1105 and the third computing unit group 1106 respectively.

[0179] 6. The processing unit 1108 uses the above-mentioned FP4 format floating-point alignment method to align the data of the first calculation unit group 1104, the second calculation unit group 11405 and the constant calculation unit 1107, and then performs floating-point addition operations.

[0180] 7. The processing unit 1108 normalizes the summation result to obtain the calculation result.

[0181] The allocation method of scaling factors and the reuse method of format conversion units in the quantization operation in the computing engine 1100 can refer to the corresponding methods in the computing engine 400 mentioned above.

[0182] The second implementation of the dot product unit 1103, namely the dot product unit 1103B, is introduced below.

[0183] Figure 14 This is another schematic diagram of the dot product unit in an embodiment of this application, corresponding to the second mode of the data processing mode. For example... Figure 14 As shown, the dot product unit 1103B includes a third calculation unit group 1106, an FP8 format dot product unit 403, and a processing unit 1108. The dot product unit 403 (see reference) Figure 5 ) including the first computing unit group 1104 (reference) Figure 5 First computing unit group 404), second computing unit group 1105 (reference) Figure 5 The second calculation unit group 405) and the constant calculation unit 1107 Figure 14 Not shown in the text, see reference. Figure 5 The constant calculation unit 406), control logic unit 1102 (such as the ...) Figure 11As shown, the data of the third precision is input into the third calculation unit group 1106, and the calculation result of the third calculation unit group 1106 is input as input data to the first calculation unit group 1104 and the second calculation unit group 1105. The processing unit 1108 processes the calculation result of the first calculation unit group 1104 and the calculation result of the second calculation unit group 1105.

[0184] The results of the operations of the two multipliers in the third calculation unit group 1106 are summed by the adder. The sum is then used as a multiplier and input into one of the multipliers in the first calculation unit group 1104 or the second calculation unit group 1105. The other multiplier of this multiplier is 1.

[0185] For example, the third computing unit group 1106 includes 64 multipliers that perform dot product operations (DOT2) on the input data A0~A63 and B0~B63, that is, dot product of two elements with two other elements. For example, dot product of [A0,B0] and [A1,B1] is A0*B0+A1*B1, thus obtaining 32 dot product results. The dot product operation method is the same as the dot product operation method of the dot product unit 1103A mentioned above.

[0186] The sum of the number of multipliers m in the first computing unit group 1104 and the number of multipliers m in the second computing unit group 1105 is 32. Therefore, each of the 32 dot product results obtained by FP4 is used as a multiplier C (i.e., C0, C1, ..., C31), and input into the corresponding multiplier of the 32 multipliers in the first and second computing unit groups respectively. The multipliers D (i.e., D0, D1, ..., D31) of the 32 multipliers in the first and second computing unit groups can be set to 1. Then, dot product operations are performed using these 32 multipliers and their corresponding adders in the first and second computing unit groups. Finally, the result is the sum of the products of the 64 multipliers in FP4. For example, the result of A0*B0+A1*B1 is converted into a predetermined format by converter c and used as input C0 of dot product unit 403. The constant C can be set to 1 and used as input D0 of dot product unit 403.

[0187] Therefore, without changing the FP8 format dot product unit 403, a third computing unit group 1106 is added to the front end to realize the FP4 format dot product operation function, thereby reducing the development cost of the computing engine 1100.

[0188] The following describes the computation flow in FP4 format for the dot product unit 1103B of the computation engine 1100:

[0189] 1. The computing engine 1100 receives matrix A and matrix B data containing 64 elements respectively, and the control logic unit 1102 distributes the matrix A and matrix B data to the third computing unit group 1106 of the DP array 1101 respectively;

[0190] 2. Adopting, for example Figure 9 The conversion unit 510 shown converts the A matrix and B matrix data of each row and each column into a predetermined format;

[0191] 3. The result of the operation of the third calculation unit group 1106 is used as one input of the first calculation unit group 1104 and the second calculation unit group 1105, and the other input is a constant 1. The first calculation unit group 1104 and the second calculation unit group 1105 perform the dot multiplication operation.

[0192] 4. The processing unit 1108 aligns the data after multiplication within the first computing unit group 1104 and the second computing unit group 1105, and then performs floating-point addition, referring to the prior art of floating-point alignment and addition.

[0193] 5. According to, Figures 6 to 8 The scaling factor allocation method is to allocate corresponding scaling factors to the first computing unit group 1104 and the second computing unit group 1105, and the processing unit 1107 performs quantization or dequantization processing on the data in the first computing unit group 1104 and the second computing unit group 1105 respectively.

[0194] 6. The processing unit 1108 uses the above-mentioned floating-point alignment method of FP8 format to align the data of the first calculation unit group 1104, the second calculation unit group 1105 and the constant calculation unit 1107, and then performs floating-point addition operations.

[0195] 7. The processing unit 1108 normalizes the summation result to obtain the calculation result.

[0196] The FP4 format dot product operation method and scaling factor configuration method in the dot product unit 1103B of the computing engine 1100 can refer to the corresponding method in the dot product unit 110A of the computing engine 1100.

[0197] The third implementation of the dot product unit 1103 is introduced below, namely the dot product unit 1103C.

[0198] Because the FP4 format has fewer data bits and a smaller representation range, its multipliers and adders have smaller areas. Therefore, the DP64, which supports the FP4 format, can be completely independent of the DP32, which supports the FP8 format.

[0199] Figure 15This is another schematic diagram of the dot product unit in an embodiment of this application, corresponding to the first mode of data processing mode. For example... Figure 15 As shown, the dot product unit 1103C includes a third calculation unit group 1106, a first calculation unit group 1104, a second calculation unit group 1105, a constant calculation unit 1107, and a processing unit 1108. The third calculation unit group 1106 is independent of the first calculation unit group 1104, the second calculation unit group 1105, and the constant calculation unit 1107. The control logic unit 1102 inputs all the data of the third precision into the third calculation unit group 1106, and the processing unit 1108 integrates the operation results of the third calculation unit group 1106 and the first calculation unit group 1104 and / or the operation results of the second calculation unit group 1105 and / or the operation results of the constant calculation unit 1107.

[0200] In some examples, since the precision of the input data of the first calculation unit group 1104 and the second calculation unit group 1105 is different from that of the third precision, the processing unit 1108 integrates the calculation results of the data with different precisions.

[0201] In the dot product unit 1103C, DP32 in the dot product unit 1103A or DP32B is extended to DP64. For a scaling factor with a group size of 16, a structure of 4 DP16 units is required, i.e., 4 calculation unit groups 1501. The dot product unit 1103C also includes a constant calculation unit 1502, which can have the same function as the constant calculation unit 1107.

[0202] When aligning the output results of the four computational unit groups 1501 of the third computational unit group 1106, a 2-bit multiplier can be used to multiply the mantissas of the FP4 format data, and then the result of the exponent addition can be aligned to the exponent 0. Exponents less than 0 are shifted to the right, and exponents greater than 0 are shifted to the left.

[0203] When aligning the sum of the results of the four computational unit groups 1501 of the third computational unit group 1106 with C, the floating-point alignment method of FP4 format in the above-mentioned dot product unit 1103A can be used to convert the dot product result in FP4 format into internal fmt format.

[0204] Therefore, in the dot product unit 1103C, the third calculation unit group 1106 is completely independent from the first calculation unit group 1104 and the second calculation unit group 1105, which further reduces the development cost of the calculation engine 1100.

[0205] The FP4 format dot product operation method, scaling factor configuration method, and FP4 format floating-point number alignment method in the dot product unit 1103C of the computing engine 1100 can refer to the corresponding methods in the dot product unit 110A of the computing engine 1100 mentioned above.

[0206] The following describes the computation flow in FP4 format for the dot product unit 1103C of the computation engine 1100:

[0207] 1. The computing engine 1100 receives matrix A and matrix B data containing 64 elements respectively, and the control logic unit 1102 distributes the matrix A and matrix B data to the third computing unit group 1106 of the DP array 1101 respectively;

[0208] 2. Adopting, for example Figure 9 The conversion unit 510 shown converts the A matrix and B matrix data of each row and each column into a predetermined format;

[0209] 3. The third calculation unit group 1106 performs multiplication operations on the data after format conversion;

[0210] 4. The processing unit 1108 aligns the data within the third calculation unit group 1106 after multiplication and addition operations;

[0211] 5. According to, Figures 6 to 8 The scaling factor allocation method is to allocate the corresponding scaling factor to the third computing unit group 1106, and the processing unit 1108 performs quantization or dequantization processing on the data in the third computing unit group 1106.

[0212] 6. The processing unit 1108 uses the above-mentioned FP4 format floating-point alignment method to align the data of the first calculation unit group 1104, the second calculation unit group 1105 and the constant calculation unit 1106, and then performs floating-point addition operations.

[0213] 7. The processing unit 1108 normalizes the summation result to obtain the calculation result.

[0214] Second aspect of the embodiments

[0215] An embodiment of the second aspect of this application provides a data processing method for a computing engine.

[0216] Figure 16 This is a schematic diagram of a data processing method for a computing engine according to an embodiment of this application. This method is applicable to applications such as... Figure 11 The computing engine 1100 is shown. (As shown) Figure 16 As shown, the data processing methods used in the computing engine include:

[0217] Operation 1601: Control logic unit 1102 reads the data to be processed and transmits it to the dot product unit array 1101 according to the acquired control instructions. When the data to be processed is third-precision data, control logic unit 1102 transmits the third-precision data to the dot product unit 1103 according to the data processing mode; and

[0218] Operation 1602: The first calculation unit group 1104, the second calculation unit group 1105 and the third calculation unit group 1106 perform dot multiplication on the input data respectively.

[0219] The data processing methods used in the computing engine also include:

[0220] Operation 1603: Processing unit 1108 processes at least two of the operation results of the first calculation unit group 1104, the operation results of the second calculation unit group 1105, and the operation results of the third calculation unit group 1106.

[0221] Figure 17 This is another schematic diagram of a data processing method for a computing engine according to an embodiment of this application. This method is applicable to, for example... Figure 4 The computing engine 400 is shown. (As shown in the image.) Figure 17 As shown, the data processing methods used in the computing engine include:

[0222] Operation 1701: Control logic unit 402 reads the data to be processed and transmits it to the dot product unit array 401 according to the acquired control instructions. When the data to be processed is data of second precision, control logic unit 402 transmits the data of second precision to the dot product unit 403 according to the data processing mode; and

[0223] Operation 1702: The first calculation unit group 404 and the second calculation unit group 405 perform dot multiplication on the input data respectively.

[0224] The data processing methods used in the computing engine also include:

[0225] Operation 1703: Processing unit 407 processes at least two of the operation results of the first calculation unit group 404 and the operation results of the second calculation unit group 405.

[0226] Third aspect of the embodiments

[0227] An embodiment of the third aspect of this application provides a processing chip.

[0228] Figure 18 This is a schematic diagram of a processing chip according to an embodiment of this application. Figure 18As shown, the processing chip 1800 includes one or more processor clusters (MPCs, Multi-Processor Clusters), for example, two processor clusters 1801 and 1802. In this application, the number of processor clusters included in the processing chip 1800 is not limited to this. A processor cluster can be a parallel computing architecture consisting of multiple independent computing nodes connected by an interconnect architecture. Each processor cluster (e.g., processor clusters 1801 and 1802) includes a computing engine 400 as described in the embodiments of the first aspect, which can support dot multiplication operations on data with FP8 precision and above.

[0229] Figure 19 This is another schematic diagram of the processing chip according to an embodiment of this application. For example... Figure 19 As shown, the processing chip 1900 includes one or more processor clusters, for example, two processor clusters 1901 and 1902. In this application, the number of processor clusters included in the processing chip 1900 is not limited to this. Each processor cluster (e.g., processor clusters 1901 and 1902) includes a computing engine 1100 as described in the embodiments of the first aspect, which can support dot multiplication operations on data with FP4 precision and above.

[0230] Fourth aspect of the embodiment

[0231] An embodiment of the fourth aspect of this application provides a computer device. Figure 20 This is a schematic diagram of a computer device, such as... Figure 20 As shown, the computer device 2000 includes a memory 2001 and at least one processing chip, such as a processing chip 1800. In this application, the number of processing chips included in the computer device 2000 is not limited to this. The memory 2001 stores a computer program that can run on the processing chip 1800, wherein when the processing chip 1800 executes the computer program, it can implement the embodiments of the second aspect as shown. Figure 17 The data processing method described above.

[0232] An embodiment of the fourth aspect of this application provides a computer device. Figure 21 This is another schematic diagram of a computer device, such as... Figure 21 As shown, the computer device 2100 includes a memory 2101 and at least one processing chip, such as a processing chip 1900. In this application, the number of processing chips included in the computer device 2100 is not limited to this. The memory 2101 stores a computer program that can run on the processing chip 1900, wherein when the processing chip 1900 executes the computer program, it can implement the embodiments of the second aspect, such as... Figure 16 The data processing method described above.

[0233] In the embodiments provided in this application, it should be understood that the disclosed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0234] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0235] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0236] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0237] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0238] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0239] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0240] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0241] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0242] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A computing engine, characterized in that, The computing engine includes an array of dot product units and a control logic unit, wherein, The array of dot product units includes multiple dot product units, each dot product unit comprising: The first set of computing units supports dot multiplication of data with a precision equal to or lower than the first precision. A second set of computing units supports dot product operations on data with a precision equal to or lower than a second precision, where the second precision is lower than the first precision; and The third computational unit group supports dot product operations on data with a third precision, which is lower than the second precision. The control logic unit reads the data to be processed according to the acquired control instructions and transmits it to the dot product unit array. When the data to be processed is the data of the third precision, the control logic unit transmits the data of the third precision to the dot product unit according to the data processing mode.

2. The computing engine as described in claim 1, characterized in that, Each dot product unit also includes: The processing unit processes at least two of the operation results of the first calculation unit group, the operation results of the second calculation unit group, and the operation results of the third calculation unit group.

3. The computing engine as described in claim 2, characterized in that, When the data processing mode is the first mode, the control logic unit inputs the first, second, and third portions of the third-precision data into the first calculation unit group, the second calculation unit group, and the third calculation unit group, respectively. The processing unit integrates the calculation results of the first calculation unit group, the second calculation unit group, and the third calculation unit group.

4. The computing engine as described in claim 2, characterized in that, When the data processing mode is the second mode, the control logic unit inputs the data with the third precision into the third calculation unit group. The calculation result of the third calculation unit group is input as input data to the first calculation unit group and the second calculation unit group. The processing unit processes the calculation results of the first calculation unit group and the calculation results of the second calculation unit group.

5. The computing engine as described in claim 4, characterized in that, The results of the operations of the two multipliers in the third calculation unit group are summed by the adder, and the sum is used as a multiplier and input into one of the multipliers in the first or second calculation unit group. The other multiplier of the multiplier is 1.

6. The computing engine as described in claim 2, characterized in that, When the data processing mode is the third mode, the control logic unit inputs all the data of the third precision into the third calculation unit group. The processing unit integrates the operation results of the third calculation unit group and the first calculation unit group and / or the operation results of the second calculation unit group and / or the operation results of the constant calculation unit that supports constant operation.

7. The computing engine as described in claim 1, characterized in that, The multiplication operation in the dot product operation performed by the third calculation unit group includes: For multiplication of floating-point numbers containing 1 decimal bit, perform integer multiplication followed by division by 2 or 4; Determine the result of the multiplication operation based on the correspondence between the two multipliers and their product in the lookup table; or Perform multiplication on the mantissas of the two multipliers, add the exponents of the two multipliers, and align the exponents after the addition to 0.

8. The computing engine as described in claim 3 or 6, characterized in that, The processing unit: The number of leading zeros in the operation result of the third calculation unit group is calculated, and the result is converted into a predetermined format by shifting. An exponent value is then calculated, and the exponent value is aligned with the operation results of at least one of the first calculation unit group, the second calculation unit group, and a constant calculation unit that supports constant operations; or... While keeping the format of the exponent bits of the calculation result in the third calculation unit group unchanged, the integer bits of the calculation result are increased.

9. The computing engine as described in claim 1, characterized in that, The control logic unit is also configured to perform the following operations: Based on the supported matrix size and scaling factor grouping size in the control instructions, calculate the register address of the scaling factor that the dot product unit array needs to read; The scaling factor is read and stored in the storage module of the computing engine; as well as The stored scaling factor is sent to the first calculation unit group, the second calculation unit group, and the third calculation unit group of each of the dot product units.

10. The computing engine as described in claim 1, characterized in that, The computing engine also includes: A format conversion unit is used to convert the data in the dot product unit array into a predetermined format. In this configuration, a predetermined number of dot product units in the dot product unit array share one format conversion unit.

11. A processing chip, characterized in that, The processing chip includes one or more processor clusters, each of the processor clusters including a computing engine as described in any one of claims 1 to 10.

12. A data processing method for a computing engine, characterized in that, The computing engine includes an array of dot product units and a control logic unit, wherein, The array of dot product units includes multiple dot product units, each dot product unit comprising: The first set of computing units supports dot multiplication of data with a precision equal to or lower than the first precision. A second set of computing units supports dot product operations on data with a precision equal to or lower than a second precision, where the second precision is lower than the first precision; and The third computational unit group supports dot product operations on data with a third precision, which is lower than the second precision. The data processing method includes: The control logic unit reads the data to be processed and transmits it to the dot product unit array according to the acquired control instructions. When the data to be processed is data of the third precision, the control logic unit transmits the data of the third precision to the dot product unit according to the data processing mode. The first calculation unit group, the second calculation unit group, and the third calculation unit group each perform dot multiplication on the input data.

13. A computer device comprising a memory and at least one processing chip, the memory storing a computer program executable on the processing chip, characterized in that, When the processing chip executes the computer program, it implements the data processing method as described in claim 12.

Images