A model processing method and apparatus

Abstract

A model processing method and apparatus, relating to the field of computer technology, is used to reduce the data bit width during AI computation. The method includes: during model training or inference, acquiring first data information in a first data format; converting the first data information into second data information corresponding to a second data format, and using the second data information for training or inference; one of the first data format and the second data format is a single-float format, and the other is a block-float format; the block-float format data information includes a-bit first shared exponent, b-bit second shared exponent, c-bit third shared exponent, and M data; the M data share the first shared exponent; the (i-1)×M / b+1 to the i×M / b data in the M data also share the i-th bit of the second shared exponent; the (j-1)×M / c+1 to the j×M / c data in the M data also share the j-th bit of the third shared exponent.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model processing method and apparatus. Background Technology

[0002] The performance of artificial intelligence (AI) computing is constrained by both algorithms and hardware. On the algorithm side, the scaling law states that the three main factors affecting model performance are computational cost, dataset size, and the number of model parameters; when not constrained by the other two factors, model performance is directly proportional to the logarithm of each individual factor, i.e., a power-law relationship. On the hardware side, with Moore's Law slowing and the benefits of computing architecture / instruction evolution gradually diminishing, low-bit data formats have become an important direction for improving hardware computing power and energy efficiency, and reducing data storage and transfer overhead.

[0003] In AI computing, low-bit data formats have evolved from 32-bit floating-point FP32 to 16-bit floating-point BF16 and FP16, and then from 16-bit floating-point FP16 to 8-bit floating-point FP8 and HiF8. However, when the data width is reduced to below 8 bits, it becomes virtually impossible to simultaneously achieve encoding of a single data point while maintaining data precision and dynamic range. Summary of the Invention

[0004] This application provides a model processing method and apparatus for reducing the bit width of data while taking into account the data accuracy and dynamic range of AI calculations.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] Firstly, a model processing method is provided, comprising: acquiring first data information in a first data format during the training or inference process of an artificial intelligence model; converting the first data information into second data information corresponding to a second data format; and using the second data information for training or inference of the artificial intelligence model; wherein one of the first data format and the second data format is a single-floating-point format and the other is a block-floating-point format; the data information corresponding to the block-floating-point format includes a-bit first shared exponent, b-bit second shared exponent, c-bit third shared exponent, and M signed bits. The original code data, where a, b, c, and M are positive integers, and c is a divisor of M and b is a divisor of c; the M signed original code data share the first shared exponent with bit a; the (i-1)×M / b+1th to the i×M / bth signed original code data in the M signed original code data also share the i-th bit in the second shared exponent, where i ranges from 1 to b; the (j-1)×M / c+1th to the j×M / cth signed original code data in the M signed original code data also share the j-th bit in the third shared exponent, where j ranges from 1 to c.

[0007] In the above technical solution, during the training or inference process of the artificial intelligence model, by converting between block floating-point format data information and single floating-point format (e.g., BF16) data information, the bit width of the data can be reduced while taking into account the data accuracy and dynamic range of AI calculations during the training or inference process.

[0008] In one possible implementation of the first aspect, the a-bit first shared exponent comprises an 8-bit exponent value, the b-bit second shared exponent comprises eight 1-bit exponent values, and the c-bit third shared exponent comprises sixteen 1-bit exponent values, where M equals 64; and / or, in one possible implementation of the first aspect, the signed original code data comprises a 1-bit sign bit, a 1-bit integer bit, and N bits of fractional bits, where N is an integer ranging from 0 to 8; and / or, the single-floating-point format is one of BF16, FP16, and FP32. In the above possible implementations, the fixed shared exponent amortization is 0.5-bit / value, compared to 0.25-bit / value for MXFP4 and 1-bit / value for MX6. This block floating-point format ensures both achievable precision and sufficient bandwidth gain. The signed original code data within the group can support 5 bits / value. Combined with the overhead of a fixed shared exponent of 0.5 bits / value, the maximum overall overhead is 5.5 bits / value. Compared to MXFP8 (8.25 bits / value), this represents a bandwidth gain of at least 8.25 / 5.5 = 1.5x. In this application, the group size of the block floating-point data can reach 64, meaning the group size is large enough to maximize the degree of fixed-point normalization and achieve good matrix multiplication energy efficiency, which is crucial for LLM training and inference. The three-level shared exponent design uses 8 bits to represent the first-level exponent, and 8 bits for the second-level and 16 bits for the third-level exponent differences in the local data. This results in an exponent representation overhead ratio of base / local = 1:3, significantly enhancing the representation of local data differences and improving the end-to-end training and inference accuracy of block floating-point data.

[0009] In one possible implementation of the first aspect, when the first data format is a single-float format and the second data format is a block-float format, the first data information includes M single-float data. Converting the first data information into the second data information includes: determining c first exponents, b second exponents, and a third exponent based on the exponent values ​​of the M single-float data included in the first data information; wherein any one of the c first exponents is determined from the (j-1)×M / c+1th single-float data to the j×th single-float data in the M single-float data. The exponent of the M / c single-floating-point data is determined by the maximum value of the first exponent; the b second exponents are determined by the maximum value of the c / b consecutive first exponents, and the third exponent is determined by the maximum value of the b second exponents; based on the c first exponents, b second exponents, and the third exponent, the first shared exponent, the second shared exponent, and the third shared exponent are determined; based on the first shared exponent, the second shared exponent, and the third shared exponent, the M single-floating-point data are converted into M signed original code data to obtain the second data information. The above possible implementation can reduce the data bit width while maintaining the data precision and dynamic range of AI computation.

[0010] In one possible implementation of the first aspect, the first shared exponent SE1, the k-th bit SE2(k) of the second shared exponent, and the k'-th bit SE3(k') of the third shared exponent satisfy the following formulas: SE1 = Emax - 2, where Emax represents the third exponent; SE2(k) = (Ebmax(k) == Emax) ? 1 : 0, where Ebmax(k) represents the k-th bit among b second exponents, and the value of k ranges from 1 to b; SE3(k') = {[Ecmax(k') - Ecmax(k') represents the k'th first exponent out of c first exponents. This indicates rounding up, and the value of k' ranges from 1 to c.

[0011] In one possible implementation of the first aspect, when the first data format is block floating-point format and the second data format is single floating-point format, converting the first data information into the second data information includes: determining the exponent value corresponding to each of the M signed original code data based on the first shared exponent, the second shared exponent, and the third shared exponent; and converting the M signed original code data into M single floating-point data based on the M signed original code data and the exponent of each of the M signed original code data to obtain the second data information. The above possible implementation enables flexible conversion between block floating-point format data information and single floating-point format (e.g., BF16) data information during model training or inference.

[0012] Secondly, a model processing method is provided, comprising: during the training or inference process of an artificial intelligence model, acquiring first data information and second data information of the first target layer of the artificial intelligence model; wherein the data format of the first data information and the second data information are both block floating-point format, and both the first data information and the second data information include a-bit first shared exponent, b-bit second shared exponent, c-bit third shared exponent, and M signed original code data, where a, b, c, and M are positive integers, and c is a divisor of M and b is a divisor of c; the M signed original code data share a-bit first shared exponent; the M signed original code data... The (i-1)×M / b+1th denoted original code data to the i×M / bth denoted original code data in the data also share the i-th bit in the second shared exponent, where the value of i ranges from 1 to b; the (j-1)×M / c+1th denoted original code data to the j×M / cth denoted original code data in the M denoted original code data also share the j-th bit in the third shared exponent, where the value of j ranges from 1 to c; the inner product calculation result of the first data information and the second data information is calculated, and this inner product calculation result is used for the training or inference of the second target layer of the artificial intelligence model, where the first target layer and the second target layer are two adjacent layers in the artificial intelligence model.

[0013] In the above technical solution, during the training or inference process of the artificial intelligence model, the calculation of any target layer of the artificial intelligence model can be performed by performing inner product calculation on the first data information and the second data information, both of which are in block floating-point format. This can reduce the bit width of the data while taking into account the data accuracy and dynamic range of AI calculation during the training or inference process.

[0014] In one possible implementation of the second aspect, the first shared exponent of bit a is an 8-bit exponent value, the second shared exponent of bit b is eight 1-bit exponent values, and the third shared exponent of bit c is sixteen 1-bit exponent values, where M equals 64; and / or, the signed original code data includes one integer bit, one integer bit, and N fractional bits, where N is an integer ranging from 0 to 8. In the above possible implementations, the fixed shared exponent amortization is 0.5 bits per value, compared to 0.25 bits per value for MXFP4 and 1 bit per value for MX6. This block floating-point format ensures both achievable precision and sufficient bandwidth gain. The signed original code data within the group can support 5 bits / value. Combined with the overhead of a fixed shared exponent of 0.5 bits / value, the maximum overall overhead is 5.5 bits / value. Compared to MXFP8 (8.25 bits / value), this represents a bandwidth gain of at least 8.25 / 5.5 = 1.5x. In this application, the group size of the block floating-point data can reach 64, meaning the group size is large enough to maximize the degree of fixed-point normalization and achieve good matrix multiplication energy efficiency, which is crucial for LLM training and inference. The three-level shared exponent design uses 8 bits to represent the first-level exponent, and 8 bits for the second-level and 16 bits for the third-level exponent differences in the local data. This results in an exponent representation overhead ratio of base / local = 1:3, significantly enhancing the representation of local data differences and improving the end-to-end training and inference accuracy of block floating-point data.

[0015] In one possible implementation of the second aspect, calculating the inner product of the first data information and the second data information includes: multiplying the M signed original code data in the first data information one-to-one with the M signed original code data in the second data information to obtain M first data products; adding the b-bit second shared exponent and c-bit third shared exponent in the first data information, and the b-bit second shared exponent and c-bit third shared exponent in the second data information, according to their correspondence, to obtain c first exponent sums; shifting the M first data products based on the c first exponent sums to obtain M second data products; accumulating the M second data products to obtain a cumulative sum; adding the a-bit first shared exponent in the first data information and the a-bit first shared exponent in the second data information to obtain a-bit second exponent sum; and determining the inner product calculation result of the first data information and the second data information based on the cumulative sum and the a-bit second exponent sum. Among the above possible implementations, accuracy can be improved while ensuring bandwidth gains; the bit width per value of block floating-point can be reduced to obtain greater bandwidth gains while ensuring availability; and greater matrix multiplication can be obtained to achieve energy efficiency gains while ensuring availability and bandwidth gains.

[0016] In one possible implementation of the second aspect, calculating the inner product of the first data information and the second data information includes: multiplying the M signed original code data in the first data information one-to-one with the M signed original code data in the second data information, and adding two adjacent first data products among the M first data products obtained by multiplication to obtain M / 2 second data products; and combining the b-bit second shared exponent and c-bit third shared exponent in the first data information with the b-bit second shared exponent and c-bit third shared exponent in the second data information. The third shared exponent is added according to the corresponding relationship to obtain c first exponent sums; based on the c first exponent sums, the M / 2 second data products are shifted to obtain M / 2 third data products; the M / 2 third data products are accumulated to obtain a product sum; the a-bit first shared exponent in the first data information and the a-bit first shared exponent in the second data information are added to obtain the a-bit second exponent sum; based on this product sum and the a-bit second exponent sum, the inner product calculation result of the first data information and the second data information is determined. Among the above possible implementations, accuracy can be improved while ensuring bandwidth gain; the bit width per value of block floating-point can be reduced to obtain greater bandwidth gain while ensuring availability; and greater matrix multiplication can be obtained to achieve energy efficiency gain while ensuring availability and bandwidth gain.

[0017] In one possible implementation of the second aspect, calculating the inner product of the first data information and the second data information includes: multiplying the M signed original code data in the first data information one-to-one with the M signed original code data in the second data information, and adding the four adjacent first data products among the M first data products obtained by multiplication to obtain M / 4 second data products; and combining the b-bit second shared exponent and c-bit third shared exponent in the first data information with the b-bit second shared exponent and c-bit third shared exponent in the second data information. The third shared exponent is added according to the corresponding relationship to obtain c first exponent sums; based on the c first exponent sums, the M / 4 second data products are shifted to obtain M / 4 third data products; the M / 4 third data products are accumulated to obtain a product sum; the a-bit first shared exponent in the first data information and the a-bit first shared exponent in the second data information are added to obtain the a-bit second exponent sum; based on this product sum and the a-bit second exponent sum, the inner product calculation result of the first data information and the second data information is determined. Among the above possible implementations, accuracy can be improved while ensuring bandwidth gain; the per-bit width of block floating-point can be reduced to obtain greater bandwidth gain while ensuring availability; and greater matrix multiplication can be obtained to achieve energy efficiency gain while ensuring availability and bandwidth gain.

[0018] Thirdly, a model processing apparatus is provided, comprising: a receiving unit for acquiring first data information in a first data format during the training or inference process of an artificial intelligence model; and a processing unit for converting the first data information into second data information corresponding to a second data format, and using the second data information for training or inference of the artificial intelligence model; wherein one of the first data format and the second data format is a single-float format and the other is a block-float format; the data information corresponding to the block-float format includes a bits of first shared exponent, b bits of second shared exponent, and c bits of third shared exponent. The system consists of an exponent and M signed original code data, where a, b, c, and M are positive integers, and c is a divisor of M and b is a divisor of c. The M signed original code data share a first shared exponent with a bits. The (i-1)×M / b+1th to the i×M / bth signed original code data in the M signed original code data also share the i-th bit in the second shared exponent, where i ranges from 1 to b. The (j-1)×M / c+1th to the j×M / cth signed original code data in the M signed original code data also share the j-th bit in the third shared exponent, where j ranges from 1 to c.

[0019] In one possible implementation of the third aspect, the first shared exponent of a bits comprises an 8-bit exponent value, the second shared exponent of b bits comprises eight 1-bit exponent values, the third shared exponent of c bits comprises sixteen 1-bit exponent values, and M equals 64.

[0020] In one possible implementation of the third aspect, the signed original code data includes 1 sign bit, 1 integer bit, and N fractional bits, where N is an integer ranging from 0 to 8.

[0021] In one possible implementation of the third aspect, the single-floating-point format is one of BF16, FP16, or FP32.

[0022] In one possible implementation of the third aspect, when the first data format is a single-float format and the second data format is a block floating-point format, the first data information includes M single-float data. The processing unit is further configured to: determine c first exponents, b second exponents, and a third exponent based on the exponent values ​​of the M single-float data included in the first data information; wherein any one of the c first exponents is determined based on the (j-1)×M / c+1th single-float data to the j×M / cth single-float data in the M single-float data. The maximum value of the exponent of the point data is determined; b second exponents are determined based on the maximum value of c / b consecutive first exponents; the third exponent is determined based on the maximum value of b second exponents; based on c first exponents, b second exponents, and the third exponent, the first shared exponent, the second shared exponent, and the third shared exponent are determined; based on the first shared exponent, the second shared exponent, and the third shared exponent, M single-floating-point data are converted into M signed original code data to obtain the second data information.

[0023] In one possible implementation of the third aspect, the first shared exponent SE1, the k-th bit SE2(k) of the second shared exponent, and the k'-th bit SE3(k') of the third shared exponent satisfy the following formulas: SE1 = Emax - 2, where Emax represents the third exponent; SE2(k) = (Ebmax(k) == Emax) ? 1 : 0, where Ebmax(k) represents the k-th bit among b second exponents, and the value of k ranges from 1 to b; SE3(k') = {[Ecmax(k') - Ecmax(k') represents the k'th first exponent out of c first exponents. This indicates rounding up, and the value of k' ranges from 1 to c.

[0024] In one possible implementation of the third aspect, when the first data format is a block floating-point format and the second data format is a single floating-point format, the processing unit is further configured to: determine the exponent value corresponding to each of the M signed original code data according to the first shared exponent, the second shared exponent, and the third shared exponent; and convert the M signed original code data into M single floating-point data according to the M signed original code data and the exponent of each of the M signed original code data to obtain the second data information.

[0025] Fourthly, a data processing device is provided for calculating the inner product of first data information and second data information in block floating-point format. Both the first and second data information include a-bit first shared exponent, b-bit second shared exponent, c-bit third shared exponent, and M signed original code data, where a, b, c, and M are positive integers, and c is a divisor of M and b is a divisor of c. The M signed original code data share the a-bit first shared exponent. The (i-1)×M / b+1th to the i×M / bthth signed original code data in the M signed original code data also share the i-th bit of the second shared exponent, where i ranges from 1 to b. The (j-1)×M / c+1th to the j×M / cthth signed original code data in the M signed original code data also share the j-th bit of the third shared exponent, where j ranges from 1 to c. The device includes a fixed-point multiplier for multiplying the M data in the first data information... The signed original code data is multiplied one-to-one with the M signed original code data in the second data information to obtain M first data products; a first fixed-point adder is used to add the b-bit second shared exponent and c-bit third shared exponent in the first data information, and the b-bit second shared exponent and c-bit third shared exponent in the second data information, according to the correspondence, to obtain c first exponent sums; a shifter is used to shift the M first data products according to the c first exponent sums to obtain M second data products; a fixed-point accumulator tree is used to accumulate the M second and third data products to obtain a product sum; a second fixed-point adder is used to add the a-bit first shared exponent in the first data information and the a-bit first shared exponent in the second data information to obtain the a-bit second exponent sum; a floating-point accumulator is used to determine the inner product calculation result of the first data information and the second data information according to the product sum and the a-bit second exponent sum.

[0026] Fifthly, a data processing device is provided for calculating the inner product of first data information and second data information in block floating-point format. Both the first and second data information include a-bit first shared exponent, b-bit second shared exponent, c-bit third shared exponent, and M signed original code data, where a, b, c, and M are positive integers, and c is a divisor of M and b is a divisor of c; the M signed original code data share a-bit first shared exponent; the (i-th)th of the M signed original code data... 1) The M × M / b+1 signed original code data to the i × M / b signed original code data also share the i-th bit in the second shared exponent, where i ranges from 1 to b; the (j-1) × M / c+1 signed original code data to the j × M / c signed original code data in the M signed original code data also share the j-th bit in the third shared exponent, where j ranges from 1 to c; the device includes: a fixed-point fusion multiplier, used to combine the M signed original code data in the first data information with the M signed original code data in the second data information. The signed original code data is multiplied one-to-one, and adjacent first data products are added to obtain M / 2 second data products; a first fixed-point adder is used to add b bits of the second shared exponent and c bits of the third shared exponent in the first data information, and b bits of the second shared exponent and c bits of the third shared exponent in the second data information, according to the correspondence, to obtain c first exponent sums; a shifter is used to shift the M / 2 second data products according to the c first exponent sums to obtain M / 2 third data products; a fixed-point accumulator is used to accumulate the M / 2 third data products to obtain a multiplicative sum; a second fixed-point adder is used to add a bits of the first shared exponent in the first data information and a bits of the first shared exponent in the second data information to obtain a bit of the second exponent sum; a floating-point accumulator is used to determine the inner product calculation result of the first data information and the second data information according to the multiplicative sum and the a bit of the second exponent sum.

[0027] Sixthly, a data processing device is provided for calculating the inner product of first data information and second data information in block floating-point format. Both the first and second data information include a-bit first shared exponent, b-bit second shared exponent, c-bit third shared exponent, and M signed original code data, where a, b, c, and M are positive integers, and c is a divisor of M and b is a divisor of c; the M signed original code data share a-bit first shared exponent; the (i-th)th of the M signed original code data... 1) The M × M / b+1 signed original code data to the i × M / b signed original code data also share the i-th bit in the second shared exponent, where i ranges from 1 to b; the (j-1) × M / c+1 signed original code data to the j × M / c signed original code data in the M signed original code data also share the j-th bit in the third shared exponent, where j ranges from 1 to c; the device includes: a fixed-point fusion multiplier, used to combine the M signed original code data in the first data information with the M signed original code data in the second data information. The signed original code data is multiplied one-to-one, and the four adjacent first data products obtained from the multiplication are added to obtain M / 4 second data products; a first fixed-point adder is used to add the b-bit second shared exponent and c-bit third shared exponent of the first data information, and the b-bit second shared exponent and c-bit third shared exponent of the second data information according to the correspondence, to obtain c first exponent sums; a shifter is used to shift the M / 4 second data products according to the c first exponent sums to obtain M / 4 third data products; a fixed-point accumulator is used to accumulate the M / 4 third data products to obtain a multiplicative sum; a second fixed-point adder is used to add the a-bit first shared exponent of the first data information and the a-bit first shared exponent of the second data information to obtain the a-bit second exponent sum; a floating-point accumulator is used to determine the inner product calculation result of the first data information and the second data information according to the multiplicative sum and the a-bit second exponent sum.

[0028] In another aspect of this application, a model processing apparatus is provided, the apparatus including a processor and a memory storing a computer program, the processor executing the computer program stored in the memory to cause the apparatus to perform: the method provided by the first aspect or any possible implementation thereof, or the method provided by the second aspect or any possible implementation thereof.

[0029] In another aspect of this application, a computer-readable storage medium is provided, which stores a computer program or instructions that, when executed, implement the method provided by the first aspect or any possible implementation thereof.

[0030] In another aspect of this application, a computer-readable storage medium is provided, which stores a computer program or instructions that, when executed, implement the method provided by the second aspect or any possible implementation thereof.

[0031] In another aspect of this application, a computer program product is provided, comprising: a computer program (also referred to as code or instructions) that, when run, causes a computer to perform the methods provided by the second aspect or any possible implementation thereof.

[0032] In another aspect of this application, a computer program product is provided, comprising: a computer program (also referred to as code or instructions) that, when run, causes a computer to perform the methods provided by the first aspect or any possible implementation thereof.

[0033] Understandably, the beneficial effects achieved by any of the devices, computer-readable storage media, and computer program products provided above can be referred to in accordance with the beneficial effects of the methods provided above, and will not be repeated here. Attached Figure Description

[0034] Figure 1 A schematic diagram of a block floating-point format provided for an embodiment of this application;

[0035] Figure 2 A schematic diagram illustrating another block floating-point format provided in an embodiment of this application;

[0036] Figure 3 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0037] Figure 4 A schematic diagram of an AI training scenario provided in an embodiment of this application;

[0038] Figure 5 A schematic flowchart illustrating a model processing method provided in an embodiment of this application;

[0039] Figure 6 A schematic diagram of data information corresponding to a block floating-point format provided in an embodiment of this application;

[0040] Figure 7 A flowchart illustrating another model processing method provided in an embodiment of this application;

[0041] Figure 8 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0042] Figure 9 This is a schematic diagram of another data processing device provided in an embodiment of this application;

[0043] Figure 10 This is a schematic diagram of the structure of another data processing device provided in the embodiments of this application;

[0044] Figure 11 This is a schematic diagram of the structure of a model processing device provided in an embodiment of this application. Detailed Implementation

[0045] The technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings. In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, or B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship, and when applied to formulas, it represents a division operation. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, a and b, a and c, b and c, a, b, and c; where a, b, and c can be single or multiple.

[0046] The embodiments of this application use terms such as "first" and "second" to distinguish objects with similar names, functions, or roles. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or order of execution. In this application, words such as "exemplary" or "for example" are used to indicate that something is being used as an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0047] In the embodiments of this application, the exponent in the data format can also be called the exponent bit, and the mantissa can also be called the fractional bit. Furthermore, the data formats involved in the embodiments of this application can include ExMy and SxPy. For the data format ExMy, E represents the exponent and M represents the mantissa; for the data format SxPy, S represents the sign bit, x represents the bit width of the integer bits, and P represents the bit width of the fractional bits. Here, bit width can refer to the bit width. E1M2 and S1P2 are equivalent; E1M2 includes a 1-bit exponent, a 2-bit mantissa, and a 1-bit sign bit, while S1P2 includes a 1-bit sign bit, a 1-bit integer bit, and 2-bit fractional bits.

[0048] Before introducing the embodiments of this application, the relevant technologies involved in this application will be described first.

[0049] The performance of artificial intelligence (AI) computing is constrained by both algorithms and hardware. On the algorithm side, the scaling law states that the three main factors affecting model performance are computational cost, dataset size, and the number of model parameters; when not constrained by the other two factors, model performance is directly proportional to the logarithm of each individual factor, i.e., a power-law relationship. On the hardware side, with Moore's Law slowing and the benefits of computing architecture / instruction evolution gradually diminishing, low-bit data formats have become an important direction for improving hardware computing power and energy efficiency, and reducing data storage and transfer overhead.

[0050] In AI computing, floating-point (FP) data formats are typically used. Low-bit data formats have evolved from 32-bit FP32 to 16-bit BF16 and FP16, and then from 16-bit FP16 to 8-bit FP8 and HiF8. However, when the data width is reduced to below 8 bits, it becomes virtually impossible to simultaneously maintain numerical precision and dynamic range while encoding a single data point.

[0051] For training large language models (LLMs), computational resources are generally a constraint. Using low-bit data formats, computational resources can be doubled, thereby simultaneously improving training performance and energy efficiency. For LLM incremental inference, bandwidth is generally a constraint. Utilizing low-bit data formats is equivalent to amplifying the bandwidth of off-chip storage and the core, meaning that the same bandwidth can transmit more data, thus improving the performance of LLM incremental inference.

[0052] In AI computing, low-bit data formats have evolved from 32-bit floating-point FP32 to 16-bit floating-point BF16 and FP16, significantly improving training and inference performance. Then, they evolved from 16-bit floating-point FP16 to 8-bit floating-point FP8 and HiF8, gradually adapting to training and LLM inference. However, when the data width is reduced to below 8 bits, it becomes crucial to balance numerical precision and dynamic range while encoding individual data points to meet the needs of training and inference.

[0053] Traditional floating-point data formats typically assign a single code to each data item. Block floating-point (BFP) formats leverage data locality, using lower bits to represent shared exponent information and distributing the overhead across a group of data. This results in a relatively small exponent information overhead for each data item, typically less than or equal to 1 bit. Therefore, BFP formats hold promise for further reducing data bit width.

[0054] In one example, Figure 1 A schematic diagram of a block floating-point format is shown. In this format, 32 data points are grouped together, sharing an 8-bit exponent represented as E8, with each data point having an amortized value of 0.25 bits. The 32 data points within a group can use the 4-bit format E2M1, also known as MXFP4 or FP4, in which case the average is 4.25 bits / data point; or, the 32 data points within a group can use the 6-bit format E2M3, also known as MXFP6 or FP6, in which case the average is 6.25 bits / data point. Figure 1 Each of the 32 data points is represented as E2M1 or E2M3.

[0055] In the example above, 32 data points share a single exponent E8, which severely impacts the ability of large data points to represent the effective bits of smaller data points. Thus, when using low-bit data within a group, it's difficult to achieve good accuracy; while using high-bit data within a group can guarantee accuracy, it incurs high overhead and prevents the computational power from doubling.

[0056] In another possible example, Figure 2A schematic diagram of another block floating-point format is shown. In this block floating-point format, 16 data points are grouped together, corresponding to a first-level shared exponent and a second-level shared exponent. The first-level shared exponent is an 8-bit exponent represented as E8, and the second-level shared exponent consists of eight 1-bit exponents, each represented as E1. Every two adjacent data points in the group share one E1 from the second-level shared exponent. The 16 data points in each group use the S1P3 data format, which is signed-magnitude encoded, including a 1-bit sign bit, a 1-bit integer bit, and 3 bits of fractional part. S1P3 can also be called MX6, meaning each data point in the group is 5 bits, and the shared exponent is amortized to 1 bit per data point, resulting in an average of 6 bits per data point.

[0057] In the example above, the 16 values ​​share a 16-bit exponent, which is a combination of E8 + 8 × E1. The exponent difference between any two adjacent data within the group is either 0 or 1, thereby improving the local data effective bit representation capability of block floating-point data. However, the 16 data within a group greatly limits the fixed-point nature of block floating-point operations, significantly reducing energy efficiency gains when used for matrix multiplication. Furthermore, the bit width gain is smaller compared to MXFP8 (average 8.25 bits / data), while the computational power is doubled, resulting in a substantial increase in overhead.

[0058] Therefore, how to design a block floating-point format to ensure that AI training and inference achieve good results in terms of end-to-end accuracy, hardware overhead, and performance is a technical problem that urgently needs to be solved. Based on this, embodiments of this application provide a model-based processing method that, when using block floating-point data for AI training and AI inference, can reduce the per-bit width of block floating-point data, improve end-to-end accuracy, and simultaneously ensure good bandwidth and energy efficiency benefits.

[0059] The technical solutions provided in this application can be applied to various computing devices, such as terminals, network devices, or servers, which are devices with computing capabilities. Optionally, in practical applications, the computing device may include, but is not limited to: mobile phones, tablets, laptops, desktop computers, handheld computers, ultra-mobile personal computers (umPCs), mobile internet devices (MIDs), netbooks, cameras, camcorders, wearable devices, in-vehicle devices (e.g., cars, bicycles, electric vehicles, airplanes, ships, trains, high-speed trains, etc.), virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, smart home devices (e.g., refrigerators, televisions, air conditioners, electricity meters, etc.), intelligent robots, workshop equipment, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, or wireless terminals in smart homes, or flying devices (e.g., intelligent robots, hot air balloons, drones, airplanes), etc.

[0060] For example, Figure 3 This is a schematic diagram of a computing device provided in this embodiment. The computing device may include a memory 101, a processor 102, a communication interface 103, and a bus 104. The memory 101, the processor 102, and the communication interface 103 are interconnected through the bus 104.

[0061] The memory 101 can be used to store data, software programs, and modules, mainly including a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc., and the data storage area can store data created during the use of the device. For example, the operating system may include Linux, Unix, or Windows operating systems; the applications (APPs) required for the at least one function may include AI-related APPs, high-performance computing (HPC)-related APPs, deep learning-related APPs, or computer graphics (CG)-related APPs, etc. In one possible example, the memory 101 includes, but is not limited to, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), or high-speed random access memory, etc. Furthermore, the memory 101 may also include other non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0062] Additionally, processor 102 is used to control and manage the operation of the computing device, such as by running or executing software programs and / or modules stored in memory 101, and by calling data stored in memory 101, to perform various functions of the computing device and process data. In one possible embodiment, processor 102 includes, but is not limited to, a central processing unit (CPU), a network processing unit (NPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, logic circuits, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 102 can also be a combination of computing functions, such as including one or more microprocessor combinations, digital signal processors, and microprocessors. In one embodiment, processor 102 may include a single instruction multiple data (SMID) processor, or a single instruction multiple threads (SIMT) processor, etc. In another embodiment, the processor 102 may include a vector computation unit, a format conversion unit, or a matrix transpose unit, etc. In yet another embodiment, the processor 102 may include one or more CPUs, GPUs, or NPUs, etc.

[0063] The communication interface 103 is used to enable communication between the computing device and external devices. The communication interface 103 may include an input interface and an output interface. The input interface can be used to acquire data information, such as data information in block floating-point format or other data formats. In some feasible embodiments, the input interface may have only one input interface or multiple input interfaces. The output interface can be used to output the calculation results in the method embodiments below. In some feasible embodiments, the calculation results may be directly output by the processor or stored in memory first and then output via memory. In other feasible embodiments, there may be only one output interface or multiple output interfaces.

[0064] Bus 104 can be a Peripheral Component Interconnect Express (PCIe) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Optionally, bus 104 may include an address bus, a data bus, and a control bus, etc.

[0065] The technical solutions provided in this application can be applied to various application scenarios, including but not limited to AI training scenarios or AI inference scenarios. For example, Figure 4 This diagram illustrates an AI training scenario, using the i-th layer of an AI model as an example. The data formats used are BF16, BF32, HiF5, and HiFx. The processing for the i-th layer during AI training can include forward and backward processing. For example, the forward processing can include: converting A_(i-1) in BF16 format to A_(i-1) in HiF5 format using a first format conversion unit; converting W_i in BF16 format to W_i in HiFx format using a second format conversion unit; performing matrix multiplication on A_(i-1) in HiF5 format and W_i in HiFx format using a matrix multiplication unit to obtain a BF16 format result; and performing non-linear vector calculations on the BF16 format result using a vector calculation unit to obtain A_i in BF16 format. For example, the backward processing may include: transposing A_(i-1) in BF16 format using a transpose unit, and converting the transposed A_(i-1) in BF16 format using a first format conversion unit. T Convert _(i-1) to HiF5 format A T _(i-1); The second format conversion unit converts E_i in BF16 format to E_i in HiF5 format; the matrix multiplication unit performs matrix multiplication on A in HiF5 format. T _(i-1) is calculated with E_i in HiF5 format to obtain G_i in FP32 format; the FP32 format G_i and FP32 format W are then optimized by the optimizer. g The variable _i is optimized to obtain W_i in BF16 format; the BF16 format W_i is transposed by the transpose unit, and the transposed W is converted by the second format conversion unit. T Convert _i to HiFx format W T_i; The first format conversion unit converts E_i in BF16 format to E_i in HiF5 format; The matrix multiplication unit performs matrix multiplication on E_i in HiF5 format and W_i in HiFx format. T _i is calculated to obtain the calculation result in BF16 format; the calculation result in BF16 format is then subjected to nonlinear vector calculation through the vector calculation unit to obtain E_(i-1) in BF16 format.

[0066] A_(i-1) and E_(i-1) above represent the activation data and error data of the (i-1)th layer, respectively; A_i, W_i, G_i, E_i and W g _i represents the activation data, weight data, weight gradient data, error data, and optimization data of the i-th layer, respectively. The first and second format conversion units described above support mutual conversion between BF16 and HiFx formats, and also support matrix quantization or format conversion by row and column. The matrix multiplication unit described above supports multiplying a row-grouped HiFx format block floating-point left matrix by a column-grouped HiFx format block floating-point right matrix, and also supports outputting data in BF16 format.

[0067] Optionally, in the AI ​​inference scenario, the inference process of each layer only includes forward processing, which is consistent with the forward processing described in the AI ​​training scenario. This embodiment of the application will not elaborate further on this.

[0068] Figure 5 This is a flowchart illustrating a model processing method provided in an embodiment of this application. The method can be applied to a computing device, which can be any of the computing devices provided above. The method includes the following steps.

[0069] S201: During the training or inference process of the AI ​​model, acquire first data information in a first data format. Optionally, the AI ​​model can be a model for processing different types of data such as images, audio / video data, or text data; for example, the AI ​​model can be an LLM.

[0070] S202: Convert the first data information into second data information corresponding to the second data format, and use the second data information to train or infer the AI ​​model; one of the first data format and the second data format is a single floating-point format and the other is a block floating-point format; the data information corresponding to the block floating-point format includes a bits of first shared exponent, b bits of second shared exponent, c bits of third shared exponent and M signed original code data, where a, b, c and M are positive integers, and c is a divisor of M and b is a divisor of c.

[0071] In one possible embodiment, the M signed original code data share the first shared exponent of bit a; the (i-1)×M / b+1th to the i×M / bth signed original code data in the M signed original code data also share the i-th bit in the second shared exponent, where i ranges from 1 to b; the (j-1)×M / c+1th to the j×M / cth signed original code data in the M signed original code data also share the j-th bit in the third shared exponent, where j ranges from 1 to c. The above block floating-point format can also be called the HiFx block floating-point format.

[0072] Optionally, the first shared exponent of bit 'a' includes an 8-bit exponent value, the second shared exponent of bit 'b' includes eight 1-bit exponent values, and the third shared exponent of bit 'c' includes sixteen 1-bit exponent values. That is, a = 8, b = 8, c = 16.

[0073] Optionally, M equals 64, with 64 signed original code data as a group.

[0074] Optionally, the signed original code data includes 1 sign bit, 1 integer bit, and N fractional bits, where N is an integer ranging from 0 to 8. This signed original code data can use a signed-magnitude encoding format and can be represented as S1PN. This signed original code data can also be called signed data, or simply data. For example, the signed original code data can be S1P0, S1P1, S1P2, or S1P3, etc.

[0075] For example, when the first shared exponent includes an 8-bit exponent value, the second shared exponent includes eight 1-bit exponent values, the third shared exponent includes sixteen 1-bit exponent values, and the M data items include 64 S1PNs, the data information corresponding to this block of floating-point format can be as follows: Figure 6 As shown. The first shared exponent can be represented as E8, the second shared exponent can be represented as 8 E1s, and the third shared exponent can be represented as 16 E1s. The above three levels of shared exponents occupy a total of 32 bits, which are shared by 64 data. The shared exponent is amortized to 0.5 bits per data.

[0076] The first shared exponent (E8) represents a numerical range of [-127, 128]. Each exponent value E1 in the second shared exponent (8 E1s) is encoded as either 0 or 1. Each exponent value E1 in the third shared exponent (16 E1s) is encoded as either 0 or 1. Each exponent value E1 in the second shared exponent corresponds to two adjacent exponent values ​​E1 in the third shared exponent. Each exponent value E1 in the third shared exponent corresponds to four adjacent data points out of the 64 data points.

[0077] Furthermore, the data information corresponding to this floating-point format can be parsed to obtain 64 values, and the parsing formula for each value can be 2. E8+E1+E1 ×S1PN. Specifically, according to the correspondence of the three-level shared exponents mentioned above, adding the three-level shared exponents together yields 16 exponent values. Each exponent value is then multiplied by the corresponding four consecutive S1PN values ​​to obtain 64 independent values. For example, the exponent values ​​corresponding to the 1st to 4th values ​​in these 64 values ​​are equal to the sum of E8, the first E1 in the second shared exponent, and the first E1 in the third shared exponent; the exponent values ​​corresponding to the 6th to 8th values ​​are equal to the sum of E8, the first E1 in the second shared exponent, and the second E1 in the third shared exponent; and so on, until the exponent values ​​corresponding to the 61st to 64th values ​​are equal to the sum of E8, the 8th E1 in the second shared exponent, and the 16th E1 in the third shared exponent.

[0078] Furthermore, the data range represented by S1PN above can be ±[0, 2-2]. -N The above three-level shared exponent, theoretically represented in the range of E8+E1+E1=[-127,128]+[0,1]+[0,1]=[-127,130], can be used in the practical range of [-127,128]. This is mainly due to the conversion between single-point and block floating-point formats. In the practical range, 128 represents NaN and Inf, where NaN refers to an abnormal value and Inf refers to infinity (∞).

[0079] Optionally, the single-point floating-point format is one of the following: BF16, FP16, or FP32. That is, the data format conversion involved in this method can include mutual conversion between BF16 and block floating-point format, mutual conversion between FP16 and block floating-point format, and mutual conversion between FP32 and block floating-point format.

[0080] The following details the process of converting first data information in a first data format into second data information in a second data format.

[0081] The first method involves converting the first data information (in single-point floating-point format) into the second data information (in block floating-point format). The first data information may include M single-point floating-point data points.

[0082] In one possible embodiment, converting the first data information into second data information may include: determining c first exponents, b second exponents, and a third exponent based on the exponent values ​​of the M single-float data included in the first data information; determining a first shared exponent, a second shared exponent, and a third shared exponent based on the c first exponents, the b second exponents, and the third exponent; and converting the M single-float data into the M signed original code data based on the first shared exponent, the second shared exponent, and the third shared exponent to obtain the second data information. Wherein, any one of the c first exponents is determined based on the maximum value of the exponents of the (j-1)×M / c+1th to the j×M / cth single-float data among the M single-float data; the b second exponents are determined based on the maximum values ​​of c / b consecutive first exponents among the c first exponents; and the third exponent is determined based on the maximum value among the b second exponents.

[0083] For example, taking a single-floating-point format of BF16, a==8, b=8, c=16, M=64 as an example, the process of determining 16 first exponents, 8 second exponents, and 1 third exponent for 64 data in BF16 format can include: extracting the exponents from the 64 data by grouping them into groups of 4 adjacent data to obtain 16 groups of exponents, and determining the maximum value of each of the 16 groups of exponents to obtain 16 first exponents, each of which is 8-bit; grouping the 16 first exponents into groups of 2 adjacent first exponents, and determining the maximum value of each of the 8 groups of exponents to obtain 8 second exponents, each of which is 8-bit; and determining the maximum value among the 8 second exponents to obtain the third exponent, which is 8-bit.

[0084] Optionally, the k-th bit SE2(k) of the first shared exponent SE1, the k-th bit SE2(k) of the second shared exponent, and the k'-th bit SE3(k') of the third shared exponent satisfy the following formula. In the formula, == represents an equality comparison operation; if they are equal, the result is 1; otherwise, the result is 0.

[0085] SE1 = Emax - 2, where Emax represents the third exponent;

[0086] SE2(k) = (Ebmax(k) == Emax) ? 1:0, where Ebmax(k) represents the kth element among the b second-order codes, and the value of k ranges from 1 to b;

[0087] Ecmax(k') represents the k'th element among the c first-order codes. This indicates rounding up, where k' ranges from 1 to c. Rounding up here can mean adding 1 to the integer part of a number that has a decimal. For example, 1.2, 1.5, or 1.9 rounded up results in 2, and 2.2, 2.3, or 2.8 rounded up results in 3.

[0088] For example, taking 16 first exponents, 8 second exponents, and 1 third exponent, with each exponent being 8 bits, the process of determining the first shared exponent SE1, the second shared exponent SE2, and the third shared exponent SE3 is illustrated. The first shared exponent is equal to the third exponent - 2, resulting in the first shared exponent SE1, which is an 8-bit exponent, i.e., E8. The second shared exponent consists of 8 1-bit exponent values. These 8 second exponents are compared with the third exponent for equality, each yielding a 1-bit exponent value, thus corresponding to the 8 1-bit exponent values ​​included in the second shared exponent. The third shared exponent consists of 16 1-bit exponent values, which can be determined as follows: The first 1-bit exponent value is equal to the following difference compared with 1, where the difference is equal to the first first exponent minus the first shared exponent SE1, and then minus the first bit of the second shared exponent; the second 1-bit exponent value is equal to the following difference compared with 1, where the difference is equal to the second first exponent minus the first shared exponent SE1, and then minus the first bit of the second shared exponent; the third ... The three 1-bit exponent values ​​are equal to the following difference compared to 1: the difference is equal to the third first exponent minus the first shared exponent SE1, and then minus the second bit of the second shared exponent. The fourth 1-bit exponent value is equal to the following difference compared to 1: the difference is equal to the fourth first exponent minus the first shared exponent SE1, and then minus the second bit of the second shared exponent. The fifth 1-bit exponent value is equal to the following difference compared to 1: the difference is equal to the first first exponent minus the first shared exponent SE1, and then minus the second bit of the second shared exponent. The first 1-bit exponent is calculated by subtracting the third bit of the second shared exponent from the first shared exponent SE1. The sixth 1-bit exponent value is equal to the following difference compared to 1: this difference is equal to the second first exponent minus the first shared exponent SE1 and then the third bit of the second shared exponent. The seventh 1-bit exponent value is equal to the following difference compared to 1: this difference is equal to the third first exponent minus the first shared exponent SE1 and then the fourth bit of the second shared exponent. The eighth 1-bit exponent value is equal to the following difference compared to 1. The exponent is compared with 1, and the difference is equal to the 4th first exponent minus the first shared exponent SE1 and then the 4th bit of the second shared exponent. The 9th 1-bit exponent value is equal to the following difference compared with 1, and the difference is equal to the 1st first exponent minus the first shared exponent SE1 and then the 5th bit of the second shared exponent. The 10th 1-bit exponent value is equal to the following difference compared with 1, and the difference is equal to the 2nd first exponent minus the first shared exponent SE1 and then the 5th bit of the second shared exponent.The 11th 1-bit exponent value is equal to the following difference compared with 1: the difference is equal to the 3rd first exponent minus the first shared exponent SE1, and then minus the 6th bit of the second shared exponent; the 12th 1-bit exponent value is equal to the following difference compared with 1: the difference is equal to the 4th first exponent minus the first shared exponent SE1, and then minus the 6th bit of the second shared exponent; the 13th 1-bit exponent value is equal to the following difference compared with 1: the difference is equal to the 1st first exponent minus the first shared exponent SE1, and then minus the 7th bit of the second shared exponent. The 14th 1-bit exponent value is equal to the following difference compared to 1: this difference is equal to the second first exponent minus the first shared exponent SE1, and then minus the 7th bit of the second shared exponent. The 15th 1-bit exponent value is equal to the following difference compared to 1: this difference is equal to the third first exponent minus the first shared exponent SE1, and then minus the 8th bit of the second shared exponent. The 16th 1-bit exponent value is equal to the following difference compared to 1: this difference is equal to the fourth first exponent minus the first shared exponent SE1, and then minus the 8th bit of the second shared exponent.

[0089] Optionally, in the process of converting M single-point floating-point data into M signed original code data according to the three-level shared exponent, it can be achieved by the formula S1PN = round(BF16 × 2^(-E)), where round represents rounding. This rounding can be done using a rounding half method, where the rounding half can be rounding half away from zero or rounding half tie to even. If the absolute value of the data carries over after rounding, i.e., equals 2, the absolute value of the data needs to be saturated to the maximum absolute value that S1PN can represent, i.e., 2 – 2^(-N).

[0090] The second method involves converting the first data information in the block floating-point format into the second data information in the single floating-point format.

[0091] In one possible embodiment, converting the first data information into the second data information may include: determining the exponent value corresponding to each of the M signed original code data according to the first shared exponent, the second shared exponent, and the third shared exponent; and converting the M signed original code data into M single-float data according to the M signed original code data and the exponent of each of the M signed original code data to obtain the second data information.

[0092] For example, when the first shared exponent includes an 8-bit exponent value, the second shared exponent includes eight 1-bit exponent values, the third shared exponent includes sixteen 1-bit exponent values, the M data includes 64 S1PNs, and the second data format is BF16, the method may include: determining the exponent E corresponding to each S1PN according to the sharing relationship between the three-level shared exponents and the 64 S1PNs, where the exponent E = E8 + E1_8 + E1_16, E8 represents the first shared exponent, E1_8 represents the exponent value in the second shared exponent corresponding to the S1PN, and E1_16 represents the exponent value in the third shared exponent corresponding to the S1PN; converting according to the formula BF16 = S1PN × 2^E, that is, multiplying each of the 64 S1PNs by the corresponding 2^E and normalizing it into 64 BF16 single-floating-point data, thus obtaining the BF16 second data information.

[0093] In the above conversion process, since BF16 can express 8 significant bits, which is more than S1PN, rounding is not involved, that is, the format conversion does not affect the numerical precision.

[0094] In practical applications, any calculation step involved in the conversion of data information in the first and second data formats can be implemented using one or more instructions. These instructions can be general instructions or special instructions; this embodiment does not impose specific limitations on them. For example, the calculation of each shared exponent in the first, second, and third shared exponents can be implemented using one or more instructions.

[0095] Understandably, when the aforementioned block floating-point format is HiF5, the data within the group is S1P3, with an average amortization of 5.5 bits / value, resulting in a 1.5x bandwidth gain when used for AI computation; when the aforementioned block floating-point format is HiF4, the data within the group is S1P2, with an average amortization of 4.5 bits / value, resulting in a 1.83x bandwidth gain when used for AI computation; when the aforementioned block floating-point format is HiF3, the data within the group is S1P1, with an average amortization of 3.5 bits / value, resulting in a 2.36x bandwidth gain when used for AI computation; and when the aforementioned block floating-point format is HiF2, the data within the group is S1P0, with an average amortization of 2.5 bits / value, resulting in a 3.3x bandwidth gain when used for AI computation.

[0096] In this embodiment of the application, by converting between data information in block floating-point format and data information in single floating-point format (e.g., BF16), the parts implemented using conventional instructions (e.g., SIMT / SIMD instructions) or the parts implemented using dedicated hardware instructions and corresponding acceleration circuits can be marked during the calculation process implemented by instructions. In this way, by merging some processes, the number of instructions can be reduced, thereby improving performance.

[0097] Furthermore, for the block floating-point format provided in this application embodiment, the fixed shared exponent amortization is 0.5 bits / value, compared to 0.25 bits / value for MXFP4 and 1 bit / value for MX6. This block floating-point format ensures both achievable precision and sufficient bandwidth gain. The signed original code data within a group can support 5 bits / value. Combined with the fixed shared exponent overhead of 0.5 bits / value, the maximum overall overhead is 5.5 bits / value, which, compared to MXFP8 (8.25 bits / value), yields a minimum bandwidth gain of 8.25 / 5.5 = 1.5x. In this application, the group size of the block floating-point data can reach 64, meaning the group size is large enough to maximize the fixed-point normalization while achieving good matrix multiplication energy efficiency, which is particularly crucial for LLM training and induction. The three-level shared exponent design uses 8 bits to represent the level 1 exponent, and uses 8 bits for level 2 and 16 bits for level 3 to express the exponent difference information of local data. In this way, the exponent expression overhead ratio is 1:3 for basic / local data, which greatly enhances the expression of local data difference information and improves the end-to-end training and inference accuracy of block floating-point data.

[0098] Figure 7 This is a flowchart illustrating a model processing method provided in an embodiment of this application. The method can be applied to a computing device, which can be any of the computing devices provided above. The method includes the following steps.

[0099] S301: During the training or inference process of the AI ​​model, obtain the first data information and the second data information of the first target layer of the AI ​​model; wherein the first data information and the second data information are both in block floating-point format, and both include a bits of first shared exponent, b bits of second shared exponent, c bits of third shared exponent and M signed original code data, where a, b, c and M are positive integers, and c is a divisor of M and b is a divisor of c.

[0100] Optionally, the AI ​​model can be a model for processing different types of data, such as images, audio and video data, or text data. For example, the AI ​​model can be an LLM.

[0101] Specifically, the M signed original code data share the first shared exponent of bit a; the (i-1)×M / b+1th to the i×M / bth signed original code data in the M signed original code data also share the ith bit in the second shared exponent, where the value of i ranges from 1 to b; the (j-1)×M / c+1th to the j×M / cth signed original code data in the M signed original code data also share the jth bit in the third shared exponent, where the value of j ranges from 1 to c.

[0102] Optionally, the first shared exponent of bit a is an 8-bit exponent value, the second shared exponent of bit b is 8 1-bit exponent values, the third shared exponent of bit c is 16 1-bit exponent values, and M equals 64.

[0103] Optionally, the signed original code data includes 1-bit integer bits, 1-bit integer bits, and N-bit fractional bits, where N is an integer ranging from 0 to 8.

[0104] For a detailed description of data information in block floating-point format, please refer to the text above. Figure 5 The relevant descriptions in the corresponding method embodiments will not be repeated here.

[0105] S302: Calculate the inner product of the first data information and the second data information. The inner product calculation result is used for training or inference of the second target layer of the AI ​​model. The first target layer and the second target layer are two adjacent layers in the AI ​​model.

[0106] Optionally, the second target layer can be the layer below the first target layer; or, the second target layer can be the layer above the first target layer. For example, when performing inner product calculation in the forward processing of the first target layer, the second target layer is the layer below the first target layer; when performing inner product calculation in the backward processing of the first target layer, the second target layer is the layer above the first target layer.

[0107] Here, the first data information and the second data information can be vectors in block floating-point format. The corresponding inner product operation can be calculated using the formula C = AB + C, where A and B can represent the first data information and the second data information, respectively. Both A and B include M block floating-point data values. Optionally, C is an accumulated data in FP32 format.

[0108] When the first shared exponent consists of 8 bits, the second shared exponent consists of 8 bits, the third shared exponent consists of 16 bits, and the M data elements consist of 64 S1PNs, A and B can be represented by the following formula. Where E8 1A E1 represents the first shared exponent in A.2A E1 represents an exponent value of the second shared exponent in A. 3A E8 represents an exponent value of the second shared exponent in A. 1B E1 represents the first shared exponent in B. 2B E1 represents an exponent value of the second shared exponent in B. 3B This represents an exponent value of the second shared exponent in B.

[0109]

[0110] Optionally, the inner product of the first data information and the second data information can be calculated in a variety of different ways, which will be introduced and explained below.

[0111] In the first method, calculating the inner product of the first data information and the second data information includes: multiplying the M signed original code data in the first data information one-to-one with the M signed original code data in the second data information to obtain M first data products; adding the b-bit second shared exponent and c-bit third shared exponent in the first data information, and the b-bit second shared exponent and c-bit third shared exponent in the second data information, according to their correspondence, to obtain c first exponent sums; shifting the M first data products based on the c first exponent sums to obtain M second data products; accumulating the M second data products to obtain a cumulative sum; adding the a-bit first shared exponent in the first data information and the a-bit first shared exponent in the second data information to obtain the a-bit second exponent sum; and determining the inner product calculation result of the first data information and the second data information based on the cumulative sum and the a-bit second exponent sum.

[0112] Optionally, adding the b-bit second shared exponent and c-bit third shared exponent in the first data information, and the b-bit second shared exponent and c-bit third shared exponent in the second data information according to their correspondence, may include: adding the b-bit second shared exponent and c-bit third shared exponent in the first data information according to their correspondence; and adding the b-bit second shared exponent and c-bit third shared exponent in the second data information according to their correspondence; then, adding the calculation results of the above two steps together.

[0113] Alternatively, the b-bit second shared exponent and c-bit third shared exponent in the first data information, as well as the b-bit second shared exponent and c-bit third shared exponent in the second data information, can be added according to their corresponding relationships. This can include: adding the b-bit second shared exponent in the first data information and the b-bit second shared exponent in the second data information according to their corresponding relationships; adding the c-bit third shared exponent in the first data information and the c-bit third shared exponent in the second data information according to their corresponding relationships; and then adding the calculation results of the above two steps together.

[0114] For example, taking the first data information and the second data information as A and B respectively, the inner product AB of the first data information and the second data information can satisfy the following formula.

[0115]

[0116] That is, first calculate the product of the 64 S1PNs in A and the 64 S1PNs in B (represented as S1PN × S1PN), resulting in 64 S2P2Ns, where S2P2N represents 1-bit sign bit, 2-bit integer bits, and 2N-bit fractional bits; then sum the corresponding exponents of the second and third level shared exponents in A and B, resulting in 16 exponent sums, which can be represented as 16 m, where m = (E1... 2A +E1 2B )+(E1 2A +E1 2B Then, based on the 16 exponents and the corresponding shifts of the 64 S2P2Ns (denoted as S2P2N < m), shifting left by a maximum of 4 bits, we obtain 64 S6P2Ns, where S6P2N represents 1-bit sign bit, 6-bit integer bits, and 2N-bit fractional bits. We then perform fixed-point addition on these 64 S6P2Ns to obtain one S12P2N, where S12P2N represents 1-bit sign bit, 12-bit integer bits, and 2N-bit fractional bits. Finally, we sum the first-level shared exponents in A and B (denoted as E8). 1A +E8 1B We obtain an E9 and a S12P2N, and perform floating-point addition (represented as 2^E9×S12P2N) to obtain the inner product.

[0117] In the second method, calculating the inner product of the first data information and the second data information includes: multiplying the M signed original code data in the first data information one-to-one with the M signed original code data in the second data information, and adding the two adjacent first data products in the resulting M first data products to obtain M / 2 second data products; and taking the b-bit second shared exponent and c-bit third shared exponent in the first data information, and the b-bit second shared exponent and c-bit third shared exponent in the second data information... Add the corresponding values ​​to obtain c first-exponential sums; based on these c first-exponential sums, shift the M / 2 second data products to obtain M / 2 third data products; accumulate these M / 2 third data products to obtain a cumulative sum; add the a-bit first shared exponent of the first data information and the a-bit first shared exponent of the second data information to obtain the a-bit second exponent sum; based on this cumulative sum and the a-bit second exponent sum, determine the inner product calculation result of the first data information and the second data information.

[0118] For example, taking the first data information and the second data information as A and B respectively, the inner product AB of the first data information and the second data information can satisfy the following formula.

[0119]

[0120] That is, first calculate the product of the 64 S1PNs in A and the 64 S1PNs in B (represented as S1PN × S1PN) to obtain 64 S2P2Ns (S2P2N represents 1-bit sign bit, 2-bit integer bit, and 2N-bit fractional bit), and add adjacent products to obtain 32 S3P2Ns, where S3P2N represents 1-bit sign bit, 3-bit integer bit, and 2N-bit fractional bit; sum the corresponding exponents of the second and third level shared exponents in A and B to obtain 16 exponent sums, which can be represented as 16 m, m = (E1 2A +E1 2B )+(E1 2A +E1 2B Then, based on the 16 exponents and the corresponding shifts of the 32 S3P2Ns (denoted as S3P2N < < m), shifting left by a maximum of 4 bits, we obtain 32 S7P2Ns, where S7P2N represents 1-bit sign bit, 7-bit integer bits, and 2N-bit fractional bits. We then perform fixed-point addition on these 32 S7P2Ns to obtain one S12P2N, where S12P2N represents 1-bit sign bit, 12-bit integer bits, and 2N-bit fractional bits. Finally, we sum the first-level shared exponents in A and B (denoted as E8). 1A +E8 1BWe obtain an E9 and a S12P2N, and perform floating-point addition (represented as 2^E9×S12P2N) to obtain the inner product.

[0121] In the third method, calculating the inner product of the first data information and the second data information includes: multiplying the M signed original code data in the first data information one-to-one with the M signed original code data in the second data information, and adding the four adjacent first data products among the M first data products obtained by multiplication to obtain M / 4 second data products; and taking the b-bit second shared exponent and c-bit third shared exponent in the first data information, as well as the b-bit second shared exponent and c-bit third shared exponent in the second data information... Add the corresponding values ​​to obtain c first-exponential sums; based on these c first-exponential sums, shift the M / 4 second data products to obtain M / 4 third data products; accumulate these M / 4 third data products to obtain a cumulative sum; add the a-bit first shared exponent of the first data information and the a-bit first shared exponent of the second data information to obtain the a-bit second exponent sum; based on this cumulative sum and the a-bit second exponent sum, determine the inner product calculation result of the first data information and the second data information.

[0122] For example, taking the first data information and the second data information as A and B respectively, the inner product AB of the first data information and the second data information can satisfy the following formula.

[0123]

[0124] That is, first calculate the product of the 64 S1PNs in A and the 64 S1PNs in B (represented as S1PN × S1PN) to obtain 64 S2P2Ns (S2P2N represents 1-bit sign bit, 2-bit integer bit, and 2N-bit fractional bit), and add the four adjacent products to obtain 16 S4P2Ns, where S4P2N represents 1-bit sign bit, 4-bit integer bit, and 2N-bit fractional bit; sum the corresponding exponents of the second and third level shared exponents in A and B to obtain 16 exponent sums, which can be represented as 16 m, m = (E1 2A +E1 2B )+(E1 2A +E1 2BThen, based on the 16 exponents and the corresponding shifts of the 16 S4P2Ns (denoted as S4P2N < < m), shifting left by a maximum of 4 bits, we obtain 16 S8P2Ns, where S8P2N represents 1-bit sign bit, 8-bit integer bits, and 2N-bit fractional bits; we then perform fixed-point addition on these 16 S8P2Ns to obtain 1 S12P2N, where S12P2N represents 1-bit sign bit, 12-bit integer bits, and 2N-bit fractional bits; finally, we sum the first-level shared exponents in A and B (denoted as E8). 1A +E8 1B We obtain an E9 and a S12P2N, and perform floating-point addition (represented as 2^E9×S12P2N) to obtain the inner product.

[0125] Figure 8 This is a schematic diagram of a data processing device provided in an embodiment of this application. The data processing device may include: a fixed-point multiplier, a first fixed-point adder, a shifter, a fixed-point accumulator tree, a second fixed-point adder, and a floating-point accumulator. This data processing device can be used to calculate the inner product of first data information and second data information in block floating-point format. A detailed description of the data information in block floating-point format can be found in the relevant descriptions in the method embodiments above, and will not be repeated here.

[0126] In this data processing device, a fixed-point multiplier is used to multiply the M signed original code data in the first data information one-to-one with the M signed original code data in the second data information to obtain M first data products. A first fixed-point adder is used to add the b-bit second shared exponent and c-bit third shared exponent in the first data information, and the b-bit second shared exponent and c-bit third shared exponent in the second data information, according to their correspondence, to obtain c first exponent sums. A shifter is used to shift the M first data products according to the c first exponent sums to obtain M second data products. A fixed-point accumulator is used to accumulate the M second data products to obtain an accumulated sum. A second fixed-point adder is used to add the a-bit first shared exponent in the first data information and the a-bit first shared exponent in the second data information to obtain the a-bit second exponent sum. The floating-point accumulator is used to determine the inner product calculation result of the first data information and the second data information based on the multiplicative sum and the second exponent sum of the a bits.

[0127] The above Figure 8 The following description uses the data processing device to calculate the inner product of the first data information and the second data information according to the first method embodiment described above as an example. For a detailed explanation of the process of calculating the inner product according to the first method, please refer to the relevant description above; this embodiment will not repeat it here.

[0128] Figure 9 This is a schematic diagram of another data processing device provided in an embodiment of this application. The data processing device may include: a fixed-point fusion multiplier, a first fixed-point adder, a shifter, a fixed-point accumulator tree, a second fixed-point adder, and a floating-point accumulator. This data processing device can be used to calculate the inner product of first and second data information in block floating-point format. For a detailed description of the data information in block floating-point format, please refer to the relevant descriptions in the method embodiments above; these will not be repeated here.

[0129] In this data processing device, a fixed-point fusion multiplier is used to: multiply the M signed original code data in the first data information one-to-one with the M signed original code data in the second data information, and add adjacent first data products among the M first data products obtained by multiplication to obtain M / 2 second data products. A first fixed-point adder is used to: add the b-bit second shared exponent and c-bit third shared exponent in the first data information, and the b-bit second shared exponent and c-bit third shared exponent in the second data information, according to their correspondence, to obtain c first exponent sums. A shifter is used to: shift the M / 2 second data products according to the c first exponent sums to obtain M / 2 third data products. A fixed-point accumulation tree is used to: accumulate the M / 2 third data products to obtain an accumulated sum. A second fixed-point adder is used to: add the a-bit first shared exponent in the first data information and the a-bit first shared exponent in the second data information to obtain the a-bit second exponent sum. The floating-point accumulator is used to determine the inner product calculation result of the first data information and the second data information based on the multiplicative sum and the second exponent sum of the a bits.

[0130] The above Figure 9 The following description uses the second method described in the above-described embodiment to illustrate the calculation of the inner product of the first data information and the second data information by the data processing device. For a detailed explanation of the process of calculating the inner product using the second method, please refer to the relevant description above; this embodiment will not repeat it here. The above-described fixed-point fusion multiplier can be used to perform calculations of “x0*y0+x1*y1+…”.

[0131] Figure 10 This is a schematic diagram of another data processing device provided in an embodiment of this application. The data processing device may include: a fixed-point fusion multiplier, a first fixed-point adder, a shifter, a fixed-point accumulator tree, a second fixed-point adder, and a floating-point accumulator. This data processing device can be used to calculate the inner product of first and second data information in block floating-point format. A detailed description of the data information in block floating-point format can be found in the relevant descriptions in the method embodiments above, and will not be repeated here.

[0132] In this data processing device, a fixed-point fusion multiplier is used to: multiply the M signed original code data in the first data information one-to-one with the M signed original code data in the second data information, and add the four adjacent first data products of the resulting M first data products to obtain M / 4 second data products. A first fixed-point adder is used to: add the b-bit second shared exponent and c-bit third shared exponent in the first data information, and the b-bit second shared exponent and c-bit third shared exponent in the second data information, according to their correspondence, to obtain c first exponent sums. A shifter is used to: shift the M / 4 second data products according to the c first exponent sums to obtain M / 4 third data products. A fixed-point accumulation tree is used to: accumulate the M / 4 third data products to obtain an accumulated sum. A second fixed-point adder is used to: add the a-bit first shared exponent in the first data information and the a-bit first shared exponent in the second data information to obtain the a-bit second exponent sum. The floating-point accumulator is used to determine the inner product calculation result of the first data information and the second data information based on the multiplicative sum and the second exponent sum of the a bits.

[0133] The above Figure 10 The following description uses the third method described in the above embodiments to illustrate the calculation of the inner product of the first data information and the second data information by the data processing device. For a detailed explanation of the calculation process using the third method, please refer to the relevant description above; this embodiment will not repeat it here. The above-described fixed-point fusion multiplier can be used to perform calculations of “x0*y0+x1*y1+…”.

[0134] Furthermore, Figures 8 to 10 The floating-point accumulator can also be used to convert the inner product calculation result into FP32 format. In the embodiments of the three data processing devices mentioned above, the floating-point accumulator is used to execute 2^E9×S12P2N+FP32 as an example for illustration.

[0135] In this embodiment, the data processing device is mainly composed of several multipliers, adders, and shifters arranged in an orderly manner. In a binary system, the impact of different components on the data bit width is as follows:

[0136] Fixed-point multipliers: double the effective bit width of the output data. Figure 8 Taking the calculation of S1PN×S1PN=S2P2N in the data processing device shown as an example, the input data is two signed original code data of {1-bit sign bit + 1-bit integer bit + N-bit fractional bits}, and the output result is one signed original code data of {1-bit sign bit + 2-bit integer bit + 2N-bit fractional bits}.

[0137] Fixed-point adders or accumulators: Carry-over will cause the high-order bits of the output data to increase. Figure 9 Taking the calculation of S1PN×S1PN+S1PN×S1PN in the data processing device shown as an example, after the multiplication is completed, it is S2P2N+S2PN. After the two numbers are added, the high bit carries over 1 bit, and finally S3P2N is obtained.

[0138] Shifters: change the data bit width. Figure 10 Taking the shifter in the data processing device shown as an example, S4P2N is shifted left by m bits, where m ranges from 0 to 4, and can be shifted left by a maximum of 4 bits. Therefore, after the shift, 4 bits need to be added to the high bits of the data, that is, S4P2N becomes S8P2N.

[0139] In the embodiments of this application, when data operations are involved, the structure of the processing device and the bit width of the signed original code data in the block floating-point data can be flexibly selected according to the actual needs of the scenario. For example, when only AI training is supported, the format of the block floating-point data can be selected as HiF5; when only AI inference is supported, the format of the block floating-point data can be selected as HiF4; when asymmetric computation is supported, a combination of multiple formats of block floating-point data can be selected.

[0140] The foregoing primarily describes the solutions provided in the embodiments of this application from the perspective of a computing device. It is understood that, in order to achieve the aforementioned functions, the computing device includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0141] This application embodiment can divide the computing device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. The following description uses the division of functional modules according to each function as an example.

[0142] When using integrated units, Figure 11A schematic diagram of a model processing device involved in the above embodiments is shown. This device can be a computing device or a module applied to a computing device. The device includes: a receiving unit 401, a processing unit 402, and a sending unit 403. In one possible embodiment, the receiving unit 401 is used to support the device in executing S201 of the above method embodiments; the processing unit 402 is used to support the device in executing S202 of the above method embodiments; and the sending unit 403 is used to support the device in outputting second data information in a second data format. In another possible embodiment, the receiving unit 401 is used to support the device in executing S301 of the above method embodiments; the processing unit 402 is used to support the device in executing S302 of the above method embodiments; and the sending unit 403 is used to support the device in outputting the inner product calculation result of the first data information and the second data information.

[0143] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here in the embodiments of this application.

[0144] Based on hardware implementation, in this embodiment, the processing unit 402 can be the processor of the device, the receiving unit 401 can be the receiver of the device, and the transmitting unit 403 can be the transmitter of the device. The transmitter can typically be integrated with the receiver as a transceiver; specifically, the transceiver can also be called a communication interface or interface circuit. Optionally, the device can be a computing device or a module applied to a computing device; the structure of the device can be referenced. Figure 3 As shown, the embodiments of this application will not be described in detail here.

[0145] It is understood that all relevant content of each step involved in the above method embodiments can be referenced in the embodiments of the processing device, and the embodiments of this application will not be repeated here.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed.

[0147] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0148] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. This readable storage medium may include various media capable of storing program code, such as a USB flash drive, external hard drive, read-only memory, random access memory, magnetic disk, or optical disk. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product.

[0149] In another embodiment of this application, a computer-readable storage medium is also provided, which stores a computer program or instructions; wherein, when a device executes the computer program or instructions, the device performs the above-described... Figure 5 The steps in the corresponding method embodiments.

[0150] In another embodiment of this application, a computer-readable storage medium is also provided, which stores a computer program or instructions; wherein, when a device runs the computer program or instructions, the device performs the aforementioned actions. Figure 7 The steps in the corresponding method embodiments.

[0151] In another embodiment of this application, a computer program product is also provided, which includes a computer program that, when executed by a device, causes the device to perform the functions described above. Figure 5 The steps of the corresponding method embodiment.

[0152] In another embodiment of this application, a computer program product is also provided, which includes a computer program that, when executed by a device, causes the device to perform the functions described above. Figure 7 The steps in the corresponding method embodiments.

[0153] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model processing method, characterized in that, The method includes: During the training or inference process of an artificial intelligence model, first data information in a first data format is acquired; The first data information is converted into second data information corresponding to the second data format, and the second data information is used to train or infer the artificial intelligence model; Wherein, one of the first data format and the second data format is a single floating-point format and the other is a block floating-point format; The data information corresponding to the block floating-point format includes a-bit first shared exponent, b-bit second shared exponent, c-bit third shared exponent, and M signed original code data, where a, b, c, and M are positive integers, and c is a divisor of M and b is a divisor of c; the M signed original code data share the a-bit first shared exponent; the (i-1)×M / b+1th to the i×M / bth signed original code data in the M signed original code data also share the i-th bit in the second shared exponent, where i ranges from 1 to b; the (j-1)×M / c+1th to the j×M / cth signed original code data in the M signed original code data also share the j-th bit in the third shared exponent, where j ranges from 1 to c.

2. The method according to claim 1, characterized in that, The first shared exponent of bit a includes 8 bits of exponent value, the second shared exponent of bit b includes 8 bits of 1-bit exponent value, the third shared exponent of bit c includes 16 bits of 1-bit exponent value, and M equals 64.

3. The method according to claim 1 or 2, characterized in that, The signed original code data includes 1 sign bit, 1 integer bit, and N fractional bits, where N is an integer ranging from 0 to 8.

4. The method according to any one of claims 1-3, characterized in that, The single-floating-point format is one of BF16, FP16, or FP32.

5. The method according to any one of claims 1-4, characterized in that, When the first data format is a single floating-point format and the second data format is a block floating-point format, the first data information includes M single floating-point data. The step of converting the first data information into second data information includes: Based on the exponent values ​​of the M single-point floating-point data included in the first data information, c first exponents, b second exponents, and a third exponent are determined; wherein, any one of the c first exponents is determined based on the maximum value of the exponents of the (j-1)×M / c+1th to the j×M / cth single-point floating-point data in the M single-point floating-point data; the b second exponents are determined based on the maximum value of c / b consecutive first exponents in the c first exponents; and the third exponent is determined based on the maximum value among the b second exponents. Based on the c first exponents, the b second exponents, and the third exponent, determine the first shared exponent, the second shared exponent, and the third shared exponent; Based on the first shared exponent, the second shared exponent, and the third shared exponent, the M single-float data are converted into the M signed original code data to obtain the second data information.

6. The method according to claim 5, characterized in that, The first shared exponent SE1, the k-th bit SE2(k) of the second shared exponent, and the k'-th bit SE3(k') of the third shared exponent satisfy the following formula: SE1 = Emax - 2, where Emax represents the third exponent; SE2(k) = (Ebmax(k) == Emax) ? 1:0, where Ebmax(k) represents the kth of the b second-order codes, and the value of k ranges from 1 to b; Ecmax(k') represents the k'th of the c first-order codes. This indicates rounding up, and the value of k' ranges from 1 to c.

7. The method according to any one of claims 1-4, characterized in that, When the first data format is block floating-point format and the second data format is single floating-point format, converting the first data information into the second data information includes: Based on the first shared exponent, the second shared exponent, and the third shared exponent, determine the exponent value corresponding to each of the M signed original code data; Based on the M signed original code data and the exponent of each of the M signed original code data, the M signed original code data are converted into M single floating-point data to obtain the second data information.

8. A model processing method, characterized in that, The method includes: During the training or inference process of the artificial intelligence model, first data information and second data information of the first target layer of the artificial intelligence model are obtained; wherein, the data format of the first data information and the second data information is block floating-point format, and the first data information and the second data information each include a-bit first shared exponent, b-bit second shared exponent, c-bit third shared exponent and M signed original code data, where a, b, c and M are positive integers, and c is a divisor of M and b is a divisor of c; the M signed original code data share the a-bit first shared exponent; the (i-1)×M / b+1th to the i×M / bth signed original code data in the M signed original code data also share the i-th bit in the second shared exponent, where the value of i ranges from 1 to b; the (j-1)×M / c+1th to the j×M / cth signed original code data in the M signed original code data also share the j-th bit in the third shared exponent, where the value of j ranges from 1 to c; The inner product of the first data information and the second data information is calculated, and the inner product calculation result is used for training or inference of the second target layer of the artificial intelligence model. The first target layer and the second target layer are two adjacent layers in the artificial intelligence model.

9. The method according to claim 8, characterized in that, The first shared exponent of bit a is an 8-bit exponent value, the second shared exponent of bit b is eight 1-bit exponent values, the third shared exponent of bit c is sixteen 1-bit exponent values, and M equals 64.

10. The method according to claim 8 or 9, characterized in that, The signed original code data includes 1 integer bit, 1 integer bit, and N fractional bits, where N is an integer ranging from 0 to 8.

11. The method according to any one of claims 8-10, characterized in that, The calculation of the inner product of the first data information and the second data information includes: Multiply the M tagged original code data in the first data information with the M tagged original code data in the second data information one by one to obtain M first data products; Add the b-bit second shared exponent and c-bit third shared exponent in the first data information, as well as the b-bit second shared exponent and c-bit third shared exponent in the second data information, according to their corresponding relationship to obtain c first exponent sums. Based on the c first exponents, the M first data products are shifted to obtain M second data products; The products of the M second data are summed to obtain the sum of the products; Add the first shared exponent of a bits in the first data information and the first shared exponent of a bits in the second data information to obtain the second exponent of a bits; Based on the sum of the multiplication and the second exponent of the a-bit, the inner product calculation result of the first data information and the second data information is determined.

12. The method according to any one of claims 8-10, characterized in that, The calculation of the inner product of the first data information and the second data information includes: Multiply the M signed original code data in the first data information with the M signed original code data in the second data information one by one, and add the two adjacent first data products in the M first data products obtained by multiplication to obtain M / 2 second data products; Add the b-bit second shared exponent and c-bit third shared exponent in the first data information, as well as the b-bit second shared exponent and c-bit third shared exponent in the second data information, according to their corresponding relationship to obtain c first exponent sums. Based on the sum of the c first exponents, the M / 2 products of the second data are shifted to obtain the M / 2 products of the third data; The products of the M / 2 third data are summed to obtain the sum of the products; Add the first shared exponent of a bits in the first data information and the first shared exponent of a bits in the second data information to obtain the second exponent of a bits. Based on the sum of the multiplications and the second exponent of the a-bits, the inner product calculation result of the first data information and the second data information is determined.

13. The method according to any one of claims 8-10, characterized in that, The calculation of the inner product of the first data information and the second data information includes: Multiply the M signed original code data in the first data information with the M signed original code data in the second data information one by one, and add the four adjacent first data products in the M first data products obtained by multiplication to obtain M / 4 second data products; Add the b-bit second shared exponent and c-bit third shared exponent in the first data information, as well as the b-bit second shared exponent and c-bit third shared exponent in the second data information, according to their corresponding relationship to obtain c first exponent sums. Based on the c first exponents, the M / 4 second data products are shifted to obtain the M / 4 third data products; The M / 4 products of the third data are summed to obtain the sum of the products; Add the first shared exponent of a bits in the first data information and the first shared exponent of a bits in the second data information to obtain the second exponent of a bits. Based on the sum of the multiplications and the second exponent of the a-bits, the inner product calculation result of the first data information and the second data information is determined.

14. A model processing device, characterized in that, The device includes: The receiving unit is used to acquire first data information in a first data format during the training or inference process of an artificial intelligence model. The processing unit is configured to convert the first data information into second data information corresponding to the second data format, and use the second data information to train or infer the artificial intelligence model. Wherein, one of the first data format and the second data format is a single floating-point format and the other is a block floating-point format; The data information corresponding to the block floating-point format includes a-bit first shared exponent, b-bit second shared exponent, c-bit third shared exponent, and M signed original code data, where a, b, c, and M are positive integers, and c is a divisor of M and b is a divisor of c; the M signed original code data share the a-bit first shared exponent; the (i-1)×M / b+1th to the i×M / bth signed original code data in the M signed original code data also share the i-th bit in the second shared exponent, where i ranges from 1 to b; the (j-1)×M / c+1th to the j×M / cth signed original code data in the M signed original code data also share the j-th bit in the third shared exponent, where j ranges from 1 to c.

15. The apparatus according to claim 14, characterized in that, The first shared exponent of bit a includes 8 bits of exponent value, the second shared exponent of bit b includes 8 bits of 1-bit exponent value, the third shared exponent of bit c includes 16 bits of 1-bit exponent value, and M equals 64.

16. The apparatus according to claim 14 or 15, characterized in that, The signed original code data includes 1 sign bit, 1 integer bit, and N fractional bits, where N is an integer ranging from 0 to 8.

17. The apparatus according to any one of claims 14-16, characterized in that, The single-floating-point format is one of BF16, FP16, or FP32.

18. The apparatus according to any one of claims 14-17, characterized in that, When the first data format is a single-float format and the second data format is a block-float format, the first data information includes M single-float data points; the processing unit is further configured to: Based on the exponent values ​​of the M single-point floating-point data included in the first data information, c first exponents, b second exponents, and a third exponent are determined; wherein, any one of the c first exponents is determined based on the maximum value of the exponents of the (j-1)×M / c+1th to the j×M / cth single-point floating-point data in the M single-point floating-point data; the b second exponents are determined based on the maximum value of c / b consecutive first exponents in the c first exponents; and the third exponent is determined based on the maximum value among the b second exponents. Based on the c first exponents, the b second exponents, and the third exponent, determine the first shared exponent, the second shared exponent, and the third shared exponent; Based on the first shared exponent, the second shared exponent, and the third shared exponent, the M single-float data are converted into the M signed original code data to obtain the second data information.

19. The apparatus according to claim 18, characterized in that, The first shared exponent SE1, the k-th bit SE2(k) of the second shared exponent, and the k'-th bit SE3(k') of the third shared exponent satisfy the following formula: SE1 = Emax - 2, where Emax represents the third exponent; SE2(k) = (Ebmax(k) == Emax) ? 1:0, where Ebmax(k) represents the kth of the b second-order codes, and the value of k ranges from 1 to b; Ecmax(k') represents the k'th of the c first-order codes. This indicates rounding up, and the value of k' ranges from 1 to c.

20. The apparatus according to any one of claims 14-17, characterized in that, When the first data format is a block floating-point format and the second data format is a single floating-point format, the processing unit is further configured to: Based on the first shared exponent, the second shared exponent, and the third shared exponent, determine the exponent value corresponding to each of the M signed original code data; Based on the M signed original code data and the exponent of each of the M signed original code data, the M signed original code data are converted into M single floating-point data to obtain the second data information.

21. A data processing device, characterized in that, The device is used to calculate the inner product of first and second data information in block floating-point format. Both the first and second data information include a-bit first shared exponent, b-bit second shared exponent, c-bit third shared exponent, and M signed original code data, where a, b, c, and M are positive integers, and c is a divisor of M and b is a divisor of c. The M signed original code data share the a-bit first shared exponent. The (i-1)×M / b+1th to the i×M / bthth signed original code data in the M signed original code data also share the i-th bit of the second shared exponent, where i ranges from 1 to b. The (j-1)×M / c+1th to the j×M / cthth signed original code data in the M signed original code data also share the j-th bit of the third shared exponent, where j ranges from 1 to c. The device includes: A fixed-point multiplier is used to multiply the M signed original code data in the first data information with the M signed original code data in the second data information one by one to obtain M first data products; The first fixed-point adder is used to add the b-bit second shared exponent and c-bit third shared exponent in the first data information, as well as the b-bit second shared exponent and c-bit third shared exponent in the second data information, according to their corresponding relationship, to obtain c first exponent sums. A shifter is used to shift the M first data products according to the c first exponents to obtain M second data products; A fixed-point accumulation tree is used to accumulate the products of the M second data to obtain a multiplicative sum. The second fixed-point adder is used to add the first shared exponent of a bits in the first data information and the first shared exponent of a bits in the second data information to obtain the second exponent sum of a bits. A floating-point accumulator is used to determine the inner product calculation result of the first data information and the second data information based on the multiplicative sum and the second exponent sum of the a bits.

22. A data processing device, characterized in that, The device is used to calculate the inner product of first and second data information in block floating-point format. Both the first and second data information include a-bit first shared exponent, b-bit second shared exponent, c-bit third shared exponent, and M signed original code data, where a, b, c, and M are positive integers, and c is a divisor of M and b is a divisor of c. The M signed original code data share the a-bit first shared exponent. The (i-1)×M / b+1th to the i×M / bthth signed original code data in the M signed original code data also share the i-th bit of the second shared exponent, where i ranges from 1 to b. The (j-1)×M / c+1th to the j×M / cthth signed original code data in the M signed original code data also share the j-th bit of the third shared exponent, where j ranges from 1 to c. The device includes: A fixed-point fusion multiplier is used to multiply the M signed original code data in the first data information with the M signed original code data in the second data information one by one, and add the two adjacent first data products in the M first data products obtained by multiplication to obtain M / 2 second data products; The first fixed-point adder is used to add the b-bit second shared exponent and c-bit third shared exponent in the first data information, as well as the b-bit second shared exponent and c-bit third shared exponent in the second data information, according to their corresponding relationship, to obtain c first exponent sums. A shifter is used to shift the M / 2 second data products according to the c first exponents to obtain an M / 2 third data product; A fixed-point accumulation tree is used to accumulate the products of the M / 2 third data to obtain the accumulated sum. The second fixed-point adder is used to add the first shared exponent of a bits in the first data information and the first shared exponent of a bits in the second data information to obtain the second exponent sum of a bits. A floating-point accumulator is used to determine the inner product calculation result of the first data information and the second data information based on the multiplicative sum and the second exponent sum of the a bits.

23. A data processing device, characterized in that, The device is used to calculate the inner product of first and second data information in block floating-point format. Both the first and second data information include a-bit first shared exponent, b-bit second shared exponent, c-bit third shared exponent, and M signed original code data, where a, b, c, and M are positive integers, and c is a divisor of M and b is a divisor of c. The M signed original code data share the a-bit first shared exponent. The (i-1)×M / b+1th to the i×M / bthth signed original code data in the M signed original code data also share the i-th bit of the second shared exponent, where i ranges from 1 to b. The (j-1)×M / c+1th to the j×M / cthth signed original code data in the M signed original code data also share the j-th bit of the third shared exponent, where j ranges from 1 to c. The device includes: A fixed-point fusion multiplier is used to multiply the M signed original code data in the first data information with the M signed original code data in the second data information one by one, and add the four adjacent first data products in the M first data products obtained by multiplication to obtain M / 4 second data products; The first fixed-point adder is used to add the b-bit second shared exponent and c-bit third shared exponent in the first data information, as well as the b-bit second shared exponent and c-bit third shared exponent in the second data information, according to their corresponding relationship, to obtain c first exponent sums. A shifter is used to shift the M / 4 second data product according to the c first exponents to obtain the M / 4 third data product; A fixed-point accumulation tree is used to accumulate the products of the M / 4 third data to obtain the accumulated sum. The second fixed-point adder is used to add the first shared exponent of a bits in the first data information and the first shared exponent of a bits in the second data information to obtain the second exponent sum of a bits. A floating-point accumulator is used to determine the inner product calculation result of the first data information and the second data information based on the multiplicative sum and the second exponent sum of the a bits.

24. A model processing device, characterized in that, The device includes a processor and a memory, the memory storing a computer program, the processor executing the computer program stored in the memory to cause the device to perform the method as described in any one of claims 1-13.

25. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on the device, cause the device to perform the method as described in any one of claims 1-13.

26. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a device, causes the device to perform the method as described in any one of claims 1-13.

Images