Inverse quantization method of model data, matrix operation method and related equipment

Abstract

The embodiment of the invention discloses an inverse quantization method of model data, a matrix operation method and related equipment, and the method comprises the steps: determining first floating point data to be inversely quantized in a model; regarding the first floating point data as first integer data with the same bit width as the first floating point data, and performing first type conversion processing of expanding the first bit width to a second bit width to generate intermediate floating point data; performing second type conversion processing for maintaining a second bit width on the intermediate floating point data to generate second integer data; performing format correction processing on the second integer data to generate inversely quantized second floating point data; according to the method, the inverse quantization process can be simplified, and the calculation efficiency can be improved, so that chips which are partially limited by instruction sets and lack special calculation instruction support for low-precision floating point data are adapted, and the overall performance of artificial intelligence model training is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence model computing technology, and in particular to a model data dequantization method, a matrix operation method, and related equipment. Background Art

[0002] In AI model training and inference scenarios, low-precision floating-point data is often used to reduce computational and storage overhead. However, current model training methods have certain limitations in hardware implementation. The instruction sets of some model training chips are incomplete and lack dedicated computational instructions for low-precision floating-point data. For example, the Ascend 910B chip does not have dedicated instructions that directly support FP8 computations. This means that before performing matrix multiplication operations, low-precision floating-point data must be dequantized into high-precision floating-point data to accommodate the chip's built-in high-precision computational instructions.

[0003] In existing model training methods, the dequantization process usually requires multiple steps of bit operations. The process is cumbersome and complicated, which not only increases the demand for computing space but also reduces computing efficiency, thereby affecting the overall performance of artificial intelligence model training. Summary of the Invention

[0004] This application provides a dequantization method, matrix operation method and related equipment involving model data, which can simplify the dequantization process, improve computing efficiency, and thus improve the overall performance of artificial intelligence model training.

[0005] This application provides the following solutions:

[0006] According to a first aspect, a method for inverse quantization of model data is provided, wherein the inverse quantization method includes: determining first floating-point data to be inversely quantized in a model; treating the first floating-point data as first integer data having the same bit width as the first floating-point data, and performing a first type conversion process to expand the first bit width to a second bit width to generate intermediate floating-point data; performing a second type conversion process to maintain the second bit width on the intermediate floating-point data to generate second integer data; performing a format correction process on the second integer data to generate inversely quantized second floating-point data; wherein the first floating-point data is stored in a first storage area; and the intermediate floating-point data, the second integer data, and the second floating-point data are time-shared and multiplexed in the same second storage area.

[0007] According to an implementable method in an embodiment of the present application, the format correction processing of the second integer data to generate the inverse quantized second floating-point data includes: obtaining format layout information of the sign bit, exponent bit, and mantissa bit in the first floating-point data; based on the format layout information, the format correction processing of the second integer data is performed to generate the inverse quantized second floating-point data.

[0008] According to an implementable method in an embodiment of the present application, the format correction processing is performed on the second integer data based on the format layout information to generate the inverse quantized second floating-point data, including: based on a preset bit mask, clearing the invalid data generated by the bit width expansion in the upper 8 bits of the second integer data; calculating a binary correction value based on the original bias value of the exponent bit in the format layout information and the exponent offset of the target floating-point format, and performing a superposition correction on the second integer data based on the binary correction value to generate bias-corrected data; based on the sign bit, exponent bit, and mantissa bit in the format layout information, performing a shift operation on the bias-corrected data with the goal of aligning it with the target floating-point format to generate alignment-corrected data; and multiplying the alignment-corrected data by a preset scaling factor to generate the inverse quantized second floating-point data.

[0009] According to an implementable method in an embodiment of the present application, the inverse quantization method is applied to an AI acceleration chip including a vector calculation unit and a matrix calculation unit, and the type conversion processing is implemented based on a bit width extension instruction built into the AI ​​acceleration chip, where the first bit width is 8 bits and the second bit width is 16 bits.

[0010] According to the second aspect, a matrix operation method for model data is provided, which includes: in a first operation unit, executing the inverse quantization method for model data described in any one of the first aspects to obtain inverse quantized second floating-point data; in a second operation unit, performing a matrix multiplication operation based on the second floating-point data to obtain a matrix operation result.

[0011] According to an implementable method in an embodiment of the present application, the matrix operation method also includes: synchronizing the first operation unit and the second operation unit to execute in parallel through a hardware event queue, so that the first operation unit triggers the second operation unit to start scheduling matrix multiplication after completing a set of inverse quantization data; wherein the operation time of the first operation unit and the second operation unit partially overlap.

[0012] According to a third aspect, a dequantization device for model data is provided, comprising: a data determination module for determining first floating-point data to be dequantized in a model; a first conversion module for treating the first floating-point data as first integer data having the same bit width as the first integer data, and performing a first type conversion process to expand the first bit width to a second bit width to generate intermediate floating-point data; a second conversion module for performing a second type conversion process on the intermediate floating-point data to maintain the second bit width, and generate second integer data; a data correction module for performing a format correction process on the second integer data, and generate dequantized second floating-point data; wherein the first floating-point data is stored in a first storage area; and the intermediate floating-point data, the second integer data, and the second floating-point data are time-shared multiplexed in the same second storage area.

[0013] According to the fourth aspect, a chip is provided, comprising: a first operation unit, used to execute the dequantization method of the model data described in any one of the first aspects to obtain dequantized second floating-point data; a second operation unit, used to perform a matrix multiplication operation based on the second floating-point data to obtain a matrix operation result.

[0014] According to the fifth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the inverse quantization method of the model data described in any one of the first aspects and the matrix operation method of the model data described in any one of the second aspects are implemented.

[0015] According to the sixth aspect, an electronic device is provided, comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the program instructions execute the dequantization method of the model data described in any one of the first aspect and the matrix operation method of the model data described in any one of the second aspect.

[0016] According to the seventh aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the inverse quantization method of the model data described in any one of the first aspects and the matrix operation method of the model data described in any one of the second aspects.

[0017] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0018] The model data dequantization method, matrix operation method and related equipment proposed in this application can achieve lossless expansion of low-precision to high-precision data by treating low-precision floating-point data as integer data with the same bit width as its bit width, and then using the bit width extension instruction supported by the chip to perform two-step type conversion processing, thereby avoiding the complex operations of extracting the sign, exponent and mantissa bit by bit in the traditional dequantization process; then, redundant data is cleared by bit masking, exponent bias is corrected and single shift alignment is performed, and the data layout is adjusted to the target floating-point format, which can streamline the multi-step tedious bit operations required for traditional dequantization to simplify the process and improve computing efficiency; in addition, the first floating-point data is stored in the first storage area, and the intermediate floating-point data, the second integer data and the second floating-point data are time-shared in the same second storage area, which can reduce the demand for computing space and effectively save computing power resources, thereby adapting to some chips that have instruction set limitations and lack support for dedicated computing instructions for low-precision floating-point data, thereby improving the overall performance of artificial intelligence model training.

[0019] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 It is an overall flow chart of the existing dequantization method in the background technology;

[0022] Figure 2 A system architecture diagram applicable to the dequantization method of model data provided in an embodiment of the present application;

[0023] Figure 3 A diagram showing the steps of a method for dequantizing model data provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of first floating-point data in the dequantization method for model data provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of generating second integer data in the dequantization method of model data provided in an embodiment of the present application;

[0026] Figure 6 A method step diagram for performing format correction processing to generate dequantized second floating-point data in the dequantization method of model data provided in an embodiment of the present application;

[0027] Figure 7 This is an overall flow chart of the dequantization method for model data provided in an embodiment of the present application;

[0028] Figure 8 This is an overall flow chart of the matrix operation method for model data provided in the embodiment of the present application;

[0029] Figure 9 A data flow diagram of a matrix operation method for model data provided in an embodiment of the present application;

[0030] Figure 10 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0032] The terms used in the embodiments of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The singular forms "a," "an," and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0033] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0034] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0035] Currently, low-precision floating-point data is widely used in AI model training and inference scenarios because it effectively reduces computational and storage overhead. However, at the hardware implementation level of model training chips, some chips have instruction set limitations and lack dedicated computational instructions supporting low-precision floating-point data. For example, the Ascend 910B chip does not have dedicated instructions that directly support FP8 computations. Therefore, before performing matrix multiplication operations, low-precision floating-point data must be dequantized to high-precision floating-point data to accommodate the chip's built-in high-precision computational instructions.

[0036] Existing dequantization methods usually require multiple steps of bit operations, which is a cumbersome and complicated process. This results in large computing space requirements, reduced computing efficiency, and affects the overall performance of artificial intelligence model training.

[0037] FP8 is an 8-bit floating point that can significantly reduce video memory usage and computing power consumption while maintaining high computing accuracy. The Ascend 910B is an AI training chip using the DaVinci architecture and has a Cube arithmetic unit designed specifically for efficient matrix multiplication. However, the Ascend 910B currently does not support dedicated computing instructions of the FP8 type. Therefore, implementing FP8 calculations on the Ascend 910B can only be achieved indirectly through algorithms.

[0038] refer to Figure 1 , Figure 1 This is a flowchart of the existing dequantization method in the background art. It is understood that in some existing technical solutions, when performing fp8 matrix multiplication operations on the Ascend 910B chip, a complex dequantization process is required because the chip's AIV and AIC instructions do not support fp8 type operations. Specifically, after the fp8 data is read into UB, it becomes tensor_fp8, and a series of combined operations must be performed using the AIV instructions to achieve dequantization. First, according to tensor_sign_temp=tensor_fp8>>7, the fp8 data is shifted right by 7 bits to obtain tensor_sign_temp, and tensor_sign_temp is bitwise ANDed with 0x1 to extract the sign bit (tensor_sign). Then, tensor_E_temp=tensor_fp8>>3, that is, the fp8 data is shifted right by 3 bits to obtain tensor_E_temp, and tensor_E_temp is bitwise ANDed with 0xF to obtain the exponent bit (tensor_E_fp8). Then, the fp8 data is bitwise ANDed with 0x7 to obtain the mantissa bit (tensor_M) to implement the bit extraction operation.

[0039] Furthermore, in subsequent steps, the sign bit (tensor_sign) is shifted left by 15 bits, the exponent bit (tensor_E_fp8) is shifted left by 10 bits after adding the bias value 8, and the mantissa bit (tensor_M) is shifted left by 7 bits. These processed bits are combined into temporary fp16 data (tensor_fp16_temp). However, the data obtained at this point requires further processing. The temporary fp16 data and the processed mantissa data must be bitwise ORed to form the complete fp16 format data. This data is then multiplied by the scale factor to obtain tensor_fp16 for matrix multiplication. This completes the dequantization from fp8 to fp16, and the fp16 matrix can be multiplied using AIC. According to statistics, the entire dequantization process currently performed on the Ascend910B chip requires at least 11 computational steps and requires storage space for at least three copies of data to temporarily store intermediate results. Furthermore, existing solutions fail to fully consider the doubling of data space usage when converting from FP8 to FP16. This can lead to data overflow or insufficient storage space during the actual dequantization process, hindering the smooth progress of dequantization. This cumbersome process not only increases the computational burden but also limits the efficiency of the Ascend 910B chip when processing FP8 data, highlighting the need to develop more efficient dequantization methods.

[0040] In summary, the current technical issues can be summarized as follows: Chips like the Ascend 910B currently lack dedicated instructions supporting the FP8 type, making direct FP8 matrix multiplication impossible. Therefore, existing dequantization schemes must be used. However, these schemes are inefficient, requiring the separate extraction of the sign bit, exponent bit, and mantissa bit, followed by correction and reassembly, requiring 11 computational steps and requiring significant computational space. Furthermore, existing schemes fail to account for the increase in space occupied by each number from 8 bits to 16 bits when converting FP8 to FP16 on the Ascend 910B, which directly impacts the implementation of dequantization.

[0041] In view of this, this application provides a new idea. In order to facilitate the understanding of this application, the system architecture on which this application is based is first described. Figure 2 , Figure 2 This is a system architecture diagram applicable to the dequantization method of model data provided in the embodiment of the present application, Figure 2 An exemplary system architecture to which the embodiments of the present application can be applied is shown. Figure 2 As shown in , the dequantization method in this application can be applied to the Ascend 910B chip, and the system architecture of the Ascend 910B chip may include:

[0042] Global memory (GM): Global memory is a storage unit in the entire system architecture, used to store large amounts of data and provide a data source for data processing throughout the chip. When performing matrix calculations, data is first stored in the GM, waiting to be called and processed, and then can interact with other parts of the AICore through the handling unit.

[0043] AICore: The AICore is the computing unit within the Ascend 910B chip, responsible for various AI computing tasks, including matrix operations. It is divided into a matrix computing component (AICube, AIC) and a vector computing component (AIVector, AIV), with data transferred between these two components via Global Memory.

[0044] AICube (AIC): The AIC is used for matrix multiplication calculations and contains multiple buffers, such as LOABuffer, LOBBuffer, and LOCBuffer, which temporarily store input and output data during matrix multiplication. The AIC also includes a Cube calculation unit, which is specifically used to perform matrix multiplication operations. During matrix multiplication, data is transferred from the GM through the transfer unit (MTE1 / MTE2 / MTE3) to the corresponding buffer in the AIC. The data is then calculated in the Cube unit, and the result is returned to the GM through the transfer unit.

[0045] AI Vector (AIV): AIV is used for vector calculation operations and includes Scalar and Vector calculation units. In the dequantization process of this application, AIV is used to dequantize fp8 into fp16. AIV includes a unified buffer (UB) for temporarily storing the data required in the vector calculation process.

[0046] Transport units (MTE1 / MTE2 / MTE3): These transport units are used to transfer data between the GM and the AICore. They can move data from the GM to the corresponding buffers of the AIC or AIV, and can also move calculation results from the AIC or AIV back to the GM.

[0047] FixPipe and fpBuffer: FixPipe is a fixed pipeline used to handle some fixed-point operations or data preprocessing tasks, while fpBuffer is used to temporarily store floating-point related data, which may play a role in matrix multiplication calculations or other floating-point operations.

[0048] Scalar unit: The Scalar unit is used to process scalar computing tasks. In AI computing, in addition to matrix and vector operations, there will also be scalar-related calculations, such as updating some control parameters and simple mathematical operations. The Scalar unit can work together with the Vector unit in AIV to complete complex computing tasks.

[0049] In some embodiments, it can be understood that the data starts from the GM and enters the corresponding buffer of the AIC or AIV in the AICore through the transport unit (MTE1 / MTE2 / MTE3). After the dequantization of fp8 to fp16 is completed in the AIV, the fp16 data is transported to the AIC buffer (such as LOABuffer or LOBBuffer) through the transport unit, and then the matrix multiplication calculation is performed in the Cube calculation unit of the AIC. The calculation result is returned to the GM through the transport unit. At the same time, when the data is processed in the AIV, it may undergo some preprocessing through the FixPipe, and the Scalar unit will also participate in related scalar calculations.

[0050] It is worth mentioning that Figure 2 The mid-range architecture divides the AI ​​Core into two independent cores, the AIC and the AIV, allowing matrix and vector calculations to be performed in parallel, improving the computational efficiency of the entire chip. The dequantization process described in this application can be applied to the AIV, enabling the AIV to dequantize fp8 data into fp16 data, which is then transferred to the AIC via the transport unit for matrix multiplication, thus implementing the fp8 general-purpose matrix multiplication function based on the Ascend 910B chip. This architecture enables dequantization and matrix multiplication operations to work efficiently together, fully leveraging the computing power of the Ascend 910B chip.

[0051] refer to Figure 3 , Figure 3 A method step diagram of the dequantization method of model data provided in an embodiment of the present application, which can be performed by Figure 2 The AIV unit in the system shown is executed. Figure 3 As shown in , the method may include the following steps:

[0052] Step S310, determining the first floating-point data to be dequantized in the model;

[0053] Step S320, treating the first floating-point data as first integer data having the same bit width as the first integer data, and performing a first type conversion process to expand the first bit width to a second bit width to generate intermediate floating-point data;

[0054] Step S330, performing a second type conversion process on the intermediate floating-point data to maintain the second bit width, to generate second integer data;

[0055] Step S340 , performing format correction processing on the second integer data to generate inverse quantized second floating-point data.

[0056] The first floating-point data is stored in the first storage area; the intermediate floating-point data, the second integer data and the second floating-point data are time-shared and multiplexed in the same second storage area.

[0057] It can be seen from the above process that the present application can achieve lossless expansion of low-precision to high-precision data by treating low-precision floating-point data as integer data with the same bit width as its bit width, and then using the cast instruction supported by the chip to perform a two-step type conversion process. For example, the first step is to convert the fp8 data into fp16 intermediate floating-point data, and the second step is to expand it to 16-bit integer data, avoiding the complex operations of extracting the sign, exponent and mantissa bit by bit in the traditional scheme; then, the redundant data is cleared by bit masking, the exponent bias is corrected and the single shift alignment is performed, and the data layout is adjusted to the target floating-point format. The multi-step tedious bit operations required for traditional inverse quantization can be streamlined to simplify the process and improve computing efficiency, thereby adapting to some chips that have instruction set limitations and lack support for dedicated computing instructions for low-precision floating-point data, thereby improving the overall performance of artificial intelligence model training.

[0058] In some embodiments, the inverse quantization method is applied to an AI acceleration chip including a vector calculation unit and a matrix calculation unit, and the type conversion processing is implemented based on the built-in bit width extension instruction in the AI ​​acceleration chip, with the first bit width being 8 bits and the second bit width being 16 bits. It is worth noting that the AI ​​acceleration chip in this application includes Ascend910B, and the built-in bit width extension instruction includes the cast instruction in the Ascend910B chip. In the AIVector unit of the Ascend910B chip, the unified buffer (UB) stores the intermediate floating-point data (fp16) and the subsequent integer data (int16) in the same physical address area through the hardware address remapping function. The physical address area is the second storage area, and the second storage area is directly overwritten with the second floating-point data (fp16) in the format correction stage to achieve zero-copy reuse of storage resources.

[0059] In some embodiments, during the dequantization process, the fp8 data in the first storage area is first regarded as int8 data and converted into fp16 data through the cast instruction; at this time, the second storage area in the UB is remapped, and the second storage area is now used to store the converted fp16 data; next, the cast instruction is used again to convert the fp16 data into int16 data, and the second storage area is still used to replace the original fp16 data; in the format correction stage, the int16 data is directly bit mask cleared, bias corrected and shifted in the second storage area to obtain fp16 data, and the previous int16 data is directly overwritten to complete zero-copy multiplexing; through the above mechanism, the intermediate floating-point data, integer data and final floating-point data are time-shared in the second storage area, and only two storage spaces (original data storage area + dynamic multiplexing area) are required to complete the entire dequantization process. The entire process does not require additional storage space to temporarily store intermediate results, and all operations are completed in the same storage area.

[0060] It can be understood that the present application only requires two storage spaces in the entire dequantization process, one for storing the original fp8 data and the other for storing the fp16 data, while the existing technology usually requires at least three storage spaces: one for storing the original fp8 data, one for storing the intermediate results (such as the sign bit, the exponent bit and the mantissa bit), and one for storing the fp16 data. Therefore, the present application can reduce the demand for computing space and effectively save computing resources.

[0061] The following describes in detail each step of the above process and the effects that can be further produced, in conjunction with the embodiments. It should be noted that the terms "first" and "second" in this disclosure do not have limitations on size, order, or quantity, but are only used to distinguish between two floating-point data. For example, "first floating-point data" and "second floating-point data" are used to distinguish between two floating-point data.

[0062] First, step S310, "Determining the first floating-point data to be dequantized in the model," is described in detail with reference to an embodiment. The first floating-point data refers to low-precision floating-point data to be converted, such as data in fp8 format, whose storage format includes a sign bit, an exponent bit, and a mantissa bit. By treating it as integer data, the chip's built-in bit-width extension instructions can be directly called to complete the initial conversion. The first type of conversion processing involves expanding the data bit width using the cast instruction, for example, expanding an 8-bit integer to a 16-bit integer. This operation leverages the chip's hardware instruction characteristics, avoiding the complexity of manually splitting data bits. The second type of conversion processing involves converting the data format while maintaining the expanded bit width, such as converting 16-bit integer data to a 16-bit floating-point format. This step establishes a standardized data foundation for subsequent format correction. Format correction processing involves bit-aligning and value-calibrating the intermediate data based on the characteristics of the original floating-point format, such as adjusting the exponent bias and mantissa alignment. Invalid data is removed using a preset bit mask, and the correction value is added to restore the format.

[0063] In some embodiments, the first floating-point data is fp8 data, and the sub-formats of the first floating-point data may include: E4M3 (4 exponent bits + 3 mantissa bits), which has a smaller dynamic range (can represent a maximum of ±448), but has a higher mantissa precision, which is suitable for reasoning tasks; E5M2 (5 exponent bits + 2 mantissa bits), which has a larger dynamic range (can represent a maximum of ±57344), but has a lower mantissa precision, which is suitable for training tasks. It is understandable that the two fp8 sub-formats are unique in their inverse quantization to fp16 formats, and different sub-formats only affect some parameters during the operation, such as different numbers of bits required for left shifting, different values ​​when the exponent bit is corrected for bias, etc., and do not affect the overall implementation logic of the present application solution.

[0064] refer to Figure 4 , Figure 4 A schematic diagram of first floating-point data in the dequantization method of model data provided in an embodiment of the present application; it can be understood that, as shown in the figure, Figure 4 This is a data diagram illustrating the fp8 type of E4M3 in the embodiment of the present application, wherein floating-point numbers generally follow the IEEE754 standard in computers, and their binary representation consists of three parts: a sign bit (Sign), an exponent part (Exponent), and a mantissa part (Mantissa / Fraction).

[0065] The floating point value V can be expressed as:

[0066] V=(-1) S ×M×2 E-B (1);

[0067] in:

[0068] S: Sign bit (1 bit), determines the positive or negative.

[0069] M: Mantissa, the formula for converting mantissa M into a value (normalized number), n is the length, b is the binary representation value, and the traversal order is from high to low:

[0070]

[0071] M=1+Fraction(3);

[0072] E: Exponent, stored as an unsigned integer, the actual exponent E real The offset value B needs to be subtracted:

[0073] E real =EB (4);

[0074] B: Bias, to allow the exponent E to express negative numbers, B = 2 k-1 -1 (k is the number of exponent bits).

[0075] Furthermore, in combination with the embodiments, the above-mentioned step S320, namely "treating the first floating-point data as the first integer data with the same bit width as the first integer data, and performing a first type conversion process from the first bit width to the second bit width to generate intermediate floating-point data", is described in detail; in some embodiments, the first floating-point data is regarded as the first integer data with the same bit width as the first integer data, and a first type conversion process from the first bit width to the second bit width is performed to generate intermediate floating-point data, including treating the 8-bit first floating-point data as the 8-bit first integer data, and performing a cast instruction conversion from 8 bits to 16 bits to generate 16-bit intermediate floating-point data.

[0076] A "cast" instruction is a hardware instruction operation that converts data from one data type to another. This can be achieved by calling the chip's built-in bit-width expansion instruction, which maintains data precision and avoids information loss. Bit-width expansion refers to expanding the storage space of data from a smaller capacity to a larger capacity. This can be achieved by padding the upper bits with zeros or duplicating the sign bit. For example, padding the upper 8 bits of an 8-bit integer with zeros to form 16-bit data. This expansion method can adapt to the input requirements of subsequent high-precision operations.

[0077] It is worth noting that the embodiment of the present application can use the redefinition function of the Ascend910B chip to realize that the first floating-point data is regarded as the first integer data with the same bit width as its bit width. Redefinition means that the data on the same block of memory is not modified and reinterpreted into another data type to facilitate the use of some calculation instructions. For example, Ascend910B does not support fp8 instructions, so the embodiment of the present application redefines fp8 into int8 to perform the above steps and dequantizes fp8 into fp16 through AIV operation.

[0078] It is worth noting that if you treat fp8 as int8, you can use the cast instruction of the Ascend hardware to convert it to fp16. Because the value range of int8 is [-128:127], and the range that fp16 can accurately represent is [-2048:2048], the cast process will not lose precision.

[0079] Tensor int8 =Tensor fp8 .view(int8) (5);

[0080] Tensor fp16 =cast(Tensor int8 ) (6);

[0081] The exact representation range of fp16 is [-2048:2048]. A floating-point number can accurately represent an integer only if it can be represented exactly in the form 1.M × 2^E, and the exponent range allows the mantissa to cover all integer bits. For float16, the mantissa has 11 significant bits (including an implicit 1), so the maximum unambiguous integer range is [-2^11, 2^11] (i.e., [-2048, 2048]).

[0082] Specifically, when processing 8-bit floating-point data, its binary bit pattern is reinterpreted as 8-bit integer data. By calling the cast instruction supported by the chip, the 8-bit integer data is expanded to a 16-bit data storage format, and the high bits are padded with zero values ​​during the expansion process to maintain numerical accuracy. The converted 16-bit intermediate floating-point data not only retains the integrity of the original quantization information, but also meets the data bit width requirements of subsequent high-precision computing units. For example, in the Ascend910B chip, this conversion process can be completed with a single hardware instruction, avoiding the calculation delay caused by multiple transfers of intermediate data.

[0083] It is understandable that compared with the existing technology, the traditional method requires multiple bit operations and temporary variable storage when processing low-precision floating-point data. However, this solution directly completes the data format conversion through a single bit width expansion instruction, effectively solving the problem of complex operation steps in the dequantization process of low-precision floating-point data. It directly completes the data bit width expansion through hardware instructions, avoiding multiple bit splitting and splicing operations in the traditional method.

[0084] Furthermore, the above-mentioned step S330, i.e., "performing a second type conversion processing on the intermediate floating-point data to maintain the second bit width to generate second integer data" is described in detail in combination with the embodiments; in some embodiments, performing a second type conversion processing on the intermediate floating-point data to maintain the second bit width to generate second integer data includes performing a second type conversion processing on the intermediate floating-point data to maintain the second bit width to generate second integer data, including performing a cast instruction conversion on the 16-bit intermediate floating-point data to maintain 16 bits to generate 16-bit second integer data.

[0085] The "cast" instruction conversion operation uses hardware instructions to convert data types. This can be achieved using the chip's built-in bit-width extension instruction, which converts floating-point data to integer data without changing the data bit width. Maintaining the second bit width means keeping the data bit width unchanged during type conversion. This can be achieved using fixed-bit-width registers, avoiding the need for additional bit-width adjustments during the conversion process.

[0086] Specifically, when the intermediate floating-point data is 16 bits, the chip's supported cast instruction is directly called to perform data format conversion. This conversion process does not change the data bit width; it simply converts the floating-point data's memory representation into an integer. During this process, the upper and lower 8 bits of the data maintain a complete mapping, avoiding data truncation or padding caused by the bit width change. By maintaining 16-bit operations, the converted second integer data can directly participate in subsequent format correction processing without the need for additional intermediate buffers or temporary variables.

[0087] refer to Figure 5 , Figure 5 In the dequantization method of model data provided in the embodiment of the present application, a schematic diagram of generating second integer data; in some embodiments, the data type of fp16 needs to be converted into data of type int16, and the floating-point number is restored to an integer, and the decimal places are directly discarded, so the conversion of fp16 to int16 is executed. The precision range of int16 is also larger than [-128:127], and there will be no loss of precision.

[0088] Tensor int16 =cast(Tensor fp16 ) (7);

[0089] like Figure 5 As shown in the results, each number occupies 16 bits. You can see that for positive numbers, the upper 8 bits are filled with 0, while for negative numbers, the upper 8 bits are filled with 1. This is because the int type uses the complement form to store data, for example:

[0090] 2'b10010011 minus 1 and inverted = -2'b01101101 = -109;

[0091] 2'b1111111110010011 minus 1 and inverted = -2'b0000000001101101 = -109;

[0092] Both express the same negative int value. In this way, the embodiment of the present application does not need to specifically extract the sign bit. The sign bit will not change during the next left shift. Therefore, after the cast is completed, the high 8-bit of the current data is filled with the sign bit, and the low 8-bit is filled with the original fp8 data. Since fp8->int8 does not require hardware calculation, the present application can losslessly expand the 8-bit data to 16 bits through two-step conversion (fp8->int8->fp16->int16).

[0093] Understandably, traditional methods typically require step-by-step bit width adjustment and format conversion when performing type conversions, such as first expanding the bit width through a bit shift operation and then separating the sign and value bits. This solution uses the cast instruction to complete the conversion, simplifying the complex process that originally required multiple bit operations into a single instruction operation. From a functional implementation perspective, the bit width expansion is completed in an efficient manner. By using a two-step cast operation, the bit width of each number is losslessly expanded to 16 bits, and dequantization is completed in four operations, which is better performance than the original solution's 11 operations. This improves the execution efficiency of the chip's computing unit while ensuring data accuracy and reduces resource consumption during the calculation process.

[0094] In some embodiments, the dequantization method of model data is applied to the Ascend910B chip, and the type conversion processing is based on a technical solution implemented by the cast instruction built into the Ascend910B chip. The instruction set of the Ascend910B chip includes a cast instruction for data type conversion. The cast instruction can directly perform bit width expansion operations on data stored in registers, such as converting 8-bit integer data into 16-bit integer data. During the dequantization process, by calling the hardware-level instructions of the chip, the additional overhead caused by data type conversion through software simulation can be avoided.

[0095] Specifically, when the Ascend910B chip performs dequantization, its arithmetic unit can directly call the built-in cast instruction set. After treating 8-bit floating-point data as 8-bit integer data, it executes the cast instruction to complete the bit width expansion from 8 bits to 16 bits, generating intermediate floating-point data. Because this instruction belongs to the chip's underlying hardware instructions, its execution process does not require multiple data transfers or temporary storage, and the data type conversion is completed directly through registers. This method fully utilizes the chip's hardware characteristics, allowing the data type conversion operation to be completed in a single instruction cycle. Compared with existing technologies, traditional dequantization methods require software algorithms on general-purpose processors to implement bit width expansion, requiring multiple shifts, masking operations, and temporary variable storage. However, this solution directly completes the core conversion operation through hardware instructions, eliminating the intermediate data storage link and reducing the number of instruction execution cycles. Specifically, for the architectural characteristics of the Ascend910B chip, this method avoids the extra computational steps caused by instruction set mismatches, effectively solving the problem of low instruction execution efficiency of existing dequantization methods under specific chip architectures, and reducing the storage space occupied during the computation process, making the entire dequantization process more suitable for the hardware architecture characteristics of artificial intelligence training chips. It is worth noting that since the embodiment of the present application is applied to the Ascend910B chip, while using fp8 type data in the reasoning of large models can save memory, it can also enable Vector and Cube units to be calculated in parallel during the dequantization process to cover most of the dequantization time.

[0096] Furthermore, the above-mentioned step S340, i.e., "performing format correction processing on the second integer data to generate inverse quantized second floating-point data", is described in detail in combination with the embodiments; in some embodiments, performing format correction processing on the second integer data to generate inverse quantized second floating-point data includes obtaining format layout information of the sign bit, exponent bit, and mantissa bit in the first floating-point data, and performing format correction processing on the second integer data based on the format layout information to generate inverse quantized second floating-point data.

[0097] Among them, the sign bit refers to the binary bit used to indicate positive and negative in floating-point data, which can be implemented by single-bit storage, occupying the most significant bit position in the floating-point format, and is used to determine the positive and negative signs of the value. The exponent bit refers to the binary bit segment used to indicate the magnitude of the value in floating-point data, which can be stored in the form of an integer with a fixed bit width, and is used to control the range scaling of floating-point numbers. The mantissa bit refers to the binary bit segment used to indicate the precision information in floating-point data, which can be stored in the form of the remaining bit width to store the decimal part, and is used to save the precision details of the significant digits. Format layout information refers to the bit allocation rules for each functional area of ​​floating-point data in binary storage, which can be obtained by parsing the IEEE standard or custom specification of the original floating-point format, and is used to guide the bit operation logic in the data format conversion process.

[0098] It is worth noting that in the embodiment of the present application, the binary data of fp8 is used (processed) as int8, because the sign bit position of fp8 and int8 is the same, negative numbers can be represented by two's complement, and can be directly corresponded; further, after int8->fp16, the original data is converted into floating-point format, and it is necessary to restore it to two's complement form through fp16->int16, and then the redundant sign bit can be cleared, the exponent can be adjusted, and the numerical accuracy can be restored through the scaling factor.

[0099] refer to Figure 6 , Figure 6 In the dequantization method for model data provided in an embodiment of the present application, a method step diagram for performing format correction processing to generate dequantized second floating-point data; in some embodiments, performing format correction processing on the second integer data to generate dequantized second floating-point data may include the following steps:

[0100] Step S610: Clearing invalid data generated by bit width expansion in the upper 8 bits of the second integer data based on a preset bit mask;

[0101] Step S620, calculating a binary correction value according to the original bias value of the exponent bit in the format layout information and the exponent offset of the target floating-point format, and performing superimposition correction on the second integer data according to the binary correction value to generate bias-corrected data;

[0102] Step S630 , performing a shift operation on the offset correction data with the goal of aligning the target floating-point format based on the sign bit, the exponent bit, and the mantissa bit in the format layout information, to generate alignment correction data;

[0103] Step S640 : multiplying the alignment-corrected data by a preset scaling factor to generate inverse-quantized second floating-point data.

[0104] In some embodiments, the present application further proposes a method for performing format correction processing on the second integer data based on the format layout information to generate inversely quantized second floating-point data, including: clearing invalid data in the upper 8 bits of the second integer data caused by bit width expansion based on a preset bit mask; calculating a binary correction value based on the original bias value of the exponent bit in the format layout information and the exponent offset of the target floating-point format, and performing superimposition correction on the second integer data based on the binary correction value to generate bias correction data; performing a shift operation on the bias correction data based on the sign bit, exponent bit and mantissa bit in the format layout information with the goal of aligning the bias correction data with the target floating-point format to generate alignment correction data; multiplying the alignment correction data by a scaling factor input by the user to generate inversely quantized second floating-point data.

[0105] A bit mask is a binary template used to filter specific bits. This can be achieved by using a mask template with the upper 8 bits set to 0 and the lower 8 bits set to 1. A bitwise AND operation is used to remove invalid data from the upper 8 bits to ensure data accuracy. A binary correction value is a value used to adjust the exponent bias. This value can be calculated by taking the difference between the original bias value and the exponent offset of the target floating-point format. For example, if the original bias is 127 and the target format offset is 1023, the correction value is the binary representation of 896. Addition is used to adjust the bias to the target range. A shift operation is a bitwise shift of data. Specifically, the corresponding fields of the corrected data can be moved to specified positions according to the definition of the sign, exponent, and mantissa bits of the target floating-point format. For example, the exponent bits can be shifted left by 23 bits and the mantissa bits can be shifted right by 7 bits to match the 32-bit floating-point format. A scaling factor is a coefficient used to restore the data dimension. This can be determined using the scaling parameters recorded during the quantization process. For example, multiplication by an integer power of 2 can restore the original floating-point value range.

[0106] Specifically, when the Ascend910B chip performs inverse quantization, it first applies a bit mask to the 16-bit integer data to eliminate the upper 8-bit residual data to avoid invalid bits interfering with subsequent calculations. Then, a correction value is generated based on the difference in exponent offset between the source data format and the target format. For example, when the source data uses an 8-bit floating-point format and the target is a 16-bit floating-point format, the intermediate data is offset compensated by an adder so that its exponent domain conforms to the representation range of the target format. Subsequently, the bias-corrected data is rearranged according to the bit field definition of the target floating-point format, such as moving the sign bit to the highest bit, the exponent bit to the middle bit segment, and the mantissa bit to fill the remaining low bits to complete data alignment. Finally, the scaling factor is applied to the aligned data through a multiplier to restore the floating-point value that meets the target precision, thereby completing the inverse quantization process. Through the above technical solution, the present application can simplify the inverse quantization process while maintaining calculation accuracy, realize floating-point format conversion through integer data operations and bit operations, significantly reduce computational complexity, and improve the execution efficiency of the Ascend910B chip when processing low-precision floating-point data.

[0107] In some embodiments, based on steps S610 to S640, taking the E4M3 format as an example, after the cast is completed, the high 8-bit of the current data is filled with the sign bit, and the low 8-bit is filled with the original fp8 data, and the dirty bit and the highest sign bit need to be removed. It can be understood that after the expansion to 16-bit, because the E of fp8 is 4 bits and the E of fp16 is 5 bits, the 5th bit of E at this time is the sign bit of fp8 before the expansion. The bias will be corrected by addition in the future. Since the hardware does not support uint16 type addition, only int16 type addition can be used. If it is a negative number, the bias of E cannot be corrected correctly. The embodiment of the present application only needs to set the highest bit to 0, because the data at this time is not a true complement.

[0108] So we need to do the following operation to set the sign bit and the 5th bit of E to 0:

[0109] Tensor uint16 =Tensor int16 .view(uint16) (8);

[0110] Tensor uint16 =Tensor uint16 &0XEFF7F (9);

[0111] Correct the bias. The bias value is 8. Because the bit shift operation has not been performed yet, the lowest bit of E is 3, so we need to add (8<<3):

[0112] Tensor int16 =Tensor int16+(8<<3) (10);

[0113] A single shift moves the sign, exponent, and mantissa to the correct positions. Since the lowest bit of E is currently at bit 3, it needs to be moved to bit 10. After the shift is completed, the float8 has been successfully converted to float16.

[0114] Tensor int16 =Tensor int16 <<7 (11);

[0115] Furthermore, multiply by scale:

[0116] Tensor float16 =Tensor float16 *scale(12);

[0117] In summary, 1. Through a two-step conversion (fp8->int8->fp16->int16), 8-bit data is losslessly expanded to 16 bits. Dirty and sign bits are removed, biases are corrected, bit shifts are performed, and scale factor precision is adjusted to obtain the correct p16 value. This effectively simplifies the bit manipulation process during the dequantization of low-precision floating-point data, reduces computing resource usage and memory bandwidth requirements, and improves the efficiency of data preprocessing before matrix multiplication operations. Standardized correction steps and a format layout guidance mechanism ensure the accuracy of conversions between floating-point formats of different precisions, while also providing a predictable operation mode for hardware instruction optimization.

[0118] refer to Figure 7 , Figure 7 The overall flow chart of the dequantization method of model data provided in the embodiment of the present application; in some embodiments, Figure 7 This article demonstrates the implementation of fp8 dequantization to fp16 on the Ascend 910B chip. The specific steps are as follows: First, float8 data is read into the UB (Uniform Buffer) to obtain tensor_fp8. UB is a buffer in the AI ​​Vector (AIV) that temporarily stores data for subsequent processing.

[0119] Redefine tensor_fp8 to int8 to obtain tensor_int8. This step uses the "redefinition" technique to treat fp8 data as int8 type, preparing for subsequent bit width expansion using the Ascend910B cast instruction.

[0120] Cast tensor_int8 to float16 to obtain tensor_float16. Since the value range of int8 is [-128:127], and the range that fp16 can accurately represent is [-2048:2048], the cast process does not lose precision.

[0121] Cast tensor_float16 to int16 to obtain tensor_int16. This expands the bit width of each number from 8-bit to 16-bit without loss, solving the bit width expansion issue when converting from fp8 to fp16.

[0122] Perform a bitwise AND operation on tensor_int16 and 0xEF7F (tensor_uint16 & 0xEF7F) to obtain tensor_uint16. This step removes the dirty bit and sets the highest bit to 0. Because E in fp8 is 4 bits and E in fp16 is 5 bits, the fifth bit of E is the sign bit of fp8 before extension. Bias correction is required later, but the hardware does not support uint16 addition and only int16 addition is supported, so the highest bit needs to be set to 0.

[0123] Add (8<<3) to tensor_int16 to correct the exponent bias, resulting in the corrected tensor_int16. Because dequantizing fp8 to fp16 requires a bias correction of 8, and the bit shift operation hasn't been performed yet, the lowest bit of E is in the third bit, so (8<<3) must be added.

[0124] Shift the corrected tensor_int16 left by 7 bits. Once the shift is complete, the float8 is successfully converted to float16. This is because the fp8 mantissa is 3 bits, while the fp16 mantissa is 10 bits. To align the highest bit of the mantissa, a 7-bit left shift is required to add 0s and move the sign, exponent, and mantissa to the correct positions.

[0125] Read scale into UB to get tensor_scale, then multiply the tensor_float16 obtained by the bit shift operation by scale to get the result of tensor_float16 multiplied by scale.

[0126] The entire process losslessly expands the bit width of each number to 16 bits through a two-step cast operation, and only four operations are required for dequantization. Compared with the 11 operations of the original solution, this can effectively improve performance. While ensuring data accuracy, it improves the execution efficiency of the chip's computing unit and reduces resource consumption during the calculation process.

[0127] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0128] According to another embodiment, the present application also provides a matrix operation method for model data, including executing an inverse quantization method for model data in a first operation unit to obtain inverse quantized second floating-point data, and executing a matrix multiplication operation based on the second floating-point data in a second operation unit to obtain a matrix operation result. Wherein, the first operation unit refers to an independent computing module for performing inverse quantization processing, which can be specifically implemented by a vector processing unit in a chip, which converts low-precision floating-point data into high-precision floating-point data by bit width expansion and format correction. Wherein, the second operation unit refers to an independent computing module for performing matrix multiplication, which can be specifically implemented by a matrix multiplication accelerator in a chip, which performs parallel calculations based on high-precision floating-point data to output operation results. Wherein, hardware event queue synchronization refers to coordinating the execution timing of different operation units through a hardware signal transmission mechanism, which can be specifically implemented by interrupt triggering or status register polling, so that inverse quantization and matrix multiplication operations can be executed in parallel in a pipelined manner.

[0129] It can be understood that the first operation unit is the AIV unit in the above example, and the second operation unit is the AIC unit in the above example. When the model data needs to perform matrix operations, the first operation unit receives the first floating-point data to be processed, generates intermediate floating-point data through bit width expansion and type conversion, and then obtains the second floating-point data that conforms to the high-precision floating-point format through format correction. At the same time, the second operation unit immediately starts the matrix multiplication calculation after receiving the data that has been partially dequantized. Through the synchronization mechanism of the hardware event queue, each time the first operation unit completes the dequantization of a set of data, it triggers the second operation unit to schedule the calculation task, so that the time window of dequantization and matrix multiplication partially overlaps, thereby reducing the waiting time.

[0130] refer to Figure 8 , Figure 8The overall flow chart of the matrix operation method for model data provided in the embodiment of the present application; in one embodiment, the lossless dequantization process from fp8 to fp16 is completed through the above embodiment, while ensuring data accuracy, the execution efficiency of the chip operation unit is improved, and the resource consumption in the calculation process is reduced. After obtaining the fp16 data, the embodiment of the present application can also efficiently perform matrix multiplication operations in the AIC to obtain the required matrix operation results. By transferring the scaled fp16 data to the AICube (AIC), the matrix multiplication operation is performed using the Cube computing unit of the AIC to obtain the matrix operation results, which is beneficial to the hardware characteristics of the Ascend910B chip and realizes efficient and low-resource-consumption fp8 matrix multiplication operations.

[0131] In some embodiments, while the chip is executing matrix operations, the hardware event queue continuously monitors the progress of the dequantization task of the first operation unit. When the first operation unit completes the conversion of a set of 8-bit floating-point data to 16-bit floating-point data, the event queue immediately sends a task start signal to the second operation unit. At this time, the second operation unit begins to read the dequantized data blocks from the shared memory to perform matrix multiplication, while the first operation unit continues to process the next set of data to be dequantized. Since there is a difference between the bit width expansion conversion time of the dequantization operation and the calculation time of the matrix multiplication, the effective working time of the two operation units can be partially overlapped by dynamically adjusting the data block size. For example, when the first operation unit completes the dequantization of 8 data blocks, the second operation unit may have just completed the multiplication operation of the first 6 data blocks. At this time, the two form two parallel execution intervals on the time axis.

[0132] refer to Figure 9 , Figure 9 A data flow diagram of the matrix operation method for model data provided in an embodiment of the present application; in some embodiments, the matrix operation method for model data also includes synchronizing the parallel execution of the first operation unit and the second operation unit through a hardware event queue, so that the first operation unit triggers the second operation unit to start scheduling matrix multiplication after completing a set of dequantized data, wherein the operation time of the first operation unit and the second operation unit partially overlap.

[0133] like Figure 9 As shown, Figure 9The figure shows the parallel computing process of AIV and AIC of the Ascend910B chip in this application, as well as the synchronization mechanism between them. Among them, AIV is responsible for dequantizing fp8 data into fp16 data. The figure shows multiple modules for dequantizing fp8 into fp16, indicating that AIV can handle the dequantization tasks of multiple copies of data at the same time, realize parallel computing, and improve data processing efficiency. AIC is responsible for performing matrix multiplication operations. In the figure, there are multiple matrix multiplication modules (matrix multiplication 1, matrix multiplication 2, etc.) in the AIC pipeline, which means that AIC can handle multiple matrix multiplication tasks at the same time, also realizing parallel computing and being able to quickly complete large-scale matrix operations.

[0134] Furthermore, the dotted line represents the synchronization signal, which is a key mechanism for coordinating the operations between the AIV and AIC. Each time the AIV completes an fp8 dequantization task to fp16, it notifies the AIC via a synchronization signal. After completing a dequantization task, the AIV passes the processed fp16 data to the AIC. Upon receiving the synchronization signal and data, the AIC immediately activates the corresponding matrix multiplication module to perform matrix multiplication on the received fp16 data. This enables efficient integration of data processing and matrix calculations, avoids data waiting and idle computing units, and improves the overall system's operational efficiency.

[0135] Among them, the hardware event queue refers to a communication mechanism used to coordinate task scheduling between different operation units and is used to send synchronization signals. It can be implemented by a ring buffer implemented by internal registers of the chip. The queue can record the task status of the operation unit in real time and trigger cross-unit event responses. Parallel execution of operation units means that the dequantization operation and the matrix multiplication operation have overlapping segments in the time dimension. It can be achieved through a pipelined task scheduling strategy, which enables the intermediate results output by the previous operation unit to be immediately obtained by the subsequent operation unit. Triggering the scheduling of matrix multiplication means automatically activating the matrix operation instruction when the dequantization data reaches the preset threshold. It can be achieved through an interrupt signal or a memory address polling mechanism, which can eliminate the idle waiting time between operation units.

[0136] In summary, the coordinated use of synchronization signals ensures efficient and orderly data flow between the AIV and AIC. Once the AIV completes data dequantization, it immediately notifies the AIC and passes the data along, which then begins matrix multiplication. The entire process seamlessly leverages the parallel computing capabilities and hardware resource efficiency of the Ascend 910B chip.

[0137] According to another embodiment, the present application also provides a dequantization device for model data, which can be set at Figure 1The AIV unit in the illustrated architecture is a dequantization device for model data, comprising: a data determination module for determining first floating-point data to be dequantized in the model; a first conversion module for treating the first floating-point data as first integer data having the same bit width as the first integer data, and performing a first type conversion process to expand the first bit width to a second bit width to generate intermediate floating-point data; a second conversion module for performing a second type conversion process on the intermediate floating-point data to maintain the second bit width, and generate second integer data; a data correction module for performing a format correction process on the second integer data to generate dequantized second floating-point data; wherein the first floating-point data is stored in a first storage area; and the intermediate floating-point data, the second integer data, and the second floating-point data are time-shared and multiplexed in the same second storage area.

[0138] Among them, the dequantization device of the model data can sequentially perform four stages of data identification, bit width expansion, format conversion and binary correction during operation. In the data identification stage, the data determination module identifies the quantization weight data stored in the FP8 format by scanning the operator input features of the model calculation graph. In the bit width expansion stage, the first conversion module inputs the FP8 floating-point number as int8 integer data and expands it to a 16-bit floating-point number through the cast instruction built into the Ascend910B chip. In the format conversion stage, the second conversion module performs fixed-point processing on the 16-bit floating-point data to generate 16-bit integer data containing the original information of the sign bit, exponent bit and mantissa bit. In the binary correction stage, the data correction module completes the format conversion by clearing the upper 8 bits of invalid data, correcting the exponent offset and adjusting the mantissa alignment according to the bit field distribution of the target FP16 format.

[0139] In some specific embodiments, the data determination module can automatically mark quantization nodes using a computational graph traversal algorithm, for example, by detecting a scale factor parameter in an operator attribute to identify quantized data. The first conversion module can call the vconv instruction in the chip instruction set to implement lossless expansion from 8 bits to 16 bits, while the second conversion module uses the vf2iz instruction to complete the floating-point to fixed-point conversion operation. The bit mask clearing operation in the data correction module can be implemented using a bitwise AND operation, for example, performing a logical AND operation on 16-bit data with 0x00FF to remove redundant data in the upper 8 bits.

[0140] In some embodiments, the dequantization device of the model data is applied to or includes the AIV in the above embodiments.

[0141] According to another embodiment, the present application also provides a chip, which may have Figure 1In the system architecture, the chip includes: a first operation unit, used to execute the inverse quantization method of the model data of any one of the first aspects to obtain the inverse quantized second floating-point data; a second operation unit, used to perform a matrix multiplication operation based on the second floating-point data to obtain a matrix operation result.

[0142] In some embodiments, the chip includes an Ascend910B chip, the first operation unit includes an AIV, and the second operation unit includes an AIC, wherein the chip improves computing efficiency through pipeline collaboration between hardware units. After the first operation unit receives an 8-bit floating-point input, it first treats it as an 8-bit integer and performs a 16-bit extension to generate intermediate floating-point data. Subsequently, the 16-bit width is maintained for secondary type conversion, and after clearing the upper 8-bit redundant data through a bit mask, the exponent offset is recalculated and the mantissa alignment is adjusted to generate 16-bit floating-point data that complies with the IEEE754 standard. The second operation unit receives the processed standardized data and calls the matrix multiplication instruction to complete the tensor calculation. The two operation units achieve overlapping calculation processes through the hardware event queue. For example, when the first unit completes the dequantization of part of the data, it triggers the second unit to start the calculation task.

[0143] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiment. The device embodiment described above is only exemplary, in which the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without making any creative efforts.

[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0145] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0146] And an electronic device comprising:

[0147] one or more processors; and

[0148] A memory associated with one or more processors, the memory being used to store program instructions, which, when read and executed by one or more processors, execute the steps of any one of the method embodiments described above.

[0149] The present application also provides a computer program product, comprising a computer program, which implements the steps of any one of the method embodiments described above when executed by a processor.

[0150] in, Figure 10 The electronic device architecture is shown as an example, and may include a processor 1010, a video display adapter 1011, a disk drive 1012, an input / output interface 1013, a network interface 1014, and a memory 1020. The processor 1010, the video display adapter 1011, the disk drive 1012, the input / output interface 1013, the network interface 1014, and the memory 1020 may be communicatively connected via a communication bus 1030.

[0151] Among them, the processor 1010 can be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.

[0152] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store an operating system 1021 for controlling the operation of the electronic device 1000, and a basic input and output system (BIOS) 1022 for controlling the low-level operations of the electronic device 1000. In addition, a web browser 1023, a data storage management system 1024, and a dequantization device 1025 for model data, etc. can also be stored. The dequantization device 1025 for the above-mentioned model data can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0153] The input / output interface 1013 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0154] The network interface 1014 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WIFI, Bluetooth, etc.).

[0155] The bus 1030 comprises a pathway for transmitting information between the various components of the device (eg, the processor 1010 , the video display adapter 1011 , the disk drive 1012 , the input / output interface 1013 , the network interface 1014 , and the memory 1020 ).

[0156] It should be noted that although the above device only shows the processor 1010, the video display adapter 1011, the disk drive 1012, the input / output interface 1013, the network interface 1014, the memory 1020, the bus 1030, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may also include only the components necessary to implement the solution of the present application, and does not necessarily include all the components shown in the figure.

[0157] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application or certain parts of the embodiments.

[0158] The above describes the technical solution provided by this application in detail. Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the contents of this specification should not be understood as limiting this application.

Claims

1. A method for dequantizing model data, wherein: The dequantization method comprises: Determining first floating-point data to be dequantized in the model; Treating the first floating-point data as first integer data having the same bit width as the first integer data, and performing a first type conversion process to expand the first bit width to the second bit width, so as to generate intermediate floating-point data; Performing a second type conversion process on the intermediate floating-point data while maintaining the second bit width to generate second integer data; Performing format correction processing on the second integer data to generate inverse-quantized second floating-point data; The first floating-point data is stored in a first storage area; the intermediate floating-point data, the second integer data and the second floating-point data are time-shared in the same second storage area.

2. The dequantization method of model data according to claim 1, characterized in that: The performing format correction processing on the second integer data to generate inverse quantized second floating-point data includes: Obtaining format layout information of a sign bit, an exponent bit, and a mantissa bits in the first floating-point data; Based on the format layout information, format correction processing is performed on the second integer data to generate inverse quantized second floating-point data.

3. The dequantization method of model data according to claim 2, characterized in that: The step of performing format correction processing on the second integer data based on the format layout information to generate inverse-quantized second floating-point data includes: Based on a preset bit mask, clear invalid data generated by bit width expansion in the upper 8 bits of the second integer data; Calculating a binary correction value according to the original bias value of the exponent bit in the format layout information and the exponent offset of the target floating-point format, and performing superimposition correction on the second integer data according to the binary correction value to generate bias-corrected data; performing a shift operation on the offset correction data with the goal of aligning the offset correction data with the target floating-point format based on the sign bit, the exponent bit, and the mantissa bit in the format layout information, to generate alignment correction data; The alignment correction data is multiplied by a preset scaling factor to generate inverse quantized second floating-point data.

4. The dequantization method of model data according to any one of claims 1 to 3, characterized in that: The inverse quantization method is applied to an AI acceleration chip including a vector calculation unit and a matrix calculation unit. The type conversion processing is implemented based on a bit width extension instruction built into the AI ​​acceleration chip. The first bit width is 8 bits and the second bit width is 16 bits.

5. A matrix operation method for model data, characterized in that: The matrix operation method comprises: In the first operation unit, executing the dequantization method of the model data according to any one of claims 1 to 4 to obtain dequantized second floating-point data; In the second operation unit, a matrix multiplication operation is performed based on the second floating-point data to obtain a matrix operation result.

6. The matrix operation method of model data according to claim 5, characterized in that: The matrix operation method further includes: The first operation unit and the second operation unit are synchronized to execute in parallel through a hardware event queue, so that the first operation unit triggers the second operation unit to start scheduling matrix multiplication after completing a set of dequantized data; wherein the operation time of the first operation unit and the second operation unit partially overlap.

7. A dequantization device for model data, characterized in that: include: A data determination module, configured to determine first floating-point data to be dequantized in the model; a first conversion module, configured to treat the first floating-point data as first integer data having the same bit width as the first integer data, and perform a first type conversion process from the first bit width to the second bit width to generate intermediate floating-point data; a second conversion module, configured to perform a second type of conversion process on the intermediate floating-point data to maintain the second bit width, so as to generate second integer data; a data correction module, configured to perform format correction processing on the second integer data to generate inverse-quantized second floating-point data; Wherein, the first floating-point data is stored in a first storage area; The intermediate floating-point data, the second integer data and the second floating-point data are time-shared and multiplexed in the same second storage area.

8. A chip, characterized in that: include: a first operation unit, configured to execute the dequantization method for model data according to any one of claims 1 to 4 to obtain dequantized second floating-point data; The second operation unit is used to perform a matrix multiplication operation based on the second floating-point data to obtain a matrix operation result.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the inverse quantization method of the model data described in any one of claims 1 to 4 and the matrix operation method of the model data described in any one of claims 5 to 6 are implemented.

10. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the program instructions execute the dequantization method of the model data described in any one of claims 1 to 4, and the matrix operation method of the model data described in any one of claims 5 to 6.

Images