Dequantization method for model data, matrix operation method, and related devices

Abstract

The embodiment of the application discloses a model data dequantization method, a matrix operation method and related equipment, wherein the dequantization method comprises the following steps: determining first floating point data to be dequantized in a model; regarding the first floating point data as first integer data with the same bit width as the first floating point data, performing first type conversion processing from the first bit width to the second bit width to generate intermediate floating point data; performing second type conversion processing on the intermediate floating point data to maintain the second bit width to generate second integer data; and performing format correction processing on the second integer data to generate second floating point data after dequantization. The application can simplify the dequantization process, improve the calculation efficiency, adapt to some chips with instruction set restrictions and lack of support for special computing instructions for low-precision floating point data, and improve the overall performance of artificial intelligence model training.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence model computation technology, and in particular to a method for inverse quantization of model data, a matrix operation method, and related equipment. Background Technology

[0002] In AI model training and inference scenarios, low-precision floating-point data is typically used for computation to reduce computational and storage overhead. However, current model training methods have certain limitations at the hardware implementation level. Some model training chips have incomplete instruction sets and lack dedicated computational instructions for low-precision floating-point data. For example, the Ascend910B chip does not have dedicated instructions to directly support fp8 calculations. This necessitates converting low-precision floating-point data to high-precision floating-point data before performing matrix multiplication operations to adapt to the chip's built-in high-precision computational instructions.

[0003] In existing model training methods, the dequantization process usually requires multiple bit operations, which is cumbersome and complex. This not only increases the computational space requirements but also reduces computational efficiency, thereby affecting the overall performance of artificial intelligence model training. Summary of the Invention

[0004] This application provides a method for dequantizing model data, a matrix operation method, and related equipment, which can simplify the dequantization process, improve computational efficiency, and thus enhance the overall performance of artificial intelligence model training.

[0005] This application provides the following solution:

[0006] According to a first aspect, a method for dequantizing model data is provided, wherein the dequantization method includes: determining first floating-point data to be dequantized in the model; treating the first floating-point data as first integer data with the same bit width, and performing a first type conversion process to expand from the first bit width to the second bit width to generate intermediate floating-point data; performing a second type conversion process on the intermediate floating-point data while maintaining the second bit width to generate second integer data; performing a format correction process on the second integer data to generate dequantized second floating-point data; wherein the first floating-point data is stored in a first storage area; and the intermediate floating-point data, the second integer data, and the second floating-point data time-division multiplex the same second storage area.

[0007] According to one achievable method in an embodiment of this application, the step of performing format correction processing on the second integer data to generate the dequantized second floating-point data includes: obtaining the format layout information of the sign bit, exponent bit, and mantissa bit in the first floating-point data; and performing format correction processing on the second integer data based on the format layout information to generate the dequantized second floating-point data.

[0008] According to one achievable method in this application embodiment, the step of performing format correction processing on the second integer data based on the format layout information to generate dequantized second floating-point data includes: based on a preset bitmask, clearing invalid data in the high 8 bits of the second integer data caused by bit width expansion; calculating a binary correction value based on the original bias value of the exponent bit in the format layout information and the exponent offset of the target floating-point format, and performing superposition correction on the second integer data based on the binary correction value to generate bias correction data; performing a shift operation on the bias correction data based on the sign bit, exponent bit, and mantissa bit in the format layout information with the aim of aligning with the target floating-point format to generate alignment correction data; and multiplying the alignment correction data by a preset scaling factor to generate dequantized second floating-point data.

[0009] According to one achievable method in an embodiment of this application, the dequantization method is applied to an AI acceleration chip that includes a vector calculation unit and a matrix calculation unit. The type conversion process is implemented based on the bit width extension instruction built into the AI ​​acceleration chip, wherein the first bit width is 8 bits and the second bit width is 16 bits.

[0010] According to the second aspect, a matrix operation method for model data is provided, the matrix operation method comprising: in a first operation unit, performing the dequantization method of the model data as described in the first aspect to obtain dequantized second floating-point data; and in a second operation unit, performing matrix multiplication operation based on the second floating-point data to obtain a matrix operation result.

[0011] According to one achievable method in this application embodiment, the matrix operation method further includes: synchronizing the parallel execution of the first operation unit and the second operation unit through a hardware event queue, so that the second operation unit is triggered to start scheduling matrix multiplication after the first operation unit completes each set of dequantized data; wherein the operation time of the first operation unit and the second operation unit partially overlaps.

[0012] According to a third aspect, a dequantization device for model data is provided, comprising: a data determination module for determining first floating-point data to be dequantized in the model; a first conversion module for treating the first floating-point data as first integer data with the same bit width, and performing a first type conversion process to expand from the first bit width to the second bit width to generate intermediate floating-point data; a second conversion module for performing a second type conversion process on the intermediate floating-point data while maintaining the second bit width to generate second integer data; and a data correction module for performing a format correction process on the second integer data to generate dequantized second floating-point data; wherein the first floating-point data is stored in a first storage area; and the intermediate floating-point data, the second integer data, and the second floating-point data time-division multiplex the same second storage area.

[0013] According to the fourth aspect, a chip is provided, comprising: a first arithmetic unit, configured to execute the dequantization method of the model data described in any one of the first aspects to obtain dequantized second floating-point data; and a second arithmetic unit, configured to perform matrix multiplication based on the second floating-point data to obtain a matrix operation result.

[0014] According to a fifth aspect, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the inverse quantization method for model data according to any one of the first aspects, and the matrix operation method for model data according to any one of the second aspects.

[0015] According to a sixth aspect, an electronic device is provided, comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, perform a method for inverse quantization of model data according to any one aspect of the first aspect, and a method for matrix operation of model data according to any one aspect of the second aspect.

[0016] According to a seventh aspect, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the inverse quantization method for model data as described in any one of the first aspects, and the matrix operation method for model data as described in any one of the second aspects.

[0017] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0018] The dequantization method, matrix operation method, and related equipment proposed in this application can treat low-precision floating-point data as integer data with the same bit width, and then use the bit width extension instructions supported by the chip to perform a two-step type conversion process, achieving lossless extension of low-precision to high-precision data, avoiding the complex operation of extracting the sign, exponent, and mantissa bit by bit in the traditional dequantization process. Subsequently, redundant data is cleared by bit masking, exponent bias is corrected, and single shift alignment is performed to adjust the data layout to the target floating-point format, which can simplify the multi-step cumbersome bit operations required by traditional dequantization, thereby simplifying the process and improving computational efficiency. In addition, the first floating-point data is stored in the first storage area, and the intermediate floating-point data, the second integer data, and the second floating-point data share the same second storage area in a time-sharing manner, which can reduce the computational space requirements and effectively save computing resources. This allows it to be adapted to some chips with instruction set limitations and lack of dedicated computational instructions for low-precision floating-point data, thereby improving the overall performance of artificial intelligence model training.

[0019] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 Here is an overall flowchart of the existing inverse quantization method in the background technology;

[0022] Figure 2 System architecture diagram applicable to the inverse quantization method for model data provided in the embodiments of this application;

[0023] Figure 3 A flowchart illustrating the method steps for the inverse quantization of model data provided in this application embodiment;

[0024] Figure 4 A schematic diagram of the first floating-point data in the dequantization method for model data provided in the embodiments of this application;

[0025] Figure 5 A schematic diagram illustrating the generation of second integer data in the dequantization method for model data provided in this application embodiment;

[0026] Figure 6 A flowchart illustrating the steps of performing format correction processing to generate dequantized second floating-point data in the dequantization method for model data provided in this application embodiment;

[0027] Figure 7 A flowchart illustrating the overall process of inverse quantization of model data provided in this application embodiment;

[0028] Figure 8 A flowchart illustrating the matrix operation method for model data provided in the embodiments of this application;

[0029] Figure 9 A schematic diagram of the data pipeline for the matrix operation method of the model data provided in the embodiments of this application;

[0030] Figure 10 A schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0032] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0033] It should be understood that the term "and / or" used in this document is only a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0034] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0035] Currently, low-precision floating-point data is widely used in AI model training and inference scenarios because it effectively reduces computational and storage overhead. However, at the hardware implementation level of model training chips, some chips have instruction set limitations and lack dedicated computational instructions for low-precision floating-point data. For example, the Ascend910B chip does not have dedicated instructions that directly support fp8 calculations. Therefore, before performing matrix multiplication operations, low-precision floating-point data must be dequantized into high-precision floating-point data to adapt to the chip's built-in high-precision computational instructions.

[0036] Existing dequantization methods typically require multiple bit operations, which are cumbersome and complex, resulting in high computational space requirements, reduced computational efficiency, and consequently impacting the overall performance of AI model training.

[0037] Among them, fp8 is an 8-bit floating-point instruction, which can significantly reduce memory usage and power consumption while maintaining high computational accuracy. The Ascend910B is an AI training chip using the DaVinci architecture, which has a Cube operation unit designed for efficient matrix multiplication. However, the Ascend910B currently does not support dedicated computation instructions of type fp8. Therefore, the computation of fp8 on the Ascend910B can only be achieved indirectly through algorithms.

[0038] refer to Figure 1 , Figure 1 The diagram below illustrates the overall flowchart of existing dequantization methods in the background technology. It's understandable that in some existing solutions, performing fp8 matrix multiplication on the Ascend910B chip involves a complex dequantization process because neither the chip's AIV nor AIC instructions support fp8 type operations. Specifically, after fp8 data is read into the UB (Unified Data Unit), it becomes a tensor_fp8, requiring a series of combined operations using AIV instructions to achieve dequantization. First, according to tensor_sign_temp = tensor_fp8 >> 7, the fp8 data is right-shifted by 7 bits to obtain tensor_sign_temp. Then, tensor_sign_temp is bitwise ANDed with 0x1 to extract the sign bit (tensor_sign). Next, tensor_E_temp = tensor_fp8 >> 3, that is, the fp8 data is right-shifted by 3 bits to obtain tensor_E_temp. Tensor_E_temp is bitwise ANDed with 0xF to obtain the exponent bit (tensor_E_fp8). Finally, the fp8 data is bitwise ANDed with 0x7 to obtain the mantissa bit (tensor_M), thus realizing the bit extraction operation.

[0039] Furthermore, in subsequent steps, the sign bit (tensor_sign) is shifted left by 15 bits, the exponent bit (tensor_E_fp8) is added to the bias value of 8 and then shifted left by 10 bits, and the mantissa bit (tensor_M) is shifted left by 7 bits. These processed bit information are combined into temporary fp16 data (tensor_fp16_temp). However, the data obtained at this point requires further processing. A bitwise OR operation is performed between the temporary fp16 data and the processed mantissa bit data to form complete fp16 format data. This data is then multiplied by a scale factor to obtain tensor_fp16 for matrix multiplication. This completes the dequantization from fp8 to fp16, and the fp16 matrix can be used for matrix multiplication using AIC. It is estimated that the entire dequantization process currently performed on the Ascend910B chip requires at least 11 computational steps and at least three copies of the data storage space to temporarily store intermediate results. Furthermore, existing solutions do not adequately consider the doubling of data space required when converting from fp8 to fp16. This could lead to data overflow or insufficient storage space during actual dequantization, affecting the smooth progress of dequantization. This cumbersome process not only increases the computational burden but also limits the efficiency of the Ascend910B chip in processing fp8 data, highlighting the necessity of developing a more efficient dequantization method.

[0040] In summary, the current technical problems can be summarized as follows: Chips such as the Ascend910B currently lack dedicated computation instructions for fp8 type, making direct fp8 matrix multiplication impossible. Therefore, existing dequantization schemes are required. However, these schemes suffer from efficiency issues, requiring the extraction of the sign bit, exponent bit, and mantissa bit separately, followed by correction and recombination, totaling 11 computational steps and consuming significant computational space. Furthermore, existing schemes do not consider the change in space required for each number from 8 bits to 16 bits when converting fp8 to fp16 on the Ascend910B, directly impacting the implementation of dequantization.

[0041] In view of this, this application provides a new approach. To facilitate understanding of this application, the system architecture on which this application is based is first described. (References) Figure 2 , Figure 2 This is a system architecture diagram applicable to the inverse quantization method for model data provided in the embodiments of this application. Figure 2 An exemplary system architecture that can be applied to embodiments of this application is shown, such as Figure 2 As shown, the inverse quantization method in this application can be applied to the Ascend910B chip, and the system architecture of the Ascend910B chip may include:

[0042] Global Memory (GM): Global Memory is the storage unit in the entire system architecture, used to store large amounts of data and provide a data source for the entire chip's data processing. When performing matrix calculations, the data is first stored in GM, waiting to be called and processed, and then can interact with other parts of AICore through the transfer unit.

[0043] AICore: AICore is the computing unit within the Ascend910B chip, responsible for various artificial intelligence computing tasks, including matrix operations. It is divided into a matrix computing part (AICube, AIC) and a vector computing part (AIVector, AIV), with data transfer between the two parts via GlobalMemory.

[0044] AICube (AIC): The AIC is used for matrix multiplication calculations and contains multiple buffers, such as LOABuffer, LOBBuffer, and LOCBuffer, which are used to temporarily store input and output data during the matrix multiplication process. The AIC also contains a Cube calculation unit, which is dedicated to performing matrix multiplication operations. During matrix multiplication, data is moved from GM to the corresponding buffer in the AIC through transport units (MTE1 / MTE2 / MTE3), then the calculation is performed in the Cube unit, and finally the result is returned to GM through the transport unit.

[0045] AIVector (AIV): AIV is used for vector computation operations and contains Scalar and Vector computation units. In the dequantization process of this application, AIV is used to dequantize fp8 into fp16. AIV includes a Unified Buffer (UB) for temporarily storing the data required during vector computation.

[0046] Transfer Units (MTE1 / MTE2 / MTE3): These transfer units are used to transfer data between GM and AICore. They can move data from GM to the corresponding buffer in AIC or AIV, and can also move computation results from AIC or AIV back to GM.

[0047] FixPipe and fpBuffer: FixPipe is a fixed-point pipeline used to handle tasks such as fixed-point operations or data preprocessing, while fpBuffer is used to temporarily store floating-point related data and may play a role in matrix multiplication or other floating-point operations.

[0048] Scalar Unit: The Scalar unit is used to handle scalar computation tasks. In AI computation, in addition to matrix and vector operations, there are also scalar-related computations, such as updating some control parameters and simple mathematical operations. The Scalar unit can work together with the Vector unit in AIV to complete complex computation tasks.

[0049] In some embodiments, it is understood that data starts from GM, enters the corresponding buffer of AIC or AIV in AICore through the transport unit (MTE1 / MTE2 / MTE3), and after fp8 to fp16 in AIV is dequantized, fp16 data is transported to the buffer of AIC (such as LOABuffer or LOBBuffer) through the transport unit. Then, matrix multiplication is performed in the Cube calculation unit of AIC, and the calculation result is returned to GM through the transport unit. At the same time, when the data is processed in AIV, it may undergo some preprocessing through FixPipe, and the Scalar unit will also participate in the relevant scalar calculation.

[0050] It is worth noting that, Figure 2 The architecture divides the AICore into two independent cores, AIC and AIV, enabling parallel matrix and vector computations and improving the overall chip's computational efficiency. The dequantization process described in this application can be applied to AIV, allowing it to dequantize fp8 data into fp16 data. This data is then transferred to the AIC via a data transfer unit for matrix multiplication, thus realizing general fp8 matrix multiplication functionality based on the Ascend910B chip. This architecture allows dequantization and matrix multiplication operations to work efficiently together, fully leveraging the computational capabilities of the Ascend910B chip.

[0051] refer to Figure 3 , Figure 3 This is a flowchart illustrating the method steps of the inverse quantization method for model data provided in this application embodiment. This method can be performed by... Figure 2 The AIV unit in the system shown executes. For example... Figure 3 As shown, the method may include the following steps:

[0052] Step S310: Determine the first floating-point data to be dequantized in the model;

[0053] Step S320: Treat the first floating-point data as the first integer data with the same bit width, and perform a first type conversion process to expand from the first bit width to the second bit width to generate intermediate floating-point data;

[0054] Step S330: Perform a second type conversion process on the intermediate floating-point data to maintain the second bit width, and generate the second integer data;

[0055] Step S340: Perform format correction processing on the second integer data to generate the dequantized second floating-point data.

[0056] The first floating-point data is stored in the first storage area; the intermediate floating-point data, the second integer data, and the second floating-point data share the same second storage area in a time-sharing manner.

[0057] As can be seen from the above process, this application can treat low-precision floating-point data as integer data with the same bit width, and then use the chip's supported cast instruction to perform a two-step type conversion process to achieve lossless expansion of low-precision to high-precision data. For example, the first step converts fp8 data to fp16 intermediate floating-point data, and the second step expands it into 16-bit integer data, avoiding the complex operation of extracting the sign, exponent, and mantissa bit by bit in the traditional scheme. Subsequently, redundant data is cleared by bit masking, exponent bias is corrected, and single shift alignment is performed to adjust the data layout to the target floating-point format. This simplifies the multi-step cumbersome bit operations required by traditional dequantization, thereby simplifying the process, improving computational efficiency, and adapting to some chips with instruction set limitations and lacking dedicated computational instructions for low-precision floating-point data, thus improving the overall performance of artificial intelligence model training.

[0058] In some embodiments, the dequantization method is applied to an AI acceleration chip that includes a vector computation unit and a matrix computation unit. The type conversion processing is implemented based on the bit-width extension instruction built into the AI ​​acceleration chip, with the first bit width being 8 bits and the second bit width being 16 bits. It is worth noting that the AI ​​acceleration chip in this application includes the Ascend910B, and the built-in bit-width extension instruction includes the cast instruction in the Ascend910B chip. In the AIVector unit of the Ascend910B chip, the unified buffer (UB) stores intermediate floating-point data (fp16) and subsequent integer data (int16) in the same physical address region through the hardware address remapping function. This physical address region is the second storage region. The second storage region is directly overwritten with the second floating-point data (fp16) during the format correction stage, realizing zero-copy reuse of storage resources.

[0059] In some embodiments, during the dequantization process, the fp8 data in the first storage area is first treated as int8 data and converted to fp16 data using a cast instruction. At this time, the second storage area in the UB is remapped, and the second storage area is now used to store the converted fp16 data. Next, the fp16 data is converted to int16 data again using a cast instruction, still using the second storage area, replacing the original fp16 data. In the format correction stage, bitmask clearing, bias correction, and shift operations are performed directly on the int16 data in the second storage area to obtain fp16 data, which directly overwrites the previous int16 data, completing zero-copy multiplexing. Through the above mechanism, intermediate floating-point data, integer data, and final floating-point data time-division multiplex the second storage area, requiring only two storage spaces (original data storage area + dynamic multiplexing area) to complete the entire dequantization process. The entire process does not require additional storage space to temporarily store intermediate results, and all operations are completed in the same storage area.

[0060] Understandably, this application only requires two storage spaces throughout the entire dequantization process: one for storing the original fp8 data and the other for storing the fp16 data. In contrast, existing technologies typically require at least three storage spaces: one for storing the original fp8 data, one for storing intermediate results (such as the sign bit, exponent bit, and mantissa bit), and one for storing the fp16 data. Therefore, this application can reduce the computational space requirement and effectively save computing resources.

[0061] The following describes in detail each step of the above process and the effects that can be further produced, with reference to the embodiments. It should be noted that the terms "first" and "second" involved in this disclosure do not have limitations on size, order, or quantity, but are only used to distinguish them by name. For example, "first floating-point data" and "second floating-point data" are used to distinguish two floating-point data.

[0062] First, step S310, namely "determining the first floating-point data to be dequantized in the model," will be described in detail with reference to the embodiment. The first floating-point data refers to the low-precision floating-point data to be converted, such as fp8 format data, whose storage format includes a sign bit, exponent bit, and mantissa bit. By treating it as integer data, the chip's built-in bit-width extension instruction can be directly called to complete the initial conversion. The first type of conversion processing refers to the data bit-width extension operation implemented through the cast instruction, such as extending an 8-bit integer to a 16-bit integer. This operation utilizes the characteristics of the chip's hardware instructions, avoiding the complexity of manually splitting data bits. The second type of conversion processing refers to the data format conversion performed while maintaining the extended bit width, such as converting 16-bit integer data to a 16-bit floating-point format. This step establishes a standardized data foundation for subsequent format correction. Format correction processing refers to the operation of bit alignment and numerical calibration of intermediate data based on the characteristics of the original floating-point format, such as adjusting the exponent bias and mantissa alignment. Invalid data is cleared through a preset bitmask, and a correction value is superimposed to achieve format restoration.

[0063] In some embodiments, the first floating-point data is fp8 data. The sub-formats of the first floating-point data may include: E4M3 (4 exponent bits + 3 mantissa bits), with a smaller dynamic range (maximum representation ±448), but higher mantissa precision, suitable for inference tasks; and E5M2 (5 exponent bits + 2 mantissa bits), with a larger dynamic range (maximum representation ±57344), but lower mantissa precision, suitable for training tasks. It is understood that the format of fp16 dequantized from the two fp8 sub-formats is unique. Different sub-formats only affect some parameters during computation, such as the number of bits required for left shift and the different values ​​for exponent bit bias correction, etc., and do not affect the overall implementation logic of the scheme in this application.

[0064] refer to Figure 4 , Figure 4 A schematic diagram of the first floating-point data in the dequantization method for model data provided in the embodiments of this application; it can be understood that, as shown in the figure... Figure 4 This is a data diagram illustrating the E4M3 fp8 type in the embodiments of this application. Floating-point numbers in computers typically follow the IEEE 754 standard, and their binary representation consists of three parts: the sign bit, the exponent, and the mantissa / fraction.

[0065] The value V of a floating-point number can be represented as:

[0066] V = (-1) S ×M×2 E-B (1);

[0067] in:

[0068] S: Sign bit (1 bit), determines positive or negative.

[0069] M: Mantissa, the formula for converting mantissa M into a value (normalized number), n is the length, b is the binary representation value, and the traversal order is from high bit to low bit:

[0070]

[0071] M = 1 + Fraction(3);

[0072] E: Exponent, stored as an unsigned integer, the actual exponent E real The bias value B needs to be subtracted:

[0073] E real =EB (4);

[0074] B: Bias, which is used to allow the exponent E to represent negative numbers. B = 2 k-1 -1 (k is the number of exponent bits).

[0075] Furthermore, the above step S320, namely "treating the first floating-point data as a first integer data with the same bit width and performing a first type conversion process from the first bit width to the second bit width to generate intermediate floating-point data", will be described in detail with reference to the embodiments. In some embodiments, treating the first floating-point data as a first integer data with the same bit width and performing a first type conversion process from the first bit width to the second bit width to generate intermediate floating-point data includes treating the 8-bit first floating-point data as an 8-bit first integer data and performing a cast instruction conversion from 8 bits to 16 bits to generate 16-bit intermediate floating-point data.

[0076] Cast instruction conversion refers to the hardware instruction operation that converts data from one type to another. This can be achieved by calling the chip's built-in bit-width extension instruction. This operation maintains data precision and avoids information loss. Bit-width extension refers to expanding the data storage space from a smaller capacity to a larger capacity. This can be achieved by padding the high-order bits with zeros or copying the sign bit. For example, padding the high 8 bits of an 8-bit integer with zeros creates a 16-bit integer. This extension method can adapt to the input requirements of subsequent high-precision calculations.

[0077] It is worth noting that, in this embodiment of the application, the redefinition function of the Ascend910B chip can be used to treat the first floating-point data as the first integer data with the same bit width. Redefinition refers to reinterpreting the data on the same memory block as another data type without modification, which facilitates the use of some calculation instructions. For example, the Ascend910B does not support the fp8 instruction, so this embodiment of the application redefines fp8 as int8 to perform the above steps, and dequantizes fp8 into fp16 through AIV operation.

[0078] It is worth noting that fp8 can be treated as int8 and converted to fp16 using the Ascend hardware's cast instruction. This is because the value range of int8 is [-128:127], while fp16 can accurately represent the range of [-2048:2048]. Therefore, the casting process will not lose precision.

[0079] Tensor int8 =Tensor fp8 .view(int8) (5);

[0080] Tensor fp16 =cast(Tensor) int8 (6);

[0081] The precise range of fp16 is [-2048:2048]. A floating-point number can precisely represent an integer if that integer can be precisely represented in the form of 1.M × 2^E, and the exponent range allows the mantissa to cover all integer bits. For float16, the mantissa has 11 significant bits (including the implicit 1), so the largest unambiguous integer range is [-2^11, 2^11] (i.e., [-2048, 2048]).

[0082] Specifically, when processing 8-bit floating-point data, its binary bit pattern is reinterpreted as 8-bit integer data. By calling the chip's supported cast instruction, this 8-bit integer data is expanded into a 16-bit data storage format, with high-order bits padded with zeros during the expansion process to maintain numerical accuracy. The converted 16-bit intermediate floating-point data retains the integrity of the original quantization information while meeting the data bit width requirements of subsequent high-precision computing units. For example, in the Ascend910B chip, this conversion process can be completed with a single hardware instruction, avoiding computational delays caused by multiple intermediate data transfers.

[0083] Understandably, compared with existing technologies, traditional methods require multiple bit operations and temporary variable storage when processing low-precision floating-point data. This solution directly completes the data format conversion through a single bit-width expansion instruction, effectively solving the problem of complex operation steps in the dequantization process of low-precision floating-point data. The data bit-width expansion is directly completed through hardware instructions, avoiding the multiple bit splitting and splicing operations in traditional methods.

[0084] Furthermore, the above step S330, namely "performing a second type conversion process to maintain the second bit width on intermediate floating-point data to generate second integer data", will be described in detail with reference to the embodiments. In some embodiments, performing a second type conversion process to maintain the second bit width on intermediate floating-point data to generate second integer data includes performing a second type conversion process to maintain the second bit width on intermediate floating-point data to generate second integer data, which includes performing a cast instruction conversion to maintain 16 bits on 16-bit intermediate floating-point data to generate 16-bit second integer data.

[0085] The `cast` instruction refers to the operation of converting data types using hardware instructions. Specifically, it can be implemented using the chip's built-in bit-width extension instruction, which can convert floating-point data to integer data without changing the data bit width. Maintaining the second bit width means keeping the data bit width unchanged during type conversion. This can be implemented using a fixed-width register to avoid introducing additional bit-width adjustment operations during the conversion process.

[0086] Specifically, when the intermediate floating-point data is 16 bits, the chip's supported cast instruction is directly invoked to perform data format conversion. This conversion process does not change the data bit width; it only converts the memory representation of the floating-point data into integer data. During this process, the high 8 bits and low 8 bits of the data maintain a complete mapping relationship, avoiding data truncation or padding due to changes in bit width. By maintaining 16-bit operation, the converted second integer data can directly participate in subsequent format correction processing without the need for additional intermediate buffers or temporary variables.

[0087] refer to Figure 5 , Figure 5 The diagram illustrates the generation of the second integer data in the dequantization method for model data provided in the embodiments of this application. In some embodiments, the data type of fp16 needs to be converted to int16 to restore the floating-point number to an integer and discard the decimal places. Therefore, the conversion of fp16 to int16 is performed. The precision range of int16 is also larger than [-128:127], so there will be no loss of precision.

[0088] Tensor int16 =cast(Tensor) fp16 (7);

[0089] like Figure 5 The results show that each number occupies 16 bits. We can see that for positive numbers, the high 8 bits are filled with 0s, while for negative numbers, the high 8 bits are filled with 1s. This is because the int type uses two's complement to store data. For example:

[0090] 2'b10010011 minus 1 and inverted = -2'b01101101 = -109;

[0091] 2'b1111111110010011 minus 1 and inverted = -2'b0000000001101101 = -109;

[0092] Both represent the same negative int value. Therefore, in this embodiment, it is not necessary to extract the sign bit. The sign bit will not change during the next left shift. So after the cast is completed, the high 8 bits of the current data are filled with the sign bit, and the low 8 bits are filled with the original fp8 data. Since fp8->int8 does not require hardware calculation, this application can extend 8 bits of data to 16 bits without loss through two-step conversion (fp8->int8->fp16->int16).

[0093] Understandably, traditional methods typically require step-by-step bit width adjustment and format conversion during type conversion, such as first expanding the bit width through bit shifting operations and then separating the sign bit and value bits. This solution completes the conversion using the cast instruction, simplifying the complex process requiring multiple bit operations into a single instruction operation. From a functional perspective, it efficiently expands the bit width by using two cast operations to losslessly expand the bit width of each number to 16 bits, completing dequantization in four operations. This is significantly better than the original solution's 11 operations, improving the execution efficiency of the chip's arithmetic unit while maintaining data accuracy and reducing resource consumption during computation.

[0094] In some embodiments, the dequantization method for model data is applied to the Ascend910B chip. The type conversion processing is based on the technical solution implemented by the cast instruction built into the Ascend910B chip. The instruction set of the Ascend910B chip includes cast instructions for data type conversion. The cast instruction can directly perform bit width expansion operations on the data stored in the register, such as converting 8-bit integer data to 16-bit integer data. During the dequantization process, by calling the hardware-level instructions of the chip, the additional overhead caused by implementing data type conversion through software simulation can be avoided.

[0095] Specifically, when the Ascend910B chip performs dequantization, its arithmetic unit can directly call the built-in cast instruction set. After treating the 8-bit floating-point data as 8-bit integer data, the cast instruction is executed to expand the bit width from 8 bits to 16 bits, generating intermediate floating-point data. Since this instruction belongs to the chip's low-level hardware instruction, its execution process does not require multiple data transfers or temporary storage; the data type conversion is directly completed through registers. This method fully utilizes the chip's hardware characteristics, allowing the data type conversion operation to be completed within a single instruction cycle. Compared with existing technologies, traditional dequantization methods require software algorithms to expand the bit width on general-purpose processors, requiring multiple shifting, masking operations, and temporary variable storage. This solution directly completes the core conversion operation through hardware instructions, eliminating intermediate data storage links and reducing the number of instruction execution cycles. Especially considering the architectural characteristics of the Ascend910B chip, this method avoids additional calculation steps caused by instruction set mismatch, effectively solving the problem of low instruction execution efficiency in existing dequantization methods under specific chip architectures, and reducing the storage space occupied during calculation, making the entire dequantization process more adaptable to the hardware architecture characteristics of artificial intelligence training chips. It is worth noting that, since the embodiments of this application are applied to the Ascend910B chip, using fp8 type data can save memory in the inference of large models, and can also enable Vector and Cube units to be computed in parallel during the dequantization process, so as to cover most of the time consumed by dequantization.

[0096] Furthermore, the above step S340, namely "performing format correction processing on the second integer data to generate dequantized second floating-point data", will be described in detail with reference to the embodiments. In some embodiments, performing format correction processing on the second integer data to generate dequantized second floating-point data includes obtaining the format layout information of the sign bit, exponent bit and mantissa bit in the first floating-point data, and performing format correction processing on the second integer data based on the format layout information to generate dequantized second floating-point data.

[0097] In floating-point data, the sign bit is the binary bit used to indicate positive or negative, typically implemented as a single bit. It occupies the most significant bit position and determines the sign of the value. The exponent bit is the binary segment representing the magnitude of the value, stored as a fixed-width integer and used to control the scaling of the floating-point number's range. The mantissa bit is the binary segment representing precision information, stored using the remaining bit width to preserve the precision details of the significant digits. Format layout information refers to the bit allocation rules for each functional area of ​​the floating-point data in binary storage. This can be obtained by parsing the IEEE standard or custom specifications of the original floating-point format and is used to guide bit manipulation logic during data format conversion.

[0098] It is worth noting that in this embodiment of the application, the binary data of fp8 is used (processed) as int8 because the sign bit positions of fp8 and int8 are the same, and negative numbers can be represented by two's complement, so they can be directly corresponded; furthermore, after int8->fp16, the original data becomes a floating-point number format, and it needs to be restored to two's complement form by fp16->int16. Then, the redundant sign bit can be cleared, the exponent can be adjusted, and the numerical precision can be restored by scaling factor.

[0099] refer to Figure 6 , Figure 6 The method for dequantizing model data provided in the embodiments of this application includes a flowchart illustrating the steps of performing format correction processing to generate dequantized second floating-point data. In some embodiments, performing format correction processing on second integer data to generate dequantized second floating-point data may include the following steps:

[0100] Step S610: Based on the preset bit mask, remove invalid data from the high 8 bits of the second integer data caused by bit width expansion;

[0101] Step S620: Calculate the binary correction value based on the original bias value of the exponent bit in the format layout information and the exponent offset of the target floating-point format, and then perform superposition correction on the second integer data based on the binary correction value to generate bias correction data.

[0102] Step S630: Based on the sign bit, exponent bit and mantissa bit in the format layout information, perform a shift operation on the offset correction data with the aim of aligning with the target floating-point format to generate alignment correction data.

[0103] Step S640: Multiply the alignment correction data by a preset scaling factor to generate the dequantized second floating-point data.

[0104] In some embodiments, this application further proposes a method for generating dequantized second floating-point data by performing format correction processing on second integer data based on format layout information, comprising: clearing invalid data in the high 8 bits of the second integer data due to bit width expansion based on a preset bit mask; calculating a binary correction value based on the original bias value of the exponent bit in the format layout information and the exponent offset of the target floating-point format, and performing superposition correction on the second integer data based on the binary correction value to generate bias correction data; performing a shift operation on the bias correction data based on the sign bit, exponent bit and mantissa bit in the format layout information with the aim of aligning with the target floating-point format to generate alignment correction data; and multiplying the alignment correction data by a scaling factor input by the user to generate dequantized second floating-point data.

[0105] In this context, a bitmask is a binary template used to filter specific bits. It can be implemented using a mask template with the high 8 bits set to 0 and the low 8 bits to 1. Invalid data in the high 8 bits is cleared using a bitwise AND operation to ensure data precision. A binary correction value is a value used to adjust the exponent bias. It is calculated by the difference between the original bias value and the exponent offset of the target floating-point format. For example, when the original bias is 127 and the target format offset is 1023, the correction value is represented by the binary representation of 896. Addition is used to adjust the bias to the target range. A shift operation refers to the bit-by-bit data movement. Specifically, it moves the corresponding fields of the corrected data to specified positions according to the definition of the sign bit, exponent bit, and mantissa bit of the target floating-point format. For example, shifting the exponent bit left by 23 bits and the mantissa bit right by 7 bits to match a 32-bit floating-point format. A scaling factor is a proportional coefficient used to restore the data's dimensions. It can be determined by the scaling parameters recorded during quantization. For example, using powers of 2 for multiplication operations restores the original floating-point value range.

[0106] Specifically, when the Ascend910B chip performs dequantization, it first applies a bitmask to the 16-bit integer data to eliminate residual data in the high 8 bits, preventing invalid bits from interfering with subsequent calculations. Next, a correction value is generated based on the difference in exponent offset between the source and target data formats. For example, when the source data is in 8-bit floating-point format and the target is in 16-bit floating-point format, an adder is used to compensate for the bias in the intermediate data, ensuring that its exponent field conforms to the representation range of the target format. Subsequently, the bias correction data is rearranged according to the bit field definition of the target floating-point format, for example, by moving the sign bit to the highest bit, shifting the exponent to the middle bit segment, and filling the remaining low bits with the mantissa, thus completing data alignment. Finally, a multiplier applies a scaling factor to the aligned data to restore a floating-point value that meets the target precision, thereby completing the dequantization process. Through this technical solution, this application simplifies the dequantization process while maintaining computational accuracy. By implementing floating-point format conversion through integer data operations and bit manipulations, it significantly reduces computational complexity and improves the execution efficiency of the Ascend910B chip when processing low-precision floating-point data.

[0107] In some embodiments, based on steps S610 to S640, taking the E4M3 format as an example, after the cast is completed, the high 8 bits of the current data are filled with the sign bit, while the low 8 bits are filled with the original fp8 data. Dirty bits and the highest sign bit need to be removed. It is understandable that after expansion to 16 bits, since the E of fp8 is 4 bits and the E of fp16 is 5 bits, the 5th bit of E is the sign bit of the fp8 data before expansion. Subsequent addition will be used to correct the bias. Since the hardware does not support uint16 type addition, only int16 type addition can be used. If it is a negative number, the bias of E cannot be correctly corrected. In this embodiment, only the highest bit needs to be set to 0, because the data at this time is not a true two's complement.

[0108] Therefore, the following operation needs to be performed to set the sign bit and the 5th bit of E to 0:

[0109] Tensor uint16 =Tensor int16 .view(uint16) (8);

[0110] Tensor uint16 =Tensor uint16 &0XEFF7F (9);

[0111] Adjust the bias value to 8. Since no shift operation has been performed yet, the least significant bit of E is 3, so we need to add (8 << 3):

[0112] Tensor int16 =Tensor int16+(8<<3) (10);

[0113] One shift moves the sign, exponent, and mantissa to the correct positions. Since the least significant bit of E is currently in the 3rd bit, it needs to be moved to the 10th bit. After the shift is completed, float8 has been successfully converted to float16.

[0114] Tensor int16 =Tensor int16 <<7 (11);

[0115] Furthermore, multiply by scale:

[0116] Tensor float16 =Tensor float16 *scale (12);

[0117] In summary, 1. By using a two-step conversion (fp8->int8->fp16->int16), 8-bit data is losslessly extended to 16 bits. This involves removing dirty bits and sign bits, correcting biases, performing shift operations, and adjusting the scale factor precision to obtain the correct p16 value. This effectively simplifies the bit manipulation process during the dequantization of low-precision floating-point data, reduces computational resource consumption and memory bandwidth requirements, and improves the efficiency of data preprocessing before matrix multiplication. Furthermore, the standardized correction steps and format layout guidance mechanism ensure the accuracy of floating-point format conversions at different precisions, while providing predictable operating modes for hardware instruction optimization.

[0118] refer to Figure 7 , Figure 7 A flowchart illustrating the overall process of inverse quantization of model data provided in embodiments of this application; in some embodiments, Figure 7 This demonstrates the process of dequantizing fp8 to fp16 on the Ascend910B chip. The specific steps are as follows: First, the float8 data is read into UB (Unified Buffer) to obtain tensor_fp8. UB is a buffer in AIVector (AIV) used to temporarily store data for subsequent processing.

[0119] The tensor_fp8 is redefined as int8, resulting in tensor_int8. This step, through the "redefinition" technique, treats the fp8 data as an int8 type, preparing for subsequent bit-width expansion using the Ascend910B's cast instruction.

[0120] Casting tensor_int8 to float16 yields tensor_float16. Since the range of int8 is [-128:127], while fp16 can accurately represent the range [-2048:2048], this casting process does not result in any loss of precision.

[0121] Casting tensor_float16 to int16 yields tensor_int16. This expands the bit width of each number from 8 bits to 16 bits losslessly, resolving the bit width expansion issue during fp8 to fp16 conversion.

[0122] Perform a bitwise AND operation between tensor_int16 and 0xEF7F (tensor_uint16&0xEF7F) to obtain tensor_uint16. This step is used to remove dirty bits and set the highest bit to 0. Because the E bit of fp8 is 4 bits and the E bit of fp16 is 5 bits, the 5th bit of E is the sign bit of the previous fp8 extension. The bias needs to be corrected later, but the hardware does not support uint16 type addition, only int16 type addition, so the highest bit needs to be set to 0.

[0123] Adding (8<<3) to tensor_int16 corrects the exponent bias, resulting in the corrected tensor_int16. Since dequantizing fp8 to fp16 requires correcting the bias value by 8, and no bit shifting has been performed yet, the least significant bit of E is in the 3rd bit, hence the addition of (8<<3).

[0124] Shifting the corrected tensor_int16 left by 7 bits successfully converts float8 to float16. This is because fp8 has a 3-bit mantissa and fp16 has a 10-bit mantissa. To align the most significant bit of the mantissa, a 7-bit left shift with zeros is needed, while also moving the sign, exponent, and mantissa to their correct positions.

[0125] Read the scale into UB to get tensor_scale, then multiply the tensor_float16 obtained by the shift operation by the scale to get the result of tensor_float16 multiplied by the scale.

[0126] The entire process extends the bit width of each number to 16-bit losslessly through two cast operations, requiring only 4 operations for dequantization. Compared to the original scheme's 11 operations, this effectively improves performance, enhances the execution efficiency of the chip's computing unit while ensuring data accuracy, and reduces resource consumption during the calculation process.

[0127] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0128] According to another embodiment, this application also provides a matrix operation method for model data, including performing a dequantization method on the model data in a first operation unit to obtain dequantized second floating-point data, and performing matrix multiplication operations on the second floating-point data in a second operation unit to obtain a matrix operation result. The first operation unit refers to an independent computing module for performing dequantization processing, which can be implemented using a vector processing unit in a chip, converting low-precision floating-point data to high-precision floating-point data through bit-width expansion and format correction. The second operation unit refers to an independent computing module for performing matrix multiplication, which can be implemented using a matrix multiplication accelerator in a chip, performing parallel computation based on high-precision floating-point data to output the operation result. Hardware event queue synchronization refers to coordinating the execution timing of different operation units through a hardware signal transmission mechanism, which can be implemented using interrupt triggering or status register polling, enabling the dequantization and matrix multiplication operations to be executed in a pipelined parallel manner.

[0129] It is understandable that the first operation unit is the AIV unit in the above example, and the second operation unit is the AIC unit in the above example. When the model data needs to undergo matrix operations, the first operation unit receives the first floating-point data to be processed, generates intermediate floating-point data through bit width expansion and type conversion, and then obtains second floating-point data conforming to the high-precision floating-point format after format correction. At the same time, after receiving the partially dequantized data, the second operation unit immediately starts matrix multiplication calculation. Through the synchronization mechanism of the hardware event queue, the second operation unit is triggered to schedule the calculation task after the first operation unit completes the dequantization of a set of data, so that the time windows of dequantization and matrix multiplication partially overlap, thereby reducing the waiting time.

[0130] refer to Figure 8 , Figure 8The overall flowchart of the matrix operation method for model data provided in the embodiments of this application is shown below. In one embodiment, the lossless inverse quantization process from fp8 to fp16 is completed through the above embodiments. While ensuring data accuracy, the execution efficiency of the chip's computing unit is improved and the resource consumption during the calculation process is reduced. After obtaining the fp16 data, the embodiments of this application can also efficiently perform matrix multiplication operations in the AIC to obtain the required matrix operation results. By moving the scaled fp16 data to the AICube (AIC), the matrix multiplication operation is performed using the Cube computing unit of the AIC. The obtained matrix operation results are used to take advantage of the hardware characteristics of the Ascend910B chip to achieve efficient and low-resource-consumption fp8 matrix multiplication operations.

[0131] In some embodiments, during the matrix operation performed by the chip, a hardware event queue continuously monitors the dequantization task progress of the first arithmetic unit. Once the first arithmetic unit completes the conversion of a set of 8-bit floating-point data to 16-bit floating-point data, the event queue immediately sends a task start signal to the second arithmetic unit. At this time, the second arithmetic unit begins reading the dequantized data blocks from shared memory to perform matrix multiplication, while the first arithmetic unit continues processing the next set of data to be dequantized. Since the bit-width expansion conversion time of the dequantization operation differs from the computation time of matrix multiplication, dynamically adjusting the data block size can cause partial overlap in the effective working time of the two arithmetic units. For example, when the first arithmetic unit completes the dequantization of 8 data blocks, the second arithmetic unit may have just finished processing the multiplication of the first 6 data blocks, thus forming two parallel execution intervals on the timeline.

[0132] refer to Figure 9 , Figure 9 This is a schematic diagram of the data pipeline for the matrix operation method of model data provided in the embodiments of this application; in some embodiments, the matrix operation method of model data further includes synchronizing the parallel execution of the first operation unit and the second operation unit through a hardware event queue, so that the second operation unit is triggered to start the scheduled matrix multiplication after the first operation unit completes each set of dequantized data, wherein the operation time of the first operation unit and the second operation unit partially overlaps.

[0133] like Figure 9 As shown, Figure 9This paper illustrates the parallel computing flow of the AIV and AIC of the Ascend910B chip in this application, as well as their synchronization mechanism. The AIV is responsible for dequantizing fp8 data into fp16 data. The diagram shows multiple modules for dequantizing fp8 to fp16, indicating that the AIV can process multiple data dequantization tasks simultaneously, achieving parallel computing and improving data processing efficiency. The AIC is responsible for performing matrix multiplication operations. The diagram shows multiple matrix multiplication modules (matrix multiplication 1, matrix multiplication 2, etc.) in the AIC pipeline, meaning that the AIC can process multiple matrix multiplication tasks simultaneously, also achieving parallel computing and enabling rapid completion of large-scale matrix operations.

[0134] Furthermore, the dashed lines represent the synchronization signal. This synchronization signal, indicated by dashed lines in the diagram, is a crucial mechanism for coordination between the AIV and AIC. Each time the AIV completes a task of dequantizing fp8 data into fp16, it notifies the AIC via the synchronization signal. After completing a dequantization task, the AIV transmits the processed fp16 data to the AIC. Upon receiving the synchronization signal and the data, the AIC immediately activates the corresponding matrix multiplication module to perform matrix multiplication operations on the received fp16 data. This allows for efficient integration of data processing and matrix computation, avoiding data waiting and idle computation units, thus improving the overall system's computational efficiency.

[0135] The hardware event queue is a communication mechanism used to coordinate task scheduling between different computing units. It sends synchronization signals and can be implemented using a circular buffer within the chip's internal registers. This queue records the task status of computing units in real time and triggers cross-unit event responses. Parallel execution of computing units refers to the overlap in time between dequantization and matrix multiplication operations. This can be achieved through a pipelined task scheduling strategy, which allows intermediate results from preceding computing units to be immediately retrieved by subsequent units. Triggered matrix multiplication scheduling means automatically activating matrix operation instructions when dequantized data reaches a preset threshold. This can be implemented through interrupt signals or a memory address polling mechanism, which eliminates idle waiting time between computing units.

[0136] In summary, through the coordination of synchronization signals, the data flow between AIV and AIC is efficient and orderly. Once AIV completes data dequantization, it immediately notifies AIC and transmits the data. AIC then begins matrix multiplication operations. The entire process is seamless, fully leveraging the parallel computing capabilities and hardware resource utilization efficiency of the Ascend910B chip.

[0137] According to another embodiment, this application also provides an inverse quantization device for model data, which can be set in... Figure 1The AIV unit in the architecture shown, a dequantization device for model data, includes: a data determination module for determining the first floating-point data to be dequantized in the model; a first conversion module for treating the first floating-point data as first integer data with the same bit width, performing a first type conversion process from the first bit width to the second bit width to generate intermediate floating-point data; a second conversion module for performing a second type conversion process on the intermediate floating-point data while maintaining the second bit width to generate second integer data; and a data correction module for performing format correction processing on the second integer data to generate the dequantized second floating-point data; wherein the first floating-point data is stored in a first storage area; the intermediate floating-point data, the second integer data, and the second floating-point data time-division multiplex the same second storage area.

[0138] The dequantization device for model data executes four stages sequentially during operation: data identification, bit width expansion, format conversion, and binary correction. In the data identification stage, the data determination module identifies the quantized weight data stored in FP8 format by scanning the operator input features of the model computation graph. In the bit width expansion stage, the first conversion module takes the FP8 floating-point number as an int8 integer input and expands it to a 16-bit floating-point number using the cast instruction built into the Ascend910B chip. In the format conversion stage, the second conversion module performs fixed-point conversion on the 16-bit floating-point data, generating a 16-bit integer containing the original information of the sign bit, exponent bit, and mantissa bit. In the binary correction stage, the data correction module completes the format conversion according to the bit field distribution of the target FP16 format by clearing invalid data in the high 8 bits, correcting the exponent offset, and adjusting the mantissa alignment.

[0139] In some specific implementations, the data determination module can automatically mark quantization nodes using a computation graph traversal algorithm, for example, by identifying quantized data through detecting the scale factor parameter in the operator attributes. The first conversion module can call the vconv instruction in the chip instruction set to achieve lossless expansion from 8 bits to 16 bits, while the second conversion module uses the vf2iz instruction to complete the floating-point to fixed-point conversion operation. The bitmask clearing operation of the data correction module can be implemented using bitwise AND operations, for example, performing a logical AND operation between 16-bit data and 0x00FF to clear the high 8 bits of redundant data.

[0140] In some embodiments, the inverse quantization device for model data is applied to or includes the AIV in the above embodiments.

[0141] According to another embodiment, this application also provides a chip that can have Figure 1The system architecture includes a chip comprising: a first arithmetic unit for executing the dequantization method of the model data of any one of the first aspects to obtain the dequantized second floating-point data; and a second arithmetic unit for performing matrix multiplication based on the second floating-point data to obtain the matrix operation result.

[0142] In some embodiments, the chip includes an Ascend910B chip, the first arithmetic unit includes an AIV, and the second arithmetic unit includes an AIC. The chip improves computational efficiency through pipelined collaboration between hardware units. After receiving an 8-bit floating-point input, the first arithmetic unit first treats it as an 8-bit integer and performs a 16-bit extension to generate intermediate floating-point data. Then, maintaining the 16-bit width, it performs a secondary type conversion, clears redundant high 8 bits using a bitmask, recalculates the exponent offset, and adjusts the mantissa alignment to generate 16-bit floating-point data conforming to the IEEE 754 standard. The second arithmetic unit receives the processed normalized data and calls matrix multiplication instructions to complete tensor calculations. The two arithmetic units overlap their computation processes through a hardware event queue; for example, the second unit is triggered to start its computation task after the first unit completes partial data dequantization.

[0143] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0145] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the methods in the foregoing method embodiments.

[0146] And an electronic device, comprising:

[0147] One or more processors; and

[0148] A memory associated with one or more processors, the memory being used to store program instructions that, when read and executed by one or more processors, perform the steps of any of the methods in the foregoing method embodiments.

[0149] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the methods in the foregoing method embodiments.

[0150] in, Figure 10 An exemplary architecture of an electronic device is shown, which may include a processor 1010, a video display adapter 1011, a disk drive 1012, an input / output interface 1013, a network interface 1014, and a memory 1020. The processor 1010, video display adapter 1011, disk drive 1012, input / output interface 1013, network interface 1014, and memory 1020 can communicate with each other via a communication bus 1030.

[0151] The processor 1010 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.

[0152] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system 1021 for controlling the operation of the electronic device 1000, and the basic input / output system (BIOS) 1022 for controlling the low-level operations of the electronic device 1000. Additionally, it can store a web browser 1023, a data storage management system 1024, and a model data inversion device 1025, etc. The aforementioned model data inversion device 1025 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0153] Input / output interface 1013 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0154] The network interface 1014 is used to connect the communication module (not shown in the figure) to enable communication and interaction between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0155] Bus 1030 includes a pathway for transmitting information between various components of the device (e.g., processor 1010, video display adapter 1011, disk drive 1012, input / output interface 1013, network interface 1014, and memory 1020).

[0156] It should be noted that although the above-described device only shows the processor 1010, video display adapter 1011, disk drive 1012, input / output interface 1013, network interface 1014, memory 1020, bus 1030, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0157] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of the embodiments of this application.

[0158] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for inverse quantization of model data, wherein, The inverse quantization method includes: Determine the first floating-point data to be dequantized in the model; The first floating-point data is treated as a first integer data with the same bit width, and a first type conversion process is performed to expand it from the first bit width to the second bit width to generate intermediate floating-point data; The intermediate floating-point data is subjected to a second type conversion process that maintains the second bit width to generate second integer data; Obtain the format layout information of the sign bit, exponent bit, and mantissa bit in the first floating-point data; Based on the format layout information, redundant data is cleared by bitmasking, exponent bias and single shift alignment are corrected, the data layout is adjusted to the target floating-point format, and the second floating-point data after dequantization is generated. The first floating-point data is stored in the first storage area; the intermediate floating-point data, the second integer data, and the second floating-point data are time-division multiplexed in the same second storage area.

2. The method for inverse quantization of model data according to claim 1, characterized in that, The process of clearing redundant data using a bitmask, correcting exponent bias and single-shift alignment, adjusting the data layout to the target floating-point format based on the format layout information, and generating the dequantized second floating-point data includes: Based on a preset bitmask, invalid data generated by bit width expansion in the high 8 bits of the second integer data is removed; Based on the original offset value of the exponent bit in the format layout information and the exponent offset of the target floating-point format, a binary correction value is calculated, and the second integer data is superimposed and corrected based on the binary correction value to generate offset correction data. Based on the sign bit, exponent bit, and mantissa bit in the format layout information, the offset correction data is shifted with the aim of aligning with the target floating-point format to generate alignment correction data. The alignment correction data is multiplied by a preset scaling factor to generate the dequantized second floating-point data.

3. The method for inverse quantization of model data according to claim 1 or 2, characterized in that, The dequantization method is applied to an AI acceleration chip that includes a vector calculation unit and a matrix calculation unit. The type conversion process is implemented based on the bit width extension instruction built into the AI ​​acceleration chip, where the first bit width is 8 bits and the second bit width is 16 bits.

4. A matrix operation method for model data, characterized in that, The matrix operation methods include: In the first arithmetic unit, the dequantization method of the model data according to any one of claims 1 to 3 is executed to obtain the dequantized second floating-point data; In the second arithmetic unit, matrix multiplication is performed based on the second floating-point data to obtain the matrix operation result.

5. The matrix operation method for model data according to claim 4, characterized in that, The matrix operation method also includes: The first and second arithmetic units are synchronized to execute in parallel through a hardware event queue, so that the second arithmetic unit is triggered to start scheduling matrix multiplication after the first arithmetic unit completes each set of dequantized data; wherein the operation time of the first and second arithmetic units partially overlaps.

6. A device for inverse quantization of model data, characterized in that, include: The data determination module is used to determine the first floating-point data to be dequantized in the model. The first conversion module is used to treat the first floating-point data as a first integer data with the same bit width, and perform a first type conversion process to expand from the first bit width to the second bit width to generate intermediate floating-point data; The second conversion module is used to perform a second type conversion process on the intermediate floating-point data while maintaining the second bit width, to generate second integer data. The data correction module is used to obtain the format layout information of the sign bit, exponent bit and mantissa bit in the first floating-point data; based on the format layout information, redundant data is cleared by bitmask, exponent bias and single shift alignment are corrected, the data layout is adjusted to the target floating-point format, and the dequantized second floating-point data is generated. The first floating-point data is stored in the first storage area; The intermediate floating-point data, the second integer data, and the second floating-point data are time-division multiplexed in the same second storage area.

7. A chip, characterized in that, include: The first arithmetic unit is used to execute the dequantization method of the model data according to any one of claims 1 to 3 to obtain the dequantized second floating-point data; The second arithmetic unit is used to perform matrix multiplication operations based on the second floating-point data to obtain the matrix operation result.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the inverse quantization method for model data as described in any one of claims 1 to 3, and the matrix operation method for model data as described in any one of claims 4 to 5.

9. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, perform the inverse quantization method of model data as described in any one of claims 1 to 3, and the matrix operation method of model data as described in any one of claims 4 to 5.

Images