Data quantification method and apparatus, and related product

Abstract

The present disclosure relates to a data quantification method and apparatus, and a related product. The apparatus in the present disclosure is comprised in an integrated-circuit apparatus, wherein the integrated circuit is integrated with one or more combined processing apparatuses, and the combined processing apparatus is an artificial-intelligence arithmetic unit, which is used for supporting various deep learning and machine learning algorithms, so as to meet intelligent processing requirements in complex scenarios in fields such as computer vision, voice, natural language processing and data mining.

Description

Method, apparatus and related product for data quantization Cross-reference to related applications

[0001] The present application claims priority to the Chinese patent application No. 202410813840.3, entitled “Method, apparatus and related product for data quantization” and filed on June 21, 2024, and the Chinese patent application No. 202411346974.5, entitled “Method, apparatus and related product for data quantization” and filed on September 25, 2024. TECHNICAL FIELD

[0002] The present disclosure generally relates to the technical field of data processing. More specifically, the present disclosure relates to a method, apparatus, integrated circuit device, board card, computing device and computer readable storage medium for data quantization. BACKGROUND

[0003] With the rapid development of deep learning technology, the size and complexity of models are increasing, which leads to a sharp rise in the demand for computing resources when the models are deployed and run. As one of the key methods to solve this problem, quantization technology can effectively reduce the amount of computation and save computing resources by reducing the data precision of the data in the model.

[0004] However, the existing quantization technology has the problem of sharp decline in task accuracy when processing original data into low-precision data types, especially ultra-low-precision data types. In view of this, there is an urgent need to provide a data quantization scheme to ensure task accuracy while quantizing data. SUMMARY

[0005] To at least solve one or more technical problems as mentioned above, the present disclosure proposes, in various aspects, a method, apparatus, integrated circuit device, board card, computing device and computer readable storage medium for data quantization.

[0006] In a first aspect, the present disclosure provides a method for data quantization, comprising: determining an absolute value maximum in a group of data to be quantized; obtaining a maximum numerical value that can be represented by a target data type after quantization of the data to be quantized; determining a scaling factor corresponding to the group of data to be quantized based on a ratio between the absolute value maximum and the maximum numerical value; converting the scaling factor from a first data type to a second data type to obtain a scaling coefficient, wherein the bit width of the second data type is smaller than the bit width of the first data type; and performing quantization processing on the group of data to be quantized using the scaling coefficient to obtain quantized data having the target data type.

[0007] In a second aspect, this disclosure provides a data quantization apparatus, comprising: a first processing circuit configured to: determine the maximum absolute value in a set of data to be quantized; obtain the maximum value that a target data type can represent after quantization of the data to be quantized; determine a scaling factor corresponding to the set of data to be quantized based on the ratio between the maximum absolute value and the maximum value; convert the scaling factor from a first data type to a second data type to output a scaling coefficient, wherein the bit width of the second data type is smaller than the bit width of the first data type; and a second processing circuit configured to quantize the set of data to be quantized using the scaling coefficient to output quantized data having the target data type.

[0008] In a third aspect, this disclosure provides an integrated circuit device, including the device described in the second aspect of this disclosure.

[0009] In a fourth aspect, this disclosure provides a board that includes the integrated circuit device described in the third aspect of this disclosure.

[0010] In a fifth aspect, this disclosure provides a data quantization computing apparatus, comprising: at least one processor; and at least one memory in communication with said at least one processor, having stored computer-readable instructions that, when loaded and executed by said at least one processor, cause said at least one processor to perform the method described in the first aspect of this disclosure.

[0011] In a sixth aspect, this disclosure provides a computer-readable storage medium storing program instructions that, when loaded and executed by a processor, cause the processor to perform the method described in the first aspect of this disclosure.

[0012] The data quantization scheme provided above, in this embodiment, determines the scaling factor based on the ratio between the maximum absolute value of the data to be quantized and the maximum value that the target data type can represent. This helps ensure the accuracy of the maximum absolute value. Furthermore, since the larger the value of the data to be quantized during quantization, the greater the loss of accuracy and the greater the impact on the task calculation results, ensuring the accuracy of the maximum absolute value helps reduce the loss of task accuracy. Therefore, compared to methods like MX quantization that calculate scaling factors, the method in this embodiment helps ensure task accuracy and processing performance. In addition, compared to quantization methods where the scaling factor uses the same data type as the original data type, this embodiment also obtains a scaling factor with a lower bit width by converting the data type. This helps reduce the storage space required to store the scaling factor and the hardware overhead required to execute the task while ensuring task accuracy. In some embodiments, the scaling factor can also be compressed to reduce the waste of storage space and bandwidth. Attached Figure Description

[0013] The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0014] Figure 1 shows a schematic diagram of the MX data format;

[0015] Figure 2 shows an exemplary flowchart of a data quantization method according to some embodiments of this disclosure;

[0016] Figure 3 illustrates an exemplary flowchart of a data quantization method according to other embodiments of this disclosure;

[0017] Figure 4 shows a schematic block diagram of a data quantization apparatus according to some embodiments of this disclosure;

[0018] Figure 5 shows a schematic block diagram of a data quantization apparatus according to other embodiments of this disclosure;

[0019] Figure 6 shows a schematic block diagram of the comparison circuit of some embodiments of this disclosure;

[0020] Figure 7 shows a schematic block diagram of a data quantization apparatus according to some of the embodiments of this disclosure;

[0021] Figure 8 shows a schematic diagram of the structure of a board according to an embodiment of this disclosure;

[0022] Figure 9 is a structural diagram showing the combined processing device in the chip of the embodiment shown in Figure 8. Detailed Implementation

[0023] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0024] It should be understood that the terms “comprising” and “including” used in this disclosure and claims indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0025] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0026] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0027] First, we provide explanations of the technical terms that may be used in this disclosure.

[0028] Floating-point numbers: The IEEE floating-point standard uses V = (-1). sign *mantissa*2 E A number is represented in the form of . Here, `sign` is the sign bit, where 0 represents a positive number and 1 represents a negative number; `E` represents the exponent (or index), which is a weighted sum of 2 raised to the power of `E` (which may be a negative power); `mantissa` represents the mantissa, a binary fraction. Floating-point numbers are represented in computers using three fields, each encoded separately:

[0029] (1) A single sign bit directly encodes the symbol s;

[0030] (2) The exponent field of the k-bit is encoded as exp=e(k-1)......e(1)e(0);

[0031] (3) The n-digit mantissa is used to encode the mantissa. However, the encoding result depends on whether the exponent field is all 0.

[0032] Fixed-point numbers are a way of representing numbers in computers, characterized by a fixed decimal point. Fixed-point numbers can represent integers or decimals, depending on the implicit position of the decimal point. A fixed-point number consists of three parts: a shared exponent, a sign bit, and a mantissa. The shared exponent means that the exponent is shared within a set of real numbers to be quantized; the sign bit indicates whether the fixed-point number is positive or negative; and the mantissa determines the number of significant digits, i.e., the precision.

[0033] Bit width: How many bits are used to represent data.

[0034] Quantization: The process of converting high-precision numbers into lower-precision data that takes up less memory.

[0035] Dequantization: The process of converting low-precision numbers into higher-precision data.

[0036] FP*: Represents a floating-point data type, where * can be an integer such as 2, 4, 6, 8, 16, 32, 64, etc. The value of * indicates the bit width of the data represented by this data type.

[0037] BF16 and TF32 are data types suitable for deep learning, which improve computational performance by sacrificing precision and range.

[0038] int*: Represents an integer data type, where * can be an integer such as 2, 4, 6, 8, 16, 32, 64, etc. The value of * indicates the bit width of the data represented by this data type.

[0039] ExMy: A data format where E represents the exponent, M represents the mantissa, x represents the exponent width, and y represents the mantissa width.

[0040] MX: refers to the Microscaling specification (see “OCP Microscaling Formats (MX) Specification Version 1.0”).

[0041] Extended MX: refers to a quantization scheme proposed in the embodiments disclosed herein.

[0042] It should be noted that in this disclosed embodiment, the data to be quantized or dequantized can be various data in the neural network, including but not limited to input neurons, weights, output neurons, gradients, etc. The neural network provided in this disclosed embodiment can be applied to various fields and can process various types of data. For example, in image recognition and processing, facial recognition, object detection, image classification, and medical image analysis can include data types such as pixel data and image features; in natural language processing (NLP), language translation, sentiment analysis, text summarization, and speech recognition can include data types such as text data, word embeddings, and sentence structure; in speech recognition and processing, intelligent assistants, automatic caption generation, and speech-to-text conversion can include data types such as audio signals and spectrograms; in recommendation systems, personalized content recommendation, product recommendation, and advertising can include data types such as user behavior data, item features, and rating data; in healthcare, disease diagnosis, drug discovery, and gene sequence analysis can include data types such as medical records, biomarker data, and genomic data; in the financial field, risk assessment, fraud detection, and stock market prediction can include data types such as transaction data, user credit scores, and market data; and in autonomous driving, vehicle perception, decision-making, and path planning can include data types such as sensor data, environmental features, and traffic signals. In gaming and entertainment, AI applications such as game AI, virtual reality, and animation generation can include data types such as game state data and user interaction data. In scientific research, applications such as physics simulation, chemical compound prediction, and astrophysical analysis can include data types such as experimental data, simulation results, and observational data. Generative AI applications, on the other hand, utilize complex algorithms, models, and rules to learn from large-scale datasets to create new, original content. This includes, but is not limited to, creating various types of content such as text, images, audio, video, and code. Correspondingly, the input neuron data processed can be unquantized or quantized image data, audio data, video data, speech data, text data, document data, etc., and the corresponding output neuron data can also be unquantized or quantized image data, audio data, video data, speech data, text data, document data, etc. The output of the neural network can include a probability score of an image belonging to a specific object category, a probability score of a document relating to a specific topic, a probability score of the correct translation of a text fragment in the target language from a text fragment in the source language, or a probability score of the correct transcription of a text fragment from spoken language, etc. These output data can also be quantized or unquantized. By quantizing / dequantizing the data, the processing efficiency of these tasks can be maintained while reducing the computational load and saving computing resources.The data quantization scheme of this disclosure embodiment can also be used in the inference and training of neural networks. By performing the quantization process described in this disclosure embodiment on the data of the neural network, the accuracy of the task can be guaranteed when the neural network performs one or more of the aforementioned processing tasks, while improving the speed and performance of the neural network model's inference and training.

[0043] Exemplary MX Quantization Scheme

[0044] Figure 1 illustrates a schematic diagram of the MX data format. As shown in Figure 1, the MX data format mainly consists of three parts: the scaling factor (X, shared scale), the scaling block size K (K, Scaling Block size), and the private elements within the block (Pi).

[0045] The scaling factor X relates to the data type or encoding of the scaling factor. The private element (Pi) also relates to its data type or encoding. The scaling block size (K) represents the number of private elements Pi within the block. All K elements Pi within the block have the same data type and therefore the same bit width. The scaling factor X is shared among these K elements. The data type of the elements and the data type of the scaling factor can be chosen independently. In a sense, MX can be viewed as a mechanism for constructing vector data types based on scalar data types. The values ​​V1,…,V represented in an MX block are... K It can be determined as follows: V i =XPi.

[0046] As further illustrated in Figure 1, W represents the number of bits used to encode the shared scaling factor X, i.e., the bit width of scaling factor X; d represents the number of bits used to represent each element Pi, i.e., the bit width of element Pi. Therefore, an MX block containing K elements can be encoded using (W+Kd) bits. Table 1 shows some common MX formats and related parameters. Table 1

[0047] In Table 1 above, E x M y This represents a scalar format notation with x exponent bits and y mantissa bits. For example, FP8(E4M3) represents an FP8 format with 1 sign bit, 4 exponent bits, and 3 mantissa bits. When y is 0 (e.g., E8M0), the format does not include the sign bit.

[0048] Furthermore, in MX, the scaling factor X is calculated as follows:

[0049] (1) Determine the largest power of the absolute value of the K elements Pi that are less than or equal to the K elements Pi;

[0050] (2) Divide the maximum power of 2 by the power of 2 of the maximum value that the data type of element Pi can represent, and set the result as the scaling factor ScaleX.

[0051] The inventors have discovered that, for MX quantization, processing the original data into ultra-low precision data types such as F4 (representing a 4-bit data format) leads to a sharp decrease in the accuracy of large models, significantly impacting the accuracy of the computational results. Taking image recognition tasks as an example, task accuracy or processing performance is reflected in the accuracy of the image recognition results, i.e., whether the image recognition computation is accurate. The quantized image data obtained after quantization using the aforementioned quantization method affects the accuracy of the image recognition results.

[0052] The inventors also discovered that if the data type of the scaling factor used in the quantization method is the same as the original data type, although the above-mentioned problem of decreased accuracy can be solved to some extent, this method will lead to greater resource overhead (such as storage space, area overhead required for hardware support, etc.).

[0053] Based on the above findings, this disclosure provides a novel data quantization scheme. This scheme determines a scaling factor based on the maximum absolute value of the data to be quantized, and obtains a scaling coefficient by converting the scaling factor to a different data type. Since the converted scaling coefficient has a non-zero mantissa, the accuracy of the quantized data is improved, resulting in processing tasks performed on the quantized data according to the embodiments of this disclosure exhibiting better accuracy. Furthermore, setting the quantization granularity based on the hardware's single-processing capacity facilitates the quantization of a block of granularity in a single hardware operation. This makes online quantization / dequantization more convenient, facilitating pipelined design and enabling neural network models using this hardware as a computing platform to perform quantization operations at various stages, including inference and training, thereby improving the computational efficiency of the neural network model.

[0054] Exemplary methods for extending MX quantization

[0055] The specific embodiments disclosed herein will now be described in detail with reference to the accompanying drawings.

[0056] Figure 2 illustrates an exemplary flowchart of a data quantization method according to some embodiments of this disclosure. As shown in Figure 2, method 200 may include: in step 201, determining the maximum absolute value among a set of data to be quantized. A set of data to be quantized may include one or more data items. In some embodiments, when a set of data to be quantized includes one data item, the absolute value of that single data item is the maximum absolute value of the set of data to be quantized. In other embodiments, when a set of data to be quantized includes multiple data items, the maximum absolute value among the set of data to be quantized can be determined by comparing the absolute values ​​of the multiple data items.

[0057] Next, in step 202, the maximum value that the target data type can represent after quantizing the data to be quantized can be obtained. The target data type is the data type of the quantized data obtained after quantizing the data to be quantized. Before quantization, the target data type can be set as needed, and its precision is generally lower than that of the original data type of the data to be quantized. In some embodiments, the original data type of the data to be quantized may include at least one of FP16, BF16, FP32, FP64, TF32, etc.; the target data type may include at least one of FP8, FP6, FP4, FP2, int8, int4, int2, int6, FP12, etc. Among them, floating-point data types such as FP8, FP6, and FP4 can support multiple data formats. Taking FP4 as an example, it can be divided in multiple ways according to the rule of "sign bit + exponent bit + mantissa bit". Usually, the sign bit occupies 1 bit, and the common combination of exponent bit and mantissa bit is E3M0 and E2M1. Similarly, FP8 can support data formats such as E4M3 and E5M2, and FP6 can support data formats such as E3M2 and E2M3.

[0058] Due to the bit width limitations of each data type, each data type has a maximum representable value. For example, the maximum representable value when the target data type is FP4 is 6; the maximum representable value when the target data type is int8 is 127. Typically, there is a one-to-one correspondence between the maximum value and the target data type, which can be obtained through methods such as table lookup.

[0059] Then, in step 203, the scaling factor corresponding to the aforementioned set of data to be quantized can be determined based on the ratio between the maximum absolute value determined in step 201 and the maximum value determined in step 202. The data type of the scaling factor (i.e., the first data type) is the same as the original data type of its corresponding data to be quantized, or the bit width of the first data type can be greater than the bit width of the original data type. Typically, the correspondence between the scaling factor and the data to be quantized is determined by the correspondence between the maximum absolute value and the data to be quantized. For example, in some embodiments, the maximum absolute value is determined by one of the aforementioned data to be quantized, and the resulting scaling factor is the one corresponding to that data to be quantized. In other embodiments, the maximum absolute value is determined from multiple data to be quantized in the aforementioned set, and the resulting scaling factor is the multiple data to be quantized.

[0060] Further, the process can proceed to step 204, where the scaling factor can be converted from a first data type to a second data type to obtain a scaling factor. The bit width of the second data type can be smaller than that of the first data type. The first data type is the data type of the scaling factor, and the second data type is the data type of the scaling factor. Data type conversion makes the bit width of the second data type smaller than that of the first data type, thus the precision of the scaling factor is less than the precision of the scaling factor. In some embodiments, the conversion of the scaling factor to the scaling factor can be implemented using a rounding instruction, such as `cvt_to_scale_dtype()`. This rounding instruction can be a hardware or software instruction at various levels, such as processor internal microinstructions, ISA instructions, or function instructions implemented in high-level languages ​​at various software levels. In other embodiments, the conversion of the scaling factor to the scaling factor supports multiple rounding modes, such as `rn(round to nearest, ties to even)`.

[0061] In some other embodiments, the second data type can be represented as ExMy, where E represents the exponent bits, M represents the mantissa bits, x represents the bit width of the exponent bits, y represents the bit width of the mantissa bits, and x + y < the bit width of the original data type. For example, assuming the original data type is FP16, then x + y < 16. The specific values of x and y can be set according to requirements or the hardware environment. x is a positive integer and y ≥ 0. In some other embodiments, 3 ≤ x + y < the bit width of the original data type. In some embodiments, 8 ≤ x + y < the bit width of the original data type. In some embodiments, the second data type can include at least one of, for example, E8M0, E8M1, E8M2, E8M3, E8M4, E8M5, E8M6, E8M7, E5M0, E5M1, E5M2, E5M3, E5M4, E5M5, E5M6, E3M0, E3M1, E3M2, E3M3, E3M4, E3M5, E3M6, E3M7, E3M8, E3M9, etc. In some preferred embodiments, y > 0. In further preferred embodiments, the second data type can be E8M2 or E8M3.

[0062] For ease of understanding, still taking the original data type as FP16 as an example, when 8 < x + y < the bit width of the original data type, the bit width (i.e., x + y) represented by the second data type ExMy of the scaling factor can be between 8 bits and 16 bits. Using such a scaling factor to perform quantization processing on the operation data of the neural network and using the quantized data obtained after quantization for inference and training of the neural network, etc., the accuracy of the neural network obtained is between the 8-bit representation (such as E8M0 represented in MX) and the original 16-bit representation (such as E5M10, E8M7, i.e., FP16, BF16). Compared with the MX format, such a scaling factor disclosed in the present disclosure is beneficial to improving the accuracy of the quantized data; compared with the case where the data type of the scaling factor is the same as the original data type, such a scaling factor disclosed in the present disclosure is beneficial to reducing hardware resource overhead or storage overhead, etc.

[0063] Furthermore, since the bit width of the mantissa bits affects the quantization accuracy, a larger bit width of the mantissa bits can provide a scaling factor with higher accuracy, thereby retaining more details of the original data during the quantization process. Therefore, compared with the scaling factor in the E8M0 format in MX quantization, when the bit width y of the mantissa bits of the scaling factor in the embodiments disclosed in the present disclosure is greater than 0, it is beneficial to improve the accuracy of the quantized data and the accuracy of the processing task.

[0064] Furthermore, this disclosed embodiment enables the scaling factor to support multiple data types by performing data type conversion, thereby allowing for more flexible adaptation to different data characteristics and precision requirements. Regarding the selection of the exponent and mantissa bit widths of the scaling factor, the distribution characteristics of the data to be quantized can be considered. A larger exponent bit width allows for the representation of more values. If the range of values ​​in a set of data to be quantized varies significantly, the exponent bit width of the scaling factor can be set larger; conversely, if the range of values ​​in a set of data to be quantized varies only slightly, the exponent bit width of the scaling factor can be set smaller, while the mantissa bit width can be set larger.

[0065] For example, suppose the second data type needs to be set to 8 bits. If the absolute value range of a set of data to be quantized involves 2... -127 ~2 127 Then the second data type can be set to E8M0; if the absolute value range of a set of data to be quantized involves 2 -4 ~2 4 Then the second data type can be set to E3M5. Of course, it is understandable that the bit width of the second data type is not limited to 8 bits, and can also be set to 5 bits, 4 bits, etc., as needed.

[0066] After determining the scaling factor through steps 201-204 above, the scaling factor can be used to quantize the corresponding data to be quantized, or to dequantize the quantized data. As further shown in Figure 2, method 200 may further include step 205, which uses the scaling factor to quantize a set of data to be quantized to obtain quantized data with the target data type. When using the scaling factor to quantize multiple data to be quantized in a set, each data to be quantized is quantized using the scaling factor to obtain multiple corresponding quantized data. In some embodiments, the quantization process may include: dividing the data to be quantized by the scaling factor to obtain an intermediate quantization result; and performing data type conversion on the intermediate quantization result to obtain quantized data with the target data type. The data type conversion here can also be implemented using a conversion instruction, such as cvt_to_scale_dtype(). The conversion instruction here can be hardware or software instructions at various levels, such as processor internal microinstructions, ISA instructions, or function instructions implemented in high-level languages ​​at various software levels.

[0067] In other embodiments, method 200 may further include: performing dequantization processing on the quantized data using a scaling factor. Dequantization processing is generally the process of restoring low-precision data to high-precision data. In some embodiments, performing dequantization processing on the quantized data using a scaling factor may include: multiplying the quantized data with the scaling factor. In other embodiments, the quantized data may be converted from a target data type to a desired data type before being multiplied with the scaling factor, wherein the precision of the desired data type is higher than the precision of the target data type. In still other embodiments, dequantization processing may restore the quantized data to the data to be quantized before the quantization operation.

[0068] As can be seen from the second data type of the scaling factor obtained from the previous conversion, the bit width of a single scaling factor is 3 to 15 bits. For example, E8M4 requires 12 bits, which typically requires 2 bytes (2B) of storage space, so it will definitely not be fully utilized. This inevitably leads to a waste of storage space and transmission bandwidth. To solve this problem, in some embodiments, two or more scaling factors can be compressed, thereby saving storage space and bandwidth.

[0069] In some embodiments, after quantizing two or more sets of data to be quantized using method 200 to obtain two or more scaling factors, these two or more scaling factors can be compressed. Then, the compressed scaling factors are used to quantize the corresponding sets of data to be quantized to output quantized data with the target data type.

[0070] Specifically, a compression process (not shown in the figure) can be included after step 204, which includes: determining the shared exponent of the Z scaling factors to be compressed, where Z ≥ 2; determining the Z new mantissas corresponding to these Z scaling factors; and bundling the shared exponent with the Z new mantissas for output for association. That is, after compression, the exponents of the Z scaling factors are reused, and their mantissas are different, thereby saving at least Z-1 exponent bits.

[0071] In some implementations, determining the shared exponent for the Z scaling factors to be compressed may involve setting the largest exponent among these Z scaling factors as the shared exponent. Choosing the largest exponent as the shared exponent ensures the precision of larger numbers as much as possible, since larger numbers typically have a greater impact on the calculation results.

[0072] In some implementations, determining the Z new mantissas corresponding to the Z scaling factors can include: setting the new mantissa of scaling factors whose exponent is less than the maximum exponent to 0; and setting the new mantissa of scaling factors whose exponent is equal to the maximum exponent to the original mantissa. It can be understood that for scaling factors whose exponent is equal to the maximum exponent, since the exponent remains unchanged, its mantissa remains unchanged; however, for scaling factors whose exponent is less than the maximum exponent, after the exponent is scaled up to the maximum exponent, to maintain equivalence, the mantissa would be proportionally reduced, exceeding the range of values ​​that the mantissa can represent. Therefore, its new mantissa is set to 0 here. This setting will result in some precision loss, but compared to the storage and bandwidth gains, this precision loss is acceptable in some cases.

[0073] In some implementations, the shared exponent and the Z new mantissas can be bundled together in sequence for association. For example, first the shared exponent, then the new mantissa of the first scaling factor, the new mantissa of the second scaling factor, and so on up to the new mantissa of the Z scaling factor.

[0074] Subsequently, in step 205, the compressed scaling factors are used to quantize the corresponding groups of data to be quantized, so as to output quantized data with the target data type. Since the exponent and mantissa of the scaling factors with an exponent smaller than the maximum exponent will change after compression, in order to maintain the consistency of quantization and dequantization, the compressed scaling factors are used to quantize the corresponding groups of data to be quantized.

[0075] In some implementations, the second data type of the Z scaling factors to be compressed is the same. For example, all Z scaling factors are of type E8M4. In this way, the shared exponent bit width after compression is 8 bits, and the Z new mantissas are 4Z bits.

[0076] In some implementations, the Z scaling factors belong to Z adjacent groups of data to be quantized. This allows the data to be processed sequentially without requiring additional information to indicate the scaling factors that are compressed together.

[0077] In some implementations, Z can be 2, 4, 8, or other values. Those skilled in the art can choose an appropriate Z value based on the number K of data to be quantized in a set of data. For example, when K is small, the value of Z can be larger; when K is large, the value of Z needs to be smaller, thereby ensuring that the benefit of using the compression scheme is greater than the benefit of amplifying K by Z times.

[0078] Similarly, when dequantizing extended MX quantized data using a compressed format, the scaling factors need to be decompressed first. Specifically, during dequantization, in response to the scaling factors being in a compressed format, the compressed scaling factors are decompressed; and the decompressed scaling factors are used to dequantize the quantized data.

[0079] Based on the compression method and bundling approach described above, the bundled Z scaling factors can be decoded into Z independent scaling factors. Specifically, the shared exponent can be combined with each of the Z new mantissas to obtain the Z decompressed scaling factors.

[0080] It is understood that in some embodiments, in addition to calculating and storing the scaling factor, the reciprocal of the scaling factor is also calculated and stored. This facilitates converting the division operation between the data to be quantized and the scaling factor into a multiplication operation between the data to be quantized and the reciprocal of the scaling factor during the quantization process. The reciprocal of the scaling factor has the same data type as the scaling factor. Therefore, the compression processing described above can also be applied to the reciprocal of the scaling factor, which will not be described in detail here.

[0081] The above description, in conjunction with Figure 2, illustrates methods for data quantification according to some embodiments of this disclosure. The following description, in conjunction with Figure 3, further illustrates methods for other embodiments of this disclosure.

[0082] Figure 3 illustrates an exemplary flowchart of a data quantization method according to other embodiments of this disclosure. As shown in Figure 3, method 300 may include: in step 301, dividing multiple data to be quantized into multiple groups and determining the number of data to be quantized in each group. Each group of data to be quantized may include one or more data to be quantized. The more groups there are, the finer the granularity of the quantization process, which is more conducive to ensuring the accuracy of the quantized data.

[0083] In some embodiments, the number of data to be quantized in a set of data to be quantized can be determined based on the single processing volume and / or the number of input ports of the processing circuit performing the quantization process; or the number of data to be quantized in a set of data to be quantized can be determined based on the single operation volume and / or the number of input ports of the arithmetic unit that performs the operation using the quantized data.

[0084] For example, the number of data points to be quantized in a group can be determined based on the single-processing capacity (i.e., the amount of data that can be processed in a single run) of the second processing circuit described in this paper (described in detail below), or based on the single-operation capacity (i.e., the amount of data that can be operated on in a single run) of an arithmetic unit that uses quantized data to perform operations such as multiply-accumulate operations or vector inner product operations, and grouped according to this granularity. With this setup, a group of data points to be quantized sharing a single scaling factor can be processed in one operation cycle, without needing to span multiple operation cycles, thus facilitating hardware implementation and fully utilizing hardware computing power.

[0085] For example, the number of data points to be quantized in a set of data can be determined based on the number of input ports of the processing circuit performing quantization or the number of input ports of the arithmetic unit that uses quantized data for calculation, and grouped accordingly. With this setup, a set of data to be quantized can be input at once using the number of hardware input ports, allowing a set of data sharing a single scaling factor to be processed in a single input, eliminating the need for multiple inputs. This improves hardware implementation, reduces hardware processing latency, and lowers I / O overhead.

[0086] For example, assuming the aforementioned processing circuit or arithmetic unit can process 32 data points in a single operation, and its corresponding number of input ports is also 32, then the data points to be quantized can be grouped into sets of 32. If the aforementioned processing circuit or arithmetic unit can process 64 data points in a single operation, and its corresponding number of input ports is less than 64, for example, 32, then the data points to be quantized can be grouped according to the number of input ports, ensuring that the number of data points to be quantized in each group is equal to the number of input ports. In this case, 32 data points are input at once, and the two groups of data can be processed in parallel; alternatively, the 64 data points to be quantized can still be grouped together and input twice to complete the processing of one group of data points at once. If the aforementioned processing circuit or arithmetic unit can process 32 data points in a single operation, and its corresponding number of input ports is more than 32, then the 32 data points to be quantized can be grouped together to achieve one-time input and processing of one group of data points.

[0087] In some embodiments, the data to be quantized can be elements of tensor data. Dividing multiple data to be quantized into multiple groups can include: dividing the multiple data to be quantized into multiple groups along at least one dimension of the tensor data. For example, suppose tensor data A(B, S), where B and S represent the dimensions of the first and second dimensions, respectively. If the division is performed along the second dimension, taking a grouping granularity of 32 as an example (i.e., each group contains 32 data to be quantized), dividing 32 consecutive data to be quantized along dimension S into a group results in (B×S / 32) groups. Taking each group as a data block, the shape of the tensor composed of each data block after grouping (i.e., the dimensions and dimensions of the tensor) can be represented as (B, S / 32), that is, the dimension of the second dimension after grouping is S / 32.

[0088] For example, suppose tensor data A'(B, S, D), where B, S, and D represent the dimensions of the first, second, and third dimensions, respectively. If we divide the data along the third dimension, taking a grouping granularity of 64 as an example, we can divide 64 consecutive data points to be quantized along the third dimension into a group, resulting in (B×S×D / 64) groups. Each group can be considered as a data block. The shape of the tensor composed of these data blocks after grouping can be represented as (B, S, D / 64), which means the size of the third dimension after grouping is D / 64.

[0089] It is understandable that dividing multiple data sets to be quantized into multiple groups is not limited to dividing them only along one dimension of the tensor data; it can also be done along multiple dimensions as needed. The grouping granularity is not limited to the aforementioned 32 or 64; it can be set as needed, for example, 16, 128, etc. The grouping granularity is related to the computing power or input port limitations of the processing circuit / arithmetic unit mentioned earlier, and will not be elaborated upon here.

[0090] Next, in this embodiment, based on the grouping results in step 301, a corresponding scaling factor can be determined for each group of data to be quantized, so that each group of data to be quantized shares a single scaling factor for quantization processing. For ease of understanding, steps 302-306 are illustrated in Figure 3 to explain the specific operation of group quantization.

[0091] As shown in Figure 3, in step 302, the maximum absolute value in each group of data to be quantized can be determined based on each group of data to be quantized. The method for determining the maximum absolute value for each group of data to be quantized is the same as or similar to step 201 described above in conjunction with Figure 2, and will not be repeated here. For the data to be quantized in tensor data, the maximum absolute value obtained by grouping can also be tensor data. The shape of the tensor formed by the maximum absolute value after grouping is related to the grouping granularity and is the same as the shape of the tensor formed by each data block after grouping (e.g., (B, S / 32) or (B, S, D / 64)).

[0092] Then, the process can proceed to step 303, where the maximum value that the target data type can represent can be obtained. The target data types of different groups of data to be quantized can be the same or different, so the maximum values ​​obtained for different groups of data to be quantized can be the same or different.

[0093] Next, in step 304, the scaling factor for each group of data to be quantized can be determined based on the ratio between the maximum absolute value and the maximum value of each group of data to be quantized. At this time, the first data type of the scaling factor is the same as the original data type of the corresponding group of data to be quantized, or the precision of the first data type can be higher than the precision of the original data type (i.e., the bit width of the first data type is greater than the bit width of the original data type). For tensor data, the shape of the tensor formed by the scaling factor obtained after grouping calculation can be the same as the shape of the tensor formed by the aforementioned maximum absolute value.

[0094] In step 305, a corresponding scaling factor is determined based on each scaling factor. Specifically, each scaling factor can be converted from a first data type to a second data type to obtain a scaling factor for each group of data to be quantized. The second data type of the scaling factors for different groups of data to be quantized can be the same or different.

[0095] Furthermore, in step 306, each group of data to be quantized can be quantized separately using a corresponding scaling factor. The scaling factor can be different for different groups of data to be quantized. The quantization process for each group of data to be quantized can be the same as or similar to step 205 described above in conjunction with Figure 2, and will not be repeated here.

[0096] It should be further explained that as the number of groups (i.e., the number of data blocks) increases, the number of scaling factors also increases accordingly. The scaling factors can represent a larger number of data points, which is beneficial for further improving data quantization accuracy. In particular, as the number of scaling factors increases, the sum of the exponent bit width and mantissa bit width increases, thus allowing for the representation of more data points than the E8M0 data format in MX quantization. Consequently, compared to MX quantization, the data accuracy represented by the scaling factors in the extended MX quantization of this disclosed embodiment is higher.

[0097] Therefore, based on the technical solution of obtaining scaling factors through data type conversion and the maximum absolute value, group quantization is used to increase the quantization granularity. A corresponding scaling factor is provided for each group of data to be quantized, which is conducive to further improving the accuracy of the quantized data and the accuracy of the executed task.

[0098] The data quantization method according to embodiments of this disclosure has been described above with reference to Figures 2 and 3. In some embodiments, the data to be quantized described above can be the input data (or neurons) and / or weight data of a neural network. In other words, the method of embodiments of this disclosure can be used to quantize the input data of a neural network, or to quantize the weight data, or to quantize both the input data and the weight data. The input data of the neural network can be data from tasks such as computer vision, speech, natural language processing, and data mining.

[0099] In some preferred embodiments, the data to be quantized described above can be the input data and weight data of the neural network. That is to say, since the method of this disclosure embodiment is beneficial to ensuring task accuracy, both the input data and weight data of the neural network can be quantized, thereby further accelerating the inference and training of large models while maintaining task accuracy.

[0100] Exemplary device for extended MX quantization

[0101] The method of the embodiments disclosed herein has been described above. The apparatus for implementing the data quantization scheme of the embodiments disclosed herein will be described in illustrative terms below with reference to several accompanying drawings.

[0102] Figure 4 shows a schematic block diagram of a data quantization apparatus according to some embodiments of this disclosure. As shown in Figure 4, the apparatus 400 may include a first processing circuit 410 and a second processing circuit 420, wherein the first processing circuit 410 is used to output a scaling factor, and the second processing circuit 420 is used to perform quantization processing. Specifically, the first processing circuit 410 may be used to: determine the maximum absolute value in a set of data to be quantized; obtain the maximum value that the target data type can represent after quantization of the data to be quantized; determine a scaling factor corresponding to a set of data to be quantized based on the ratio between the maximum absolute value and the maximum value; convert the scaling factor from a first data type to a second data type to output a scaling factor, wherein the bit width of the second data type is smaller than the bit width of the first data type. The second processing circuit 420 may be used to quantize a set of data to be quantized using the scaling factor to output quantized data with the target data type.

[0103] In some embodiments, the first processing circuit 410 may be further configured to: determine the number of data to be quantized in a set of data to be quantized based on the single processing volume and / or the number of input ports of the second processing circuit 420; or determine the number of data to be quantized in a set of data to be quantized based on the single operation volume and / or the number of input ports of the arithmetic unit that performs operations using quantized data.

[0104] In some embodiments, the first data type may be the same as the original data type of the data to be quantized, or the bit width of the first data type may be greater than the bit width of the original data type; and the second data type may be represented as E. x M y x + y < bit width of the primitive data type, where E represents the exponent bits, M represents the mantissa bits, x represents the exponent bit width, and y represents the mantissa bit width.

[0105] In other embodiments, the original data type of the data to be quantized may include at least one of, for example, FP16, BF16, FP32, FP64, TF32, etc.; the second data type may include at least one of, for example, E8M0, E8M1, E8M2, E8M3, E8M4, E8M5, E8M6, E8M7, E5M0, E5M1, E5M2, E5M3, E5M4, E5M5, E5M6, E3M0, E3M1, E3M2, E3M3, E3M4, E3M5, E3M6, E3M7, E3M8, E3M9, etc.; the target data type may include at least one of, FP8, FP6, FP4, FP2, int8, int4, int2, int6, FP12, etc. In still other embodiments, y > 0.

[0106] In some other embodiments, the data to be quantized can be the input data and / or weight data of a neural network.

[0107] In some embodiments, the apparatus 400 may further include a compression circuit 430. The compression circuit 430 is configured to compress two or more scaling factors after quantization of two or more sets of data to be quantized. In this case, the second processing circuit 420 is configured to quantize the corresponding sets of data to be quantized using the compressed scaling factors, respectively, to output quantized data with the target data type.

[0108] Correspondingly, when device 400 needs to dequantize extended MX quantized data using a compressed format, the scaling factors must first be decompressed. In these embodiments, device 400 may also include a decompression circuit (not shown). The decompression circuit is configured to decompress the compressed scaling factors in response to the scaling factors being in a compressed format. At this time, the circuit in device 400 that performs the dequantization process (e.g., the second processing circuit 420) can be configured to perform dequantization processing on the quantized data using the decompressed scaling factors.

[0109] The operations performed by the first processing circuit 410, the second processing circuit 420, the compression circuit 430, and the decompression circuit have been described in detail above with reference to Figures 2 and 3, and will not be repeated here. Other implementations of the apparatus according to the embodiments of this disclosure will now be described exemplarily with reference to Figure 5.

[0110] Figure 5 shows a schematic block diagram of a data quantization apparatus according to other embodiments of this disclosure. As shown in Figure 5, the apparatus 500 may include a first processing circuit 410 and a second processing circuit 420, wherein the first processing circuit 410 may include: a comparison circuit 411, which can be used to determine the maximum absolute value in a set of data to be quantized; a first division circuit 412, which can be used to obtain the maximum value that the target data type after quantization of the data to be quantized can represent, and determine a scaling factor corresponding to a set of data to be quantized based on the ratio between the maximum absolute value and the maximum value; and a first rotation circuit 413, which can be used to convert the scaling factor from a first data type to a second data type to output a scaling coefficient. In some embodiments, the first division circuit 412 can obtain the maximum value by means of a lookup table or the like.

[0111] In some embodiments, the second processing circuit 420 may include a second division circuit 421 and a second rotation circuit 422, wherein the second division circuit 421 can be used to perform a division operation on the data to be quantized and the scaling factor, and the second rotation circuit 422 can be used to perform a rotation operation on the result of the division operation (i.e., the intermediate quantization result) to output the result of quantization of the data to be quantized (i.e., the quantized data).

[0112] In other embodiments, the second processing circuit 420 can also be used to perform dequantization processing on the quantized data using scaling factors. For example, the second processing circuit 420 may also include a multiplication circuit for multiplying the quantized data and the scaling factors. In still other embodiments, the second rotation circuit 422 can also be used to perform rotation operations on the quantized data to convert the quantized data from the target data type to the data type required for dequantization processing; the multiplication circuit is used to multiply the rotated quantized data and the scaling factors to obtain the dequantization result. When the required data type is the original data type of the aforementioned data to be quantized, dequantization processing can restore the quantized data to the data to be quantized before the quantization operation.

[0113] It is understood that the dequantization process is not limited to being performed by the second processing circuit 420 as described above, but can also be performed by other circuits. For example, in some embodiments, the apparatus 500 may further include a third processing circuit 510 (shown in a dashed box in the figure) for dequantizing the quantized data using a scaling factor. The third processing circuit 510 may receive, for example, quantized data output by the second processing circuit 420, and may also receive scaling factors output by the first processing circuit 410. In some embodiments, a storage circuit may be further included between the third processing circuit 510 and the second processing circuit 420, or the second processing circuit 420 may include a storage circuit for storing quantized data so that the third processing circuit 510 can read the quantized data from the storage circuit for dequantization processing.

[0114] In some embodiments, when a piece of data to be quantized is input to the comparison circuit 411, the comparison circuit 411 can output the absolute value of that piece of data as the maximum absolute value. In some embodiments, when multiple pieces of data to be quantized are input to the comparison circuit 411, the comparison circuit 411 can output the maximum absolute value among the multiple pieces of data to be quantized. In still some embodiments, the comparison circuit 411 can also be used to divide the multiple pieces of data to be quantized into multiple groups and output the maximum absolute value of each group of data to be quantized; the first division circuit 412 can be further used to determine a scaling factor corresponding to each group of data to be quantized; and the first revolution circuit 413 can be further used to determine a scaling coefficient corresponding to each group of data to be quantized based on each scaling factor. In some embodiments, the data to be quantized is an element of tensor data, and the comparison circuit 411 can be further used to divide the multiple pieces of data to be quantized of the tensor data into multiple groups in at least one dimension of the tensor data.

[0115] It is understood that the above description is exemplary and not restrictive. For example, the first processing circuit for obtaining the scaling factor may not be limited to the comparison circuit, the first division circuit, and the first rotation circuit described above, but may also be implemented using, for example, a lookup table (circuit). Since the dividend (the maximum value that the target data type can represent) is a constant, a lookup table or similar method can be used to directly determine the value based on the input data, thereby implementing the division operation and data type conversion functions described above to obtain the scaling factor. Furthermore, to facilitate understanding of the grouping function implemented by the comparison circuit, an exemplary description will be provided below with reference to Figure 6.

[0116] Figure 6 shows a schematic block diagram of a comparison circuit according to some embodiments of this disclosure. As shown in Figure 6, the comparison circuit 411 may include a comparator tree composed of multiple levels of comparators, for example, it may include layers a, b, c, ..., z, each level including one or more comparators, with the number of comparators decreasing sequentially at each level. The outputs of multiple comparators in the upper level of an adjacent level can be input to the comparators in the lower level. In practical applications, the outputs of the required levels can be selected as needed by means of software or hardware to achieve grouping of multiple data to be quantized and to obtain the maximum absolute value of each group of data to be quantized. For ease of understanding, further examples will be provided below with reference to Figure 6.

[0117] As further illustrated in Figure 6, layer a includes comparators a1, a2, a3, ..., an, where n represents the number of comparators in layer a; layer b includes comparators b1, b2, etc.; layer c includes comparator c1, etc.; and layer z includes comparator z1. It is understood that the comparator tree shown in the figure is exemplary, and the number of levels in the comparator tree can be set more or less as needed, and the number of comparators included in each level can also be set as required.

[0118] In some embodiments, the number of comparators *n* in layer *a* can be the same as the number of input data to be quantized, so that each data point to be quantized is input into a corresponding comparator in layer *a*. As the number of comparators in subsequent layers decreases sequentially, the output can be gradually reduced to obtain the desired grouping result and the maximum absolute value of each group. For example, assuming the input layer receives 128 data points to be quantized, and layer *a* has 128 comparators, then each data point to be quantized is input into a corresponding comparator in layer *a*. The output of layer *a* can then divide the 128 data points to be quantized into 128 groups. Assuming layer *b* includes 64 comparators, then the output of every two comparators in layer *a* can be input into a comparator in the next layer, layer *b*. For example, the outputs of comparators *a1* and *a2* in layer *a* can be input into comparator *b1* in layer *b*. Layer *b* then outputs 64 maximum absolute values, thus dividing the 128 data points to be quantized into 64 groups. Assuming layer c contains 32 comparators, the outputs of every two comparators in layer b can be input into a comparator in the next layer c. For example, the outputs of comparators b1 and b2 in layer b can be input into comparator c1 in layer c. At this point, layer c outputs 32 maximum absolute values, effectively dividing the 128 data points to be quantized into 32 groups. This process continues until the last layer z outputs a single maximum absolute value, thus dividing the 128 data points to be quantized into one group.

[0119] Taking this embodiment as an example, in practical applications, assuming that 128 data points to be quantized need to be divided into groups of four, resulting in 32 groups, the output of layer C can be selected as the output of comparator circuit 411 through methods such as programming, logic control, or selectors. This allows the first division circuit to determine the scaling factor corresponding to each of the 32 groups of data to be quantized based on the 32 outputs of layer C (i.e., the 32 maximum absolute values). Therefore, the comparator tree provides multiple granularity grouping methods, allowing users to flexibly select the output of a certain level in the comparator tree according to the required quantization granularity.

[0120] Furthermore, it should be noted that the comparator tree structure is not limited to the two comparators in each level shown in the diagram whose outputs correspond to one comparator in the next level. It can also be configured to have, for example, three or more comparators whose outputs correspond to one comparator in the next level. For instance, assuming level a has 128 comparators, level b can have 4 comparators. That is, the outputs of the 32 comparators in level a correspond to one comparator in level b. Therefore, the output of level b can divide the 128 data points to be quantized into 4 groups and output the maximum absolute value of the 32 data points in each group.

[0121] In other embodiments, the number of comparators n in layer a can be different from the number of input data to be quantized, as illustrated below. Assuming the number of comparators in layer a is less than the number of data to be quantized, multiple data to be quantized can be input into a single comparator in layer a. This achieves the division of multiple data to be quantized into fewer groups for output in layer a. In this embodiment, the data to be quantized input to each comparator in layer a can also be controlled before layer a (e.g., in the input layer) by using methods such as address decoders, multiplexers (MUX), or other logic circuits.

[0122] For example, suppose the input layer receives 128 data points to be quantized, and layer 'a' has 64 comparators. Then, every two data points to be quantized are input into one of the corresponding comparators in layer 'a'. At this point, the output of layer 'a' divides the 128 data points into 64 groups. The number of comparators in subsequent layers is halved sequentially until the last layer outputs the maximum absolute value, at which point the 128 data points are divided into one group.

[0123] The foregoing description, in conjunction with Figure 6, exemplarily illustrates the scheme for grouping and outputting the maximum absolute value using a comparison circuit according to an embodiment of this disclosure. It is understood that the structure of the comparison circuit shown in the figure and the preceding description are exemplary and not restrictive. For example, the last level of comparison circuit 411 may not be limited to including only one comparator as shown in the figure, but may also include multiple comparators as needed, such as layer c in the figure as the last level. Furthermore, comparator circuit 411 may not be limited to including only one comparator tree, but may also include multiple comparator trees as needed, thereby enabling, for example, parallel grouping. The comparison circuit of this embodiment of the disclosure may not be limited to being implemented using the aforementioned comparator tree, but may also be implemented based on radix sorting or other methods.

[0124] It is also understood that the grouping method in this disclosed embodiment is not limited to the comparison circuit, but can also be implemented by combining software. For example, by setting grouping parameters (including information such as the number of data to be quantized in each group K) through software, the comparison circuit can read K consecutive data to be quantized as a group according to these grouping parameters, and directly output the maximum absolute value in each group of data to be quantized.

[0125] Figure 7 shows a schematic block diagram of a data quantization apparatus according to some embodiments of the present disclosure. As shown in Figure 7, the apparatus 700 may include a first processing circuit 410, a second processing circuit 420, and an arithmetic circuit 710, wherein the arithmetic circuit 710 can be used to perform arithmetic operations related to the data to be quantized, and the arithmetic circuit 710 may include the second processing circuit 420, that is, the second processing circuit 420 may be integrated within the arithmetic circuit 710.

[0126] The computational operations related to the data to be quantized described above can be operations that directly or indirectly use the data to be quantized for computation. For example, in some embodiments, the computation circuit 710 may include a multiplication circuit 711 (shown in a dashed box in the figure) and / or an addition circuit 712 (shown in a dashed box in the figure), etc., to perform multiplication and / or addition operations on the data to be quantized. In some scenarios, the data to be quantized may not require preprocessing such as multiplication and / or addition operations, and can be directly input into the first processing circuit 410 to calculate the scaling factor.

[0127] Further, the second processing circuit 420 can receive the scaling factor output by the first processing circuit 410 for quantization processing of the data to be quantized. In some embodiments, the arithmetic circuit 710 may include a second division circuit, and the second processing circuit 420 may include a second rotation circuit. In this case, the second division circuit in the arithmetic circuit 710 can be used to perform a division operation on the data to be quantized and the scaling factor, and the second rotation circuit in the second processing circuit 420 can be used to perform a rotation operation based on the result of the division operation to obtain the quantized data. In other embodiments, the multiplication circuit 711 in the arithmetic circuit 710 can also be used to perform a multiplication operation on the quantized data and the scaling factor to obtain an inverse quantization result.

[0128] The apparatus including the arithmetic circuit according to the embodiments of the present disclosure has been described above with reference to FIG7. It is understood that the above description is exemplary and not restrictive. For example, the second processing circuit 420 may not be integrated into the arithmetic circuit 710 as shown in the figure, or it may be set independently of the arithmetic circuit 710 as needed.

[0129] Exemplary hardware architecture

[0130] Figure 8 shows a schematic diagram of a board structure according to an embodiment of this disclosure. As shown in Figure 8, the board 800 includes one or more chips 801 (i.e., an implementation of the integrated circuit device described herein), which is a system-on-a-chip (SoC) that integrates one or more combined processing devices. The combined processing device can be an artificial intelligence computing unit used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in the field of cloud intelligence. A significant characteristic of cloud intelligence applications is the large amount of input data, which places high demands on the platform's storage and computing capabilities. The board 800 of this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

[0131] Chip 801 is connected to external device 803 via external interface device 802. External device 803 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 803 to chip 801 via external interface device 802. The calculation results from chip 801 can be transmitted back to external device 803 via external interface device 802. Depending on the application scenario, external interface device 802 may have different interface forms, such as PCIe (Peripheral Component Interconnect express) interface.

[0132] The board 800 also includes a storage device 804 for storing data, which includes one or more memory cells 805. The storage device 804 is connected to and transmits data with the controller 806 and the chip 801 via a bus. The controller 806 in the board 800 is configured to regulate the state of the chip 801. Therefore, in one application scenario, the controller 806 may include a microcontroller, also known as a microcontroller unit (MCU).

[0133] Figure 9 is a structural diagram showing the combined processing device in the chip 801 of the embodiment shown in Figure 8. As shown in Figure 9, the combined processing device 900 includes a computing device 901, an interface device 902, a processing device 903, and a storage device 904.

[0134] The computing device 901 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 903 through the interface device 902 to jointly complete the user-specified operations.

[0135] Interface device 902 is used to transmit data and control commands between computing device 901 and processing device 903. For example, computing device 901 can obtain input data from processing device 903 via interface device 902 and write it to on-chip storage device of computing device 901. Further, computing device 901 can obtain control commands from processing device 903 via interface device 902 and write them to on-chip control cache of computing device 901. Alternatively or optionally, interface device 902 can also read data from storage device of computing device 901 and transmit it to processing device 903.

[0136] Processing device 903, as a general-purpose processing device, performs basic controls including but not limited to data transfer and starting / stopping computing device 901. Depending on the implementation, processing device 903 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, computing device 901 disclosed herein can be considered as having a single-core structure or a homogeneous multi-core structure. However, when computing device 901 and processing device 903 are considered together, they are considered to form a heterogeneous multi-core structure.

[0137] The storage device 904 is used to store data to be processed. It may be DRAM (Dynamic Random Access Memory), DDR (Double Data Rate) memory, typically 16G or larger, and is used to store data of computing device 901 and / or processing device 903.

[0138] This disclosure also provides a data quantization computing apparatus, including at least one processor; and at least one memory in communication with the at least one processor, having stored computer-readable instructions that, when loaded and executed by the at least one processor, cause the at least one processor to perform the aforementioned data quantization method.

[0139] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various circuits in the device embodiments described above, this document has decomposed them based on their logical functions, but in actual implementation, there may be other decomposition methods. The physical implementation of the hardware structure of the circuit may include, but is not limited to, physical devices, which may include, but are not limited to, devices such as transistors or memristors. In view of this, the various devices described herein (e.g., computing devices or other processing devices) can be implemented by appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage unit or storage device can be any suitable storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), ROM (read-only memory), and RAM (random access memory), etc.

[0140] In this disclosed embodiment, the term "instruction" can include software instructions, hardware instructions, firmware instructions, or any combination thereof. Software instructions typically refer to statements or commands in a programming language; they are high-level abstractions, such as function calls, machine code, and bytecode. Hardware instructions, also known as machine instructions or instruction set architecture (ISA) instructions, are low-level commands that computer hardware can directly recognize and execute, such as processor instructions and CPU instructions. Firmware instructions include, for example, opcodes and microcode. Furthermore, different CPU architectures and different instruction sets can have different hardware instruction sets. Based on the complexity of the computer instruction set, hardware design, execution speed, compiler complexity, and instruction format, they can be further classified as: CISC (Complex Instruction Set Computer), RISC (Reduced Instruction Set Computer), and VLIW (Very Long Instruction Word).

[0141] In summary, the embodiments disclosed herein determine the scaling factor based on the maximum absolute value of the data to be quantized, and obtain the scaling coefficient by rotating the scaling factor. This allows the quantized data obtained by quantization based on the scaling coefficient to help ensure the accuracy and results of the processing task when performing the processing task, while also helping to reduce resource consumption.

[0142] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0143] While numerous embodiments of this disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and intent of this disclosure. It should be understood that various alternatives to the embodiments of this disclosure described herein may be employed in the practice of this disclosure. The appended claims are intended to define the scope of this disclosure and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A method for data quantization, comprising: determining an absolute value maximum in a set of data to be quantized; obtaining a maximum numerical value that can be represented by a target data type of the data to be quantized after quantization; determining a scaling factor corresponding to the set of data to be quantized based on a ratio between the absolute value maximum and the maximum numerical value; converting the scaling factor from a first data type to a second data type to obtain a scaling coefficient, wherein a bit width of the second data type is smaller than a bit width of the first data type; and quantizing the set of data to be quantized using the scaling coefficient to obtain quantized data having the target data type.

2. The method of claim 1, further comprising: determining a number of data to be quantized in the set of data to be quantized according to a single processing amount and / or a number of input ports of a processing circuit that performs the quantization.

3. The method of claim 1 or 2, wherein the first data type is the same as an original data type of the data to be quantized, or a bit width of the first data type is greater than a bit width of the original data type; and E represents an exponent bit, M represents a mantissa bit, x represents an exponent bit width, and y represents a mantissa bit width.

4. The method of any one of claims 1-3, wherein the original data type of the data to be quantized comprises at least one of FP16, BF16, FP32, FP64, and TF32; the second data type comprises at least one of E8M0, E8M1, E8M2, E8M3, E8M4, E8M5, E8M6, E8M7, E5M0, E5M1, E5M2, E5M3, E5M4, E5M5, E5M6, E3M0, E3M1, E3M2, E3M3, E3M4, E3M5, E3M6, E3M7, E3M8, and E3M9; and the target data type comprises at least one of FP8, FP6, FP4, FP2, int8, int4, int2, int6, and FP12.

5. The method of claim 3, wherein y>0.

6. The method of claim 1, wherein the data to be quantized is an element of tensor data, and the method further comprises: dividing a plurality of data to be quantized of the tensor data into a plurality of sets in at least one dimension of the tensor data.

7. The method of any one of claims 1-6, wherein the data to be quantized is input data and / or weight data of a neural network.

8. The method of claim 1, further comprising: dequantizing the quantized data using the scaling coefficient.

9. The method of any one of claims 1-8, further comprising: compressing two or more scaling coefficients of two or more sets of data to be quantized after quantization; and ​ ​ ​ ​ The second data type is represented as E x M y , x + y < bit width of original data type, wherein, ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ Quantize the compressed scaling coefficients respectively to the corresponding group of data to be quantized to obtain quantized data of the target data type. 10.The method of claim 9, wherein the compressing comprises: determining a shared exponent of Z scaling coefficients to be compressed, Z≥2; determining Z new mantissas corresponding to the Z scaling coefficients; and binding the shared exponent and the Z new mantissas for associated use. 11.The method of claim 10, wherein determining a shared exponent of Z scaling coefficients to be compressed comprises: setting a maximum exponent in the Z scaling coefficients as the shared exponent. 12.The method of claim 11, wherein determining Z new mantissas corresponding to the Z scaling coefficients comprises: setting a new mantissa of a scaling coefficient with an exponent less than the maximum exponent as 0; and setting a new mantissa of a scaling coefficient with an exponent equal to the maximum exponent as an original mantissa. 13.The method of any of claims 10-12, further comprising one or more of: the second data type of the Z scaling coefficients to be compressed is the same; the Z scaling coefficients belong to Z adjacent groups of data to be quantized; Z is 2 or 4 or 8. 14.The method of any of claims 10-13, further comprising: decompressing the compressed scaling coefficients in response to the scaling coefficients being in a compressed format; and inverse quantizing the quantized data using the decompressed scaling coefficients.

15. The method of claim 14, wherein, The decompressing comprises: combining the shared exponent and the Z new mantissas respectively to obtain Z decompressed scaling coefficients. 16.An apparatus for data quantization, comprising: a first processing circuit configured to determine an absolute value maximum in a group of data to be quantized; obtain a maximum value representable by a target data type of the data to be quantized after quantization; determine a scaling factor corresponding to the group of data to be quantized based on a ratio between the absolute value maximum and the maximum value; convert the scaling factor from a first data type to a second data type to output a scaling coefficient, wherein a bit width of the second data type is less than a bit width of the first data type; and a second processing circuit configured to quantize the group of data to be quantized using the scaling coefficient to output quantized data of the target data type. 17.The apparatus of claim 16, wherein the first processing circuit comprises: a comparison circuit configured to determine an absolute value maximum in a group of data to be quantized; a first division circuit configured to obtain a maximum value representable by a target data type of the data to be quantized after quantization, and determine a scaling factor corresponding to the group of data to be quantized based on a ratio between the absolute value maximum and the maximum value; and a first conversion circuit configured to convert the scaling factor from a first data type to a second data type to output a scaling coefficient. 18.The apparatus of claim 16, wherein the first processing circuit is further configured to: ​ ​ determining the number of the to-be-quantized data in the group of to-be-quantized data according to a single processing amount and / or a number of input ports of the second processing circuit; or determining the number of the to-be-quantized data in the group of to-be-quantized data according to a single operation amount and / or a number of input ports of an operation unit using the quantized data for operation.

19. The apparatus of claim 16, further comprising an operation circuit configured to perform an operation related to the to-be-quantized data, and the operation circuit comprises the second processing circuit, or the second processing circuit is independently arranged from the operation circuit.

20. The apparatus of any one of claims 16-19, wherein the first data type is the same as an original data type of the to-be-quantized data, or a bit width of the first data type is greater than a bit width of the original data type; and The second data type is represented as E x M y , x + y < bit width of original data type, wherein, E represents an exponent bit, M represents a mantissa bit, x represents an exponent bit width, and y represents a mantissa bit width.

21. The apparatus of any one of claims 16-20, wherein the original data type of the to-be-quantized data comprises at least one of FP16, BF16, FP32, FP64, and TF32; and the second data type comprises at least one of E8M0, E8M1, E8M2, E8M3, E8M4, E8M5, E8M6, E8M7, E5M0, E5M1, E5M2, E5M3, E5M4, E5M5, E5M6, E3M0, E3M1, E3M2, E3M3, E3M4, E3M5, E3M6, E3M7, E3M8, and E3M9. the target data type comprises at least one of FP8, FP6, FP4, FP2, int8, int4, int2, int6, and FP12.

22. The apparatus of claim 20, wherein y>0.

23. The apparatus of claim 16, wherein the to-be-quantized data is an element of tensor data, and the first processing circuit is further configured to: divide a plurality of to-be-quantized data of the tensor data into a plurality of groups in at least one dimension of the tensor data.

24. The apparatus of any one of claims 16-23, wherein the to-be-quantized data is input data and / or weight data of a neural network.

25. The apparatus of claim 16, wherein the second processing circuit is further configured to: dequantize the quantized data using the scaling coefficients; or the apparatus further comprises a third processing circuit configured to dequantize the quantized data using the scaling coefficients.

26. The apparatus of any one of claims 16-25, further comprising a compression circuit configured to: compress two or more scaling coefficients corresponding to two or more groups of to-be-quantized data after quantization; and the second processing circuit is configured to quantize the to-be-quantized data in the corresponding groups using the compressed scaling coefficients respectively to output quantized data having the target data type.

27. The apparatus of claim 26, wherein the compression circuit is further configured to: determining a shared exponent of Z scaled coefficients to be compressed, Z≥2; determining Z new mantissas corresponding to the Z scaled coefficients; and binding the shared exponent and the Z new mantissas together for associated use.

28. The apparatus of claim 27, wherein the compression circuit is further configured to: set a largest exponent among the Z scaled coefficients as the shared exponent.

29. The apparatus of claim 28, wherein the compression circuit is further configured to: set a new mantissa of a scaled coefficient with an exponent smaller than the largest exponent as 0; and set a new mantissa of a scaled coefficient with an exponent equal to the largest exponent as an original mantissa.

30. The apparatus of any one of claims 27-29, further comprising one or more of: the second data type of the Z scaled coefficients to be compressed is the same; the Z scaled coefficients belong to Z adjacent groups of data to be quantized; Z is 2 or 4 or 8.

31. The apparatus of any one of claims 27-30, further comprising a decompression circuit configured to: decompress the compressed scaled coefficients in response to the scaled coefficients being in a compressed format; and the second processing circuit or the third processing circuit is configured to utilize the decompressed scaled coefficients to perform inverse quantization on the quantized data.

32. The apparatus of claim 31, wherein, the decompression circuit is configured to: combine the shared exponent and the Z new mantissas respectively to obtain Z decompressed scaled coefficients.

33. An integrated circuit apparatus comprising the apparatus of any one of claims 16-32.

34. A board card comprising the integrated circuit apparatus of claim 33.

35. A computing apparatus for data quantization, comprising: at least one processor; and at least one memory in communication with the at least one processor, having computer readable instructions stored thereon that, when loaded and executed by the at least one processor, cause the at least one processor to perform the method of any one of claims 1-15.

36. A computer readable storage medium having program instructions stored thereon that, when loaded and executed by a processor, cause the processor to perform the method of any one of claims 1-15. ​

Images