Data quantization method and apparatus based on lookup table, processor, medium

Abstract

At least one embodiment of the present disclosure provides a lookup table based data quantization method, device, processor and non-transitory computer readable storage medium. The method comprises: receiving data to be quantized, wherein the data to be quantized is of a first precision type; obtaining an index of the data to be quantized for a lookup table based on a first exponent part and a first mantissa part of the data to be quantized, wherein the index comprises a high bit part of the first mantissa part of the data to be quantized; performing a lookup in the lookup table based on the index, and obtaining a quantization result of a second precision type for the data to be quantized according to the lookup result, wherein the second precision type has lower precision than the first precision type. At least one embodiment of the present disclosure saves hardware storage space by using a smaller number of index structures, improves lookup table efficiency, and efficiently performs quantization. Furthermore, the bit width of the lookup table index can be adjusted according to the precision requirement, further reducing the error of the quantization result and maintaining the accuracy of the quantization result.

Description

Technical Field

[0001] At least one embodiment of this disclosure relates to the field of data processing, and more specifically, to a lookup table-based data quantization method, a lookup table-based data quantization apparatus, a processor, and a non-transitory computer-readable storage medium. Background Technology

[0002] When computing deep learning models on Graphics Processing Units (GPUs) or General-Purpose Computing on Graphics Processing Units (GPGPUs), high-precision numerical inputs can lead to significant computational and storage overhead. Therefore, deep learning models typically require quantization. This involves reducing the precision of the data representation (e.g., converting from 32-bit floating-point (FP32) to 4-bit floating-point (FP4)) to reduce model storage requirements, accelerate inference speed, and lower power consumption. This is particularly important in resource-constrained scenarios (such as mobile devices and edge devices), but it also introduces quantization errors, requiring technical means to balance accuracy and efficiency.

[0003] For example, the FP32 number data format has 32 bits (1 sign bit + 8 exponent bits + 23 mantissa bits), and the numerical range is approximately ±3.4 × 10⁻⁶. 38 The FP4 data format has only 4 bits (1 sign bit + 2 exponent bits + 1 mantissa bit), and its numerical range is only [-1, +2] (for example, when the bias = 1). The FP4 format only requires 1 / 8 of the storage space of FP32, and the FP4 format is a non-uniform quantization, with the numerical intervals being denser near zero and sparser far from zero. Therefore, it is often used for non-uniformly distributed data in deep learning models.

[0004] However, there is still a need to improve current quantification methods by accelerating execution, maintaining accuracy, and saving storage space. Summary of the Invention

[0005] According to one aspect of this disclosure, at least one embodiment provides a data quantization method based on a lookup table, comprising: receiving data to be quantized, wherein the data to be quantized is of a first precision type; obtaining an index of the data to be quantized to a lookup table based on a first exponent portion and a first mantissa portion of the data to be quantized, wherein the index includes the high-order portion of the first mantissa portion of the data to be quantized; performing a search in the lookup table based on the index; and obtaining a quantization result of a second precision type for the data to be quantized based on the search result, wherein the precision of the second precision type is lower than the precision of the first precision type.

[0006] In some embodiments, the second precision type is a 4-bit floating-point number FP4, and the high-order part of the first mantissa portion of the data to be quantized includes the high-order 3 bits of the mantissa of the data to be quantized.

[0007] In some embodiments, the data quantization method further includes: in response to the data to be quantized being a single input data, taking the data to be quantized as the maximum value; or, in response to the data to be quantized being one of a plurality of combined input data included in a data block, obtaining the maximum value among the absolute values ​​of each input data in the plurality of combined input data; and decomposing the maximum value into a maximum exponent part and a maximum mantissa part.

[0008] In some embodiments, the index further includes an exponential difference, wherein the exponential difference is the difference between the maximum exponential portion and the exponent of the data to be quantized, and the method further includes limiting the range of values ​​for the exponential difference by means of a threshold.

[0009] In some embodiments, the data quantization method further includes setting a threshold such that when the difference in the exponential portion exceeds the threshold, the resulting quantization will exceed the expression range of the second precision type.

[0010] In some embodiments, the second precision type is a 4-bit floating-point number FP4, and the difference in the exponent portion of the index is 3 bits for the E2M1 format of the 4-bit floating-point number FP4 or 2 bits for the E1M2 format of the 4-bit floating-point number FP4.

[0011] In some embodiments, the index is 9 bits for E2M1 format or 8 bits for E1M2 format.

[0012] In some embodiments, the index further includes the mantissa of a scaling factor used for data quantization, and the method further includes obtaining the range of values ​​for the exponent of the scaling factor by using a threshold of the difference in the exponent portions. In some embodiments, the second precision type is a 4-bit floating-point number FP4, and the mantissa of the scaling factor is 3 bits for a 4-bit floating-point number FP4.

[0013] In some embodiments, performing a lookup in the lookup table based on the index and obtaining a second-precision type quantization result for the data to be quantized based on the lookup result includes: determining the optimal scaling factor and quantization result by: obtaining multiple different candidate scaling factors based on the number of bits and / or the value range of the exponent of the scaling factor; obtaining multiple candidate quantization results by using the mantissa of each of the multiple different candidate scaling factors and the lookup table; calculating multiple differences between the multiple dequantized results of the multiple candidate quantization results after dequantization with the candidate scaling factors and the data to be quantized; determining the candidate scaling factor with the smallest difference as the optimal scaling factor; and using the candidate quantization result corresponding to the optimal scaling factor as the second-precision type quantization result of the data to be quantized.

[0014] In some embodiments, multiple candidate quantization results are obtained using the mantissa of each of the multiple different candidate scaling factors and a lookup table, including the following steps: using the mantissa of each of the multiple different candidate scaling factors to find multiple first quantization results in the lookup table; adding or subtracting one or more positive integers to the encoding space of each first quantization result to obtain one or more first candidate quantization results; calculating the deviation between the result of each first quantization result after dequantization using the candidate scaling factor and the data to be quantized, and calculating one or more deviations between the result of one or more first candidate quantization results after dequantization using the candidate scaling factor and the data to be quantized; determining the quantization result with the smallest deviation as the candidate quantization result for the candidate scaling factor.

[0015] According to another aspect of this disclosure, at least one embodiment provides a lookup table-based data quantization apparatus, comprising: a receiving module configured to receive data to be quantized, wherein the data to be quantized is of a first precision type; an obtaining module configured to obtain an index of the data to be quantized in a lookup table based on a first exponent portion and a first mantissa portion of the data to be quantized, wherein the index includes the high-order portion of the first mantissa portion of the data to be quantized; and a searching module configured to perform a search in the lookup table based on the index, and obtain a quantization result of a second precision type for the data to be quantized based on the search result, wherein the precision of the second precision type is lower than the precision of the first precision type.

[0016] According to another aspect of this disclosure, at least one embodiment provides a lookup table-based data quantization apparatus, comprising: at least one memory storing computer instructions; and at least one processor configured to execute the computer instructions in the at least one memory to perform a method according to at least one embodiment of this disclosure.

[0017] According to another aspect of this disclosure, at least one embodiment provides a non-transitory computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions, when executed by at least one processor, cause at least one processor to perform a method according to at least one embodiment of this disclosure.

[0018] According to another aspect of this disclosure, at least one embodiment provides a processor, wherein the processor includes at least one computing unit, wherein the computing unit includes a lookup table-based data quantization device and a lookup table according to at least one embodiment of this disclosure.

[0019] According to another aspect of this disclosure, at least one embodiment provides a computer program product including computer instructions, wherein, when executed by at least one processor, the computer instructions cause the at least one processor to perform a method according to at least one embodiment of this disclosure. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A schematic diagram of a general-purpose graphics processing unit (GPGPU) is shown.

[0022] Figure 2 A flowchart of a data quantization method based on a lookup table (LUT) according to at least one embodiment of the present disclosure is shown.

[0023] Figure 3 A complete quantization process example according to at least one embodiment of the present disclosure is shown, including a process for simplifying the lookup table, compensating for the precision of the index, and selecting the optimal scaling factor.

[0024] Figure 4 A block diagram of a lookup table-based data quantization apparatus according to at least one embodiment of the present disclosure is shown.

[0025] Figure 5 A block diagram of a lookup table-based data quantization apparatus according to at least one embodiment of the present disclosure is shown.

[0026] Figure 6 A block diagram of a lookup table-based data quantization apparatus according to at least one embodiment of the present disclosure is shown.

[0027] Figure 7 A schematic block diagram of an electronic device according to at least one embodiment of the present disclosure is shown. Detailed Implementation

[0028] Referring now to specific embodiments of this disclosure, examples of which are illustrated in the accompanying drawings. Although this application will be described in conjunction with specific embodiments, it will be understood that it is not intended to limit this application to the described embodiments. Rather, it is intended to cover variations, modifications, and equivalents included within the spirit and scope of this disclosure. It should be noted that the method steps described herein can be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of both.

[0029] In this article, "multiple" refers to two or more. "Or" describes the relationship between related objects, indicating that there can be three relationships. For example, A or B can represent: A alone, both A and B, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0030] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0031] GPUs or GPGPUs can be used for data computation in neural networks and deep learning. Specific applications include image processing (real-time video processing and analysis, image super-resolution and enhancement, style transfer and generation), computer vision (face recognition and authentication, object detection and tracking), and other computationally intensive tasks (gene sequencing and bioinformatics, financial analysis and risk management, virtual reality and augmented reality). When GPUs or GPGPUs process large amounts of data, quantization is needed to reduce data representation precision, thereby reducing data storage requirements, accelerating inference speed, and lowering power consumption.

[0032] Figure 1 A schematic diagram of a general-purpose graphics processing unit (GPGPU) is shown.

[0033] like Figure 1 As shown, a general-purpose graphics processor is actually an array of programmable multiprocessors. For example, a programmable multiprocessor can be a streaming processor cluster (SPC), such as including... Figure 1 The diagram shows streaming processor clusters 1, ..., M, where M is a positive integer greater than 1. In a general-purpose graphics processor, one streaming processor cluster handles one computational task, or multiple streaming processor clusters handle one computational task. Multiple streaming processor clusters share data through a global cache or global memory.

[0034] like Figure 1 As shown, taking streaming processor cluster 1 as an example, one streaming processor cluster includes multiple computing units, such as... Figure 1The system is structured as Computation Unit 1, Computation Unit 2, ..., Computation Unit N, where N is a positive integer. Each Computation Unit (CU) performs arithmetic and logical operations, such as accumulation, reduction, and standard addition, subtraction, multiplication, and division. A Computation Unit includes multiple cores (also called computational kernels), each of which includes an Arithmetic Logic Unit (ALU), a floating-point unit, etc. These cores are used to execute specific computational tasks. Furthermore, the Computation Unit also includes registers (e.g., ...). Figure 1 The register file and shared memory in a computing unit are used to store source and destination data related to computing tasks in a hierarchical manner. The shared memory in a computing unit is used to share data between the cores of that computing unit.

[0035] like Figure 1 As shown, each computing unit also provides a tensor core for performing tensor-related computations, such as tensor shrinking operations. Tensor cores can accelerate tensor operations such as matrix multiplication. Tensor cores in multiple computing units can be scheduled and controlled uniformly.

[0036] like Figure 1 As shown, each streaming processor cluster also provides a buffer for caching data across the N computing units within the streaming processor cluster.

[0037] In parallel computing, computational tasks are typically executed by multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block distribution module. Figure 1 (Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0038] In each computing unit, the thread beam scheduling / distribution module ( Figure 1 (Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple computing cores within the computing unit can run thread bundles. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can execute concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory-executed instructions are issued to shared memory within the computing unit or further issued to intermediate-level caches, global caches, or global memory (e.g., [example cache]). Figure 1High Bandwidth Memory (HBM) is used for read and write operations.

[0039] GPUs or GPGPUs process a large number of floating-point numbers during deep learning and neural network calculations. However, different precision types of floating-point numbers occupy different amounts of storage space. The goal is to convert higher precision floating-point numbers, which occupy more storage space, into lower precision floating-point numbers, which occupy less storage space, in order to speed up data storage requirements, accelerate inference speed, and reduce power consumption.

[0040] Floating-point numbers are represented in scientific notation as:

[0041] Value = (-1) sign ×1.X_mantissa×

[0042] Sign bit: 1 bit, indicating positive or negative (0 for positive, 1 for negative).

[0043] index( ) Bit: 2 bits, directly representing the exponent value (e.g., 00=0, 01=1, 10=2, 11=3), a discrete value ranging from 0 to 3. It can be converted into the actual exponent by subtracting the bias. After biasing by 1, it is a discrete value ranging from -1 to 2.

[0044] The mantissa (X_mantissa) bit: 1 bit, stores the fractional part (implicitly the first bit is 1, such as 1. X_mantissa).

[0045] Bias: A fixed value used to map unsigned exponents to signed ranges, such as 1.

[0046] Max-Normalization quantization is a data quantization method that linearly transforms high-precision data to a fixed range to reduce precision. Max-Normalization quantization typically uses a single scaling factor to map the range of input data to the expression range of the target precision type. Specifically, given an input tensor X, its quantization formula is:

[0047] Q = round(X / (max(|X|) / ))

[0048] in, This represents the maximum value of the target precision. For example, the 4-bit floating-point format FP4 is frequently used in artificial intelligence (AI) computing. In the E2M1 format of FP4, = 6, in E1M2 format =3.5.

[0049] FP4 quantization maps data to a 4-bit floating-point representation. A typical representable range for FP4 is [-1.5, -0.25] ∪ [0.25, 1.5] (a symmetric range). The data X needs to be mapped to this range using a scaling factor SF, i.e., X / SF. According to the aforementioned formula Q = round(X / (max(|X|) / ... Therefore, the scaling factor SF is usually the maximum value of X, max(|X|), divided by the maximum range that the target precision type can express. ,Right now, ,here , This is the maximum range for the target precision type (e.g., 6 for E2M1, 3.5 for E1M2). It is a very small floating-point number (e.g.) =1e -8 , 1e -9 ,…,1e -12 (etc.) is used to prevent division by zero. This scaling factor is typically a single, fixed value for X. The scaling factor SF can also be expressed in scientific notation as described above, including the sign bit, mantissa, and exponent. Note that to reduce the size of the scaling factor, it is often quantized to a lower precision.

[0050] However, this single scaling factor approach performs poorly when the input data is unevenly distributed. Furthermore, the numerous operations such as division during quantization impose significant area overhead and latency on the hardware, making it difficult to deploy efficiently in high-performance accelerators used for neural network processing. To reduce the various overheads of real-time computation, lookup tables are designed to quickly retrieve quantized data from the input data. However, if the lookup table utilizes all 11 bits of the index (8 bits...), it becomes problematic. (The last few digits of the data to be quantized) and 3 digits (The mantissa portion of the scaling factor) requires 2048 entries / table, occupying more than 6KB of storage space. Additionally, the exponent portion needs alignment, resulting in significant hardware overhead; an 8-bit wide shifter is required to handle the exponent difference. - ).

[0051] The defects in the above solutions were discovered by the inventors of this disclosure through careful research and creative labor. The discovery process of the above problems and the solutions proposed by at least one embodiment disclosed below for the above problems are all creative contributions.

[0052] According to at least one embodiment of this disclosure, a data quantization method and apparatus based on a lookup table are proposed. The method involves receiving data to be quantized, wherein the data is of a first precision type; obtaining an index of the data to be quantized in a lookup table based on a first exponent portion and a first mantissa portion of the data to be quantized, wherein the index includes the high-order bits of the first mantissa portion of the data to be quantized; performing a lookup in the lookup table based on the index; and obtaining a quantization result of a second precision type for the data to be quantized based on the lookup result, wherein the precision of the second precision type is lower than that of the first precision type. The complete quantization process (including candidate scaling factor evaluation and error calculation) is encoded into a unified lookup table. This saves hardware storage space and improves lookup efficiency through a less-bit index structure, while efficiently performing quantization. Furthermore, the bit width of the lookup table index can be adjusted according to precision requirements. Further, by limiting the range of values ​​for the exponent differences included in the index using a threshold, the number of index entries is reduced, further saving hardware storage space and improving lookup efficiency. Furthermore, by adding or subtracting one or more positive integers to the encoding space of the quantization result obtained from the lookup table, one or more candidate quantization results are obtained. By comparing the deviation of the second-precision type quantization result after dequantization with the scaling factor and the deviation of the one or more candidate quantization results after dequantization with the scaling factor and the data to be quantized, the quantization result with the smallest deviation is selected as the optimal quantization result for the data to be quantized for the scaling factor, thereby reducing the error of the quantization result and maintaining the accuracy of the quantization result. Further, based on the number of bits and / or the value range of the scaling factor's exponent, multiple different candidate scaling factors are obtained. Multiple candidate quantization results are obtained using the mantissa of each candidate scaling factor and the lookup table. Multiple differences between the multiple dequantization results of the multiple candidate quantization results after dequantization with the candidate scaling factors and the data to be quantized are calculated. The candidate scaling factor with the smallest difference is determined as the optimal scaling factor. The candidate quantization result corresponding to the optimal scaling factor is used as the second-precision type quantization result of the data to be quantized, which can obtain a more suitable scaling factor and corresponding quantization result to reduce the error of the quantization result and maintain the accuracy of the quantization result.

[0053] It should be noted that the data to be quantized and the quantization result of the method or apparatus provided in at least one embodiment of this disclosure may have different physical meanings depending on the application scenario. For example, the method or apparatus provided in at least one embodiment of this disclosure can be applied in fields such as speech processing, image processing, text processing, and video processing.

[0054] For example, in the field of speech processing, the data to be quantized and the quantization result can be the input, output data, intermediate calculation results, etc. in tasks such as feature extraction, speech enhancement, and speech recognition.

[0055] For example, in the field of image processing, the data to be quantized and the quantization result can be the input data, output data, and intermediate calculation results in tasks such as image recognition, feature extraction, image segmentation, object detection, image classification, and scene reconstruction.

[0056] For example, in the field of text processing, the data to be quantified and the quantification results can be input data, output data, intermediate calculation results, etc. in tasks such as text classification, sentiment analysis, and text generation.

[0057] For example, in the field of video processing, the data to be quantized and the quantization result can be the input, output data, intermediate calculation results, etc., as in the field of image processing, or the input, output data, intermediate calculation results, etc., as in video processing-specific operators such as optical flow operators (used to estimate motion between video frames) and target tracking operators (used to track specific targets in video).

[0058] Of course, this disclosure is not limited to this. For other application scenarios or fields, as long as data quantification is required, the method of at least one embodiment of this disclosure can be applied, which will not be described in detail here.

[0059] Taking the quantization of FP16 precision type to FP4 precision type as an example, the quantization process can be represented by the following formula:

[0060]

[0061]

[0062]

[0063]

[0064]

[0065]

[0066]

[0067] Here, X represents the input data to be quantized, for example, FP16 precision type. This represents the data after X has been normalized using a scaling factor. This indicates the quantization result for the target precision type (e.g., FP4 precision type). This represents the last part of the data to be quantized. This represents the exponential portion of the data to be quantified. SF represents the scaling factor. This represents the mantissa portion of the scaling factor. This represents the exponential part of the scaling factor. This represents the maximum exponential portion (which will be described later in the preprocessing section). This represents the exponential portion of the data to be quantified. This represents the difference in the exponential portion. It is a threshold introduced according to at least one embodiment of this disclosure, representing a limitation. The size is This limits the number of bits in the lookup table index, which will be explained in detail below.

[0068] Analysis revealed that the quantification results can be derived from... (The last few digits of the data to be quantified) (The mantissa of the scaling factor) The difference between the exponents is obtained, so they can be considered as indexes for the lookup table.

[0069] Among them, the last part ( The sign bit (F) is the mantissa of the data to be quantized in scientific notation (i.e., the significant digits after removing the sign bit and exponent). The data to be quantized is a floating-point number, such as FP16, that needs to be quantized.

[0070] The mantissa of the scaling factor ( The scaling factor (SF) is the mantissa of the quantization scaling factor, used to adjust the input range to the target quantization range. The scaling factor maps input values ​​to the quantization range. The scaling factor is typically expressed as SF = SF_max. sign × × (1 + ), of which SF sign It is the sign bit. It refers to the last digit, for example, 0.5 is represented as 1.5 × . This represents the exponent of the scaling factor. The scaling factor scales the input data X to the quantization range. , For example, if the input range is [-1.0, 1.0] and the target FP4 range is [-0.5, 0.5], then SF = 0.5 (i.e., ...). =0, =-1).

[0071] Exponential part difference ( - (or Δ_exp) is the difference between the maximum exponent portion and the exponent portion of the data to be quantized, used to align the exponent to the quantization range. The exponent of the data to be quantized, X, is... Maximum exponential part The exponent is obtained by decomposing the absolute value of the input data, such as vector data, into its maximum value. The difference Δ_exp = - Used to adjust the exponent of the data to be quantized, X, to the target range. Input X=0.3 ( =-2), if =1, then Δ_exp=1 - (-2)=3.

[0072] Therefore, in order to quickly retrieve the quantized data from the input data, the quantization result can be pre-calculated and a lookup table can be designed as follows for fast retrieval of the quantization result, where the index in the lookup table can be set to... , , An index value composed of these three elements can be used to look up the corresponding quantized result in the index table. Therefore, consider using specific... value, value, Values ​​form an index to look up the corresponding quantization result in the lookup table, thereby speeding up the quantization process.

[0073] According to at least one embodiment of this disclosure, in order to compress the size of the lookup table, it is desirable to compress the number of bits in the index portion and / or the number of value entries, thereby reducing storage space and speeding up the lookup efficiency, while at the same time it is desirable to reduce the error of the quantization result and maintain the accuracy of the quantization result.

[0074] Figure 2 A flowchart of a lookup table-based data quantization method 200 according to at least one embodiment of the present disclosure is shown.

[0075] The quantification method 200 includes steps 210, 220 and 230.

[0076] In step 210, the data to be quantized is received, wherein the data to be quantized is of the first precision type.

[0077] In some embodiments, method 200 may further include preprocessing: in response to the data to be quantized being a single input data, taking the data to be quantized as the maximum value; or in response to the data to be quantized being one of a plurality of combined input data included in a data block, obtaining the maximum value among the absolute values ​​of each input data in the plurality of combined input data; and decomposing the maximum value into a maximum exponent part and a maximum mantissa part.

[0078] For example, input 16 BF16 elements (X0 to X...) 15 (This forms a vector data block to be quantized, where each element is a (combined) input data) (where j is an integer in the range of 0 to 15), for multiple combinations of input data in the data block. Calculate the maximum absolute value ,Will Decomposed into the largest exponential part and the largest last digit : Here, the maximum exponent part and the maximum mantissa part refer to the maximum value. The exponent and mantissa parts obtained by decomposing using scientific notation.

[0079] In step 220, an index of the data to be quantized to the lookup table is obtained based on the first exponent part and the first mantissa part of the data to be quantized, wherein the index includes the high-order part of the first mantissa part of the data to be quantized.

[0080] The tail portion of the data to be quantified It is usually 7 bits, but experiments have shown that a portion of the lower bits... The impact on the result is relatively small. Therefore, in some embodiments, in step 210, only a few high-order bits of the tail of the data to be quantized can be taken as part of the index of the lookup table, thereby reducing the number of index bits, reducing the number of index value entries, thereby reducing the storage space of the lookup table, and at the same time having less impact on the quantization result.

[0081] In some embodiments, the second precision type is a 4-bit floating-point number FP4, and the high-order part of the first mantissa portion of the data to be quantized includes the high-order 3 bits of the mantissa of the data to be quantized.

[0082] Specifically, the following data was obtained through experiments: the tail portion of the data to be quantified. The lower 4 bits The impact is minimal, with a maximum of only 11.7%. Therefore, in some embodiments, only the tail portion of the data to be quantized is considered. The most significant 3 bits are used as part of the index in the lookup table, thus reducing the number of bits in the index. Alternatively, the mantissa of the data to be quantized could also be used. The high-order 4 or 5 bits can be used as part of the index, which is not restricted here. However, considering the storage space of the lookup table and the impact on the results, using the high-order 3 bits will achieve the greatest benefit.

[0083] In step 230, a lookup table is performed based on the index, and a quantization result of the second precision type for the data to be quantized is obtained based on the lookup result, wherein the precision of the second precision type is lower than that of the first precision type.

[0084] For example, the first precision type is 16-bit floating-point number FP16, 32-bit floating-point number FP32, 64-bit floating-point number FP64, etc. For example, if we need to quantize the data to be quantized from the first precision type to the quantized result of the second precision type, such as the second precision type being 4-bit floating-point number FP4, etc.

[0085] A lookup table can consist of an index and quantization results. Directly retrieving quantization results from the lookup table using an index can speed up the quantization process.

[0086] As before, the index can be X_mantissa (the mantissa of the data to be quantized). (The mantissa of the scaling factor) (The difference in the exponential part) constitutes the total.

[0087] Additionally, the mantissa portion of the scaling factor It is usually 3 digits, and the value range is usually, for example... .

[0088] In some embodiments, the index further includes an exponential difference, wherein the exponential difference is the largest exponential portion (i.e., calculated in the preceding preprocessing). ) and the exponent of the data to be quantified (i.e., the exponential part of the data to be quantified by scientific notation). The difference between the exponential and tangential components. Method 200 also includes limiting the range of values ​​for the exponential component difference by using a threshold.

[0089] Specifically, the difference in the exponential portion The value range is limited to [0, threshold], thus reducing the number of possible values ​​and consequently reducing the space occupied by the index. The threshold represents the upper limit of the difference between the exponent of the largest and smallest values ​​in the input data. When this difference exceeds the threshold, even the smallest input data cannot be represented using the target data type. Therefore, cases exceeding the threshold are meaningless, and it is unnecessary to use extra space to represent such meaningless differences. Thus, the exponent portion of the difference... The value range is limited to [0, threshold], which not only narrows the difference in the index, but also reduces the overall difference. The number of value entries is reduced, and quantization accuracy is also guaranteed, because only a portion of the index that has little or no impact on the accuracy of the quantization result is abandoned.

[0090] The choice of threshold has a significant impact on the quantization results (hereinafter referred to as...). (This represents the maximum value of the target data type). If the threshold value is small, the scaling factor value will be large, which may not fully utilize the range of the target data type. In this case, the maximum value of the input data cannot be accurately mapped to... Furthermore, this can lead to small values ​​in the input data being processed as 0, resulting in significant precision loss for small values. If the threshold value is large, the scaling factor is generally small, which may cause the quantization result to exceed the expression range of the target data type. Specifically, large values ​​in the input data may be larger than 0 after quantization. Because these values ​​are outside the range of expression, they will be processed into... This allows the use of the entire value range of the target data type, resulting in higher precision for smaller values ​​in the input data. However, a drawback is that the maximum value in the input will be saturated, which can lead to an incorrect ratio between the maximum and minimum values. Therefore, how to set the threshold value needs to be considered.

[0091] In some embodiments, method 200 further includes setting a threshold such that when the difference in the exponential portion exceeds the threshold, the resulting quantization will exceed the expression range of the second precision type.

[0092] Specifically, for cases where quantization is of FP4 precision type, the threshold can be determined according to... (The maximum value of the mantissa in floating-point representation), the mantissa of the scaling factor. And whether FP4 is in E2M1 or E1M2 format is dynamically adjusted. E2M1 and E1M2 are two different format definitions for 4-bit floating-point numbers (FP4). The core difference lies in the bit allocation of the exponent and mantissa, which affects the numerical range and quantization precision. In E2M1 precision type, the exponent part has a bit width of 2 and the mantissa part has a bit width of 1, while in E1M2, the exponent part has a bit width of 1 and the mantissa part has a bit width of 2.

[0093] In some embodiments, the threshold can be determined as follows:

[0094] For E2M1, if ( * 8) <= (6 * ), threshold = 4; else threshold = 3 (that is, if ( * 8) <= (6 * If the threshold is 4, then set threshold = 4; otherwise, set threshold = 3.

[0095] For E1M2, if ( * 4) <= (3.5 * ), threshold = 3; else threshold = 2 (that is, if ( * 4) <= (3.5 * If the threshold is 3, then set threshold = 3; otherwise, set threshold = 2.

[0096] The reason for setting the above formula is explained below.

[0097] For E2M1, it can represent the following data: {-6.0, -4.0, -3.0, -2.0, -1.5, -1.0, -0.5, 0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0}, where the ratio of its maximum value to its minimum non-zero value is 6 / 0.5 = 12, and this value falls within the range of 8(2). 3 ) and 16 (2 4 The threshold can be between 3 and 4.

[0098] When threshold=3, for different In this case, quantization results may not fully utilize the expressive range of the target data type. For example, when... When the maximum value of mantissa is 2'b1.111 (decimal 1.875), and the maximum value of mantissa is 2'b1.0000000 (decimal 1.0), then... = - 3, then the quantization result of the maximum input value is:

[0099]

[0100] Since the closest value that E2M1 can express to 4.26 is 4, then the maximum value is... When the value is 1.0, it will be quantized to 4 instead of the maximum value of 6 of E2M1. In this case, the value of 6 in the range of E2M1 is wasted.

[0101] When threshold=4, the result of the above example will be 8.533. Since the maximum value of E2M1 is 6, 8.533 will be processed as 6.

[0102] For E1M2, it can represent the following data: {-3.5, -3.0, -2.5, -2.0, -1.5, -1.0, -0.5, 0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5}, where the ratio of its maximum value to its minimum non-zero value is 3.5 / 0.5 = 7. This value falls within the range of 4(2). 2 ) and 8 (2) 3 The threshold can be between 2 and 3.

[0103] When threshold=2, such as When the maximum value of mantissa is 2'b1.111 (decimal 1.875), and the maximum value of mantissa is 2'b1.0000000 (decimal 1.0), then... = -2, then the quantization result of the maximum input value is:

[0104]

[0105] Since the closest value that E2M1 can express to 2.13 is 2, then this maximum value is... When the value is 1.0, it will be quantized to 2 instead of the maximum value of 3.5 of E2M1. In this case, the range {2.5, 3, 3.5} in the E1M2 expression is wasted.

[0106] When threshold=3, the result of the above example will be 4.26. Since the maximum value of E1M2 is 3.5, 4.26 will be processed into 3.5.

[0107] Thus, for E2M1, only 3 bits are needed, with a maximum value of 4, to represent it. – For E1M2, only 2 bits are needed, and the maximum value is 3 to represent it. – This reduces the number of indexes in the lookup table. – This reduces the number of value entries in a portion of the index, thereby reducing the storage space occupied by the index.

[0108] Therefore, in some embodiments, the second precision type is a 4-bit floating-point number FP4, and the difference in the exponent portion of the index is 3 bits for the E2M1 format of the 4-bit floating-point number FP4 or 2 bits for the E1M2 format of the 4-bit floating-point number FP4.

[0109] In some embodiments, the index is 9 bits for the E2M1 format of a 4-bit floating-point number FP4 or 8 bits for the E1M2 format of a 4-bit floating-point number FP4.

[0110] In summary, this reduces the number of bits in the index from 11 bits to 9 bits (for E1M2, It has 3 bits, with X_mantissa being the most significant 3 bits. – (3 bits) or 8 bits (for E1M2, It has 3 bits, with X_mantissa being the most significant 3 bits. – (By using 2 bits), the bit width of the index is reduced, the number of valid index entries is reduced, and thus the storage space occupied by the lookup table is reduced.

[0111] Table 1 below shows the index bit width and index composition, as well as the number of valid entries required for the E2M1 and E1M2 formats for FP4, respectively.

[0112] Table 1

[0113] Format Index bit width Composition Number of valid value entries E2M1 9 bits [[8:6]SF mantissa + [5:3]exp_shift + [2:0]top3_mantissa]] 8*5*8=320 E1M2 8 bits [[7:5] SF mantissa + [4:3] exp_shift + [2:0] top3_mantissa]] 8*4*8=256

[0114] In line E2M1, exp_shift = MIN( - , 4), for E2M1, we get 3 or 4.

[0115] In line E1M2, exp_shift = MIN( - , 3), for E1M2, we get 2 or 3.

[0116] And top3_mantissa = X_mantissa[7:5] only requires 3 high bits.

[0117] Therefore, the effective entries for the index designed for E2M1 are 8*5*8=320, and the effective value entries for the index designed for E1M2 are 8*4*8=256. Thus, compared to the 2048 value entries obtained from an 11-bit index, the simplified index scheme according to at least one embodiment of this disclosure uses approximately 84.375% fewer value entries. The index portion of the lookup table can be constructed from the aforementioned effective value entries, while the quantization result portion consists of, for example, 4 bits of FP4 precision type, namely, 1 sign bit + 2 exponent bits + 1 mantissa bit. In this way, the number of effective value entries for the index is reduced, thereby further reducing the overall storage space of the lookup table and improving lookup efficiency.

[0118] In some embodiments, the index also includes the mantissa of the scaling factor used for data quantization.

[0119] In some embodiments, method 200 further includes obtaining the range of values ​​for the exponent of the scaling factor by using a threshold of the difference in the exponent portion.

[0120] For example, the range of values ​​for the scaling factor's exponent can be obtained by using the threshold of the aforementioned difference in the exponent portion. = – threshold, which also gives the number of bits needed to represent the exponent of the scaling factor, and then further selects the optimal scaling factor SF from the available values ​​of the number of bits of the scaling factor SF and / or the range of values ​​mentioned above, and obtains the optimal quantization result.

[0121] In some embodiments, the second precision type is a 4-bit floating-point number FP4, and the mantissa of the scaling factor is 3 bits for the 4-bit floating-point number FP4.

[0122] In some embodiments, step 230 further includes determining the optimal scaling factor and quantization result by: obtaining multiple different candidate scaling factors based on the number of bits and / or the range of values ​​of the exponent of the scaling factor; obtaining multiple candidate quantization results by using the mantissa of each of the multiple different candidate scaling factors and a lookup table; calculating multiple differences between the multiple dequantized results of the multiple candidate quantization results after dequantization with the candidate scaling factors and the data to be quantized; determining the candidate scaling factor with the smallest difference as the optimal scaling factor; and using the candidate quantization result corresponding to the optimal scaling factor as the second precision type quantization result of the data to be quantized.

[0123] For example, if the exponent of the scaling factor is set to 3 digits, then there are 2 possible values. 3 =8, based on the mantissa of the scaling factor (e.g., determined to be 2), for example, 8 different candidate scaling factors SF can be obtained. i : SF0, SF1, SF2, SF3, SF4, SF5, SF6, SF 7。 Alternatively, the range of values ​​for the scaling factor's exponent can be obtained, for example, by using a threshold of the aforementioned difference in the exponent portion, to determine, for example, eight different values. This allows for the selection of such candidate scaling factors SF. i Candidate quantization result q obtained by table lookup i Then, after dequantizing the candidate quantization result (for example, by using the candidate quantization result q...), i Multiply by the candidate scaling factor SF i The scaling factor is selected by comparing the inverse quantization result of the input data (X0 to X16) with the data to be quantized. The closest candidate scaling factor is then chosen as the optimal scaling factor. Here, X can represent, for example, the 16 BF16 elements of the input (X0 to X16). 15 One of the elements in ).

[0124] Specifically, the difference between the candidate quantization result q0 obtained by looking up the candidate scaling factor SF0 after dequantization (i.e., q0 * SF0) and the data to be quantized can be calculated, i.e., |q0 * SF0 - X|.

[0125] The difference between the candidate quantization result X1 obtained by looking up the candidate scaling factor SF1 after dequantization (i.e., q1* SF1) and the data to be quantized can be calculated as |q1* SF1-X|.

[0126] The difference between the candidate quantization result q2 obtained by looking up the candidate scaling factor SF2 after dequantization (i.e., q2 * SF2) and the data to be quantized can be calculated as |q2 * SF2 - X|.

[0127] The difference between the candidate quantization result q3 obtained by looking up the candidate scaling factor SF3 after dequantization (i.e., q3 * SF3) and the data to be quantized can be calculated as |q3 * SF3 - X|.

[0128] The difference between the candidate quantization result q4 obtained by looking up the candidate scaling factor SF4 after dequantization (i.e., q4 * SF4) and the data to be quantized can be calculated as |q4 * SF4 - X|.

[0129] The difference between the candidate quantization result q5 obtained by looking up the candidate scaling factor SF5 after dequantization (i.e., q5 * SF5) and the data to be quantized can be calculated as |q5 * SF5 - X|.

[0130] The difference between the candidate quantization result q6 obtained by looking up the candidate scaling factor SF6 after dequantization (i.e., q6 * SF6) and the data to be quantized can be calculated as |q6 * SF6 - X|.

[0131] The difference between the candidate quantization result q7 obtained by looking up the candidate scaling factor SF7 after dequantization (i.e., q7 * SF7) and the data to be quantized can be calculated as |q7 * SF7 - X|.

[0132] Alternatively, the above calculation of the difference can be modified to calculate the normalized mean-square error (NMSE). Other methods for calculating the difference between two values ​​can also be applied, such as mean square error, the square of the difference, etc.

[0133] By comparing the above multiple differences, the candidate scaling factor corresponding to the smallest difference is obtained as the optimal scaling factor. At the same time, the candidate quantization result corresponding to the optimal scaling factor is obtained as the quantization result of the second precision type of the data to be quantized. For example, if |q7*SF7-X| is determined to be the smallest, then SF7 is determined as the optimal scaling factor, and q7 is the candidate quantization result corresponding to the optimal scaling factor SF7 as the quantization result of the target precision type.

[0134] However, since the above index is simplified and its precision is reduced, it may have errors with the true value. Therefore, the above candidate quantization results obtained by looking up the table through the index constructed by the candidate scaling factor may have occasional jumps in the floating-point domain, deviating from the true quantization results.

[0135] Therefore, to compensate for the precision of the index and prevent occasional jumps in the floating-point domain of the found quantization result, after the simplified index constructed using the aforementioned candidate scaling factors finds the quantization result in the lookup table, one or more alternative quantization results are found near the original quantization result. For example, one or more alternative quantization results can be obtained by adding / subtracting one or more different positive integers from the encoding space of the original quantization result. Then, by comparing the deviation between the original quantization result after dequantization and the data to be quantized, and the deviation between the one or more alternative quantization results after dequantization and the data to be quantized, the quantization result with the smallest deviation is selected as the optimal quantization result, thus determining the optimal quantization result for the candidate scaling factor SF. i Candidate quantization results q i Here, the encoding space of the quantization result is a finite discrete set of all possible codewords after quantization. Adding or subtracting one or more different positive integers to the encoding space of the quantization result will yield several quantization results that are close to the original quantization result. In this way, a quantization result that is closer to the true quantization result can be obtained.

[0136] Specifically, in some embodiments, multiple candidate quantization results are obtained by using the mantissa of each of the multiple different candidate scaling factors and a lookup table, including the following steps: using the mantissa of each of the multiple different candidate scaling factors to find multiple first quantization results in the lookup table; adding or subtracting one or more positive integers to the encoding space of each first quantization result to obtain one or more first candidate quantization results; calculating the deviation between the result of each first quantization result after dequantization with the candidate scaling factor and the data to be quantized, and calculating one or more deviations between the result of one or more first candidate quantization results after dequantization with the candidate scaling factor and the data to be quantized; determining the quantization result with the smallest deviation as the candidate quantization result for the candidate scaling factor.

[0137] In other words, for the i-th candidate scaling factor SF i It can generate a total of one original quantization result and multiple alternative quantization results: Original quantization result With multiple alternative quantification results or (Add / subtract N from the encoding space). Typically, N can be 1, 2, or 3, allowing for multiple alternative quantization results near the encoding space of the original quantization result. i is used to distinguish different candidate scaling factors.

[0138] Then, for each candidate scaling factor SF i Compare the original quantization result with one or more alternative quantization results in dequantization (i.e., using the original quantization result). or one or more alternative quantification results Multiply by the candidate scaling factor SF i (After) and the data to be quantified The deviation is used to select the quantization result with the smallest deviation as the candidate scaling factor SF. i Candidate quantization results q i This is because the smallest deviation indicates that the error between the quantized result and the data to be quantized is the smallest. Therefore, selecting the quantized result with the smallest deviation as the optimal quantization result for that scaling factor can reduce quantization error and improve the accuracy of neural networks or deep learning results calculated using the quantized result. This, in turn, helps to improve the accuracy of results in practical applications such as image recognition.

[0139] In other words, by comparing the deviation obtained from the original quantization results With one or more alternative quantification results One or more deviations obtained The quantization result with the smallest deviation is selected as the candidate scaling factor SF. i The optimal quantization result q i Here, the calculation of deviation can also be modified to calculate the normalized mean-square error (NMSE). Other methods for calculating the deviation between two values ​​can also be applied, such as mean square error, the square of the difference, etc., which will not be elaborated here.

[0140] In one embodiment, the encoding space of the original quantization result can be reduced by 1, i.e., the original quantization result can be selected. With an alternative quantification result As two candidate values, then if Select the original quantization result As the optimal quantification result, if Select alternative quantification results As for the candidate scaling factor SF i The optimal quantization result q i .

[0141] Thus, by using it on a certain candidate scaling factor SF i After the simplified index finds the quantization result in the lookup table, it retrieves one or more candidate quantization results near its encoding space, and selects the quantization result with the smallest quantization error as the candidate scaling factor SF. i The optimal quantization result q i This can reduce the error in the quantification results and maintain their accuracy.

[0142] Then, based on these candidate scaling factors SF i The optimal quantization result q i Following the above method, from these candidate scaling factors SFi Choose the optimal scaling factor.

[0143] Figure 3 A complete quantization process example according to at least one embodiment of the present disclosure is shown, including a process 300 for simplifying the lookup table, compensating for the precision of the index, and selecting the optimal scaling factor.

[0144] In step 310, data input and preprocessing are performed.

[0145] The input consists of 16 BF16 elements (X0 to X...) 15 ) data blocks.

[0146] Calculate the maximum absolute value , where j = 0, 1, 2, ..., 15.

[0147] Will Decomposed into the largest exponential part and the largest last digit : .

[0148] In step 320, candidate scaling factors are generated.

[0149] The mantissa of the scaling factor If it is 3 bits, then 2 can be generated. 3 =8 candidates .

[0150] For each Determine the threshold and the exponential part of the scaling factor as follows: By setting a threshold so that when the difference in the exponential part exceeds the threshold, the resulting quantization will exceed the range of the second precision type.

[0151] The threshold can be determined as follows:

[0152] For E2M1, if ( * 8) <= (6 * If the threshold is 4, then set threshold = 4; otherwise, set threshold = 3.

[0153] For E1M2, if ( * 4) <= (3.5 * If the threshold is 3, then set threshold = 3; otherwise, set threshold = 2.

[0154] The exponential part of the scaling factor = – threshold.

[0155] Then, make the scaling factor SF = ( << 3) | (i.e., for) Perform a left shift operation of 3 bits, and then shift the result by 3 bits. and Perform a bitwise OR operation. Assuming... The value is 2 (00000010 in binary). The value is 5 (00000101 in binary). For Perform a left shift operation of 3 bits: 00000010 << 3 = 00010000 (i.e., 16 in decimal). The left-shifted... and Perform a bitwise OR operation: 00010000 | 00000101 = 00010101 (i.e., 21 in decimal). Therefore, the final scaling factor SF value is 21.

[0156] In step 330, the index of the lookup table is constructed.

[0157] For E2M1, construct a 9-bit composite index index[8:0]:

[0158] index[8:6] = (3-digit scaling factor mantissa)

[0159] index[5:3] = MIN( – , 4)

[0160] index[2:0] = X_mantissa[6:4].

[0161] For E1M2, construct an 8-bit composite index index[7:0]:

[0162] index[7:5] = (3-digit scaling factor mantissa)

[0163] index[4:3] = MIN( – , 3)

[0164] index[2:0] = X_mantissa[6:4].

[0165] In step 340, for each The obtained index is used to find each quantization result for error optimization and selection of search results.

[0166] Through each The obtained index is used to directly look up the lookup table and obtain each... Candidate values ​​for at least two quantization results: and (Minus 1 in the encoding field), comparison and Choose the result with the smallest error As the The optimal search result for the corresponding scaling factor. Thus, each... The corresponding scaling factor lookup quantization result.

[0167] In step 350, the optimal scaling factor is selected.

[0168] calculate The corresponding deviation, such as mean square error (MSE)

[0169]

[0170] Compare the MSE values ​​corresponding to the eight candidate scaling factors.

[0171] Select the candidate scaling factor with the smallest MSE. and the corresponding search quantification results (The optimal search result obtained in step 340) is used as the final quantification result.

[0172] Thus, according to at least one embodiment of this disclosure, hardware storage space is saved by using fewer index structures, table lookup efficiency is improved, and quantization is performed efficiently. At the same time, the lookup table index bit width can be adjusted according to accuracy requirements, further reducing the error of the quantization result and maintaining the accuracy of the quantization result.

[0173] Figure 4 A block diagram of a lookup table-based data quantization apparatus 400 according to at least one embodiment of the present disclosure is shown.

[0174] The data quantization device 400 includes a receiving module 410, an obtaining module 420, and a searching module 430.

[0175] The receiving module 410 is configured to receive data to be quantized, wherein the data to be quantized is of the first precision type.

[0176] The acquisition module 420 is configured to obtain an index of the data to be quantized for a lookup table based on the first exponent portion and the first mantissa portion of the data to be quantized, wherein the index includes the high-order portion of the first mantissa portion of the data to be quantized.

[0177] The lookup module 430 is configured to perform a lookup in the lookup table based on the index and obtain a second-precision type quantization result for the data to be quantized based on the lookup result, wherein the precision of the second-precision type is lower than that of the first-precision type.

[0178] In some embodiments, the second precision type is a 4-bit floating-point number FP4, and the high-order part of the first mantissa portion of the data to be quantized includes the high-order 3 bits of the mantissa of the data to be quantized.

[0179] In some embodiments, the data quantization apparatus 400 further includes a decomposition module (not shown) configured to: in response to the data to be quantized being a single input data, treat the data to be quantized as the maximum value; or, in response to the data to be quantized being one of a plurality of combined input data comprising a data block, obtain the maximum value among the absolute values ​​of each input data in the plurality of combined input data; and decompose the maximum value into a maximum exponent portion and a maximum mantissa portion.

[0180] In some embodiments, the index further includes an exponential difference, wherein the exponential difference is the difference between the maximum exponential portion and the exponent of the data to be quantized, wherein the device 400 further includes a limiting module configured to limit the range of values ​​of the exponential difference by means of a threshold.

[0181] In some embodiments, the apparatus 400 further includes a setting module configured to set a threshold such that when the difference in the exponential portion exceeds the threshold, the resulting quantization will exceed the expression range of the second precision type.

[0182] In some embodiments, the second precision type is a 4-bit floating-point number FP4, and the difference in the exponent portion of the index is 3 bits for the E2M1 format of the 4-bit floating-point number FP4 or 2 bits for the E1M2 format of the 4-bit floating-point number FP4.

[0183] In some embodiments, the index is 9 bits for E2M1 format or 8 bits for E1M2 format.

[0184] In some embodiments, the index further includes the mantissa of a scaling factor for data quantization, and the apparatus further includes a obtaining module configured to obtain a range of values ​​for the exponent of the scaling factor by means of a threshold of the difference in the exponent portion.

[0185] In some embodiments, the second precision type is a 4-bit floating-point number FP4, and the mantissa of the scaling factor is 3 bits for the 4-bit floating-point number FP4.

[0186] In some embodiments, the lookup module 430 is configured to: obtain multiple different candidate scaling factors based on the number of bits and / or the range of values ​​of the exponent of the scaling factor; obtain multiple candidate quantization results using the mantissa of each of the multiple different candidate scaling factors and a lookup table; calculate multiple differences between the multiple dequantized results of the multiple candidate quantization results after dequantization with the candidate scaling factors and the data to be quantized; determine the candidate scaling factor with the smallest difference as the optimal scaling factor; and use the candidate quantization result corresponding to the optimal scaling factor as the second precision type quantization result of the data to be quantized.

[0187] In some embodiments, the lookup module 430 is configured to obtain multiple candidate quantization results by using the mantissa of each of the multiple different candidate scaling factors and a lookup table through the following steps: obtaining multiple candidate quantization results by using the mantissa of each of the multiple different candidate scaling factors and a lookup table includes: finding multiple first quantization results in the lookup table using the mantissa of each of the multiple different candidate scaling factors; adding or subtracting one or more positive integers to the encoding space of each first quantization result to obtain one or more first candidate quantization results; calculating the deviation between the result of each first quantization result after dequantization with the candidate scaling factor and the data to be quantized, and one or more deviations between the result of one or more first candidate quantization results after dequantization with the candidate scaling factor and the data to be quantized; and determining the quantization result with the smallest deviation as the candidate quantization result for the candidate scaling factor.

[0188] Figure 5 A block diagram of a lookup table-based data quantization apparatus 500 according to at least one embodiment of the present disclosure is shown.

[0189] The apparatus 500 may include a processor 510 and a memory 520, the memory 520 being coupled to the processor 510 and storing computer instructions therein for performing steps of various methods of at least one embodiment of the present disclosure when executed by the processor 510.

[0190] The processor 510 may include, but is not limited to, one or more processors or microprocessors.

[0191] The memory 520 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (such as hard disks, floppy disks, solid-state drives, removable disks, CD-ROMs, DVD-ROMs, Blu-ray discs, etc.).

[0192] Figure 6 A block diagram of a processor 600 according to at least one embodiment of the present disclosure is shown.

[0193] The processor 600 includes at least one computing unit 610. The computing unit 610 includes, for example: Figure 4 Alternatively, the data quantization device 400 or 500 based on a lookup table, as shown in Figure 5, and the lookup table. The calculation unit has already been described above and will not be repeated here.

[0194] It should be noted that in at least one embodiment of this disclosure, device 400 or 500 or processor 600 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited and can be constructed from analog devices, digital chips, or other suitable methods according to circuit principles.

[0195] For example, device 400 or 500 or processor 600 may be implemented in hardware, software or a combination of hardware and software, and this disclosure does not impose any specific limitations on this.

[0196] The apparatus 400 or 500 or processor 600 provided in at least one embodiment of this disclosure can achieve similar technical effects to the methods described above, and will not be repeated here.

[0197] Figure 7 A schematic block diagram of an electronic device according to at least one embodiment of the present disclosure is shown.

[0198] like Figure 7 As shown, the electronic device 700 is, for example, suitable for implementing the methods provided in the embodiments of this disclosure. It should be noted that... Figure 7 The components of the electronic device 700 shown are merely exemplary and not limiting. The electronic device 700 may have other components depending on the actual application requirements.

[0199] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to non-transitory computer-readable instructions stored in memory to achieve various functions.

[0200] For example, when the computer-readable instructions are executed by the processing device 701, one or more steps of the method according to any of the above embodiments can be performed. It should be noted that a detailed description of the method's processing procedure can be found in the relevant descriptions in the embodiments of the above methods.

[0201] For example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 703 and / or cache memory, etc., whereby computer-readable instructions can be loaded from storage device 708 into RAM 703 to execute. Non-volatile memory may include, for example, read-only memory (ROM) 702, hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. Various applications and various data, such as style images, and various data used and / or generated by the applications, may also be stored in the computer-readable storage medium.

[0202] For example, the processing device 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0203] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, flash memory, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown, and the electronic device 700 may alternatively implement or possess more or fewer devices. For example, a processing device 701 can control other components in the electronic device 700 to perform desired functions. The processing device 701 may be a device with data processing capabilities and / or program execution capabilities, such as a central processing unit (CPU), a tensor processor (TPU), or a graphics processing unit (GPU). The central processing unit (CPU) may be an x86, ARM, RISC-V architecture, etc. The GPU may be directly integrated into the SOC, directly integrated onto the motherboard, or built into the northbridge chip of the motherboard.

[0204] In one embodiment, the at least one computer instruction may also be compiled into or comprise a computer program product or software product, wherein one or more computer instructions, when executed by a processor, perform the steps of the various functions or methods in the embodiments described herein.

[0205] Non-transitory computer-readable storage media are also disclosed according to at least one embodiment of the present disclosure.

[0206] Instructions, such as computer instructions, are stored on a non-transitory computer-readable storage medium. When the computer instructions are executed by a processor, the various methods described above can be performed. Non-transitory computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disks, floppy disks, solid-state drives, removable disks, CD-ROMs, DVD-ROMs, Blu-ray discs, etc.). For example, a non-transitory computer-readable storage medium can be connected to a computing device such as a computer, and then, when the computing device executes the computer instructions stored on the computer-readable storage medium, the various methods described above can be performed.

[0207] According to at least one embodiment of this disclosure, a computer program product is also disclosed, which can perform the methods, steps, and operations given herein. For example, such a computer program product may be a computer software package, computer code instructions, or a computer-readable tangible medium having computer instructions tangibly stored (or encoded) thereon, which can be executed by a processor to perform the operations described herein. The computer program product may include packaging materials.

[0208] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The term “such as / for example” as used herein refers to the phrase “such as / for example but not limited to,” and is used interchangeably with it.

[0209] The flowcharts and method descriptions in this disclosure are merely illustrative examples and are not intended to require or imply that the steps of the various embodiments must be performed in the given order. As those skilled in the art will recognize, the steps in the above embodiments can be performed in any order. Words such as "then," "next," etc., are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods. Furthermore, any reference to a singular element, such as the use of the articles "a," "one," or "the," is not to be construed as limiting that element to the singular.

[0210] Furthermore, the steps and apparatus in the various embodiments herein are not limited to any one embodiment. In fact, new embodiments can be conceived by combining relevant steps and apparatus in the various embodiments herein based on the concepts of this disclosure, and these new embodiments are also included within the scope of this disclosure.

[0211] The above methods can be implemented in hardware, software, firmware, or any combination thereof.

[0212] Furthermore, modules or other suitable means for carrying out the methods and techniques described herein can be downloaded from a server wirelessly when appropriate. Alternatively, the various methods described herein can be provided via a storage component so that the various methods are available when coupled to the storage component. Additionally, any other suitable techniques for providing the methods and techniques described herein to the device can be utilized.

[0213] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit at least one embodiment of the present disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method of data quantization based on a lookup table, characterized by, The method comprises: receiving data to be quantized, wherein the data to be quantized is of a first precision type; obtaining an index of the data to be quantized in the lookup table based on a first exponent part and a first mantissa part of the data to be quantized, wherein the index comprises a high bit part of the first mantissa part of the data to be quantized; performing a lookup in the lookup table based on the index, and obtaining a quantization result of a second precision type for the data to be quantized according to a lookup result, wherein the second precision type has a lower precision than the first precision type, and the high bit part of the first mantissa part of the data to be quantized comprises 3, 4 or 5 high bits of a mantissa of the data to be quantized, The method further comprises: in response to the data to be quantized being a single input data, taking the data to be quantized as a maximum value; or, in response to the data to be quantized being one of a plurality of combined input data included in a data block, obtaining a maximum value in absolute values of each input data in the plurality of combined input data; decomposing the maximum value into a maximum exponent part and a maximum mantissa part, The index further comprises an exponent part difference value, wherein the exponent part difference value is a difference between the maximum exponent part and an exponent of the data to be quantized.

2. The data quantization method of claim 1, wherein, The second precision type is a 4-bit floating point number FP4.

3. The data quantization method of claim 1, wherein, The method further comprises: limiting a value range of the exponent part difference value by a threshold value.

4. The data quantization method of claim 3, wherein, The method further comprises: setting the threshold value to a value such that, when the exponent part difference value exceeds the threshold value, the obtained quantization result will be out of an expression range of the second precision type.

5. The data quantization method of claim 3, wherein, The second precision type is a 4-bit floating point number FP4, The exponent part difference value in the index is 3 bits for an E2M1 format of the 4-bit floating point number FP4 or 2 bits for an E1M2 format of the 4-bit floating point number FP4.

6. The data quantization method of claim 5, wherein, The index is 9 bits for the E2M1 format or 8 bits for the E1M2 format.

7. The data quantization method of claim 3, wherein, The index further comprises a mantissa of a scaling factor for data quantization, The method further comprises: obtaining a value range of an exponent of the scaling factor by a threshold value of the exponent part difference value.

8. The data quantization method of claim 7, wherein, The second precision type is a 4-bit floating point number FP4, and the mantissa of the scaling factor is 3 bits for the 4-bit floating point number FP4.

9. The data quantization method of claim 7, wherein, The obtaining of the quantization result of the second precision type for the data to be quantized based on the index in the lookup table according to the lookup result comprises: determining an optimal scaling factor by: obtaining a plurality of different candidate scaling factors according to a bit number and / or the value range of the exponent of the scaling factor; obtaining a plurality of candidate quantization results by using the mantissa of each candidate scaling factor in the plurality of different candidate scaling factors and the lookup table respectively; calculating a plurality of difference distances between the plurality of candidate quantization results and the data to be quantized after being dequantized by the candidate scaling factors respectively; determining a candidate scaling factor with the smallest difference distance as the optimal scaling factor; taking a candidate quantization result corresponding to the optimal scaling factor as the quantization result of the second precision type for the data to be quantized.

10. The data quantization method of claim 9, wherein, The obtaining the multiple candidate quantization results respectively by using the mantissa of each candidate scaling factor in the multiple different candidate scaling factors and the lookup table comprises the following steps: Finding multiple first quantization results in the lookup table by using the mantissa of each candidate scaling factor in the multiple different candidate scaling factors; Adding or subtracting one or more positive integers to each first quantization result to obtain one or more first alternative quantization results; Calculating the deviation of the result of each first quantization result after being inverse quantized by the candidate scaling factor from the to-be-quantized data and one or more deviations of the result of the one or more first alternative quantization results after being inverse quantized by the candidate scaling factor from the to-be-quantized data; Determining the quantization result with the minimum deviation as the candidate quantization result for the candidate scaling factor.

11. A data quantization apparatus based on a lookup table, characterized by, The device comprises: A receiving module configured to receive to-be-quantized data, wherein the to-be-quantized data is of a first precision type; An obtaining module configured to obtain an index of the to-be-quantized data for the lookup table based on a first exponent part and a first mantissa part of the to-be-quantized data, wherein the index comprises a high-bit part of the first mantissa part of the to-be-quantized data; A lookup module configured to perform lookup in the lookup table based on the index and obtain a quantization result of a second precision type for the to-be-quantized data according to the lookup result, wherein the second precision type has lower precision than the first precision type, and the high-bit part of the first mantissa part of the to-be-quantized data comprises 3 bits, 4 bits or 5 bits of high bits of the mantissa of the to-be-quantized data, The device further comprises: A decomposition module configured to, in response to the to-be-quantized data being a single input data, decompose the to-be-quantized data as a maximum value; or, in response to the to-be-quantized data being one of multiple combined input data included in a data block, obtain a maximum value in absolute values of each input data in the multiple combined input data; Decompose the maximum value into a maximum exponent part and a maximum mantissa part, The index further comprises an exponent part difference value, wherein the exponent part difference value is a difference value between the maximum exponent part and the exponent of the to-be-quantized data.

12. A data quantization apparatus based on a lookup table, characterized by, The device comprises: At least one memory storing computer instructions; At least one processor configured to execute the computer instructions in the at least one memory to perform the method according to any one of claims 1-10.

13. A processor, comprising: The processor comprises at least one computing unit, wherein the computing unit comprises the lookup table-based data quantization device and the lookup table according to claim 11 or 12.

14. A non-transitory computer-readable storage medium, comprising: Having computer instructions stored thereon, When the computer instructions are executed by at least one processor, the at least one processor executes the method according to any one of claims 1-10.

Images