Processing apparatus and quantization method for quantization and inverse quantization of numeric data

Abstract

The invention relates to a processing device and a quantization method for quantization and inverse quantization of numeric data. In one or more aspects, a processing apparatus for numeric data quantization includes processing circuitry to determine a maximum exponent from a set of exponents of a set of digit representations of a set of digits, obtain a set of scaled exponents based on the maximum exponent, and quantize the scaled exponents based on the set of scaled exponents. And one of (i) obtaining a set of quantized significant numbers based on the set of digit representations and a set of mantissas of the set of scaled exponents, or (ii) obtaining a set of quantized mantissas based on the set of mantissas. The processing circuitry is configured to output a set of quantized digit representations of the set of digits based on the set of quantized significant numbers or based on the set of quantized mantissas and the set of scaled exponents; and outputting an offset exponential scaling factor based on the maximum exponent.

Description

TECHNICAL FIELD

[0001] One embodiment of the present disclosure relates to a processing device and quantization method, and particularly, to a processing device and quantization method for quantizing or dequantizing numerical data. BACKGROUND

[0002] Recent developments in the field of electronic devices and systems include a need for increased computing power and capacity in order to process complex computing tasks, such as training machine learning models and / or inference tasks based on machine learning models. In some applications of neural network-based machine learning models, startup data and / or weight data for particular layers of the neural network are received and / or output as floating point data. As the volume of startup data and the computational complexity (e.g., the number of layers and the number of nodes at each layer of the neural network, as well as associated weight data) increases, the size and complexity of the corresponding processing devices, including processing circuitry and memory, also increases. The cost for manufacturing such processing devices, the time needed for transferring data between processing circuitry and memory, the power consumption for operating the processing devices to perform the corresponding computations, and the time needed for completing the computations also increase. SUMMARY

[0003] One embodiment of the present disclosure provides a processing device for quantizing numerical data, comprising a memory and processing circuitry. The processing circuitry is coupled with the memory and the processing circuitry is configured to determine a maximum exponent from a set of exponents of a set of digit representations of a set of numbers. A set of scaled exponents is obtained based on subtracting the maximum exponent from each of the set of exponents. One of the following operations is performed. A set of quantized significands is obtained based on a set of digit representations of the set of numbers and the set of scaled exponents, or a set of quantized mantissas is obtained based on the set of digit representations; a set of quantized digit representations of the set of numbers is output to the memory based on the set of quantized significands or based on the set of quantized mantissas and the set of scaled exponents; and an exponent scaling factor of an offset associated with the set of quantized digit representations is output to the memory based on the maximum exponent.

[0004] An embodiment of the disclosure provides a processing device for inverse quantization of numeric data, including a memory and a processing circuit. The processing circuit is coupled with the memory and the processing circuit is configured to perform the following operations. Extract a mantissa of an inverse quantized digital representation of numeric data, the mantissa being a result of processing a first set of quantized digital representations of a first set of numbers and a second set of quantized digital representations of a second set of numbers. Extract an exponent adjustment from the numeric data. Obtain a combined exponent scaling factor based on a first exponent scaling factor associated with the first set of quantized digital representations and a second exponent scaling factor associated with the second set of quantized digital representations. Obtain an unbiased exponent of the numeric data based on the combined exponent scaling factor and the exponent adjustment. Output the inverse quantized digital representation of the numeric data to the memory, the inverse quantized digital representation including the mantissa of the inverse quantized digital representation and an exponent of the inverse quantized digital representation based on the unbiased exponent of the numeric data.

[0005] An embodiment of the disclosure provides a method for quantization of numeric data, including the following steps. Determine a maximum exponent from a set of exponents of a set of digital representations of a set of numbers. Obtain, by a processing circuit, a set of scaled exponents based on subtracting the maximum exponent from each of the set of exponents. Perform, by the processing circuit, one of the following operations. Obtain a set of quantized significands based on a set of mantissas of the set of digital representations and the set of scaled exponents, or obtain a set of quantized mantissas based on the set of mantissas; output a set of quantized digital representations of the set of numbers based on the set of quantized significands or based on the set of quantized mantissas and the set of scaled exponents; and output an offset exponent scaling factor associated with the set of quantized digital representations based on the maximum exponent. BRIEF DESCRIPTION OF DRAWINGS

[0006] Aspects of one or more embodiments of the disclosure are set forth with reference to the accompanying drawings that form a part of this disclosure. Variations in Figure One Best mode for carrying out the disclosure is thus disclosed. It is to be understood that the drawings are solely for purposes of illustrating the concepts of the disclosure and that the presented concepts can be implemented in any manner and with any number of variations not necessarily directly

[0007] Figure 1 A block diagram of a processing device according to some embodiments;

[0008] Figure 2A A block diagram of a micro scaled quantizer according to some embodiments

[0009] Figure 2B A block diagram of a micro scaled inverse quantizer according to some embodiments;

[0010] Figure 3A A processing flow diagram of an example of a processing flow for quantization of numeric data according to some embodiments;

[0011] Figure 3BA process flow diagram for a numerical data dequantization process instance according to some embodiments;

[0012] Figure 4A A process flow diagram for another numerical data quantization process instance according to some embodiments;

[0013] Figure 4B A process flow diagram for another numerical data dequantization process instance according to some embodiments;

[0014] Figure 5 A process flow diagram for a method of numerical data quantization according to some embodiments;

[0015] Figure 6 A process flow diagram for a method of numerical data dequantization according to some embodiments;

[0016] Figure 7 A block diagram of a computing device usable in connection with one or more embodiments according to some embodiments.

[0017] [SYMBOL DESCRIPTION]

[0018] 100: processing device

[0019] 110: processing circuitry

[0020] 120: microscale scaling quantizer / MX quantizer

[0021] 122: data

[0022] 124: first portion

[0023] 126: second portion

[0024] 130: microscale scaling correlation processor / MX correlation processor

[0025] 132: numerical data

[0026] 140: microscale scaling dequantizer / MX dequantizer

[0027] 142: dequantized digital representation

[0028] 150: post-processor

[0029] 152: post-processed data

[0030] 160: memory

[0031] 162: weight data

[0032] 164: enable data

[0033] 166: output data

[0034] 200A: microscale quantizer

[0035] 200B: microscale dequantizer

[0036] 210: maximum finder

[0037] 212: maximum exponent

[0038] 220: subtracter

[0039] 222: scaled exponent

[0040] 230: significand generator

[0041] 232: output data

[0042] 240: data format converter

[0043] 250: hidden bit finder

[0044] 252: information

[0045] 260: mantissa extractor

[0046] 262: mantissa

[0047] 270: exponent adjustment extractor

[0048] 272: exponent adjustment

[0049] 282: first adder

[0050] 284: combined exponent scale factor

[0051] 286: second adder

[0052] 288: unshifted exponent

[0053] 290: data format converter

[0054] 300A: processing flow diagram

[0055] 300B: processing flow diagram

[0056] 302: input data

[0057] 304: quantized digit representation

[0058] 306: associated offset exponent scale factor

[0059] 310: microscale quantizer

[0060] 312: stage

[0061] 314: stage

[0062] 316: stage

[0063] 318: stage

[0064] 322: numeric data

[0065] 326: dequantized numeric representation

[0066] 330: microscaled dequantizer

[0067] 332: stage

[0068] 334: stage

[0069] 336: stage

[0070] 342: stage

[0071] 344: stage

[0072] 346: stage

[0073] 400A: process flow diagram

[0074] 400B: process flow diagram

[0075] 402: input data

[0076] 404: quantized numeric representation

[0077] 406: offset exponent scale factor

[0078] 410: microscaled quantizer

[0079] 412: stage

[0080] 414: stage

[0081] 416: stage

[0082] 418: stage

[0083] 422: numeric data

[0084] 426: dequantized numeric representation

[0085] 430: microscaled dequantizer

[0086] 432: stage

[0087] 434: stage

[0088] 436: stage

[0089] 438: smoking cessation

[0090] 442: stage

[0091] 444: stage

[0092] 446: phase

[0093] 500: method of quantizing numeric data

[0094] 510: block

[0095] 520: block

[0096] 530: block

[0097] 540: block

[0098] 550: block

[0099] 600: method of dequantizing numeric data

[0100] 610: block

[0101] 620: block

[0102] 630: block

[0103] 640: block

[0104] 650: block

[0105] 700: computing device instance

[0106] 702: hardware processor

[0107] 704: memory

[0108] 706: set of executable instructions

[0109] 708: bus

[0110] 710: I / O interface

[0111] 712: network interface

[0112] 714: network

[0113] 742: user interface (UI) instructions DETAILED DESCRIPTION

[0114] The following disclosure provides many different embodiments, or examples, for implementing different characteristics of the provided subject matter. The following description describes specific examples of components and configurations to simplify the present disclosure. Of course, these components and configurations are merely examples and are not intended to be limiting. For example, in the following description, the formation of a first feature over or on a second feature can include embodiments in which the first and second features are formed directly on each other, and can also include embodiments in which additional features can be formed between the first and second features such that the first and second features are not directly on each other. Furthermore, the present disclosure can repeat reference numerals and / or letters in various examples and / or throughout the specification. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and / or configurations discussed.

[0115] In addition, spatially relative terms, such as "beneath", "below", "lower", "above", "upper", and the like, can be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. The spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientations depicted in the figures. The devices can be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. Moreover, the term "about" can mean "including" or "comprising." In the present disclosure, the phrase "one or more of' a list of items suggests that at least one of the items can be present and one or more of the items can be present. For example, the phrase "one or more of: A, B, and C" means that only A can be present, or only B can be present, or only C can be present, or A and B can be present, or A and C can be present, or B and C can be present, or A, B, and C can be present.

[0116] In some applications, a set of digits in one format of a longer bit length is to be converted first to another format of a shorter bit length in order to improve processing efficiency without significantly sacrificing processing accuracy. For example, a set of numeric data in which each number is in a 16-bit floating point format or a 32-bit floating point format can be converted to a set of quantized representations in which each number is in a microscaled format (e.g., with a bit length of 8 bits, 6 bits, or even 4 bits). In some embodiments, the conversion from the longer bit length format to the shorter bit length format is referred to as a numeric data quantization procedure; and the conversion from the shorter bit length format to the longer bit length format is referred to as a numeric data dequantization procedure.

[0117] In some applications, numerical data quantization and dequantization are performed by a host controller of a computing device, such as a central processing unit (CPU), a graphic processing unit (GPU), a tensor processing unit (TPU), or the like. In some embodiments, such quantization and / or dequantization processing involves repeated use of data for quantization and / or dequantization, which can result in substantial energy consumption and latency. Moreover, the involved operations in quantization and / or dequantization are nominally complex (e.g., logarithm, exponential computation, and / or division operation), and thus are expensive in terms of energy.

[0118] In some embodiments, a numerical data quantization procedure based on an embodiment of the present disclosure avoids performing logarithm and / or exponential computation, replaces multiplication operation in linear space with shift and / or addition operation in exponential space, and replaces division operation in linear space with subtraction operation in exponential space. As a result, computational complexity and conversion speed are improved. In some embodiments, a numerical data dequantization procedure based on an embodiment of the present disclosure also avoids performing logarithm and / or exponential computation, and provides a convenient method to convert numerical data from exponential space back to linear space. As a result, thanks to the use of the micro-scaling format as discussed above, the result can still be obtained in linear space without adversely increasing computational complexity and conversion cost.

[0119] Figure 1 A block diagram of a processing device 100 according to some embodiments. Figure 1 The processing device 100 in FIG. 1 is a simplified non-limiting example of a portion of a computing device. In some embodiments, the processing device 100 corresponds to at least a portion of an artificial intelligence (AI) acceleration device. In some embodiments, the AI acceleration device includes one or a combination of one or more central processing units (CPUs), one or more graphic processing units (GPUs), one or more tensor processing units (TPUs), application-specific integrated circuits (ASICs), and / or other types of processing units or circuits.

[0120] As Figure 1As illustrated in the middle, the processing device 100 includes a processing circuit 110 that includes a micro-scale quantizer (120, labeled as "MX Quantizer"), a micro-scale correlation processor (130, labeled as "MX Correlation Processor"), a micro-scale dequantizer (140, labeled as "MX Dequantizer"), and a post-processor 150. The processing device 100 further includes a memory 160 that includes memory units configured as storage areas to store at least weight data 162, activation data 164, and output data 166.

[0121] In some embodiments, the micro-scale quantizer 120 is configured to receive data 122 from the memory 160, where the input data 122 includes the weight data 162 and the activation data 164. In some embodiments, the weight data 162 includes a first set of digital representations of a first set of numbers, where the weight data 162 corresponds to weighting factors from one layer to a subsequent layer of a neural network or filter coefficients of a convolutional neural network. In some embodiments, the activation data 164 includes a second set of digital representations of a second set of numbers, where the activation data 164 corresponds to node values of one layer of a neural network. In some embodiments, the first set of digital representations and the second set of digital representations are in a 16-bit floating-point format (e.g., based on the Institute of Electrical and Electronics Engineers (IEEE) half-precision floating-point format, also known as FP16 format) or a 32-bit floating-point format (e.g., based on the IEEE single-precision floating-point format, also known as FP32 format). In some embodiments, the micro-scale quantizer 120 is configured to generate micro-scale output data (including a first portion 124 and a second portion 126) based on a numerical data quantization procedure. In some embodiments, the first portion 124 of the micro-scale output data includes a first set of quantized digital representations of the first set of numbers and / or a second set of quantized digital representations of the second set of numbers. In some embodiments, the second portion 126 of the micro-scale output data includes a first offset exponent scaling factor associated with the first set of quantized digital representations of the first set of numbers and / or a second offset exponent scaling factor associated with the second set of quantized digital representations. In some embodiments, the combination of the first set of quantized digital representations and the associated first offset exponent scaling factor and / or the combination of the second set of quantized digital representations and the associated second offset exponent scaling factor is consistent with the Open Compute Project (OCP) micro-scale format and micro-scale data format, such as MXFP8, MXFP6, MXFP4, MXINT8, or MXINT4. In some embodiments, the micro-scale quantizer 120 is configured to send the first portion 124 of the micro-scale output data to the micro-scale correlation processor 130 and to send the second portion 126 of the micro-scale output data to the micro-scale dequantizer 140.

[0122] In some embodiments, the mini-scaling correlation processor 130 is to receive the first portion 124 of the mini-scaling output and output numerical data 132 that is a result of processing the first set of quantized digital representations and the second set of quantized digital representations. In some embodiments, and by way of non-limiting example, the mini-scaling correlation processor 130 is to determine the result based on multiply-accumulate (MAC) operations of the first set of quantized digital representations and the second set of quantized digital representations as illustrated in an embodiment of the present disclosure. In some embodiments, the mini-scaling correlation processor 130 is to determine a result of processing the first portion 124 of the mini-scaling output data based on one or more other operations in a technical field of artificial intelligence (AI) computing, machine learning, language / text processing (e.g., for large language models (LLMs)), data encoding / decoding, audio processing, and / or graphics processing.

[0123] In some embodiments, because the first portion 124 of the mini-scaling output data is in a data format having a bit length that is less than the bit length of the input data 122 (due to the quantization of the mini-scaling quantizer 120 at the cost of precision), the size and complexity of the mini-scaling correlation processor 130 can be reduced compared to its counterpart that directly processes the input data 122. In some embodiments, the benefits and improvements of using the mini-scaling data format as discussed in an embodiment of the present disclosure include enabling more scaling compute units, higher energy efficiency (e.g., measured in operations per watt or TOPS / W), and higher area efficiency (e.g., measured in operations per square millimeter or TOPS / mm 2 while reducing memory bandwidth and capacity requirements.

[0124] In some embodiments, the mini-scaling dequantizer 140 is to receive the numerical data 132 from the mini-scaling correlation processor 130 and output dequantized digital representations 142 of the numerical data 132. In some embodiments, the dequantized digital representations 142 are in a 16-bit floating point format (e.g., FP16 format) or a 32-bit floating point format (e.g., FP32 format). In some embodiments, the post-processor 150 is to receive the dequantized digital representations 142 of the numerical data 132 from the mini-scaling dequantizer 140, perform one or more post-processing operations, and output post-processed data 152 to the memory 160. In some embodiments, the post-processed data 152 corresponds to at least a portion of the output data 166 stored in the memory 160. In some embodiments, the one or more post-processing operations performed by the post-processor 150 include introducing non-linearity to the dequantized digital representations 142 of the numerical data 132, merging the dequantized digital representations 142 of the numerical data 132, and / or other suitable operations.

[0125] In some embodiments, each of the one or more components of processing circuitry 110 is implemented, in whole or in part, based on one or more processors executing a set of instructions or computer program code stored in memory 160 and / or another memory included in processing circuitry 110, unless otherwise specified in an embodiment in this disclosure. In some embodiments, processing circuitry 110 includes one or more units based on a compute-in-memory (CIM) architecture configuration.

[0126] In many applications, the standard data format for AI workloads is typically FP32 or FP16. The micro-scaling (MX) format, as introduced by OCP, corresponds to quantizing data in FP32 or FP16 format to a shorter bit length (i.e., 8 bits or below) format, such as MX floating point format (MXFP8, MXFP6, or MXFP4) or MX integer format (MXINT8 or MXINT4). In some non-limiting application examples (e.g., with respect to operations of deep neural networks, vision transformers, and / or large language models), the accuracy degradation caused by implementing processing circuitry that processes data using MX format instead of using FP32 / FP16 format is exchanged for various improvements of less than 3%, such as 2x operations per second (TOPS), 3.7x TOPS / W, 5.2x TOPS / mm2, and / or 0.36x less than the area of a wafer. 2

[0127] Figure 2A A block diagram of micro-scaling quantizer 200A in accordance with some embodiments. In some embodiments, micro-scaling quantizer 200A corresponds to micro-scaling quantizer 120 in Figure 1 As illustrated in Figure 2A Micro-scaling quantizer 200A is used to receive input data 122 and output a first portion of micro- scaled output data 124 and a second portion of micro-scaled output data 126 as described in Figure 1

[0128] Micro-scaling quantizer 200A includes a maximum finder 210, a subtractor 220, a significand generator 230, and a data format converter 240. In some embodiments, input data 122 includes a first portion of input data 122A and a second portion of input data 122B corresponding to Figure 1 ​​In some embodiments, the maximum finder 210 is configured to determine a maximum exponent 212 from a set of exponents of the set of digit representations. In some embodiments, the maximum finder 210 is configured to output the maximum exponent 212 to the subtractor 220, the significand generator 230, and the data format converter 240.

[0129] In some embodiments, the subtractor 220 is configured to obtain a set of scaled exponents 222 based on subtracting the maximum exponent 212 from each of the set of exponents of the set of digit representations. In some embodiments, the subtractor 220 is configured to output the set of scaled exponents 222 to the significand generator 230. In some embodiments, the subtractor 220 is also configured to output the set of scaled exponents 222 to the data format converter 240.

[0130] In some embodiments, in a case where the miniscaled output data is based on an MX integer format (e.g., MXINT8 or MXINT4), the significand generator 230 is configured to obtain a set of quantized significands based on the set of digit representations of the set of digits and a set of exponents of the set of scaled exponents. In some embodiments, in a case where the miniscaled output data is based on an MX floating point format (e.g., MXFP8, MXFP6, or MXFP4), the significand generator 230 is configured to obtain a set of quantized exponents based on the set of exponents. In some embodiments, the significand generator 230 is configured to output the set of quantized significands or the set of quantized exponents (e.g., output data 232) to the data format converter 240.

[0131] In some embodiments, in a case where the miniscaled output data is based on an MX integer format (e.g., MXINT8 or MXINT4), the data format converter 240 is configured to output a set of quantized digit representations of the set of digits based on the set of quantized significands (output data 232) as a first portion 124 of the miniscaled output data. In some embodiments, in a case where the miniscaled output data is based on an MX floating point format (e.g., MXFP8, MXFP6, or MXFP4), the data format converter 240 is configured to output a set of quantized digit representations of the set of digits based on the set of quantized exponents (output data 232) and the set of scaled exponents 222 as the first portion 124 of the miniscaled output data. In some embodiments, the data format converter 240 is configured to output an offset exponent scaling factor associated with the set of quantized digit representations based on the maximum exponent 212 as a second portion 126 of the miniscaled output data. In some embodiments, the offset exponent scaling factor has a bit length of 8 bits.

[0132] In some embodiments, in a case where the microscaled output data is based on MX integer format (e.g., MXINT8 or MXINT4), the significand generator 230 is to obtain a set of shifted significands based on the set of mantissas and the set of scaled exponents, and round the set of shifted significands to a target bit length to become a set of quantized significands. In some embodiments, the significand generator 230 is further to convert the set of mantissas to a set of significands based on restoration of a first non-zero significand digit to each of the set of mantissas, and right shift the set of significands by a corresponding number of bits indicated by the set of scaled exponents to become the set of shifted significands. In some embodiments, the target bit length is 7 bits (e.g., for output in MXINT8 format). In some embodiments, the set of quantized digit representations comprises a set of 2’s complement integer values of a set of sign bits of the set of quantized significands based on the set of digit representations of the set of numbers. In some embodiments, a bit length of each of the set of 2’s complement integer values is 8 bits (e.g., for output in MXINT8 format).

[0133] In some embodiments, in a case where the microscaled output data is based on M floating point format (e.g., MXFP8, MXFP6, or MXFP4), the significand generator 230 is to obtain an unbiased exponent scaling factor based on subtracting a target offset from a maximum exponent, and obtain a set of scaled exponents based on subtracting the unbiased exponent scaling factor from each of the set of exponents, each exponent in the set of scaled exponents having a target exponent bit length. In some embodiments, the target exponent bit length ranges from 5 bits to 2 bits. In some embodiments, the significand generator 230 is further to round the set of mantissas to a target bit length to become a set of quantized mantissas. In some embodiments, the target mantissa bit length ranges from 3 bits to 1 bit.

[0134] In some embodiments, in the case that the microscaled output data is based on MX floating point format (e.g., MXFP8, MXFP6, or MXFP4) conditions, each of the set of quantized digit representations includes a corresponding one of the set of sign bits of the set of digit representations, a corresponding one of the set of scaled exponents, and a corresponding one of the set of quantized mantissas. In some embodiments, each of the set of quantized digit representations has a bit length of 8 bits (e.g., MXFP8), each of the set of scaled exponents has a bit length of 4 bits (e.g., 4-bit exponent), and each of the set of quantized mantissas has a bit length of 3 bits (3-bit mantissa) (also referred to as MXFP8(E4M3) format). In some embodiments, each of the set of quantized digit representations has a bit length of 8 bits (e.g., MXFP8), each of the set of scaled exponents has a bit length of 5 bits (e.g., 5-bit exponent), and each of the set of quantized mantissas has a bit length of 2 bits (2-bit mantissa) (also referred to as MXFP8(E5M2) format). In some embodiments, each of the set of quantized digit representations has a bit length of 6 bits (e.g., MXFP6), each of the set of scaled exponents has a bit length of 2 bits (e.g., 2-bit exponent), and each of the set of quantized mantissas has a bit length of 3 bits (3-bit mantissa) (also referred to as MXFP6(E2M3) format). In some embodiments, each of the set of quantized digit representations has a bit length of 6 bits (e.g., MXFP6), each of the set of scaled exponents has a bit length of 3 bits (e.g., 3-bit exponent), and each of the set of quantized mantissas has a bit length of 2 bits (2-bit mantissa) (also referred to as MXFP6(E3M2) format). In some embodiments, each of the set of quantized digit representations has a bit length of 4 bits (e.g., MXFP4), each of the set of scaled exponents has a bit length of 2 bits (e.g., 2-bit exponent), and each of the set of quantized mantissas has a bit length of 1 bit (1-bit mantissa) (also referred to as MXFP4(E2M1) format).

[0135] In some embodiments, the numerical data quantization procedure of the example based on Figure 2A avoids performing logarithm and / or exponent calculations, replaces multiplication operations in linear space with shift and / or addition operations in exponent space, and replaces division operations in linear space with subtraction operations in exponent space. As a result, computational complexity and conversion speed are improved.

[0136] Figure 2B is a block diagram of a microscaled dequantizer 200B according to some embodiments. In some embodiments, the microscaled dequantizer 200B corresponds to the microscaled dequantizer 140 in Figure 1 . As shown in FIG. 20B, the microscaled dequantizer 200B includes a set of scaled exponent decoders 202, a set of scaled mantissa decoders 204, and a set of digit decoders 206. Figure 2BAs shown in the diagram, the microscale dequantizer 200B is used to receive the numerical data 132 and the second part 126 of the microscale output data, and outputs as follows: Figure 1 The dequantized bit representation 142 of the numerical data 132 described herein.

[0137] The microscale dequantizer 200B includes a hidden bit finder 250, a mantissa extractor 260, an exponent adjustment extractor 270, a first adder 282, a second adder 286, and a data format converter 290. In some embodiments, the numeric data 132 includes numeric data that processes a first set of numbers (e.g., corresponding to...). Figure 1 The first group of quantized digits and the second group of digits (e.g., corresponding to the weighted data 162) Figure 1 The result of the second set of quantized bits in the start data 164). In some embodiments, the hidden bit finder 250 is used to identify the first non-zero significant digits of the numerical data and provide this information 252 to the mantissa extractor 260 and the exponent adjustment extractor 270. In some embodiments, the hidden bit finder 250 is incorporated into the mantissa extractor 260 and / or the exponent adjustment extractor 270. In some embodiments, the functionality of the hidden bit finder 250 is embedded in the mantissa extractor 260 and / or the exponent adjustment extractor 270, and the hidden bit finder 250 is therefore omitted. In some embodiments, the functionality of the hidden bit finder 250 is not required for subsequent processing by the mantissa extractor 260 and / or the exponent adjustment extractor 270, and the hidden bit finder 250 is therefore omitted.

[0138] In some embodiments, mantissa extractor 260 is used to extract the mantissa 262 of the inverse quantized bit representation of numeric data 132 and outputs the mantissa 262 to data format converter 290. In some embodiments, mantissa extractor 260 is used to further extract the mantissa based on removing a first non-zero significant digit from the numeric data (e.g., based on information 252 from hidden digit finder 250). In some embodiments, exponent adjustment extractor 270 is used to extract exponent adjustment 272 from numeric data 132 and outputs exponent adjustment 272 to second adder 286. In some embodiments, exponent adjustment extractor 270 is used to further extract exponent adjustment 272 based on the bit position of the first non-zero significant digit within the numeric data (e.g., based on information 252 from hidden digit finder 250).

[0139] In some embodiments, the first adder 282 is to obtain a combined exponent scale factor 284 based on a first exponent scale factor associated with the first set of quantized digit representations and a second exponent scale factor associated with the second set of quantized digit representations (e.g., included in the second portion 126 of the micro-scaled output data). In some embodiments, the second adder 286 is to obtain an unbiased exponent 288 of the numeric data based on the combined exponent scale factor 284 from the first adder 282 and the exponent adjustment 272 from the exponent adjustment extractor 270.

[0140] In some embodiments, the data format converter 290 is to output the dequantized digit representation 142 of the numeric data. In some embodiments, the digit representation is based on the unbiased exponent 288 of the numeric data including a mantissa 262 of the digit representation and an exponent of the digit representation.

[0141] In some embodiments, in a case where the first set of quantized digit representations and the second set of quantized digit representations are based on the MX integer format (e.g., MXINT8 or MXINT4), the second adder 286 is to obtain the unbiased exponent 288 based on an addition of the combined exponent scale factor 284 and the exponent adjustment 272. In some embodiments, in a case where the first set of quantized digit representations and the second set of quantized digit representations are based on the MX floating point format (e.g., MXFP8, MXFP6, or MXFP4), the second adder 286 is to obtain the unbiased exponent 288 based on an addition of a maximum product exponent of the first set of quantized digit representations and the second set of quantized digit representations, the combined exponent scale factor 284, and the exponent adjustment 272. In some embodiments, the maximum product exponent is an unbiased exponent value and the combined exponent scale factor is another unbiased exponent value.

[0142] In some embodiments, the dequantized digit representation of the numeric data is based on FP16 and includes a sign bit extracted from the numeric data 132, an exponent having a bit length of 5 bits, and a mantissa having a bit length of 10 bits. In some embodiments, the dequantized digit representation of the numeric data is based on FP32 and includes a sign bit extracted from the numeric data 132, an exponent having a bit length of 8 bits, and a mantissa having a bit length of 23 bits.

[0143] In some embodiments, the numeric data dequantization procedure is based on Figure 2B Examples of the numeric data dequantization procedure also avoid performing logarithm and / or exponent calculations and provide a convenient method to convert numeric data from an exponent space back to a linear space. As a result, thanks to the use of the micro-scaled format as discussed in Figure 2A the results can still be obtained in the linear space without increasing the computational complexity and conversion cost.

[0144] Figure 3A A processing flow diagram 300A for a numerical data quantization processing flow instance according to some embodiments. The processing flow diagram 300A includes various stages corresponding to operations performed by a microscale quantizer 310, which corresponds to Figure 1 the microscale quantizer 120 in FIG. 1 and / or Figure 2A the microscale quantizer 200A in FIG. 2. In some embodiments, the numerical data quantization instance receives two sets of digital representations of numbers as input data 302, and outputs two sets of quantized digital representations 304 and two associated offset exponent scaling factors 306. In some embodiments, each set of quantized mantissa representations includes 8, 16, 32, or 64 entries.

[0145] In this non-limiting example, each set of quantized digital representations includes 4 entries for illustration purposes. In this non-limiting example, the input data 302 is based on the FP16 format, and the quantized digital representations 304 and associated offset exponent scaling factors 306 are based on the MXINT8 format. For example, the input data 302 includes a set of weights and a boot input asset as follows.

[0146] Weight (FP16) Activation input (FP16) 0011 1011 1111 1000 1011 0111 0101 0001 0011 0000 0101 1001 0011 1010 0101 1110 1100 0000 10100001 0011 1010 1000 0111 0010 1000 1011 1001 1010 1011 0000 1010

[0147] In some embodiments, each of the digital representations includes a sign bit at the leftmost bit, followed by a 5-bit exponent and then a 10-bit mantissa.

[0148] At stage 312, the maximum exponent of each set of digital representations included in the input data is determined, for example, by the maximum finder 210 in FIG. 1. In this example, the maximum exponent of the weights is “10000”, and the maximum exponent of the boot input is “01110”. In some embodiments, the maximum exponent at this stage is an offset post-offset exponent based on 15. Thus, the un-offset exponent of the weights is effectively “1”, and the un-offset exponent of the boot input is “-1”. Figure 2A

[0149] At stage 314, a set of scaled exponents of the weights and a set of scaled exponents of the boot input are obtained, for example, by the subtractor 220 in FIG. 2. In some embodiments, the scaled exponents are calculated based on subtracting the maximum exponent from the corresponding exponent. For example, the set of scaled exponents are as follows. Figure 2A

[0150] Scaled exponent of weight Scaled exponent of activation input 2 1 4 1 0 0 6 4

[0151] At stage 316, the corresponding sets of quantized significands are obtained, for example, by the quantizer 230 in FIG. 2. In some embodiments, the quantized significands are calculated based on dividing the corresponding mantissa by the corresponding scaled exponent. For example, the sets of quantized significands are as follows. Figure 2A ​​quantized significands are obtained by converting the corresponding mantissas to significands by adding the hidden bits, right shifting the significands by a number of bits based on the corresponding scaled exponents, and rounding the shifted significands to a target bit length. In this non-limiting example, the target bit length is 7. For example, the shifted significands are as follows.

[0152] Shifted significand of weight Shifted significand of activation input 001 11 1111 1000 01 11 0101 0001 00001 00 0101 1001 01 10 0101 1110 1 00 1010 0001 1 10 1000 0111 0000001 00 1011 1001 00001 11 0000 1010

[0153] In addition, the quantized significands after rounding are as follows.

[0154] Quantized significand of weight Quantized significand of activation input 0100000 0111011 0000100 0100110 1001010 1101000 0000001 0000111

[0155] At stage 318, the array maximum exponents and the array quantized significands are collected and configured consistently with an output format, such as MXINT8 in this example. In some embodiments, the quantized significands are converted to 8-bit 2’s complement (labeled as “2’s complement” in the table below) integer values to become the corresponding quantized digit representations 304. For example, the quantized digit representations are as follows.

[0156] Quantized digit representation of weight (two's complement) Quantized digit representation of activation input (two's complement) 0010 0000 1100 0101 0000 0100 0010 0110 1011 0110 0110 1000 0000 0001 1111 1001

[0157] In some embodiments, the maximum exponent of the weights and the maximum exponent of the activation inputs are converted to 8-bit offset exponent scaling factors with an offset of 127. For example, the offset exponent scaling factor associated with the set of quantized digit representations of the weights is “1000 0000”, and the offset exponent scaling factor associated with the set of quantized digit representations of the activation inputs is “0111 1110”.

[0158] Figure 3B A process flow diagram 300B for a numerical data dequantization process example according to some embodiments. The process flow diagram 300B includes various stages corresponding to operations performed by a micro-scale dequantization 330, which corresponds to Figure 1 the micro-scale dequantization 140 in Figure 2B the micro-scale dequantization 200B in In some embodiments, the numerical data dequantization example receives numerical data 322, which is a first set of quantized digit representations of a first set of numbers (e.g., quantized digit representations of weights from Figure 3A and a second set of quantized digit representations of a second set of numbers (e.g., quantized digit representations of activation inputs from Figure 3AThe result of the quantized bit representation of the start input. In some embodiments, the numerical data dequantization instance also receives an exponential scaling factor 306 of the offset associated with the first set of quantized bit representations and the second set of quantized bit representations. In some embodiments, the numerical data dequantization instance outputs the dequantized bit representation 326 of the numerical data 322.

[0159] In this non-limiting example, for illustrative purposes, the numerical data 322 corresponds to the processing from... Figure 3A The result of the quantized bit representation. For example, the numeric data 322 is the complement of 2 in "111101.101100100001". In this example, the two leftmost bits are the sign bits, and the unsigned binary value of the numeric data 322 is "0010.010011011111". At stage 332, the first non-zero significant digit of the numeric data (in the form of an unsigned binary value) is, for example, represented by... Figure 2B The hidden bit finder 250 identifies the bits. In this example, the first non-zero significant digit is the second digit to the left of the point separator (i.e., "2"). 1 (Digital).

[0160] At stage 334, based on the information from stage 332, the dequantized bit representation of the mantissa of 326 is, for example, by... Figure 2B The mantissa extractor 260 extracts the mantissa based on the unsigned binary value of the numerical data 322. In some embodiments, the mantissa is based on the dequantized bit representation 326, which in this non-limiting instance is in FP16 format and rounded to a bit length of 10 bits. In this non-limiting instance, the extracted mantissa is "0010011100" (rounded). Furthermore, at stage 336, based on information from stage 332, the exponent is adjusted, for example by... Figure 2B The exponent adjustment extractor 270 extracts from the numerical data 322. In this non-limiting instance, the exponent is adjusted to "1" because the extracted mantissa begins at the first digit to the left of the point separator.

[0161] At stage 342, the exponential scaling factor 306 based on the offset, and the combined exponential scaling factor, for example, is derived from... Figure 2B The first adder 282 in the input is used to obtain the combined exponential scaling factor. In some embodiments, the combined exponential scaling factor is obtained by adding the unoffset equivalent of the set of quantized bits representing the weights to the unoffset equivalent of the set of quantized bits representing the starting input to the associated exponential scaling factor. In this non-limiting example, the combined exponential scaling factor is 0.

[0162] At stage 344, the unoffset exponent of numerical data 322 is, for example, derived from... Figure 2Ba second adder 286 in the stage 336 based on the exponent adjustment from the stage 342 and the combined exponent scaling factor from the stage 344. In some embodiments, the unbiased exponent is obtained based on adding the exponent adjustment and the combined exponent scaling factor. In this non-limiting example, the unbiased exponent is 1.

[0163] At the stage 346, the dequantized digital representation 326 of the numeric data 322 is obtained, for example, by Figure 2B the data format converter 290 in the stage 336. In some embodiments, the dequantized digital representation 326 is based on FP16. In some embodiments, the dequantized digital representation 326 includes the sign bit from the numeric data 322, the mantissa from the stage 334, and the biased exponent based on the unbiased exponent from the stage 344. In this non-limiting example, the dequantized digital representation 326 of the numeric data 322 in FP16 format is "11000000 1001 1100".

[0164] Figure 4A A process flow diagram 400A for another numeric data quantization process instance according to some embodiments. The process flow diagram 400A includes various stages corresponding to operations performed by a micro-scaled quantizer 410, which corresponds to the micro-scaled quantizer 120 in Figure 1 and / or the micro-scaled quantizer 200A in Figure 2A In some embodiments, the numeric data quantization instance receives two sets of digital representations of two sets of numbers as input data 402, and outputs two sets of quantized digital representations 404 and two associated biased exponent scaling factors 406. In some embodiments, each set of quantized mantissa representations includes 8, 16, 32, or 64 entries.

[0165] In this non-limiting example, each set of quantized digital representations includes 4 entries for illustration purposes. In this non-limiting example, the input data 402 is based on FP16 format, and the quantized digital representations 404 and the associated biased exponent scaling factors 406 are based on MXFP8(E4M3) format. In this non-limiting example, the input data 402 includes the same set of weights and the start input asset as the instance in Figure 3A

[0166] At the stage 412, the maximum exponent of each set of digital representations included in the input data is obtained, for example, by Figure 2A ​The maximum exponent of the weights and the maximum exponent of the boot input are determined by the maximum finder 210. In this example, the maximum exponent of the weights is "10000" and the maximum exponent of the boot input is "01110". In some embodiments, the maximum exponent at this stage is the post-offset exponent based on the 15-bit offset. Thus, the un-offset exponent of the weights is effectively "1" and the un-offset exponent of the boot input is effectively "-1". Furthermore, to match the MXFP8(E4M3) format, the un-offset exponent is further scaled by subtracting the target offset (e.g., 8 for MXFP8(E4M3)) from the un-offset exponent. Thus, the scaled exponents become the un-offset exponent scaled factor of the weights is "-7" and the un-offset exponent scaled factor of the boot input is "-9".

[0167] At stage 414, the set of scaled exponents of the weights and the set of scaled exponents of the boot input are obtained, for example, by the subtractor 220 in the MXFP8(E4M3) format. Figure 2A In some embodiments, the scaled exponents are calculated based on subtracting the corresponding un-offset exponent scaled factor from the corresponding un-offset exponent. For example, the set of scaled exponents are as follows.

[0168] Scaled exponent of weight Scaled exponent of activation input 6 7 4 7 8 8 2 4

[0169] The set of scaled exponents are converted to 4-bit binary values as follows.

[0170] Scaled exponent of weight Scaled exponent of activation input 1101 1110 1011 1110 1111 1111 1001 1011

[0171] At stage 416, the corresponding set of quantized mantissas are obtained, for example, by the significand generator 230 in the MXFP8(E4M3) format. Figure 2A In some embodiments, the quantized mantissas are obtained by rounding the corresponding mantissas to the target bit length. In this non-limiting example, the target bit length is 3. For example, the quantized mantissas are as follows.

[0172] Quantized mantissa of weight Quantized mantissa of activation input 111 111 001 001 001 101 001 110

[0173] At stage 418, the set of scaled exponents and the set of quantized mantissas are collected along with the sign bits included in the input data 402 and configured to conform to the output format, for example, the MXFP8(E4M3) in this example. For example, the quantized digit representations are as follows.

[0174] Quantized digit representation of weight Quantized digit representation of activation input 0110 1111 1111 0111 0101 1001 0111 0001 1111 1001 0111 1101 0100 1001 1101 1110

[0175] In some embodiments, the unoffset exponential scaling factor of the weights and the unoffset exponential scaling factor of the start input are converted to an 8-bit offset exponential scaling factor with an offset of 127. For example, the exponential scaling factor of the offset associated with the set of quantized bits of the weights is "01111000", and the exponential scaling factor of the offset associated with the set of quantized bits of the start input is "01110110".

[0176] Figure 4B A flowchart 400B is provided as an example of another numerical data dequantization processing procedure according to some embodiments. Flowchart 400B includes various stages corresponding to operations performed by a microscale dequantizer 430, which corresponds to... Figure 1 The microscale inverse quantizer 140 and / or Figure 2B The miniature dequantizer 200B is described. In some embodiments, the numerical data dequantization instance receives numerical data 422, which is a first set of quantized bit representations of the first set of numbers (e.g., from...). Figure 4A The weights are represented by quantized bits) and the second set of quantized bits represents the second set of numbers (e.g., from...). Figure 4A (The quantized bit representation of the start input). In some embodiments, the numerical data dequantization instance also receives an exponential scaling factor 406 of the offset associated with the first set of quantized bit representations and the second set of quantized bit representations. In some embodiments, the numerical data dequantization instance outputs the dequantized bit representation 426 of the numerical data 422.

[0177] In this non-limiting example, for illustrative purposes, the numerical data 422 corresponds to the processing from... Figure 4A The result of the quantized bit representation. For example, the numeric data 422 is the complement of 2 in "111101.1100010100100010". In this example, the two leftmost bits are the sign bits, and the unsigned binary value of the numeric data 422 is "0010.0011101011011110". At stage 432, the first non-zero significant digit of the numeric data (in the form of an unsigned binary value) is, for example, represented by... Figure 2B The hidden bit finder 250 identifies the bits. In this example, the first non-zero significant digit is the second digit to the left of the point separator (i.e., "2"). 1 (Digital).

[0178] At stage 434, based on the information from stage 432, the dequantized bit representation of the mantissa of 426 is, for example, by... Figure 2BThe mantissa extractor 260 extracts the mantissa based on the unsigned binary value of the numerical data 422. In some embodiments, the mantissa is based on the dequantized bit representation 426, which in this non-limiting instance is in FP16 format and rounded to a bit length of 10 bits. In this non-limiting instance, the extracted mantissa is "0001110110" (rounded). Furthermore, at stage 436, based on information from stage 432, the exponent is adjusted, for example by... Figure 2B The exponent adjustment extractor 270 extracts from the numerical data 422. In this non-limiting instance, the exponent is adjusted to "1" because the extracted mantissa begins at the first digit to the left of the point separator.

[0179] At stage 442, the offset-based exponential scaling factor 406, and the combined exponential scaling factor, for example, are derived from... Figure 2B The first adder 282 in the input is used to obtain the combined exponential scaling factor. In some embodiments, the combined exponential scaling factor is obtained by adding the unoffset equivalent of the set of quantized bits representing the weights to the unoffset equivalent of the set of quantized bits representing the starting input to the associated exponential scaling factor. In this non-limiting example, the combined exponential scaling factor is -16.

[0180] At stage 443, the modified index adjustment is obtained based on the following operations: for example, through... Figure 2B The second adder 286 adds the exponent adjustment from stage 436 and the maximum product exponent from the first set of quantized bit representations and the second set of quantized bit representations from the numerical data 422. In this non-limiting example, the exponent adjustment is modified to 17.

[0181] At stage 444, the unoffset exponent of the numerical data 422 is, for example, by... Figure 2B The second adder 286 is obtained based on the modified exponent adjustment from stage 443 and the combined exponent scaling factor from stage 442. In some embodiments, the unoffset exponent is obtained by adding the modified exponent adjustment and the combined exponent scaling factor. In this non-limiting example, the unoffset exponent is 1.

[0182] At stage 446, the dequantized bit representation 426 of the numerical data 422 is, for example, derived from... Figure 2B The data is obtained by the data format converter 290. In some embodiments, the unquantized bit representation 426 is based on FP16. In some embodiments, the dequantized bit representation 426 includes the sign bit from the numeric data 422, the mantissa from stage 434, and an offset index based on the unoffset index from stage 444. In this non-limiting example, the dequantized bit representation 426 of the numeric data 422 in FP16 format is "1100 0000 0111 0110".

[0183] Figure 5 is a flowchart of a method 500 of quantizing numerical data according to some embodiments. In some embodiments, the various operations of the method 500 are performed by the microscale quantizer 120 in Figure 1 or the microscale quantizer 200A in Figure 2A In some embodiments, the method 500 corresponds to the processing flow instance in Figure 3A or the processing flow instance in Figure 4A In some embodiments, the method 500 corresponds to one or more operations performed entirely or partially based on the computing device 700 as illustrated in Figure 7 As illustrated in Figure 5 the method 500 includes blocks 510-550.

[0184] At block 510, a maximum exponent is determined from a set of exponents of a set of digit representations of a set of numbers. In some embodiments, the set of digit representations of the set of numbers correspond to at least a portion of the data 122 in Figure 1 and Figure 2A the input data 302 in Figure 3A or the input data in Figure 4A In some embodiments, the set of digit representations of the set of numbers correspond to the weight data 162 in Figure 1 in the FP16 format or the FP32 format. In some embodiments, the set of digit representations of the set of numbers correspond to the activation data 164 in Figure 1 in the FP16 format or the FP32 format. In some embodiments, the block 510 corresponds to operations performed by the maximum finder 210 in Figure 2A In some embodiments, the block 510 corresponds to operations at the stage 312 in Figure 3A or the stage 412 in Figure 4A

[0185] At block 520, a set of scaled exponents is obtained by processing circuitry (e.g., of the microscale quantizer 120, the microscale quantizer 200A, or the computing device 700) based on subtracting the maximum exponent from each of the set of exponents. In some embodiments, the block 520 corresponds to operations performed by the subtracter 220 in Figure 2A In some embodiments, the block 520 corresponds to operations at the stage 314 in Figure 3A or the stage 414 in Figure 4A

[0186] ​​In some embodiments corresponding to outputting microscaled output data based on an MX floating-point format (e.g., MXFP8, MXFP6, or MXFP4), the set of scaled exponents is obtained based on obtaining an unscaled exponent scale factor based on subtracting a target offset from a maximum exponent, and obtaining the set of scaled exponents based on subtracting the unscaled exponent scale factor from each of the set of exponents, each exponent of the set of scaled exponents having a target exponent bit length. In some embodiments, the target exponent bit length ranges from 5 bits to 2 bits.

[0187] At block 530, in some embodiments corresponding to outputting microscaled output data based on an MX integer format (e.g., MXINT8 or MXINT4), the processing circuit obtains a set of quantized significands based on the set of digit representations of the set of numbers and the set of scaled exponents. In some embodiments, the set of quantized digit representations includes a set of 2’s complement integer values of a set of sign bits of the set of quantized significands based on the set of digit representations of the set of numbers. In some embodiments, each of the set of 2’s complement integer values has a bit length of 8 bits. In some embodiments, block 530 corresponds to operations performed by significand generator 230 in FIG. 7. Figure 2A In some embodiments, block 530 corresponds to operations at stage 316 in FIG. 7. Figure 3A

[0188] In some embodiments, the set of quantized significands is obtained based on obtaining a set of shifted significands based on the set of exponents and the set of scaled exponents, and rounding the set of shifted significands to a target bit length to become the set of quantized significands. In some embodiments, the set of quantized significands is further obtained based on converting the set of exponents to a set of significands based on restoring a first non-zero significand digit to each of the set of exponents, and right-shifting the set of significands by a corresponding number of bits indicated by the set of scaled exponents to become the set of shifted significands. In some embodiments, the target bit length is 7 bits.

[0189] At block 530, in some embodiments corresponding to outputting microscaled output data based on an MX floating-point format (e.g., MXFP8, MXFP6, or MXFP4), the processing circuit obtains a set of quantized exponents based on the set of exponents. In some embodiments, block 530 corresponds to operations performed by exponent generator 230 in FIG. 7. Figure 4A In some embodiments, block 530 corresponds to operations at stage 416 in FIG. 7. Figure 2A

[0190] ​​In some embodiments, the set of quantized mantissas is obtained based on rounding the set of mantissas to a target bit length to become the set of quantized mantissas. In some embodiments, the target mantissa bit length ranges from 3 bits to 1 bit.

[0191] At block 540, in some embodiments corresponding to outputting microscaled output data based on an MX integer format (e.g., MXINT8 or MXINT4), a set of quantized digit representations of the set of numbers is output to memory (e.g., memory 160) based on the set of quantized significands. In some embodiments, block 540 corresponds to operations performed by data format converter 240 in Figure 3A FIG. 3. In some embodiments, block 540 corresponds to operations at stage 318 in Figure 2A FIG. 4.

[0192] At block 540, in some embodiments corresponding to outputting microscaled output data based on an MX floating point format (e.g., MXFP8, MXFP6, or MXFP4), a set of quantized digit representations of the set of numbers is output to memory (e.g., memory 160) based on the set of quantized mantissas and the set of scaled exponents. In some embodiments, each of the set of quantized digit representations includes a corresponding one of a set of sign bits of the set of digit representations of the set of numbers, a corresponding one of the set of scaled exponents, and a corresponding one of the set of quantized mantissas. In some embodiments, block 540 corresponds to operations performed by data format converter 240 in Figure 4A FIG. 4. In some embodiments, block 540 corresponds to operations at stage 418 in Figure 2A FIG. 4.

[0193] In some embodiments based on the MXFP8 (E4M3) format, each of the set of quantized digit representations has a bit length of 8 bits, each of the set of scaled exponents has a bit length of 4 bits, and each of the set of quantized mantissas has a bit length of 3 bits. In some embodiments based on the MXFP8 (E5M2) format, each of the set of quantized digit representations has a bit length of 8 bits, each of the set of scaled exponents has a bit length of 5 bits, and each of the set of quantized mantissas has a bit length of 2 bits. In some embodiments based on the MXFP6 (E2M3) format, each of the set of quantized digit representations has a bit length of 6 bits, each of the set of scaled exponents has a bit length of 2 bits, and each of the set of quantized mantissas has a bit length of 3 bits. In some embodiments based on the MXFP6 (E3M2) format, each of the set of quantized digit representations has a bit length of 6 bits, each of the set of scaled exponents has a bit length of 3 bits, and each of the set of quantized mantissas has a bit length of 2 bits. In some embodiments based on the MXFP4 (E2M1) format, each of the set of quantized digit representations has a bit length of 4 bits, each of the set of scaled exponents has a bit length of 2 bits, and each of the set of quantized mantissas has a bit length of 1 bit.

[0194] At block 550, the offset exponent scaling factor associated with the set of quantized digit representations is output to memory (e.g., memory 160) based on the maximum exponent. In some embodiments, the offset exponent scaling factor has a bit length of 8 bits. In some embodiments, block 550 corresponds to the operations performed by data format converter 240 in Figure 3A In some embodiments, block 550 corresponds to the operations at stage 318 in Figure 4A In some embodiments, block 550 corresponds to the operations at stage 418 in Figure 6 In some embodiments, block 550 corresponds to the operations at stage 418 in

[0195] Figure 1 A flowchart of a method 600 of dequantizing numerical data according to some embodiments. In some embodiments, the various operations of method 600 are performed by microscale dequantization 140 in Figure 2B In some embodiments, method 600 corresponds to the processing flow instance in Figure 3B In some embodiments, method 600 corresponds to the processing flow instance in Figure 4B In some embodiments, method 600 corresponds to the processing flow instance in Figure 7 In some embodiments, method 600 corresponds to one or more operations performed entirely or partially by computing device 700 as illustrated in Figure 6 In some embodiments, method 600 corresponds to one or more operations performed entirely or partially by computing device 700 as illustrated in Figure 2B In some embodiments, method 600 corresponds to one or more operations performed entirely or partially by computing device 700 as illustrated in

[0196] At block 610, the mantissa of the dequantized bit representation of the numerical data is extracted. In some embodiments, the numerical data is the result of processing a first set of quantized bit representations of a first set of numbers and a second set of quantized bit representations of a second set of numbers. In some embodiments, the first set of numbers and the second set of numbers are based on MX integer format (e.g., MXINT8 or MXINT4) or MX floating-point format (e.g., MXFP8, MXFP6, or MXFP4). In some embodiments, the final dequantized bit representation of the numerical data is in FP16 or FP32 format. In some embodiments, block 610 corresponds to the data obtained by... Figure 3B The operation performed by the mantissa extractor 260. In some embodiments, block 610 corresponds to... Figure 4B Phase 334 or Figure 2B The operation at stage 434.

[0197] At block 620, the exponential adjustment from the numerical data is extracted. In some embodiments, block 620 corresponds to the data obtained from... Figure 3B The operation performed by the index adjustment extractor 270. In some embodiments, block 620 corresponds to... Figure 4B Stage 336 or Figure 2B The operation at stage 436.

[0198] In some embodiments, method 600 further includes identifying a first non-zero significant digit of the numerical data (e.g., corresponding to the digits formed by...). Figure 3B The operations performed by the hidden bit finder 250 in the middle, and Figure 4B Phase 332 or Figure 2B (The operation at stage 432). In some embodiments, the mantissa is further extracted based on the removal of the first non-zero significant digits from the numeric data. In some embodiments, the exponent adjustment is further extracted based on the position of the first non-zero significant digits within the numeric data.

[0199] At block 630, the combined exponential scaling factor is obtained by processing circuitry (e.g., microscale dequantizer 140, microscale dequantizer 200B, or computing device 700) based on a first exponential scaling factor associated with a first set of quantized bit representations and a second exponential scaling factor associated with a second set of quantized bit representations. In some embodiments, block 630 corresponds to the... Figure 3B The operation performed by the first adder 282 in the block. In some embodiments, block 630 corresponds to... Figure 4B Phase 342 or Figure 2B The operation at stage 442.

[0200] At block 640, an unbiased exponent of the numeric data is obtained by the processing circuit based on the combined exponent scaling factor and the exponent adjustment. In some embodiments, block 640 corresponds to the operation performed by the second adder 286 in Figure 3B In some embodiments, block 630 corresponds to the operation performed by the stage 344 in Figure 4B In some embodiments, block 630 corresponds to the operation performed by the stage 438, 443, and 444 in Figure 2B In some embodiments, block 630 corresponds to the operation performed by the stage 438, 443, and 444 in

[0201] In some embodiments, in a case where the first set of numbers and the second set of numbers are based on MX integer format (e.g., MXINT8 or MXINT4), the unbiased exponent is obtained based on an addition of the combined exponent scaling factor and the exponent adjustment, and the combined exponent scaling factor is an unbiased exponent value. In some embodiments, in a case where the first set of numbers and the second set are based on MX floating point format (e.g., MXFP8, MXFP6, or MXFP4), the unbiased exponent of the numeric data is obtained based on an addition of a maximum product exponent of the first set of quantized digit representations and the second set of quantized digit representations, the combined exponent scaling factor, and the exponent adjustment, the maximum product exponent is an unbiased exponent value, and the combined exponent scaling factor is another unbiased exponent value.

[0202] At block 650, the dequantized digit representation of the numeric data is output. In some embodiments, the digit representation includes a mantissa of the digit representation of the mantissa, and an exponent of the dequantized digit representation based on the unbiased exponent of the numeric data. In some embodiments, block 650 corresponds to the operation performed by the data format converter 290 in Figure 3B In some embodiments, block 630 corresponds to the operation performed by the stage 346 in Figure 4B In some embodiments, block 630 corresponds to the operation performed by the stage 438, 443, and 444 in Figure 7 In some embodiments, block 630 corresponds to the operation performed by the stage 438, 443, and 444 in

[0203] Figures 3A to 6 A block diagram of a computing device instance 700 according to some embodiments is usable in connection with one or more embodiments. In some embodiments, an embodiment of the present disclosure described herein is directed to a method and / or operations performed by the computing device 700 as a whole or in part according to some embodiments. ​

[0204] ​In some embodiments, computing device 700 is a general purpose computing device or a special purpose computing device. In some embodiments, computing device 700 includes one or more hardware processors 702 and memory 704. In some embodiments, memory 704 includes a non-transitory computer-readable storage medium encoded, i.e., storing, a set of executable instructions 706 (i.e., computer program code). Execution of instructions 706 by one or more hardware processors 702 represents (at least in part) a processing device that implements, in accordance with one or more embodiments (hereinafter, a referenced program and / or method), a portion or all of the methods described herein and / or operations. In some embodiments, in addition to computer executable instructions 706, memory 704 also stores processing information 707 that facilitates execution of a portion or all of the referenced program and / or method.

[0205] One or more hardware processors 702 are electrically coupled to memory 704 via bus 708. One or more hardware processors 702 are also electrically coupled to I / O interface 710 by bus 708. Network interface 712 is also electrically connected to one or more hardware processors 702 via bus 708. Network interface 712 is connected to network 714 (which in some embodiments is not part of computing device 700) so that one or more hardware processors 702 and memory 704 can connect to external elements via network 714. One or more hardware processors 702 are used to execute instructions 706 encoded in memory 704 so as to enable computing device 700 to be used to perform a portion or all of the referenced program and / or described method of an embodiment of the present disclosure. In one or more embodiments, one or more hardware processors include a CPU, a GPU, a TPU, an ASIC, suitable processing circuitry, or any combination thereof.

[0206] In one or more embodiments, memory 704 includes electronic, magnetic, optical, electromagnetic, infrared, and / or semiconductor systems (or devices or apparatuses). For example, memory 704 includes semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), rigid disks, and / or optical disks. In one or more embodiments that use optical disks, memory 704 includes compact disk-read only memory (CD-ROM), compact disk-read / write (CD-R / W), and / or digital video disc (DVD).

[0207] In some embodiments, network interface 712 comprises a wireless network interface, such as Bluetooth, WIFI, WIMAX, GPRS, or WCDMA; or a wired network interface, such as ETHERNET, USB, or IEEE- 1364. In one or more embodiments, part or all of the referenced programs and / or methods are implemented based on two or more computing devices 700.

[0208] Computing device 700 is to receive information via I / O interface 710. The information received via I / O interface 710 includes one or more of the following: instructions, weight data, initialization data for a neural network model, and / or other parameters for processing by one or more hardware processors 702. The information is communicated to one or more hardware processors 702 via bus 708. Computing device 700 is to implement a user interface (UI) based on executing user interface (UI) instructions 742 stored on memory 704. Computing device 700 is to receive user input based on user operation of the UI via I / O interface 710.

[0209] In some embodiments, the programs are implemented as functions of the programs stored in a non-transitory computer readable recording medium. Examples of the non-transitory computer readable recording medium include, but are not limited to, external / removable and / or internal / embedded storage or memory units, such as one or more of the following: optical discs, such as DVDs; magnetic discs, such as hard discs; semiconductor memories, such as ROMs, RAMs, memory cards, and the like.

[0210] In some aspects, a processing device for quantization of numerical data includes a memory and a processing circuit coupled with the memory. In some embodiments, the processing circuit is to: determine a maximum exponent from a set of exponents of a set of digital representations of a set of numbers; obtain a set of scaled exponents based on subtracting the maximum exponent from each of the set of exponents; and perform one of: (i) obtain a set of quantized significands based on the set of digital representations of the set of numbers and the set of scaled exponents, or (ii) obtain a set of quantized exponents based on the set of scaled exponents. In some embodiments, the processing circuit is to: output a set of quantized digital representations of the set of numbers to the memory based on the set of quantized significands or based on the set of quantized exponents and the set of scaled exponents; and output an exponent scaling factor of an offset associated with the set of quantized digital representations to the memory based on the maximum exponent.

[0211] In some embodiments, the processing circuitry to obtain the set of quantized significands is further to: obtain a set of shifted significands based on the set of significands and the set of scaled exponents. Round the set of shifted significands to a target bit length to become the set of quantized significands.

[0212] In some embodiments, the processing circuitry to obtain the set of quantized significands is further to: convert the set of significands to a set of significands based on restoration of a first non-zero significand digit to each of the set of significands. Shift the set of significands right by a corresponding number of bits indicated by the set of scaled exponents to become the set of shifted significands.

[0213] In some embodiments, the target bit length is 7 bits.

[0214] In some embodiments, the set of quantized digits comprises a set of two's complement integers of the set of quantized significands based on a set of sign bits of the set of digit representations of the set of numbers.

[0215] In some embodiments, each of the set of two's complement integers has a bit length of 8 bits.

[0216] In some embodiments, the processing circuitry to obtain the set of scaled exponents is further to: obtain an unscaled exponent scaling factor based on subtracting a target offset from a maximum exponent. Obtain the set of scaled exponents based on subtracting the unscaled exponent scaling factor from each of the set of exponents, each exponent in the set of scaled exponents having a target exponent bit length.

[0217] In some embodiments, the target exponent bit length ranges from 5 bits to 2 bits.

[0218] In some embodiments, the processing circuitry to obtain the set of quantized significands is further to: round the set of significands to a target significand bit length to become the set of quantized significands.

[0219] In some embodiments, the target significand bit length ranges from 3 bits to 1 bit.

[0220] In some embodiments, each of the set of quantized digits comprises a corresponding sign bit of a set of sign bits of the set of digit representations of the set of numbers, a corresponding scaled exponent of the set of scaled exponents, and a corresponding significand of the set of quantized significands.

[0221] In some embodiments, each of the set of quantized digit representations has a bit length of 8 bits, each of the set of scaled exponents has a bit length of 4 bits, and each of the set of quantized mantissas has a bit length of 3 bits. Each of the set of quantized digit representations has a bit length of 8 bits, each of the set of scaled exponents has a bit length of 5 bits, and each of the set of quantized mantissas has a bit length of 2 bits. Each of the set of quantized digit representations has a bit length of 6 bits, each of the set of scaled exponents has a bit length of 2 bits, and each of the set of quantized mantissas has a bit length of 3 bits. Each of the set of quantized digit representations has a bit length of 6 bits, each of the set of scaled exponents has a bit length of 3 bits, and each of the set of quantized mantissas has a bit length of 2 bits. Each of the set of quantized digit representations has a bit length of 4 bits, each of the set of scaled exponents has a bit length of 2 bits, and each of the set of quantized mantissas has a bit length of 1 bit.

[0222] In some embodiments, the exponent scaling factor of the offset has a bit length of 8 bits.

[0223] In some aspects, a processing device for dequantization of numeric data includes a memory and a processing circuit coupled with the memory. In some embodiments, the processing circuit is to: extract a mantissa of a dequantized digit representation of numeric data, the mantissa being a result of processing a first set of numbers of a first set of quantized digit representations and a second set of numbers of a second quantized digit representation; extract an exponent adjustment from the numeric data; obtain a combined exponent scaling factor based on a first exponent scaling factor associated with the first set of quantized digit representations and a second exponent scaling factor associated with the second quantized digit representation; obtain an unbiased exponent of the numeric data based on the combined exponent scaling factor and the exponent adjustment; and output the dequantized digit representation of the numeric data to the memory. In some embodiments, the dequantized digit representation includes the mantissa of the dequantized digit representation, and an exponent of the dequantized digit representation based on the unbiased exponent of the numeric data.

[0224] In some embodiments, the processing circuit is further to: identify a first non-zero significant digit of the numeric data. The mantissa is further extracted based on removal of the first non-zero significant digit from the numeric data, and the exponent adjustment is further extracted based on a digit position of the first non-zero significant digit within the numeric data.

[0225] In some embodiments, the unbiased exponent is obtained based on addition of the combined exponent scaling factor and the exponent adjustment. The combined exponent scaling factor is an unbiased exponent value.

[0226] In some embodiments, the unbiased exponent of the numeric data is obtained based on an addition of a maximum product exponent, a combined exponent scaling factor, and an exponent adjustment, of the first set of quantized digit representations and the second set of quantized digit representations. The maximum product exponent is an unbiased exponent value, and the combined exponent scaling factor is another unbiased exponent value.

[0227] In some aspects, a method of quantizing numeric data includes determining a maximum exponent from a set of exponents of a set of digit representations of a set of numbers; obtaining, by the processing circuit, a set of scaled exponents based on subtracting the maximum exponent from each of the set of exponents; and performing, by the processing circuit, one of: (i) obtaining a set of quantized significands based on a set of significands of the set of digit representations and the set of scaled exponents, or (ii) obtaining a set of quantized exponents based on the set of significands. In some embodiments, the method includes outputting a set of quantized digit representations of the set of numbers based on the set of quantized significands or based on the set of quantized exponents and the set of scaled exponents; and outputting an offset exponent scaling factor associated with the set of quantized digit representations based on the maximum exponent.

[0228] In some embodiments, obtaining the set of quantized significands includes obtaining a set of shifted significands based on the set of significands and the set of scaled exponents. Rounding the set of shifted significands to a target bit length to become the set of quantized significands.

[0229] In some embodiments, obtaining the set of scaled exponents includes obtaining an unbiased exponent scaling factor based on subtracting a target offset from the maximum exponent. Obtaining the set of scaled exponents based on subtracting the unbiased exponent scaling factor from each of the set of exponents, each exponent of the set of scaled exponents having a target exponent bit length.

[0230] In some aspects, a method of dequantizing numeric data includes extracting a significand of a dequantized digit representation of numeric data, the significand being a result of processing a first set of quantized digit representations of a first set of numbers and a second set of quantized digit representations of a second set of numbers; extracting an exponent adjustment from the numeric data; obtaining, by the processing circuit, a combined exponent scaling factor based on a first exponent scaling factor associated with the first set of quantized digit representations and a second exponent scaling factor associated with the second set of quantized digit representations; obtaining, by the processing circuit, an unbiased exponent of the numeric data based on the combined exponent scaling factor and the exponent adjustment; and outputting the dequantized digit representation of the numeric data. In some embodiments, the dequantized digit representation includes a significand of a significand of the digit representation, and an exponent of the dequantized digit representation based on the unbiased exponent of the numeric data.

[0231] In some aspects, a processing device for quantization of numeric data includes a maximum value finder configured to determine a maximum exponent from a set of exponents of a set of digit representations of a set of numbers, an extractor to obtain a set of scaled exponents based on a subtraction of the maximum exponent from each of the set of exponents, and a significand generator to (i) obtain a set of quantized significands based on the set of digit representations of the set of numbers and a set of mantissas of the set of scaled exponents, or (ii) obtain a set of quantized mantissas based on the set of mantissas. In some embodiments, the processing device includes a data format converter to output a set of quantized digit representations of the set of numbers based on the set of quantized significands or based on the set of quantized mantissas and the set of scaled exponents, and output an exponent scaling factor of an offset associated with the set of quantized digit representations based on the maximum exponent.

[0232] In some aspects, a processing device for dequantization of numeric data includes a mantissa extractor to extract a mantissa of a dequantized digit representation of numeric data, the mantissa being a result of processing a first set of quantized digit representations of a first set of numbers and a second set of quantized digit representations of a second set of numbers, an exponent adjustment extractor to extract an exponent adjustment from the numeric data, a first adder to obtain a combined exponent scaling factor based on a first exponent scaling factor associated with the first set of quantized digit representations and a second exponent scaling factor associated with the second set of quantized digit representations, a second adder to obtain an unbiased exponent of the numeric data based on the combined exponent scaling factor and the exponent adjustment, and a data format converter to output the dequantized digit representation of the numeric data. In some embodiments, the dequantized digit representation includes the mantissa of a dequantized digit representation, and an exponent of the dequantized digit representation based on the unbiased exponent of the numeric data.

[0233] The foregoing outlines features of a few embodiments or examples so that those skilled in the art can better understand the present disclosure an embodiment. Those skilled in the art should appreciate that they can readily use the present disclosure an embodiment as a basis for designing or modifying other processes and structures for carrying out the same purposes and / or achieving the same advantages of the embodiments or examples introduced herein. Those skilled in the art should also realize that such equivalent constructions do not depart from the spirit and scope of the present disclosure an embodiment, and that they can make various changes, substitutions and alterations herein without departing from the spirit and scope of the present disclosure an embodiment.

Claims

1. A processing device for quantization of numerical data, characterized in that, Comprising: a memory; and processing circuitry coupled with the memory and configured to: determine a maximum exponent from a set of exponents of a set of digit representations of a set of numbers; obtain a set of scaled exponents based on subtracting the maximum exponent from each of the set of exponents; perform one of: obtain a set of quantized significands based on a set of significands of the set of digit representations of the set of numbers and the set of scaled exponents, or obtain a set of quantized exponents based on the set of significands; output a set of quantized digit representations of the set of numbers to the memory based on the set of quantized significands or based on the set of quantized exponents and the set of scaled exponents; and output an exponent scaling factor of an offset associated with the set of quantized digit representations to the memory based on the maximum exponent. The processing circuitry to obtain the set of quantized significands is further configured to:

2. The processing device of claim 1, wherein, obtain a set of shifted significands based on the set of significands and the set of scaled exponents; and round the set of shifted significands to a target bit length to become the set of quantized significands. The processing circuitry to obtain the set of quantized significands is further configured to:

3. The processing device of claim 2, wherein, convert the set of significands to a set of significands based on restoration of a first non-zero significand digit to each of the set of significands; and shift the set of significands right by a corresponding number of bits indicated by the set of scaled exponents to become the set of shifted significands. The set of quantized digit representations comprises:

4. The processing device of claim 1, wherein, a set of two's complement integers of the set of quantized significands based on a set of sign bits of the set of digit representations of the set of numbers. The processing circuitry to obtain the set of scaled exponents is further configured to:

5. The processing device of claim 1, wherein, obtain an unbiased exponent scaling factor based on subtracting a target offset from the maximum exponent; and obtain the set of scaled exponents based on subtracting the unbiased exponent scaling factor from each of the set of exponents, each exponent in the set of scaled exponents having a target exponent bit length. Comprising: a memory; and 6. A processing device for inverse quantization of numerical data, characterized by, processing circuitry coupled with the memory and configured to: extract a significand of a dequantized digit representation of a numerical data, the significand being a result of processing a first set of quantized digit representations of a first set of numbers and a second set of quantized digit representations of a second set of numbers; extract an exponent adjustment from the numerical data; obtain a combined exponent scaling factor based on a first exponent scaling factor associated with the first set of quantized digit representations and a second exponent scaling factor associated with the second set of quantized digit representations; obtain an unbiased exponent of the numerical data based on the combined exponent scaling factor and the exponent adjustment; and output the dequantized digit representation of the numerical data to the memory, the dequantized digit representation comprising the significand of the dequantized digit representation, and an exponent of the dequantized digit representation based on the unbiased exponent of the numerical data. The processing circuitry is further configured to: identify a first non-zero significand digit of the numerical data, wherein 7. The processing device of claim 6, wherein, the significand is further extracted based on removal of the first non-zero significand digit from the numerical data, and ​ ​ ​ The index adjustment is further based on extracting a position of the first non-zero significant digit within the numeric data.

8. The processing device of claim 6, wherein, The unbiased exponent is obtained based on an addition of the combined exponent scaling factor and the index adjustment; and The combined exponent scaling factor is an unbiased exponent value.

9. The processing device of claim 6, wherein, The unbiased exponent of the numeric data is obtained based on an addition of a maximum product exponent of the first set of quantized digit representations and the second set of quantized digit representations, the combined exponent scaling factor, and the index adjustment, The maximum product exponent is an unbiased exponent value, and The combined exponent scaling factor is another unbiased exponent value.

10. A method of quantizing numerical data, characterized by, comprising: determining a maximum exponent from a set of exponents of a set of digit representations of a set of numbers; obtaining, by a processing circuit, a set of scaled exponents based on subtracting the maximum exponent from each of the set of exponents; performing, by the processing circuit, one of: obtaining a set of quantized significands based on a set of significands of the set of digit representations and the set of scaled exponents, or obtaining a set of quantized significands based on the set of significands; outputting a set of quantized digit representations of the set of numbers based on the set of quantized significands or based on the set of quantized significands and the set of scaled exponents; and outputting, based on the maximum exponent, an offset exponent scaling factor associated with the set of quantized digit representations. ​

Images