Nanoscaling floating -point for large language models

Abstract

Block decoding in an artificial neural network is provided. An encoded block comprising a plurality of encoded values, each encoded value comprising a mantissa, is read. The encoded block's scaling information, which includes an exponent and a mantissa, is read. Each of the plurality of encoded values is decoded according to the scaling information to produce a plurality of decoded values.

Description

HCU-07325NANOSCALING FLOATING-POINT FOR LARGE LANGUAGE MODELSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application Nos. 63 / 683095, filed August 14, 2024, and 63 / 704314, filed October 7, 2024, each of which is hereby incorporated by reference in its entirety.BACKGROUND

[0002] Embodiments of the present disclosure relate to block decoding, and more specifically, to augmenting block floating-point for machine learning using a mantissa, an adaptive exponent, and / or code recycling.BRIEF SUMMARY

[0003] According to embodiments of the present disclosure, methods and systems of and computer program products for block decoding in an artificial neural network are provided. An encoded block comprising a plurality of encoded values, each encoded value comprising a mantissa, is read. The encoded block’s scaling information, which includes an exponent and a mantissa, is read. Each of the plurality of encoded values is decoded according to the scaling information to produce a plurality of decoded values.BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0004] Fig. 1A depicts an exemplary block floating-point format.

[0005] Fig. IB depicts an exemplary microscaling floating-point format.

[0006] Fig. 1C depicts an encoded block and its scaling information according to various embodiments of the present disclosure.Page 1 of 31FOLEYHOAGUS12475408.1HCU-07325

[0007] Fig. 2 depicts the quantization of a floating-point vector using an exemplary microscaling floating-point format.

[0008] Fig. 3 is a graph illustrating the performance of the microscaling floating-point format in the low-bit realm, as exemplified by the profiled distribution of scaled weights for several large language models.

[0009] Fig. 4 depicts an example quantization of a floating-point vector according to the microscaling floating-point format and according to an exemplary embodiment of the present disclosure.

[0010] Fig. 5A depicts example quantizations of floating-point vectors according to exemplary microscaling and block floating-point formats, with vacant quantization levels indicated by red crosses.

[0011] Fig. 5B depicts the augmentation of a mantissa vector using an exponent, according to various embodiments of the present disclosure.

[0012] Fig. 5C depicts floating-point formats corresponding to two possible exponent values, according to various embodiments of the present disclosure.

[0013] Fig. 6 depicts a code recycling scheme according to various embodiments of the present disclosure.

[0014] Fig. 7 presents an algorithm for quantizing a full-precision vector into a low-bit nanoscaling vector, according to various embodiments of the present disclosure.

[0015] Fig. 8 depicts an on-the-fly dequantization workflow for deployment on exemplary hardware, according to various embodiments of the present disclosure.

[0016] Fig. 9 depicts quantization error improvements of various embodiments of the present disclosure over the microscaling floating-point format.Page 2 of 31FOLEYHOAGUS12475408.1HCU-07325

[0017] Fig. 10 presents weight-only quantization results for the block floating-point format, the microscaling floating-point format, and various embodiments of the present disclosure on various large language models, using various weights and activations.

[0018] Fig. 11A depicts the perplexity -to-memory -footprint tradeoffs of weight-only quantization on the large language model Llama3-8B, using the block floating-point format, the microscaling floating-point format, and various embodiments of the present disclosure.

[0019] Fig. 11B depicts the perplexity -to-memory -footprint tradeoffs of weight-and-KV- cache quantization on the large language model Llama3-8B, using the block floating-point format, the microscaling floating-point format, and various embodiments of the present disclosure.

[0020] Fig. 11C depicts the perplexity -to-memory -footprint tradeoffs of weight-only quantization on the large language model Llama2-7B, using the block floating-point format, the microscaling floating-point format, and various embodiments of the present disclosure.

[0021] Fig. 11D depicts the perplexity -to-memory -footprint tradeoffs of weight-and-KV- cache quantization on the large language model Llama2-7B, using the block floating-point format, the microscaling floating-point format, and various embodiments of the present disclosure.

[0022] Fig. 12 depicts the accuracy degradation of various large language models on the MMLU-Social Science dataset, using the block floating-point format, the microscaling floating-point format, and various embodiments of the present disclosure.

[0023] Fig. 13A depicts the perplexities of various remapped values after code recycling for the microscaling floating-point format on Llama3 (8B).

[0024] Fig. 13B depicts the perplexities of various remapped values after code recycling for the block floating-point format on Llama3 (8B).Page 3 of 31FOLEYHOAGUS12475408.1HCU-07325

[0025] Fig. 14 depicts perplexity-to-memory-footprint tradeoffs under different block sizes on Llama3 (8B), using the block floating-point format, the microscaling floating-point format, and various embodiments of the present disclosure.

[0026] Fig. 15 depicts the accuracy degradation of various large language models on the Arc- Challenge dataset, using the block floating-point format, the microscaling floating-point format, and various embodiments of the present disclosure.

[0027] Fig. 16 depicts weight-only quantization results for the block floating-point format, the microscaling floating-point format, and various embodiments of the present disclosure on reasoning tasks of Gemma2 (2B), using various weights.

[0028] Fig. 17 depicts weight-only quantization results for the block floating-point format, the microscaling floating-point format, and various embodiments of the present disclosure on reasoning tasks of Llama2 (7B), using various weights.

[0029] Fig. 18 depicts weight-only quantization results for the block floating-point format, the microscaling floating-point format, and various embodiments of the present disclosure on reasoning tasks of Llama3 (8B), using various weights.

[0030] Fig. 19 depicts weight-only quantization results for the block floating-point format, the microscaling floating-point format, and various embodiments of the present disclosure on reasoning tasks of Llama3.1 (8B), using various weights.

[0031] Fig. 20 depicts weight-only quantization results for the block floating-point format, the microscaling floating-point format, and various embodiments of the present disclosure on reasoning tasks of Mistral (7B), using various weights.

[0032] Fig. 21 depicts weight-only quantization results for the block floating-point format, the microscaling floating-point format, and various embodiments of the present disclosure on reasoning tasks of Llama2 (13B), using various weights.Page 4 of 31FOLEYHOAGUS12475408.1HCU-07325

[0033] Fig. 22A depicts the perplexities of various remapped values after code recycling for the microscaling floating-point format on Llama3.1 (8B).

[0034] Fig. 22B depicts the perplexities of various remapped values after code recycling for the block floating-point format on Llama3.1 (8B).

[0035] Fig. 23A depicts the perplexities of various remapped values after code recycling for the microscaling floating-point format on Llama2 (7B).

[0036] Fig. 23B depicts the perplexities of various remapped values after code recycling for the block floating-point format on Llama2 (7B).

[0037] Fig. 24A depicts an exemplary MSE-based quantization method according to embodiment of the present disclosure.

[0038] Fig. 24B depicts another exemplary threshold-based quantization method according to embodiment of the present disclosure.

[0039] Fig. 25 is chart providing a perplexity comparison of MSE-based NxFP with threshold-based NxFP quantization.

[0040] Fig. 26A is a flowchart illustrating a method to decompress a nanoscaling format according to embodiment of the present disclosure.

[0041] Fig. 26B depicts an NxFP-enabled LLM accelerator for dequantization according to embodiment of the present disclosure.

[0042] Fig. 27 is a table analyzing the perplexity degradation of quantizing both weights and activations on modern LLMs.

[0043] Fig. 28 is a graph is provided comparing the MMMU accuracy of Llama3.2-vision using 3 -bit MxFP and NxFP.

[0044] Fig. 29 is a graph providing a runtime comparison of MxFP, MSE-based NxFP quantization, and threshold-based NxFP quantization algorithms.Page 5 of 31FOLEYHOAGUS12475408.1HCU-07325

[0045] Fig. 30 is a graph providing a runtime comparison of MxFP and NxFP when KV cache is quantized.

[0046] Fig. 31 contains graphs illustrating the perplexity of different remapped values on code-recycled MxFP4 for Llama3-8B.

[0047] Fig. 32 is a graph showing the perplexity of NxFP by threshold-based quantization flow and MSE-based quantization flow.

[0048] Fig. 33 is a flowchart illustrating a method of block decoding in an artificial neural network according to embodiments of the present disclosure.

[0049] Fig. 34 is a schematic view of a computing node according to embodiments of the present disclosure.DETAILED DESCRIPTION

[0050] Large language models (LLMs), which are transforming various industries, suffer from memory traffic and capacity challenges when applied to large datasets. To improve the scalability of LLMs, a microscaling format (MxFP), where the conventional block floatingpoint format (BFP) is augmented with exponents, may be employed. However, microscaling degrades the quality of generative inference when the bitwidths are lower than 6. Examples of low-bit floating-point formats include FP16, BF16, and adaptivFloat. Various artificial intelligence hardware, including Google’s TPUs and NVIDIA’ s Al 00 GPUs, utilize such low-bit floating-point formats for improved efficiency, specifically reduced memory footprint or accelerated token generation speed. To give an example of microscaling’s performance degradation in this low-bit realm, quantizing both weights and activations of LLaMA-7B using MxFP4 degrades the perplexity from 9.488 (FP16) to 27.201 (MxFP4).Page 6 of 31FOLEYHOAGUS12475408.1HCU-07325

[0051] Therefore, there remains a need for methods that preserve perplexity in various maching learning models such as generative language models, and there is a particular need for such methods that are compatible with sub-6-bit microscaling standards.

[0052] The present disclosure provides a “nanoscaling” floating-point datatype (NxFP), which addresses this need for models including modern LLMs. In various embodiments, the microscaling format is augmented using a mantissa and an exponent in the scaling information. Some embodiments include recycling code via the remapping of the binary code representing -0 to a more useful value. NxFP was formulated based on the profiling of modern LLMs and the identification of challenges of low-bit microscaling formats, including vacant quantization levels, inefficient approximation of outliers, and wasted code. Experimental results across various modern LLMs and deployment scenarios demonstrate that NxFP outperforms the microscaling standard in memory footprint, quantization error, perplexity, and reasoning accuracy.

[0053] For instance, NxFP improves the perplexity by up to 0.64 at 4-bit precision, or reduces the memory footprint by up to 16% while maintaining a perplexity degradation of less than 0.1. NxFP also improves accuracy by up to 30% on the MMLU benchmark. In addition, NxFP allows for efficient on-the-fly dequantization, quantization, and structural memory layout, aiding in NxFP deployment. Regarding quantization, various alternative approaches provide either calibration-based quantization or finetuning to deal with outliers. However, such quantization methods generally introduce deployment friction and the risk of overfitting on the calibration set. NxFP, on the other hand, focuses on direct quantization without calibration or finetuning, thus minimizing deployment friction and overfitting. Such direct-cast inference also directly reflects the efficiencies of different number systems. The number system proposed in the present disclosure is orthogonal to and can be integrated with many post-training quantization techniques.Page 7 of 31FOLEYHOAGUS12475408.1HCU-07325

[0054] Regarding the experiments described herein, the quantized LLMs are evaluated on language generative performance using perplexity on the Wikitext2 dataset for its ability to reflect the performance of LLMs. The best perplexity values are reported. The reasoning capabilities of the quantized LLMs are also evaluated on the MMLU and CommonSenseQA benchmarks. All evaluations are run on servers using NVIDIA’ s Al 00 GPU with 80 GB HBM.

[0055] With reference now to Fig. 1A, a schematic of an encoded block and its associated scaling information according to the block floating-point format (BFP) is shown. Each block 101 includes a plurality of encoded values 102, where each encoded value comprises a mantissa 103 and a sign 104. The scaling information comprises a shared exponent 105. The sharing of the scaling information across the vector of mantissas allows for low area overheads.

[0056] Referring now to Fig. IB, a schematic of an encoded block and its associated scaling information according to the microscaling floating-point format (MxFP) is shown. Each block 111 includes a plurality of encoded values 112, where each encoded value comprises a mantissa 115, a sign 113, and additionally an exponent 114 for element-wise scaling, called a “microexponent.” This augmentation of the block floating-point datatype with microexponents (Eg) as second-level, per-element scaling factors is the key of the microscaling format. Lastly, the scaling information associated with the encoded block comprises a shared exponent 116.

[0057] Open Compute Project (OCP) has defined four standards in the microscaling family, MxINT8, MxFP8 (E5M2 / E4M3), MxFP6 (E3M2 / E2M3), and MxFP4 (E2M1), where the suffix denotes the element format and the bitwidth. MxFP8 and MxFP6 have defined multiple optimized configurations to optimize for different scenarios, for example, MxFP8 with 4-bit exponent and 3 -bit mantissa (E4M3) prioritizes precision and is optimized for inference. InPage 8 of 31FOLEYHOAGUS12475408.1HCU-07325 relation to the block floating-point format (BFP), MxINT8 is fundamentally BFP but with sign magnitude replaced by two’s complement. Specifically, MxFP augments conventional BFP with microexponents by incorporating two exponent fields per vector: one shared exponent and multiple microexponents. The shared exponent is responsible for vector-wise scaling, and the microexponents serve as element-wise scaling.

[0058] Referring now to Fig. 1C, a schematic of an encoded block and its associated scaling information according to the nanoscaling floating-point format of the present disclosure, is shown. Each block 121 includes a plurality of encoded values 122, where each encoded values comprise a mantissa 125, a sign 123, and a microexponent 124. The scaling information comprises a shared exponent 126 and additionally a mantissa 127, called a “nanomantissa,” and a flag 128, indicating whether the microexponent should be applied, thus providing an “adaptive microexponent.” In some embodiments, code recycling is performed. Specifically, binary code representing -0 is remapped to more useful values, e.g., half of the smallest quantization level, to improve the quantization error.

[0059] Fig. 2 depicts the quantization of an example FP16 floating-point vector according to the microscaling floating-point format. First, the mantissa binary corresponding to the FP16 vector is obtained, and each mantissa is aligned with the most significant exponent value. Second, the shared exponent is configured to be the largest exponent value in the vector. Third, the microexponents serve as exponent offsets to track each element’s leading one. Finally, the mantissa slices the trailing bit while assuming the leading one exists.

[0060] In Fig. 3, the performance of the microscaling floating-point format (MxFP) in the low-bit realm is illustrated. There are several challenges of low-bit MxFP, including vacant quantization levels, inaccurate tracking of largest values, and wasted code. To understand these challenges, the distribution of weight values was profiled after scaling by Eshared for each 32-wide weight vector on the modern LLMs Llama3, Llama3.1, Phi3, Llama2, andPage 9 of 31FOLEYHOAGUS12475408.1HCU-07325Mistral. As shown, the scaled weights have a normal distribution ranging from -8 to +8, with a relatively large standard deviation. The element format, FP4, quantizes these full-precision values into quantization levels ranging from -6 to +6. The first challenge of low-bit MxFP is that the shared exponent cannot efficiently scale the weight value to capture the largest value. For example, the maximum quantization level of MxFP4 is 6, which can hardly quantize weights with values 7~8. Second, since MxFP4 allocates more quantization levels near zero, there are vacant quantization levels between 4 and 6. Values in this range cannot be quantized efficiently, leading to significant quantization errors. Third, MxFP4 adopts sign-magnitude and wastes two binary codes to represent -0 and +0, where the overheads grow as the bitwidth is scaled down.

[0061] Fig. 4 presents an example of quantizing a Llama3 vector using (a) pure MxFP4 and (b) MxFP4 with nanomantissa. In various embodiments of the present disclosure, a 2 -bit mantissa field (a nanomantissa) may be incorporated in the shared scaling factor to help low- bit MxFP accurately track the largest value. Conceptually, the shared nanomantissa helps low-bit MxFP formats boost their precision and track the full-precision values more accurately. In the example shown, while the most considerable scaled weight value is -7.4, MxFP can only approximate it using -6 (-1.5 x 22). If MxFP4 is enabled with a nanomantissa, the nanomantissa can be configured to scale the entire vector by 1.25, which best approximates -7.4 using -7.5. As a result, the LI quantization error is significantly reduced from 1.4 to 0.1. To set the nanomantissa value, the value can be nullified by setting it to zero or calculated via the quantization algorithm presented in Fig. 7.

[0062] Referring now to Fig. 5A, example quantizations of floating-point vectors according to the microscaling and block floating-point formats are shown, with vacant quantization levels indicated by crosses. As shown, different blocks have distinct distributions and levels of clustering in their values. Each block is quantized using MxFP4 and BFP4 to show that thePage 10 of 31FOLEYHOAGUS12475408.1HCU-07325 issue of vacant quantization levels arises when there is a mismatch between the low-bit format and the per-block distribution. When quantizing value-clustered block 1 (Bl), BFP4 outperforms MxFP4 because the latter suffers from vacant quantization levels around 5, as denoted by the red crosses in MxFP4. When quantizing value-scattered block 2 (B2), MxFP4 outperforms BFP4 because BFP4 suffers from a more significant quantization error around 0, as denoted by the red crosses in BFP4.

[0063] Fig. 5B depicts the augmentation of a microscaling vector through the use of an exponent, an “adaptive microexponent,” according to various embodiments of the present disclosure. The adaptive microexponent is implemented through an index bit that indicates whether or not a block uses a microexponent. Specifically, microexponent fields are only allocated for vectors with scattered distributions. This adaptive microexponent system gives each block the flexibility to select its most suitable format and to choose between MxFP and BFP, and thereby mitigates the challenge of vacant quantization levels.

[0064] In Fig. 5C, two possible formats that each block can take, according to various embodiments of the present disclosure, are illustrated. When a given vector is clustered in value, the index bit can be set as 0 to choose the BFP format, which allocates all the bits to be mantissas. This format optimizes precision. Conversely, when a vector contains outliers, setting the index bit as 1 configures the block as MxFP, ensuring a great dynamic range and more quantization levels around zero.

[0065] Referring now to Fig. 6, a code recycling workflow, according to various embodiments of the present disclosure, is depicted. Code recycling mitigates the issue of wasted code in sign-magnitude format. The binary code representing -0 in sign-magnitude numbers is remapped to a useful value, thereby improving the quantization error. The arrows, as shown, illustrate the process of sweeping for different remapped values. A remapped value should simultaneously improve the perplexity and have low implementation overheads. ThePage 11 of 31FOLEYHOAGUS12475408.1HCU-07325 remapped value is set to be 2xbased on the empirical profiling of three representative LLMs, z.e., Llama2, Llama3, and Llama3.1. During the dequantization, the smallest number can be right-shifted by one bit to obtain the remapped value during decoding.

[0066] Referring now to Fig. 7, an algorithm for quantizing a full-precision vector into a low- bit nanoscaling vector, according to various embodiments of the present disclosure, is shown. First, the largest absolute valueis found from the full -preci si on vectorSecond, the largest exponentis found. Third, the largest absolute value is normalized and rounded to get a 2 -bit nanomantissa for scaling. Fourth, the original vector is scaled and quantized into MxFP and BFP formats. Then, the quantization error (MSE) is evaluated to determine which format is better, and the format indicator flagis set. The above quantization is repeated with the nanomantissa set to zero to get the MSE-optimized binary. It is noted that the quantization function contains code recycling, which maps -0 to be half of the smallest quantization level. Also, this quantization flow can be adjusted to satisfy different scenarios based on the user's requirements. In terms of applications, this quantization flow can be used for NxFP deployment on various hardware.

[0067] Referring now to Fig. 8, an on-the-fly dequantization workflow for efficient deployment on exemplary hardware, according to various embodiments of the present disclosure, is shown. First, the format index bit ( ) is used to identify the format of a given block and slice out the corresponding fields. Second, the wasted code is decoded. Specifically, if the element code is 10...0, it is remapped to half of the smallest value by rightshifting the smallest quantization value by one bit. Third, the nanomantissa is multiplied with the element-wise mantissa to get the expanded mantissa field. Fourth, the microexponents are added to the shared exponent to form the expanded exponent field, as multiplying two power- of-two numbers equates to summing the exponents. Fifth, for conversion to BF16 (Is8e7m)Page 12 of 31FOLEYHOAGUS12475408.1HCU-07325 or FP16, the mantissa and exponent are properly padded with zeros as least-significant bits. Lastly, computation is initiated using appropriate instructions for different platforms, such as calling MatrixMultiply() on Google TPU or tdp() on the Intel Xeon 6 processor.

[0068] Referring now to Fig. 9, the quantization error (MSE) improvement of NxFP4 over MxFP4 using direct-cast quantization, where the benefits of different techniques can be isolated, is shown. Nanomantissa (NM), adaptive microexponent (NM+AM), and code recycling (NM+AM+CR) techniques were cumulatively added to isolate their benefits. In brief, NxFP4 reduces the quantization error by up to 45%. The quantization error improvement is similar across all models. The contribution of each technique can be further isolated as follows. First, the nanomantissa helps reduce the quantization error by up to 26%. Then, the adaptive microexponent further reduces the quantization error by 14%. Finally, code recycling helps reduce the quantization error by 4.7%.

[0069] Referring now to Fig. 10, analysis of the perplexity degradation of weight-only quantization for the block floating-point format, the microscaling floating-point format, and various embodiments of the present disclosure on various modern large language models, using various weights and activations, is shown. The best perplexity values are bolded. NxFP is compared against BFP and MxFP, where NxFP consistently achieves the best perplexity under 4~6 bits. The three tested NxFP techniques, highlighted in green, successfully improve the perplexity, especially at ultra-low bits (z.e., 4-bit). Since the trend is similar across different models, Llama3 can be used as a main illustrative example. The original FP16 perplexity is 6.14. Under 6-bit weight, MxFP (6.18) and BFP (6.21) degrade the perplexity by 0.4 and 0.7, respectively. On the other hand, NxFP (6.17) achieves the smallest perplexity degradation (0.03). Similarly, BFP5 (6.38) and MxFP5 (6.31) degrade perplexity by 0.24 and 0.17, respectively. NxFP5 (6.25), again, achieves the smallest perplexity degradation of 0.11.Page 13 of 31FOLEYHOAGUS12475408.1HCU-07325Finally, using 4-bit weight, the perplexity degradation of BFP, MxFP, and NxFP are 0.93, 0.81, and 0.43, respectively.

[0070] Referring now to Fig. 11 A, the perplexity -to-footprint Pareto frontier of NxFP with standard MxFP and BFP on Llama3 (8B) is shown. The zone of < 0.1 perplexity degradation compared to FP16 is highlighted in green. NxFP consistently improves the perplexity using a smaller model footprint. First, NxFP4 can significantly reduce the perplexity by 0.38 compared to MxFP4. Second, NxFP5 reduces 0.93 GB (13%) memory footprint with only 0.08 perplexity degradation compared to MxFP6. Third, NxFP5 can satisfy the < 0.1 perplexity degradation requirement and reduces 10 GB footprint compared to FP16.

[0071] Fig. 11B presents the tradeoffs when quantizing both the weights and the KV cache. The weights and KV cache specifically were quantized because they generally dominate the memory footprint. The zone of < 0.1 perplexity degradation compared to FP16 is highlighted in green. First, NxFP4 can significantly reduce the perplexity by 0.64 compared to MxFP4. Furthermore, NxFP5 reduces the memory footprint by 1.02 GB (16%) with only 0.09 perplexity degradation compared to MxFP6, which nearly satisfies the < 0.1 perplexity degradation requirement compared to FP16.

[0072] Fig. 11C depicts the perplexity -to-memory -footprint tradeoffs of weight-only quantization on the large language model Llama2 (7B), using the block floating-point format, the microscaling floating-point format, and various embodiments of the present disclosure. The zone of < 0.1 perplexity degradation compared to FP16 is highlighted in green.Compared to MxFP6, NxFP5 can reduce the memory footprint by 0.78 GB (13%).

[0073] Fig. 11D depicts the perplexity-to-memory-footprint tradeoffs of weight-and-KV- cache quantization on the large language model Llama2 (7B), using the block floating-point format, the microscaling floating-point format, and various embodiments of the present disclosure. The zone of < 0.1 perplexity degradation compared to FP16 is highlighted inPage 14 of 31FOLEYHOAGUS12475408.1HCU-07325 green. Compared to MxFP6, NxFP5 can reduce the memory footprint by 0.85 GB (16%). Compared to FP16, NxFP5 reduces the footprint by 8.43 GB while only degrading the perplexity by 0.05.

[0074] Referring now to Fig. 12, a summary of the accuracy degradation of various LLMs, i.e., Llama3, Llama3.1, Gemma2, Llama2, and Mistral, on MMLU- Social Science is shown. There are three takeaways. First, NxFP significantly mitigates the accuracy degradation on low bitwidths, i.e., 4-bit and 3 -bit. Specifically, NxFP improves the accuracy by up to 30.2% on MMLU- Social Science compared to MxFP and BFP. Second, the smaller language models, e.g., Gemma2-2B, are generally less quantizable than larger language models (Llama2-13B). Third, Mistral is surprisingly the most quantizable model in the 7B category.

[0075] Referring now to Fig. 13A, the perplexity of different remapped values on coderecycled MxFP4 for Llama3-8B is shown. Due to implementation overheads, the middlepoints between the original quantization levels, rather than all remapped values, were swept. The dotted line shows the baseline perplexity of MxFP4. As shown, half of the smallest quantization number and the middle-point between the 1st- and 2nd-largest quantization values improve the perplexity the most.

[0076] Fig. 13B shows the perplexity of different remapped values on code-recycled BFP4 for Llama3-8B. Due to implementation overheads, the middle-points between the original quantization levels, rather than all remapped values, were swept. The dotted line shows the baseline perplexity of BFP4. As shown, half of the smallest quantization number improves the perplexity the most.

[0077] Referring now to Fig. 14, a comparison of the perplexity-to-footprint tradeoffs under different block sizes (BS) on Llama3 (8B) is shown. NxFP4 consistently outperforms MxFP4 and BFP4 regardless of block size. Additionally, MxFP4 outperforms BFP4 when the block size is large. This is because microexponents can ensure sufficient element-wise dynamicPage 15 of 31FOLEYHOAGUS12475408.1HCU-07325 range when the values are much more scattered. In other experiments in this paper, a block size of 32 was used, as is defined in the microscaling standard.

[0078] Referring now to Fig. 15, a summary of the accuracy degradation of various LLMs, i.e., Llama3, Llama3.1, Gemma2, Llama2, and Mistral, on Arc-Challenge tasks is shown. NxFP significantly mitigates the accuracy degradation on low bitwidths, i.e., 4-bit and 3 -bit. Specifically, NxFP improves the accuracy by up to 14.0% compared to MxFP and BFP.

[0079] Referring now to Fig. 16, the detailed accuracy degradation of Gemma2 (2B) on 6- bit / 5-bit / 4-bit / 3-bit weights on the entire MMLU and CommonSenseQA benchmarks is shown. First, it is observed that the 6-bit quantization can preserve the reasoning capabilities of full-precision Gemma2 (2B). Second, NxFP can improve the accuracy by up to 18.2% compared to MxFP and BFP. Third, different tasks have different sensitivities under quantization. Using 3 -bit NxFP, PIQA only degrades accuracy by 4.6% while Arc-Easy degrades the accuracy by 10.7%.

[0080] Referring now to Fig. 17, the detailed accuracy degradation of Llama2 (7B) is similarly shown. It is observed that NxFP can improve the accuracy by up to 10.0% compared to MxFP and BFP.

[0081] Referring now to Fig. 18, the detailed accuracy degradation of Llama3 (8B) is shown. It is observed that NxFP can improve the accuracy by up to 28.3% compared to MxFP and BFP.

[0082] Referring now to Fig. 19, the detailed accuracy degradation of Llama3.1 (8B) is shown. It is observed that NxFP can improve the accuracy by up to 30.2% compared to MxFP and BFP.

[0083] Referring now to Fig. 20, the detailed accuracy degradation of Mistral (7B) is shown. It is observed that NxFP can improve the accuracy by up to 8.0% compared to MxFP and BFP.Page 16 of 31FOLEYHOAGUS12475408.1HCU-07325

[0084] Referring now to Fig. 21, the detailed accuracy degradation of Llama2 (13B) is shown. It is observed that NxFP can improve the accuracy by up to 14.6% compared to MxFP and BFP.

[0085] Referring now to Fig. 22A, the perplexity improvement for MxFP4 when recycling code 10...0 to different values on Llama3.1 (8B) is shown. Setting the remapped value to be t2 smallest quantization value and setting the remapped value to be. 3 . are two of the most beneficial options; they improve the perplexity by up to 0.1.

[0086] Referring now to Fig. 22B, the perplexity improvement for BFP4 when recycling code 10...0 to different values on Llama3.1 (8B) is shown. Setting the remapped value to be 1 smallest quantization value is the most beneficial option; it improves the perplexity by 0.2.

[0087] Referring now to Fig. 23A, the Wikitext2 perplexity for MxFP4 when recycling code J10. . .0 on Llama2 (7B) is shown. Setting the remapped value to be 2 smallest quantization value and setting the remapped value to beare two of the most beneficial options; they improve the perplexity by up to 0.06.

[0088] Referring now to Fig. 23B, the Wikitext2 perplexity for BFP4 when recycling code110. . .0 on Llama2 (7B) is shown. Setting the remapped value to be X smallest quantization value is the most beneficial option; it improves the perplexity by 0.09. Since half of the smallest quantization value is consistently one of the most beneficial remapped values in terms of perplexity improvement across different LLMs, setting this value as the final remapped value is recommended.

[0089] Referring now to Figs. 24A-B, exemplary embodiments of quantization methods are shown. The NxFP format is not bounded to just one quantization algorithm. In somePage 17 of 31FOLEYHOAGUS12475408.1HCU-07325 embodiments, the quantization algorithm may be MSE-based or threshold-based. While minimization of quantization error through selecting quantization configurations that yield the MSE can improve the fidelity of the quantized representation to the original data, an exclusive focus on error minimization may inadvertently lead to overfitting. Overfitting may occur when the quantization process becomes excessively tailored to the specific characteristics of the training or calibration data, capturing noise or outlier behavior that does not generalize well to unseen data. Optimization methods allow for a tolerable level of error, which may include not only minimizing error but also balancing computational complexity, memory footprint, and generalization performance. For example, threshold-based quantization methods may optimize computational efficiency and robustness by tolerating a controlled level of error, rather than strictly minimizing it. When utilizing quantization algorithms, it may be advantageous to optimize for overall system performance rather than to minimize error in isolation, thereby reducing the risk of overfitting and promoting better generalization to diverse data distributions.

[0090] Fig. 24A presents an MSE-based quantization method, which optimizes quantization error. The MSE-based quantization method evaluates multiple potential quantization configurations for each vector (block) and selects the optimized quantization configuration. The system may be quantized using different setups and individually select the MSE- optimized setup for each block.

[0091] Fig. 24B presents a threshold-based quantization method, which optimizes computational complexity. The threshold-based quantization method estimates the severity of outliers and correspondingly quantizes using a suitable format depending on the level of severity. Vectors containing outliers may be quantized using MxFP, while vectors without outliers may be quantized using the BFP format. In some embodiments, the threshold-based quantization methods may require a user to define one or more thresholds to divide inputPage 18 of 31FOLEYHOAGUS12475408.1HCU-07325 signal range into intervals. Optimal threshold values may be dependent on the model architecture, input data, or other considerations. In some embodiments, the optimal threshold range is 1.2-1.6.

[0092] Referring now to Fig. 25, a perplexity comparison is provided of MSE-based NxFP with threshold-based NxFP quantization. This analyzes the perplexity degradation of MxFP4, NVFP4, MSE-based NxFP4, and threshold-based NxFP4 quantization flow. Threshold-based flow achieves promising perplexity improvement while significantly reducing the overheads (as shown in Fig. 29). As shown in Fig. 32, some embodiments have a threshold of 1.4 to optimize perplexity across different models.

[0093] Referring now to Figs. 26A-B, an exemplary dequantization flow of the nanoscaling format NxFP is illustrated. The NxFP format can be enabled using dequantization on either off-the-shelf hardware or future Al accelerator to address the memory wall challenge. This deployment method may reduce memory footprint while maintaining great hardware compatibility. Specifically, the system allows data to be quantized into Mx format for storage in off-chip memory and then decompressed for execution on FP16 / BF16 cores.

[0094] Fig. 26A displays a flowchart of steps taken to decompress nanoscaling format and run off-the-shelf hardware. The decompression steps of NxFP are slicing fields, remapping wasted code, and multiplying NanoMantissa with per-element mantissa. First, the fields are sliced. The format index bit (fmt) is used to identify the format of a given block and slice out the corresponding fields. Next, the wasted code is decoded. If the element code is 10...0, it is remapped to half of the smallest value by right-shifting the smallest quantization value by one bit. Subsequently, the NanoMantissas are multiplied with the element-wise mantissa to get the expanded mantissa field. Next, the microexponents are added to the shared exponent to form the expanded exponent field, as multiplying two power-of-two numbers equates toPage 19 of 31FOLEYHOAGUS12475408.1HCU-07325 summing the exponents. For conversion to BF16 (Is8e7m) or FP16, the mantissa, and exponent are properly padded with zeros as LSBs to perform Matmul.

[0095] Fig. 26B displays a NxFP-enabled LLM accelerator, where dequantization unit only occupies <3% area. In some embodiments, a 16nm FinFET accelerator can support dequantizing from nxFP to BF16 utilizing accustom hardware unit, therefore eliminating the runtime overhead of format conversion. The custom DQU, which is smaller than the SoC, saves space within the computing system. This showcases the efficiency of enabling NxFP on an LLM decoding accelerator. These operations incur low-cost fixed point addition and bitwise logiclogics and can be efficiently accelerated by mapping to vector engines.

[0096] Referring to Fig. 27, a table is provided analyzing the perplexity degradation of quantizing both weights and activations on modern LLMs. The NxFP consistently achieves the smallest perplexity among all bitwidths.

[0097] Referring to Fig. 28, a graph is provided comparing the MMMU accuracy of Llama3.2-vision using 3-bit MxFP and NxFP. The accuracy of NxFP is significantly better than traditional MxFP standard. Specifically, the accuracy of NxFP is 7.5%~20.3% higher than that of MxFP. This trend demonstrates that the NxFP datatype outperforms the low-bit MxFP standard.

[0098] Referring to Fig. 29, a runtime comparison is provided of MxFP, MSE-based NxFP quantization, and threshold-based NxFP quantization algorithms. The time of quantizing different number of blocks using MSE-based NxFP quantization, threshold-based NxFP quantization, and MxFP quantization are analyzed. While MSE-based quantization method optimizes the quantization error and is suitable for quantizing stationary datatype like weights, it requires the largest complexity and runtime overheads. Threshold-based quantization method significantly reduces quantization overheads and achieves MxFP-Page 20 of 31FOLEYHOAGUS12475408.1HCU-07325 comparable quantization complexity. Hence, threshold-based quantization is suitable for quantizing varying datatypes, e.g., KV cache and 558 activations.

[0099] Referring to Fig. 30, a runtime comparison is provided of MxFP and NxFP when KV cache is quantized. End-to-end performance is profiled of an extended SOTA KV cache quantization framework, KIVI, to compare the end-to-end runtime of quantizing KV cache using different datatype. Fig. 30 shows the end-to-end performance when quantizes NxFP and MxFP, which the performance degradation is within 8%.

[0100] Referring to Fig. 31, the perplexity of different remapped values on code-recycled MxFP4 for Llama3-8B is shown. The dotted line shows the baseline perplexity of MxFP4. One half smallest quantization number and the middle-point between the 1st and 2nd-largest- quantization-value improves the perplexity the most. This also shows the perplexity of different remapped values on coderecycled BFP4. The dotted line shows the baseline perplexity of BFP4. One half smallest quantization number improves the perplexity the most.

[0101] Referring to Fig. 32, the perplexity of NxFP by threshold-based quantization flow and MSE-based quantization flow are illustrated. Fig. 32 analyzes different largest-to-2nd-largest ratio to select between BFP / MxFP across three large language models. Although different models have slightly different optimal ratio, it is shown that a ratio of 1.4 can achieve near optimal perplexity across different models. In addition, users can also sweep to determine optimal ratio for new models.

[0102] Referring now to Fig. 33, a flowchart is provided illustrating a method of block decoding in an artificial neural network according to embodiments of the present disclosure. At 3301, an encoded block is read comprising a plurality of encoded values having a first format. Each encoded value comprises a mantissa. At 3302, scaling information associated with the encoded block is read. The scaling information comprises an exponent and aPage 21 of 31FOLEYHOAGUS12475408.1HCU-07325 mantissa. At 3303, each of the plurality of encoded values is decoded according to the scaling information to produce a plurality of decoded values.

[0103] Referring now to Fig. 34 a schematic of an example of a computing node is shown. Computing node 10 is only one example of a suitable computing node and is not intended to suggest any limitation as to the scope of use or functionality of embodiments described herein. Regardless, computing node 10 is capable of being implemented and / or performing any of the functionality set forth hereinabove.

[0104] In computing node 10 there is a computer system / server 12, which is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like.

[0105] Computer system / server 12 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. Computer system / server 12 may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.Page 22 of 31FOLEYHOAGUS12475408.1HCU-07325

[0106] As shown in Fig. 34, computer system / server 12 in computing node 10 is shown in the form of a general -purpose computing device. The components of computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including system memory 28 to processor 16.

[0107] Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Express (PCIe), and Advanced Microcontroller Bus Architecture (AMBA).

[0108] Computer system / server 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer system / server 12, and it includes both volatile and non-volatile media, removable and non-removable media.

[0109] System memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable,Page 23 of 31FOLEYHOAGUS12475408.1HCU-07325 non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media can be provided. In such instances, each can be connected to bus 18 by one or more data media interfaces. As will be further depicted and described below, memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the disclosure.

[0110] Program / utility 40, having a set (at least one) of program modules 42, may be stored in memory 28 by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, may include an implementation of a networking environment. Program modules 42 generally carry out the functions and / or methodologies of embodiments as described herein.

[0111] Computer system / server 12 may also communicate with one or more external devices 14 such as a keyboard, a pointing device, a display 24, etc.; one or more devices that enable a user to interact with computer system / server 12; and / or any devices (e.g., network card, modem, etc.) that enable computer system / server 12 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interfaces 22.Still yet, computer system / server 12 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via network adapter 20. As depicted, network adapter 20 communicates with the other components of computer system / server 12 via bus 18. It should be understood that although not shown, other hardware and / or software components could be used in conjunction with computer system / server 12. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.Page 24 of 31FOLEYHOAGUS12475408.1HCU-07325

[0112] The present disclosure may be embodied as a system, a method, and / or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0113] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD- ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiberoptic cable), or electrical signals transmitted through a wire.

[0114] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers,Page 25 of 31FOLEYHOAGUS12475408.1HCU-07325 firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0115] Computer readable program instructions for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user’s computer, partly on the user’s computer, as a stand-alone software package, partly on the user’s computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user’s computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0116] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer programPage 26 of 31FOLEYHOAGUS12475408.1HCU-07325 products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0117] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram block or blocks.

[0118] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0119] The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion ofPage 27 of 31FOLEYHOAGUS12475408.1HCU-07325 instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.Page 28 of 31FOLEYHOAGUS12475408.1

Claims

HCU-07325CLAIMSWhat is claimed is:

1. A method of block decoding in an artificial neural network, the method comprising: reading an encoded block comprising a plurality of encoded values, each encoded value comprising a mantissa; reading scaling information associated with the encoded block, the scaling information comprising an exponent and a mantissa; and decoding each of the plurality of encoded values according to the scaling information to produce a plurality of decoded values.

2. The method of claim 1, wherein decoding each of the plurality of encoded values comprises scaling its mantissa proportionate to the mantissa of the scaling information and exponentially with respect to the exponent of the scaling information.

3. The method of claim 1, wherein each of the plurality of encoded values further comprises a sign.

4. The method of claim 3, wherein each of the plurality of encoded values further comprises an exponent.

5. The method of claim 4, wherein decoding each of the plurality of encoded values comprises scaling its mantissa proportionate to the mantissa of the scaling information and exponentially with respect to its exponent and the exponent of the scaling information.

6. The method of any one of claims 1-5, wherein each of the plurality of encoded values is a weight.

7. The method of any one of claims 1-5, wherein each of the plurality of encoded values is an activation.

8. The method of any one of claims 1-5, wherein the scaling information further comprises a format field identifying a format.Page 29 of 31FOLEYHOAGUS12475408.1HCU-073259. The method of claim 8, wherein decoding each of the plurality of encoded values comprises applying the format to the plurality of encoded values.

10. The method of claim 1, wherein decoding each of the plurality of encoded values comprises replacing those of the plurality of encoded values corresponding to a first predetermined value with a second predetermined value.

11. A system comprising: a computing node comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor of the computing node to cause the processor to perform a method according to any one of claims 1-10.

12. A computer program product for block decoding in an artificial neural network, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform a method according to any one of claims 1-10.Page 30 of 31FOLEYHOAGUS12475408.1

Images