Data quantization and dequantization methods and systems based on tensor memory accelerators
By integrating quantization and dequantization hardware units into the tensor memory accelerator, the problem of separating quantization operations from data transport is solved, enabling real-time data quantization and dequantization with zero overhead, thus improving GPU performance and energy efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- INST OF COMPUTING TECH CHINESE ACAD OF SCI
- Filing Date
- 2026-01-30
- Publication Date
- 2026-05-26
AI Technical Summary
In existing technologies, the separation of quantization operations and data transfer leads to performance loss, insufficient energy efficiency, and wasted memory bandwidth, resulting in low hardware efficiency. The separation of quantization operations and data transfer also lacks hardware coordination, leading to redundant data transfer, kernel startup delays, and excessive consumption of computing core resources.
The Tensor Memory Accelerator (TMA) integrates configurable quantization and dequantization hardware units into its data transport path. By using extended TMA descriptors, it achieves simultaneous data transport and precision conversion. Employing a multi-stage pipeline and multi-channel parallel architecture, it automatically splits and reassembles the data stream to achieve real-time quantization with zero overhead.
It achieves simultaneous quantization and dequantization at the hardware level, improving data transfer efficiency, reducing chip area and power consumption, simplifying the programming model, and enhancing the parallel processing capability and throughput of the hardware unit.
Smart Images
Figure CN122089553A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of graphics processing unit (GPU) and high-performance computing chip architecture, and to a method, apparatus and system for real-time data quantization and dequantization for GPU tensor memory accelerator (TMA), which is particularly suitable for data-intensive applications such as artificial intelligence training and inference, and scientific computing. Background Technology
[0002] In the fields of large-scale deep learning training and high-performance computing, graphics processing units (GPUs) have become core computing hardware. With the rapid increase in the computing power of their cores (such as Tensor Cores), data transfer efficiency has become a key bottleneck restricting the overall performance of GPUs. In traditional architectures, the threads responsible for performing computations (CUDA Threads) need to personally manage the movement of data between global memory and shared memory. This includes complex address calculations, initiating data transfer instructions, and waiting for the transfer to complete. This model severely consumes thread resources and causes computing units to be idle while waiting for data, resulting in low overall system efficiency.
[0003] Secondly, as artificial intelligence models continue to grow in terms of scale, depth, number of parameters, and activation tensor size (typical examples include Transformer, LLM, and Diffusion models), the data size of weight data and intermediate activation values is growing rapidly. This further amplifies the data transfer pressure between memory and computing units, exacerbating the already prominent bandwidth bottleneck and causing the computing process to exhibit an even greater imbalance of "sufficient computing power but insufficient bandwidth."
[0004] Existing technical solutions include:
[0005] 1) Dedicated Tensor Memory Accelerator
[0006] To address the problem of inefficient data transfer, modern high-performance GPUs (such as NVIDIA's Hopper architecture H100) have introduced the Tensor Memory Accelerator (TMA) as an important component of their memory subsystem. TMA is a hardware engine dedicated to handling tensor data transfer. It understands the multidimensional structure of tensors and can efficiently schedule and manage the asynchronous transfer of large blocks of data between global memory and shared memory with a single programming effort.
[0007] like Figure 1This demonstrates the data transfer process before and after the introduction of Tensor Memory Accelerator (TMA) (A100 architecture vs. H100 architecture). In the A100 architecture, when computation requires data, the thread must first manually generate memory addresses and then read the data from GPU memory using the LDGSTS instruction. Therefore, the thread is not only responsible for initiating read and write operations but also for waiting for the data transfer to complete. During this process, the thread cannot perform other computational tasks, and the computing resources of the Tensor Core are idle. The H100 architecture introduces a dedicated hardware unit, the Tensor Memory Accelerator (TMA). The thread is no longer responsible for specific address calculations but instead describes a data block at a higher level (such as defining the tensor's starting address, shape, stride, etc.) and submits this task descriptor to the TMA unit. After receiving the descriptor, the TMA unit automatically generates all memory addresses at the hardware level and efficiently manages the entire data block transfer process. This achieves the decentralization and hardwareization of the data transfer task. Its core purpose is to liberate the computing core from complex memory address calculations and data transfer scheduling, allowing it to focus on the computation itself.
[0008] 2) Quantization and Inverse Quantization Techniques
[0009] As mentioned earlier, with the continuous development of artificial intelligence models towards larger scale and deeper layers, the data size of model weights and intermediate activation values is growing rapidly. The industry commonly uses data quantization / dequantization techniques to address this critical issue. Data quantization involves converting high-precision data into low-precision data through a series of operations to reduce data volume, lower memory access bandwidth requirements, and save storage space. The advantages of data quantization are as follows: First, high-precision data is several times larger than low-precision data, thus saving significant bandwidth and time when transmitting low-precision data. Second, low-precision operations are faster and more energy-efficient than high-precision operations in terms of hardware. For example, the FP64 computing speed of the GB200 GPU based on the NVIDIA Blackwell architecture is 90 TFLOPS, while the INT8 computing speed reaches 20 PFLOPS. With the newly introduced FP4 precision, the peak computing power reaches as high as 40 PFLOPS, with a performance difference of hundreds of times between different precisions. In addition, low-precision formats (such as FP4 and INT8) are far superior to high-precision formats (such as FP32 and FP64) in terms of power consumption and chip area. For example, compared with FP32 addition, INT8 addition is 30 times more energy efficient and 116 times more area efficient.
[0010] Currently, the mainstream approach to data quantization and dequantization is primarily implemented through software programs (such as CUDA). Common quantization implementations generally involve three steps: First, data reading and copying are performed. The program reads floating-point tensors from global memory and loads them into registers or shared memory accessible by the computational core. Then, the following operations are performed on each tensor element:
[0011]
[0012] Where x represents the original floating-point data, S is the scaling factor, and Z is the zero point offset. The upper and lower bounds of the target data type are defined by `round` and `clamp`, respectively, representing rounding and saturation clipping operations. For the dequantization process, an inverse transformation is performed: x = (Q − Z) × S. Finally, data is stored by writing the quantized data back to video memory or shared memory for subsequent calculations or reuse.
[0013] After investigation and analysis, the entire process of performing a data quantization operation in the software implementation scheme is as follows: First, a set of high-precision floating-point data generated by the Tensor Core is stored in shared memory. To perform precision conversion, the software must additionally start a quantization kernel. The execution process of this quantization kernel is as follows: the floating-point tensor is reloaded from shared memory into the CUDA Core's registers, element-wise quantization is performed, and the quantization result is written back to shared memory. After quantization is complete, TMA is started, moving the low-precision data from shared memory to global memory, completing the final data storage. In the above process, data transmission and data precision conversion operations are closely linked, but the existing TMA only acts as a "data porter," responsible for moving data from the source address to the target address unchanged, without the ability to change the data content or format during the data transport path.
[0014] The lack of coordination between TMA and the quantization kernel leads to three additional costs: First, redundant data transport, requiring data to travel back and forth through a path of "shared memory → CUDA kernel → shared memory → TMA transport." This is equivalent to performing two reads and two writes on the same data, significantly increasing memory bandwidth usage and latency. Second, kernel startup and scheduling overhead; each quantization requires an additional CUDA kernel to be started. For small tensors or high-frequency quantization tasks, the kernel startup latency can even exceed the quantization computation time itself. Third, the consumption of computational core resources. Quantization itself is a low-arithmetic-intensity operation, but it requires a large number of CUDA cores to perform simple scalar operations. When computational cores are forced to switch from main computation tasks to quantization tasks, overall throughput decreases.
[0015] Therefore, there is an urgent need to research a new hardware architecture capable of real-time quantization in order to eliminate the above-mentioned software execution overhead. Summary of the Invention
[0016] To address the performance losses, energy inefficiencies, and wasted memory bandwidth caused by the separation of quantization operations and data transfer in existing technologies, as well as the low hardware efficiency and low memory bandwidth utilization, this invention proposes a real-time data quantization and dequantization method, device, and system based on a TMA (Tensor Memory Accelerator). This invention aims to achieve "zero-overhead" precision conversion of data during the transfer process through innovative hardware integration, thereby achieving ultimate performance.
[0017] In a first aspect, embodiments of this application provide a data quantization and dequantization system based on a tensor memory accelerator (TMA). The system includes: a quantization and dequantization module integrated into the data transport path within the TMA of a GPU. The quantization and dequantization module comprises multiple parallel processing channels, each processing channel including:
[0018] Splitting and expansion modules: These modules are used to split the continuous data blocks to be transferred into multiple basic processing units and expand them to a preset fixed bit width, based on the quantization block size and the source data precision. Each channel executes either quantization mode or dequantization mode according to the operation mode.
[0019] Quantization module: The basic processing unit that receives the expanded quantization mode and performs scaling, zero-point correction, rounding and saturation operations in sequence;
[0020] Dequantization module: Used to receive the extended basic processing unit of the dequantization mode, and sequentially perform zero-point correction, scaling, precision clipping and rounding operations;
[0021] Assembly module: Used to receive the data processed by each parallel channel, reassemble it into a data block that conforms to the target memory layout according to the target data precision and quantization block size, and write it into the target memory.
[0022] In this embodiment of the invention, the quantization module includes:
[0023] First scaling module: Used to execute quantization commands and scale the source data, including a floating-point multiplier, a reciprocal taker, and an arithmetic unit;
[0024] First zero-point correction module: used to select whether to add an offset based on the quantization method;
[0025] Rounding module: Used to perform truncation and rounding operations according to rules by checking specified data bits;
[0026] Saturation module: Used to prevent overflow, it includes comparators and multiplexers.
[0027] In this embodiment of the invention, the above-mentioned inverse quantization module includes:
[0028] The second zero-point correction module is used to select whether to add an offset based on the inverse quantization method.
[0029] The second scaling module is used to execute dequantization commands to scale the source data. It includes a floating-point multiplier, a reciprocal taker, and an arithmetic unit.
[0030] Precision cropping module: Used to crop redundant precision based on the target data precision and according to rules;
[0031] Rounding module: Used to perform rounding operations according to rules by checking specified data bits.
[0032] In this embodiment of the invention, the quantization and dequantization modules are uniformly configured and scheduled through an extended TMA descriptor, and the data transfer and precision conversion are completed synchronously.
[0033] The extended TMA descriptor is issued through the GPU's command queue, parsed and executed by the TMA hardware, and includes: type code, first configuration field, second configuration field and function description.
[0034] In this embodiment of the invention, the first configuration field includes the following fields: quantization function enable, operation mode, source data precision encoding, target data precision encoding, quantization block size, and reserved field.
[0035] The second configuration field includes the following fields: scaling factor, zero offset, quantization algorithm, rounding mode, and reserved fields.
[0036] Secondly, embodiments of this application provide a data quantization and dequantization method based on a tensor memory accelerator, employing the aforementioned data quantization and dequantization system based on a tensor memory accelerator, and the method includes:
[0037] Module splitting and expansion steps: Based on the quantization block size and source data precision, the continuous data block to be transferred is split into multiple basic processing units and expanded to a preset fixed bit width. Each channel executes quantization mode or dequantization mode according to the operation mode.
[0038] Quantization steps: The basic processing unit receives the expanded quantization mode and sequentially performs scaling, zero-point correction, rounding, and saturation operations;
[0039] Dequantization step: Receive the extended basic processing unit after the dequantization mode, and sequentially perform zero-point correction, scaling, precision clipping and rounding operations;
[0040] Assembly steps: Receive the data processed by each parallel channel, reassemble it into a data block that conforms to the target memory layout according to the target data precision and quantization block size, and write it into the target memory.
[0041] In this embodiment of the invention, the quantization step includes:
[0042] The first scaling step: Execute the quantization command to scale the source data, including the floating-point multiplier, the reciprocal taker, and the arithmetic unit;
[0043] First zero-point calibration step: Choose whether to add an offset based on the quantization method;
[0044] Rounding steps: By checking the specified data bits, truncation and rounding operations are performed according to the rules;
[0045] Saturation step: Prevents overflow, includes comparators and multiplexers.
[0046] In this embodiment of the invention, the above-mentioned inverse quantization step includes:
[0047] The second zero-point calibration step: Select whether to add an offset based on the inverse quantization method;
[0048] The second scaling step: Execute the dequantization command to scale the source data, including the floating-point multiplier, the reciprocal unit, and the arithmetic unit;
[0049] Precision cropping steps: Based on the target data precision, perform cropping of excess precision according to the rules;
[0050] Rounding steps: By checking the specified data bits, rounding operations are performed according to the rules.
[0051] Thirdly, embodiments of this application provide a Tensor Memory Accelerator (TMA), including: a quantization and dequantization module in the data transport path integrated within the TMA of a GPU; and
[0052] Control module: Parses the TMA descriptor from the source data, initializes the data transfer parameters, and schedules and manages the status of the entire data transfer process;
[0053] Address generation module: Based on the dimension, stride and tensor layout information in the descriptor, calculate the multidimensional addresses of the source and destination ends, and perform indexing of the source tensor and the target tensor;
[0054] Data acquisition module: Receives the task after the descriptor is parsed, initiates memory read requests and prefetches source data blocks in sequence, and sends the data to the quantization and dequantization module to complete the calculation in parallel through multiple pipelines;
[0055] Data conversion module: Based on the target address generated by the address generation module and the data output by the quantization pipeline, complete the write operation at the target end;
[0056] The quantization and dequantization module contains multiple parallel processing channels, each of which includes:
[0057] Splitting and expansion modules: These modules are used to split the continuous data blocks to be transferred into multiple basic processing units and expand them to a preset fixed bit width, based on the quantization block size and the source data precision. Each channel executes either quantization mode or dequantization mode according to the operation mode.
[0058] Quantization module: The basic processing unit that receives the expanded quantization mode and performs scaling, zero-point correction, rounding and saturation operations in sequence;
[0059] Dequantization module: Used to receive the extended basic processing unit of the dequantization mode, and sequentially perform zero-point correction, scaling, precision clipping and rounding operations;
[0060] Assembly module: Used to receive the data processed by each parallel channel, reassemble it into a data block that conforms to the target memory layout according to the target data precision and quantization block size, and write it into the target memory.
[0061] Fourthly, embodiments of this application provide a graphics processing unit (GPU), including a GPU chip, a tensor accelerator (TMA), a memory, and a computer program stored in the memory and executable on the processor. The TMA employs the data quantization and dequantization system based on a tensor memory accelerator as described above. When the processor executes the program, it implements the steps of the data quantization and dequantization method based on a tensor memory accelerator as described above.
[0062] Compared with existing technologies, it has the following outstanding advantages:
[0063] 1) The method and system of this invention propose to integrate configurable quantization and dequantization hardware units in the data transport path of TMA; technical effect: quantization and dequantization are offloaded from software computing tasks and made into hardware, making them an inherent attribute of data transport, realizing near-zero overhead precision conversion, and enabling data transport and precision conversion to be completed synchronously in a single hardware pipeline.
[0064] 2) The method and system of this invention propose to achieve bidirectional hardware reuse for quantization and dequantization through a unified reconfigurable data path and arithmetic logic; Technical effect: By controlling signals and configuring parameters, the same set of physical hardware resources can be dynamically switched and reused bidirectionally. Compared with designing two independent units, this solution significantly reduces chip area and power consumption.
[0065] 3) The method and system of this invention propose an extended TMA descriptor format, which transmits complete quantization control words and parameters through a single instruction / descriptor; Technical effect: By defining a dedicated descriptor containing parameters such as operating mode, source / target precision, scaling factor, zero point, and rounding mode, precise control of the hardware quantization unit is achieved. The software can initiate complex tensor transfer tasks with real-time data conversion through a single configuration, simplifying the programming model.
[0066] 4) The method and system of this invention propose a parallel processing channel design and data stream reassembly mechanism based on quantization block size and source data precision; Technical effect: The hardware can automatically split the input data stream into multiple basic units according to the task granularity and distribute them to multiple parallel processing channels for simultaneous processing. Finally, the results are reassembled into data blocks that conform to the target memory layout, improving the parallel processing capability and throughput of the hardware unit. Attached Figure Description
[0067] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0068] Figure 1 Schematic diagrams of existing A100 and H100 architectures;
[0069] Figure 2 This is a schematic diagram of the data quantization and dequantization system based on a tensor memory accelerator according to the present invention;
[0070] Figure 3 This is a schematic diagram of the quantization / dequantization hardware pipeline component in an embodiment of the present invention;
[0071] Figure 4 This is a schematic diagram of the data quantization and dequantization method based on tensor memory accelerator of the present invention;
[0072] Figure 5 This is a schematic diagram of the TMA hardware architecture according to an embodiment of the present invention. Detailed Implementation
[0073] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.
[0074] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.
[0075] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0076] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0077] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0078] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0079] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0080] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.
[0081] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.
[0082] This invention proposes a real-time data quantization and dequantization method, device, and system based on TMA (Tensor Memory Accelerator), aiming to solve the problems of performance loss, insufficient energy efficiency, and wasted memory bandwidth caused by the separation of quantization operation and data transfer in the prior art.
[0083] In traditional solutions, quantization relies on software kernel execution, resulting in redundant data transfer, kernel scheduling overhead, and computational core usage. This invention integrates configurable quantization / dequantization hardware units into the TMA data path and extends the TMA descriptor to support multiple quantization methods, enabling precision conversion during data transfer and achieving "zero-overhead quantization." The hardware employs a multi-stage pipeline and multi-channel parallel architecture, automatically splitting and reassembling data streams according to task granularity, thereby significantly improving throughput. Through collaborative innovation at the hardware and architecture levels, this invention solves the fundamental efficiency bottleneck of data precision conversion in AI computing, providing a sound technical solution for next-generation high-performance, high-energy-efficiency AI computing platforms.
[0084] In AI training and inference systems, data frequently undergoes form and precision transformations between different computational stages. Examples include padding in convolutional neural networks, tiling and layout transformations in matrix multiplication, and quantization and dequantization, which have become widely used in large-scale model training in recent years. These transformations are generally low-arithmetic-intensity and essentially involve preprocessing or post-processing the data to be moved, thus being highly coupled with memory transfer processes. In current GPU architectures, these operations rely on software kernels, incurring additional overhead. Quantization is a prime example of this type of operation, occurring very frequently in large model training and inference, but currently it still relies entirely on software kernel execution, lacking coordination with TMA (Thinking Mapping) and unable to be completed within the data transfer path. Therefore, from both a general and scalability perspective, there is an urgent need to achieve hardware-level real-time quantization. To address this, we propose this invention—a real-time data quantization and dequantization method, apparatus, and system based on TMA. The aim is to eliminate CUDA kernel startup overhead, GPU computing power overhead, shared memory, register, and video memory transport overhead, and scheduling synchronization overhead by combining the data transport process with data precision conversion. This transforms quantization from "a computational task requiring kernel startup and core usage" into "a data transfer attribute seamlessly completed within the data transport path," achieving zero-overhead quantization.
[0085] The purpose of this invention is to provide a real-time data quantization and dequantization method, apparatus, and system based on a GPU tensor memory accelerator, to solve problems such as the separation of quantization operations and data transfer, low hardware efficiency, and low memory bandwidth utilization in existing technologies. This invention aims to achieve "zero-overhead" precision conversion of data during the transfer process through innovative hardware-level integration, thereby achieving ultimate performance.
[0086] The system of this application embodiment will be described in detail below with reference to specific embodiments:
[0087] like Figure 2 and Figure 3 As shown, the core idea of this invention is to integrate the quantization and dequantization hardware units as dedicated logic directly into the TMA data transfer path, and to perform unified configuration and scheduling through the extended TMA instruction set, thereby achieving the synchronous completion of data transfer and precision conversion.
[0088] like Figure 2 As shown, this invention proposes a data quantization and dequantization system based on a tensor memory accelerator (TMA). The system includes a quantization and dequantization module integrated into the data transport path of the TMA within the GPU. The quantization and dequantization module contains multiple parallel processing channels, each of which includes:
[0089] Splitting module and expansion module 101: used to split the continuous data block to be transported into multiple basic processing units and expand them into a preset fixed bit width according to the quantization block size and the source data precision. Each channel executes quantization mode or dequantization mode according to the operation mode.
[0090] Quantization module 102: Used to receive the extended basic processing unit of the quantization mode, and sequentially perform scaling, zero-point correction, rounding and saturation operations;
[0091] Inverse quantization module 103: Used to receive the extended basic processing unit of the inverse quantization mode and sequentially perform zero-point correction, scaling, precision clipping and rounding operations;
[0092] Assembly module 104: Used to receive the data processed by each parallel channel, reassemble it into a data block that conforms to the target memory layout according to the target data precision and quantization block size, and write it into the target memory.
[0093] In this embodiment of the invention, the quantization module 102 includes:
[0094] First scaling module: Used to execute quantization commands and scale the source data, including a floating-point multiplier, a reciprocal taker, and an arithmetic unit;
[0095] First zero-point correction module: used to select whether to add an offset based on the quantization method;
[0096] Rounding module: Used to perform truncation and rounding operations according to rules by checking specified data bits;
[0097] Saturation module: Used to prevent overflow, it includes comparators and multiplexers.
[0098] In this embodiment of the invention, the inverse quantization module 103 includes:
[0099] The second zero-point correction module is used to select whether to add an offset based on the inverse quantization method.
[0100] The second scaling module is used to execute dequantization commands to scale the source data. It includes a floating-point multiplier, a reciprocal taker, and an arithmetic unit.
[0101] Precision cropping module: Used to crop redundant precision based on the target data precision and according to rules;
[0102] Rounding module: Used to perform rounding operations according to rules by checking specified data bits.
[0103] In this embodiment of the invention, the quantization and dequantization modules are uniformly configured and scheduled through an extended TMA descriptor, and the data transfer and precision conversion are completed synchronously.
[0104] The extended TMA descriptor is issued through the GPU's command queue, parsed and executed by the TMA hardware, and includes: type code, first configuration field, second configuration field and function description.
[0105] In this embodiment of the invention, the first configuration field includes the following fields: quantization function enable, operation mode, source data precision encoding, target data precision encoding, quantization block size, and reserved field.
[0106] The second configuration field includes the following fields: scaling factor, zero offset, quantization algorithm, rounding mode, and reserved fields.
[0107] Specifically, the main contents of the specific embodiments of the present invention include:
[0108] 1) Design of Extended TMA Descriptor and Configuration Interface
[0109] This invention defines a dedicated descriptor format for the GPU's Tensor Memory Accelerator (TMA) for configuring and controlling real-time quantization and dequantization operations during data transfer. This descriptor is issued via the GPU's command queue and parsed and executed by the TMA hardware.
[0110] Table 1 Extended TMA Descriptor Format
[0111]
[0112] As shown in Table 1, the configuration field of the quantization-related descriptor is described in detail below:
[0113] Configuration Domain 1: Quantization Control Word
[0114] QuantEnable(1bit): Enables quantization. 1 indicates that real-time precision conversion is enabled in the data transfer path.
[0115] OperationMode (2 bits): Operation mode. 01 indicates quantization; 10 indicates dequantization.
[0116] SrcPrecision(3 bits): Source data precision encoding. Specifies the precision of the input data (e.g., INT4, INT8, FP16, FP32).
[0117] DstPrecision(3 bits): Target data precision encoding. Specifies the precision of the output data.
[0118] BlockSize (16 bits): Quantization block size. Defines the number of consecutive elements processed in a single precision conversion operation, used to optimize data locality and parallel processing.
[0119] Reserved (39 bits): Reserved fields are used for feature extensions.
[0120] Configuration Domain 2: Quantization Parameters
[0121] ScaleFactor (32 bits): Scaling factor. Represented in IEEE 754 FP32 format, it is the core parameter in the precision conversion formula, determining the dynamic range and scale of the conversion.
[0122] ZeroPoint (8 bits): Zero-point offset. Used for asymmetric quantization to align the zeros of integer and floating-point values.
[0123] QuantMethod (4 bits): Quantization algorithm. Choose from symmetric quantization, asymmetric quantization, non-uniform quantization, etc.
[0124] RoundMode (4 bits): Rounding mode. The rounding method used in precision conversion (such as rounding to the nearest even number or rounding to zero).
[0125] Reserved (16 bits): Reserved fields are used for feature extensions.
[0126] 2) Data path of integrated quantization / dequantization hardware unit
[0127] Within the TMA's internal data transfer path, we have integrated a configurable precision conversion hardware unit. This unit serves as an essential link in the data transmission from the source address to the destination address, achieving real-time processing with zero overhead.
[0128] The pipelined processing of the quantization / dequantization unit is as follows: Figure 3 As shown, each block diagram represents a hardware computing unit that implements a corresponding function. First, the TMA control logic, based on the number of elements defined by the quantization block size (BlockSize) in the descriptor and the element bit width determined by the source data precision (SrcPrecision), splits the continuous data block to be transferred into multiple basic processing units and expands them to a fixed bit width. These units are then distributed to multiple parallel processing channels. Each channel configures its internal data path according to its operation mode and performs mathematical operations independently. The quantization path sequentially performs scaling, zero-point correction (asymmetric quantization), rounding (applying the specified rounding algorithm), and saturation operations; the dequantization path sequentially performs zero-point correction, scaling, precision clipping, and rounding operations. Finally, the data processed by each channel is reassembled into data blocks conforming to the target memory layout according to the target data precision (DstPrecision) and quantization block size, and written to the target memory.
[0129] Each processing channel integrates two nearly symmetrical reconfigurable computing pipelines. Each pipeline contains independent scaling, zero-point correction, rounding, and saturation units, which execute quantization and dequantization commands respectively. The scaling unit includes floating-point multipliers, reciprocal units, and other arithmetic units for quantization methods to scale the source data; the zero-point correction unit selects whether to add an offset based on the quantization method; the rounding unit supports multiple rounding modes, performing truncation and rounding operations according to rules by checking specified data bits; and the saturation unit is used to prevent overflow and is implemented by logic units such as comparators and multiplexers.
[0130] In standard operating mode, quantization and dequantization commands each occupy a set of computing units. After receiving a command, the TMA configures the specified computing unit in each channel according to the descriptor, so quantization and dequantization commands can be executed in parallel. To efficiently reuse computing units, our invention also supports a full quantization mode, in which both sets of computing pipelines in each channel perform symmetric quantization calculations to achieve a doubling of single-vector throughput.
[0131] As described above, the system of the present invention can be implemented well.
[0132] Compared with the prior art, the present invention has the following outstanding advantages and beneficial effects:
[0133] Compared with existing technologies (software quantization schemes and single-function TMA data transfer schemes), the overall beneficial effects of this invention are reflected in three aspects: performance, energy efficiency, and architecture.
[0134] In terms of performance, hardware pipelining eliminates the time overlap between precision conversion and data transfer, thus removing the latency associated with kernel startup and scheduling in software solutions, while also increasing effective memory bandwidth. The implementation of dedicated hardware also significantly improves instruction execution efficiency, resulting in a substantial performance boost.
[0135] In terms of energy efficiency, the energy efficiency of dedicated hardware circuits executing specific algorithms is far superior to the software implementation of general-purpose computing cores. Simultaneously, hardware reuse technology avoids the power consumption of redundant units, saving power. Secondly, quantization before data is written to global video memory ensures that low-precision compressed data is always transmitted on the system bus, reducing the effective memory bandwidth requirement to 1 / 2 to 1 / 4 of the original, saving bandwidth and storage space.
[0136] In terms of architecture, software developers only need to configure parameters through the extended descriptor interface, without having to write and maintain complex quantization kernel code, which reduces the complexity of AI framework and algorithm development. At the same time, it tightly integrates the two originally separate system functions of data transfer and precision conversion, simplifying the overall hardware design of the AI accelerator.
[0137] In summary, this invention solves the fundamental efficiency bottleneck of data precision conversion in AI computing through collaborative innovation at the hardware and architecture levels, providing a technical solution for the next generation of high-performance, high-energy-efficiency AI computing platforms.
[0138] Secondly, such as Figure 4 As shown, this application provides a data quantization and dequantization method based on a tensor memory accelerator, employing the aforementioned data quantization and dequantization system based on a tensor memory accelerator. The method includes:
[0139] Module splitting and expansion steps 201: Based on the quantization block size and source data precision, the continuous data block to be transferred is split into multiple basic processing units and expanded to a preset fixed bit width. Each channel executes quantization mode or dequantization mode according to the operation mode.
[0140] Quantization step 202: Receive the extended basic processing unit of the quantization mode and perform scaling, zero-point correction, rounding and saturation operations in sequence;
[0141] Dequantization step 203: Receive the extended basic processing unit after the dequantization mode, and sequentially perform zero-point correction, scaling, precision clipping and rounding operations;
[0142] Assembly step 204: Receive the data processed by each parallel channel, reassemble it into a data block that conforms to the target memory layout according to the target data precision and quantization block size, and write it into the target memory.
[0143] In this embodiment of the invention, the quantization step 202 includes:
[0144] The first scaling step: Execute the quantization command to scale the source data, including the floating-point multiplier, the reciprocal taker, and the arithmetic unit;
[0145] First zero-point calibration step: Choose whether to add an offset based on the quantization method;
[0146] Rounding steps: By checking the specified data bits, truncation and rounding operations are performed according to the rules;
[0147] Saturation step: Prevents overflow, includes comparators and multiplexers.
[0148] In this embodiment of the invention, the above-mentioned inverse quantization step 203 includes:
[0149] The second zero-point calibration step: Select whether to add an offset based on the inverse quantization method;
[0150] The second scaling step: Execute the dequantization command to scale the source data, including the floating-point multiplier, the reciprocal unit, and the arithmetic unit;
[0151] Precision cropping steps: Based on the target data precision, perform cropping of excess precision according to the rules;
[0152] Rounding steps: By checking the specified data bits, rounding operations are performed according to the rules.
[0153] Thirdly, embodiments of this application provide a Tensor Memory Accelerator (TMA), including: a quantization and dequantization module in the data transport path integrated within the TMA of a GPU; and
[0154] Control module: Parses the TMA descriptor from the source data, initializes the data transfer parameters, and schedules and manages the status of the entire data transfer process;
[0155] Address generation module: Based on the dimension, stride and tensor layout information in the descriptor, calculate the multidimensional addresses of the source and destination ends, and perform indexing of the source tensor and the target tensor;
[0156] Data acquisition module: Receives the task after the descriptor is parsed, initiates memory read requests and prefetches source data blocks in sequence, and sends the data to the quantization and dequantization module to complete the calculation in parallel through multiple pipelines;
[0157] Data conversion module: Based on the target address generated by the address generation module and the data output by the quantization pipeline, complete the write operation at the target end;
[0158] The quantization and dequantization module contains multiple parallel processing channels, each of which includes:
[0159] Splitting and expansion modules: These modules are used to split the continuous data blocks to be transferred into multiple basic processing units and expand them to a preset fixed bit width, based on the quantization block size and the source data precision. Each channel executes either quantization mode or dequantization mode according to the operation mode.
[0160] Quantization module: The basic processing unit that receives the expanded quantization mode and performs scaling, zero-point correction, rounding and saturation operations in sequence;
[0161] Dequantization module: Used to receive the extended basic processing unit of the dequantization mode, and sequentially perform zero-point correction, scaling, precision clipping and rounding operations;
[0162] Assembly module: Used to receive the data processed by each parallel channel, reassemble it into a data block that conforms to the target memory layout according to the target data precision and quantization block size, and write it into the target memory.
[0163] like Figure 5The diagram illustrates the overall hardware architecture of a TMA module. The Control Unit is responsible for parsing the TMA descriptor from the SM, initializing the transfer parameters, and scheduling and managing the state of the entire data transfer process. The AddressGenerator is responsible for calculating the multi-dimensional addresses of the source and destination tensors based on the dimension, stride, and tensor layout information in the descriptor, ensuring correct indexing of the source and target tensors. After descriptor parsing, the task is sent to the Issue Queue & Data Fetch module, which sequentially initiates memory read requests, prefetches source data blocks, and sends the data to the backend quantization / dequantization unit. The quantization unit performs calculations in parallel through multiple pipelines. Finally, the Data Transfer Engine completes the write operation at the target end based on the target address generated by the AddressGenerator and the data output from the quantization pipelines.
[0164] The control logic of this invention is meticulously designed. By parsing the quantization control word and parameters in the descriptor, it can dynamically configure the hardware data path, achieving flexible and precise control over various complex algorithm processes, including symmetric and asymmetric quantization, with comprehensive functionality. More importantly, this invention decomposes the core computations involved in quantization into multi-stage pipelines and adopts a multi-channel parallel processing architecture, deeply embedding data precision conversion into the data transport process. While ensuring functional completeness, it achieves extreme efficiency in transmission and computation, realizing near-zero overhead real-time quantization.
[0165] Fourthly, embodiments of this application provide a graphics processing unit (GPU), including a GPU chip, a tensor accelerator (TMA), a memory, and a computer program stored in the memory and executable on the processor. The TMA employs the data quantization and dequantization system based on a tensor memory accelerator as described above. When the processor executes the program, it implements the steps of the data quantization and dequantization method based on a tensor memory accelerator as described above.
[0166] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0167] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A data quantization and dequantization system based on a tensor memory accelerator, characterized in that, The system includes: a quantization and dequantization module integrated into the data transport path of the Tensor Memory Accelerator (TMA) within the GPU; the quantization and dequantization module comprises multiple parallel processing channels, each processing channel including: Splitting and expansion modules: These modules are used to split the continuous data blocks to be transferred into multiple basic processing units and expand them to a preset fixed bit width, based on the quantization block size and the source data precision. Each channel executes either quantization mode or dequantization mode according to the operation mode. Quantization module: Used to receive the basic processing unit after the quantization mode is extended, and to perform scaling, zero-point correction, rounding and saturation operations in sequence; Dequantization module: Used to receive the basic processing unit after the dequantization mode is extended, and to perform zero-point correction, scaling, precision clipping and rounding operations in sequence; Assembly module: Used to receive the data processed by each parallel channel, reassemble it into a data block that conforms to the target memory layout according to the target data precision and quantization block size, and write it into the target memory.
2. The data quantization and dequantization system based on a tensor memory accelerator according to claim 1, characterized in that, The quantization module includes: First scaling module: Used to execute quantization commands and scale the source data, including a floating-point multiplier, a reciprocal taker, and an arithmetic unit; First zero-point correction module: used to select whether to add an offset based on the quantization method; Rounding module: Used to perform truncation and rounding operations according to rules by checking specified data bits; Saturation module: Used to prevent overflow, it includes comparators and multiplexers.
3. The data quantization and dequantization system based on a tensor memory accelerator according to claim 1, characterized in that, The inverse quantization module includes: The second zero-point correction module is used to select whether to add an offset based on the inverse quantization method. The second scaling module is used to execute dequantization commands to scale the source data. It includes a floating-point multiplier, a reciprocal taker, and an arithmetic unit. Precision cropping module: Used to crop redundant precision based on the target data precision and according to rules; Rounding module: Used to perform rounding operations according to rules by checking specified data bits.
4. The data quantization and dequantization system based on a tensor memory accelerator according to claim 1, characterized in that, The quantization and dequantization modules are uniformly configured and scheduled through an extended TMA descriptor, and the data transfer and precision conversion are completed synchronously. The extended TMA descriptor is issued through the GPU's command queue, parsed and executed by the TMA hardware, and includes: a type code, a first configuration field, a second configuration field, and a function description.
5. The data quantization and dequantization system based on a tensor memory accelerator according to claim 1, characterized in that, The first configuration field includes the following fields: quantization function enable, operation mode, source data precision encoding, target data precision encoding, quantization block size, and reserved fields; The second configuration field includes the following fields: scaling factor, zero offset, quantization algorithm, rounding mode, and reserved fields.
6. A data quantization and dequantization method based on a tensor memory accelerator, employing the data quantization and dequantization system based on a tensor memory accelerator as described in any one of claims 1-5, characterized in that, The method includes: Module splitting and expansion steps: Based on the quantization block size and source data precision, the continuous data block to be transferred is split into multiple basic processing units and expanded to a preset fixed bit width. Each channel executes quantization mode or dequantization mode according to the operation mode. Quantization steps: The basic processing unit, after receiving the expanded quantization mode, sequentially performs scaling, zero-point correction, rounding, and saturation operations; Dequantization step: The basic processing unit, after receiving the extended dequantization mode, sequentially performs zero-point correction, scaling, precision clipping, and rounding operations; Assembly steps: Receive the data processed by each parallel channel, reassemble it into a data block that conforms to the target memory layout according to the target data precision and quantization block size, and write it into the target memory.
7. The data quantization and dequantization method based on tensor memory accelerator according to claim 6, characterized in that, The quantization step includes: The first scaling step: Execute the quantization command to scale the source data, including the floating-point multiplier, the reciprocal taker, and the arithmetic unit; First zero-point calibration step: Choose whether to add an offset based on the quantization method; Rounding steps: By checking the specified data bits, truncation and rounding operations are performed according to the rules; Saturation step: Prevents overflow, includes comparators and multiplexers.
8. The data quantization and dequantization method based on tensor memory accelerator according to claim 6, characterized in that, The inverse quantization step includes: The second zero-point calibration step: Select whether to add an offset based on the inverse quantization method; The second scaling step: Execute the dequantization command to scale the source data, including the floating-point multiplier, the reciprocal unit, and the arithmetic unit; Precision cropping steps: Based on the target data precision, perform cropping of excess precision according to the rules; Rounding steps: By checking the specified data bits, rounding operations are performed according to the rules.
9. A tensor memory accelerator (TMA), characterized in that, include: Quantization and dequantization modules integrated into the data transport path within the Tensor Memory Accelerator (TMA); and Control module: Parses the TMA descriptor from the source data, initializes the data transfer parameters, and schedules and manages the status of the entire data transfer process; Address generation module: Based on the dimension, stride and tensor layout information in the descriptor, calculate the multidimensional addresses of the source and destination ends, and perform indexing of the source tensor and the target tensor; Data acquisition module: Receives the task after the descriptor is parsed, sequentially initiates memory read requests and prefetches source data blocks, and sends the data to the quantization and dequantization module, and completes the calculation in parallel through multiple pipelines; Data conversion module: Based on the target address generated by the address generation module and the data output by the quantization pipeline, complete the write operation at the target end; The quantization and dequantization module comprises multiple parallel processing channels, each of which includes: Splitting and expansion modules: These modules are used to split the continuous data blocks to be transferred into multiple basic processing units and expand them to a preset fixed bit width, based on the quantization block size and the source data precision. Each channel executes either quantization mode or dequantization mode according to the operation mode. Quantization module: Used to receive the basic processing unit after the quantization mode is extended, and to perform scaling, zero-point correction, rounding and saturation operations in sequence; Dequantization module: Used to receive the basic processing unit after the dequantization mode is extended, and to perform zero-point correction, scaling, precision clipping and rounding operations in sequence; Assembly module: Used to receive the data processed by each parallel channel, reassemble it into a data block that conforms to the target memory layout according to the target data precision and quantization block size, and write it into the target memory.
10. A graphics processing unit (GPU), comprising a GPU chip, a TMA, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, The TMA employs a data quantization and dequantization system based on a tensor memory accelerator as described in any one of claims 1-5, and the processor, when executing the program, implements the steps of the data quantization and dequantization method based on a tensor memory accelerator as described in any one of claims 6-8.