High-fidelity key cache compression method and system based on spectral quantization
By using a high-fidelity key cache compression framework based on spectral quantization, the storage overhead problem caused by RoPE in large language models is solved, and efficient key cache compression and inference acceleration are achieved at extremely low bit rates.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANDONG UNIV
- Filing Date
- 2026-03-17
- Publication Date
- 2026-06-23
AI Technical Summary
Existing technologies face the problem that storage overhead increases linearly with sequence length when dealing with key-value caching of large language models. In particular, when using Rotated Position Encoding (RoPE), traditional quantization methods cannot effectively compress the high-frequency oscillations of key vectors, resulting in excessive memory usage and affecting inference efficiency.
A high-fidelity key cache compression framework based on spectral quantization (SpectrumQuant) is adopted. By extracting the main frequency, using hybrid bit width allocation and high frequency pre-emphasis techniques, the key cache data is mapped from the time domain to the frequency domain. The discrete cosine transform (DCT) is used for signal analysis, and high-fidelity compression at extremely low bit rates is achieved through frequency domain hybrid quantization and hardware fusion calculation.
Maintaining model performance at extremely low bit rates, significantly reducing GPU memory usage, increasing inference throughput, solving the RoPE-induced oscillation problem, and achieving inference speed comparable to full-precision models.
Smart Images
Figure CN122268377A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a high-fidelity key caching compression method and system based on spectral quantization, belonging to the field of artificial intelligence and deep learning technology. Background Technology
[0002] With the continuous development of artificial intelligence and deep learning technologies, large language models (LLMs) have demonstrated remarkable capabilities in natural language processing tasks, gradually becoming a research hotspot in both academia and industry. Among these, as model architectures evolve towards ultra-long contexts (e.g., 1M tokens), efficiently processing massive amounts of historical information has become a key challenge. In the inference process of LLMs, key-value (KV) caching, as a core mechanism for reusing historical states to avoid redundant computation, plays a crucial role in generating long sequences. By caching the key and value tensors at each layer, the model does not need to recalculate the attention mechanism of historical tokens at each generation step, which greatly improves inference speed.
[0003] However, traditional key-value (KV) caching mechanisms face the serious problem of storage overhead increasing linearly with sequence length. As the length of input prompts and generated sequences increases, the GPU memory usage of the KV cache rises sharply, quickly exceeding the GPU memory usage of the model weights themselves, becoming the primary bottleneck restricting the throughput of long text inference and the maximum context length. In resource-constrained hardware environments, this often leads to "out-of-memory" (OOM) errors or forces the system to drastically reduce the batch size, thus severely impairing the service's concurrency and response speed.
[0004] To alleviate this memory crisis, key-value (KV) cache quantization techniques have emerged. These methods aim to map high-precision floating-point (FP16 / BF16) key-value tensors to low-bit integers (such as INT4 or INT2), thereby significantly reducing memory usage while maintaining model performance. Traditional quantization methods typically employ asymmetric uniform quantization strategies, determining the scaling factor and zeros by calculating the maximum and minimum values of the tensor. While these methods alleviate memory pressure to some extent, they fall short when facing the Rotary Positional Embeddings (RoPE) mechanism commonly used in modern large models.
[0005] To address the aforementioned issues, scholars have recently begun to delve into the impact of RoPE on the quantization process. RoPE, as a positional encoding scheme that effectively captures relative positional information, is widely used in state-of-the-art models such as LLaMA and Qwen. However, by introducing position-dependent rotation operations between channel pairs, RoPE inevitably disrupts the amplitude consistency of the key vector along the channel dimension. This rotation operation introduces high-frequency, high-amplitude periodic oscillations along the temporal dimension, causing drastic fluctuations in the dynamic range of the data. Existing time-domain-based quantization methods often fail to effectively capture this complex oscillation pattern, resulting in quantization noise overwhelming the original semantic signal and causing a severe degradation in model performance at low bit settings.
[0006] Although some work has attempted to address this issue through targeted quantization strategies (such as grouped quantization and outlier preservation), existing technologies still face the following two main challenges: (1) RoPE-induced oscillations are difficult to compress effectively in the time domain. Since RoPE is a rotation that changes with position in the time domain, the numerical distribution of the bond vector exhibits high irregularity. Simple time-domain quantization (whether channel-by-channel or token-by-token) often requires a large bit width to maintain accuracy, or suffers a huge loss of accuracy at low bit widths. Existing time-domain methods have failed to fundamentally solve the problem of distribution disruption caused by rotation.
[0007] (2) The computational overhead and hardware efficiency of frequency domain methods are difficult to balance. Although frequency domain transformation can theoretically concentrate signal energy, the introduction of spectral transformation operations such as discrete cosine transform (DCT) will bring additional computational latency. If there is a lack of in-depth optimization for modern GPU hardware (such as SRAM and HBM interaction), the increase in inference latency caused by frequency domain transformation may offset the performance gains brought by quantization, making it difficult to meet the needs of real-time inference. Summary of the Invention
[0008] To address the issues of decreased accuracy, insufficient compression ratio, and high computational overhead of frequency domain methods in existing KV cache quantization methods when processing RoPE rotation position encoding, this invention proposes a high-fidelity key cache compression framework (SpectrumQuant) based on spectral quantization for long context LLM inference; this invention also proposes a key cache compression and inference acceleration system based on the aforementioned framework.
[0009] The SpectrumQuant framework constructed in this invention consists of a Dominant Frequency Extraction module, a Hybrid Bit-width Allocation mechanism, a High-Frequency Pre-emphasis technique, and a Fused On-Chip Computation kernel. Based on spectral analysis of RoPE-induced oscillations, this framework utilizes Discrete Cosine Transform (DCT) to transform complex time-domain oscillations into a sparse frequency-domain representation. Through refined modeling of the spectral energy distribution and hardware-level optimization, it achieves extreme compression and efficient inference with lossless accuracy.
[0010] First, this invention re-examines the quantization problem of key buffers from the perspective of spectral analysis. Through visualization and theoretical derivation, it is found that the periodic oscillations induced by RoPE exhibit a high degree of energy concentration in the frequency domain (DCT domain): the vast majority of spectral energy is concentrated on a very small number of adjacent frequency coefficients (i.e., the dominant frequency), while the remaining residual signal exhibits extremely small fluctuations. Based on this finding, the SpectrumQuant framework proposed in this invention utilizes DCT transformation along the time dimension to decouple the originally difficult-to-compress time-domain signal into a high-energy dominant frequency component and a low-energy residual component, laying the foundation for subsequent efficient encoding.
[0011] Secondly, the main frequency extraction module and hybrid bit-width allocation mechanism proposed in this invention aim to maximize compression fidelity. The main frequency extraction module is responsible for identifying and separating the highest-energy coefficients (main frequencies) in the spectrum, and storing this key information in a high-precision format to ensure accurate reconstruction of the RoPE rotation characteristics. For the residual spectrum after removing the main frequencies, this invention utilizes a hybrid bit-width allocation mechanism to assign different quantization bit widths based on the different contributions of low-frequency and high-frequency signals to the reconstruction quality. Specifically, a relatively high bit width is allocated to the low-frequency band containing semantic structural information, while an extremely low bit width is allocated to the high-frequency band mainly containing noise. This non-uniform bit-width allocation strategy ensures the overall compression ratio while preserving the semantic features crucial to model inference to the greatest extent.
[0012] Building upon this foundation, this invention introduces a high-frequency pre-emphasis technique to address the quantization challenges arising from differences in dynamic range across different frequency bands. Since the amplitude of high-frequency residual signals is typically much smaller than that of low-frequency signals, direct uniform quantization leads to the loss of high-frequency information. The pre-emphasis technique scales the high-frequency coefficients before quantization, aligning their dynamic range with that of the low-frequency signal. This allows different frequency bands to share quantization parameters (such as scaling factors), further reducing metadata storage overhead and improving the overall compression ratio.
[0013] To eliminate the additional latency introduced by frequency domain processing, this invention develops a deeply customized Fused Triton Kernels. This kernel utilizes the GPU's on-chip shared memory (SRAM) for end-to-end computation, fusing operations such as transformation, extraction, and packing during quantization, and inverse transformation and sparse accumulation during decoding. In particular, this invention designs a "Deferred Inverse Transform" strategy, avoiding explicit reconstruction of the time-domain tensor and directly calculating and accumulating attention scores in the frequency domain. This overcomes the memory bandwidth bottleneck, achieving inference latency comparable to or even better than the FP16 baseline.
[0014] Through the collaborative design of main frequency extraction, hybrid bit width allocation, high frequency pre-emphasis and fusion computing kernel, the SpectrumQuant framework proposed in this invention can effectively neutralize the oscillation effect caused by RoPE at an extremely low bit compression rate (such as 3.71 bits / token), maintain performance almost consistent with the full-precision model in long context tasks, significantly reduce memory usage and improve inference throughput.
[0015] The technical solution of the present invention is as follows: High-fidelity key caching compression methods based on spectral quantization include: Step S1: Spectral Transformation and Feature Analysis Process; Receive the raw key buffer data from the model, map the raw key buffer data from the time domain to the frequency domain, and use the energy concentration characteristics unique to Discrete Cosine Transform (DCT) to deeply analyze the signal structure; Step S2: Frequency domain hybrid quantization coding process; through the main frequency extraction mechanism, high frequency pre-emphasis technology and hybrid bit width allocation strategy, the frequency domain coefficients are compressed into a sparse representation with extremely low bits; Step S3: Fusion decoding and sparse accumulation process; using hardware-aware fusion operators and delay inverse transform technology, attention scores are calculated directly in the compressed domain.
[0016] According to a preferred embodiment of the present invention, the specific implementation process of step S1 includes: Step S1.1: Grouping processing of the key buffer and modeling of the rotation position encoded signal, i.e., the RoPE signal; specifically including: Mathematical modeling and preprocessing of the key vectors in the Transformer model; let the key vectors of the Transformer model be... The layer input hidden state is The tensor dimension is ,in Indicates the sequence length of the current context. This represents the feature dimension of the hidden layer; for any given attention head, the key vector... That is, the original key cache data is generated through linear transformation projection; the first key cache data is generated through linear transformation projection. Each channel is located at the position The mathematical form of the rotational transformation is as follows: ; Among them, frequency parameters It decays as the channel index increases; This refers to the position of the rotated feature within the sequence. , These represent the positions before and after rotation, respectively. The key vector in the th Components on each channel; , These represent the positions before and after rotation, respectively. The key vector in the th Components on each channel; key vector Logically divide the token into several groups of fixed size, denoted as group size . All mathematical transformations, parameter calculations, and quantization operations, including the execution of DCT transformation, main frequency extraction, mixed bit width allocation and high frequency pre-emphasis processing, shared parameter quantization calculation and data packaging in steps S1.2 to S2.3, are performed independently and in parallel within each group. Step S1.2: Perform Discrete Cosine Transform (DCT) and energy concentration characteristic analysis; use DCT-II transform, i.e., Discrete Cosine Transform (DCT), to transform the time-domain key vector... Mapped to the frequency domain, for lengths of input sequence Transformed spectral coefficients The calculation formula is as follows: ; in, , The normalization coefficient is defined as: when hour, ;when hour, ; yes The Each component.
[0017] According to a preferred embodiment of the present invention, the specific implementation process of step S2 includes: A strategy of extracting the main frequency and sparse coding is adopted to separate and protect the high-energy signal carrying key semantics; Hybrid bit width allocation and high-frequency pre-emphasis achieve extreme compression in residual signal processing through a frequency-domain-aware adaptive quantization mechanism. Based on the difference in contribution of low-frequency and high-frequency components to the reconstruction quality, the residual spectrum is divided into low-frequency and high-frequency bands. At the same time, a high-frequency pre-emphasis factor is introduced to stretch the distribution of high-frequency signals before quantization and share quantization parameters with low-frequency signals, thereby minimizing metadata overhead.
[0018] According to a preferred embodiment of the present invention, the specific implementation process of step S2 includes: Step S2.1: Main frequency extraction and residual signal separation; after obtaining the spectral coefficient matrix Next, firstly, the main frequency extraction operation is performed; for each independent channel... Automatically scan and identify the largest absolute value The coefficients are taken as the set of main frequencies, denoted as . The mathematical expression for the main frequency extraction process is: ; in, Represents the spectrum coefficient matrix In the Components on each channel This means retrieving the largest value from a vector. The extracted main frequency value is denoted as (the number of elements is missing in the original text). The corresponding frequency index is denoted as ; Spectrum coefficient matrix The middle corresponds to The coefficients for position are set to zero, thus obtaining the residual spectrum. The calculation logic is as follows: ; in, , Representing the residual spectrum and the original spectral coefficients in the sequence, respectively. The position Components on each channel; Step S2.2: Hybrid bit width allocation and high-frequency pre-emphasis processing; the residual spectrum... Logically divided into low-frequency bands along the frequency dimension and high frequency band low frequency band The index range is from arrive High frequency band The index range is from arrive ; High-frequency pre-emphasis technology is introduced, using a preset emphasis factor before quantization. The high-frequency band coefficients are amplitude stretched to align the dynamic range with the low-frequency signal; the pre-emphasized high-frequency band is denoted as... The calculation formula is as follows: ; Step S2.3: Shared parameter quantization calculation and data packaging; specifically including: based on the concatenated data vector Calculate the scaling factor shared at the group level. and zero point The calculation formula is as follows: ; ; in, The reference bit width is used; an adaptation factor is introduced. The final quantification formulas are as follows: ; ; in, ; Finally, the quantized integers and The entire encoding and compression process is completed by using bitwise operations to perform compact packaging and storage.
[0019] Further preferred, It is 1 / 5.
[0020] According to a preferred embodiment of the present invention, the specific implementation process of step S3 includes: Step S3.1: Fusion dequantization and high-frequency inverse scaling recovery; design a deep fusion kernel based on the Triton language; first, read the packaged compressed data from the video memory and perform dequantization; the low-frequency band recovery formula is: ; in, It is a low-frequency signal obtained after dequantization. and These are the storage scaling factor and zero point, respectively; For high-frequency band recovery, an adaptation factor is also applied. Inverse transform and pre-emphasis factor The inverse transform is used to restore the original amplitude ratio. The formula for high-frequency band recovery is: ; in, It is the high-frequency signal after dequantization; After the above steps, the reconstructed residual spectrum is obtained in the shared memory SRAM inside the chip. Residual spectrum It is by and A complete spectrum signal assembled from multiple parts; Step S3.2: Deferred Inverse Transform (Deferred IDCT) and Attention Score Calculation; including: Let the query vector Query be The DCT basis matrix is Attention score is derived mathematically. The calculation process is rewritten as follows: ; The fusion operator first calculates the query vector. With frequency domain coefficients The dot product is used to obtain the frequency domain attention score; subsequently, the DCT basis matrix stored in the on-chip shared memory SRAM is used. Projecting the frequency domain fraction back into the time domain; Step S3.3: Sparse frequency accumulation and final output; based on the read frequency index Sum of values Calculations and updates are performed only at the corresponding non-zero positions: ; Finally, the accumulator The information stored in the middle is the attention score in the time domain. The final attention weights are obtained after the Softmax operation.
[0021] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program, implements the steps of the above-described high-fidelity key cache compression method based on spectral quantization.
[0022] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described high-fidelity key cache compression method based on spectral quantization.
[0023] A high-fidelity key caching compression and inference system based on spectral quantization includes: The spectral transformation and extraction module is configured to: convert the time-domain key vector into a frequency-domain representation based on the discrete cosine transform, and identify and separate the dominant frequency components with concentrated energy, thereby realizing the sparsity decomposition of the signal; The hybrid bit-width coding module is configured to: align the dynamic range of different frequency bands through high-frequency pre-emphasis technology, and use a hybrid precision strategy to perform non-uniform quantization on low-frequency and high-frequency residual signals in order to minimize storage footprint; The fusion decoding inference module is configured to: in the inference generation stage, use fusion operators to directly calculate attention scores and perform sparse accumulation in the frequency domain, avoiding explicit time-domain reconstruction and achieving low-latency long text inference.
[0024] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention proposes a unified SpectrumQuant framework to address the problems of RoPE-induced oscillations in long-context LLM inference, which lead to difficulties in key cache quantization, limited compression ratio, and severe accuracy loss. It constructs a frequency domain compression scheme composed of spectral domain transformation, main frequency extraction, and hybrid bit width allocation. By transforming complex time-domain oscillations into a sparse frequency domain representation, this invention fundamentally solves the "RoPE dilemma," achieving high-fidelity compression of the key cache at extremely low bit rates (e.g., an average of 3.7 bits / token), effectively compensating for the shortcomings of existing spatial or temporal quantization methods in capturing the characteristics of rotation position encoding.
[0025] 2. The proposed main frequency extraction and hybrid bit-width allocation mechanism balances signal fidelity and storage efficiency. It ensures the accuracy of RoPE location information by accurately preserving the main frequency coefficients, which concentrate most of the energy; and it allocates bit resources based on the information entropy difference between low-frequency and high-frequency signals (e.g., 4 bits for low-frequency and 2 bits for high-frequency) through a hybrid bit-width strategy. This mechanism better conforms to the spectral energy distribution of the key cache, significantly improves the cosine similarity of the reconstructed signal, and ensures the accuracy of attention calculation in long text tasks.
[0026] 3. The high-frequency pre-emphasis technique proposed in this invention is based on signal processing principles and adaptively scales the high-frequency residual signal, solving the problem of quantization parameter sharing caused by large differences in the dynamic range of different frequency bands. This mechanism aligns the amplitude distribution of high and low frequency bands, allowing the entire group to share a single quantization scaling factor, thereby significantly reducing metadata storage overhead. This not only improves the overall compression ratio but also avoids the problem of high-frequency signal amplitudes being overwhelmed by quantization noise, enhancing the model's ability to capture local details.
[0027] 4. The hardware-aware fusion computation and latency inverse transform mechanism proposed in this invention can combine the on-chip memory characteristics of the GPU to integrate quantization, packing, dequantization, and attention computation into a high-efficiency Triton kernel. During the decoding stage, this mechanism utilizes the orthogonality of the DCT transform to directly calculate the dot product of the query and the key in the frequency domain, avoiding intermediate full-precision time-domain tensor read / write operations in GPU memory. This design effectively overcomes the limitation of traditional frequency-domain methods that increase inference latency due to high transform overhead. This allows SpectrumQuant to achieve or even surpass the inference speed of the FP16 baseline while significantly saving GPU memory, significantly improving service throughput in long-context scenarios. Attached Figure Description
[0028] Figure 1 This is a flowchart of the high-fidelity key caching compression method based on spectral quantization of the present invention. Detailed Implementation
[0029] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.
[0030] Example 1 The high-fidelity key cache compression framework (SpectrumQuant) based on spectral quantization for long-context large-scale language model (LLM) inference proposed in this invention consists of the following core processing flows working closely together: 1) Spectral transformation and feature analysis module: This module, as the front end of data processing, is responsible for receiving the raw key cache data from the model, mapping it from the time domain to the frequency domain, and using the energy concentration characteristics unique to Discrete Cosine Transform (DCT) to deeply analyze the signal structure, thereby fundamentally solving the high-frequency oscillation problem introduced by Rotated Position Encoding (RoPE); 2) Frequency domain hybrid quantization encoding module: As the core of data compression, it compresses the frequency domain coefficients into a sparse representation with extremely low bits through a main frequency extraction mechanism, high-frequency pre-emphasis technology, and hybrid bit width allocation strategy, achieving the best balance between storage efficiency and signal reconstruction accuracy while significantly reducing memory usage; 3) Fusion decoding inference module: As the back end of inference execution, it uses hardware-aware fusion operators and delay inverse transform technology during the generation stage to directly calculate the attention score in the compressed domain, avoiding heavy memory read and write operations, and achieving lossless recovery and accelerated inference.
[0031] High-fidelity key caching compression methods based on spectral quantization, such as Figure 1 As shown, it includes: Step S1: Spectral Transformation and Feature Analysis Process; As the front end of data processing, it receives the original key cache data from the model, maps the original key cache data from the time domain to the frequency domain, and uses the energy concentration characteristics unique to Discrete Cosine Transform (DCT) to deeply analyze the signal structure; thereby fundamentally solving the high-frequency oscillation problem introduced by Rotated Position Encoding (RoPE); Since the periodic oscillations introduced by Rotated Position Encoding (RoPE) are difficult to compress in the time domain, this module applies DCT along the token dimension and uses the isomorphism between the RoPE rotation mechanism and the cosine basis function to transform the originally widely distributed oscillation signal into a highly sparse pulse signal in the frequency domain, thereby revealing the hidden energy concentration characteristics.
[0032] Step S2: Frequency domain hybrid quantization coding process; as the core of data compression, through the main frequency extraction mechanism, high frequency pre-emphasis technology and hybrid bit width allocation strategy, the frequency domain coefficients are compressed into a sparse representation with extremely low bits; while significantly reducing the memory usage, the best balance between storage efficiency and signal reconstruction accuracy is achieved. Step S3: Fusion Decoding and Sparse Accumulation Process; As the backend of inference execution, during the generation phase, hardware-aware fusion operators and inverse delay transform technology are used to directly calculate the attention score in the compressed domain. This avoids heavy GPU memory read / write operations, achieving lossless recovery and accelerated inference. The hardware-aware fusion computation and inverse delay transform mechanism is used to eliminate the computational overhead of frequency domain processing. This mechanism develops a deeply customized Triton fusion kernel, which avoids explicit full-size time-domain tensor reconstruction (i.e., inverse delay transform) during the decoding phase. Instead, it utilizes the orthogonality property of DCT to directly calculate the dot product of the query vector and the compressed key vector in the frequency domain and performs sparse accumulation through on-chip SRAM, thereby breaking through the memory bandwidth bottleneck and achieving end-to-end latency comparable to full-precision inference.
[0033] Example 2 The difference between the high-fidelity key caching compression method based on spectral quantization described in Example 1 and the method described in Example 1 is as follows: The specific implementation process of step S1 includes: Step S1.1: Grouping processing of the key buffer and modeling of the rotation position encoded signal, i.e., the RoPE signal; specifically including: A detailed mathematical modeling and preprocessing of the key vectors in the Transformer model (an existing network architecture model) is performed; let the key vectors of the Transformer model be... The layer input hidden state is The tensor dimension is ,in Indicates the sequence length of the current context. This represents the feature dimension of the hidden layer; for any given attention head, the key vector... That is, the original key cache data is generated through linear transformation projection; in order to effectively capture the relative position information in the sequence, modern large models widely adopt Rotated Position Encoding (RoPE). For ease of analysis and processing, the first key cache data is... Each channel pair (corresponding to the feature dimension) and In position The mathematical form of the rotational transformation is as follows:
[0034] ; Among them, frequency parameters It decays as the channel index increases; This refers to the position of the rotated feature within the sequence. , These represent the positions before and after rotation, respectively. The key vector in the th Components on each channel; , These represent the positions before and after rotation, respectively. The key vector in the th Components on each channel; This rotation-based operation introduces severe high-frequency oscillations in the time domain, severely disrupting the numerical smoothness of the channel dimension and rendering traditional quantization methods ineffective. To effectively handle this non-stationary signal during quantization, the key vector... Logically divide the token into several fixed-size groups along the token dimension (i.e., the time dimension), denoted as group size . (In this embodiment, for example, setting) All mathematical transformations, parameter calculations, and quantization operations, including the execution of DCT transformation, main frequency extraction, mixed bit width allocation and high frequency pre-emphasis processing, shared parameter quantization calculation, and data packaging in steps S1.2 to S2.3, are performed independently and in parallel within each group. This grouping strategy not only adapts to the inference characteristics of LLM streaming generation but also maintains the coherence of local location information, providing a stable data foundation for subsequent processing.
[0035] Mathematical modeling refers to analyzing the mathematical form of rotational transformations to obtain the above formula. Preprocessing refers to "logically dividing the token into several fixed-size groups along the token dimension (i.e., the time dimension)".
[0036] Step S1.2: Perform Discrete Cosine Transform (DCT) and energy concentration characteristic analysis; in order to transform the RoPE-induced periodic oscillations into an easily compressible sparse representation, the time-domain key vector is transformed using the DCT-II transform, i.e., Discrete Cosine Transform (DCT). Mapped to the frequency domain, due to the high mathematical isomorphism between the RoPE rotation mechanism and the DCT cosine basis functions, the DCT transform can extremely effectively concentrate the complex oscillatory energy dispersed in the time domain onto a few coefficients in the frequency domain. For a length of... input sequence Transformed spectral coefficients The calculation formula is as follows:
[0037] ; in, , The normalization coefficient is defined as: when hour, ;when hour, ; yes The Each component.
[0038] By performing the above transformation, the RoPE signal, which originally exhibited drastic amplitude fluctuations in the time domain, was converted into a sparse pulse signal in the frequency domain. In-depth statistical analysis revealed that the vast majority of the spectral energy was highly concentrated on a very small number of adjacent coefficients corresponding to the inherent frequency of RoPE (i.e., the dominant frequency described later), while the remaining broad spectral region manifested as a low-amplitude residual signal. This energy concentration characteristic provides a solid physical foundation for subsequent sparse coding.
[0039] The specific implementation process of step S2 includes: A strategy of dominant frequency extraction and sparse coding is employed to separate and protect high-energy signals carrying key semantics. Based on spectral analysis, the vast majority of energy is concentrated on a very small number of adjacent frequency coefficients. This strategy identifies and extracts the signal with the largest amplitude in each channel. One coefficient is used as the "master frequency" and stored with high precision (such as FP16 or INT8); while the remaining coefficients are regarded as residual signals, thereby significantly reducing data redundancy while ensuring accurate reconstruction of location information.
[0040] The Hybrid Bit Width Allocation and High-Frequency Pre-Emphasis (SpectrumQuant) framework achieves extreme compression in residual signal processing through a frequency-domain-aware adaptive quantization mechanism. Based on the difference in contribution of low-frequency and high-frequency components to the reconstruction quality, the residual spectrum is divided into low-frequency and high-frequency bands, assigned higher (e.g., 4-bit) and lower (e.g., 2-bit) bit widths, respectively. Simultaneously, to address the insufficient quantization grid utilization caused by the small dynamic range of high-frequency signals, a high-frequency pre-emphasis factor is introduced to stretch the high-frequency signal distribution before quantization, sharing quantization parameters with the low-frequency signal, thereby minimizing metadata overhead.
[0041] The specific implementation process of step S2 includes: Step S2.1: Main frequency extraction and residual signal separation; obtain the spectral coefficient matrix through the above transformation. Next, in order to preserve the positional semantic information that is crucial to inference accuracy to the greatest extent, the main frequency extraction operation is first performed; for each independent channel Automatically scan and identify the largest absolute value The coefficients are taken as the set of main frequencies, denoted as . The mathematical expression for the main frequency extraction process is: ; in, Represents the spectrum coefficient matrix In the Components on each channel This means retrieving the largest value from a vector. The extracted main frequency value is denoted as (the number of elements is missing in the original text). ( The element in the middle), the corresponding frequency index is denoted as ; To ensure accuracy, this embodiment It uses the high-precision FP16 format for storage, and The data is stored in INT8 format, and these critical data are not involved in subsequent lossy compression quantization to ensure accurate reconstruction of the RoPE rotation characteristics. Subsequently, to obtain a residual signal that is easy to compress, the spectral coefficient matrix is... The middle corresponds to The coefficients for position are set to zero, thus obtaining the residual spectrum. The calculation logic is as follows:
[0042] ; in, , Representing the residual spectrum and the original spectral coefficients in the sequence, respectively. The position Components on each channel; Because the high-energy main frequency component has been removed, the residual signal... The dynamic range was significantly reduced. This processing method greatly reduced the truncation error and quantization noise that may be generated in the subsequent quantization process, laying the foundation for high-fidelity compression.
[0043] Based on the extraction of the dominant frequency and the separation of the residual, a sparse representation of the spectral data of each channel is generated. The extraction of the dominant frequency is used to capture the energy peak of RoPE modulation and to preserve the core semantics by storing its value and frequency index. The separation of the residual takes the spectral coefficients after removing the dominant frequency as the object to be quantized and uses its significantly reduced dynamic range to reduce quantization error.
[0044] Step S2.2: Hybrid Bit Width Allocation and High-Frequency Pre-Emphasis Processing; To further compress the residual signal to the extreme, this invention employs a hybrid bit width strategy based on frequency domain characteristics. According to signal processing theory, the low-frequency part of the spectrum usually contains the global structure and main trends of the signal, while the high-frequency part is more characterized by local details or random noise. Therefore, this invention uses a hybrid bit width strategy based on frequency domain characteristics to allocate the residual spectrum. Logically divided into low-frequency bands along the frequency dimension and high frequency band low frequency band The index range is from arrive High frequency band The index range is from arrive ;
[0045] However, directly quantizing these two parts presents a challenge: the amplitude of high-frequency signals is usually much smaller than that of low-frequency signals, and if quantization parameters are shared, high-frequency information is easily lost. To solve this problem, this invention introduces a high-frequency pre-emphasis technique, which uses a preset emphasis factor before the quantization operation. The high-frequency band coefficients are amplitude stretched to align the dynamic range with the low-frequency signal; the pre-emphasized high-frequency band is denoted as... The calculation formula is as follows:
[0046] ; The key value of this step is that it allows the high-frequency band and the low-frequency band to share the same set of quantization parameters (i.e., scaling factor and zero point), thereby avoiding the additional metadata overhead of storing parameters separately for different frequency bands and improving the overall compression ratio.
[0047] Step S2.3: Shared parameter quantization calculation and data packaging; specifically including: based on the concatenated data vector Calculate the scaling factor shared at the group level. and zero point The calculation formula is as follows: ; ; in, The reference bit width is used; this invention employs a non-uniform bit allocation strategy, that is, a higher bit width is used for the low-frequency band containing important structural information (in this embodiment, it is set to...). ), while using a lower bit width for the high-frequency band (in this embodiment, setting ), To accommodate the lower bit width of the high-frequency band, an adaptation factor was introduced. The final quantification formulas are as follows:
[0048] ; ; in, In this embodiment, it is... .
[0049] Finally, the quantized integers and The entire encoding and compression process is completed by using bitwise operations to perform compact packaging and storage.
[0050] It is 1 / 5.
[0051] The specific implementation process of step S3 includes: Step S3.1: Fusion Dequantization and High-Frequency Inverse Scaling Recovery; During the decoding process in the inference stage, in order to minimize the additional latency caused by frequency domain processing, a deep fusion kernel based on the Triton language is designed; First, the packaged compressed data is read from the video memory and dequantized; The low-frequency band recovery formula is: ; in, It is a low-frequency signal obtained after dequantization. and These are the storage scaling factor and zero point, respectively; For high-frequency band recovery, an adaptation factor is also applied. Inverse transform and pre-emphasis factor The inverse transform is used to restore the original amplitude ratio. The formula for high-frequency band recovery is: ; in, It is the high-frequency signal after dequantization; After the above steps, the reconstructed residual spectrum is obtained in the shared memory SRAM inside the chip. Residual spectrum It is by and A complete spectrum signal assembled from multiple parts; Step S3.2: Deferred Inverse Transform (DCT) and Attention Score Calculation; To avoid explicitly converting the spectral coefficients back to the time-domain key vector (i.e., performing a full IDCT transform), which would result in huge memory read / write overhead and peak memory usage, this invention utilizes the orthogonality of the DCT transform and the associative law of matrix multiplication to design an innovative delayed inverse transform strategy. This includes:
[0052] Let the query vector Query be The DCT basis matrix is The traditional process for calculating attention scores is usually expressed as: ,in This invention uses mathematical derivation to calculate attention scores. The calculation process is rewritten as follows:
[0053] ; In practice, the fusion operator first calculates the query vector. With frequency domain coefficients (Including reconstructed residuals) and clock speed The frequency domain attention score is obtained by taking the dot product of the DCT basis matrix stored in the on-chip shared memory SRAM. The frequency domain fraction is projected back to the time domain; this process is completed entirely in a pipelined manner in registers and SRAM, completely avoiding the intermediate full-precision time domain tensor memory read and write, and greatly breaking through the memory bandwidth bottleneck.
[0054] Step S3.3: Sparse frequency accumulation and final output; For the unquantized frequency portion, due to its high sparsity in the frequency domain, the fusion kernel uses sparse accumulation for efficient processing. Based on the read frequency index... Sum of values Calculations and updates are performed only at the corresponding non-zero positions:
[0055] ; This sparse computation design avoids expanding sparse clock frequency data into a dense matrix, greatly saving memory bandwidth and computational resources. Finally, the accumulator... The information stored in the middle is the attention score in the time domain. This result can be directly used in subsequent Softmax operations to generate attention weights, thus completing the key cache access stage in the entire long context inference process. The final attention weights are obtained after the Softmax operation.
[0056] Table 1 shows the comparative method of the present invention; Table 1 Table 2 shows the score comparison between the present invention and the comparison method on the benchmark dataset LongBench.
[0057] Table 2 Table 3 shows the score comparison between the present invention and the comparison method on the benchmark dataset IF-Eval.
[0058] Table 3 Table 4 shows the score comparison between the present invention and the comparison method on the benchmark dataset MMLU.
[0059] Table 4 Table 5 shows the score comparison between the present invention and the comparison method on the benchmark dataset GSM8K.
[0060] Table 5 As shown in Tables 2 to 5, compared with other methods, SpectrumQuant achieves a lower average bit width (3.71 bits) and its average score in various long context tasks is not only better than other quantization methods, but also surpasses the FP16 baseline in some metrics, proving that the present invention has significant superiority and robustness in long text reasoning scenarios.
[0061] Example 3 A computer device includes a memory and a processor. The memory stores a computer program, and the processor, when executing the computer program, implements the steps of the high-fidelity key cache compression method based on spectral quantization described in Embodiment 1 or 2.
[0062] Example 4 A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the high-fidelity key cache compression method based on spectral quantization described in Embodiment 1 or 2.
[0063] Example 5 A high-fidelity key caching compression and inference system based on spectral quantization includes: The spectral transformation and extraction module is configured to: convert the time-domain key vector into a frequency-domain representation based on the discrete cosine transform, and identify and separate the dominant frequency components with concentrated energy, thereby realizing the sparsity decomposition of the signal; The hybrid bit-width coding module is configured to: align the dynamic range of different frequency bands through high-frequency pre-emphasis technology, and use a hybrid precision strategy to perform non-uniform quantization on low-frequency and high-frequency residual signals in order to minimize storage footprint; The fusion decoding inference module is configured to: in the inference generation stage, use fusion operators to directly calculate attention scores and perform sparse accumulation in the frequency domain, avoiding explicit time-domain reconstruction and achieving low-latency long text inference.
Claims
1. A high-fidelity key caching compression method based on spectral quantization, characterized in that, include: Step S1: Spectral transformation and feature analysis process; It receives raw key buffer data from the model, maps the raw key buffer data from the time domain to the frequency domain, and uses the energy concentration characteristics unique to Discrete Cosine Transform (DCT) to deeply analyze the signal structure. Step S2: Frequency domain hybrid quantization coding process; through the main frequency extraction mechanism, high frequency pre-emphasis technology and hybrid bit width allocation strategy, the frequency domain coefficients are compressed into a sparse representation with extremely low bits; Step S3: Fusion decoding and sparse accumulation process; By utilizing hardware-aware fusion operators and inverse delay transform techniques, attention scores are calculated directly in the compressed domain.
2. The high-fidelity key caching compression method based on spectral quantization according to claim 1, characterized in that, The specific implementation process of step S1 includes: Step S1.1: Grouping processing of the key buffer and modeling of the rotation position encoded signal, i.e., the RoPE signal; specifically including: Mathematical modeling and preprocessing of the key vectors in the Transformer model; let the key vectors of the Transformer model be... The layer input hidden state is The tensor dimension is ,in Indicates the sequence length of the current context. This represents the feature dimension of the hidden layer; for any given attention head, the key vector... That is, the original key cache data is generated through linear transformation projection; the first key cache data is generated through linear transformation projection. Each channel is located at the position The mathematical form of the rotational transformation is as follows: ； Among them, frequency parameters It decays as the channel index increases; This refers to the position of the rotated feature within the sequence. , These represent the positions before and after rotation, respectively. The key vector in the th Components on each channel; , These represent the positions before and after rotation, respectively. The key vector in the th Components on each channel; key vector Logically divide the token into several groups of fixed size, denoted as group size . All mathematical transformations, parameter calculations, and quantization operations, including the execution of DCT transformation, main frequency extraction, mixed bit width allocation and high frequency pre-emphasis processing, shared parameter quantization calculation and data packaging in steps S1.2 to S2.3, are performed independently and in parallel within each group. Step S1.2: Perform Discrete Cosine Transform (DCT) and energy concentration characteristic analysis; use DCT-II transform, i.e., Discrete Cosine Transform (DCT), to transform the time-domain key vector... Mapped to the frequency domain, for lengths of input sequence Transformed spectral coefficients The calculation formula is as follows: ； in, , The normalization coefficient is defined as: when hour, ;when hour, ; yes The Each component.
3. The high-fidelity key caching compression method based on spectral quantization according to claim 1, characterized in that, The specific implementation process of step S2 includes: A strategy of extracting the main frequency and sparse coding is adopted to separate and protect the high-energy signal carrying key semantics; Hybrid bit width allocation and high-frequency pre-emphasis achieve extreme compression in residual signal processing through a frequency-domain-aware adaptive quantization mechanism. Based on the difference in contribution of low-frequency and high-frequency components to the reconstruction quality, the residual spectrum is divided into low-frequency and high-frequency bands. At the same time, a high-frequency pre-emphasis factor is introduced to stretch the distribution of high-frequency signals before quantization and share quantization parameters with low-frequency signals, thereby minimizing metadata overhead.
4. The high-fidelity key caching compression method based on spectral quantization according to claim 3, characterized in that, The specific implementation process of step S2 includes: Step S2.1: Main frequency extraction and residual signal separation; after obtaining the spectral coefficient matrix Next, firstly, the main frequency extraction operation is performed; for each independent channel... Automatically scan and identify the largest absolute value The coefficients are taken as the set of main frequencies, denoted as . The mathematical expression for the main frequency extraction process is: ； in, Represents the spectrum coefficient matrix In the Components on each channel This means retrieving the largest value from a vector. The extracted main frequency value is denoted as (the number of elements is missing in the original text). The corresponding frequency index is denoted as ; Spectrum coefficient matrix The middle corresponds to The coefficients for position are set to zero, thus obtaining the residual spectrum. The calculation logic is as follows: ； in, , Representing the residual spectrum and the original spectral coefficients in the sequence, respectively. The position Components on each channel; Step S2.2: Hybrid bit width allocation and high-frequency pre-emphasis processing; the residual spectrum... Logically divided into low-frequency bands along the frequency dimension and high frequency band low frequency band The index range is from arrive High frequency band The index range is from arrive ; High-frequency pre-emphasis technology is introduced, using a preset emphasis factor before quantization. The high-frequency band coefficients are amplitude stretched to align the dynamic range with the low-frequency signal; the pre-emphasized high-frequency band is denoted as... The calculation formula is as follows: ； Step S2.3: Shared parameter quantization calculation and data packaging; specifically including: based on the concatenated data vector Calculate the scaling factor shared at the group level. and zero point The calculation formula is as follows: ； ； in, The reference bit width is used; an adaptation factor is introduced. The final quantification formulas are as follows: ； ； in, ; Finally, the quantized integers and The entire encoding and compression process is completed by using bitwise operations to perform compact packaging and storage.
5. The high-fidelity key caching compression method based on spectral quantization according to claim 4, characterized in that, It is 1 / 5.
6. The high-fidelity key buffer compression method based on spectral quantization according to any one of claims 1-5, characterized in that, The specific implementation process of step S3 includes: Step S3.1: Fusion dequantization and high-frequency inverse scaling recovery; design a deep fusion kernel based on the Triton language; first, read the packaged compressed data from the video memory and perform dequantization; the low-frequency band recovery formula is: ； in, It is a low-frequency signal obtained after dequantization. and These are the storage scaling factor and zero point, respectively; For high-frequency band recovery, an adaptation factor is also applied. Inverse transform and pre-emphasis factor The inverse transform is used to restore the original amplitude ratio. The formula for high-frequency band recovery is: ； in, It is the high-frequency signal after dequantization; After the above steps, the reconstructed residual spectrum is obtained in the shared memory SRAM inside the chip. Residual spectrum It is by and A complete spectrum signal assembled from multiple parts; Step S3.2: Deferred Inverse Transform (Deferred IDCT) and Attention Score Calculation; including: Let the query vector Query be The DCT basis matrix is Attention score is derived mathematically. The calculation process is rewritten as follows: ； The fusion operator first calculates the query vector. With frequency domain coefficients The dot product is used to obtain the frequency domain attention score; subsequently, the DCT basis matrix stored in the on-chip shared memory SRAM is used. Projecting the frequency domain fraction back into the time domain; Step S3.3: Sparse frequency accumulation and final output; based on the read frequency index Sum of values Calculations and updates are performed only at the corresponding non-zero positions: ； Finally, the accumulator The information stored in the middle is the attention score in the time domain. The final attention weights are obtained after the Softmax operation.
7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the high-fidelity key cache compression method based on spectral quantization as described in any one of claims 1-6.
8. A computer-readable storage medium having a computer program stored thereon, the program, when executed by a processor, implementing the steps of the high-fidelity key cache compression method based on spectral quantization as described in any one of claims 1-6.
9. A high-fidelity key caching compression and inference system based on spectral quantization, characterized in that, include: The spectral transformation and extraction module is configured to: convert the time-domain key vector into a frequency-domain representation based on the discrete cosine transform, and identify and separate the dominant frequency components with concentrated energy, thereby realizing the sparsity decomposition of the signal; The hybrid bit-width coding module is configured to: align the dynamic range of different frequency bands through high-frequency pre-emphasis technology, and use a hybrid precision strategy to perform non-uniform quantization on low-frequency and high-frequency residual signals in order to minimize storage footprint; The fusion decoding inference module is configured to: in the inference generation stage, use fusion operators to directly calculate attention scores and perform sparse accumulation in the frequency domain, avoiding explicit time-domain reconstruction and achieving low-latency long text inference.