An adaptive kv cache compression method, system and device for an audio-text multimodal large model

Abstract

The present application relates to the cross field of artificial intelligence and computer architecture, in particular to an adaptive KV cache compression method, system and device for an audio-text multi-modal large model. The method comprises opening a full-precision hot table and a low-bit cold table in the video memory; writing the newly generated audio or text Token in the FP16 format into the hot table, and when the number of Tokens in the hot table reaches a preset threshold, gradually migrating the Tokens in the hot table to the cold table; identifying the modal type of the Token during the migration process, selecting a corresponding quantization grouping strategy according to the identified modal type, and performing INT2 compression storage to the cold table after online calculation of the quantization parameters of the Token to be migrated. Through the cooperation of the above steps, the present application realizes the improvement of the model inference throughput and the natural full-duplex human-computer interaction under the premise of ensuring the audio synthesis quality.

Description

Technical Field

[0001] This invention relates to the intersection of artificial intelligence and computer architecture, specifically to a KV caching compression method, system, and device for audio-text multimodal large language models. Background Technology

[0002] In recent years, large language models based on the Transformer architecture have made groundbreaking progress in the field of natural language processing. With the continuous evolution of technology, the model's capabilities have expanded from single text processing to the understanding and generation of multimodal data such as visual and auditory data. Although existing models have achieved multimodal fusion at the architectural level, they still face severe challenges in key-value caching during actual deployment, especially when dealing with long speech dialogue scenarios.

[0003] In the Transformer's self-attention mechanism, to avoid redundant computation, the system must store the key and value vectors of all historical tokens in GPU memory. The GPU memory usage of the KV cache increases linearly with the sequence length. This problem is particularly prominent for audio multimodal models, mainly manifested in the following aspects:

[0004] 1. The high density of audio tokens. Text tokens typically correspond to a single word or character, while audio tokens are generated at a fixed frame rate (e.g., 25Hz or 32Hz). This means that just one minute of voice interaction can generate 1500 to 1920 tokens, far exceeding the amount of text with the same information. For a 10-minute conversation, the sequence length can quickly exceed 15k tokens, causing a dramatic expansion of the key-value cache.

[0005] 2. Memory bandwidth bottleneck. Large audio models strive for extremely low first-word latency and high streaming generation speed. During the decoding phase, each new token requires reading the entire key-value cache from GPU memory for attention calculation. When the cache size becomes too large, GPU memory bandwidth becomes the primary bottleneck, leading to idle computing units and a significant decrease in inference speed.

[0006] 3. Limited VRAM. On edge devices or consumer-grade GPUs (such as the NVIDIA L40 or 4090), VRAM capacity is extremely limited (typically 24GB to 48GB). Full-precision KV cache (FP16) can quickly exhaust VRAM resources, causing Out-Of-Memory errors, forcing the model to truncate historical dialogue, and severely impacting the user experience. Summary of the Invention

[0007] In view of this, this invention proposes an adaptive KV cache compression method, system, and device for large audio-text multimodal models. By using dual-zone hybrid precision storage, combined with modality-aware differential quantization, progressive token migration, and a full-duplex interrupt-aware cache reset mechanism, it solves the problems of video memory capacity bottleneck, memory bandwidth bottleneck, and interrupt response latency in long context inference of large audio-text multimodal models. While ensuring the quality of audio synthesis, it improves the model inference throughput and realizes natural full-duplex human-computer interaction.

[0008] The technical solution adopted in this invention is as follows:

[0009] An adaptive key-value buffer compression method for large audio-text multimodal models includes:

[0010] A full-precision hot table and a low-bit cold table are allocated in the video memory; the hot table is an FP16 full-precision storage area based on a circular buffer, and the cold table is an INT2 low-bit compressed storage area based on block addressing.

[0011] Newly generated audio or text tokens are written to the hot table in FP16 format. When the number of tokens in the hot table reaches a preset threshold, the tokens in the hot table are gradually migrated to the cold table.

[0012] The modality type of the token is identified during the migration process, including text modality and audio modality;

[0013] Based on the identified modality type, a corresponding quantization grouping strategy is selected. After the quantization parameters of the token to be migrated are calculated online, they are compressed using INT2 and stored in a cold table. The quantization grouping strategy is an adaptive online quantization strategy that detects outliers. Different grouping dimensions and group sizes are used for text modalities and audio modalities. The online calculated quantization parameters include a scaling factor. and zero point .

[0014] Furthermore, the preset threshold is ; The preset number of full-precision stored tokens for the hot table, B is the number of token batches in a single migration; the gradual migration of tokens from the hot table to the cold table adopts a batch pop-up mechanism, specifically as follows:

[0015] When the number of tokens in the hot list reaches At that time, the oldest B tokens in the hot table are taken as the tokens to be migrated, and after quantization and compression, they are added to the cold table. At the same time, the hot table write pointer is moved forward by B steps to free up the hot table storage space.

[0016] Furthermore, The value is 128. The value is 32, and the heat meter is used to store the most recently generated data. Each token is stored in a cold table, which is used to store historical tokens that have overflowed from the hot table.

[0017] Furthermore, the calculation formula for the online quantization parameters is as follows:

[0018] ;

[0019] ;

[0020] in, For each set of values ​​to be quantified, This is the scaling factor for this set of values. This is the zero point of the set of values. This is a rounding operation; the calculation of the quantization parameters is completed in the on-chip SRAM and does not occupy DRAM bandwidth.

[0021] Furthermore, the quantization grouping strategy is as follows:

[0022] For the key vector of the text modality, it is grouped by the channel dimension, with a group size of 32;

[0023] For the Value vector of the text modality, it is grouped by the Token dimension, with a group size of 32;

[0024] For the key vector of the audio modality, it is grouped by the channel dimension, with a group size of 16;

[0025] For the value vector of the audio modality, it is grouped by the token dimension, with a group size of 32.

[0026] Furthermore, the above adaptive KV cache compression method also includes a full-duplex interrupt handling step S5:

[0027] When an interrupt signal is detected, the historical token data in the cold table remains unchanged, the write pointer of the hot table is reset to the starting position of the current generation round, and the unarchived token data in the hot table that has been crossed by the pointer is logically marked as invalid, without the need to perform dequantization or memory release operations, thus achieving lossless state rollback.

[0028] An adaptive key-value (KV) caching compression system for large audio-text multimodal models includes: a key-value caching optimization controller and a large audio-text multimodal model;

[0029] The KV cache optimization controller is communicatively connected to the audio-text multimodal large model and is used to execute the above-mentioned adaptive KV cache compression method for the audio-text multimodal large model.

[0030] Furthermore, the KV cache optimization controller is hardware logic residing on the chip, or a kernel driver that runs with a higher priority than the large model inference task, used to intercept all read and write requests from the audio-text multimodal large model to the KV cache, and to perform address mapping and quantization scheduling; the audio-text multimodal large model is a model based on the Transformer Decoder architecture, used to process interleaved audio and text token sequences.

[0031] An adaptive key-value buffer compression device for large audio-text multimodal models includes:

[0032] At least one AI accelerator, wherein the AI ​​accelerator is a GPU or an NPU;

[0033] Memory, which stores computer-executable instructions;

[0034] When the executable instructions are executed by at least one AI accelerator, the above-mentioned adaptive KV buffer compression method for audio-text multimodal large models is implemented.

[0035] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described adaptive KV buffer compression method for audio-text multimodal large models.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] (1) Breaking the limit of video memory capacity: This invention adopts dual-zone mixed precision storage of hot table in FP16 format and cold table in INT2 format, reducing the video memory usage of KV cache to 1 / 8 of the original full precision storage, so that a single 24GB video memory graphics card can support audio dialogue context for several hours, completely solving the model "forgetting" problem in long voice interaction.

[0038] (2) Significantly improves inference speed: This invention reduces the amount of KV data transmitted to the memory through batch quantization compression, which alleviates the pressure on memory bandwidth. At the same time, the migration process overlaps with the calculation process to hide the delay, ensuring that the audio-text multimodal large model maintains high throughput when generating text and audio in parallel, and effectively reducing the first character delay.

[0039] (3) Ensure audio generation quality: The modal perception differential quantization strategy adopted is adapted to the feature distribution pattern of text and audio, especially the 16 fine-grained channel grouping of audio key vector, which effectively isolates outliers, prevents quantization noise from damaging the timbre and clarity of speech, and makes the listening effect comparable to the full-precision model, avoiding problems such as muteness and popping.

[0040] (4) Achieve natural full-duplex interaction: Based on the interrupt reset mechanism of hot / cold table structure, microsecond-level state rollback can be achieved without complex video memory operations. The model can be interrupted by the user at any time and respond quickly to new instructions, breaking the traditional mechanical interaction mode of "you speak and I listen", and greatly improving the naturalness and human-like experience of human-computer interaction.

[0041] (5) High deployment flexibility: The hardware and software collaborative design scheme does not require modification of the core structure of the audio-text multimodal large model. It only achieves cache management and quantization scheduling through the KV cache optimization controller. It can be adapted to the existing mainstream Transformer Decoder architecture model, and is compatible with various AI accelerators such as GPU and NPU. It can be deployed on different hardware platforms such as servers and edge devices. Attached Figure Description

[0042] Figure 1 Flowchart of management and batch pop-up migration in the adaptive KV cache compression method of the example;

[0043] Figure 2 This is a flowchart of modality-aware differential quantization and cold table storage in the adaptive KV cache compression method of the embodiment;

[0044] Figure 3 This is a schematic diagram of the full-duplex interrupt-aware cache reset mechanism in the adaptive KV cache compression method of the embodiment. Detailed Implementation

[0045] This embodiment provides an adaptive key-value (KV) cache compression method, system, and device for large audio-text multimodal models, used to optimize memory usage and response speed during model inference. It is applicable to large audio multimodal models based on the TransformerDecoder architecture, specifically LLaMA-Omni2 or Mini-Omni2, running on servers or edge devices equipped with high-performance AI accelerators. The core implementation achieves efficient KV cache compression, adaptive modal quantization, and fast full-duplex interrupt response, reducing memory usage while ensuring high-quality audio generation and smooth interaction. The AI ​​accelerator is a GPU or a dedicated NPU.

[0046] Example 1: This example provides an adaptive key-value (KV) cache compression system for large audio-text multimodal models, including an audio-text multimodal model and a KV cache optimization controller. The two are communicatively connected and collaboratively manage the KV cache. The audio-text multimodal model, based on the Transformer Decoder architecture, is responsible for processing interleaved text and audio sequences and generating corresponding tokens. Text tokens come from a text tokenizer, and audio tokens come from an audio adapter or a speech tokenizer. The KV cache optimization controller is either on-chip hardware logic or a kernel driver with a higher priority than the large model inference task. Its core function is to intercept all read and write requests to the KV cache, perform address mapping, quantization scheduling, modality recognition, and interrupt handling. It is the core control unit for implementing adaptive KV cache compression.

[0047] Example 2: This example provides an adaptive key-value (KV) cache compression method for this model. Through the collaborative execution of a KV cache optimization controller and an audio-text multimodal large model, hierarchical storage and efficient migration of tokens are achieved, thus completing the optimized control of the KV cache. Specifically, it includes the following steps:

[0048] S1. Cache Initialization: The KV cache optimization controller allocates two logical storage areas in the video memory—a hot table and a cold table—and completes the initialization configuration. This dual-area hybrid precision storage architecture constructs a hierarchical memory management unit to adapt to the streaming generation characteristics of large multimodal audio models. The specific design of the two logical storage areas is as follows:

[0049] The hot table adopts a circular buffer design and is an FP16 full-precision storage area used to store the most recently generated Lres tokens. In this embodiment, L_res is set to 128, covering the model's current thinking and speaking window, ensuring that the recent context (i.e., high-frequency access tokens) with recency bias characteristics in the hot table have no precision loss, thus ensuring the accuracy of inference and generation.

[0050] The cold table adopts a block addressing design and is an INT2 (2-bit) low-bit compressed storage area used to store historical tokens that overflow from the hot table. The low-bit compression greatly reduces the memory usage while retaining complete context information and avoiding context loss.

[0051] S2. Token Writing and Progressive Migration: This step employs a batch pop mechanism based on a dual-zone hybrid precision storage architecture to achieve efficient token migration. The core purpose of this mechanism is to ensure that the hot table always stores the latest and most relevant Lres tokens at full precision, avoiding wasting precision resources on outdated tokens. When the hot table reaches a set capacity threshold, the oldest tokens are migrated to the cold table in batches. This ensures context integrity while continuously freeing up storage space for new tokens, adapting to the characteristics of streaming model generation, and providing a data foundation for modality recognition and quantization operations during subsequent migration. This embodiment utilizes the recency bias of the Transformer attention mechanism and employs a progressive sliding window strategy to manage the KV cache. Figure 1 As shown, the specific operation process is as follows:

[0052] Newly generated audio or text tokens are first written to a hot sheet in FP16 full-precision format. When the number of tokens in the hot sheet reaches a preset threshold Lres+B, a migration procedure is triggered. The oldest B tokens in the hot sheet are used as the data to be migrated and enter the migration process. Modality recognition and quantization are then performed during the migration process. In this embodiment, Lres is set to 128, B is set to 32, and the threshold is 128+32=160.

[0053] Modality identification during the transfer process: This step employs a modality-aware asymmetric grouping quantization strategy because audio features, such as Whisper spectral features and SNAC token embeddings, differ significantly from text features in their outlier distribution. Traditional unified grouping quantization can easily lead to distortion problems such as muffled or popped sounds in audio generation. Specifically, some channels in Whisper features correspond to specific frequency band energies, and the activation values ​​of silent and speech segments differ significantly. SNAC token embeddings are high-dimensional vectors, and the variance of the embedding vectors varies across different codebook levels, requiring differentiated grouping to adapt to this characteristic. Placing modality identification during the transfer process allows for precise matching of quantization strategies to the tokens to be transferred, avoiding invalid identification and improving transfer and quantization efficiency. Specific grouping configurations are shown in Table 1.

[0054] Table 1: Configuration table of differentiated quantization grouping strategies for different modalities.

[0055]

[0056] The specific operation process is as follows: During the migration of tokens from the hot table to the cold table, the KV cache optimization controller intercepts the B tokens to be migrated, identifies their modal types, and loads the corresponding quantization grouping parameters according to the modal types. Specifically, the tokens from the text tokenizer are marked as follows: Tokens from audio adapters or voice tokenizers are The controller loads the corresponding quantization grouping parameters, including grouping dimension and group size, from the preset configuration register based on the identified modal labels, laying the foundation for subsequent quantization operations.

[0057] S4. Online Quantization and Cold Table Storage: After completing modal identification, the quantization grouping strategy is loaded according to the modal type identified in step S3, and the B tokens to be migrated are grouped; for each group value g, the quantization parameters are calculated in real time in the on-chip SRAM. and After INT2 compression is completed, the compressed token block is written to the cold table, while the hot table write pointer is slid to free up storage space, completing the entire migration, identification, and quantization process. This step employs an adaptive online quantization strategy that detects outliers, achieving extremely low-loss compression of the tokens to be migrated to the cold table, such as... Figure 2 As shown, the specific operation includes the following sub-steps:

[0058] 1) Calculation of quantization parameters: For each group of values ​​to be quantized At the instant the token is migrated from the hot table to the cold table, the KV cache optimization controller uses on-chip SRAM to temporarily calculate the quantization parameters without occupying DRAM bandwidth. The calculation formula is as follows: ; ;

[0059] in, Scaling factor The value is zero, and round is the rounding operation. This linear mapping ensures that the full range of the value g is fully utilized in the 2-bit (4) quantization levels, minimizing rounding errors.

[0060] 2) Differentiated grouping: Combining the modal grouping strategy in step 3, a customized grouping method is adopted for outlier patterns with different Key and Value vectors to avoid a single outlier increasing the quantization scaling factor of the entire layer: Key vector: Grouped along the channel dimension to isolate persistent high-amplitude feature channels in independent groups;

[0061] Value vector: Grouped along the Token dimension, it restricts transient spikes affecting a single Token to the local space of that Token.

[0062] All quantization parameters are calculated in real time based on the tokens input in the current batch, without any offline calibration steps. This allows for continuous adaptation to changes in the current numerical distribution, ensuring low loss during the compression process and working in synergy with modality recognition during the migration process to improve overall efficiency.

[0063] S5: Full-Duplex Interrupt Handling: For full-duplex functionality in a large multimodal audio model, i.e., simultaneous voice output and real-time monitoring, supporting bidirectional synchronous interaction, it leverages the advantages of a dual-zone storage architecture and designs pointer rollback logic to achieve microsecond-level interrupt response and lossless state rollback, forming a cache reset mechanism based on full-duplex interrupt awareness. This mechanism eliminates the need for complex garbage collection, dequantization, or memory release operations, avoiding memory fragmentation, achieving microsecond-level state switching, perfectly supporting full-duplex interaction, and enhancing the naturalness and human-like experience of the interaction. Figure 3 As shown, the specific operation includes the following sub-steps:

[0064] When an interrupt signal is detected, i.e., the model outputs a special state Token irq, the KV cache optimization controller keeps the historical Token data in the cold table unchanged, and only resets the write pointer of the hot table to the beginning of the current generation round. The unarchived Token data in the hot table that is crossed by the rollback pointer is the interrupted erroneous generation content. It still exists physically in SRAM, but is logically marked as invalid. The next generation will directly overwrite this part of the address, completing the lossless state rollback, and supporting users to interrupt and switch interactively in real time.

[0065] Example 3: This example also provides an adaptive KV buffer compression device for large audio-text multimodal models, including:

[0066] At least one AI accelerator, which is a GPU or a dedicated NPU, is used to provide hardware support for model inference and quantization computation;

[0067] Memory, which stores computer-executable instructions;

[0068] When the executable instructions in memory are executed by at least one AI accelerator, all the steps of the above adaptive KV cache compression method are implemented.

[0069] Example 4: This example also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, i.e., an AI accelerator, it implements all the steps of the above-described adaptive KV caching compression method for audio-text multimodal large models, and can be used for program deployment and reuse.

[0070] In summary, this invention deeply analyzes the pain points of audio-text multimodal large models in long context reasoning. Through modality-aware differential quantization and streaming-friendly memory management, it successfully breaks through the memory wall, providing key technical support for deploying an all-around voice assistant on limited hardware.

Claims

1. An adaptive KV cache compression method for an audio-text multimodal large model, characterized in that, include: Create a full-precision hot table and a low-bit cold table in the video memory. The hot table is an FP16 full-precision storage area based on a circular buffer, and the cold table is an INT2 low-bit compressed storage area based on block addressing. Newly generated audio or text tokens are written to the hot table in FP16 format. When the number of tokens in the hot table reaches a preset threshold, the tokens in the hot table are gradually migrated to the cold table. The modality type of the token is identified during the migration process, including text modality and audio modality; Select the corresponding quantization grouping strategy based on the identified modality type, and after calculating the quantization parameters of the token to be migrated online, compress it using INT2 and store it in a cold table; The quantization grouping strategy is an adaptive online quantization strategy that perceives outliers. Text modalities and audio modalities use different grouping dimensions and group sizes. The online-calculated quantization parameters include scaling factors. and zero point .

2. The method according to claim 1, characterized in that, The preset threshold is ; The preset number of full-precision stored tokens for the hot table, B is the number of token batches in a single migration; the gradual migration of tokens from the hot table to the cold table adopts a batch pop-up mechanism, specifically as follows: When the number of tokens in the hot list reaches At that time, the oldest B tokens in the hot table are taken as the tokens to be migrated, and after quantization and compression, they are added to the cold table. At the same time, the hot table write pointer is moved forward by B steps to free up the hot table storage space.

3. The method according to claim 2, characterized in that, The value is 128. The value is 32, and the heat meter is used to store the most recently generated data. Each token is stored in a cold table, which is used to store historical tokens that have overflowed from the hot table.

4. The method according to claim 1, characterized in that, The formula for calculating the online quantization parameters is as follows: ； ； in, For each set of values ​​to be quantified, This is the scaling factor for this set of values. This is the zero point of the set of values. This is a rounding operation; the calculation of the quantization parameters is completed in the on-chip SRAM and does not occupy DRAM bandwidth.

5. The method according to claim 1, characterized in that, The quantization grouping strategy is as follows: For the key vector of the text modality, it is grouped by the channel dimension, with a group size of 32; For the Value vector of the text modality, it is grouped by the Token dimension, with a group size of 32; For the key vector of the audio modality, it is grouped by the channel dimension, with a group size of 16; For the value vector of the audio modality, it is grouped by the token dimension, with a group size of 32.

6. The method according to claim 1, characterized in that, The above adaptive KV buffer compression method also includes a full-duplex interrupt handling step S5: When an interrupt signal is detected, the historical token data in the cold table remains unchanged, the write pointer of the hot table is reset to the starting position of the current generation round, and the unarchived token data in the hot table that has been crossed by the pointer is logically marked as invalid, without the need to perform dequantization or memory release operations, thus achieving lossless state rollback.

7. An adaptive key-value (KV) buffer compression system for large audio-text multimodal models, characterized in that, The system includes: a KV cache optimization controller and an audio-text multimodal large model; The KV cache optimization controller is communicatively connected to the audio-text multimodal large model and is used to perform the method as described in any one of claims 1 to 6.

8. The system according to claim 7, characterized in that, The KV cache optimization controller is hardware logic residing on the chip, or a kernel driver that runs at a higher priority than the large model inference task. It is used to intercept all read and write requests from the audio-text multimodal large model to the KV cache and to perform address mapping and quantization scheduling. The audio-text multimodal large model is a model based on the Transformer Decoder architecture, used to process interleaved audio and text token sequences.

9. An adaptive key-value (KV) buffer compression device for large audio-text multimodal models, characterized in that, include: At least one AI accelerator, wherein the AI ​​accelerator is a GPU or an NPU; Memory, which stores computer-executable instructions; When the executable instructions are executed by at least one AI accelerator, the method described in any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the above-described adaptive KV buffer compression method for audio-text multimodal large models.

Images