Long context audio inference method and system based on semantic anchor key-value cache compression

Abstract

The application discloses a long-context audio inference method and system based on semantic anchor key-value cache compression, and relates to the technical field of speech processing and natural language processing. Long-context original audio data is obtained, and a timestamped transcription text embedding and an initial audio embedding are generated; the initial audio embedding is divided into semantic interval and background interval feature slices based on the timestamp, the semantic interval slices are pooled and fused with the transcription text embedding to generate a semantic anchor embedding; the semantic anchor embedding and the background interval slices are spliced into a compressed input sequence and input into a large model base to construct an initial key-value cache; when autoregressive decoding, a time decay coefficient is used to update the cumulative attention score of each key-value pair, and only the key-value pairs with the highest cumulative score and the latest key-value pairs are retained when the cache reaches the upper limit, and the decoding output inference result is continued. The application greatly reduces the video memory overhead of long audio inference, significantly compresses the key-value cache, and effectively maintains the semantic understanding accuracy.

Description

Technical Field

[0001] This invention relates to the fields of speech processing and natural language processing, and in particular to a long contextual audio inference method and system based on semantic anchor key-value caching compression. Background Technology

[0002] In recent years, speech large language models (SLLM) have demonstrated outstanding capabilities in tasks such as summarizing long meeting minutes, multi-turn dialogues, and long audio logical reasoning. Modern models mostly adopt a unified end-to-end architecture, directly perceiving and reasoning about audio features, thereby more accurately capturing complex multi-dimensional information such as speech content, prosody, and emotion.

[0003] However, when processing long-context audio inference, the model faces severe computational and memory challenges. Due to the quadratic complexity of the self-attention mechanism in the Transformer architecture, and the linear growth of the key-value cache (KVCache) in the pre-filling and decoding stages, the memory overhead for processing long audio sequences is enormous, severely restricting the real-time deployment of the model and the application expansion in long audio scenarios.

[0004] Existing technologies primarily alleviate pressure by compressing the key-value cache (KV Cache) of text modalities, but their effectiveness is limited when directly applied to speech modalities. On the one hand, speech input has extremely high temporal redundancy and extremely low unit information density, and the original audio features exhibit severe dispersion in the attention matrix, making it difficult for traditional expulsion methods to accurately identify semantic anchors and easily leading to semantic drift. On the other hand, existing methods lack a dynamic compression mechanism in the decoding stage, failing to efficiently filter meaningless background tokens and resulting in significant waste of GPU memory.

[0005] Therefore, how to effectively achieve dynamic compression of KV Cache in speech modalities, while reducing the consumption of computing resources and maintaining the fidelity of long audio inference and the depth of semantic understanding, is a key problem that urgently needs to be solved in the current field. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the difficulties of the prior art in processing long context audio inference, which is caused by high memory occupation of key-value cache (KV Cache) and low density of audio feature information, resulting in low inference efficiency and semantic understanding distortion. The present invention provides a long context audio inference method and system based on semantic anchoring key-value cache compression.

[0007] Therefore, the technical solution adopted by the present invention is as follows:

[0008] In a first aspect, this invention proposes a long-context audio inference method based on semantic anchored key-value caching compression, comprising the following steps:

[0009] S1. Obtain long-context raw audio data, the raw audio data including the speech part of the user command and the audio content to be processed; perform text recognition on the raw audio data, generate a timestamped transcription fragment and map it into a transcription text embedding, and encode the raw audio data into an initial audio embedding.

[0010] S2. Based on the timestamp boundary, the initial audio embedding is divided into semantic audio interval feature slices and background audio interval feature slices; pooling aggregation is performed on the semantic audio interval feature slices to obtain the converged audio embedding; the converged audio embedding is fused with the corresponding transcribed text embedding to generate semantic anchoring embedding.

[0011] S3. Semantic anchoring embedding and background audio interval feature slices are concatenated in chronological order to form a compressed input sequence. The compressed input sequence is then input into the large model base and an initial key-value cache is constructed through a pre-filling stage.

[0012] S4. In the autoregressive decoding stage, the cumulative attention score of each key-value pair is updated using a preset time decay coefficient. When the key-value cache reaches the storage budget limit, the key-value pairs with the highest cumulative scores and the latest key-value pairs are retained, while the remaining key-value pairs are evicted. Decoding continues with the updated key-value cache until the inference result for the user instruction is output.

[0013] Furthermore, in S2, the timestamps carried by each transcribed segment include start and end times, and the semantic audio interval feature slice is the embedded frame interval covered by the start and end times of each transcribed segment; the background audio interval feature slice is the embedded frame interval corresponding to the time gap between two adjacent semantic audio interval feature slices; when the timestamps of adjacent transcribed segments are connected at the boundary, it is determined that there is no background audio interval feature slice between them.

[0014] Further, in S2, the fusion specifically involves: adding the converged audio embeddings and transcribed text embeddings corresponding to the same transcribed fragment element by element; and traversing all transcribed fragments to generate a complete semantic anchoring embedding sequence.

[0015] Furthermore, the length of the compressed input sequence is less than the length of the initial audio embedding; the pre-padding stage performs parallel self-attention calculations on all lexical units in the compressed input sequence to generate an initial key-value cache.

[0016] Furthermore, in S4, at each decoding step, a single-step attention score is calculated based on the query vector of the currently generated lexical and all key vectors in the key-value cache; the cumulative attention score vector of the previous decoding step is multiplied by the time decay coefficient and then added to the single-step attention score to obtain the cumulative attention score vector of the current decoding step; the time decay coefficient is between 0 and 1.

[0017] Furthermore, the latest key-value pairs are not included in the ranking of the top-ranked pairs, and the upper limit of the storage budget is a preset proportion of the initial key-value cache length.

[0018] Furthermore, the transcribed text embedding in S1 is obtained by mapping each transcribed fragment to a text embedding layer, and the initial audio embedding is obtained by encoding the original audio data by a pre-trained audio encoder.

[0019] Furthermore, the pooling aggregation operation compresses the sequence length of each semantic audio region feature slice to be consistent with the sequence length of the corresponding transcribed text embedding.

[0020] Secondly, this invention proposes a long context audio inference system based on semantic anchor key-value cache compression, which is used to implement the aforementioned long context audio inference method based on semantic anchor key-value cache compression.

[0021] The beneficial effects of this invention are as follows:

[0022] This invention introduces a semantic anchoring mechanism to compress redundant speech modal information into a high-density semantic representation aligned with the text, thereby reducing the caching pressure in the pre-filling stage from the source. At the same time, by combining time decay and dynamic attention evaluation mechanisms, it achieves adaptive eviction of long-term historical information in the decoding stage, which greatly reduces memory overhead and computational complexity. Furthermore, it effectively preserves key acoustic features and semantic logic in long-context audio inference tasks, significantly improving the model's inference speed and ability to process long sequences. Attached Figure Description

[0023] Figure 1 This is a flowchart of the long context audio inference method based on semantic anchoring key-value caching compression proposed in this invention.

[0024] Figure 2 This is a schematic diagram of the overall architecture proposed in this invention.

[0025] Figure 3 This is an architecture diagram of the long context audio inference system based on semantic anchoring key-value caching compression proposed in this invention. Detailed Implementation

[0026] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments. The embodiments described are merely examples of the present disclosure and do not limit the scope of the invention. The technical features of each embodiment of the present invention can be combined accordingly, provided that there is no conflict between them.

[0027] The accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0028] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all steps. For example, some steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances. To further clarify the technical details and implementation process of the present invention, the following is combined with... Figure 1 The flowchart shown and Figure 2 The overall architecture diagram shown below provides a detailed explanation of this method.

[0029] This embodiment uses the Qwen3-Omni-30B model as its basic architecture for inference processing on a 20-minute long audio input containing multiple inflection points. The specific operation steps are as follows:

[0030] Step 1: Multimodal input processing and feature extraction.

[0031] This step acquires long-context raw audio data, which includes the speech portion of the user command and the audio content to be processed. An automatic speech recognition model is used to extract timestamped transcribed text, where the transcribed fragments corresponding to the user command and the transcribed fragments corresponding to the audio content to be processed are uniformly included in the transcribed text sequence. An embedding is obtained through embedding mapping. An audio encoder is used to encode the raw audio data into a raw audio embedding, where the audio frames corresponding to the user command and the audio frames corresponding to the audio content to be processed are uniformly included in the initial audio embedding.

[0032] The user instructions refer to control statements issued by the user in natural language to guide the large model base to understand, reason, or generate audio content. Examples include requests for summarizing audio content, retrieval queries for specific information, question-and-answer based on audio content, and instructions for time-series analysis of audio events.

[0033] In one specific embodiment of the present invention, an automatic speech recognition (ASR) model is used to process raw audio data with long context. , generate containing The transcribed text sequence of each fragment and its corresponding timestamp boundary set ,in They represent the first The start and end times of each transcribed segment in the audio. Indicates the first One transcribed fragment.

[0034] Further, a text embedding layer is used to map transcribed fragments to transcribed text embeddings. This places text features and audio features in the same semantic space, where N represents the number of transcribed fragments and D represents the embedding dimension. Each transcribed fragment... The corresponding text vector embedding is , , For text feature dimensions, and The audio dimensions are the same; is the length of the text token corresponding to the i-th transcribed segment.

[0035] Additionally, a pre-trained audio encoder is used to encode the raw audio into continuous raw audio embeddings. ,in The total number of frames representing audio features. For feature dimension, It is an audio encoder.

[0036] Step 2: Semantic interval division based on timestamps.

[0037] Embed the original audio using the timestamps output by ASR. The transcription fragments were dynamically divided into two disjoint subsets: semantic audio regions containing pronunciation and background audio regions without pronunciation or semantic meaning. The semantic regions refer to each transcribed segment. Corresponding start and end timestamps The inner interval, or background interval, refers to the gap between two adjacent semantic intervals. Therefore, based on the mapping relationship between timestamps and original audio embedding features, for semantic audio intervals containing pronunciation, the temporal range is extracted. Corresponding semantic audio region feature slices For background audio intervals without sound or semantic meaning, extract the time range. Corresponding background audio region feature slices This enables the semantic structured decomposition of long audio data.

[0038] Step 3: Semantic anchoring embedding generation.

[0039] For each semantic audio region feature slice Perform average pooling to obtain converged audio features with lengths aligned to single-text embeddings. :

[0040]

[0041]

[0042] in, This is a pooling operation.

[0043] Aggregate audio features With corresponding text vector embedding Add to generate the first High information density semantic anchoring embedding of each transcription fragment :

[0044]

[0045] Repeat the above steps to iterate through N transcription fragments and generate a complete semantic anchoring embedding sequence. .

[0046] Step 4: Pre-population construction of compressed key-value cache.

[0047] Semantic anchoring embeddings and background audio region feature slices are concatenated according to the timestamp order of the original audio to form a compressed long contextual input sequence, for example... .like and If they overlap, then there is no background region between two adjacent transcription segments. Empty.

[0048] Each independent feature representation in the input sequence is defined as a word element. This sequence... During the pre-filling stage of the Transformer layer, a forward propagation is performed, and the model internally computes self-attention in parallel for all words to generate a complete initial key-value cache. ,in This is used to identify the initial key-value cache length generated during the pre-filling stage; These correspond to the key vector and value vector at the m-th cache position, respectively. Compared to full processing, the cache length in the pre-filling stage in this embodiment is reduced to approximately 25% of the original sequence, significantly reducing the latency of first character generation.

[0049] Step 5: Autoregressive decoding and dynamic cache eviction based on temporal decay.

[0050] During the decoding phase, the first step in the autoregressive decoding process of the large language model is... Step 1: The currently generated tokens produce a query vector. The key vector matrix is ​​represented as ,in Indicate the number of key-value pairs in the key-value cache at the t-th decoding step. Calculate the single-step attention score for the current decoding step. :

[0051]

[0052] Among them, attention score Each component in the equation corresponds to a single-step attention score for a key-value pair. It is the dimension of the key.

[0053] To measure the importance of different historical key-value pairs, a time decay coefficient is introduced. Utilize the attention score of the current step Update the cumulative attention score vector for all cached key-value pairs. :

[0054]

[0055] in, To control the rate of decay of historical attention weights, This is the cumulative attention score vector from the previous decoding step. This indicates that zeros are padded at the end of the vector to align with the key-value pairs added in the current decoding step.

[0056] Each layer of attention mechanism independently maintains a cumulative attention score vector. .

[0057] This embodiment sets a time decay coefficient. At each decoding step update, the old attention score is multiplied by The attention weight generated in the current step is then added to the current attention weight, resulting in lower cumulative scores for historical key-value pairs that are further back in time, naturally creating a preference for recent data. The system monitors the memory usage of the key-value cache in real time. When the number of key-value pairs in the cache reaches the preset storage budget limit, dynamic eviction is performed: extracting key-value pairs from the cache. The indexes of the top k key-value pairs in the median value are used as the final historical token indexes. In addition, the system also retains the latest M key-value pairs to ensure the continuity of the reasoning logic. The latest M key-value pairs do not participate in the Topk sort.

[0058] Based on the index set of the top k key-value pairs and the latest M key-value pairs. Filter the current key-value cache and generate an updated key-value cache matrix. .

[0059] Step Six: Output of Reasoning Results.

[0060] After processing through the above mechanism, the model maintains only a high-density key-value cache that is retained after dynamic eviction in each decoding step. Based on this cache, attention is calculated and the probability distribution of the next word is generated. The target text sequence is output through sampling or a greedy strategy, thereby completing the semantic reasoning task of continuous audio content and outputting the reasoning result text while maintaining the ability to understand long context.

[0061] It should be noted that user commands are recorded in speech form at the beginning of the original audio data or at a specific location. After ASR recognition, the semantic anchoring embedding generated by fusing the transcribed text embedding with the corresponding converged audio embedding is naturally included in the compressed input sequence. The large model base can obtain the audio context content and task intent simultaneously through this sequence.

[0062] This embodiment, when processing a diverse audio dataset of 80 hours, not only maintains high accuracy inference output, but also achieves stability and low latency in long text output with limited hardware memory configuration, demonstrating the robustness and efficiency of this method in handling complex long context scenarios.

[0063] Application examples

[0064] This application example aims to verify the performance of the method of this invention in long audio context reasoning and general audio understanding tasks. The experiment uses the Qwen3-Omni-Instruct-30B model as the backbone network. The effectiveness is evaluated by comparing the accuracy of the method of this invention (VoxZip) with existing mainstream KV Cache compression techniques (including StreamingLLM, SnapKV, PyramidKV, and ChunkKV) in different task scenarios. 1. Performance Evaluation of Long Audio Context Reasoning

[0065] This experiment selected Vox-Infinity, AudioMarathon, and SPIRAL as benchmark datasets for long audio context inference. With a 25% KV Cache budget, the accuracy of each method in understanding and inferring long audio sequences was tested. The evaluation results are shown in Table 1.

[0066] Table 1 Performance Comparison of Long Audio Contextual Reasoning

[0067]

[0068] Table 1 shows that, under a 25% cache constraint, existing comparative methods (StreamingLLM, SnapKV, PyramidKV, and ChunkKV) all show significant declines in various metrics, with average accuracy not exceeding 45%. In contrast, the VoxZip method proposed in this invention achieves 72.88% on the Vox-Infinity dataset and 57.82% on the AudioMarathon dataset, with an average accuracy of 69.75%, far surpassing other compression methods and even exceeding the baseline level of FullKV (67.07%). This indicates that the semantic anchoring mechanism of this invention successfully overcomes the semantic drift problem in long contexts by reconstructing information density.

[0069] General Audio Understanding Performance Evaluation: To evaluate the performance of this invention on fine-grained acoustic perception tasks, this application example was further tested on three general audio understanding benchmark datasets: MMAU, MMSU, and MMAR. The evaluation results are shown in Table 2.

[0070] Table 2 Comparison of General Audio Understanding Performance

[0071]

[0072] Experimental results show that, with a 25% KV cache budget, the method of this invention achieves an accuracy of 68.57% on the MMAR dataset, significantly higher than the 57.73% accuracy of the full cache benchmark. In the MMAU and MMSU tasks, this invention also maintains performance comparable to or even better than full cache. In contrast, existing methods exhibit significant information loss in general audio understanding tasks, especially on the MMAR task, where the accuracy of comparable methods is mostly between 40% and 55%.

[0073] Experimental conclusions: The above application examples fully demonstrate that the key-value caching compression method based on semantic anchoring and temporal decay provided by this invention can effectively compensate for the information density imbalance in speech modalities while significantly reducing memory usage. It also exhibits excellent robustness and accuracy in both long context reasoning and refined acoustic feature perception tasks, achieving the dual technical objectives of improving reasoning efficiency and understanding fidelity.

[0074] It should also be noted that the long context audio inference method based on semantic anchor key-value cache compression in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, another preferred embodiment of the present invention also provides a long context audio inference system based on semantic anchor key-value cache compression, corresponding to the long context audio inference method based on semantic anchor key-value cache compression provided in the above embodiments, such as... Figure 3 As shown, it includes:

[0075] The audio transcription and encoding module is used to acquire long-context raw audio data, which includes the speech part of the user command and the audio content to be processed; to perform text recognition on the raw audio data, generate time-stamped transcription fragments and map them into transcribed text embeddings, and to encode the raw audio data into an initial audio embedding.

[0076] The semantic anchoring embedding generation module is used to divide the initial audio embedding into semantic audio interval feature slices and background audio interval feature slices based on the timestamp boundary; perform pooling aggregation operation on the semantic audio interval feature slices to obtain converged audio embeddings; and fuse the converged audio embeddings with the corresponding transcribed text embeddings to generate semantic anchoring embeddings.

[0077] The compressed input construction and pre-filling module is used to concatenate semantic anchoring embeddings and background audio interval feature slices in chronological order to form a compressed input sequence. The compressed input sequence is then input into the large model base, and an initial key-value cache is constructed through the pre-filling stage.

[0078] The dynamic cache management and decoding module is used to update the cumulative attention score of each key-value pair using a preset time decay coefficient during the autoregressive decoding stage. When the key-value cache reaches the storage budget limit, it retains the key-value pairs with the highest cumulative scores and the latest key-value pairs, evicts the remaining key-value pairs, and continues decoding with the updated key-value cache until the inference result for the user instruction is output.

[0079] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0080] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0081] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A long-context audio inference method based on semantic anchoring key-value caching compression, characterized in that, Includes the following steps: S1. Obtain long-context raw audio data, the raw audio data including the speech part of the user command and the audio content to be processed; perform text recognition on the raw audio data, generate a timestamped transcription fragment and map it into a transcription text embedding, and encode the raw audio data into an initial audio embedding. S2. Based on the timestamp boundary, the initial audio embedding is divided into semantic audio interval feature slices and background audio interval feature slices; Pooling and aggregation operations are performed on the semantic audio interval feature slices to obtain the converged audio embeddings. The converged audio embeddings are then fused with the corresponding transcribed text embeddings to generate semantic anchoring embeddings. S3. Semantic anchoring embedding and background audio interval feature slices are concatenated in chronological order to form a compressed input sequence. The compressed input sequence is then input into the large model base and an initial key-value cache is constructed through a pre-filling stage. S4. In the autoregressive decoding stage, the cumulative attention score of each key-value pair is updated using a preset time decay coefficient. When the key-value cache reaches the storage budget limit, the key-value pairs with the highest cumulative scores and a number of the newest key-value pairs are retained, while the remaining key-value pairs are evicted. Decoding continues with the updated key-value cache until the inference result for the user's command is output.

2. The long context audio inference method based on semantic anchoring key-value caching compression according to claim 1, characterized in that, In S2, the timestamps carried by each transcribed segment include start and end times. The semantic audio interval feature slice is the embedded frame interval covered by the start and end times of each transcribed segment. The background audio interval feature slice is the embedded frame interval corresponding to the time gap between two adjacent semantic audio interval feature slices. When the timestamps of adjacent transcribed segments are connected at the boundary, it is determined that there is no background audio interval feature slice between them.

3. The long context audio inference method based on semantic anchoring key-value caching compression according to claim 1, characterized in that, In S2, the fusion specifically involves: adding the converged audio embeddings and transcribed text embeddings corresponding to the same transcribed fragment element by element; and traversing all transcribed fragments to generate a complete semantic anchoring embedding sequence.

4. The long context audio inference method based on semantic anchoring key-value caching compression according to claim 1, characterized in that, The length of the compressed input sequence is less than the length of the initial audio embedding; the pre-filling stage performs parallel self-attention calculations on all lexical units in the compressed input sequence to generate an initial key-value cache.

5. The long context audio inference method based on semantic anchoring key-value caching compression according to claim 1, characterized in that, In S4, at each decoding step, a single-step attention score is calculated based on the query vector of the currently generated lexical and all key vectors in the key-value cache; the cumulative attention score vector of the previous decoding step is multiplied by the time decay coefficient and then added to the single-step attention score to obtain the cumulative attention score vector of the current decoding step; the time decay coefficient is between 0 and 1.

6. The long context audio inference method based on semantic anchoring key-value caching compression according to claim 1, characterized in that, The latest key-value pairs are not included in the ranking of the top-ranked pairs, and the upper limit of the storage budget is a preset proportion of the initial key-value cache length.

7. The long context audio inference method based on semantic anchoring key-value caching compression according to claim 1, characterized in that, The pooling aggregation operation adjusts the sequence length of each semantic audio region feature slice to match the sequence length of the corresponding transcribed text embedding.

8. A long-context audio inference system based on semantic anchor key-value caching compression, used to implement the long-context audio inference method based on semantic anchor key-value caching compression as described in claim 1, characterized in that, The system includes: The audio transcription and encoding module is used to acquire long-context raw audio data, which includes the speech part of the user command and the audio content to be processed; to perform text recognition on the raw audio data, generate timestamped transcription fragments and map them into transcribed text embeddings, and to encode the raw audio data into an initial audio embedding. The semantic anchoring embedding generation module is used to divide the initial audio embedding into semantic audio interval feature slices and background audio interval feature slices based on the timestamp boundary; perform pooling aggregation operation on the semantic audio interval feature slices to obtain converged audio embeddings; and fuse the converged audio embeddings with the corresponding transcribed text embeddings to generate semantic anchoring embeddings. The compressed input construction and pre-filling module is used to concatenate semantic anchoring embeddings and background audio interval feature slices in chronological order to form a compressed input sequence. The compressed input sequence is then input into the large model base, and an initial key-value cache is constructed through the pre-filling stage. The dynamic cache management and decoding module is used to update the cumulative attention score of each key-value pair using a preset time decay coefficient during the autoregressive decoding stage. When the key-value cache reaches the storage budget limit, it retains the key-value pairs with the highest cumulative scores and the latest key-value pairs, evicts the remaining key-value pairs, and continues decoding with the updated key-value cache until the inference result for the user instruction is output.

9. The long-context audio inference system based on semantic anchoring key-value caching compression according to claim 8, characterized in that, In the semantic anchoring embedding generation module, the converged audio embedding is fused with the corresponding transcribed text embedding. Specifically, the converged audio embedding corresponding to the same transcribed fragment is added element-wise to the transcribed text embedding to generate the semantic anchoring embedding.

Images