Search enhancement generation method and apparatus, electronic device, and storage medium

Abstract

The application relates to the technical field of data processing, in particular to a retrieval enhancement generation method and device, electronic equipment and a storage medium, wherein user multi-modal data of a current streaming unit is acquired, a first user sequence is constructed by encoding, first stage reasoning is performed, a retrieval identifier is output and a user side key value cache is cached, when retrieval is required, multi-path parallel retrieval is performed, knowledge tokens are added to the tail of the first user sequence, second stage reasoning is performed by reusing the user side key value cache, reply content and knowledge abstracts are generated, the full text key value cache is released, the abstract key value cache is written into a global history pool, the user side key value cache and the abstract key value cache are sent to a Performer flow matching module, audio and video hidden variables are generated and decoded and output. The application realizes external knowledge retrieval and fusion synchronized with the current streaming unit without destroying the original block causal time sequence and double pipeline mechanism.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a search enhancement generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] In scenarios such as real-time digital humans and embodied interaction, end-to-end full-duplex streaming multimodal bases like Wan-Streamer typically use a streaming unit (Block) of about 160ms as the minimum inference cycle. They employ intra-block bidirectional and inter-block causal attention masks, combined with Thinker-Performer dual pipelines, to control the model-side response within about 200ms, in order to support full-duplex behaviors such as listening, interrupting, question answering, and idle.

[0003] These types of bases generally do not provide retrieval augmentation generation (RAG) capabilities for interfacing with external knowledge bases. Directly incorporating external textual knowledge into the context can easily disrupt the block causal sequence, leading to inconsistencies between lip movements, facial expressions, and the content being explained, and causing the global key-value (KV) to continuously increase in long conversations. The base itself also cannot reliably determine when to perform a retrieval. Therefore, existing cascaded digital human RAG solutions, which send retrieval results into independent text branches without altering the original block causality and dual pipelines as much as possible, are difficult to jointly model with the user's audio and video in the same attention computation within the current streaming unit. Asynchronous thinking layer solutions such as DuplexOmni decouple interaction from thinking, often resulting in knowledge results lagging behind the current interaction block by approximately 160ms. Block-Attention solutions are geared towards paragraph-level KV reuse in pure text-based large language models, do not cover multimodal pipelines, and do not distinguish between temporary knowledge within blocks and persistent summaries between blocks. Other solutions use external interactive state machines to control retrieval, relying on external interactive tags to switch retrieval strategies, which interrupts the listening and interruption behaviors already learned by the base. Furthermore, if the full text of the search is written to the global key-value cache for a long time, the memory usage will continue to increase as the conversation lengthens, and the existing solutions lack a search mechanism designed for the approximately 160ms streaming unit constraint.

[0004] Therefore, how to achieve external knowledge retrieval and fusion synchronized with the current streaming unit in streaming inference of large audio and video models without destroying the original block causal timing and dual pipeline mechanism is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a retrieval enhancement generation method, apparatus, electronic device, and storage medium that enables external knowledge retrieval and fusion synchronized with the current streaming unit in streaming reasoning of large audio and video models without disrupting the original block causal timing and dual pipeline mechanism.

[0006] The first aspect of this application provides a retrieval enhancement generation method applied to a large audio-visual model with a streaming unit as the inference cycle, wherein the streaming unit is the smallest time unit for streaming inference of the large audio-visual model, and the method includes: Obtain the user multimodal data of the current streaming unit; The user multimodal data is encoded into latent variables to construct the first user sequence of the current streaming unit; Perform the first stage of inference on the first user sequence, output the retrieval identifier and cache the user-side key value cache; When the retrieval identifier indicates that a retrieval is required, a multi-path parallel retrieval is performed based on the latent variable to obtain knowledge fragments and encode them as knowledge tokens. The knowledge token is appended to the end of the first user sequence, the user-side key value cache is reused, and the second-stage reasoning is performed on the appended second user sequence to generate response content and knowledge summary; the second user sequence is the sequence formed by appending the knowledge token to the end of the first user sequence. Release the full-text key-value cache of the current streaming unit after use, and write the summary key-value cache corresponding to the knowledge summary into the global history pool for subsequent streaming units to read; The user-side key-value cache and the summary key-value cache are sent as conditional features to the Performer stream matching module of the audio and video big model to generate audio and video latent variables and decode and output them.

[0007] In an optional implementation, the multi-path parallel retrieval includes text semantic retrieval, hot word inverted index retrieval, audio intent retrieval, image entity retrieval, and image-text-audio joint retrieval. The latent variables include text latent variables, audio latent variables, and visual latent variables. The multi-path parallel retrieval based on the latent variables includes: The text latent variables, audio latent variables, and visual latent variables are projected onto the same retrieval dimension using their respective MLPs, and then scaled using LayerNorm to obtain text query features, audio query features, and visual query features respectively. The text query features, audio query features, and visual query features are fused using a fusion MLP to obtain joint query features; Based on the textual latent variables, hot word hypotheses are obtained through CTC decoding; The text semantic retrieval, the hot word inverted index retrieval, the audio intent retrieval, the video entity retrieval, and the image-text-audio joint retrieval are executed in parallel to obtain the hit score of each retrieval path; wherein, the text semantic retrieval is performed based on the text query features, the hot word inverted index retrieval is performed based on the hot word hypothesis, the audio intent retrieval is performed based on the audio query features, the video entity retrieval is performed based on the visual query features, and the image-text-audio joint retrieval is performed based on the joint query features; The comprehensive score is determined based on the hit scores of each retrieval path and the cross-path consistency reward item; the cross-path consistency reward item is used to characterize the consistency of the same candidate knowledge fragment being hit simultaneously in multiple retrieval paths. The top-K knowledge segments are selected in descending order of the overall scores. The Top-K knowledge fragments are encoded into knowledge tokens of the same dimension as the audio-visual large model, and a knowledge start token and a knowledge end token are added to the beginning and end of the knowledge tokens, respectively.

[0008] In an optional implementation, determining the overall score based on the hit scores of each retrieval path and the cross-path consistency reward includes: For each retrieval path, the hit scores of each candidate knowledge fragment output by the target path retrieval path are normalized by the temperature parameter softmax to obtain the path weight of each candidate knowledge fragment corresponding to the target path retrieval path; the target path retrieval path is any retrieval path; the path weight is used to represent the relative priority of each candidate knowledge fragment from the perspective of that retrieval path. For the same candidate knowledge segment, the hit scores of each retrieval path are multiplied by their corresponding path weights and summed, and then the cross-path consistency reward is added to obtain the comprehensive score of the same candidate knowledge segment.

[0009] In an optional implementation, the step of performing a first-stage inference on the first user sequence, outputting a retrieval identifier, and caching a user-side key value cache includes: Input the first user sequence into the Transformer of the audio and video big model, perform autoregressive inference, and generate at least one token; When the generated token contains a retrieval trigger token, the retrieval trigger token is output as the retrieval identifier; The key-value cache corresponding to the first user sequence generated during the first stage of inference is cached as the user-side key-value cache; The retrieval identifier is determined by the RAG-specific token output by the autoregressive output of the audio and video large model, and does not depend on the input of an external interactive state machine.

[0010] In an optional implementation, the second-stage reasoning on the appended second user sequence to generate response content and knowledge summary includes: Input the second user sequence into the Transformer of the audio and video large model; The user-side key-value cache is reused to calculate self-attention for the knowledge token, and cross-attention within the block is calculated for the knowledge token and the user-side key-value cache. After completing the attention calculation on the knowledge token, autoregressive decoding is performed to generate a response token sequence; Decode the tokens in the response token sequence, excluding the digest token, into text, and output this as the response content. In the response token sequence, when the audio and video big model outputs a summary token, the text content corresponding to the summary token is output as the knowledge summary; The knowledge digest has a length of 3 to 8 tokens.

[0011] In an optional implementation, after writing the summary key-value cache corresponding to the knowledge summary to the global history pool, the method further includes: After the first stage of inference of the subsequent streaming unit is completed, the semantic similarity between the first user sequence of the current streaming unit and each digest key-value cache in the global history pool is calculated; When the semantic similarity exceeds a preset threshold, the corresponding summary key value is cached and read, and participates in the second stage of reasoning of the subsequent streaming unit; When the number of summary key-value caches in the global history pool reaches a preset threshold, multiple summary key-value caches with semantic similarity exceeding a preset threshold are merged into a single merged summary key-value cache, and summary key-value caches with a reference count lower than a preset threshold and an existence time exceeding a preset time threshold are removed from the global history pool.

[0012] In an optional implementation, the RAG-specific tokens used by the retrieval identifier and the knowledge token are implemented by adding a retrieval trigger token, a knowledge start token, a knowledge end token, and a summary token to the text vocabulary. The audio-visual large model is pre-fine-tuned in the following way: In the text vocabulary of the audio and video big model, a retrieval trigger token, a knowledge start token, a knowledge end token, and a summary token are added as RAG-specific tokens. The embedding of the new tokens is initialized with the mean of the general real word embedding. Freeze the audio and video encoder, audio and video variational autoencoder and the first 80% of the Transformer layers of the large audio and video model, and train only the top 2 to 3 layers of the Transformer and the embedding of the RAG special token. A training sample is constructed, with each sample aligned to a single streaming unit. The training sample includes a first-stage input sequence and a first-stage target token, where the first-stage target token is either the retrieval trigger token or an empty token. For samples to be retrieved, a knowledge token sequence is appended to the first-stage input sequence. The knowledge token sequence includes a knowledge start token, a knowledge token, and a knowledge end token. The second-stage target includes a response token and the summary token. The training samples include idle scene samples, listening scene samples, question-and-answer scene samples, and interruption scene samples. During the training process, a course sampling strategy is adopted. The first 10% of the training steps are mainly composed of question-and-answer scene samples. In the middle 80% of the training steps, the four types of samples are relatively balanced: idle scene samples, listening scene samples, question-and-answer scene samples, and interruption scene samples. In the last 10% of the training steps, the proportion of idle scene samples is increased. During the training phase, behavioral labels and auxiliary classification heads are used to assist training. During deployment, the auxiliary classification head is removed, the behavioral labels are not input, and the decision to perform a retrieval is made solely based on whether the autoregression of the audio and video large model outputs the retrieval trigger token.

[0013] A second aspect of this application provides a retrieval enhancement generation apparatus applied to a large audio-visual model with a streaming unit as the inference cycle, wherein the streaming unit is the smallest time unit for streaming inference of the large audio-visual model, and the apparatus includes: The acquisition module is used to acquire the user multimodal data of the current streaming unit; The encoding module is used to encode the user multimodal data into latent variables and construct the first user sequence of the current streaming unit; The first-stage reasoning module is used to perform first-stage reasoning on the first user sequence, output the retrieval identifier and cache the user-side key value cache. The retrieval module is used to perform multi-path parallel retrieval based on the latent variables when the retrieval identifier indicates that a retrieval is required, to obtain knowledge fragments and encode them into knowledge tokens; The second-stage reasoning module is used to append the knowledge token to the end of the first user sequence, reuse the user-side key-value cache, perform second-stage reasoning on the appended second user sequence, and generate response content and knowledge summary; the second user sequence is the sequence formed by appending the knowledge token to the end of the first user sequence. The cache management module is used to release the full-text key-value cache of the current streaming unit after use, and write the summary key-value cache corresponding to the knowledge summary into the global history pool for subsequent streaming units to read. The output module is used to send the user-side key-value cache and the summary key-value cache as conditional features to the Performer stream matching module of the audio and video big model, generate audio and video latent variables, and decode and output them.

[0014] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the retrieval enhancement generation method.

[0015] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described retrieval enhancement generation method.

[0016] In summary, the search enhancement generation method, apparatus, electronic device, and storage medium provided in this application have at least one of the following beneficial effects: 1. The reasoning process is divided into a first-stage reasoning and a second-stage reasoning. In the first stage, reasoning is performed only on the user sequence to output a retrieval identifier. This retrieval identifier is determined by the model's autoregressive output and does not rely on an external interaction state machine. This preserves the learning methods of listening and interrupting behaviors in the original block causal sequence of the base, and avoids the disruption of the sequence behavior caused by the hard cutting of the external state machine. At the same time, the first stage caches the user-side key-value cache for subsequent reuse. 2. When a retrieval identifier indicates that a retrieval is needed, multi-path parallel retrieval is performed based on the latent variables of the current streaming unit. The retrieved knowledge is encoded as a knowledge token and appended to the end of the first user sequence. The second-stage inference is then performed on the appended second user sequence. The entire retrieval and fusion process is completed within the time window of the current streaming unit, ensuring that external knowledge can be jointly modeled with the user multimodal data of the current streaming unit in the same attention computation, avoiding knowledge results lagging behind the current interaction block. 3. The second-stage inference reuses the user-side key-value cache of the first stage cache and only incrementally calculates the attention related to the knowledge token. This allows the inference latency to be controlled within the streaming unit cycle of about 160ms after the introduction of retrieval and knowledge fusion operations in the current streaming unit, so as not to disrupt the original response rhythm of the base.

[0017] 4. After the full-text key-value cache of the current streaming unit is used up, it is released. Only the summary key-value cache corresponding to the knowledge summary is written to the global history pool for subsequent streaming units to read. Furthermore, the user-side key-value cache and the summary key-value cache are sent to the Performer streaming matching module as conditional features. By distinguishing between "temporary full knowledge within a block" and "persistent summary between blocks", the memory growth of knowledge-type key-value cache under long dialogue is effectively controlled while ensuring that subsequent streaming units can reference historical knowledge. It is evident that this application achieves external knowledge retrieval and fusion synchronized with the current streaming unit without disrupting the original block causal timing and dual pipeline mechanism. Attached Figure Description

[0018] Figure 1 This is a schematic flowchart illustrating the inference process of a Wan-Streamer base according to an embodiment of this application; Figure 2 This is a schematic flowchart illustrating the reasoning process of a retrieval enhancement generation method according to an embodiment of this application; Figure 3 This is a schematic flowchart illustrating a search enhancement generation method according to an embodiment of this application; Figure 4 This is an interactive schematic diagram illustrating a retrieval enhancement generation method according to an embodiment of this application; Figure 5 This is a schematic diagram illustrating a five-way parallel retrieval process according to an embodiment of this application; Figure 6 This is a schematic diagram illustrating a hierarchical key-value (KV) lifecycle management method according to an embodiment of this application; Figure 7 This is a functional block diagram of a search enhancement generation device shown in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0020] The following will clearly and completely describe the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are all within the scope of protection of the present invention. Furthermore, all connections / linkages involved in the patent do not simply refer to direct contact between components, but rather to the ability to form a better connection structure by adding or reducing connecting accessories according to specific implementation conditions. The various technical features in this invention can be combined interactively without contradicting each other.

[0021] The retrieval enhancement generation method provided in this application is applied to large audio-visual models with a streaming unit (Block) as the inference cycle. The streaming unit is the smallest time unit for streaming inference in the large audio-visual model. As an example, the large audio-visual model can adopt a Wan-Streamer architecture, which uses a streaming unit of approximately 160ms as the minimum inference cycle, employs intra-block bidirectional and inter-block causal attention masks, and combines a Thinker-Performer dual pipeline to control the model-side response within approximately 200ms, supporting full-duplex behaviors such as listening, interrupting, question answering, and idle. It should be noted that the method in this application is not limited to the Wan-Streamer architecture but can be applied to any large audio-visual model with a streaming unit as the inference cycle.

[0022] Reference Figure 1 As shown, the inference process of Wan-Streamer is roughly divided into two pipeline stages: (1) Thinker completes user multimodal encoding, Transformer inference, context KV maintenance, and sends semantic condition features (i.e., updated KV) to Performer. The native part mainly performs interleaved multimodal autoregressive generation and does not include knowledge base retrieval, knowledge injection, and summary persistence; (2) Performer performs stream matching according to the condition features given by Thinker, generates synchronous speech and video latent variables, and uses them to decode and generate speech and video. The native conditions do not contain external knowledge base content. Listening, interrupting, questioning and answering, and idle behavior are learned by interleaved token temporal learning. No explicit interaction status labels are input during inference.

[0023] This application does not modify the backbone network structure, streaming segmentation method, or block causal masking rules of the base, nor does it modify... Figure 1 The multimodal causal Encoder / Decoder, model causal inference module, and Performer stream matching module are shown; however, without modifying the original structure mentioned above, retrieval enhancement generation is achieved through the following method: (1) Replace the original forward reasoning in Thinker with two-stage reasoning plus five-way parallel retrieval; (2) A new full-duplex adaptation scheduling module is added to replace the original KV management unit in Thinker and manage the KV of two-stage inference; (3) A model fine-tuning scheme was designed to enable the model to generate RAG retrieval tokens, and to receive text / audio / video inputs and output text / audio / video and RAG tokens.

[0024] Reference Figure 2The diagram shown is a flowchart illustrating the reasoning process of a retrieval enhancement generation method according to an embodiment of this application. The system architecture of this embodiment includes: (1) a user input module, used to collect the original text, audio, and video data of the current block; (2) a Thinker module, used for encoding, two-stage RAG reasoning, built-in retrieval, hierarchical KV, and decoding preparation; (3) a full-duplex adaptation scheduling module, used to coordinate retrieval start and stop, listen to the pre-retrieval cache, KV recycling, and summary maintenance; (4) a Performer module, used to receive the updated KV after two-stage reasoning by Thinker as conditional features, generate audiovisual latent variables, and send them back; (5) a digital rendering output module, used to decode the latent variables into external audio and video.

[0025] In this embodiment, the large audio and video model that executes the retrieval enhancement generation method needs to be fine-tuned and trained in advance to enable it to autonomously decide whether to conduct a retrieval and generate a knowledge summary. Figure 2 The model shown in the Thinker module is the finely tuned model. The specific fine-tuning method is as follows.

[0026] (1) Vocabulary expansion and embedding initialization.

[0027] First, add a retrieval trigger token (e.g., to the text vocabulary of the large audio-visual model) to the vocabulary.<RAG_QUERY> ), knowledge start token (e.g.)<RAG_START> ), knowledge end token (e.g.)<RAG_END> ) and digest tokens (e.g.<RAG_SUMMARY> This is a RAG-specific token. The embedding of the new token is initialized with the average of the general content word embeddings, the original word embeddings are frozen, and only the newly added token participates in the update.

[0028] (2) Parameter freezing strategy.

[0029] To inject RAG capabilities while preserving the original capabilities of the base, a layered freezing fine-tuning strategy is adopted: the audio / video encoder, audio / video variational autoencoder, and the first 80% of the Transformer layers of the large audio / video model are frozen, and only the top 2 to 3 layers of the Transformer and the embedding of the RAG-specific token are trained. Specifically, the audio / video causal encoder and Causal Audio / Visual VAE remain frozen to maintain the original compressed representation; the first 80% of the Transformer layers are frozen to fix the block causality and intra-block bidirectional low-level behavior; the top 2 to 3 layers of the Transformer (including FFN) are trainable to learn the cross-correlation between user latent variables and knowledge tokens; the shared text embedding is locally trainable and can only be updated with the addition of RAG tokens; the text prediction head and optional RAG classification auxiliary head are trainable for the generation of tool tokens, responses, summaries, and other objectives; the Performer stream matching network is frozen without changing the generation backbone.

[0030] For ease of understanding, the specific freeze and trainable range are shown in Table 1 below.

[0031] Table 1:

[0032] (3) Construction of training samples.

[0033] Training samples are constructed, with each sample aligned to a single streaming unit. The training samples include a first-stage input sequence and a first-stage target token, which is either the retrieval trigger token or an empty token. For samples to be retrieved, a knowledge token sequence is appended to the first-stage input sequence. This knowledge token sequence includes a knowledge start token, a knowledge token, and a knowledge end token. The second-stage target token includes a response token and the summary token.

[0034] Training samples include idle scenario samples, listening scenario samples, question-and-answer scenario samples, and interruption scenario samples. Idle scenarios are not retrieved in approximately 95% of cases; Listening scenarios are marked for retrieval in approximately 70% of cases to practice pre-retrieval; QA scenarios are not retrieved if the global summary is sufficient, otherwise they are retrieved; Interruption scenarios are marked for retrieval in approximately 80% of cases to update intent.

[0035] For example, each sample is aligned to a single 160 ms block. Idle / Listening / QA / Interruption are only used for training annotations, not for inference.

[0036] Example of a single-sample structure: The first-stage input is the user-side sequence. The first phase objective is output.<RAG_QUERY> Or leave blank (do not search); append when searching is required.<RAG_START> +Knowledge+<RAG_END> The second phase objective includes responding with a token and...<RAG_SUMMARY> And a short summary.

[0037] Example of labeling ratios: Idle approximately 95% - no retrieval; Listening approximately 70% - tagged for retrieval to practice pre-retrieval; QA - no retrieval if the global summary is sufficient, otherwise retrieval; Interruption approximately 80% - tagged for retrieval to update intent.

[0038] (4) Course sampling strategy and training phase division.

[0039] During training, a course sampling strategy was adopted: the first 10% of training steps mainly used question-and-answer scenario samples to familiarize the model with the use of RAG tokens and establish a complete link of "questioning → retrieval identifier → knowledge summary"; in the middle 80% of training steps, idle scenario samples, listening scenario samples, question-and-answer scenario samples, and interruption scenario samples were relatively balanced and jointly trained according to the course sampling method. In each iteration, the forward propagation and inference processes were kept consistent and executed according to a two-stage inference process (first, the first stage of inference was executed to output the retrieval identifier, and then the retrieval identifier was used to determine whether to execute the retrieval and the second stage of inference). Backpropagation only updated the trainable parameters specified in Table 1; in the last 10% of training steps, the proportion of idle scenario samples was increased and the learning rate was reduced to reduce the false trigger rate in the idle state. At the same time, the model performance was monitored according to indicators such as trigger accuracy, invalid retrieval rate, and summary quality. When the indicators reached the preset threshold or no longer improved after several consecutive training steps, the training was terminated early (early stop).

[0040] (5) Joint loss function.

[0041] During training, a joint loss function is used for optimization: ; in, For text cross-entropy loss; The stream matching loss is used to suppress audiovisual drift after knowledge injection; Constraint knowledge boundary tokens; Constraints on whether to output a retrieval identifier (which may include an autoregressive token and a lightweight binary classification header); The summary cross-entropy loss can be applied, and a length constraint can be added to ensure the summary falls within approximately 3–8 tokens. Example weights: α=1.0, β=0.2, γ=0.8, δ=1.2, ε=0.9.

[0042] (6) Training and reasoning are separated.

[0043] During training, behavior labels and an auxiliary classification head are used to assist in training. During deployment, the auxiliary classification head is removed, and the behavior labels are not input. The decision to perform a retrieval is based solely on whether the autoregression of the large audio / video model outputs the retrieval trigger token. That is, training can use behavior labels, auxiliary heads, and multiple losses; deployment removes the auxiliary head, does not input behavior labels, and relies solely on whether autoregression occurs.<RAG_QUERY> The retrieval is determined by (or by a conventional equivalent field).

[0044] Through the aforementioned fine-tuning, the model learns how to autonomously determine retrieval timing during streaming inference, how to encode retrieved knowledge into knowledge token sequences for attention computation, and how to generate short summaries. During deployment, only the backbone network and parameters required for model inference are retained, while the auxiliary classification head used during training is removed. No behavior labels are input, and the model relies solely on whether its autoregression outputs a retrieval trigger token to determine whether to perform a retrieval. This ensures that the inference phase does not depend on an external state machine and aligns the inference behavior with the behavior during training, thereby avoiding temporal deviations caused by hard-slicing of the state machine.

[0045] Reference Figure 3 The diagram shown is a flowchart illustrating a retrieval enhancement generation method according to an embodiment of this application. The retrieval enhancement generation method is applied to a large audio-visual model with a streaming unit as the inference cycle. The streaming unit is the smallest time unit for streaming inference of the large audio-visual model. The retrieval enhancement generation method includes the following steps.

[0046] S31, Obtain the user multimodal data of the current streaming unit.

[0047] In some embodiments, the user input module collects user multimodal data for the current streaming unit (Block). The user multimodal data includes text data, audio data, and video data. The streaming unit is the smallest time unit for streaming inference in the audio / video large model; as an example, the duration of the streaming unit is approximately 160ms. It should be noted that 160ms is only an exemplary implementation; different audio / video large models with different architectures can use different streaming unit durations, and this application does not limit this.

[0048] S32, the user multimodal data is encoded into latent variables to construct the first user sequence of the current streaming unit.

[0049] In some embodiments, the encoding unit in the Thinker module encodes the collected user multimodal data into latent variables, including text latent variables, audio latent variables, and visual latent variables. The text latent variables are converted into text tokens by a text segmenter; the audio latent variables are expanded into an audio token sequence according to time steps; and the visual latent variables are expanded into a visual token sequence according to spatial and temporal dimensions. Then, the tokens of the three modalities are interleaved according to a causal timeline. Within the time window of the current streaming unit, audio and visual tokens are arranged in their corresponding acquisition time order, and text tokens are inserted into their corresponding time positions according to the input order, forming an interleaved sequence of visual, audio, and text input tokens. Start and end tokens are added to the beginning and end of the sequence, respectively, to construct the first user sequence S1 of the current streaming unit. This first user sequence is used for subsequent first-stage inference.

[0050] In some embodiments, the features used for retrieval are latent variables output by the encoding unit, rather than the original waveform or pixel data. Latent variables are high-dimensional abstract representations compressed by the audio / video encoder, with low dimensionality and aligned to a unified semantic space. Direct retrieval on the original waveform or pixels results in high-dimensionality modalities, making it difficult to unify them into a comparable vector space. Furthermore, the computational cost of retrieving the original data is far greater than that of latent space retrieval, easily disrupting the approximately 160ms inference rhythm of the streaming unit and making latency difficult to control. This application achieves retrieval by performing the search in the latent variable space, thus ensuring retrieval accuracy while also meeting the real-time requirements of streaming inference.

[0051] S33, perform the first stage of reasoning on the first user sequence, output the retrieval identifier and cache the user-side key value cache.

[0052] In some embodiments, the fine-tuned model already has the ability to receive text / audio / video and output text / audio / video and RAG tokens. (See also...) Figure 4 As shown, in the first stage, the Thinker module only performs inference on the first user sequence S1, outputs the RAG retrieval identifier, and caches the user-side key value cache. .

[0053] In an optional implementation, the step of performing a first-stage inference on the first user sequence, outputting a retrieval identifier, and caching a user-side key value cache includes: Input the first user sequence into the Transformer of the audio and video big model, perform autoregressive inference, and generate at least one token; When the generated token contains a retrieval trigger token, the retrieval trigger token is output as the retrieval identifier; The key-value cache corresponding to the first user sequence generated during the first stage of inference is cached as the user-side key-value cache; The retrieval identifier is determined by the RAG-specific token output by the autoregressive output of the audio and video large model, and does not depend on the input of an external interactive state machine.

[0054] In some embodiments, refer to Figure 4 As shown, firstly, the first user sequence is input into the Transformer of the audio-visual large model, and autoregressive inference is performed to generate at least one token. The autoregressive inference refers to the process by which the model predicts the next token based on the generated tokens one by one.

[0055] Secondly, when the generated token contains a retrieval trigger token (e.g.)<RAG_QUERY> When the generated token does not contain a retrieval trigger token, it indicates that the current streaming unit does not need to perform a retrieval and can directly enter the response generation process. The retrieval identifier is determined by the RAG-specific token output by the autoregressive output of the audio / video large model and does not depend on the input of an external interactive state machine. That is to say, no explicit interactive state labels (such as "idle", "question and answer", "interrupt", etc.) need to be input during the inference process; the model relies entirely on its own output token to determine whether to perform a retrieval. This preserves the original sequential behaviors of the base, such as listening, interrupting, question and answer, and idle, and avoids the destruction of the original behaviors by the hard cutting of the external state machine.

[0056] Finally, the key-value cache corresponding to the first user sequence generated during the first stage of inference is cached as the user-side key-value cache. This key-value cache records the key and value vectors of each token in the first user sequence, for reuse in subsequent second-stage inference.

[0057] Figure 4 "in The word "caching" indicates this caching operation. This is achieved through caching in the first stage. In the second phase, there is no need to recalculate the key value of the user-side token, which greatly reduces the amount of computation.

[0058] It should be noted that the retrieval identifier described in this embodiment is determined by the RAG-specific token output by the model's autoregression, and does not rely on external interactive state machine input. If inference relies on an external interactive state machine (such as switching retrieval strategies through manual rules or external labels), it is necessary to identify the user's intent in advance and assign a state label. This not only conflicts with the end-to-end, no-explicit-state-label modeling approach of the base, but also interrupts the native temporal behaviors such as listening and interruption already learned by the base. This application determines whether to retrieve data by outputting the retrieval identifier through the model's autoregression, avoiding behavioral deviations caused by hard switching of external state machines, and making the pre-retrieval during the listening phase more natural—the model can autonomously decide whether to pre-retrieve during the listening process without external instructions.

[0059] S34, when the retrieval identifier indicates that a retrieval is required, perform multi-path parallel retrieval based on the latent variable to obtain knowledge fragments and encode them as knowledge tokens.

[0060] This full-duplex adaptation scheduling module contains only scheduling logic, no model weights, and is only enabled during the inference phase. For example... Figure 4 As shown in this embodiment, after the first stage of inference is completed, the full-duplex adaptation scheduling module reads the retrieval identifier of this block. When the retrieval identifier indicates that retrieval is required, it coordinates the model inference step on the GPU and the retrieval operation on the CPU / retrieval engine, so that retrieval and model calculation are executed in parallel, reducing serial waiting.

[0061] In some embodiments, when the retrieval identifier output in step S3 indicates that a retrieval is required (i.e., the model outputs a retrieval trigger token), multi-path parallel retrieval is performed. See also... Figure 4 As shown, the five-way retrieval is encapsulated in a two-stage inference kernel. It reuses the text, audio, and visual latent variables obtained by the encoding unit to construct queries, performs parallel recall, fusion and sorting, and then encodes them into knowledge tokens of the same dimension as the base. A full-duplex adaptation scheduling module coordinates the model inference steps on the GPU and the retrieval operations on the CPU / retrieval engine, reducing serial waiting time. That is, when the retrieval identifier...<RAG_QUERY> If the value is not empty, a five-way parallel search is performed, and the knowledge token is appended to the end of the sequence, while the user prefix remains unchanged. Reuse.

[0062] In an optional implementation, the multi-path parallel retrieval includes text semantic retrieval, hot word inverted index retrieval, audio intent retrieval, image entity retrieval, and image-text-audio joint retrieval. The latent variables include text latent variables, audio latent variables, and visual latent variables. The multi-path parallel retrieval based on the latent variables includes: The text latent variables, audio latent variables, and visual latent variables are projected onto the same retrieval dimension using their respective MLPs, and then scaled using LayerNorm to obtain text query features, audio query features, and visual query features respectively. The text query features, audio query features, and visual query features are fused using a fusion MLP to obtain joint query features; Based on the textual latent variables, hot word hypotheses are obtained through CTC decoding; The text semantic retrieval, the hot word inverted index retrieval, the audio intent retrieval, the video entity retrieval, and the image-text-audio joint retrieval are executed in parallel to obtain the hit score of each retrieval path; wherein, the text semantic retrieval is performed based on the text query features, the hot word inverted index retrieval is performed based on the hot word hypothesis, the audio intent retrieval is performed based on the audio query features, the video entity retrieval is performed based on the visual query features, and the image-text-audio joint retrieval is performed based on the joint query features; The comprehensive score is determined based on the hit scores of each retrieval path and the cross-path consistency reward item; the cross-path consistency reward item is used to characterize the consistency of the same candidate knowledge fragment being hit simultaneously in multiple retrieval paths. The top-K knowledge segments are selected in descending order of the overall scores. The Top-K knowledge fragments are encoded into knowledge tokens of the same dimension as the audio-visual large model, and a knowledge start token and a knowledge end token are added to the beginning and end of the knowledge tokens, respectively.

[0063] In this embodiment, the multi-path parallel retrieval adopts a five-path parallel retrieval architecture, including text semantic retrieval (R1), hot word inverted index retrieval (R2), audio intent retrieval (R3), image entity retrieval (R4), and image-text-audio joint retrieval (R5). If only text semantic retrieval and hot word inverted index retrieval are retained, the visual entity information in the image cannot be retrieved and utilized, and the user's colloquial intent expression is difficult to accurately identify. In multimodal interaction scenarios (such as when a user points to an object in the image and asks a question, or describes the image content through voice), both recall and precision will significantly decrease. This application fully utilizes audio, visual, and cross-modal joint features by executing and merging five-path retrieval in parallel, covering proper nouns and multimodal questions, effectively reducing the missed recall rate of single-path retrieval. The latent variables include text latent variables, audio latent variables, and visual latent variables. Figure 5 The processing flow of five-way parallel retrieval is shown.

[0064] Specifically, the multi-path parallel retrieval based on the latent variables includes the following sub-steps.

[0065] S41, Feature projection and scale unification.

[0066] The textual, audio, and visual latent variables output by the encoding unit in step S2 are projected onto the same retrieval dimension using their respective Multilayer Perceptron (MLP), and then scaled using LayerNorm to obtain textual, audio, and visual query features, respectively. Each modality is mapped to the retrieval space via an independent MLP, and scaled using LayerNorm, allowing features from different modalities to be compared in the same space.

[0067] S42, Joint query feature generation.

[0068] The text query features, audio query features, and visual query features obtained in sub-step S41 are fused using a fusion MLP to obtain joint query features. Based on the scale-aligned three-way features, a trained MLP is used for feature fusion for subsequent retrieval.

[0069] S43, Hot word hypothesis generation.

[0070] Hot word hypotheses are obtained through CTC decoding based on the latent variables in the text. Text branches are then processed by CTC to obtain hot word hypotheses (such as proper nouns like model number and SKU), which are used for subsequent hot word inverted index retrieval. CTC (Connectionist Temporal Classification) is a decoding method for sequence modeling that allows for end-to-end training without pre-aligning inputs and outputs.

[0071] S44 performs a five-way search in parallel.

[0072] The text semantic retrieval, the hot word inverted index retrieval, the audio intent retrieval, the visual entity retrieval, and the image-text-audio joint retrieval are executed in parallel to obtain the hit score of each retrieval. The correspondence between the retrieval paths is as follows: the text semantic retrieval is executed based on the text query features; the hot word inverted index retrieval is executed based on the hot word hypothesis; the audio intent retrieval is executed based on the audio query features; the visual entity retrieval is executed based on the visual query features; and the image-text-audio joint retrieval is executed based on the joint query features.

[0073] That is, R1 text semantics; R2 hot keyword inversion (model, SKU, etc.); R3 audio intent; R4 visual entities; R5 image-text-audio joint query. The normalized hit score of R2 is used as... Entering the subsequent integration, with other paths They will be scored together.

[0074] The five-way search is executed in parallel, which improves search efficiency and helps to cover proper nouns and multimodal questions, reducing the single-way missed recall rate.

[0075] S45, determine the overall score.

[0076] The overall score is determined based on the hit scores of each retrieval path and the cross-path consistency reward. The cross-path consistency reward is used to characterize the consistency of the same candidate knowledge fragment being hit simultaneously in multiple retrieval paths.

[0077] Specifically, for each retrieval path, the hit scores of each candidate knowledge fragment output by that retrieval path (i.e., the target retrieval path, which can be any retrieval path) are normalized using softmax with the temperature parameter τ to obtain the path weight of each candidate knowledge fragment corresponding to that retrieval path. That is, the cosine similarity between each independent query vector q and its retrieved knowledge block k is... The path weights are obtained by performing softmax on the similarity of each path using the temperature parameter τ. : ; in, τ represents the hit score output by this retrieval path, and τ is the temperature parameter. The path weight is used to represent the relative priority of each candidate knowledge fragment from the perspective of this retrieval path. Softmax allows high-scoring fragments to receive exponentially amplified path weights, while low-scoring fragments have their path weights suppressed, thus prioritizing the fragments most confident in each retrieval path within a limited knowledge injection window.

[0078] Then, for the same candidate knowledge fragment, the hit scores from each path are summed according to their respective path weights, and a cross-path consistency reward is added to obtain the comprehensive score S. That is, the hit scores from each retrieval are multiplied by their corresponding path weights, summed, and then added to the cross-path consistency reward to obtain the comprehensive score S of the candidate knowledge fragment. ; Where C represents the consistency ratio of simultaneous hits across multiple paths (i.e., the cross-path consistency reward item), and λ is the consistency weight. The cross-path consistency reward mechanism adds extra rewards to candidate segments that are simultaneously hit by multiple retrievals, which is equivalent to adding a "cooperative confirmation" signal on the basis of weighted fusion, preventing high-scoring segments generated by noise in one retrieval path from misleading the results. Multi-path consensus is considered a higher measure of knowledge reliability.

[0079] S46, Select Top-K knowledge fragments.

[0080] The top-K knowledge segments are selected in descending order of their comprehensive scores S. The value of K in the top-K can be preset according to the actual application scenario and hardware conditions; for example, the top 3 to 5 knowledge segments with the highest comprehensive scores can be selected.

[0081] S47, encoded as a knowledge token and with added boundary markers.

[0082] The selected Top-K knowledge fragments are encoded into knowledge tokens of the same dimension as the audio-visual large model, and a knowledge start token (e.g., ...) is added to the beginning and end of each knowledge token.<RAG_START> ) and knowledge end tokens (e.g.<RAG_END> The dimension of the knowledge token is consistent with the feature dimension of the base model, so it can be directly input into the Transformer for subsequent calculations.

[0083] It should be noted that in this embodiment, the full-duplex adaptation scheduling module is set as a scheduling logic independent of the inference kernel, separated from the model weights, and only enabled during the inference phase. By separating the scheduling logic from the inference kernel, retrieval I / O and GPU inference steps can be executed in parallel, avoiding serial waiting; at the same time, when a failure occurs on the retrieval side, the scheduling module can isolate the fault, allowing the system to still follow the unknowing chatter or listening path. If all scheduling logic is written into the same process as the inference kernel, the retrieval I / O and GPU steps are prone to serialization, and a failure on the retrieval side will cause the entire inference process to crash, resulting in poor system robustness. This application improves the reliability and maintainability of the system through the architecture design of separating scheduling and weights.

[0084] S35, append the knowledge token to the end of the first user sequence, reuse the user-side key value cache, perform second-stage reasoning on the appended second user sequence, and generate response content and knowledge summary.

[0085] In some embodiments, refer to Figure 4 As shown, in the second stage, the knowledge token (including start and end markers) generated in step S4 is first appended to the end of the first user sequence S1 to form the second user sequence S2. Then, the user-side key-value cache cached in step S3 is reused. ), and perform the second phase of inference on the appended second user sequence.

[0086] In an optional implementation, the second-stage reasoning on the appended second user sequence to generate response content and knowledge summary includes: Input the second user sequence into the Transformer of the audio and video large model; The user-side key-value cache is reused to calculate self-attention for the knowledge token, and cross-attention within the block is calculated for the knowledge token and the user-side key-value cache. After completing the attention calculation on the knowledge token, autoregressive decoding is performed to generate a response token sequence; Decode the tokens in the response token sequence, excluding the digest token, into text, and output this as the response content. In the response token sequence, when the audio and video big model outputs a summary token, the text content corresponding to the summary token is output as the knowledge summary; The knowledge digest has a length of 3 to 8 tokens.

[0087] In some embodiments, refer to Figure 4 As shown, the second-stage reasoning includes the following sub-steps.

[0088] S51, input the second user sequence into the Transformer of the audio and video large model.

[0089] The second user sequence S2, with an appended knowledge token, is input into the Transformer in preparation for inference.

[0090] S52, reuse Computational knowledge attention.

[0091] Reuse the user-side key-value cache cached in step S3 ( The second stage calculates self-attention for the knowledge token and intra-block cross-attention between the knowledge token and the user-side key-value cache. In other words, the second stage primarily incrementally calculates knowledge-side self-attention and intra-block cross-attention between the user and knowledge; the user-side key-value cache is not repeatedly calculated, thus significantly reducing computational load. Within a block, the user token and knowledge token are bidirectionally visible (intra-block bidirectional attention), achieving deep fusion of knowledge and user multimodal features within the same block.

[0092] It should be noted that this embodiment employs a bidirectional attention masking method within the block, where both the user token and knowledge token are visible. This means the knowledge token can follow the user token, and vice versa. If a one-way masking method were used, where knowledge can only follow the user and the user cannot follow knowledge, the user's audiovisual representation could not be adjusted according to the knowledge, leading to misalignment between lip movements, facial expressions, and knowledge points, resulting in an interaction method closer to plain text customer service. This application, through a bidirectional attention mechanism within the block, ensures that the digital human's audio-visual generation remains consistent with the knowledge content.

[0093] S53, autoregressive decoding generates a response token sequence.

[0094] After completing the attention calculation on the knowledge tokens, autoregressive decoding is performed to generate a response token sequence. The response token sequence includes ordinary tokens corresponding to the response content, as well as possible digest tokens.

[0095] S54, output the reply content.

[0096] The tokens in the response token sequence, excluding the digest token, are decoded into text and output as the response content. This response content incorporates retrieved external knowledge and can be directly used to drive the speech and image generation of the digital human.

[0097] S55, Output knowledge summary.

[0098] In the response token sequence, when the audio / video large model outputs a digest token (e.g.<RAG_SUMMARY> When the knowledge digest is generated, the text content corresponding to the digest token is output as the knowledge digest. The knowledge digest is 3 to 8 tokens in length and is a highly condensed knowledge representation. The short digest design allows subsequent streaming units to reference historical knowledge with minimal memory overhead.

[0099] S36, release the full-text key-value cache of the current streaming unit after use, and write the summary key-value cache corresponding to the knowledge summary into the global history pool for subsequent streaming units to read.

[0100] In some embodiments, refer to Figure 4 and Figure 6 As shown, after completing the second stage of reasoning and generating the response content and knowledge summary, hierarchical key-value lifecycle management is executed. (Refer to...) Figure 6 As shown, this embodiment divides the key-value cache involved in the current streaming unit inference process into the following three categories: (1) User-side key-value cache.

[0101] This is a key-value cache corresponding to the first user sequence cached during the first stage of inference, i.e. Figure 6 China's "Phase One Output" As shown in the image. It can be reused within this block (for calculating the cross-attention between the knowledge token and the user sequence during the second-stage inference) and can also be used as part of the condition features to participate in the condition construction sent to the Performer.

[0102] (2) Temporary full-text key-value cache within the block.

[0103] This is a key-value cache corresponding to the full-text knowledge retrieved in this block, i.e. Figure 6 "Retrieved knowledge token" and " The temporary knowledge (KV) within the block is shown in the figure. Valid only within this block, used to calculate the self-attention of knowledge tokens and the cross-attention with the user side during the second-stage reasoning. For example... Figure 6 As shown, after this block ends, It is released immediately and not written to the global history pool. This "temporary full data within a block" management method avoids the long-term occupation of video memory for full-text retrieval.

[0104] (3) Summary key-value cache.

[0105] The knowledge summary described in step S55 (<RAG_SUMMARY> The corresponding key-value cache, i.e. Figure 6 middle" As shown in the "KV Persistent Knowledge Summary". Figure 6 As shown, After generation, it is written to a global persistent key-value cache pool (i.e., the global history pool) for subsequent streaming unit causal reading. The global history pool stores both the key-value pairs of the historical dialogue and the key-value pairs of the historical summary, for use in the next block loading.

[0106] like Figure 6 As shown, after the current 160ms streaming cell ends, Released, and The data is persisted to the global history pool. At the start of the next block, the model loads the historical dialogue key-value pairs from the global history pool. This allows subsequent streaming units to reference historical knowledge with minimal memory overhead.

[0107] Full-duplex adaptation scheduling module and hierarchical KV unit linkage: release (i.e., the full-text key-value cache of the current streaming unit), merge summaries of the same topic, and evict expired summaries. Through this linkage mechanism, while ensuring that subsequent streaming units can reference historical knowledge, the length of the global history pool is effectively controlled, avoiding the continuous growth of global key-value pairs under long dialogues.

[0108] In addition, the full-duplex adaptation scheduling module works with the hierarchical KV unit to write the retrieval results into a short-term cache in the streaming unit during the listening phase, so that the streaming unit in the subsequent question-and-answer phase can reuse them preferentially; when an interruption occurs, the summary key-value cache that has been written into the global history pool is retained, and the existing knowledge context is not cleared due to the interruption.

[0109] After writing the digest key-value cache to the global history pool, the process also includes maintaining the global history pool. Specifically: After the first stage of reasoning in subsequent streaming units is completed, the semantic similarity between the first user sequence of the current streaming unit and each summary key-value cache in the global history pool is calculated. When the semantic similarity exceeds a preset threshold, the corresponding summary key-value cache is read and participates in the second stage of reasoning in subsequent streaming units, thereby enabling historical knowledge to be referenced by the current streaming unit.

[0110] When the number of digest key-value caches in the global history pool reaches a preset threshold, the full-duplex adaptation scheduling module, in conjunction with the hierarchical KV unit, merges multiple digest key-value caches with semantic similarity exceeding a preset threshold into a single merged digest key-value cache (i.e., merges digests with similar topics), and removes digest key-value caches with citation counts below a preset threshold and existence times exceeding a preset time threshold from the global history pool (i.e., removes long-unused digests). By merging digests with similar topics and removing long-unused digests, the length of the global history pool can be controlled, preventing unlimited growth. After writing digests and dialogue tokens, the global history pool is used by the next block.

[0111] S37, the user-side key-value cache and the summary key-value cache are sent as conditional features to the Performer stream matching module of the audio and video big model to generate audio and video latent variables and decode and output them.

[0112] In some embodiments, refer to Figure 2 and Figure 4 As shown, after completing the second stage of inference and KV management, Thinker caches the user-side key-value pairs. ) and the digest key-value cache ( The conditional features are then sent to the Performer stream matching module of the audio-visual big model. The Performer stream matching module is a component in the audio-visual big model used to generate synchronized speech and image latent variables. After receiving the conditional features from Thinker, it performs stream matching to generate synchronized speech and image latent variables. Finally, the digital rendering output module decodes the audio-visual latent variables into output audio-visual data, driving the digital human to output speech and images aligned with the knowledge points.

[0113] Because the knowledge token and the user's multimodal features are fused with bidirectional attention within the same block, and the fused KV is used as a conditional feature to deliver to the Performer, the lip movements and facial expressions output by the digital human can be consistent with the content being explained, thus avoiding the problem of misaligned lip movements caused by knowledge remaining in the text branch.

[0114] Compared to existing technologies that rely on the baseline of writing all knowledge into the global key-value pair for an extended period and the baseline of decoupling the asynchronous thinking layer from the current interaction block, this application eliminates reliance on external interaction state machines during inference. Instead, the model's autoregressive output of retrieval identifiers determines whether to perform a retrieval, ensuring that the causal mask of the main block and the Thinker-Performer path are largely maintained. This prevents temporal behaviors such as listening and interrupting from being disrupted by external state machines. Furthermore, knowledge and user multimodal features are bidirectionally fused within the same block before being distributed to the Performer, making it easier for the digital human's lip movements, expressions, and explanations to remain consistent, avoiding the lip-syncing issues caused by knowledge remaining only in the text branch. Regarding memory control, this application employs a hierarchical key-value management rule of "releasing full-text key-value pairs within blocks and retaining only summaries between blocks," resulting in a reduction of over 85% in knowledge-based key-value usage compared to the full-text persistent baseline. This facilitates maintaining a streaming inference rhythm of approximately 160ms. Meanwhile, by reusing the user-side key-value pairs cached in the first stage, and only incrementally calculating knowledge-related attention in the second stage, the five-way retrieval can be controlled within approximately 30ms, with a single-block end-to-end latency of approximately 200ms. After retaining the summary in the global history pool, similar follow-up questions can reuse existing summaries without repeated retrieval, reducing the number of repeated retrievals by approximately 60%. Furthermore, this application separates the scheduling logic from the model weights, allowing the system to still use non-knowledge-based chat and listening paths even when a retrieval failure occurs, improving system robustness. The streaming units in the listening stage can pre-cachate retrieval results and retain the summary context when interrupted, further optimizing the full-duplex interactive experience. Five-way parallel retrieval is beneficial for covering proper nouns and multimodal questions, effectively reducing the missed recall rate of single-way retrieval.

[0115] Reference Figure 7 The diagram shown is a functional block diagram of the retrieval enhancement generation device according to an embodiment of this application.

[0116] In some embodiments, the retrieval enhancement generation device 70 is applied to a large audio-visual model with a streaming unit as the inference cycle. The streaming unit is the smallest time unit for streaming inference of the large audio-visual model and may include multiple functional modules composed of computer program segments. The computer programs of each program segment of the retrieval enhancement generation device 70 may be stored in the memory of an electronic device and executed by at least one processor to perform the retrieval enhancement generation function. Based on the function performed, it can be divided into multiple functional modules. The functional modules may include: an acquisition module 701, an encoding module 702, a first-stage inference module 703, a retrieval module 704, a second-stage inference module 705, a cache management module 706, and an output module 707. The term "module" in this application refers to a series of computer program segments that can be executed by at least one processor and perform a fixed function, and which are stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0117] The acquisition module 701 is used to acquire the user multimodal data of the current streaming unit.

[0118] The encoding module 702 is used to encode the user multimodal data into latent variables and construct the first user sequence of the current streaming unit.

[0119] The first-stage inference module 703 is used to perform first-stage inference on the first user sequence, output the retrieval identifier, and cache the user-side key value cache.

[0120] The retrieval module 704 is used to perform multi-path parallel retrieval based on the latent variables when the retrieval identifier indicates that a retrieval is required, to obtain knowledge fragments and encode them into knowledge tokens.

[0121] The second-stage reasoning module 705 is used to append the knowledge token to the end of the first user sequence, reuse the user-side key-value cache, perform second-stage reasoning on the appended second user sequence, and generate response content and knowledge summary.

[0122] The cache management module 706 is used to release the full-text key-value cache of the current streaming unit after it is used up, and write the summary key-value cache corresponding to the knowledge summary into the global history pool for subsequent streaming units to read.

[0123] The output module 707 is used to send the user-side key-value cache and the summary key-value cache as conditional features to the Performer stream matching module of the audio and video big model, generate audio and video latent variables, and decode and output them.

[0124] It should be understood that the various variations and specific embodiments of the search enhancement generation method provided in the above embodiments are also applicable to the search enhancement generation device of this embodiment. Through the foregoing detailed description of the search enhancement generation method, those skilled in the art can clearly understand the implementation method of the search enhancement generation device in this embodiment. For the sake of brevity, it will not be described in detail here.

[0125] See Figure 8 The diagram shown is a schematic representation of the structure of an electronic device according to an embodiment of this application. In a preferred embodiment of this application, the electronic device 8 includes a memory 81, at least one processor 82, and at least one communication bus 83.

[0126] Those skilled in the art should understand that Figure 8 The structure of the electronic device shown does not constitute a limitation of the embodiments of this application. It can be a bus structure or a star structure. The electronic device 8 may also include more or fewer other hardware or software than shown, or different component arrangements.

[0127] In some embodiments, the electronic device 8 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital processors, and embedded devices. The electronic device 8 may also include user equipment, which includes, but is not limited to, any electronic product capable of human-computer interaction with a user via a keyboard, mouse, remote control, touchpad, or voice control device, such as a personal computer, tablet computer, smartphone, or digital camera.

[0128] In the embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, computer-readable storage media, and electronic devices can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple components or modules may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices, components, or modules may be electrical, mechanical, or other forms.

[0129] The components described as separate parts may or may not be physically separate. The components shown as components may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the components can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each component can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0131] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drive, portable hard drive, read-only memory (ROM). Various media that can store program code, such as only memory, random access memory (RAM), magnetic disks or optical disks.

[0132] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0133] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0134] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A retrieval enhancement generation method, characterized in that, The method, applied to a large audio-visual model with a streaming unit as the inference cycle, wherein the streaming unit is the smallest time unit for streaming inference of the large audio-visual model, includes: Obtain the user multimodal data of the current streaming unit; The user multimodal data is encoded into latent variables to construct the first user sequence of the current streaming unit; Perform the first stage of inference on the first user sequence, output the retrieval identifier and cache the user-side key value cache; When the retrieval identifier indicates that a retrieval is required, a multi-path parallel retrieval is performed based on the latent variable to obtain knowledge fragments and encode them as knowledge tokens. The knowledge token is appended to the end of the first user sequence, the user-side key value cache is reused, and the second-stage reasoning is performed on the appended second user sequence to generate response content and knowledge summary; the second user sequence is the sequence formed by appending the knowledge token to the end of the first user sequence. Release the full-text key-value cache of the current streaming unit after use, and write the summary key-value cache corresponding to the knowledge summary into the global history pool for subsequent streaming units to read; The user-side key-value cache and the summary key-value cache are sent as conditional features to the Performer stream matching module of the audio and video big model to generate audio and video latent variables and decode and output them.

2. The retrieval enhancement generation method according to claim 1, characterized in that, The multi-path parallel retrieval includes text semantic retrieval, hot word inverted index retrieval, audio intent retrieval, image entity retrieval, and image-text-audio joint retrieval. The latent variables include text latent variables, audio latent variables, and visual latent variables. The multi-path parallel retrieval based on the latent variables includes: The text latent variables, audio latent variables, and visual latent variables are projected onto the same retrieval dimension using their respective MLPs, and then scaled using LayerNorm to obtain text query features, audio query features, and visual query features respectively. The text query features, audio query features, and visual query features are fused using a fusion MLP to obtain joint query features; Based on the textual latent variables, hot word hypotheses are obtained through CTC decoding; The text semantic retrieval, the hot word inverted index retrieval, the audio intent retrieval, the video entity retrieval, and the image-text-audio joint retrieval are executed in parallel to obtain the hit score of each retrieval path; wherein, the text semantic retrieval is performed based on the text query features, the hot word inverted index retrieval is performed based on the hot word hypothesis, the audio intent retrieval is performed based on the audio query features, the video entity retrieval is performed based on the visual query features, and the image-text-audio joint retrieval is performed based on the joint query features; The comprehensive score is determined based on the hit scores of each retrieval path and the cross-path consistency reward item; the cross-path consistency reward item is used to characterize the consistency of the same candidate knowledge fragment being hit simultaneously in multiple retrieval paths. The top-K knowledge segments are selected in descending order of the overall scores. The Top-K knowledge fragments are encoded into knowledge tokens of the same dimension as the audio-visual large model, and a knowledge start token and a knowledge end token are added to the beginning and end of the knowledge tokens, respectively.

3. The retrieval enhancement generation method according to claim 2, characterized in that, The determination of the overall score based on the hit score of each retrieval path and the cross-path consistency bonus includes: For each retrieval path, the hit scores of each candidate knowledge fragment output by the target path retrieval path are normalized by the temperature parameter softmax to obtain the path weight of each candidate knowledge fragment corresponding to the target path retrieval path; the target path retrieval path is any retrieval path; the path weight is used to represent the relative priority of each candidate knowledge fragment from the perspective of that retrieval path. For the same candidate knowledge segment, the hit scores of each retrieval path are multiplied by their corresponding path weights and summed, and then the cross-path consistency reward is added to obtain the comprehensive score of the same candidate knowledge segment.

4. The retrieval enhancement generation method according to claim 1, characterized in that, The step of performing first-stage inference on the first user sequence, outputting a retrieval identifier, and caching the user-side key value cache includes: Input the first user sequence into the Transformer of the audio and video big model, perform autoregressive inference, and generate at least one token; When the generated token contains a retrieval trigger token, the retrieval trigger token is output as the retrieval identifier; The key-value cache corresponding to the first user sequence generated during the first stage of inference is cached as the user-side key-value cache; The retrieval identifier is determined by the RAG-specific token output by the autoregressive output of the audio and video large model, and does not depend on the input of an external interactive state machine.

5. The retrieval enhancement generation method according to claim 1, characterized in that, The second-stage reasoning process for the appended second user sequence, generating response content and knowledge summary, includes: Input the second user sequence into the Transformer of the audio and video large model; The user-side key-value cache is reused to calculate self-attention for the knowledge token, and cross-attention within the block is calculated for the knowledge token and the user-side key-value cache. After completing the attention calculation on the knowledge token, autoregressive decoding is performed to generate a response token sequence; Decode the tokens in the response token sequence, excluding the digest token, into text, and output this as the response content. In the response token sequence, when the audio and video big model outputs a summary token, the text content corresponding to the summary token is output as the knowledge summary; The knowledge digest has a length of 3 to 8 tokens.

6. The retrieval enhancement generation method according to claim 1, characterized in that, After writing the summary key-value cache corresponding to the knowledge summary to the global history pool, the method further includes: After the first stage of inference of the subsequent streaming unit is completed, the semantic similarity between the first user sequence of the current streaming unit and each digest key-value cache in the global history pool is calculated; When the semantic similarity exceeds a preset threshold, the corresponding summary key value is cached and read, and participates in the second stage of reasoning of the subsequent streaming unit; When the number of summary key-value caches in the global history pool reaches a preset threshold, multiple summary key-value caches with semantic similarity exceeding a preset threshold are merged into a single merged summary key-value cache, and summary key-value caches with a reference count lower than a preset threshold and an existence time exceeding a preset time threshold are removed from the global history pool.

7. The retrieval enhancement generation method according to any one of claims 1 to 6, characterized in that, The retrieval identifier and the RAG-specific token used by the knowledge token are implemented by adding a retrieval trigger token, a knowledge start token, a knowledge end token, and a summary token to the text vocabulary. The audio-visual large model is pre-fine-tuned in the following ways: In the text vocabulary of the audio and video big model, a retrieval trigger token, a knowledge start token, a knowledge end token, and a summary token are added as RAG-specific tokens. The embedding of the new tokens is initialized with the mean of the general real word embedding. Freeze the audio and video encoder, audio and video variational autoencoder and the first 80% of the Transformer layers of the large audio and video model, and train only the top 2 to 3 layers of the Transformer and the embedding of the RAG special token. Construct training samples, each sample aligned to a single streaming unit, the training samples including a first-stage input sequence and a first-stage target token, the first-stage target token being the retrieval trigger token or an empty token; For the samples to be retrieved, a knowledge token sequence is appended after the input sequence in the first stage. The knowledge token sequence includes a knowledge start token, a knowledge token, and a knowledge end token. The second stage target includes a response token and the summary token. The training samples include idle scene samples, listening scene samples, question-and-answer scene samples, and interruption scene samples. During the training process, a course sampling strategy is adopted. The first 10% of the training steps are mainly composed of question-and-answer scene samples. In the middle 80% of the training steps, the four types of samples are relatively balanced: idle scene samples, listening scene samples, question-and-answer scene samples, and interruption scene samples. In the last 10% of the training steps, the proportion of idle scene samples is increased. During the training phase, behavioral labels and auxiliary classification heads are used to assist training. During deployment, the auxiliary classification head is removed, the behavioral labels are not input, and the decision to perform a retrieval is made solely based on whether the autoregression of the audio and video large model outputs the retrieval trigger token.

8. A retrieval enhancement generation device, characterized in that, An audio-visual large-scale model applied to which streaming units are used as inference cycles, wherein the streaming unit is the smallest time unit for streaming inference of the audio-visual large-scale model, the device includes: The acquisition module is used to acquire the user multimodal data of the current streaming unit; The encoding module is used to encode the user multimodal data into latent variables and construct the first user sequence of the current streaming unit; The first-stage reasoning module is used to perform first-stage reasoning on the first user sequence, output the retrieval identifier and cache the user-side key value cache. The retrieval module is used to perform multi-path parallel retrieval based on the latent variables when the retrieval identifier indicates that a retrieval is required, to obtain knowledge fragments and encode them into knowledge tokens; The second-stage reasoning module is used to append the knowledge token to the end of the first user sequence, reuse the user-side key-value cache, perform second-stage reasoning on the appended second user sequence, and generate response content and knowledge summary; the second user sequence is the sequence formed by appending the knowledge token to the end of the first user sequence. The cache management module is used to release the full-text key-value cache of the current streaming unit after use, and write the summary key-value cache corresponding to the knowledge summary into the global history pool for subsequent streaming units to read. The output module is used to send the user-side key-value cache and the summary key-value cache as conditional features to the Performer stream matching module of the audio and video big model, generate audio and video latent variables, and decode and output them.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the retrieval enhancement generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the retrieval enhancement generation method according to any one of claims 1 to 7.

Images